Develop Reference

Entity Framework Core 6 adding (or referencing when already exists) data in a Many-to-Many relationship

I have been struggling to get a many-to-many relationship up and running. I have followed the Microsoft's example for EF Core 5+. My Join table is getting created and I can add entries to all three tables named: SpecificationTest, SpecificationDeviceSpecificationTest and SpecificationDevice.
I am however having some difficulties populating the join table, when the 'SpecificationDevice' already exists.
The simplified entities look as follows:
public class SpecificationTest
{
[Key]
[DatabaseGenerated(DatabaseGeneratedOption.Identity)]
public int Id { get; set; }
public ICollection<SpecificationDevice> SpecificationDevices { get; set; }
}
public class SpecificationDevice
{
[Key]
[DatabaseGenerated(DatabaseGeneratedOption.Identity)]
public int Id { get; set; }
[Required]
public string Name { get; set; }
[Required]
public string SerialNumber { get; set; }
public ICollection<SpecificationTest> SpecificationTests { get; set; }
}
The join table is automatically created, but I have added the following in the OnModelCreating method as well, even though it is possibly redundant:
modelBuilder.Entity<SpecificationTest>()
.HasMany(a => a.SpecificationDevices)
.WithMany(a => a.SpecificationTests);
modelBuilder.Entity<SpecificationDevice>()
.HasMany(a => a.SpecificationTests)
.WithMany(a => a.SpecificationDevices);
The DbContext also contains the following DbSets:
public DbSet<SpecificationTest> SpecificationTests { get; set; }
public DbSet<SpecificationDevice> SpecificationDevices { get; set; }
By design, every SpecificationTest can have multiple SpecificationDevices. However, the SpecificationDevice table itself can only contain a single entry per unique device (Unique by combination of Serial and Name).
Adding a new SpecificationDevice works as expected. EF creates the entry in the SpecificatonDevice table, and correctly populates the SpecificationDeviceSpecificationTest join table as well.
The problem I am having is that if the SpecificationDevice being added already exists, I merely want to select the ID of that device and add it to the Join table. Currently it tries to add another entry with the ID I have selected, and then throws the following exception:
Error: System.InvalidOperationException: The instance of entity type 'SpecificationDevice' cannot be tracked because another instance with the key value '{Id: 1}' is already being tracked.
Here is the code I use to try and add the entities. Disclaimer, I use the Repository pattern with a UnitOfWork to do the work, but the code comes down to the same.
Note: The specificationTest.SpecificationDevices collection already contains data at this stage, where the SpecificationDevice can be a new device or an existing device.
var existingDevices = await dbContext.SpecificationDevices.AsNoTracking().ToArrayAsync();
foreach (var specDevice in specificationTest.SpecificationDevices)
{
var existingSpecDevice = existingDevices.FirstOrDefault(a => a.Name.Equals(specDevice.Name) && a.SerialNumber.Equals(specDevice.SerialNumber));
if (existingSpecDevice is not null)
{
specDevice.Id = existingSpecDevice.Id;
}
}
await dbContext.SpecificationTests.AddAsync(specificationTest);
await dbContext.SaveChangesAsync();
I would expect that because I selected the ID of the existing SpecificationDevice, that EF will automatically populate the Join table with the correct values (SpecificationTestId just created and the already existing SpecificationDeviceId) This is not the case and I get the exception in the block quote above.
The approach of selecting the ID above works perfectly when trying to add a one-to-one entry when the unique entry already exists, but not with Many-to-many relationship.
Am I doing something wrong? Am I missing something?

Why do I need to .Include() collections

I wrote a query which is pretty simple:
var locations = await _context.Locations
.Include(x => x.LocationsOfTheUsers)
.Include(x => x.Address)
.ThenInclude(x => x.County)
.Where(CalculateFilters(searchObj))
.ToListAsync(cancellationToken);
And everytime LocationsOfTheUsers were null so I decided to .Include(x => x.LocationsOfTheUsers) and I received results as expected but I'm not sure why do I have to include this collections since it's defined like this:
public class Location
{
public string Title { get; set; }
public long? RegionId { get; set; }
public Region Region { get; set; }
public long? AddressId { get; set; }
public Address Address { get; set; }
public long? CountyId { get; set; }
public County County { get; set; }
public ICollection<LocationsOfTheUsers> LocationsOfTheUsers { get; set; }
}
I thought this will be automatically included since it exist as ICollection in Location class.
So why is .Include() on LocationsOfTheUsers needed here?
Thanks guys
Cheers

In entity framework the non-virtual properties represent the columns of the tables, the virtual properties represent the relations between the tables (one-to-many, many-to-many, ...)
So your property should have been defined as:
public virtual ICollection<LocationsOfTheUsers> LocationsOfTheUsers { get; set; }
One of the slower parts of a database query is the transfer of the selected data from the database management system to your local process. Hence it is wise to limit the selected data to the values you actually plan to use.
If you have a one-to-many relation between Schools and Students, and you ask for School [10] you don't want automatically to fetch its 2000 Students.
Even if you would like to have "School [10] with all its Students" it would not be efficient to use Include to also fetch the Students. Every Student will have a foreign key SchoolId with a Value of [10]. If you would use Include you would transfer this foreign key 2000 times. What a waste!
When using entity framework always use Select to fetch data and select only the properties that you actually plan to use. Only use Include if you plan to change the included items.
This way you can separate your database table structure from the actual query. If your database structure changes, only the query changes, users of your query don't notice the internal changes.
Apart from better performance and more robustness against changes, readers of your code can more easily see what values are in their query.
Certainly don't use Include to save you some typing. Having to debug one error after future changes will take way more time than you will ever save by typeing include instead of Select
Finally: limit your data early in your process, so put the Where in front.
So your query should be:
var predicate = CalculateFilters(searchObj)
var queryLocations = dbContext.Locations
.Where(predicate)
.Select(location => new
{
// Select only the location properties that you plan to use
Id = location.Id,
Name = location.Name,
// Locations Of the users:
UserLocations = location.LocationsOfTheUsers
.Select(userLocation => new
{
// again: only the properties that you plan to use
Id = userLocation.Id,
...
// Not needed, you already know the value
// LocationId = userLocation.LocationId
})
.ToList(),
Address = new
{
Street = location.Address.Street,
PostCode = location.Addrress.PostCode,
...
County = location.Address.County.Name // if you only want one property
// or if you want more properties:
County = new
{
Name = location.Address.County.Name,
Abbr = location.Address.Count.Abbr,
...
}),
},
});

I thought this will be automatically included since it exist as ICollection in Location class.
Well, it's not automatically included, probably for performance reasons as the graph of related entities and their recursive child entities may be rather deep.
That's why you use eager loading to explicitly include the related entities that you want using the Include method.
The other option is to use lazy loading which means that the related entities are loaded as soon as you access the navigation property in your code, assuming some prerequisites are fulfilled and that the context is still around when this happens.
Please refer to the docs for more information.

I believe you are using EntityFrameworkCore. In EntityFramework (EF6), lazy loading is enabled by default, However, in EntityFrameworkCore, lazy loading related entities is handled by a separate package Microsoft.EntityFrameworkCore.Proxies.
To enable the behaviour you are seeking, install the above package and add the following code
protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
{
optionsBuilder.UseLazyLoadingProxies();
}
After this, the related entities will be loaded without the Include call.

Entity Framework Core Generating SQL With Ambiguous Column Names

I am using Entity Framework Core 2.2.6. I'm going to try and make this question concise and apologies in advance if it ends up being a wall of text.
The error I am seeing is an ambiguous column name in the SQL Entity Framework Core generates.
So my situation is this: I have two entities with a many-to-one relationship. The "parent" entity implements
an interface that has a property that is of type IChildEntity. Here are the interfaces:
public interface IParentEntity
{
IChildEntity Child { get; set; }
string Prop1 { get; set; }
string Prop2 { get; set; }
}
public interface IChildEntity
{
string ChildProp1 { get; set; }
string ChildProp2 { get; set; }
}
I am using ef core's fluent api and in order to set up the relationship between parent and child
I am using a concrete type of ChildEntity and defining a IChildEntity property to conform to the
interface and just passing things through to the concrete type:
public class ChildEntity : IChildEntity
{
public long ID {get; set;}
public string ChildProp1 { get; set; }
public string ChildProp2 { get; set; }
}
public class ParentEntity : IParentEntity
{
public long ID { get; set; }
public string Prop1 { get; set; }
public string Prop2 { get; set; }
public long ChildID { get; set; }
// Navigation property so EF Core can create the relationship
public ChildEntity MappedChild { get; private set; }
// this is to adhere to the interface
// just pass this through to the backing concrete instance
[NotMapped]
public IChildEntity Child
{
get => MappedChild;
set => MappedChild = (ChildEntity)value;
}
}
Then in OnModelCreating I set up the relationship like so:
protected override void OnModelCreating(ModelBuilder modelBuilder)
{
modelBuilder.Entity<ParentEntity>()
.HasOne(e => e.MappedChild)
.WithMany()
.HasForeignKey(e => e.ChildID);
}
This works and the relationship gets set up as expected, however I am finding when I do a query it can generate
some SQL that can result in an ambigous column error in some database engines. Here is the example query:
MyContext.ParentEntity
.Include(p => p.MappedChild)
.Where(p => p.Prop1.Equals("somestring")
.FirstOrDefault()
The SQL that gets generated is similar to:
SELECT p."ID", p."ChildID", p."Prop1", p."Prop1", "p.MappedChild"."ID", "p.MappedChild"."ChildProp1", "p.MappedChild"."ChildProp2"
FROM "ParentEntity" AS p
INNER JOIN "ChildEntity" AS "p.MappedChild" ON p."ChildID" = "p.MappedChild"."ID"
WHERE p."Prop1" = 'somestring'
ORDER BY "p.MappedChild"."ID"
LIMIT 1
The problem here is we are selecting two columns with the name ID and not aliasing. Some databases will be ok with this
but some will not. A work around I can do for this is to do two separate queries to get the entity and the child entity:
var parent = MyContext.ParentEntity
.Where(p => p.Prop1.Equals("somestring")
.FirstOrDefault()
MyContext.Entry(parent).Reference(p => s.MappedChild).Load();
But this is less than ideal since it does multiple queries and is a bit less elegant than just using Include()
Because this seems like such a common use case and I couldn't find any bug reports against EF Core for this type of
behavior it is my suspicion that I am doing something wrong here that is resulting in EFCore not aliasing column names
for this type of query. I was thinking it could be the bit of trickery I have to do to ensure my entity implements it's interface
(this is something I can't due to constraints in the codebase and other integrations) but the more I look at it the less likely that
seems to me since we are directly dealing with the "mapped" property in EF related code and it's completely unaware of the interface.
My questions are - can anyone see something in my implementation that would cause this? Could anyone
suggest a better workaround than what I have here? Any advice here would be appreciated. Thanks much.

This is an old Entity framework bug with the Oracle company products bug including the MySQL database and Oracle database (12.1 and older).
I see the
ORA-00918: column ambiguously defined
error mostly when:
Selecting one entity with including parent entity.
Selecting one entity with value object own one command
This error appears when using Find, First, FirstOrDefault, Last, Single and all single entity selector commands.
I tested many solutions and check generated sql statement to find out a very unique way without any performance overhead:
// This the way of getting one entity from oracle 12.1 without throwing Oracle exception => ORA-00918: column ambiguously defined without any extra overhead
var entities = await dbSet.Where(x => x.Id == id).Take(1).ToListAsync();
var entity = entities.FirstOrDefault();
Another Sample:
var entities = await dbSet.OrderByDescending(x => x.Id).Take(1).ToListAsync();
var entity = entities.FirstOrDefault();
At the end of your IQueryable Linq add Take(1) and get all with .ToList() or .ToListAsync() to execute the statement and fetch a list with one record. Then use Enumerable Single Entity Selector to change the list to an entity.
That’s all.

Null returned when trying to access child object

I am in the process of migrating to EF6 from Linq To Sql, and I have the autogenerated object
public partial class PCU
{
[System.Diagnostics.CodeAnalysis.SuppressMessage("Microsoft.Usage", "CA2214:DoNotCallOverridableMethodsInConstructors")]
public PCU()
{
this.PUs = new HashSet<PU>();
}
public int ID { get; set; }
public int FileNumberID { get; set; }
public Nullable<int> PartnerID { get; set; }
public virtual Company Company { get; set; }
public virtual File File { get; set; }
[System.Diagnostics.CodeAnalysis.SuppressMessage("Microsoft.Usage", "CA2227:CollectionPropertiesShouldBeReadOnly")]
public virtual ICollection<PU> PUs { get; set; }
}
where PartnerID is the Foreign key for company
when I call:
var company = dc.Set<PCU>().FirstOrDefault(c => c.FileNumber == fileNumber).Company;
I get a Null object, however if I call:
var company = dc.Set<PCU>().Where(c => c.FileNumber == fileNumber).Select(x => x.Company).First();
It returns the company object as expected. I have both LazyLoading and ProxyCreation enabled.
I understand I could use:
var company = dc.Set<PCU>().Include(x => x.Company).FirstOrDefault(c => c.FileNumber == fileNumber).Company;
however, as this is existing code, and I have the same problem for hundreds of different objects, this will mean massive amounts of changes. Is there an easier way to achieve this?

At first it indeed looks like:
dc.Set<PCU>().FirstOrDefault(c => c.FileNumber == fileNumber).Company
is similar to:
dc.Set<PCU>().Where(c => c.FileNumber == fileNumber).Select(x => x.Company).First()
but in case the foreign key 'Company' is null while using 'FirstOrDefault', returning 'Company' will obviously return null.
The second case, selects a valid 'Company' FK from the result set which was created by the 'Where' condition, and returns the first one from that set, this is why the 'Where' query returns a 'Company'.
If you don't wish to alter existing code, it seems to me that the best solution for you will be to actually see why you have null foreign keys in your database in the first place.
If it's the way its supposed to be (e.g., a null 'Company' entry could exist) then you'll have to take it into account in your queries, hence changing them to return only existing 'Company' entries.
Edit: I take it back, I missed the 'LazyLoading enabled' part 🤔

As a follow up, I believe the cause of the error is the name of the ForeignKey (PartnerID), and if it were named "CompanyID" it would work fine.
I have had to bite the bullet, and had to implement
var company = dc.Set<PCU>().Include(x => x.Company).FirstOrDefault(c => c.FileNumber == fileNumber).Company;
where neccesary. There does not seem to be another workaround, except for renaming the columns in my DB (which I can't do).

EntityFramework load / update Entities

i am struggeling for a while now to understand how EF loads / updates entities.
First of all i wanna explain what my app (WPF) is about. I am developing
an application where users can store Todo Items in Categories, these categories are predefined by the application. Each user can read all items but can only delete / update his own items. It's a multiuser system, means the application is running multiple times in the network accessing the same sql server database.
When a user is adding/deleting/updating items the UI on all the other running apps has to update.
My model looks like this:
public class Category
{
public int Id { get; set; }
public string Name { get; set; }
public List<Todo> Todos { get; set; }
}
public class Todo
{
public int Id { get; set; }
public string Content { get; set; }
public DateTime LastUpdate { get; set; }
public string Owner { get; set; }
public Category Category { get; set; }
public List<Info> Infos { get; set; }
}
public class Info
{
public int Id { get; set; }
public string Value { get; set; }
public Todo Todo { get; set; }
}
I am making the inital load like this, which works fine:
Context.dbsCategories.Where(c => c.Id == id).Include(c => c.Todos.Select(t => t.Infos)).FirstOrDefault();
Now i was trying to load only the Todos which are from the current user therefore i tried this:
Context.dbsCategories.Where(c => c.Id == id).Include(c => c.Todos.Where(t => t.Owner == Settings.User).Select(t => t.Infos)).FirstOrDefault();
This does not work because it's not possible to filter within include, so I tried this:
var cat = Context.dbsCategories.Where(c => c.Id == id).FirstOrDefault();
Context.dbsTodos.Where(t => t.Category.Id == cat.Id && t.Owner == Settings.User).Include(t=>t.Infos);
After executing the second line where i look for the Todo Items, these Items were automatically added to cat's Todos collection. Why? I would have expected that i have to add them manually to cat's Todos collection.
Just for my understanding what is EF doing here exactly?
Now to my main problem -> the synchronization of the data between database and client. I am using a long running Context which lives as long as the application is running to save changes to the database which are made on owned items. The user does not have the possibility to manipulate / delete data from other users this is guarantee by the user interface.
To synchronize the data i build this Synch Method which will run every 10 second, right now it's triggere manually.
Thats my synchronization Code, which only synchronizes Items to the client that do not belong to it.
private async Task Synchronize()
{
using (var ctx = new Context())
{
var database = ctx.dbsTodos().Where(x => x.Owner != Settings.User).Select(t => t.Infos).AsNoTracking();
var loaded = Context.dbsTodos.Local.Where(x => x.Owner != Settings.User);
//In local context but not in database anymore -> Detachen
foreach (var detach in loaded.Except(database, new TodoIdComparer()).ToList())
{
Context.ObjectContext.Detach(detach);
Log.Debug(this, $"Item {detach} detached");
}
//In database and local context -> Check Timestamp -> Update
foreach (var update in loaded.Intersect(database, new TodoIdTimeStampComparer()))
{
await Context.Entry(update).ReloadAsync();
Log.Debug(this, $"Item {update} updated");
}
//In database but not in local context -> Attach
foreach (var attach in database.ToList().Except(loaded, new TodoIdComparer()))
{
Context.dbsTodos().Attach(attach);
Log.Debug(this, $"Item {attach} attached");
}
}
}
I am having following problems / issues of unknow origin with it:
Detaching deleted Items seems to work, right now i am not sure if only the Todo Items are detached or also the Infos.
Updating Items works only for the TodoItem itsself, its not reloading the Infos within? How can i reload the whole entity with all it's relations?
I am thankful for every help on this, even if you are saying it's all wrong what i am doing here!
Attaching new Items and Infos does not work so far? What am i doing wrong here?
Is this the right approach to synchronize data between client and database?
What am i doing wrong here? Is there any "How to Sync" Tutorial? I have not found anything helpful so far?
Thanks!

My, you do like to deviate from entity framework code-first conventions, do you?
(1) Incorrect class definitions
The relations between your tables are Lists, instead of ICollections, they are not declared virtual and you forgot to declare the foreign key
There is a one-to-many relation between Todo and Category: every Todo belongs to exactly one Category (using a foreign key), every Category has zero or more Todos.
You choose to give Category a property:
List<Todo> Todos {get; set;}
Are you sure that category.Todos[4] has a defined meaning?
What would category.Todos.Insert(4, new Todo()) mean?
Better stick to an interface where you can't use functions that have no proper meaning in your database: use ICollection<Todo> Todos {get; set;}. This way you'll have only access to functions that Entity Framework can translate to SQL.
Besides, a query will probably be faster: you give entity framework the possibility to query the data in its most efficient way, instead of forcing it to put the result into a List.
In entity framework the columns of a table are represented by non-virtual properties; the virtual properties represent the relations between the tables (one-to-many, many-to-many)
public class Category
{
public int Id { get; set; }
public string Name { get; set; }
... // other properties
// every Category has zero or more Todos (one-to-many)
public virtual ICollection<Todo> Todos { get; set; }
}
public class Todo
{
public int Id { get; set; }
public string Content { get; set; }
... // other properties
// every Todo belongs to exactly one Category, using foreign key
public int CategoryId { get; set }
public virtual Category Category { get; set; }
// every Todo has zero or more Infos:
public virtual ICollection<Info> Infos { get; set; }
}
You'll probably guess Info by now:
public class Info
{
public int Id { get; set; }
public string Value { get; set; }
... // other properties
// every info belongs to exactly one Todo, using foreign key
public int TodoId {get; set;}
public virtual Todo Todo { get; set; }
}
Three major improvements:
ICollections instead of Lists
ICollections are virtual, because it is not a real column in your table,
foreign key definitions non-virtual: they are real columns in your tables.
(2) Use Select instead of Include
One of the slower parts of a database query is the transport of the selected data from the Database Management System to your local process. Hence it is wise to limit the amount of transported data.
Suppose Category with Id [4] has a thousand Todos. Every Todo of this Category will have a foreign key with a value 4. So this same value 4 will be transported 1001 times. What a waste of processing power!
In entity framework use Select instead of Include to query data and select only the properties you actually plan to use. Only use Include if you plan to update the Selected data.
Give me all Categories that ... with their Todos that ...
var results = dbContext.Categories
.Where(category => ...)
.Select(category => new
{
// only select properties that you plan to use
Id = category.Id,
Name = category.Name,
...
Todos = category.Todos
.Where(todo => ...) // only if you don't want all Todos
.Select(todo => new
{
// again, select only the properties you'll plan to use
Id = todo.Id,
...
// not needed, you know the value:
// CategoryId = todo.CategoryId,
// only if you also want some infos:
Infos = todo.Infos
.Select(info => ....) // you know the drill by now
.ToList(),
})
.ToList(),
});
(3) Don't keep DbContext alive for such a long time!
Another problem is that you keep your DbContext open for quite some time. This is not how a dbContext was meant. If your database changes between your query and your update, you'll have troubles. I can hardly imagine that you query so much data that you need to optimize it by keeping your dbContext alive. Even if you query a lot of data, the display of this huge amount of data would be the bottle-neck, not the database query.
Better fetch the data once, dispose the DbContext, and when updating fetch the data again, update the changed properties and SaveChanges.
fetch data:
RepositoryCategory FetchCategory(int categoryId)
{
using (var dbContext = new MyDbContext())
{
return dbContext.Categories.Where(category => category.Id == categoryId)
.Select(category => new RepositoryCategory
{
... // see above
})
.FirstOrDefault();
}
}
Yes, you'll need an extra class RepositoryCategory for this. The advantage is, that you hide that you fetched your data from a database. Your code would hardly change if you'd fetch your data from a CSV-file, or from the internet. This is way better testable, and also way better maintainable: if the Category table in your database changes, users of your RepositoryCategory won't notice it.
Consider creating a special namespace for the data you fetch from your database. This way you can name the fetched Category still Category, instead of RepositoryCategory. You even hide better where you fetched your data from.
Back to your question
You wrote:
Now i was trying to load only the Todos which are from the current user
After the previous improvements, this will be easy:
string owner = Settings.User; // or something similar
var result = dbContext.Todos.Where(todo => todo.Owner == owner)
.Select(todo => new
{
// properties you need
})