I've got two entity classes User and Connection, and User has a collection of Connection.
class User
{
public string username {get; set;}
public ICollection<Connection> Connections {get; set;}
}
class Connection
{
public string ConnectionId {get; set;}
public string RoomName {get; set;}
}
I've got this SQL Query which retrieves a list of connections for a user
// this code can be moved to the database as a stored procedure or view
var sql = #"SELECT Users.UserName, Users.UserPix,
Connections.ConnectionId, Connections.RoomName, Connections.DateCreated
FROM Users CROSS JOIN Connections
WHERE (Users.UserName = #p0) AND (Connections.RoomName = #p1)";
return _context.Users.SqlQuery(sql, username, roomName).FirstOrDefault();
It returns a user object with an empty connection list, rather than populating the connections with data returned from the database.
I've tried replacing cross-join with inner-join but still the same result
I don't know how else to modify the sql query so that the actual data is returned. How can I do this or is there something I'm missing?
Since you aren't doing anything special here (there aren't even any projections) you can just return the entities directly from the context, including the navigation properties you want with them:
e.g. given these entities:
class User
{
public string username { get; set; }
public ICollection<Connection> Connections { get; set; }
}
class Connection
{
public string ConnectionId { get; set; }
public string RoomName { get; set; }
}
A one-to-many relationship should exist in your database between User -> Connection
In order to get all connections for a particular user, or all connections for all users, or any combination of queries you can think of with filters/aggregates etc on users/connections whilst preserving the relationship you can just use .Include() from QueryableExtensions in System.Data.Entity
Signature is as so if you are interested:
public static IQueryable<T> Include<T, TProperty>(this IQueryable<T> source, Expression<Func<T, TProperty>> path) where T : class
Which is a good thing about EF, you can eager load the child entities on demand without having to mess with the entity design or add new entity classes which don't contain the navigation properties
So basically it boils down to this:
using(YourDbContext context = new YourDbContext())
{
var query = context.Users.Include(user => user.Connections);
// Do stuff with query
}
Don't forget about deferred execution - if you don't know what deferred execution is, and how a query is built up in EF it's worth looking it up - the general rule of thumb is; the less queries that hit the database and the less crunching you do in C# (for database style operations like aggregates) the quicker your app is going to perform - so make sure you don't enumerate the results until you've fully built the query or you are going to be doing all the number crunching on the .NET side using LINQ to objects, not in SQL where it should be!
So calling .ToList() or enumerating the above query will get you the results
EF will translate anything you have done to the correct SQL dialect using the provider you have chosen (likely System.Data.SqlClient). You don't need to write any SQL...
Some other examples of queries you might do:
// Get me users called Fred including their connection details
context.Users.Include(x => x.Connections).Where(u => u.Username == "Fred")
// Get me users that are currently connected to "Game Room"
context.Users.Include(x => x.Connections).Where(u => u.Connections.Any(c => c.RoomName == "Game Room")
None of this requires you to write any SQL - put a SQL trace on when running these queries to see what EF does, and usually it will write queries better than you ever could :) (only sometimes it doesn't and it's usually when you are doing something silly)
Edit
Ok I see you are trying to filter both the user and the connections that come back over the wire, in that case you either need to explicitly load the navigation property as a separate query or use projection to do the filtering
e.g.
Explicit Loading
var user = context.Users.First(x => x.UserName == username);
context.Entry(user).Collection(x => x.Connections).Query().Where(x => x.RoomName == roomName).Load();
That does result in two queries
Projection
var usersConnections = context.Users
.Where(u => u.UserName == userName)
.Select(u => new
{
User = u,
Connections = u.Connections.Where(c => c.RoomName == roomName)
});
This results in an anonymous type which contains a User property and a Connections property. You can always project into a known type too if you need to send this over some sort of domain boundary
This will be a single query against the datasource
To do this I first retrieve/check if the user exist, then I retrieve their connections and add it to the list of connection for that user
public User GetUserAndConnections(string username, string roomname)
{
var user = _context.Users.Find(username);
if (user != null)
{
var connections =
_context.Users.Where(u => u.UserName == username)
.SelectMany(x => x.Connections.Where(p => p.RoomName == roomName))
.ToList();
user.AddExistingConnections(connections);
}
return user;
}
Related
I'm trying my hand at LINQ for the first time and just wanted to post a small question to make sure if this was the best way to go about it. I want a list of every value in a table. So far this is what I have, and it works, but is this the best way to go about collecting everything in a LINQ friendly way?
public static List<Table1> GetAllDatainTable()
{
List<Table1> Alldata = new List<Table1>();
using (var context = new EFContext())
{
Alldata = context.Tablename.ToList();
}
return Alldata;
}
For simple entities, that is an entity that has no references to other entities (navigation properties) your approach is essentially fine. It can be condensed down to:
public static List<Table1> GetAllDatainTable()
{
using (var context = new EFContext())
{
return context.Table1s.ToList();
}
}
However, in most real-world scenarios you are going to want to leverage things like navigation properties for the relationships between entities. I.e. an Order references a Customer with Address details, and contains OrderLines which each reference a Product, etc. Returning entities this way becomes problematic because any code that accepts the entities returned by a method like this should be getting either complete, or completable entities.
For instance if I have a method that returns an order, and I have various code that uses that order information: Some of that code might try to get info about the order's customer, other code might be interested in the products. EF supports lazy loading so that related data can be pulled if, and when needed, however that only works within the lifespan of the DbContext. A method like this disposes the DbContext so Lazy Loading is off the cards.
One option is to eager load everything:
using (var context = new EFContext())
{
var order = context.Orders
.Include(o => o.Customer)
.ThenInclude(c => c.Addresses)
.Include(o => o.OrderLines)
.ThenInclude(ol => ol.Product)
.Single(o => o.OrderId == orderId);
return order;
}
However, there are two drawbacks to this approach. Firstly, it means loading considerably more data every time we fetch an order. The consuming code may not care about the customer or order lines, but we've loaded it all anyways. Secondly, as systems evolve, new relationships may be introduced that older code won't necessarily be noticed to be updated to include leading to potential NullReferenceExceptions, bugs, or performance issues when more and more related data gets included. The view or whatever is initially consuming this entity may not expect to reference these new relationships, but once you start passing around entities to views, from views, and to other methods, any code accepting an entity should expect to rely on the fact that the entity is complete or can be made complete. It can be a nightmare to have an Order potentially loaded in various levels of "completeness" and code handling whether data is loaded or not. As a general recommendation, I advise not to pass entities around outside of the scope of the DbContext that loaded them.
The better solution is to leverage projection to populate view models from the entities suited to your code's consumption. WPF often utilizes the MVVM pattern, so this means using EF's Select method or Automapper's ProjectTo method to populate view models based each of your consumer's needs. When your code is working with ViewModels containing the data views and such need, then loading and populating entities as needed this allows you to produce far more efficient (fast) and resilient queries to get data out.
If I have a view that lists orders with a created date, customer name, and list of products /w quantities we define a view model for the view:
[Serializable]
public class OrderSummary
{
public int OrderId { get; set; }
public string OrderNumber { get; set; }
public DateTime CreatedAt { get; set; }
public string CustomerName { get; set; }
public ICollection<OrderLineSummary> OrderLines { get; set; } = new List<OrderLineSummary>();
}
[Serializable]
public class OrderLineSummary
{
public int OrderLineId { get; set; }
public int ProductId { get; set; }
public string ProductName { get; set; }
public int Quantity { get; set; }
}
then project the view models in the Linq query:
using (var context = new EFContext())
{
var orders = context.Orders
// add filters & such /w Where() / OrderBy() etc.
.Select(o => new OrderSummary
{
OrderId = o.OrderId,
OrderNumber = o.OrderNumber,
CreatedAt = o.CreatedAt,
CustomerName = o.Customer.Name,
OrderLines = o.OrderLines.Select( ol => new OrderLineSummary
{
OrderLineId = ol.OrderLineId,
ProductId = ol.Product.ProductId,
ProductName = ol.Product.Name,
Quantity = ol.Quantity
}).ToList()
}).ToList();
return orders;
}
Note that we don't need to worry about eager loading related entities, and if later down the road an order or customer or such gains new relationships, the above query will continue to work, only being updated if the new relationship information is useful for the view(s) it serves. It can compose a faster, less memory intensive query fetching fewer fields to be passed over the wire from the database to the application, and indexes can be employed to tune this even further for high-use queries.
Update:
Additional performance tips: Generally avoid methods like GetAll*() as a lowest common denominator method. Far too many performance issues I come across with methods like this are in the form of:
var ordersToShip = GetAllOrders()
.Where(o => o.OrderStatus == OrderStatus.Pending)
.ToList();
foreach(order in ordersToShip)
{
// do something that only needs order.OrderId.
}
Where GetAllOrders() returns List<Order> or IEnumerable<Order>. Sometimes there is code like GetAllOrders().Count() > 0 or such.
Code like this is extremely inefficient because GetAllOrders() fetches *all records from the database, only to load them into memory in the application to later be filtered down or counted etc.
If you're following a path to abstract away the EF DbContext and entities into a service / repository through methods then you should ensure that the service exposes methods to produce efficient queries, or forgo the abstraction and leverage the DbContext directly where data is needed.
var orderIdsToShip = context.Orders
.Where(o => o.OrderStatus == OrderStatus.Pending)
.Select(o => o.OrderId)
.ToList();
var customerOrderCount = context.Customer
.Where(c => c.CustomerId == customerId)
.Select(c => c.Orders.Count())
.Single();
EF is extremely powerful and when selected to service your application should be embraced as part of the application to give the maximum benefit. I recommend avoiding coding to abstract it away purely for the sake of abstraction unless you are looking to employ unit testing to isolate the dependency on data with mocks. In this case I recommend leveraging a unit of work wrapper for the DbContext and the Repository pattern leveraging IQueryable to make isolating business logic simple.
I've been trying to take advantage of a new way of creating many-to-many relationships - nice article about EF 5 many-to-many relationships.
The article states that you no longer need to define relation class and the framework does the job for you.
However, for a couple of hours now I've been struggling to add an existing entity to the collection of another entity.
My models
public record Bottle
{
[Key]
public int Id { get; set; }
[Required]
public string Username { get; set; }
// some other properties
public Collection<User> Owners { get; set; }
}
public record User
{
[Key]
public int Id { get; set; }
// some other properties
public Collection<Bottle> Bottles { get; set; }
}
Say that I want to add a new bottle to the database. I also know owners of that bottle. I had thought that this bit of code could work:
public async Task<int> AddBottle(BottleForAddition bottle)
{
var bottleEntity = mapper.Map<Bottle>(bottle);
bottleEntity.Owners = bottle
.OwnerIds // List<int>
.Select(id => new User { Id = id })
.ToCollection(); // my extension method
var createdEntity = await context.AddEntityAsync(bottleEntity);
await context.SaveChangesAsync();
return createdEntity.Entity.Id;
}
but sadly it does not work (BottleForAddition is DTO with almost the same properties).
I get this error:
Unable to create bottle (error: Microsoft.EntityFrameworkCore.DbUpdateException: An error occurred while updating the entries. See the inner exception for details.
Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 19: 'NOT NULL constraint failed: Users.Username'.
at Microsoft.Data.Sqlite.SqliteException.ThrowExceptionForRC(Int32 rc, sqlite3 db)
at Microsoft.Data.Sqlite.SqliteDataReader.NextResult()
at ...
So I came up with this
public async Task<int> AddBottle(BottleForAddition bottle)
{
var bottleEntity = mapper.Map<Bottle>(bottle);
bottleEntity.Owners = (await context.Users
.Where(u => bottle.OwnerIds.Contains(u.Id))
.ToListAsync())
.ToCollection();
var createdEntity = await context.AddEntityAsync(bottleEntity);
await context.SaveChangesAsync();
return createdEntity.Entity.Id;
}
That works but I have to fetch Users from the database.
Do you know about a better way how to deal with it?
The Users table in the database has a Username field does not allow NULL
You are creating new User entities from the OwnerIds which doesn't have Username value set
EF is trying to insert a new user to the Users table
Combining the pieces of information above, you'll get a clear picture why the error message says -
SQLite Error 19: 'NOT NULL constraint failed: Users.Username'.
Then comes the real question, why EF is trying to insert new users at all. Obviously, you created the User entities from the OwnerIds to add already existing users to the list, not to insert them.
Well, I'm assuming that the AddEntityAsync() method you are using (I'm not familiar with it) is an extension method, and inside it, you are using the DbContext.Add() or DbSet<TEntity>.Add() method. Even if that is no the case, apparently AddEntityAsync() at least works similarly as them.
The Add() method causes the entity in question (Bottle) and all it's related entities (Users) present in the entity-graph to be marked as Added. An entity marked as Added implies - This is a new entity and it will get inserted on the next SaveChanges call. Therefore, with your first approach, EF tried to insert the User entities you created. See details - DbSet<TEntity>.Add()
In your second approach, you fetched the existing User entities first. When you fetch existing entities using the DbContext, EF marks them as Unchanged. An entity marked as Unchanged implies - This entity already exists in the database and it might get updated on the next SaveChanges call. Therefore, in this case the Add method caused only the Bottle entity to be marked as Added and EF didn't try to re-insert any User entities you fetched.
As a general solution, in a disconnected scenario, when creating new entity with an entity-graph (with one or more related entities) use the Attach method instead. The Attach method causes any entity to be marked as Added only if it doesn't have the primary-key value set. Otherwise, the entity is marked as Unchanged. See details - DbSet<TEntity>.Attach()
Following is an example -
var bottleEntity = mapper.Map<Bottle>(bottle);
bottleEntity.Owners = bottle
.OwnerIds // List<int>
.Select(id => new User { Id = id })
.ToCollection(); // my extension method
await context.Bottles.Attach(bottleEntity);
await context.SaveChangesAsync();
Not related to the issue :
Also, since you are already using AutoMapper, if you define your BottleForAddition DTO as something like -
public class BottleForAddition
{
public int Id { get; set; }
public string Username { get; set; }
// some other properties
public Collection<int> Owners { get; set; } // the list of owner Id
}
then you will be able to configure/define your maps like -
this.CreateMap<BottleForAddition, Bottle>();
this.CreateMap<int, User>()
.ForMember(d => d.Id, opt => opt.MapFrom(s => s));
which could simplify the operation code like -
var bottleEntity = mapper.Map<Bottle>(bottle);
await context.Bottles.Attach(bottleEntity);
await context.SaveChangesAsync();
Fetching the Users is generally the correct course of action. This allows you to make the associations but also helps validate that the reference IDs passed from the client are valid. Fetching entities by ID is generally quite fast, so I'd consider avoiding async/await for this operation. async is suited for large or high-frequency operations where server responsiveness could be "hung up". Using it everywhere just leads to slower operations overall.
EF will want to use proxies for navigation properties both for lazy loading (not to be relied on as a crutch, but useful to avoid errors as a worst-case) as well as for change tracking.
public record Bottle
{
[Key]
public int Id { get; set; }
[Required]
public string Username { get; set; }
// some other properties
public virtual ICollection<User> Owners { get; set; } = new List<User>();
}
then in the applicable code...
var bottleEntity = mapper.Map<Bottle>(bottle);
var users = context.Users
.Where(u => bottle.OwnerIds.Contains(u.Id))
.ToList();
foreach(var user in users)
bottleEntity.Users.Add(user);
// Or since dealing with a new Entity could do this...
//((List<User>)bottleEntity.Users).AddRange(users);
await context.SaveChangesAsync();
return bottleEntity.Id;
It might be tempting to just create the users and attach them to the DbContext and much of the time this would work, except if there is ever the possibility that the DbContext might have been tracking an instance of any of those to-be-attached users, which will result in a runtime error that an entity with the same ID is already being tracked.
var bottleEntity = mapper.Map<Bottle>(bottle);
var proxyUsers = bottle.OwnerIds
.Select(x => new User { Id = x }).ToList();
foreach(var user in proxyUsers)
{
context.Users.Attach(user);
bottleEntity.Users.Add(user);
}
await context.SaveChangesAsync();
return bottleEntity.Id;
This requires either turning off all entity tracking or remember to always query entities with AsNoTracking which can lead to additional work and intermitted bugs appearing if this isn't adhered to consistently. To deal with possible tracked entities is a fair bit more work:
var bottleEntity = mapper.Map<Bottle>(bottle);
var proxyUsers = bottle.OwnerIds
.Select(x => new User { Id = x }).ToList();
var existingUsers = context.Users.Local
.Where(x => bottle.OwnerIds.Contains(x.Id)).ToList();
var neededProxyUsers = proxyUsers.Except(existingUsers, new UserIdComparer()).ToList();
foreach(var user in neededProxyUsers)
context.Users.Attach(user);
var users = neededProxyUsers.Union(existingUsers).ToList();
foreach(var user in users)
bottleEntity.Users.Add(user);
await context.SaveChangesAsync();
return bottleEntity.Id;
Any existing tracked entity needs to be found and referenced in place of an attached user reference. The other caveat of this approach is that the "proxy" users created for non-tracked entities are not complete user records so later code expecting to get User records from the DbContext could receive these attached proxy rows and result in things like null reference exceptions etc. for fields that were not populated.
Hence, fetching the references from the EF DbContext to get the relatable entities is generally the best/simplest option.
I wrote a query which is pretty simple:
var locations = await _context.Locations
.Include(x => x.LocationsOfTheUsers)
.Include(x => x.Address)
.ThenInclude(x => x.County)
.Where(CalculateFilters(searchObj))
.ToListAsync(cancellationToken);
And everytime LocationsOfTheUsers were null so I decided to .Include(x => x.LocationsOfTheUsers) and I received results as expected but I'm not sure why do I have to include this collections since it's defined like this:
public class Location
{
public string Title { get; set; }
public long? RegionId { get; set; }
public Region Region { get; set; }
public long? AddressId { get; set; }
public Address Address { get; set; }
public long? CountyId { get; set; }
public County County { get; set; }
public ICollection<LocationsOfTheUsers> LocationsOfTheUsers { get; set; }
}
I thought this will be automatically included since it exist as ICollection in Location class.
So why is .Include() on LocationsOfTheUsers needed here?
Thanks guys
Cheers
In entity framework the non-virtual properties represent the columns of the tables, the virtual properties represent the relations between the tables (one-to-many, many-to-many, ...)
So your property should have been defined as:
public virtual ICollection<LocationsOfTheUsers> LocationsOfTheUsers { get; set; }
One of the slower parts of a database query is the transfer of the selected data from the database management system to your local process. Hence it is wise to limit the selected data to the values you actually plan to use.
If you have a one-to-many relation between Schools and Students, and you ask for School [10] you don't want automatically to fetch its 2000 Students.
Even if you would like to have "School [10] with all its Students" it would not be efficient to use Include to also fetch the Students. Every Student will have a foreign key SchoolId with a Value of [10]. If you would use Include you would transfer this foreign key 2000 times. What a waste!
When using entity framework always use Select to fetch data and select only the properties that you actually plan to use. Only use Include if you plan to change the included items.
This way you can separate your database table structure from the actual query. If your database structure changes, only the query changes, users of your query don't notice the internal changes.
Apart from better performance and more robustness against changes, readers of your code can more easily see what values are in their query.
Certainly don't use Include to save you some typing. Having to debug one error after future changes will take way more time than you will ever save by typeing include instead of Select
Finally: limit your data early in your process, so put the Where in front.
So your query should be:
var predicate = CalculateFilters(searchObj)
var queryLocations = dbContext.Locations
.Where(predicate)
.Select(location => new
{
// Select only the location properties that you plan to use
Id = location.Id,
Name = location.Name,
// Locations Of the users:
UserLocations = location.LocationsOfTheUsers
.Select(userLocation => new
{
// again: only the properties that you plan to use
Id = userLocation.Id,
...
// Not needed, you already know the value
// LocationId = userLocation.LocationId
})
.ToList(),
Address = new
{
Street = location.Address.Street,
PostCode = location.Addrress.PostCode,
...
County = location.Address.County.Name // if you only want one property
// or if you want more properties:
County = new
{
Name = location.Address.County.Name,
Abbr = location.Address.Count.Abbr,
...
}),
},
});
I thought this will be automatically included since it exist as ICollection in Location class.
Well, it's not automatically included, probably for performance reasons as the graph of related entities and their recursive child entities may be rather deep.
That's why you use eager loading to explicitly include the related entities that you want using the Include method.
The other option is to use lazy loading which means that the related entities are loaded as soon as you access the navigation property in your code, assuming some prerequisites are fulfilled and that the context is still around when this happens.
Please refer to the docs for more information.
I believe you are using EntityFrameworkCore. In EntityFramework (EF6), lazy loading is enabled by default, However, in EntityFrameworkCore, lazy loading related entities is handled by a separate package Microsoft.EntityFrameworkCore.Proxies.
To enable the behaviour you are seeking, install the above package and add the following code
protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
{
optionsBuilder.UseLazyLoadingProxies();
}
After this, the related entities will be loaded without the Include call.
I was reading about query interceptors. I was disappointed because thats more like a filter instead of an interceptor. In other words you can eather include records or not include them. You are not able to modify records for instance.
If I want to create a query interceptor for my entity Users I could then do something like:
[QueryInterceptor("Users")] // apply to table users
public Expression<Func<User, bool>> UsersOnRead()
{
return cust => cust.IsDeleted == false;
}
What if I instead create the operation: NOTE IS VERY IMPORTANT TO HAVE THE OPERATION NAME JUST LIKE THE ENTITY NAME OTHERWISE IT WILL NOT WORK
[WebGet]
public IEnumerable<User> Users()
{
return this.CurrentDataSource.Users.Where(x=>x.IsDeleted==false);
}
Placing this method instead of the query interceptor makes my service behave exactly the same. Plus I have more power! Is taking this approach a better solution?
I played around a little more with this and one of the issues is navigation properties won't be filtered. Let's say you have an entity called SalesPeople that has a link into IEnumberable of Customers
If you do
[QueryInterceptor("Customers")] // only show active customers
public Expression<Func<Customers, bool>> ActiveCustomers()
{
return cust => cust.IsDeleted == false;
}
when you query your OData feed like WCFDataService.svc/SalesPeople?$expand=Customers the results set for Customers will still have the filter applied.
But this
[WebGet]
public IQueryable<Customers> Customers()
{
return this.CurrentDataSource.Customers.Where(x=>x.IsDeleted==false);
}
When running OData query like WCFDataService.svc/Customers you will have the filtered list on active customers, but when running this WCFDataService.svc/SalesPeople?$expand=Customers the results set for the Customers will include deleted customers.
I was writing a web app and learning entity framework along the way. I am curios if I have done something wrong though. I haven't been using dispose or using statements when querying.
Example of a Repository of mine:
public User GetUserById(int sessionId)
{
var user = (from u in db.Users where u.Id == sessionId select u).SingleOrDefault();
return user;
}
My User table I selected from would have foreign keys which would be associated with other tables. The cool thing, or so I thought, about entity is that I could return the entire User object (built by entity) which then would allow me to do something like this User.MyOtherTable.SomeColumn in my controller.
The problem is I relied on this and went about my merry way in my controller of grabbing the user information as well as utilizing information from the other table. I am now realizing that if I close that connection like the code below:
public User GetUserById(int sessionId)
{ using(db)
{
var user = (from u in db.Users where u.Id == sessionId select u).SingleOrDefault();
return user;
}
}
My controller no long has access to User.MyOtherTable.SomeColumn as this will be null. My true question is how important is it for me use dispose in my entity application?
I would strongly recommend that you use using statements, and also that you stop relying on lazy-loading.
Instead of selecting User objects with lazy-load properties, work out the full data you need and project that to a Model class, e.g.
public class UserWithOtherStuffModel
{
public int Id { get; set; }
public string Name { get; set; }
public string OtherStuff { get; set; }
}
Then in your method:
public UserWithOtherStuffModel GetUserById(int sessionId)
{
using(db)
{
var user = (from u in db.Users
where u.Id == sessionId
select new UserWithOtherStuffModel{
Id = u.Id,
Name = u.Name,
OtherStuff = u.MyOtherTable.SomeColumn
}).SingleOrDefault();
return user;
}
}
Not only does this mean you limit the number of database calls, but you have a data contract that isn't tied to your database model. If you move that column, you simply update the query to point to the new spot/name, the view can remain unchanged.
So when you dispose your context, you aren't going to be able to reference properties that are lazy loaded. You'll need to make your context live longer either by having the context disposed when the controller is disposed (I don't remember if MVC does this per request or not) or by controlling the lifetime of the context using some attributes on the Action.
Some examples on how to resolve this can be found in this question, which is pretty much the same as what you are asking here.