Prevent SELECT N+1 issue in Entity Framework query using Joins - c#

I'm trying to query something from an indirectly related entity into a single-purpose view model. Here's a repro of my entities:
public class Team {
[Key]
public int Id { get; set; }
public string Name { get; set; }
public List<Member> Members { get; set; }
}
public class Member {
[Key]
public int Id { get; set; }
public string Name { get; set; }
}
public class Pet {
[Key]
public int Id { get; set; }
public string Name { get; set; }
public Member Member { get; set; }
}
Each class is in a DbSet<T> in my database context.
This is the view model I want to construct from a query:
public class PetViewModel {
public string Name { get; set; }
public string TeamItIndirectlyBelongsTo { get; set; }
}
I do so with this query:
public PetViewModel[] QueryPetViewModel_1(string pattern) {
using (var context = new MyDbContext(connectionString)) {
return context.Pets
.Where(p => p.Name.Contains(pattern))
.ToArray()
.Select(p => new PetViewModel {
Name = p.Name,
TeamItIndirectlyBelongsTo = "TODO",
})
.ToArray();
}
}
But obviously there's still a "TODO" in there.
Gotcha: I can not change the entities at this moment, so I can't just include a List<Pet> property or a Team property on Member to help out. I want to fix things inside the query at the moment.
Here's my current solution:
public PetViewModel[] QueryPetViewModel_2(string pattern) {
using (var context = new MyDbContext(connectionString)) {
var petInfos = context.Pets
.Where(p => p.Name.Contains(pattern))
.Join(context.Members,
p => p.Member.Id,
m => m.Id,
(p, m) => new { Pet = p, Member = m }
)
.ToArray();
var result = new List<PetViewModel>();
foreach (var info in petInfos) {
var team = context.Teams
.SingleOrDefault(t => t.Members.Any(m => m.Id == info.Member.Id));
result.Add(new PetViewModel {
Name = info.Pet.Name,
TeamItIndirectlyBelongsTo = team?.Name,
});
}
return result.ToArray();
}
}
However, this has a "SELECT N+1" issue in there.
Is there a way to create just one EF query to get the desired result, without changing the entities?
PS. If you prefer a "plug and play" repro containing the above, see this gist.

You've made the things quite harder by not providing the necessary navigation properties, which as #Evk mentioned in the comments do not affect your database structure, but allow EF to supply the necessary joins when you write something like pet.Member.Team.Name (what you need here).
The additional problem with your model is that you don't have a navigation path neither from Team to Pet nor from Pet to Team since the "joining" entity Member has no navigation properties.
Still it's possible to get the information needed with a single query in some not so intuitive way by using the existing navigation properties and unusual join operator like this:
var result = (
from team in context.Teams
from member in team.Members
join pet in context.Pets on member.Id equals pet.Member.Id
where pet.Name.Contains(pattern)
select new PetViewModel
{
Name = pet.Name,
TeamItIndirectlyBelongsTo = team.Name
}).ToArray();

Related

Lambda expression used inside Include is not valid. Include not working

I am trying to include two relationships with the Include() method.
using (var db = new DBContext()) {
var result = db.ProduktKundeArtNummer
.Include(p => p.Kunde)
.Include(d => d.Produkt)
.ToList();
}
}
When I execute this snippet I get the following error message:
Lambda expression used inside Include is not valid.
I created my context by scaffolding my mysql database.
This is the class it is referring to:
public partial class ProduktKundeArtNummer {
[Key]
public int Produkt { get; set; }
[Key]
public int Kunde { get; set; }
[Required]
[StringLength(255)]
public string Artikelnummer { get; set; }
[ForeignKey(nameof(Kunde))]
[InverseProperty("ProduktKundeArtNummer")]
public virtual Kunde KundeNavigation { get; set; }
[ForeignKey(nameof(Produkt))]
[InverseProperty("ProduktKundeArtNummer")]
public virtual Produkt ProduktNavigation { get; set; }
}
I already googled a lot and as far as I know my snippet should work.
Does anyone know where I messed up?
Edit:
Using a string navigator works fine, but I would rather use the lambda expressions.
using (var db = new DBContext()) {
var result = db.ProduktKundeArtNummer
.Include("KundeNavigation")
.Include("ProduktNavigation")
.ToList();
}
}
You are trying to Include integer properties which are not a Navigation properties. Include is a declaration which navigation properties should be loaded with entity.
Probably you need this:
using (var db = new DBContext()) {
var result = db.ProduktKundeArtNummer
.Include(p => p.KundeNavigation)
.Include(d => d.ProduktNavigation)
.ToList();
}

LINQ GroupBy Aggregation with AutoMapper

Trying to get a query to work, but honestly not sure how (or if it's even possible) to go about it as everything I have tried hasn't worked.
Querying a total of 6 tables: Person, PersonVote, PersonCategory, Category, City, and FirstAdminDivision.
PersonVote is a user review table for people and contains a column called Vote that is a decimal accepting a value from 1-5 (5 being "best"). FirstAdminDivision would be synonymous with US states, like California. Person table has a column called CityId which is the foreign key to City. The other tables I believe are mostly self-explanatory so I won't comment unless needed.
My goal is create a query that returns a list of the "most popular" people which would be based on the average of all votes on the PersonVote table for a particular person. For instance, if a person has 3 votes and all 3 votes are "5" then they would be first in the list...don't really care about secondary ordering at this point...eg...like most votes in a tie would "win".
I have this working without AutoMapper, but I love AM's ability to do projection using the ProjectTo extension method as the code is very clean and readable and would prefer to use that approach if possible but haven't had any luck getting it to work.
Here is what I have that does work....so basically, I am trying to see if this is possible with ProjectTo instead of LINQ's Select method.
List<PersonModel> people = db.People
.GroupBy(x => x.PersonId)
.Select(x => new PersonModel
{
PersonId = x.FirstOrDefault().PersonId,
Name = x.FirstOrDefault().Name,
LocationDisplay = x.FirstOrDefault().City.Name + ", " + x.FirstOrDefault().City.FirstAdminDivision.Name,
AverageVote = x.FirstOrDefault().PersonVotes.Average(y => y.Vote),
Categories = x.FirstOrDefault().PersonCategories.Select(y => new CategoryModel
{
CategoryId = y.CategoryId,
Name = y.Category.Name
}).ToList()
})
.OrderByDescending(x => x.AverageVote)
.ToList();
By looking at your code sample I tried to determine what your models would be in order to setup an example. I only implemented using a few of the properties to show the functionality:
public class People
{
public int PeronId { get; set; }
public string Name { get; set; }
public City City { get; set; }
public IList<PersonVotes> PersonVoes { get; set; }
}
public class PersonVotes
{
public int Vote { get; set; }
}
public class City
{
public string Name { get; set; }
}
public class FirstAdminDivision
{
public string Name { get; set; }
}
public class PersonModel
{
public int PersonId { get; set; }
public string Name { get; set; }
public string LocationDisplay { get; set; }
public double AverageVote { get; set; }
}
To use the ProjectTo extension I then initialize AM through the static API:
Mapper.Initialize(cfg =>
{
cfg.CreateMap<IEnumerable<People>, PersonModel>()
.ForMember(
model => model.LocationDisplay,
conf => conf.MapFrom(p => p.FirstOrDefault().City.Name))
.ForMember(
model => model.AverageVote,
conf => conf.MapFrom(p => p.FirstOrDefault().PersonVoes.Average(votes => votes.Vote)));
});
So given the following object:
var people = new List<People>()
{
new People
{
PeronId = 1,
City = new City
{
Name = "XXXX"
},
PersonVoes = new List<PersonVotes>
{
new PersonVotes
{
Vote = 4
},
new PersonVotes
{
Vote = 3
}
}
}
};
I would then a have query:
var result = people
.GroupBy(p => p.PeronId)
.Select(peoples => peoples)
.AsQueryable()
.ProjectTo<PersonModel>();
I'm just using in memory objects so that is why I convert to IQueryable to use the ProjectTo extension method in AM.
I'm hoping this was what you're looking for. Cheers,
UPDATED FOR LINQ TO ENTITIES QUERY:
var result = db.People
.GroupBy(p => p.PersonId)
.ProjectTo<PersonModel>(base.ConfigProvider) // AM requires me to pass Mapping Provider here.
.OrderByDescending(x => x.AverageVote)
.ToList();

LINQ to Entities query error

I am encountered an error that I am not familier with. I tried to google with no success.
I wrote the following query where I am having this error.
The entity or complex type 'MyWebProject.Models.UserDetail' cannot be constructed in a LINQ to Entities query.
The query:
UsersContext db = new UsersContext();
var userdata = (from k in db.UserDetails
where k.UserId == WebSecurity.CurrentUserId
select new UserDetail()
{
FullName = k.FullName,
Email = k.Email,
About = k.About,
Link = k.Link,
UserSchool = new School()
{
SchoolId = k.UserSchool.SchoolId,
SchoolName = k.UserSchool.SchoolName
},
UserCourse = new Course()
{
CourseId=k.UserCourse.CourseId,
CourseName=k.UserCourse.CourseName
},
Country=k.Country
}).FirstOrDefault();
Class:
public class UserDetail
{
public int Id { get; set; }
public int UserId { get; set; }
public string FullName { get; set; }
public string Link { get; set; }
public bool? Verified { get; set; }
public string Email { get; set; }
public string About { get; set; }
public School UserSchool { get; set; }
public Course UserCourse { get; set; }
public string Country { get; set; }
}
public class School
{
public int SchoolId { get; set; }
public string SchoolName { get; set; }
public string Country { get; set; }
}
public class Course
{
public int CourseId { get; set; }
public string CourseName { get; set; }
public School School { get; set; }
}
Any idea what went wrong??
It looks like it is due to how you are creating the complex properties School and Course in the middle of the query. It would be better to select the User (remove the select transformation), then use navigation properties to access those objects instead of building them manually. The navigation are meant for this as long as you have the proper relations built with foreign keys.
UsersContext db = new UsersContext();
var userdata = (from k in db.UserDetails
where k.UserId == WebSecurity.CurrentUserId})
.FirstOrDefault();
// access navigation properties which will perform the joins on your behalf
// this also provides for lazy loading which would make it more effecient. (it wont load the school object until you need to access it)
userdata.School
userdata.Course
MSDN article about navigation properties: http://msdn.microsoft.com/en-us/library/vstudio/bb738520(v=vs.100).aspx
This should give you what you want. It will load your objects as part of the query (and not rely on lazy loading).
UsersContext db = new UsersContext();
var userdata = db.UserDetails.Include(x => x.UserSchool)
.Include(x => x.UserCourse)
.Include(x => x.Country)
.Where(x => x.UserId == WebSecurity.CurrentUserId)
.FirstOrDefault();
I think it's because your entity has the same name of the object you're trying to create. Try renaming the object you want to return back. If you want to return the same type as your entity try the eager loading with .Include("relationshipname") feature.
A great answer from #Yakimych is given below.
You cannot (and should not be able to) project onto a mapped entity. You can, however, project onto an annonymous type or onto a DTO:
public class ProductDTO
{
public string Name { get; set; }
// Other field you may need from the Product entity
}
And your method will return a List of DTO's.
public List<ProductDTO> GetProducts(int categoryID)
{
return (from p in db.Products
where p.CategoryID == categoryID
select new ProductDTO { Name = p.Name }).ToList();
}
Mapped entities in EF basically represent database tables. If you project onto a mapped entity, what you basically do is partially load an entity, which is not a valid state. EF won't have any clue how to e.g. handle an update of such an entity in the future (the default behaviour would be probably overwriting the non-loaded fields with nulls or whatever you'll have in your object). This would be a dangerous operation, since you would risk losing some of your data in the DB, therefore it is not allowed to partially load entities (or project onto mapped entities) in EF.
For more details please go to the following link:
The entity cannot be constructed in a LINQ to Entities query

IDbSetExtensions.AddOrUpdate and relationships

IDbSetExtensions.AddOrUpdate is meant to help write code that works the same whether the database is empty or populated. But linking objects needs different code. When the database is empty, objects don't have IDs yet and you link them by assigning the navigational property. When the objects already exist, however, navigational properties don't work and you need to set the foreign keys directly. Navigational properties do work for proxies in both cases, at the cost of forfeiting POCOs. Edit: Actually, proxies don't work when both entities are old.
This sample crashes in the second SaveChanges call, when EF tries to set CountryID to 0:
public class Country
{
public virtual int ID { get; set; }
public virtual string Name { get; set; }
}
public class Person
{
public virtual int ID { get; set; }
public virtual string Name { get; set; }
public virtual int CountryID { get; set; }
public virtual Country Country { get; set; }
}
public class Context : DbContext
{
public DbSet<Person> Person { get; set; }
public DbSet<Country> Country { get; set; }
}
class Program
{
static void Foo()
{
using (var db = new Context())
{
//var c = new Country();
var c = db.Country.Create();
c.Name = "usa";
db.Country.AddOrUpdate(x => x.Name, c);
//var p = new Person();
var p = db.Person.Create();
p.Name = "billg";
p.Country = c;
db.Person.AddOrUpdate(x => x.Name, p);
db.SaveChanges();
}
}
static void Main()
{
Database.SetInitializer<Context>(new DropCreateDatabaseAlways<Context>());
Foo();
Foo();
}
}
How is AddOrUpdate used?
IDbSetExtensions.AddOrUpdate is meant to help write code that works the same whether the database is empty or populated.
AddOrUpdate is meant to be used only in Seed method of code first migrations. It is not supposed to be used in normal code because it has big overhead and some limitations. Overhead is additional query to database and reflection. Limitation is that it checks only the main entity you are passing but not its relations. Each relation is supposed to be handled by separate call to AddOrUpdate:
static void Foo()
{
using (var db = new Context())
{
var c = new Country() {Name = "abc"};
db.Country.AddOrUpdate(x => x.Name, c);
var p = new Person()
{
Name = "me",
CountryID = c.ID,
Country = c
};
db.Person.AddOrUpdate(x => x.Name, p);
db.SaveChanges();
}
}

Can I create nested classes when using Linq-To-Entities?

I'm still learning Entity Framework and Linq-To-Entities, and I was wondering if a statement of this kind is possible:
using (var context = new MyEntities())
{
return (
from a in context.ModelSetA.Include("ModelB")
join c in context.ModelSetC on a.Id equals c.Id
join d in context.ModelSetD on a.Id equals d.Id
select new MyModelA()
{
Id = a.Id,
Name = a.Name,
ModelB = new MyModelB() { Id = a.ModelB.Id, Name = a.ModelB..Name },
ModelC = new MyModelC() { Id = c.Id, Name = c.Name },
ModelD = new MyModelD() { Id = d.Id, Name = d.Name }
}).FirstOrDefault();
}
I have to work with a pre-existing database structure, which is not very optimized, so I am unable to generate EF models without a lot of extra work. I thought it would be easy to simply create my own Models and map the data to them, but I keep getting the following error:
Unable to create a constant value of type 'MyNamespace.MyModelB'. Only
primitive types ('such as Int32, String, and Guid') are supported in
this context.
If I remove the mapping for ModelB, ModelC, and ModelD it runs correctly. Am I unable to create new nested classes with Linq-To-Entities? Or am I just writing this the wrong way?
What you have will work fine with POCOs (e.g., view models). Here's an example. You just can't construct entities this way.
Also, join is generally inappropriate for a L2E query. Use the entity navigation properties instead.
I have created your model (how I understand it) with EF 4.1 in a console application:
If you want to test it, add reference to EntityFramework.dll and paste the following into Program.cs (EF 4.1 creates DB automatically if you have SQL Server Express installed):
using System.Linq;
using System.Data.Entity;
namespace EFNestedProjection
{
// Entities
public class ModelA
{
public int Id { get; set; }
public string Name { get; set; }
public ModelB ModelB { get; set; }
}
public class ModelB
{
public int Id { get; set; }
public string Name { get; set; }
}
public class ModelC
{
public int Id { get; set; }
public string Name { get; set; }
}
public class ModelD
{
public int Id { get; set; }
public string Name { get; set; }
}
// Context
public class MyContext : DbContext
{
public DbSet<ModelA> ModelSetA { get; set; }
public DbSet<ModelB> ModelSetB { get; set; }
public DbSet<ModelC> ModelSetC { get; set; }
public DbSet<ModelD> ModelSetD { get; set; }
}
// ViewModels for projections, not entities
public class MyModelA
{
public int Id { get; set; }
public string Name { get; set; }
public MyModelB ModelB { get; set; }
public MyModelC ModelC { get; set; }
public MyModelD ModelD { get; set; }
}
public class MyModelB
{
public int Id { get; set; }
public string Name { get; set; }
}
public class MyModelC
{
public int Id { get; set; }
public string Name { get; set; }
}
public class MyModelD
{
public int Id { get; set; }
public string Name { get; set; }
}
class Program
{
static void Main(string[] args)
{
// Create some entities in DB
using (var ctx = new MyContext())
{
var modelA = new ModelA { Name = "ModelA" };
var modelB = new ModelB { Name = "ModelB" };
var modelC = new ModelC { Name = "ModelC" };
var modelD = new ModelD { Name = "ModelD" };
modelA.ModelB = modelB;
ctx.ModelSetA.Add(modelA);
ctx.ModelSetB.Add(modelB);
ctx.ModelSetC.Add(modelC);
ctx.ModelSetD.Add(modelD);
ctx.SaveChanges();
}
// Run query
using (var ctx = new MyContext())
{
var result = (
from a in ctx.ModelSetA.Include("ModelB")
join c in ctx.ModelSetC on a.Id equals c.Id
join d in ctx.ModelSetD on a.Id equals d.Id
select new MyModelA()
{
Id = a.Id,
Name = a.Name,
ModelB = new MyModelB() {
Id = a.ModelB.Id, Name = a.ModelB.Name },
ModelC = new MyModelC() {
Id = c.Id, Name = c.Name },
ModelD = new MyModelD() {
Id = d.Id, Name = d.Name }
}).FirstOrDefault();
// No exception here
}
}
}
}
This works without problems. (I have also recreated the model from the database (which EF 4.1 had created) in EF 4.0: It works as well. Not surprising since EF 4.1 doesn't change anything in LINQ to Entities.)
Now the question is why you get an exception? My guess is that there is some important difference in your Models or ViewModels or your query compared to the simple model above which is not visible in your code example in the question.
But the general result is: Projections into nested (non-entity) classes work. (I'm using it in many situations, even with nested collections.) Answer to your question title is: Yes.
What Craig posted does not seem to work for nested entities. Craig, if I am misunderstood what you posted, please correct me.
Here is the workaround I came up with that does work:
using (var context = new MyEntities())
{
var x = (
from a in context.ModelSetA.Include("ModelB")
join c in context.ModelSetC on a.Id equals c.Id
join d in context.ModelSetD on a.Id equals d.Id
select new { a, b, c }).FirstOrDefault();
if (x == null)
return null;
return new MyModelA()
{
Id = x.a.Id,
Name = x.a.Name,
ModelB = new MyModelB() { Id = x.a.ModelB.Id, Name = x.a.ModelB..Name },
ModelC = new MyModelC() { Id = x.c.Id, Name = x.c.Name },
ModelD = new MyModelD() { Id = x.d.Id, Name = x.d.Name }
};
}
Since Entity Framework can't handle creating nested classes from within the query, I simply returned an anonymous object from my query containing the data I wanted, then mapped it to the Model

Categories

Resources