Using Entity Framework Core 2.2 I have the following query:
IQueryable<User> users = _context.Users.AsNoTracking();
User user = await users
.Include(x => x.UserSkills)
.ThenInclude(x => x.Skill)
.FirstOrDefaultAsync(x => x.Id == 1);
var userSkills = user.UserSkills.ToList();
IQueryable<Lesson> lessons = _context.Lessons.AsNoTracking();
var test = lessons
.Where(x => x.IsEnabled)
.Where(x => x.LessonSkills.All(y => userSkills.Any(z => y.SkillId == z.SkillId)))
.ToList();
I am looking to get User Skills contains all Lesson Skills.
When I run this query I get the following error:
Exception thrown: 'System.InvalidOperationException' in System.Private.CoreLib.dll:
'Error generated for warning 'Microsoft.EntityFrameworkCore.Query.QueryClientEvaluationWarning:
The LINQ expression 'where ([y].SkillId == [z].SkillId)' could not be translated and will be evaluated locally.'.
How to change the query to solve this problem?
Update
I need to extend this query with an extra option (y.SkillLevelId <= z.SkillLevelId):
var test = lessons
.Where(x => x.IsEnabled)
.Where(x => x.LessonSkills.All(y => userSkills.Any(z =>
y.SkillId == z.SkillId
&&
y.SkillLevelId <= z.SkillLevelId)))
.ToList();
userSkills is in-memory collection, and from my experience with EF6 and EF Core so far I can say that the only reliable translatable construct with in-memory collections is Enumerable.Contains method on primitive type in-memory collection.
So the following solves the problem is question.
First (should be outside the query expression tree):
var userSkillIds = user.UserSkills.Select(x => x.SkillId);
Then instead of
.Where(x => x.LessonSkills.All(y => userSkills.Any(z => y.SkillId == z.SkillId)))
use the equivalent (but translatable):
.Where(x => x.LessonSkills.All(y => userSkillIds.Contains(y.SkillId)))
Update: If you can't use Contains, the options you have until EF Core starts supporting it are (1) EntityFrameworkCore.MemoryJoin package (I personally haven't tested it, but the idea is interesting), (2) manually building Or based predicate with Expression class (hard and works for small memory collections) and (3) replace the memory collection with real IQueryable<>, for instance
var userSkills = users
.Where(x => x.Id == 1)
.SelectMany(x => x.UserSkills);
and use the original query.
The Problem
I have a LINQ query (against Entity Framework) that use Include to include some navigation properties. One of those properties uses ThenInclude to include its own property collection. When I run the query, the first level properties are included on the primary object but the sub-collection (the one using ThenInclude) is always empty.
However, if I change the query to use Fluent API form, the query works and the sub-collection is actually included. Why does this work for the Fluent form and not the normal LINQ query?
Example
//FAIL - This returns Benefits but Benefits.Dates.Count = 0 on all Benefits
var list1 = (from s in _context.Subscribers
.Include(s => s.Dates)
.Include(s => s.Benefits)
.ThenInclude(b => b.Dates)
where s.Id == 13643
select new { benefits = s.Benefits }).ToList();
//SUCCESS - This returns Benefits and Benefits.Dates.Count is > 0 on the ones with Dates
var list2 = _context.Subscribers
.Include(s => s.Dates)
.Include(s => s.Benefits)
.ThenInclude(b => b.Dates)
.Where(s => s.Id == 13643)
.Select(s => new { benefits = s.Benefits}).ToList();
Am I mistaken that these queries should give the same output?
Update
I just tried manually linking things using LINQ and I am able to get Benefit dates included -- granted, its not apples to apples as the resulting set is different, but the point is that Include(b => b.Dates) seems to work in this case.
var list3 = (from s in _context.Subscribers.Include(s => s.Dates)
join b in _context.Benefits.Include(b => b.Dates) on s.Id equals b.SubscriberId
select new {benefits = b}).ToList();
I'm beginning to wonder if ThenInclude() may be a little more restrictive in where/when it can be used?
Update 2
I just noticed a warning in my Debug Output window that led me to this link about ignored includes. This seems to be on the right track as the Debug Output clearly indicates that these includes are being ignored.
What doesn't make sense, though, is that I'm actually selecting to anonymous in both examples and only one of them seems to ignore the includes. Why one and not the other?
This is just a guess: since You have Datas, in both Subscribers and Benefits, there is a chance that you are not querying the correct model/entity... you can try confirming the entity type like below (assuming Benefit is your Entity type).
var list1 = (from s in _context.Subscribers
.Include(s => s.Dates)
.Include(s => s.Benefits)
.ThenInclude(b => (b as Benefit).Dates)
where s.Id == 13643
select new { benefits = s.Benefits }).ToList();
var list2 = _context.Subscribers
.Include(s => s.Dates)
.Include(s => s.Benefits)
.ThenInclude(b => (b as Benefit).Dates) // <-- I suggest renaming s to b
.Where(s => s.Id == 13643)
.Select(s => new { benefits = s.Benefits}).ToList();
My model has a navigation property and this navigation property has another sub navigation property. I need to use a where clause on sub navigation property to filter results.
I'm trying to use linq query but unable to get the results
_context.Set<Job>().Include(x=>x.Premises).ThenInclude(y=>y.Station.Where(s=>s.)
The following sql join gives me desired results
select *
from [dbo].[JOB] J inner join
[dbo].[PREMISES] P on J.PremisesId = P.Id inner join
[dbo].[STATION] S on P.StationCode=S.StationCode
where S.StationCode = '001'
Any ideas?
Notice these similar LINQ statements:
var jobs = db.Jobs
.Include(j => j.Premises)
.Include(j => j.Premises.Select(p => p.Stations))
.ToList();
var stations = db.Stations
.Include(s => s.Premise)
.Include(s => s.Premise.Job)
.ToList();
While your return type is different, you are essentially holding the same data in memory. I could use the second to get all jobs too:
var jobs_from_stations = stations.Select(s => s.Premise.Job).Distinct();
Both jobs_from_stations and jobs will contain the exact same data.
There is a difference in filtering though.
If you were to add a Where() clause in this query, it would work differently.
The first query would filter in scope of the Job entity, whereas the second would filter in scope of the Station entity.
Since you are currently trying to filter based on a station property, that suggests using the second query:
var stations = db.Stations
.Include(s => s.Premise)
.Include(s => s.Premise.Job)
.Where(s => s.StationCode == "001")
.ToList();
If you want the return type to be a list of jobs:
var jobs = db.Stations
.Include(s => s.Premise)
.Include(s => s.Premise.Job)
.Where(s => s.StationCode == "001")
.Select(s => s.Premise.Job)
.Distinct()
.ToList();
Note that it would still be possible to use the first query instead, but it becomes more verbose and unwieldy:
var jobs = db.Jobs
.Include(j => j.Premises)
.Include(j => j.Premises.Select(p => p.Stations))
.Where(j => j.Premises.Any(p => p.Stations.Any(s => s.StationCode == "001")))
.ToList();
As a rule of thumb, I always try to start from the child and work my way up. As you see in the above example, it makes the filtering easier. But maybe you also noticed that it keeps the Include() statements simple too:
.Include(s => s.Premise)
.Include(s => s.Premise.Job)
instead of
.Include(j => j.Premises)
.Include(j => j.Premises.Select(p => p.Stations))
While these two examples are functionally equivalent, having to add a Select() for every level becomes very cumbersome if you want to include entities that are several relationships removed from where you started.
I'm trying to filter on the initial query. I have nested include leafs off a model. I'm trying to filter based on a property on one of the includes. For example:
using (var context = new BloggingContext())
{
var blogs = context.Blogs
.Include(blog => blog.Posts)
.ThenInclude(post => post.Author)
.ToList();
}
How can I also say .Where(w => w.post.Author == "me")?
Entity Framework core 5 is the first EF version to support filtered Include.
How it works
Supported operations:
Where
OrderBy(Descending)/ThenBy(Descending)
Skip
Take
Some usage examples (from the original feature request and the github commmit)
:
Only one filter allowed per navigation, so for cases where the same navigation needs to be included multiple times (e.g. multiple ThenInclude on the same navigation) apply the filter only once, or apply exactly the same filter for that navigation.
context.Customers
.Include(c => c.Orders.Where(o => o.Name != "Foo")).ThenInclude(o => o.OrderDetails)
.Include(c => c.Orders).ThenInclude(o => o.Customer)
or
context.Customers
.Include(c => c.Orders.Where(o => o.Name != "Foo")).ThenInclude(o => o.OrderDetails)
.Include(c => c.Orders.Where(o => o.Name != "Foo")).ThenInclude(o => o.Customer)
Another important note:
Collections included using new filter operations are considered to be loaded.
That means that if lazy loading is enabled, addressing one customer's Orders collection from the last example won't trigger a reload of the entire Orders collection.
Also, two subsequent filtered Includes in the same context will accumulate the results. For example...
context.Customers.Include(c => c.Orders.Where(o => !o.IsDeleted))
...followed by...
context.Customers.Include(c => c.Orders.Where(o => o.IsDeleted))
...will result in customers with Orders collections containing all orders.
Filtered Include and relationship fixup
If other Orders are loaded into the same context, more of them may get added to a customers.Orders collection because of relationship fixup. This is inevitable because of how EF's change tracker works.
context.Customers.Include(c => c.Orders.Where(o => !o.IsDeleted))
...followed by...
context.Orders.Where(o => o.IsDeleted).Load();
...will again result in customers with Orders collections containing all orders.
The filter expression
The filter expression should contain predicates that can be used as a stand-alone predicate for the collection. An example will make this clear. Suppose we want to include orders filtered by some property of Customer:
context.Customers.Include(c => c.Orders.Where(o => o.Classification == c.Classification))
It compiles, but it'll throw a very technical runtime exception, basically telling that o.Classification == c.Classification can't be translated because c.Classification can't be found. The query has to be rewritten using a back-reference from Order to Customer:
context.Customers.Include(c => c.Orders.Where(o => o.Classification == o.Customer.Classification))
The predicate o => o.Classification == o.Customer.Classification) is "stand alone" in the sense that it can be used to filter Orders independently:
context.Orders.Where(o => o.Classification == o.Customer.Classification) // No one would try 'c.Classification' here
This restriction may change in later EF versions than the current stable version (EF core 5.0.7).
What can (not) be filtered
Since Where is an extension method on IEnumerable it's clear that only collections can be filtered. It's not possible to filter reference navigation properties. If we want to get orders and only populate their Customer property when the customer is active, we can't use Include:
context.Orders.Include(o => o.Customer.Where( ... // obviously doesn't compile
Filtered Include vs filtering the query
Filtered Include has given rise to some confusion on how it affects filtering a query as a whole. The rule of the thumb is: it doesn't.
The statement...
context.Customers.Include(c => c.Orders.Where(o => !o.IsDeleted))
...returns all customers from the context, not only the ones with undeleted orders. The filter in the Include doesn't affect the number of items returned by the main query.
On the other hand, the statement...
context.Customers
.Where(c => c.Orders.Any(o => !o.IsDeleted))
.Include(c => c.Orders)
...only returns customers having at least one undeleted order, but having all of their orders in the Orders collections. The filter on the main query doesn't affect the orders per customer returned by Include.
To get customers with undeleted orders and only loading their undeleted orders, both filters are required:
context.Customers
.Where(c => c.Orders.Any(o => !o.IsDeleted))
.Include(c => c.Orders.Where(o => !o.IsDeleted))
Filtered Include and projections
Another area of confusion is how filtered Include and projections (select new { ... }) are related. The simple rule is: projections ignore Includes, filtered or not. A query like...
context.Customers
.Include(c => c.Orders)
.Select(c => new { c.Name, c.RegistrationDate })
...will generate SQL without a join to Orders. As for EF, it's the same as...
context.Customers
.Select(c => new { c.Name, c.RegistrationDate })
It gets confusing when the Include is filtered, but Orders are also used in the projection:
context.Customers
.Include(c => c.Orders.Where(o => !o.IsDeleted))
.Select(c => new
{
c.Name,
c.RegistrationDate,
OrderDates = c.Orders.Select(o => o.DateSent)
})
One might expect that OrderDates only contains dates from undeleted orders, but they contain the dates from all Orders. Again, the projection completely ignores the Include. Projection and Include are separate worlds.
How strictly they lead their own lives is amusingly demonstrated by this query:
context.Customers
.Include(c => c.Orders.Where(o => !o.IsDeleted))
.Select(c => new
{
Customer = c,
OrderDates = c.Orders.Select(o => o.DateSent)
})
Now pause for a moment and predict the outcome...
The not so simple rule is: projections don't always ignore Include. When there is an entity in the projection to which the Include can be applied, it is applied. That means that Customer in the projection contains its undeleted Orders, whereas OrderDates still contains all dates. Did you get it right?
Not doable.
There is an on-going discussion about this topic:
https://github.com/aspnet/EntityFramework/issues/1833
I'd suggest to look around for any of the 3rd party libraries listed there, ex.: https://github.com/jbogard/EntityFramework.Filters
You can also reverse the search.
{
var blogs = context.Author
.Include(author => author.posts)
.ThenInclude(posts => posts.blogs)
.Where(author => author == "me")
.Select(author => author.posts.blogs)
.ToList();
}
Not sure about Include() AND ThenInclude(), but it's simple to do that with a single include:
var filteredArticles =
context.NewsArticles.Include(x => x.NewsArticleRevisions)
.Where(article => article.NewsArticleRevisions
.Any(revision => revision.Title.Contains(filter)));
Hope this helps!
Although it's (still in discussion) not doable with EF Core, I've managed to do it using Linq to Entities over EF Core DbSet. In your case instead of:
var blogs = context.Blogs
.Include(blog => blog.Posts)
.ThenInclude(post => post.Author)
.ToList()
.. you'll have:
await (from blog in this.DbContext.Blogs
from bPost in blog.Posts
from bpAuthor in bPost.Author
where bpAuthor = "me"
select blog)
.ToListAsync();
I used below package
Use Z.EntityFramework.Plus
IncludeFilter and IncludeFilterByPath two methods are which you can use.
var list = context.Blogs.IncludeFilter(x => x.Posts.Where(y => !y.IsSoftDeleted))
.IncludeFilter(x => x.Posts.Where(y => !y.IsSoftDeleted)
.SelectMany(y => y.Comments.Where(z => !z.IsSoftDeleted)))
.ToList();
Here is the example https://dotnetfiddle.net/SK934m
Or you can do like this
GetContext(session).entity
.Include(c => c.innerEntity)
.Select(c => new Entity()
{
Name = c.Name,
Logo = c.Logo,
InnerEntity= c.InnerEntity.Where(s => condition).ToList()
})
Interesting case and it worked!!
If you have table/model user(int id, int? passwordId, ICollection<PwdHist> passwordHistoryCollection) where collection is history of passwords. Could be many or none.
And PwdHistory(int id, int UserId, user User). This has a quasi relationship via attributes.
Needed to get user, with related current password record, while leaving historical records behind.
User user = _userTable
.Include(u => u.Tenant)
.Include(u => u.PwdHistory.Where(p => p.Id == p.PwdUser.PasswordId))
.Where(u => u.UserName == userName)
.FirstOrDefault();
Most interesting part is .Include(u => u.PwdHistory.Where(p => p.Id == p.PwdUser.PasswordId))
works with user and many passwords
works with user and no passwords
works with no user
We can use by extension
public static IQueryable<TEntity> IncludeCondition<TEntity, TProperty>(this IQueryable<TEntity> query, Expression<Func<TEntity, TProperty>> predicate, bool? condition) where TEntity : class where TProperty : class
{
return condition == true ? query.Include(predicate) : query;
}
Usage;
_context.Tables.IncludeCondition(x => x.InnerTable, true)
This task can be accomplished with two queries. For example:
var query = _context.Employees
.Where(x =>
x.Schedules.All(s =>
s.ScheduleDate.Month != DateTime.UtcNow.AddMonths(1).Month &&
s.ScheduleDate.Year != DateTime.UtcNow.AddMonths(1).Year) ||
(x.Schedules.Any(s =>
s.ScheduleDate.Month == DateTime.UtcNow.AddMonths(1).Month &&
s.ScheduleDate.Year == DateTime.UtcNow.AddMonths(1).Year) &&
x.Schedules.Any(i => !i.ScheduleDates.Any())));
var employees = await query.ToListAsync();
await query.Include(x => x.Schedules)
.ThenInclude(x => x.ScheduleDates)
.SelectMany(x => x.Schedules)
.Where(s => s.ScheduleDate.Month == DateTime.UtcNow.AddMonths(1).Month &&
s.ScheduleDate.Year == DateTime.UtcNow.AddMonths(1).Year).LoadAsync();
I have a data model like this
I would like to load all the related entities from a Reconciliation into a Reconciliation object.
For now the only way I could find to load all the related entites to a single Recon is in multiple Lists. But I want to load every related entities in a Reconciliation object. If possible in an elegant way.
Reconciliation recon = db.Reconciliations
.Where(r => r.ReconNum == 382485).First();
List<ReconciliationDetail> reconDetails = recon.ReconciliationDetails.ToList();
List<JrnlEntryDetail> jrnlDetails = reconDetails.Select(r => r.JrnlEntryDetail).ToList();
List<JrnlEntry> jrnl = jrnlDetails.Select(j => j.JrnlEntry).ToList();
List<ARInvoice> invoices = jrnl.SelectMany(j => j.ARInvoices).ToList();
List<ARInvoiceDetail> invoicesDetail = invoices
.SelectMany(i => i.ARInvoiceDetails).ToList();
List<ARCredMemo> credmemos = jrnl.SelectMany(j => j.ARCredMemoes).ToList();
List<ARCredMemoDetail> credmemosDetail = credmemos
.SelectMany(c => c.ARCredMemoDetails).ToList();
List<IncomingPay> incomingPays = jrnl.SelectMany(j => j.IncomingPays).ToList();
List<IncomingPayDetail> incomingPaysDetail = incomingPays
.SelectMany(i => i.IncomingPayDetails).ToList();
// ... and so on for outgoing pays, AP Invoices AP Cred Memo ...etc
I have also tried to load it with Include and Select but I get this exception :
The Include path expression must refer to a navigation property defined on the type. Use dotted paths for reference navigation properties and the Select operator for collection navigation properties.
And I don't get how I could load every childs of JrnlEntry using Include and Select
Reconciliation recon = db.Reconciliations
.Where(r => r.ReconNum == 382485)
.Include(r => r.ReconciliationDetails
.Select(d => d.JrnlEntryDetail)
.Select(jd => jd.JrnlEntry)
.SelectMany(j => j.ARInvoices).SelectMany(i => i.ARInvoiceDetails))
Edit
Managed to do it this way too but it's not very beautiful :
Reconciliation recon = db.Reconciliations
.Where(r => r.ReconNum == 382485)
.Include(r => r.ReconciliationDetails.Select(rd => rd.JrnlEntryDetail)
.Select(jd => jd.JrnlEntry).Select(j => j.ARInvoices.Select(i => i.ARInvoiceDetails)))
.Include(r => r.ReconciliationDetails.Select(rd => rd.JrnlEntryDetail)
.Select(jd => jd.JrnlEntry).Select(j => j.ARCredMemoes.Select(c => c.ARCredMemoDetails)))
.Include(r => r.ReconciliationDetails.Select(rd => rd.JrnlEntryDetail)
.Select(jd => jd.JrnlEntry).Select(j => j.IncomingPays.Select(i => i.IncomingPayDetails)))
.Include(r => r.ReconciliationDetails.Select(rd => rd.JrnlEntryDetail)
.Select(jd => jd.JrnlEntry).Select(j => j.OutgoingPays.Select(o => o.OutgoingPayDetails)))
.Include(r => r.ReconciliationDetails.Select(rd => rd.JrnlEntryDetail)
.Select(jd => jd.JrnlEntry).Select(j => j.APInvoices.Select(o => o.APInvoiceDetails)))
.Include(r => r.ReconciliationDetails.Select(rd => rd.JrnlEntryDetail)
.Select(jd => jd.JrnlEntry).Select(j => j.APCredMemoes.Select(o => o.APCredMemoDetails)))
.Include(r => r.ReconciliationDetails.Select(rd => rd.JrnlEntryDetail)
.Select(jd => jd.JrnlEntry).Select(j => j.JrnlEntryDetails))
There are two ways to perform Eager Loading in Entity Framework:
ObjectQuery.Include Method
Explicit loading using either DbEntityEntry.Collection Method or DbEntityEntry.Reference Method along with their respective Load methods
There are also manners to write Raw SQL Queries against database:
DbSet.SqlQuery Method deals with entities
Database.SqlQuery Method deals with arbitrary types
Database.ExecuteSqlCommand Method for arbitrary DDL/DML
For the case, when you're attempting to load nearly entire database, it would be good idea to execute dedicated store procedure against it.
Try with just .Include(r => r.ReconciliationDetails) initially. Then add the .Select() statements one-by-one. At what point does the exception reappear? The .SelectMany() call looks a bit suspicious to me!
A second question that might help identify the problem... After you run the code that contains all the ToList() calls, is your recon entity complete? i.e. are all its navigation properties populated? This should be the case because of the automatic 'fixup' behavior of Entity Framework.
With EF, sometimes it is more efficient to load a complex object graph with several calls rather than chained Include() calls. Check the generated SQL and see what is most efficient in your case.
Not sure if it's to late but could you benefit from structuring your code such as
var acctName = "someName";
var detailList = _repository.Include(e => e.JrnlEntryDetail).Filter(c => c.JrnlEntryDetail.Any(e => e.AcctName == acctName)).Get().ToList();