I'm looking for suggestions on how to write a query. For each Goal, I want to select the first Task (sorted by Task.Sequence), in addition to any tasks with ShowAlways == true. (My actual query is more complex, but this query demonstrates the limitations I'm running into.)
I tried something like this:
var tasks = (from a in DbContext.Areas
from g in a.Goals
from t in g.Tasks
let nextTaskId = g.Tasks.OrderBy(tt => tt.Sequence).Select(tt => tt.Id).DefaultIfEmpty(-1).FirstOrDefault()
where t.ShowAlways || t.Id == nextTaskId
select new CalendarTask
{
// Member assignment
}).ToList();
But this query appears to be too complex.
System.InvalidOperationException: 'Processing of the LINQ expression 'OrderBy<Task, int>(
source: MaterializeCollectionNavigation(Navigation: Goal.Tasks(< Tasks > k__BackingField, DbSet<Task>) Collection ToDependent Task Inverse: Goal, Where<Task>(
source: NavigationExpansionExpression
Source: Where<Task>(
source: DbSet<Task>,
predicate: (t0) => Property<Nullable<int>>((Unhandled parameter: ti0).Outer.Inner, "Id") == Property<Nullable<int>>(t0, "GoalId"))
PendingSelector: (t0) => NavigationTreeExpression
Value: EntityReferenceTask
Expression: t0
,
predicate: (i) => Property<Nullable<int>>(NavigationTreeExpression
Value: EntityReferenceGoal
Expression: (Unhandled parameter: ti0).Outer.Inner, "Id") == Property<Nullable<int>>(i, "GoalId"))),
keySelector: (tt) => tt.Sequence)' by 'NavigationExpandingExpressionVisitor' failed. This may indicate either a bug or a limitation in EF Core. See https://go.microsoft.com/fwlink/?linkid=2101433 for more detailed information.'
The problem is the line let nextTaskId =.... If I comment out that, there is no error. (But I don't get what I'm after.)
I'll readily admit that I don't understand the details of the error message. About the only other way I can think of to approach this is return all the Tasks and then sort and filter them on the client. But my preference is not to retrieve data I don't need.
Can anyone see any other ways to approach this query?
Note: I'm using the very latest version of Visual Studio and .NET.
UPDATE:
I tried a different, but less efficient approach to this query.
var tasks = (DbContext.Areas
.Where(a => a.UserId == UserManager.GetUserId(User) && !a.OnHold)
.SelectMany(a => a.Goals)
.Where(g => !g.OnHold)
.Select(g => g.Tasks.Where(tt => !tt.OnHold && !tt.Completed).OrderBy(tt => tt.Sequence).FirstOrDefault()))
.Union(DbContext.Areas
.Where(a => a.UserId == UserManager.GetUserId(User) && !a.OnHold)
.SelectMany(a => a.Goals)
.Where(g => !g.OnHold)
.Select(g => g.Tasks.Where(tt => !tt.OnHold && !tt.Completed && (tt.DueDate.HasValue || tt.AlwaysShow)).OrderBy(tt => tt.Sequence).FirstOrDefault()))
.Distinct()
.Select(t => new CalendarTask
{
Id = t.Id,
Title = t.Title,
Goal = t.Goal.Title,
CssClass = t.Goal.Area.CssClass,
DueDate = t.DueDate,
Completed = t.Completed
});
But this also produced an error:
System.InvalidOperationException: 'Processing of the LINQ expression 'Where<Task>(
source: MaterializeCollectionNavigation(Navigation: Goal.Tasks (<Tasks>k__BackingField, DbSet<Task>) Collection ToDependent Task Inverse: Goal, Where<Task>(
source: NavigationExpansionExpression
Source: Where<Task>(
source: DbSet<Task>,
predicate: (t) => Property<Nullable<int>>((Unhandled parameter: ti).Inner, "Id") == Property<Nullable<int>>(t, "GoalId"))
PendingSelector: (t) => NavigationTreeExpression
Value: EntityReferenceTask
Expression: t
,
predicate: (i) => Property<Nullable<int>>(NavigationTreeExpression
Value: EntityReferenceGoal
Expression: (Unhandled parameter: ti).Inner, "Id") == Property<Nullable<int>>(i, "GoalId"))),
predicate: (tt) => !(tt.OnHold) && !(tt.Completed))' by 'NavigationExpandingExpressionVisitor' failed. This may indicate either a bug or a limitation in EF Core. See https://go.microsoft.com/fwlink/?linkid=2101433 for more detailed information.'
This is a good example for the need of full reproducible example. When trying to reproduce the issue with similar entity models, I was either getting a different error about DefaulIfEmpty(-1) (apparently not supported, don't forget to remove it - the SQL query will work correctly w/o it) or no error when removing it.
Then I noticed a small deeply hidden difference in your error messages compared to mine, which led me to the cause of the problem:
MaterializeCollectionNavigation(Navigation: Goal.Tasks (<Tasks>k__BackingField, DbSet<Task>)
specifically the DbSet<Task> at the end (in my case it was ICollection<Task>). I realized that you used DbSet<T> type for collection navigation property rather than the usual ICollection<T>, IEnumerable<T>, List<T> etc., e.g.
public class Goal
{
// ...
public DbSet<Task> Tasks { get; set; }
}
Simply don't do that. DbSet<T> is a special EF Core class, supposed to be used only from DbContext to represent db table, view or raw SQL query result set. And more importantly, DbSets are the only real EF Core query roots, so it's not surprising that such usage confuses the EF Core query translator.
So change it to some of the supported interfaces/classes (for instance, ICollection<Task>) and the original problem will be solved.
Then removing the DefaultIfEmpty(-1) will allow successfully translating the first query in question.
I don't have EF Core up and running, but are you able to split it up like this?
var allTasks = DbContext.Areas
.SelectMany(a => a.Goals)
.SelectMany(a => a.Tasks);
var always = allTasks.Where(t => t.ShowAlways);
var next = allTasks
.OrderBy(tt => tt.Sequence)
.Take(1);
var result = always
.Concat(next)
.Select(t => new
{
// Member assignment
})
.ToList();
Edit: Sorry, I'm not great with query syntax, maybe this does what you need?
var allGoals = DbContext.Areas
.SelectMany(a => a.Goals);
var allTasks = DbContext.Areas
.SelectMany(a => a.Goals)
.SelectMany(a => a.Tasks);
var always = allGoals
.SelectMany(a => a.Tasks)
.Where(t => t.ShowAlways);
var nextTasks = allGoals
.SelectMany(g => g.Tasks.OrderBy(tt => tt.Sequence).Take(1));
var result = always
.Concat(nextTasks)
.Select(t => new
{
// Member assignment
})
.ToList();
I would recommend you start by breaking up this query into individual parts. Try iterating through the Goals in a foreach with your Task logic inside. Add each new CalendarTask to a List that you defined ahead of time.
Overall breaking this logic up and experimenting a bit will probably lead you to some insight with the limitations of Entity Framework Core.
I think we might separate the query into two steps. First, query each goals and get the min Sequence task and store them(maybe with a anonymous type like {NextTaskId,Goal}). Then, we query the temp data and get the result. For example
Areas.SelectMany(x=>x.Goals)
.Select(g=>new {
NextTaskId=g.Tasks.OrderBy(t=>t.Sequence).FirstOrDefault()?.Id,
Tasks=g.Tasks.Where(t=>t.ShowAlways)
})
.SelectMany(a=>a.Tasks,(a,task)=>new {
NextTaskId = a.NextTaskId,
Task = task
});
I tried to create the linq request but I'm not sure about the result
var tasks = ( from a in DbContext.Areas
from g in a.Goals
from t in g.Tasks
join oneTask in (from t in DbContext.Tasks
group t by t.Id into gt
select new {
Id = gt.Key,
Sequence = gt.Min(t => t.Sequence)
}) on new { t.Id, t.Sequence } equals new { oneTask.Id,oneTask.Sequence }
select new {Area = a, Goal = g, Task = t})
.Union(
from a in DbContext.Areas
from g in a.Goals
from t in g.Tasks
where t.ShowAlways
select new {Area = a, Goal = g, Task = t});
I currently don't have EF Core, but do you really need to compare this much?
Wouldn't querying the tasks be sufficient?
If there is a navigation property or foreign key defined I could imaging using something like this:
Tasks.Where(task => task.Sequence == Tasks.Where(t => t.GoalIdentity == task.GoalIdentity).Min(t => t.Sequence) || task.ShowAlways);
The Problem
I have a LINQ query (against Entity Framework) that use Include to include some navigation properties. One of those properties uses ThenInclude to include its own property collection. When I run the query, the first level properties are included on the primary object but the sub-collection (the one using ThenInclude) is always empty.
However, if I change the query to use Fluent API form, the query works and the sub-collection is actually included. Why does this work for the Fluent form and not the normal LINQ query?
Example
//FAIL - This returns Benefits but Benefits.Dates.Count = 0 on all Benefits
var list1 = (from s in _context.Subscribers
.Include(s => s.Dates)
.Include(s => s.Benefits)
.ThenInclude(b => b.Dates)
where s.Id == 13643
select new { benefits = s.Benefits }).ToList();
//SUCCESS - This returns Benefits and Benefits.Dates.Count is > 0 on the ones with Dates
var list2 = _context.Subscribers
.Include(s => s.Dates)
.Include(s => s.Benefits)
.ThenInclude(b => b.Dates)
.Where(s => s.Id == 13643)
.Select(s => new { benefits = s.Benefits}).ToList();
Am I mistaken that these queries should give the same output?
Update
I just tried manually linking things using LINQ and I am able to get Benefit dates included -- granted, its not apples to apples as the resulting set is different, but the point is that Include(b => b.Dates) seems to work in this case.
var list3 = (from s in _context.Subscribers.Include(s => s.Dates)
join b in _context.Benefits.Include(b => b.Dates) on s.Id equals b.SubscriberId
select new {benefits = b}).ToList();
I'm beginning to wonder if ThenInclude() may be a little more restrictive in where/when it can be used?
Update 2
I just noticed a warning in my Debug Output window that led me to this link about ignored includes. This seems to be on the right track as the Debug Output clearly indicates that these includes are being ignored.
What doesn't make sense, though, is that I'm actually selecting to anonymous in both examples and only one of them seems to ignore the includes. Why one and not the other?
This is just a guess: since You have Datas, in both Subscribers and Benefits, there is a chance that you are not querying the correct model/entity... you can try confirming the entity type like below (assuming Benefit is your Entity type).
var list1 = (from s in _context.Subscribers
.Include(s => s.Dates)
.Include(s => s.Benefits)
.ThenInclude(b => (b as Benefit).Dates)
where s.Id == 13643
select new { benefits = s.Benefits }).ToList();
var list2 = _context.Subscribers
.Include(s => s.Dates)
.Include(s => s.Benefits)
.ThenInclude(b => (b as Benefit).Dates) // <-- I suggest renaming s to b
.Where(s => s.Id == 13643)
.Select(s => new { benefits = s.Benefits}).ToList();
I am trying to cut this linq down
var sys = db.tlkpSystems
.Where(a => db.tlkpSettings.Where(e => e.Hidden < 3)
.Select(o => o.System)
.ToList().Contains(a.System)) //cannot get this part in?
.OrderBy(a => a.SystemName).ToList();
foreach (var item in sys)
model.Add(new SettingSystem {
System = item.System,
SystemName = item.SystemName
});
I have tried the following:
List<SettingSystem> model = new List<SettingSystem>();
model = db.tlkpSettings.Where(e => e.Hidden < 3)
.OrderBy(e => e.Setting)
.Select(e => new SettingSystem
{
System = e.System,
SystemName = e.Setting
}).ToList();
How can I call the .Contains(a.System) part in my query?
Thanks
Some general rules when working with LINQ to Entities:
Avoid using ToList inside the query. It prevents EF to build a correct SQL query.
Don't use Contains when working with entities (tables). Use Any or joins.
Here is your query (in case System is not an entity navigation property):
var sys = db.tlkpSystems
.Where(a => db.tlkpSettings.Any(e => e.Hidden < 3 && e.System == a.System))
.OrderBy(a => a.SystemName).ToList();
As an addendum, there is also AsEnumerable for when you must pull a query into memory (such as calling methods within another clause). This is generally better than ToList or ToArray since it'll enumerate the query, rather than enumerating, putting together a List/Array, and then enumerating that collection.
this code written by #Rahul Singh in this post Convert TSQL to Linq to Entities :
var result = _dbContext.ExtensionsCategories.ToList().GroupBy(x => x.Category)
.Select(x =>
{
var files = _dbContext.FileLists.Count(f => x.Select(z => z.Extension).Contains(f.Extension));
return new
{
Category = x.Key,
TotalFileCount = files
};
});
but this code have problem when used inside database context and we should use ToList() like this to fix "Only primitive types or enumeration types are supported in this context" error :
var files = _dbContext.FileLists.Count(f => x.Select(z => z.Extension).ToList().Contains(f.Extension));
the problem of this is ToList() fetch all records and reduce performance, now i wrote my own code :
var categoriesByExtensionFileCount =
_dbContext.ExtensionsCategories.Select(
ec =>
new
{
Category = ec.Category,
TotalSize = _dbContext.FileLists.Count(w => w.Extension == ec.Extension)
});
var categoriesTOtalFileCount =
categoriesByExtensionFileCount.Select(
se =>
new
{
se.Category,
TotalCount =
categoriesByExtensionFileCount.Where(w => w.Category == se.Category).Sum(su => su.TotalSize)
}).GroupBy(x => x.Category).Select(y => y.FirstOrDefault());
the performance of this code is better but it have much line of code, any idea about improve performance of first code or reduce line of second code :D
Regards, Mojtaba
You should have a navigation property from ExtensionCategories to FileLists. If you are using DB First, and have your foreign key constraints set up in the database, it should do this automatically for you.
If you supply your table designs (or model classes), it would help a lot too.
Lastly, you can rewrite using .ToList().Contains(...) with .Any() which should solve your immediate issue. Something like:
_dbContext.FileLists.Count(f => x.Any(z => z.Extension==f.Extension)));
Today I ran into a problem with Entity Framework. I'm not sure if this is a weird bug or that i'm doing something wrong. I've already looked all over the forum for any possible solutions, but none I found worked for me.
I have the following LINQ query:
return (from sp in context.ServiceProviders.DefaultIfEmpty()
join pl in context.Platforms on sp.Id equals pl.ServiceProviderId into innerPl
from pl in innerPl.DefaultIfEmpty()
join pp in context.Participants on pl.Id equals pp.PlatformId into innerPp
from pp in innerPp.DefaultIfEmpty()
join ps in context.Paymentsettlements on pp.Id equals ps.ParticipantId into innerPs
from ps in innerPs.Where(ps => ps.ConfirmedOn.HasValue && ps.ExportDate.HasValue && !ps.StatisticsDate.HasValue).DefaultIfEmpty()
select sp).Include(sp => sp.Environment)
.Include(sp => sp.Platforms.Select(pl => pl.Participants.Select(pp => pp.Paymentsettlements.Select(ps => ps.Requester))))
.Include(sp => sp.Platforms.Select(pl => pl.Participants.Select(pp => pp.Paymentsettlements.Select(ps => ps.Payer))))
.ToList();
The result i'm looking for is that i always get the ServiceProvider no matter if there are objects inside the ServiceProvider. I am getting this result at the moment, but the where I've put in the query does not get taken into account. The following where does not make any difference:
innerPs.Where(ps => ps.ConfirmedOn.HasValue && ps.ExportDate.HasValue && !ps.StatisticsDate.HasValue).DefaultIfEmpty()
If the StatisticsDate has a value, those Paymentsettlements also are given in the output.
I've already tried to put the WHERE statement on the context.Paymentsettlements object.
I hope anyone can help me with this problem.
Kind regards,
Rob H
Actually you are doing left join and then selecting ServiceProviders. Here you are getting all providers. Then you are including all child elements: select sp).Include(sp => sp.Environment). This won't work. It will include all rows.
What you can really do is select to anonymous type like
select new {sp, ps }
Unfortunately there is no way of filtering in included objects. Include is something like all or nothing.
You can read about it:
How to filter nested collection Entity Framework objects?
EF Query With Conditional Include
I've finally made another (hacky) solution. Here is my final code:
using (var context = new BetaalplatformContext())
{
var dienstverleners = context.Dienstverleners.Include(dv => dv.Omgeving)
.Include(dv => dv.Platformen)
.Include(dv => dv.Platformen.Select(pl => pl.Deelnemers))
.Include(dv => dv.Platformen.Select(pl => pl.Deelnemers.Select(dn => dn.Betaalregelingen)))
.Include(dv => dv.Platformen.Select(pl => pl.Deelnemers.Select(dn => dn.Betaalregelingen.Select(br => br.Aanvrager))))
.Include(dv => dv.Platformen.Select(pl => pl.Deelnemers.Select(dn => dn.Betaalregelingen.Select(br => br.Betaler))))
.ToList();
dienstverleners.ForEach(
dv => dv.Platformen.ForEach(
pl => pl.Deelnemers.ForEach(
dn => dn.Betaalregelingen = dn.Betaalregelingen
.Where(br2 => br2.BevestigdOp.HasValue && br2.ExportDatum.HasValue && !br2.StatistiekDatum.HasValue)
.ToList()
)
)
);
return dienstverleners;
}
This way i am abled to keep my models intact (I don't like to use anonymous objects).