I am attempting to get a list of Order Ids from a EF linq query. The sql query is returning back quickly but I think the EF framework is trying to create the full entity. I only want the ID of the order. It seems that it creates the whole entity and then it parses it out to only the id. Which seems to be a complete waste of resources.
Orders are a complex object that includes lots of child entitys. I dont need anything but the Ids of the orders in the list. Orders are organized into OrderCollection which is a many to many relationship.
The basic query in English is get the order ids in the specified order collection and have a cart date newer then the date specified and only send the specified page (skip and take).
example:
_repo.Orders.Where(o => o.OrderCollection.Any(r => r.Id == RoutingRuleId)).ToList()
.Where(o => o.OrderDate >= StartDateTime)
.OrderBy(x => x.OrderDate )
.Skip(RecordsToSkipCount)
.Take(BatchSize).Select(x => new { x.Id }).ToArray();
The sql runs in just 102ms for this in debug mode. But afterwards I see the memory go to up 4GB before failing. The batchsize is only 100. Its like it grabbing everything.
I tried moving the select around but that failed also or gave syntax errors or poor performance in running the sql (SQL taking 16 seconds).
Example
_repo.Orders.Select(x => new { x.Id, x.OrderCollection, x.OrderDate})
.Where(o => o.OrderCollection.Any(r => r.Id == RoutingRuleId)).ToList()
.Where(o => o.OrderDate >= StartDateTime)
.OrderBy(x => x.OrderDate )
.Skip(RecordsToSkipCount)
.Take(BatchSize).Select(x => new { x.Id }).ToArray();
The database has millions of records.
What you have is roughly;
List<Order> list = _repo.Orders
.Where(o => o.OrderCollection.Any(r => r.Id == RoutingRuleId))
.ToList();
list.Where(o => o.OrderDate >= StartDateTime)
.OrderBy(x => x.OrderDate )
.Skip(RecordsToSkipCount)
.Take(BatchSize)
.Select(x => new { x.Id })
.ToArray();
That first .ToList is forcing EF Core to load every order with a matching routing rule into memory. The rest of the expression is then using IEnumerable extension methods to process those results.
I think you want to rearrange that to;
IQueryable<Order> query = _repo.Orders
.Where(o => o.OrderCollection.Any(r => r.Id == RoutingRuleId)
&& o.OrderDate >= StartDateTime)
.OrderBy(x => x.OrderDate )
.Skip(RecordsToSkipCount)
.Take(BatchSize)
.Select(x => new { x.Id });
query.ToArray();
Creating an IQueryable doesn't trigger EF Core to execute any SQL. An IQueryable is just a description of the query you would like to run. Then it's the .ToArray method that will finally cause EF Core to compile and execute an sql statement.
I'm looking for suggestions on how to write a query. For each Goal, I want to select the first Task (sorted by Task.Sequence), in addition to any tasks with ShowAlways == true. (My actual query is more complex, but this query demonstrates the limitations I'm running into.)
I tried something like this:
var tasks = (from a in DbContext.Areas
from g in a.Goals
from t in g.Tasks
let nextTaskId = g.Tasks.OrderBy(tt => tt.Sequence).Select(tt => tt.Id).DefaultIfEmpty(-1).FirstOrDefault()
where t.ShowAlways || t.Id == nextTaskId
select new CalendarTask
{
// Member assignment
}).ToList();
But this query appears to be too complex.
System.InvalidOperationException: 'Processing of the LINQ expression 'OrderBy<Task, int>(
source: MaterializeCollectionNavigation(Navigation: Goal.Tasks(< Tasks > k__BackingField, DbSet<Task>) Collection ToDependent Task Inverse: Goal, Where<Task>(
source: NavigationExpansionExpression
Source: Where<Task>(
source: DbSet<Task>,
predicate: (t0) => Property<Nullable<int>>((Unhandled parameter: ti0).Outer.Inner, "Id") == Property<Nullable<int>>(t0, "GoalId"))
PendingSelector: (t0) => NavigationTreeExpression
Value: EntityReferenceTask
Expression: t0
,
predicate: (i) => Property<Nullable<int>>(NavigationTreeExpression
Value: EntityReferenceGoal
Expression: (Unhandled parameter: ti0).Outer.Inner, "Id") == Property<Nullable<int>>(i, "GoalId"))),
keySelector: (tt) => tt.Sequence)' by 'NavigationExpandingExpressionVisitor' failed. This may indicate either a bug or a limitation in EF Core. See https://go.microsoft.com/fwlink/?linkid=2101433 for more detailed information.'
The problem is the line let nextTaskId =.... If I comment out that, there is no error. (But I don't get what I'm after.)
I'll readily admit that I don't understand the details of the error message. About the only other way I can think of to approach this is return all the Tasks and then sort and filter them on the client. But my preference is not to retrieve data I don't need.
Can anyone see any other ways to approach this query?
Note: I'm using the very latest version of Visual Studio and .NET.
UPDATE:
I tried a different, but less efficient approach to this query.
var tasks = (DbContext.Areas
.Where(a => a.UserId == UserManager.GetUserId(User) && !a.OnHold)
.SelectMany(a => a.Goals)
.Where(g => !g.OnHold)
.Select(g => g.Tasks.Where(tt => !tt.OnHold && !tt.Completed).OrderBy(tt => tt.Sequence).FirstOrDefault()))
.Union(DbContext.Areas
.Where(a => a.UserId == UserManager.GetUserId(User) && !a.OnHold)
.SelectMany(a => a.Goals)
.Where(g => !g.OnHold)
.Select(g => g.Tasks.Where(tt => !tt.OnHold && !tt.Completed && (tt.DueDate.HasValue || tt.AlwaysShow)).OrderBy(tt => tt.Sequence).FirstOrDefault()))
.Distinct()
.Select(t => new CalendarTask
{
Id = t.Id,
Title = t.Title,
Goal = t.Goal.Title,
CssClass = t.Goal.Area.CssClass,
DueDate = t.DueDate,
Completed = t.Completed
});
But this also produced an error:
System.InvalidOperationException: 'Processing of the LINQ expression 'Where<Task>(
source: MaterializeCollectionNavigation(Navigation: Goal.Tasks (<Tasks>k__BackingField, DbSet<Task>) Collection ToDependent Task Inverse: Goal, Where<Task>(
source: NavigationExpansionExpression
Source: Where<Task>(
source: DbSet<Task>,
predicate: (t) => Property<Nullable<int>>((Unhandled parameter: ti).Inner, "Id") == Property<Nullable<int>>(t, "GoalId"))
PendingSelector: (t) => NavigationTreeExpression
Value: EntityReferenceTask
Expression: t
,
predicate: (i) => Property<Nullable<int>>(NavigationTreeExpression
Value: EntityReferenceGoal
Expression: (Unhandled parameter: ti).Inner, "Id") == Property<Nullable<int>>(i, "GoalId"))),
predicate: (tt) => !(tt.OnHold) && !(tt.Completed))' by 'NavigationExpandingExpressionVisitor' failed. This may indicate either a bug or a limitation in EF Core. See https://go.microsoft.com/fwlink/?linkid=2101433 for more detailed information.'
This is a good example for the need of full reproducible example. When trying to reproduce the issue with similar entity models, I was either getting a different error about DefaulIfEmpty(-1) (apparently not supported, don't forget to remove it - the SQL query will work correctly w/o it) or no error when removing it.
Then I noticed a small deeply hidden difference in your error messages compared to mine, which led me to the cause of the problem:
MaterializeCollectionNavigation(Navigation: Goal.Tasks (<Tasks>k__BackingField, DbSet<Task>)
specifically the DbSet<Task> at the end (in my case it was ICollection<Task>). I realized that you used DbSet<T> type for collection navigation property rather than the usual ICollection<T>, IEnumerable<T>, List<T> etc., e.g.
public class Goal
{
// ...
public DbSet<Task> Tasks { get; set; }
}
Simply don't do that. DbSet<T> is a special EF Core class, supposed to be used only from DbContext to represent db table, view or raw SQL query result set. And more importantly, DbSets are the only real EF Core query roots, so it's not surprising that such usage confuses the EF Core query translator.
So change it to some of the supported interfaces/classes (for instance, ICollection<Task>) and the original problem will be solved.
Then removing the DefaultIfEmpty(-1) will allow successfully translating the first query in question.
I don't have EF Core up and running, but are you able to split it up like this?
var allTasks = DbContext.Areas
.SelectMany(a => a.Goals)
.SelectMany(a => a.Tasks);
var always = allTasks.Where(t => t.ShowAlways);
var next = allTasks
.OrderBy(tt => tt.Sequence)
.Take(1);
var result = always
.Concat(next)
.Select(t => new
{
// Member assignment
})
.ToList();
Edit: Sorry, I'm not great with query syntax, maybe this does what you need?
var allGoals = DbContext.Areas
.SelectMany(a => a.Goals);
var allTasks = DbContext.Areas
.SelectMany(a => a.Goals)
.SelectMany(a => a.Tasks);
var always = allGoals
.SelectMany(a => a.Tasks)
.Where(t => t.ShowAlways);
var nextTasks = allGoals
.SelectMany(g => g.Tasks.OrderBy(tt => tt.Sequence).Take(1));
var result = always
.Concat(nextTasks)
.Select(t => new
{
// Member assignment
})
.ToList();
I would recommend you start by breaking up this query into individual parts. Try iterating through the Goals in a foreach with your Task logic inside. Add each new CalendarTask to a List that you defined ahead of time.
Overall breaking this logic up and experimenting a bit will probably lead you to some insight with the limitations of Entity Framework Core.
I think we might separate the query into two steps. First, query each goals and get the min Sequence task and store them(maybe with a anonymous type like {NextTaskId,Goal}). Then, we query the temp data and get the result. For example
Areas.SelectMany(x=>x.Goals)
.Select(g=>new {
NextTaskId=g.Tasks.OrderBy(t=>t.Sequence).FirstOrDefault()?.Id,
Tasks=g.Tasks.Where(t=>t.ShowAlways)
})
.SelectMany(a=>a.Tasks,(a,task)=>new {
NextTaskId = a.NextTaskId,
Task = task
});
I tried to create the linq request but I'm not sure about the result
var tasks = ( from a in DbContext.Areas
from g in a.Goals
from t in g.Tasks
join oneTask in (from t in DbContext.Tasks
group t by t.Id into gt
select new {
Id = gt.Key,
Sequence = gt.Min(t => t.Sequence)
}) on new { t.Id, t.Sequence } equals new { oneTask.Id,oneTask.Sequence }
select new {Area = a, Goal = g, Task = t})
.Union(
from a in DbContext.Areas
from g in a.Goals
from t in g.Tasks
where t.ShowAlways
select new {Area = a, Goal = g, Task = t});
I currently don't have EF Core, but do you really need to compare this much?
Wouldn't querying the tasks be sufficient?
If there is a navigation property or foreign key defined I could imaging using something like this:
Tasks.Where(task => task.Sequence == Tasks.Where(t => t.GoalIdentity == task.GoalIdentity).Min(t => t.Sequence) || task.ShowAlways);
I am trying to filter out the second part of the tables (UserRoles.IsDeleted==false). Is there any advice how i can do that?
var Users = context.Users.Where(r => r.IsDeleted == IsDeleted).ToList<User>();
Users = context.Users.Include(x => x.UserRoles.Select(y=>y.IsDeleted==false)).ToList();
Thank you
You can do the following to filter using the second part:
var Users = context.Users.Where(r => r.IsDeleted == IsDeleted).ToList<User>();
if(condition)
{
Users = Users.where(y => y.IsDeleted == false)).ToList();
}
There are two options to filter related entities
Doing a projection.
Unfortunately, when you use Include method, you can't filter the related entities as you intend to do. You need to project your query to a DTO object or a anonymous object, as the below example.
var query=context.Users.Include(x => x.UserRoles)
.Where(r => r.IsDeleted == IsDeleted)
.Select(u=> new{ ...,
Roles=x => x.UserRoles.Where(y=>!y.IsDeleted)})
A second option could be using Explicitly Loading. But this is in case you can load the related entities of one specific entity,eg,.
var user=context.Users.FirstOrDefault(r.IsDeleted == IsDeleted);//Getting a user
context.Entry(user)
.Collection(b => b.UserRoles)
.Query()
.Where(y=>!y.IsDeleted)
.Load();
You can do this inside of a foreach per each entity you get from the first query,
var query=context.Users.Where(r => r.IsDeleted == IsDeleted);
foreach(var u in query)
{
context.Entry(u)
.Collection(b => b.UserRoles)
.Query()
.Where(y=>!y.IsDeleted)
.Load();
}
but it's going to be really inefficient because you are going to do a roundtrip to your DB per each entity. My advice is use the first option, projecting the query.
I have two models: Thing and ThingStatus. Thing has an Id and some other fields. ThingStatus is a model which stores Status enum corresponding to id of Thing. Now I want to fetch Things that have Status != Completed.
What I try to do now looks like this:
var unfinishedIds = session.QueryOver<ThingStatus>()
.Where(t => t.Status != StatusEnum.Completed)
.Select(t => t.Id)
.List<long>()
.ToArray();
var unfinishedThings = session.QueryOver<Thing>()
.WhereRestriction(t => t.Id)
.IsIn(unfinishedIds)
.List<Thing>();
As far as I understand, in such case unfinishedIds will be fetched from database and only after that used as a filter in unfinishedThings query. Is there any way to avoid that and have the query optimizer select the right way to do that? I've heard there are some futures available with nhibernate but I'm not sure how they'd help here.
You can use a subquery if you can't create a NHibernate relationship between the two entities. No relationship --> no JoinAlias (or JoinQueryOver) possible.
With a subquery:
var unfinishedIds = QueryOver.Of<ThingStatus>()
.Where(t => t.Status != StatusEnum.Completed)
.Select(t => t.Id);
var unfinishedThings = session.QueryOver<Thing>()
.WithSubquery.WhereProperty(t => t.Id).In(unfinishedIds)
.List<Thing>();
(note the use of QueryOver.Of<>)
The query is equivalent to writing:
SELECT * FROM Things WHERE Id IN (SELECT Id FROM ThingsStatuses WHERE Status <> 'Completed')
I am trying to get the Previous and Next Record, but I am getting an exception at run time. What am I doing wrong? I found this solution to my problem on the internet. I get the basic idea, but I am not very experienced with LINQ.
LINQ to Entities does not recognize the method 'System.Linq.IQueryable1[CXW__DAQ___Web.Models.Report] SkipWhile[Report(System.Linq.IQueryable1[CXW_DAQWeb.Models.Report],System.Linq.Expressions.Expression1[System.Func2[CXW__DAQ_Web.Models.Report,System.Boolean]])' method, and his method cannot be translated into a store expression.
here is my relevant code.
var NextRecord = db.Reports.OrderBy(i => i.ID)
.SkipWhile(i => i.ID != id)
.Skip(1)
.First();
int nextID = NextRecord.ID;
var PrevRecord = db.Reports.OrderBy(i => i.ID)
.Reverse()
.SkipWhile(i => i.ID != id)
.Skip(1)
.Last();
int prevID = PrevRecord.ID;
What the error message is trying to tell you is that LINQ to Entities has no way to translate the IEnumerable.SkipWhile() method into SQL. You'll first have to force the remainder of the query to use LINQ to Objects first instead:
var nextRecord =
db.Reports.OrderBy(i => i.ID)
.AsEnumerable()
.SkipWhile(i => i.ID != id)
.Skip(1)
.First();
The obvious downside here being that your entire table is going to be ordered and then read into memory.