Develop Reference

Suggestions on why this linq ef core query wont execute

Good morning,
I have the following query in EF Core which works as required if I do not add into the the where clause a condition for the createdtimestamp. If I add other conditions clauses to filter down the data as shown below the query will execute without issue.
var q = (
from trckhead in DbContext.TrackingBatchHeader
from userdets in DbContext.UserDetails
.Where(u => u.UserId == trckhead.ClosedByUserId).DefaultIfEmpty()
from trckkeys in DbContext.TrackingBatchesItemKey
.Where(t => t.TrackingNo == trckhead.TrackingNo).DefaultIfEmpty()
from trcklink1 in DbContext.TrackingBatchesLink
.Where(x => x.TrackingNo == trckhead.TrackingNo && x.TrackingType == "I").DefaultIfEmpty()
from trcklink2 in DbContext.TrackingBatchesLink
.Where(x => x.TrackingNo == trckhead.TrackingNo && x.TrackingType == "T").DefaultIfEmpty()
join trckref in DbContext.TrackingBatchesReference on trckhead.TrackingBatchType equals trckref
.TrackingBatchType
join mthsite in DbContext.SiteDetail on trckhead.SiteCode equals mthsite.SiteCode
join userdets2 in DbContext.UserDetails on trckhead.CreatedByUserId equals userdets2.UserId
where
(string.IsNullOrEmpty(searchRequest.Status) || trckhead.Status == searchRequest.Status) &&
trckhead.CreatedTimestamp >= DateTime.Now.AddMonths(-11)
select new TrackingBatchSearchResultDto
{
TrackingBatchId = trckhead.TrackingNo,
SiteCode = trckhead.SiteCode,
TrackingBatchType = trckhead.TrackingBatchType,
Status = trckhead.Status,
Created = trckhead.CreatedTimestamp,
CreatedById = trckhead.CreatedByUserId,
ClosedById = trckhead.ClosedByUserId,
UserDescription = trckhead.UsersDescription,
TrackingBatchDescription = trckref.TrackingBatchTypeDescription,
SiteName = mthsite.Title,
CreatedByName = userdets2.FullUserName ?? string.Empty,
ClosedByName = userdets.FullUserName ?? string.Empty,
Link1 = trcklink1.TrackingData ?? string.Empty,
Link2 = trcklink2.TrackingData ?? string.Empty
}).Distinct();
return await q.ToListAsync();
If I keep the date clause in the query the exception that is raised shows the following. I would have thought the server evaluation wouldnt be happy if I was passing a function into the query but checking against a simple date has thrown me. I could filter the results by date after obtaining the whole data set but the number of records returned can be massive so I would rather filter it down from a database query. Any help would be appreciated.
The LINQ expression 'DbSet<TrackingBatchHeader>
.SelectMany(
source: t => DbSet<UserDetail>
.Where(u => u.UserId == t.ClosedByUserId)
.DefaultIfEmpty(),
collectionSelector: (t, c) => new TransparentIdentifier<TrackingBatchHeader, UserDetail>(
Outer = t,
Inner = c
))
.SelectMany(
source: ti => DbSet<TrackingBatchesItemKey>
.Where(t0 => t0.TrackingNo == ti.Outer.TrackingNo)
.DefaultIfEmpty(),
collectionSelector: (ti, c) => new TransparentIdentifier<TransparentIdentifier<TrackingBatchHeader, UserDetail>, TrackingBatchesItemKey>(
Outer = ti,
Inner = c
))
.SelectMany(
source: ti0 => DbSet<TrackingBatchesLink>
.Where(t1 => t1.TrackingNo == ti0.Outer.Outer.TrackingNo && t1.TrackingType == "I")
.DefaultIfEmpty(),
collectionSelector: (ti0, c) => new TransparentIdentifier<TransparentIdentifier<TransparentIdentifier<TrackingBatchHeader, UserDetail>, TrackingBatchesItemKey>, TrackingBatchesLink>(
Outer = ti0,
Inner = c
))
.SelectMany(
source: ti1 => DbSet<TrackingBatchesLink>
.Where(t2 => t2.TrackingNo == ti1.Outer.Outer.Outer.TrackingNo && t2.TrackingType == "T")
.DefaultIfEmpty(),
collectionSelector: (ti1, c) => new TransparentIdentifier<TransparentIdentifier<TransparentIdentifier<TransparentIdentifier<TrackingBatchHeader, UserDetail>, TrackingBatchesItemKey>, TrackingBatchesLink>, TrackingBatchesLink>(
Outer = ti1,
Inner = c
))
.Join(
outer: DbSet<TrackingBatchesReference>,
inner: ti2 => ti2.Outer.Outer.Outer.Outer.TrackingBatchType,
outerKeySelector: t3 => t3.TrackingBatchType,
innerKeySelector: (ti2, t3) => new TransparentIdentifier<TransparentIdentifier<TransparentIdentifier<TransparentIdentifier<TransparentIdentifier<TrackingBatchHeader, UserDetail>, TrackingBatchesItemKey>, TrackingBatchesLink>, TrackingBatchesLink>, TrackingBatchesReference>(
Outer = ti2,
Inner = t3
))
.Join(
outer: DbSet<SiteDetail>,
inner: ti3 => ti3.Outer.Outer.Outer.Outer.Outer.SiteCode,
outerKeySelector: s => s.SiteCode,
innerKeySelector: (ti3, s) => new TransparentIdentifier<TransparentIdentifier<TransparentIdentifier<TransparentIdentifier<TransparentIdentifier<TransparentIdentifier<TrackingBatchHeader, UserDetail>, TrackingBatchesItemKey>, TrackingBatchesLink>, TrackingBatchesLink>, TrackingBatchesReference>, SiteDetail>(
Outer = ti3,
Inner = s
))
.Join(
outer: DbSet<UserDetail>,
inner: ti4 => ti4.Outer.Outer.Outer.Outer.Outer.Outer.CreatedByUserId,
outerKeySelector: u0 => u0.UserId,
innerKeySelector: (ti4, u0) => new TransparentIdentifier<TransparentIdentifier<TransparentIdentifier<TransparentIdentifier<TransparentIdentifier<TransparentIdentifier<TransparentIdentifier<TrackingBatchHeader, UserDetail>, TrackingBatchesItemKey>, TrackingBatchesLink>, TrackingBatchesLink>, TrackingBatchesReference>, SiteDetail>, UserDetail>(
Outer = ti4,
Inner = u0
))
.Where(ti5 => ti5.Outer.Outer.Outer.Outer.Outer.Outer.Outer.CreatedTimestamp >= (Nullable<DateTime>)DateTime.Now.AddMonths(-11))' could not be translated. Either rewrite the query in a form that can be translated, or switch to client evaluation explicitly by inserting a call to either AsEnumerable(), AsAsyncEnumerable(), ToList(), or ToListAsync(). See https://go.microsoft.com/fwlink/?linkid=2101038 for more information.

Plainly speaking, neglecting either establish navigation properties for the relationships between entities and instead using manual outer joins using "from" plus then adding the ultimate insult to poorly formed queries with Distinct is trying to use EF like a hammer to pound a square peg through a round hole.
With proper relationships, that query should be trivial.
Your datetime condition error message does not match your query.
** Edit: Removed details on Date comparison - This does not appear to be directly related as I did verify that the Nullable Date/DateTime comparisons do work with .AddMonth() in EF Core.
Expanding out conditional logic can help simplify the queries being executed.
For example:
var query = (/*build your base query*/);
if (!string.IsNullOrEmpty(searchRequest.Status))
query = query.Where(x => x.Status == searchRequest.Status);
if (searchRequest.DateFrom.HasValue)
{
var fromDate = searchRequest.DateFrom.Value.AddMonths(-11);
query = query.Where(x => x.CreatedTimestamp >= fromDate);
}
var results = query.Select(...).Distinct().ToList();
EF can perform some magic to build queries provided you give it enough information to do it properly. Seeing it build something like ti5.Outer.Outer.Outer.Outer.Outer.Outer.Outer.CreatedTimestamp should be sending off alarm bells. There may be something deep in your query lurking that is tripping a Client-side evaluation, or it is simply a case that a combination of joins and such has tipped over a complexity limit or exposed a bug in EF Core. The starting point would be to look to simplify that query expression by leveraging navigation properties rather than joins, and externalizing the conditional logic where you can.
If you have no recourse but to try and query across relationships that you are not prepared to map out properly with navigation properties, I would recommend exploring building your query in SQL as a Stored Procedure and then defining an Entity that can map to the resulting record from that Sproc.

Retrieving the last record in each group with EF Core

I'm trying to do the retrieve the last record in each group like it does here
https://stackoverflow.com/a/20770779/4789608
but in Entity Framework Core. Based on what's on the link, I can get the SQL to work to bring back the correct data using
select *
from [Location]
where [LocationModelId] in (select max([LocationModelId])
from [Location]
group by [UserModelId])
or
select m1.*
from [Location] m1
left outer join [Location] m2 on (m1.[LocationModelId]< m2.[LocationModelId]
and m1.[UserModelId] = m2.[UserModelId])
This is the closest I've gotten based on that link
locationDetails = _context.Location
.GroupBy(p => p.UserModelId)
.Select(p => p.FirstOrDefault(w => w.UserModelId == p.Max(m => m.UserModelId)))
.OrderBy(p => p.DateCreated)
.ToList();
which returns this error message so, it's definitely not working.
The LINQ expression 'GroupByShaperExpression:\r\nKeySelector: l.UserModelId, \r\nElementSelector:EntityShaperExpression: \r\n EntityType: LocationModel\r\n ValueBufferExpression: \r\n ProjectionBindingExpression: EmptyProjectionMember\r\n IsNullable: False\r\n\r\n .FirstOrDefault(w => w.UserModelId == GroupByShaperExpression:\r\n KeySelector: l.UserModelId, \r\n ElementSelector:EntityShaperExpression: \r\n EntityType: LocationModel\r\n ValueBufferExpression: \r\n ProjectionBindingExpression: EmptyProjectionMember\r\n IsNullable: False\r\n\r\n .Max(m => m.UserModelId))' could not be translated. Either rewrite the query in a form that can be translated, or switch to client evaluation explicitly by inserting a call to 'AsEnumerable', 'AsAsyncEnumerable', 'ToList', or 'ToListAsync'

This query should get you desired output. It is not fast as just SQL with window functions, but it should be acceptable.
var unique = _context.Location
.Select(x => new {x.UserModelId})
.Distinct();
var query =
from u in unique
from l in _context.Location
.Where(x => x.UserModelId == u.UserModelId)
.OrderByDescending(x.DateCreated)
.Take(1)
select l;
var locationDetails = query.ToList();

Unable to use GroupBy on Subselect in EF Core 5/6

I am currently trying to implement a Report for a customer using Entity Framework Core 5 (also tested 6) and came across a problem, that I couldn't resolve.
I am trying to query a List of all customers / bank accounts with their total sales per month.
Therefore I am trying to group all of the customers bookings, that fit the criteria, by their Month and sum up their amounts.
Shouldn't be that hard of a task.
This is my Query:
context.Kontens
.Select(k => new
{
Creditor = context.DebitorKreditors
.Include(k => k.Kontakt)
.FirstOrDefault(d => d.Nr == k.KtoNr),
Sums = k.BuchZeilenKtos
.Where(b => b.Buch.Belegdatum.Year == DateTime.Now.Year
&& (
(b.GegenKto.KtoNr >= xxxxxx && b.GegenKto.KtoNr <= xxxxxx)
|| (b.GegenKto.KtoNr >= xxxxxx && b.GegenKto.KtoNr <= xxxxxx)
|| (b.GegenKto.KtoNr >= xxxxxx && b.GegenKto.KtoNr <= xxxxxx)
)
)
.GroupBy(b => b.Buch.Belegdatum.Month)
.Select(g => new
{
Sum = g.Sum(b => b.Betrag * (b.SollHaben == "-" || b.SollHaben == "H" ? -1 : 1)),
Month = g.Key
}),
Sum334 = k.BuchZeilenKtos
.Where(b => b.GegenKto.KtoNr >= xxxxxx && b.GegenKto.KtoNr <= xxxxxx)
.Sum(b => b.Betrag * (b.SollHaben == "-" || b.SollHaben == "H" ? -1 : 1))
})
.Where(k => k.Creditor.Kreditor && k.Sums.Any())
.OrderBy(k => k.Creditor.Nr)
.ToList()
Translations:
BuchzeilenKtos = Bookings
Kontens = Bank Accounts
DebitorKreditors = Customers
Buch = Booking
The bank account numbers have been removed due to privacy reasons.
With this I am getting the following error complaining about missing key columns:
System.InvalidOperationException: Unable to translate collection subquery in projection since it uses 'Distinct' or 'Group By' operations and doesn't project key columns of all of it's tables which are required to generate results on client side. Missing column: b.ID. Either add column(s) to the projection or rewrite query to not use 'GroupBy'/'Distinct' operation.
I tried numerous things including changing the GroupBy to also include the key columns:
.GroupBy(b => new
{
b.Buch.Belegdatum.Month,
b.Id,
b.GegenKto,
b
})
But this should produce wrong results anyway from what I understand about LINQ Grouping.
Nevertheless, Now EF is complaining about not being able to translate the query and wants me to change to clientside processing, which isn't an option due to the very high amount of bookings being processed:
System.InvalidOperationException: The LINQ expression 'DbSet<BuchZeilen>()
.Where(b => EF.Property<Nullable<Guid>>(EntityShaperExpression:
EntityType: Konten
ValueBufferExpression:
ProjectionBindingExpression: EmptyProjectionMember
IsNullable: False
, "Id") != null && object.Equals(
objA: (object)EF.Property<Nullable<Guid>>(EntityShaperExpression:
EntityType: Konten
ValueBufferExpression:
ProjectionBindingExpression: EmptyProjectionMember
IsNullable: False
, "Id"),
objB: (object)EF.Property<Nullable<Guid>>(b, "KtoId")))
.Join(
inner: DbSet<Buchung>(),
outerKeySelector: b => EF.Property<Nullable<Guid>>(b, "BuchId"),
innerKeySelector: b0 => EF.Property<Nullable<Guid>>(b0, "Id"),
resultSelector: (o, i) => new TransparentIdentifier<BuchZeilen, Buchung>(
Outer = o,
Inner = i
))
.LeftJoin(
inner: DbSet<Konten>(),
outerKeySelector: b => EF.Property<Nullable<Guid>>(b.Outer, "GegenKtoId"),
innerKeySelector: k0 => EF.Property<Nullable<Guid>>(k0, "Id"),
resultSelector: (o, i) => new TransparentIdentifier<TransparentIdentifier<BuchZeilen, Buchung>, Konten>(
Outer = o,
Inner = i
))
.Where(b => b.Outer.Inner.Belegdatum.Year == DateTime.Now.Year && b.Inner.KtoNr >= 311000 && b.Inner.KtoNr <= 319999 || b.Inner.KtoNr >= 334000 && b.Inner.KtoNr <= 334999 || b.Inner.KtoNr >= 340000 && b.Inner.KtoNr <= 340999)
.GroupBy(
keySelector: b => new {
Month = b.Outer.Inner.Belegdatum.Month,
Id = b.Outer.Outer.Id,
GegenKto = b.Inner,
b = b.Outer.Outer
},
elementSelector: b => b.Outer.Outer)' could not be translated. Either rewrite the query in a form that can be translated, or switch to client evaluation explicitly by inserting a call to 'AsEnumerable', 'AsAsyncEnumerable', 'ToList', or 'ToListAsync'.
I've read, that EF Core 5 has removed certain functionality on GroupBy/Distinct for correlated sub queries but I am not sure whether that is my exact scenario. In my opinion the query shouldn't be too specific for EF to handle.
Does anybody know how I can achieve, what I am trying to do or whether it is possible at least?

Rewriting an SQL statement as a LINQ query results in runtime error

As a follow-up to this question of mine I have got this SQL:
SELECT Documents.*
FROM Documents
WHERE Documents.ID IN
(
SELECT Keywords.DocumentID
FROM Keywords
WHERE
Keywords.Keyword = 'KeywordA' OR
Keywords.Keyword = 'KeywordB'
GROUP BY Keywords.DocumentID
HAVING COUNT(Keywords.Keyword) = 2
)
I did use Linqer to convert the query to C# to use with Entity Framework Core 5:
from Document in db.Document
where
(from Keyword in db.Keyword
where
Keyword.Value == "KeywordA" ||
Keyword.Value == "KeywordB"
group Keyword by new {
Keyword.DocumentId
} into g
where g.Count(p => p.Value != null) == 2
select new {
g.Key.DocumentId
}).Contains(new { DocumentId = Document.DocumentId })
select Document
This compiles successfully, but upon running the query, I get an error:
The LINQ expression '<see below>' could not be translated. Either rewrite the query in a form that can be translated, or switch to client evaluation explicitly by inserting a call to 'AsEnumerable', 'AsAsyncEnumerable', 'ToList', or 'ToListAsync'. See https://go.microsoft.com/fwlink/?linkid=2101038 for more information.
The formatted LINQ expression from the above error message reads:
DbSet<Keyword>()
.Where(k => k.Value == "KeywordA" || k.Value == "KeywordB")
.GroupBy(
keySelector: k => new { DocumentId = k.DocumentId },
elementSelector: k => k)
.Where(e => e
.Count(p => p.Value != null) == 2)
.Select(e => new { DocumentId = e.Key.DocumentId })
.Any(p => p == new { DocumentId = EntityShaperExpression:
EntityType: Document
ValueBufferExpression:
ProjectionBindingExpression: EmptyProjectionMember
IsNullable: False
.DocumentId })
I really do not understand what's wrong here. I could only imagine that Linqer is too old to generate valid C# code for use in EF Core 5.
My question
Could someone give me a hint on what I'm doing wrong here and how to resolve the issue? (I.e. how to rewrite the C# query)

EF Core query translation still doesn't support many LINQ constructs, and all that without documentation of what exactly is/is not supported really puts us on the trial-and-error path, thus it's not surprising that external tools cannot produce a "proper" translation.
In this particular case the problem is Contains call with complex argument (event though it is a class with single member). Because they support only Contains with primitive argument.
So the minimal change needed to make this work is to replace
select new
{
g.Key.DocumentId
}).Contains(new { DocumentId = Document.DocumentId })
with
select g.Key.DocumentId).Contains(Document.DocumentId)

It looks like the .Contains(new { DocumentId = Document.DocumentId }) part is your problem. It's having trouble translating it into an expression because Document hasn't been evaluated at that point.
If you have the FK setup, you could refactor it this way:
from d in db.Documents
where d.Keywords
.Where(k => (new[] { "keyword A", "keyword B" }).Contains(k.Keyword))
.Count() == 2
select d

Try this:
var result =
from Document in db.Document
where
(from Keyword in db.Keyword
where
Keyword.Value == "KeywordA" ||
Keyword.Value == "KeywordB"
group Keyword by Keyword.DocumentId
into g
where g.Count(p => p.Value != null) == 2
select g.Key).Any(d => d == Document.DocumentId)
select Document;

If having 2 separate queries, 1 for the keywords and 1 for the documents, is not an issue, this should work too:
var matchedDocumentIds = db.Keywords
.Where(keyword => keyword.Keyword == "KeywordA" || keyword.Keyword == "KeywordB")
.GroupBy(keyword => keyword.DocumentID)
.Where(grp => grp.Count() > 2)
.Select(grp => grp.Key);
var filteredDocs = db.Documents.Where(doc => matchedDocumentIds.Any(matchedDocId => matchedDocId == doc.ID));
This assumes there is a foreign key in place between the 2 tables.

How to use Linq or Lambda with IQueryable to GroupBy and get the first/last record on the collection in C#?

I'm using Entity framework core 3.1 to connect to the database, what I'm trying is to get the Highest and lowest product details which grouped by its category id. based on answers shown on this question and this question I tried to write the following code:
using Linq :
//NOTE: 'Products' is an IQueryable for all the products in the database.
var highPrice = from a in Products
group a by a.CategoryId
into prods
select prods.OrderByDescending(a => a.Price).First();
using Lambda:
//NOTE: 'Products' is an IQueryable for all the products in the database.
var highPrice = Products.GroupBy(a => a.CategoryId, (key, a) => a.OrderByDescending(a => a.Price).First()).ToList();
Both works fine only if I converted the IQueryble of Products to IEnumerable using for example .ToList() , but when running without converting it pops the following exception:
System.InvalidOperationException
HResult=0x80131509
Message=The LINQ expression '(GroupByShaperExpression:
KeySelector: (a.CategoryId),
ElementSelector:(EntityShaperExpression:
EntityType: Product
ValueBufferExpression:
(ProjectionBindingExpression: EmptyProjectionMember)
IsNullable: False))
.OrderByDescending(a => a.Price)' could not be translated. Either rewrite the query in a form that can be translated, or switch to client evaluation explicitly by inserting a call to either AsEnumerable(), AsAsyncEnumerable(), ToList(), or ToListAsync().
Is there any way to get the first/Last records of IQueryable collection without converting to IEnumerable ??

Not all Linq methods can be translated to a SQL query (that's why your expression doesn't compile).
So, you can obtain the same result using the Take() method combined with order by clause.
Following your example:
var mostExpensiveProductByCategory = dbContext.Products
.GroupBy(x => x.CategoryId).Select(x => new
{
CategoryId = x.Key,
Product = x.OrderByDescending(p => p.Price).Take(1)
})
.ToList();
var cheapestProductByCategory = dbContext.Products
.GroupBy(x => x.CategoryId).Select(x => new
{
CategoryId = x.Key,
Product = x.OrderBy(p => p.Price).Take(1)
})
.ToList();
UPDATE:
To achieve your requirement and avoid in-memory grouping, I would suggest you to work with navigation properties, see below:
var mostExpensiveProductByCategory = dbContext.Categories.Select(x => new
{
x.CategoryId,
Products = x.Products.OrderByDescending(p => p.Price).Take(1)
}).ToList();
This will produce the following query output:
SELECT [c].[CategoryId], [t0].[ProductId], [t0].[CategoryId], [t0].[Price]
FROM [Categories] AS [c]
LEFT JOIN ( SELECT [t].[ProductId], [t].[CategoryId], [t].[Price]
FROM ( SELECT [p].[ProductId], [p].[CategoryId], [p].[Price], ROW_NUMBER() OVER(PARTITION BY [p].[CategoryId] ORDER BY [p].[Price] DESC) AS [row]
FROM [Products] AS [p] ) AS [t] WHERE [t].[row] <= 1 ) AS [t0] ON [c].[CategoryId] = [t0].[CategoryId]
ORDER BY [c].[CategoryId], [t0].[CategoryId], [t0].[Price] DESC, [t0].[ProductId]