Develop Reference

GroupBy with EF Core 6.0 and SQL Server

I have a Blazor Web Application that has been working and in the field for a few months. I want to extend the DB querying to the group of similar "Detections".
It was written starting with .NET 5, and just today was updated to .NET 6 trying and get this working.
I would like to know how to get the results ordered by TimeStamp (a DateTime property). I have a working example with an in-memory DB, but production will be in SQL Server. I am not that great in SQL, but I have played around with it for a while in Management Studio with no luck.
Commenting out the OrderByDescending() groups things properly, but the results are not in the correct order. It seems the EF translation process is completely removing that line, it makes no difference in the generated query or the result set.
var results = context.Detections
//Line below makes no change ignored by SQL Server. Works when using in memory DB.
//.OrderByDescending(det => det.TimeStamp)
.GroupBy(det => new
{
Year = det.TimeStamp.Year,
Month = det.TimeStamp.Month,
Day = det.TimeStamp.Day,
Hour = det.TimeStamp.Hour,
})
.Select(grp => new
{
Count = grp.Count(),
Detection = grp.OrderByDescending(det => det.TimeStamp).First(),
})
//The following line will not translate
//.OrderByDescending(det => det.Detection.TimeStamp)
.ToList();
If any of this matters:
Visual Studio 2022 (4.8.04084)
.Net 6.0
SQL Server 2019 (15.0.2080.9)
*All NuGet packages related to EF have been updated to 6.0
Edit for clarification
The above code segment produces the following SQL query.
SELECT [t].[c], [t0].[Id], [t0].[TimeStamp]
FROM (
SELECT COUNT(*) AS [c], DATEPART(year, [d].[TimeStamp]) AS [c0], DATEPART(month, [d].[TimeStamp]) AS [c1], DATEPART(day, [d].[TimeStamp]) AS [c2], DATEPART(hour, [d].[TimeStamp]) AS [c3]
FROM [Detections] AS [d]
WHERE [d].[TimeStamp] > DATEADD(day, CAST(-16.0E0 AS int), GETUTCDATE())
GROUP BY DATEPART(year, [d].[TimeStamp]), DATEPART(month, [d].[TimeStamp]), DATEPART(day, [d].[TimeStamp]), DATEPART(hour, [d].[TimeStamp])
) AS [t]
OUTER APPLY (
SELECT TOP(1) [d0].[Id], [d0].[TimeStamp]
FROM [Detections] AS [d0]
WHERE ([d0].[TimeStamp] > DATEADD(day, CAST(-30.0E0 AS int), GETUTCDATE())) AND (((([t].[c0] = DATEPART(year, [d0].[TimeStamp])) AND ([t].[c1] = DATEPART(month, [d0].[TimeStamp]))) AND ([t].[c2] = DATEPART(day, [d0].[TimeStamp]))) AND ([t].[c3] = DATEPART(hour, [d0].[TimeStamp])))
ORDER BY [d0].[TimeStamp] DESC
) AS [t0]
It produces results similar to the following. Notice not sorted by time.
1 628591 2021-11-02 14:34:06.0442966
10 628601 2021-11-12 05:43:27.7015291
150 628821 2021-11-12 21:59:27.6444236
20 628621 2021-11-12 06:17:13.7798282
50 628671 2021-11-12 15:17:23.8893856
If I add ORDER BY [t0].TimeStamp DESC at the end of that SQL query in Management Studio I get the results I am looking for (see below). I just need to know how to write that in LINQ.
150 628821 2021-11-12 21:59:27.6444236
50 628671 2021-11-12 15:17:23.8893856
20 628621 2021-11-12 06:17:13.7798282
10 628601 2021-11-12 05:43:27.7015291
1 628591 2021-11-02 14:34:06.0442966
Adding .OrderByDescending(det => det.Detection.TimeStamp) at the end before ToList() was my first thought, but that "could not be translated". I will need to do some pagination with these results so I would really like to do the sorting in SQL.

GroupBy has to do its own Ordering so that 'ignores' is not totally unexpected.
Move it to below the grouping:
var results = context.Detections
//.OrderByDescending(det => det.TimeStamp)
.GroupBy(det => new
{
Year = det.TimeStamp.Year,
Month = det.TimeStamp.Month,
Day = det.TimeStamp.Day,
Hour = det.TimeStamp.Hour,
})
// .OrderByDescending(grp => grp.Key) // may have to split into y/m/d/h again
.OrderByDescending(grp => grp.Key.Year)
.ThenByDescending( grp => grp.Key.Month)
.ThenByDescending( grp => grp.Key.Day)
.ThenByDescending( grp => grp.Key.Hour)
.Select(grp => new
{
Count = grp.Count(),
Detection = grp.OrderByDescending(det => det.TimeStamp).First(),
})
.ToList();
When EF supports it, the Ordering and Grouping might become a little easier with
.GroupBy(det => new
{
Date = det.TimeStamp.Date,
Hour = det.TimeStamp.Hour,
})

For anyone looking at this in the future.
I was able to make this work by declaring and populating a TimeStamp property and using the OrderByDescending() at the end. I am not sure if this is the best solution, but it did solve my problem.
var results = context.Detections
.GroupBy(det => new
{
Year = det.TimeStamp.Year,
Month = det.TimeStamp.Month,
Day = det.TimeStamp.Day,
Hour = det.TimeStamp.Hour,
})
.Select(grp => new
{
Count = grp.Count(),
TimeStamp = grp.OrderByDescending(det => det.TimeStamp).First().TimeStamp,
Detection = grp.OrderByDescending(det => det.TimeStamp).First(),
})
.OrderByDescending(det => det.TimeStamp)
.ToList();

group by linq to entity query to get one record having latest timestamp by joining tables

There are two tables and using linq query to get records. From second table, there can be multiple rows corresponding to first table with date timestamp... based on below query, I am getting all records, but is there a way we can get the row from second table which has latest timestamp ?
Table Parent
ID Name
1 M
2 N
3 O
4 P
5 Q
Table Child
Id fkID DateTime
1 2 01/12/2021 09:12:20
2 2 01/12/2021 09:13:20
3 2 01/12/2021 09:14:20
4 2 01/12/2021 09:15:20
5 2 01/12/2021 **09:16:20**
Linq query:
from p in Parent
join c in Child on p.id equals c.fkId into cJoin
from cJoin in cJoin.DefaultIfEmpty()
select new TempResponse
{
Id = p.Id,
Name = p.Name,
Date = c.Date
}
I am getting 10 records using above query but just need 5 records i.e. from child table instead of all 5 records, we need a record that has latest time stamp
**expected output**
1 M
2 N 01/12/2021 09:16:20
this record is 5'th record from child table because this one has latest date time stamp
( latest record )
3 O
4 P
5 Q
Is there any way we can use group by and get the record that has latest time stamp from second table ?

Assuming you have defined navigation properties for the FK, I would use a query like;
dbContext.Child.Where(c => c.Date == c.Parent.Child.Max(c2 => c2.Date))

I believe you can use:
var ans = from p in Parent
join cmax in (
from c in Child
group c by c.fkId into cg
select cg.OrderByDescending(c => c.Date).FirstOrDefault())
on p.Id equals cmax.fkId into cJoin
from c in cJoin.DefaultIfEmpty()
select new TempResponse {
Id = p.Id,
Name = p.Name,
Date = c != null ? c.Date : null
};
Note that the order of results seems to vary on SQL Server unless you add another orderby clause before the select to force an order.

Simplifying LINQ query in C#

Table1
Table1ID Name Graduation Version Hobbies
1 A Degree 1 B
2 A Degree 2 C
3 A Degree 3 D
Table2
Table2ID Table1ID Name Graduation Version Address Surname Date
1 1 A Degree 1 A A 08-10-2019
2 2 A Degree 2 A A 08-10-2019
3 3 A Degree 3 A A
//I want to check if any version greater than highest version exists in Table1 .where Date column is not null in Table2
Suppose for the combination of Name and Degree , the highest version is 2 in Table2 since Date is null for Table2, I want to check if any record greater than 2 exists in Table1, if yes add it to a new List
Here is what I am doing.
List<Table2> groupByTable2 = //Operations on Table2 and get highest Version record from db
List<Table1> check = new List<Table1>();
List<Table1> check2 = await _table1.GetAll().ToListAsync();
Foreach(var a in groupByTable2)
{
List<Table1> check4 = check2.Where(x => x.Name == a.Name && x.Graduation == a.Graduation).ToList();
If(check4.Any(x=>x.Version > a.Version))
{
check.Add(check2.Where(x=>x.Table1ID == a.Table1ID).First());
}
}
Now my check contains a record where ID is 3. But is there any simpler way to achieve this in simpler way with readability and performance?

I hope I understood what you are trying to achieve. You could try the following.
var result = table2.Where(x=>x.Date!=null)
.GroupBy(x=> new {x.Name, x.Graduation})
.SelectMany(x=> x.OrderByDescending(c=>c.Version).Take(1))
.Join(table1,t2=>t2.Table1ID,t1=>t1.Table1ID,(t2,t1)=>t1)
.ToList();
result.AddRange(table1.Where(x=> result.Any(c=>c.Name.Equals(x.Name)
&& c.Graduation.Equals(x.Graduation)
&& c.Version < x.Version)));
The idea is to first use GroupBy and Join to get the List of Items with highest Version number in Table1 that has a valid date in Table2. Then, use List.AddRange to add remaining higher versions from Table1.

Slow performance in LINQ query

I have this t-sql query that is from an old system that run on Cold Fusion. This query takes less than one second to return the records.
select dateDiff(month, dateAdd(hour, 11, createdAt), {ts '2015-02-28 23:59:59'}) p, count(*) c
from account
where createdAt <= {ts '2015-02-28 23:59:59'}
and accountType = 'business'
and dateDiff(month, dateAdd(hour, 11, createdAt), {ts '2015-02-28 23:59:59'}) <12
group by dateDiff(month, dateAdd(hour, 11, createdAt), {ts '2015-02-28 23:59:59'})
order by dateDiff(month, dateAdd(hour, 11, createdAt), {ts '2015-02-28 23:59:59'})
I am now converting this to the new system using .NET and LINQ.
I managed to write this LINQ query which gives me the same results.
from a in db.Accounts
where SqlFunctions.DateDiff("Month", SqlFunctions.DateAdd("Hour", 11, a.createdAt), "2015-02-28 23:59:59") < 12
&& a.accountType == "business"
group a by SqlFunctions.DateDiff("Month", a.createdAt, "2015-02-28 23:59:59") into grp
orderby SqlFunctions.DateDiff("Month", grp.FirstOrDefault().createdAt, "2015-02-28 23:59:59")
select new ProgressViewModel.Data
{
date = SqlFunctions.DateDiff("Month", grp.FirstOrDefault().createdAt, "2015-02-28 23:59:59"),
amount = grp.Count()
});
However, this query takes no less than 5 seconds to run, while with the first one (t-sql) it takes less than 1 second.
By using Glimpse, we could see the t-sql that that LINQ query generates. It has multiple sub selects and it is 5 times longer than the fast query.
How could I improve the LINQ query?

Try something like this to bring it in to memory before the grouping:
from ca in (
from a in db.Accounts
where SqlFunctions.DateDiff("Month", SqlFunctions.DateAdd("Hour", 11, a.createdAt), "2015-02-28 23:59:59") < 12 && a.accountType == "business"
select a.createdAt).ToArray()
group a by new /* month diff */ into grp
orderby grp.Key
select new ProgressViewModel.Data
{
date = grp.key,
amount = grp.Count()
});

I really doubt you actually want to use FirstOrDefault() at any point in your code.
BTW It looks like you are using LinqToSQL as your Linq provider. That thing is nasty, inefficient and downright buggy. You should switch to EntityFramework if its at all possible
Given that...perhaps you should try this...
var date = new Date(2015, 2, 28).AddDays(1);
var query = from account in context.Accounts
where account.CreatedAt < date
where account.accountType == "business"
group account by
SqlFunctions.DateDiff(
"Month",
SqlFunctions.DateAdd(
"Hour", 11, a.createdAt),
date)
into g
where g.Key < 12
order by g.Key ascending
select new
{
MonthsAgo = g.Key,
Count = g.Count(),
};

Taking a quick look I would investigate your section grp.FirstOrDefault - is this really what you want to do?

I would definitely go for a parametrized Stored Procedure in this case. You should also consider creating a covering index on the tables you need. These steps usually boost performance by a very noticable amount.

Entity Framework: Efficiently grouping by month

I've done a bit of research on this, and the best I've found so far is to use an Asenumerable on the whole dataset, so that the filtering occurs in linq to objects rather than on the DB. I'm using the latest EF.
My working (but very slow) code is:
var trendData =
from d in ExpenseItemsViewableDirect.AsEnumerable()
group d by new {Period = d.Er_Approved_Date.Year.ToString() + "-" + d.Er_Approved_Date.Month.ToString("00") } into g
select new
{
Period = g.Key.Period,
Total = g.Sum(x => x.Item_Amount),
AveragePerTrans = Math.Round(g.Average(x => x.Item_Amount),2)
};
This gives me months in format YYYY-MM, along with the total amount and average amount. However it takes several minutes every time.
My other workaround is to do an update query in SQL so I have a YYYYMM field to group natively by. Changing the DB isn't an easy fix however so any suggestions would be appreciated.
The thread I found the above code idea (http://stackoverflow.com/questions/1059737/group-by-weeks-in-linq-to-entities) mentions 'waiting until .NET 4.0'. Is there anything recently introduced that helps in this situation?

The reason for poor performance is that the whole table is fetched into memory (AsEnumerable()). You can group then by Year and Month like this
var trendData =
(from d in ExpenseItemsViewableDirect
group d by new {
Year = d.Er_Approved_Date.Year,
Month = d.Er_Approved_Date.Month
} into g
select new
{
Year = g.Key.Year,
Month = g.Key.Month,
Total = g.Sum(x => x.Item_Amount),
AveragePerTrans = Math.Round(g.Average(x => x.Item_Amount),2)
}
).AsEnumerable()
.Select(g=>new {
Period = g.Year + "-" + g.Month,
Total = g.Total,
AveragePerTrans = g.AveragePerTrans
});
edit
The original query, from my response, was trying to do a concatenation between an int and a string, which is not translatable by EF into SQL statements. I could use SqlFunctions class, but the query it gets kind ugly. So I added AsEnumerable() after the grouping is made, which means that EF will execute the group query on server, will get the year, month, etc, but the custom projection is made over objects (what follows after AsEnumerable()).

When it comes to group by month i prefer to do this task in this way:
var sqlMinDate = (DateTime) SqlDateTime.MinValue;
var trendData = ExpenseItemsViewableDirect
.GroupBy(x => SqlFunctions.DateAdd("month", SqlFunctions.DateDiff("month", sqlMinDate, x.Er_Approved_Date), sqlMinDate))
.Select(x => new
{
Period = g.Key // DateTime type
})
As it keeps datetime type in the grouping result.

Similarly to what cryss wrote, I am doing the following for EF. Note we have to use EntityFunctions to be able to call all DB providers supported by EF. SqlFunctions only works for SQLServer.
var sqlMinDate = (DateTime) SqlDateTime.MinValue;
(from x in ExpenseItemsViewableDirect
let month = EntityFunctions.AddMonths(sqlMinDate, EntityFunctions.DiffMonths(sqlMinDate, x.Er_Approved_Date))
group d by month
into g
select new
{
Period = g.Key,
Total = g.Sum(x => x.Item_Amount),
AveragePerTrans = Math.Round(g.Average(x => x.Item_Amount),2)
}).Dump();
A taste of generated SQL (from a similar schema):
-- Region Parameters
DECLARE #p__linq__0 DateTime2 = '1753-01-01 00:00:00.0000000'
DECLARE #p__linq__1 DateTime2 = '1753-01-01 00:00:00.0000000'
-- EndRegion
SELECT
1 AS [C1],
[GroupBy1].[K1] AS [C2],
[GroupBy1].[A1] AS [C3]
FROM ( SELECT
[Project1].[C1] AS [K1],
FROM ( SELECT
DATEADD (month, DATEDIFF (month, #p__linq__1, [Extent1].[CreationDate]), #p__linq__0) AS [C1]
FROM [YourTable] AS [Extent1]
) AS [Project1]
GROUP BY [Project1].[C1]
) AS [GroupBy1]