Develop Reference

Why is Group By translating in Order By by Entity Framework?

I am using Entity Framework in .NET 7.
I have 3 entities:
Course that contains a ProfessorId among other things
Grade that has a CourseId among other things
Professor
I want to get all the courses that are assigned to a professor and have at least 1 grade associated with them and filter them in a Dictionary<string, CourseViewModel> where string is the semester.
I have written the following LINQ query:
var professorGradedCourses = _dbContext.Courses
.Where(course => course.ProfessorId == professorId && course.Grades.Any())
.Select(course => new CourseViewModel
{
Title = course.Title,
Semester = course.Semester,
})
.GroupBy(course => course.Semester)
.OrderBy(course => course.Key)
.ToDictionary(group => group.Key, group => group.ToList());
When that executes I get an exception saying it can't be translated.
If I remove the OrderBy and keep only the GroupBy, it works and the translated SQL in Microsoft SQL Server is:
SELECT [c].[Semester], [c].[Title]
FROM [Courses] AS [c]
WHERE [c].[ProfessorId] = #__professorId_0
AND EXISTS (SELECT 1
FROM [Grades] AS [g]
WHERE [c].[Id] = [g].[CourseId])
ORDER BY [c].[Semester]
As you can see it adds ORDER BY anyway, even though I have removed it and kept only GroupBy(). Can someone explain why is that? What if I wanted to order by descending would that be possible? Also the weird thing is that if I remove GroupBy() and keep only OrderBy() and replace the ToDictionary with ToList, it works and the exact same query is produced (only now I can't really use the results without further actions).

LINQ GroupBy :
Groups the elements of a sequence.
SQL GROUP BY :
A SELECT statement clause that divides the query result into groups of rows, usually by performing one or more aggregations on each group. The SELECT statement returns one row per group.
They aren't equivalent. The main difference is LINQ GroupBy return a collection by key, when SQL GROUP BY return ONE element (column) by key.
If the projection ask ONE element by key, then EF Core translate LINQ GroupBy to SQL GROUP BY :
// Get the number of course by semester
context
.Courses
.GroupBy(c => c.Semester)
.Select(cs => new { Semester = cs.Key, Count = cs.Count() })
.ToList();
Translated to :
SELECT [c].[Semester], COUNT(*) AS [Count]
FROM [Courses] AS [c]
GROUP BY [c].[Semester]
But if the projection ask several element, then EF Core translate LINQ GroupBy to SQL ORDER BY and group by itself.
context
.Courses
.Select(c => new { c.Id, c.Semester })
.GroupBy(c => c.Semester)
.ToDictionary(cs => cs.Key, cs => cs.ToList());
Translated to :
SELECT [c].[Semester], [c].[Id]
FROM [Courses] AS [c]
ORDER BY [c].[Semester]
If the result is :
Semester
Id
2023 S1
1
2023 S1
4
2023 S2
2
...
...
Then EF Core read like :
Read first row : Semester is "2023 S1"
No group
Then create a group and add the row in.
Read second row : Semester is "2023 S1"
The key is the same that precedent element
Then Add the row in the group
Read the third row : Semester is "2023 S2"
The key is different that precedent element
Then create a new group and the row in.
And so on...
You understand the interest of sorting.
About the error, I don't know that EF Core can't. The query sound legit. Maybe this should not be implemented at this time.
About that you try, to convert a sorted grouping enumeration to a dictionary. This is weird because the dictionary isn't sortable. Then this sorts elements and put them in loose.
If Dictionary seem sorted, it's a coincidence, not a feature. In intern, the dictionary sort element by key's has code, that is generally the sorted order... But not every time.
If you want a sorted dictionary, you can use SortedDictyonary. But it can be tricky if you need a custom sort rule, like :
context
.Courses
.Select(c => new { c.Id, c.Semester })
.GroupBy(c => c.Semester)
.ToImmutableSortedDictionary(cs => cs.Key, cs => cs.ToList(), new ReverseComparer<string>());
public class ReverseComparer<T> : IComparer<T>
{
private IComparer<T> _comparer = Comparer<T>.Default;
public int Compare(T? x, T? y)
{
return _comparer.Compare(x, y) * -1;
}
}

The exception you are encountering is most likely due to the fact that the OrderBy clause cannot be translated into SQL by Entity Framework. The OrderBy clause is executed in memory after the data has been retrieved from the database, which is why it works when you remove it and keep only the GroupBy clause.
However, if you want to order the dictionary by descending, you can simply call the Reverse method on the ToDictionary result:
var professorGradedCourses = _dbContext.Courses
.Where(course => course.ProfessorId == professorId && course.Grades.Any())
.Select(course => new CourseViewModel
{
Title = course.Title,
Semester = course.Semester,
})
.GroupBy(course => course.Semester)
.OrderByDescending(course => course.Key)
.ToDictionary(group => group.Key,
group => group.ToList())
.Reverse();
This way, the dictionary will be sorted in descending order based on the semester.
Give this a try and let me know how it works for you.
EDIT:
Converting the IEnumerable back to a Dictionary should work like this:
var professorGradedCourses = _dbContext.Courses
.Where(course => course.ProfessorId == professorId && course.Grades.Any())
.Select(course => new CourseViewModel
{
Title = course.Title,
Semester = course.Semester,
})
.GroupBy(course => course.Semester)
.OrderByDescending(course => course.Key)
.ToDictionary(group => group.Key,
group => group.ToList())
.Reverse()
.ToDictionary(pair => pair.Key,
pair => pair.Value);

Field expression GroupBy not returning included objects

In this code:
var dbrepayments = _context.Repayments.Include("Loan").Include("Loan.Borrower").Include("Loan.LoanProduct")
.Where(c => c.PaidOn == null && c.DateOfRepayment <= today)
.GroupBy(c => c.Loan.Id, (key, g) => g.OrderByDescending(c => c.Id).FirstOrDefault())
.OrderBy(c => c.DateOfRepayment);
_context is ApplicationDbContext type that I am using to get results from database using Code-First approach.
The problem is when I try to iterate through dbrepayments and get the value of Loan, Loan.Borrower, and Loan.LoanProduct objects they are showing as null. But when I remove GroupBy, these objects are returned correctly.

I'd wager the issue here is the element selector in your GroupBy statement:
(key, g) => g.OrderByDescending(c => c.Id).FirstOrDefault()
This didn't make a lot of sense when I first read it. You are taking repayments grouped by loan, but then trying to select just the last repayment for each loan? Followed by ordering those first repayments by date.
I believe this will give you the results you're looking for with the eager loaded relationships:
var dbrepayments = _context.Repayments.Include("Loan").Include("Loan.Borrower").Include("Loan.LoanProduct")
.Where(c => c.PaidOn == null && c.DateOfRepayment <= today)
.GroupBy(c => c.Loan.Id)
.Select(c => c.OrderByDescending(x => x.Id).FirstOrDefault())
.OrderBy(c => c.DateOfRepayment);
GroupBy will respect Include but if you are using a select expression, that overrides it. You cannot add Include inside the selector as that is working with IEnumerable of the expected results. Instead, group the results by loan as expected, but then Select from the results to get the latest repayment. This will give you a list of the latest repayments that you can then order.

Linq to List: Is there a way to include all the columns after applying a groupby on list

I have a list that is getting its values from the database and then this is converted to a datatable, but before doing that i want to apply groupby on the list and get all the columns instead of just a key and a value. After a new list is created using groupby datatable does not show all the columns instead it has only two columns which says capacity and count.
var groupedResults = Results.GroupBy(x => x.PROJECT_ID)
.Select(y => y.ToList())
.ToList();
Results is a list which contains around 14 columns or keys in this case with all project related properties project name, id etc. When I use Results and convert that to datatable I do not have any issue but when I use groupedResults list as shown above and convert that to a datatable it does not have all the 14 columns and an exception is raised as column not found. Is there a way to select all the keys as in original list.
Thank you

Have you tried wwith something like:
var groupedResults = Results.GroupBy(x => new {x.PROJECT_ID, obj = x})
.Select(y => y.Key.obj.ToList())
.ToList();

Try something like this
var e = Results.GroupBy(x => x.PROJECT_ID) .Select(y =>new { ProjectId = y.Key,
Count=y.Count()})
.Join(Results,x=>x.ProjectId,y=>y.PROJECT_ID,(x1,y1) => new
{x1.ProjectId,x1.Count,y1.Col1,y1.Col2 /* Add all columns you need */} )
.ToList();

how can i select last row in a group

I have a problem getting last row in a each group. I am using Linq query to retrieve the groups.
Here is my LINQ query.
return View(db.tblMsgs.OrderByDescending(a => a.Id)
.GroupBy(a => new { a.Sender, a.Receiver }).Select(x => x.FirstOrDefault())
.Where(a => a.Receiver == username).ToList());
using FirstOfDefault() I am getting first row in a group.
Using LastOrDefault() I am getting run time exception.

That's what I run into too, in some time now.
After a little bit of research I found out that only way that works as it must is to reverse the list that you want the get the last item of and get the first item.
The reason behind this is that SQL languages does not have a statement as SELECT BOTTOM but SELECT TOP. Hence our LastOrDefault query could not be translated into SQL.
The possible way of doing so is to OrderByDescending method.
return View(db.tblMsgs.OrderByDescending(a => a.Id)
.GroupBy(a => new { a.Sender, a.Receiver }).Select(x => x.OrderByDescending(y => y.SomeAttribute).FirstOrDefault())
.Where(a => a.Receiver == username).ToList());
Edit:
Only thing you should be choosy about is the column to order by. It can be the id field if it is an auto incremented number value, or an add date of the row(better be generated by server or it can cause problems).

Linq using Distinct() in C# Lambda Expression

SFC.OrderFormModifiedMonitoringRecords
.SelectMany(q => q.TimeModify, w => w.DateModify)
.Distinct()
.OrderBy(t => t)
.SelectMany(t => new { RowID = t.rowID, OFnum = t.OFNo });
It's Error did i missed something or is it Completely Coded The Wrong Way? After this i'm gonna use this on a Foreach method to gather up multiple data without the duplicates.

The delegate you pass to SelectMany must return an IEnumerable and is for collapsing multiple collections into one. So yes, something's definitely wrong here. I think you've confused it with Select which simply maps one collection to another.
Without knowing what your goal is, it's hard to know exactly how to fix it, but I'm guessing you want something like this:
SFC.OrderFormModifiedMonitoringRecords
.OrderBy(t => t.DateModify)
.ThenBy(t => t.TimeModify)
.Select(t => new { RowID = t.rowID, OFnum = t.OFNo })
.Distinct();
Or in query syntax:
(from t in SFC.OrderFormModifiedMonitoringRecords
orderby t.DateModify, t.TimeModify
select new { RowID = t.rowID, OFnum = t.OFNo })
.Distinct();
This will order the records by DateModify then by TimeModify, select two properties, rowID and OFNo and return only distinct pairs of values.