Develop Reference

How to get total available rows from paginated ef core query

Thanks in advance for taking time to read this question.
I have a view in my database, lets call it Members_VW
In my .net 5 API, I'm trying to get a paginated response for the list of members from the view with search parameters. I need to also return the total number of responses for the front end to know in how many pages the results will be returned in.
Currently the Members_VW is made with a query like:
select
col1, col2, col3
from
table1 1
inner join table2 2 on 1.key = 2.key
inner join tble3 3 on 3.key = 2.key
where
defaultcondition1 = '1'
and
defaultcondition2 = '2'
I referred to this answer and tried using CTE which ended up changing my view to using a query like this:
with cte1 as (
select
col1, col2, col3
from
table1 1
inner join table2 2 on 1.key = 2.key
inner join tble3 3 on 3.key = 2.key
where
defaultcondition1 = '1'
and
defaultcondition2 = '2')
cte2 as (
select count(*) over() from cte1 )
select
*
from
cte1, cte2
But this didn't work because it would always return the total number of rows in cte1 without any of the filters applied.
So, I continued to try to construct queries to return the total number of rows after the conditions are applied and found that this query works:
select
col1, col2, col3, count(*) over()
from
table1 1
inner join table2 2 on 1.key = 2.key
inner join tble3 3 on 3.key = 2.key
where
defaultcondition1 = '1'
and
defaultcondition2 = '2'
Currently, I'm trying to implement the same query with EF Core but am struggling to implement that.
I've tried implementing the solution provided here, but as one of the comments suggests, this implementation is no longer allowed.
I am trying to avoid an implementation where I use a raw query. Is there anyway to get the result from count(*) over() without using a raw query?
The following is my current implementation:
IQueryable<MembersVW> membersQuery = _context.MembersVW;
membersQuery = membersQuery.Where(u => u.MemberId == memberid);
membersQuery = membersQuery.OrderBy(m => m.MemberId).Skip(page * size).Take(size);
When I do:
membersQuery = membersQuery.Count()
I'm returned with the following error:
Error CS0029 Cannot implicitly convert type 'int' to 'System.Linq.IQueryable<PersonalPolicyAPI.Models.VwPersonalPolicyMember>'
Again, thanks for reading my question, appreciate any help you can offer. 🙏🏾

I've read your question about can it be done with one query. While I'm not aware of any way to do it with 1 query I can offer one more solution that will help with your concern about performance and 2 queries. I do this frequently. 😁 Try:
//execute both queries at the same time instead of sequentially
var countqry = membersQuery.CountAsync();
var pageqry = membersQuery.OrderBy(m => m.MemberId).Skip(page * size).Take(size).ToListAsync();
//wait for them both to complete
Task.WaitAll(countqry, pageqry);
//use the results
var count = countqry.Result;
var page = pageqry.Result;

membersQuery.Count() returns integer not the queryable
you can do
int count = membersQuery.Count();
List<MemberVW> = membersQuery.OrderBy(m => m.MemberId).Skip(page * size).Take(size).ToList();
and you can return with
public class MemberVwWithCount {
public int Count{get;set;}
public List<MemberVW> Members {get; set;}
}

You try to assign the Count Value, which is an Integer, to the variable of your query, which is an IQueryable. That's all there is to it.
If you want to do it in one single query, as you suggest in one of your comments, you can first execute the query to get all Entries, then count the result, and then filter the result with skip/take. This is most probably not the most efficient way to do this, but it should work.
I'd also suggest to use AsNoTracking() if you do not modify any data in this function/api.
EDIT:
I'd suggest this solution for now. The counting is fast, as it actually doesn't fetch any data and just counts the rows. It is still two queries tho, gonna try to combine it & edit my answer later.
var count = await yourContext.YourTable.CountAsync();
var data = await yourContext.YourTable
.OrderBy(x => x.YourProp)
.Skip(10).Take(10)
//.AsNoTracking()
.ToListAsync();
EDIT2:
Okay, so, I couldn't get it to just make on DB-Call yet, however, I could combine it syntactically. However, the approach in my first edit is easier to read and does basically the same. Still, gonna dig deeper into this, there's gotta be a funky way to do this.
var query = yourContext.YourTable.AsQueryable();
var result = await query.OrderBy(x => x.Prop)
.Select(x => new {Data = x, Count = query.Count()} )
.Skip(50).Take(50)
.AsNoTracking()
.ToListAsync();
var count = result.FirstOrDefault()?.Count ?? 0; //If empty/null return 0
var data = result.Select(x => x.Data).ToList();

In membersQuery = membersQuery.Count() line you are assigning integer value to a queryable list, which is incorrect. You can get the list item counts after your query like this i.e.
membersQuery = membersQuery.OrderBy(m => m.MemberId).Skip(page * size).Take(size);
int totalCount = membersQuery.Count();
To get count column in same list, you first need to add Count property in your MembersVW class and then use LINQ projection to add column value.
Solution-1:
memberQuery = membersQuery.Select(p => new MembersVW
{
col1 = p.col1
col2 = p.col2
col3 = p.col3
count = totalCount
});
Solution-2:
With LINQ foreach loop i.e.
membersQuery.ForEach(item =>
{
item.count = totalCount;
});

LINQ/EF Optimising group by query with count

I'm trying to do a very simple group by with a count but running into issues as LINQ is throwing some very horrible SQL which is timing out. I'm running EF7 and ASP NET 5.
Here's my code -
var counts = (from log in db.LogTable
where proj.Contains(log.projId)
group log by log.callTypeId into grp
select new
{
key = (from temp in callTypes where temp.Id == grp.Key select temp),
count = grp.Count()
});
callTypes is cached, LogTable contains over 1m records.
I've even tried removing the where clause, changing the key to select from the callTypes cache and just return the id, and also adding an orderby callTypeId but neither of these made any improvements.
Query being executed is as follows -
SELECT [log].[Id], [log].[callTypeId], [all other columns...]
FROM [LogTable] AS [log]
WHERE [log].[callTypeId] IN (1, 2, 3, 4, 5)
Does anyone have any idea whether there is a way I can force this to "SELECT callTypeId, count(*) FROM LogTable WHERE... GROUP BY callTypeId" to stop it from returning all the records and doing the grouping on the application side please? Or do I need to revert to Stored Procedures ??
UPDATE -
As explained above, replacing (from temp in callTypes where temp.Id == grp.Key select temp) with just grp.Key gives the same query so this is not the issue.
Many thanks!

How to get records from tables in custom order of ids?

I have some Ids store in below variable:
List<int> Ids;
Now I want to get records based on above Ids but with same order as it is in above Ids.
For eg: Records are like this in database:
Employee:
Id
1
2
3
4
5
Now if Ids array holds Ids like this : 4,2,5,3,1 then I am trying to get records in this order order only:
Query:
var data = context.Employee.Where(t => Ids.Contains(t.Id)).ToList();
But above query is giving me output like it is in table:
Id
1
2
3
4
5
Expected output :
Id
4
2
5
3
1
Update:I have already tried this below solution but as this is entity framework it didn't work out:
var data = context.Employee.Where(t => Ids.Contains(t.Id))
.OrderBy(d => Ids.IndexOf(d.Id)).ToList();
For above solution to make it working I have to add to list :
var data = context.Employee.Where(t => Ids.Contains(t.Id)).ToList()
.OrderBy(d => Ids.IndexOf(d.Id)).ToList();
But I don't want to load data in memory and then filter out my record.

Since the order in which the data is returned when you do not specify an ORDER BY is not determined, you have to add an ORDER BY to indicate how you want it sorted. Unfortunately you have to order based on objects/values in-memory, and cannot use that to order in your SQL query.
Therefore, the best you can do is to order in-memory once the data is retrieved from the database.
var data = context.Employee
// Add a criteria that we only want the known ids
.Where(t => Ids.Contains(t.Id))
// Anything after this is done in-memory instead of by the database
.AsEnumerable()
// Sort the results, in-memory
.OrderBy(d => Ids.IndexOf(d.Id))
// Materialize into a list
.ToList();

Without stored procedures you can use Union and ?: that are both canonical functions.
I can't immagine other ways.
?:
You can use it to assign a weigth to each id value then order by the weigth. Also, you have to generate ?: using dynamic linq.
What is the equivalent of "CASE WHEN THEN" (T-SQL) with Entity Framework?
Dynamically generate LINQ queries
Union
I think this is the more simple way to obtain it. In this case you can add a Where/Union for each Id.
EDIT 1
About using Union you can use code similar to this
IQueryable<Foo> query = context.Foos.AsQueryable();
List<int> Ids = new List<int>();
Ids.AddRange(new[] {3,2,1});
bool first = true;
foreach (int id in Ids)
{
if (first)
{
query = query.Where(_ => _.FooId == id);
first = false;
}
else
{
query = query.Union(context.Foos.Where(_ => _.FooId == id));
}
}
var results = query.ToList();
This generate the followiong query
SELECT
[Distinct2].[C1] AS [C1]
FROM ( SELECT DISTINCT
[UnionAll2].[C1] AS [C1]
FROM (SELECT
[Distinct1].[C1] AS [C1]
FROM ( SELECT DISTINCT
[UnionAll1].[FooId] AS [C1]
FROM (SELECT
[Extent1].[FooId] AS [FooId]
FROM [Foos] AS [Extent1]
WHERE [Extent1].[FooId] = #p__linq__0
UNION ALL
SELECT
[Extent2].[FooId] AS [FooId]
FROM [Foos] AS [Extent2]
WHERE [Extent2].[FooId] = #p__linq__1) AS [UnionAll1]
) AS [Distinct1]
UNION ALL
SELECT
[Extent3].[FooId] AS [FooId]
FROM [Foos] AS [Extent3]
WHERE [Extent3].[FooId] = #p__linq__2) AS [UnionAll2]
) AS [Distinct2]
p__linq__0 = 3
p__linq__1 = 2
p__linq__2 = 1
EDIT 2
I think the best approach is in memory approach because it has the same network load, EF does not generate the ugly query that could not work on databases different from SQL Server and code is more readable. In your particular application could be that union/where is better. So, generally I would suggest you to try memory approach then, if you have [performance] issues, you can check if union/where is better.

one query with groupby / count / select new

I have to make in c# a query with linq to sql. I can handle it in sql but in linq to
sql is the result not what I wanted to get.
So there is a table with:
a day, in datetime with date and time
and a kind of id
I have to count the ids for each date, the time isn't important. So the result
should be something like:
day: 2013-11-12 amountIDs: 4
People said to me, I can make a select new query and in this query I can set the day
and could count the ids, or I make a group by day. I read similar question, but it doesn't work in my case.
Could somebody help me?
I tried it with the statement below, but the days have to be grouped, so now the output is foreach datetime, like this
day: 12.12.2013 12:00:00 amountIDs: 1
day: 12.12.2013 12:10:10 amountIDs: 1
In sql I made this statement:
SELECT CONVERT(VARCHAR(20), data.dayandtime, 106) AS day, count(data.amountIds) as ids
FROM data
WHERE ( data.dayandtime >= DATEADD(day, -28, getdate()) AND (data.type = 100) AND (data.isSomething = 0) )
GROUP BY CONVERT(VARCHAR(20), data.dayandtime, 106), data.isSomthing and it works.
I saw similar cases, where people made a : from-select-new xyz statement, than I made a view of it and tried to group just the view. Like this
var query = data.GroupBy(g => g.day.Value).ToList();

var qry = from data in dbContext
group data by data.day into dataGrpd
select new
{
day= dataGrpd.Key,
amountIDs= dataGrpd.Select(x => x.Id).Distinct().Count()
};
Check This

Entity Framework: Efficiently grouping by month

I've done a bit of research on this, and the best I've found so far is to use an Asenumerable on the whole dataset, so that the filtering occurs in linq to objects rather than on the DB. I'm using the latest EF.
My working (but very slow) code is:
var trendData =
from d in ExpenseItemsViewableDirect.AsEnumerable()
group d by new {Period = d.Er_Approved_Date.Year.ToString() + "-" + d.Er_Approved_Date.Month.ToString("00") } into g
select new
{
Period = g.Key.Period,
Total = g.Sum(x => x.Item_Amount),
AveragePerTrans = Math.Round(g.Average(x => x.Item_Amount),2)
};
This gives me months in format YYYY-MM, along with the total amount and average amount. However it takes several minutes every time.
My other workaround is to do an update query in SQL so I have a YYYYMM field to group natively by. Changing the DB isn't an easy fix however so any suggestions would be appreciated.
The thread I found the above code idea (http://stackoverflow.com/questions/1059737/group-by-weeks-in-linq-to-entities) mentions 'waiting until .NET 4.0'. Is there anything recently introduced that helps in this situation?

The reason for poor performance is that the whole table is fetched into memory (AsEnumerable()). You can group then by Year and Month like this
var trendData =
(from d in ExpenseItemsViewableDirect
group d by new {
Year = d.Er_Approved_Date.Year,
Month = d.Er_Approved_Date.Month
} into g
select new
{
Year = g.Key.Year,
Month = g.Key.Month,
Total = g.Sum(x => x.Item_Amount),
AveragePerTrans = Math.Round(g.Average(x => x.Item_Amount),2)
}
).AsEnumerable()
.Select(g=>new {
Period = g.Year + "-" + g.Month,
Total = g.Total,
AveragePerTrans = g.AveragePerTrans
});
edit
The original query, from my response, was trying to do a concatenation between an int and a string, which is not translatable by EF into SQL statements. I could use SqlFunctions class, but the query it gets kind ugly. So I added AsEnumerable() after the grouping is made, which means that EF will execute the group query on server, will get the year, month, etc, but the custom projection is made over objects (what follows after AsEnumerable()).

When it comes to group by month i prefer to do this task in this way:
var sqlMinDate = (DateTime) SqlDateTime.MinValue;
var trendData = ExpenseItemsViewableDirect
.GroupBy(x => SqlFunctions.DateAdd("month", SqlFunctions.DateDiff("month", sqlMinDate, x.Er_Approved_Date), sqlMinDate))
.Select(x => new
{
Period = g.Key // DateTime type
})
As it keeps datetime type in the grouping result.

Similarly to what cryss wrote, I am doing the following for EF. Note we have to use EntityFunctions to be able to call all DB providers supported by EF. SqlFunctions only works for SQLServer.
var sqlMinDate = (DateTime) SqlDateTime.MinValue;
(from x in ExpenseItemsViewableDirect
let month = EntityFunctions.AddMonths(sqlMinDate, EntityFunctions.DiffMonths(sqlMinDate, x.Er_Approved_Date))
group d by month
into g
select new
{
Period = g.Key,
Total = g.Sum(x => x.Item_Amount),
AveragePerTrans = Math.Round(g.Average(x => x.Item_Amount),2)
}).Dump();
A taste of generated SQL (from a similar schema):
-- Region Parameters
DECLARE #p__linq__0 DateTime2 = '1753-01-01 00:00:00.0000000'
DECLARE #p__linq__1 DateTime2 = '1753-01-01 00:00:00.0000000'
-- EndRegion
SELECT
1 AS [C1],
[GroupBy1].[K1] AS [C2],
[GroupBy1].[A1] AS [C3]
FROM ( SELECT
[Project1].[C1] AS [K1],
FROM ( SELECT
DATEADD (month, DATEDIFF (month, #p__linq__1, [Extent1].[CreationDate]), #p__linq__0) AS [C1]
FROM [YourTable] AS [Extent1]
) AS [Project1]
GROUP BY [Project1].[C1]
) AS [GroupBy1]