Develop Reference

How to get total available rows from paginated ef core query

Thanks in advance for taking time to read this question.
I have a view in my database, lets call it Members_VW
In my .net 5 API, I'm trying to get a paginated response for the list of members from the view with search parameters. I need to also return the total number of responses for the front end to know in how many pages the results will be returned in.
Currently the Members_VW is made with a query like:
select
col1, col2, col3
from
table1 1
inner join table2 2 on 1.key = 2.key
inner join tble3 3 on 3.key = 2.key
where
defaultcondition1 = '1'
and
defaultcondition2 = '2'
I referred to this answer and tried using CTE which ended up changing my view to using a query like this:
with cte1 as (
select
col1, col2, col3
from
table1 1
inner join table2 2 on 1.key = 2.key
inner join tble3 3 on 3.key = 2.key
where
defaultcondition1 = '1'
and
defaultcondition2 = '2')
cte2 as (
select count(*) over() from cte1 )
select
*
from
cte1, cte2
But this didn't work because it would always return the total number of rows in cte1 without any of the filters applied.
So, I continued to try to construct queries to return the total number of rows after the conditions are applied and found that this query works:
select
col1, col2, col3, count(*) over()
from
table1 1
inner join table2 2 on 1.key = 2.key
inner join tble3 3 on 3.key = 2.key
where
defaultcondition1 = '1'
and
defaultcondition2 = '2'
Currently, I'm trying to implement the same query with EF Core but am struggling to implement that.
I've tried implementing the solution provided here, but as one of the comments suggests, this implementation is no longer allowed.
I am trying to avoid an implementation where I use a raw query. Is there anyway to get the result from count(*) over() without using a raw query?
The following is my current implementation:
IQueryable<MembersVW> membersQuery = _context.MembersVW;
membersQuery = membersQuery.Where(u => u.MemberId == memberid);
membersQuery = membersQuery.OrderBy(m => m.MemberId).Skip(page * size).Take(size);
When I do:
membersQuery = membersQuery.Count()
I'm returned with the following error:
Error CS0029 Cannot implicitly convert type 'int' to 'System.Linq.IQueryable<PersonalPolicyAPI.Models.VwPersonalPolicyMember>'
Again, thanks for reading my question, appreciate any help you can offer. 🙏🏾

I've read your question about can it be done with one query. While I'm not aware of any way to do it with 1 query I can offer one more solution that will help with your concern about performance and 2 queries. I do this frequently. 😁 Try:
//execute both queries at the same time instead of sequentially
var countqry = membersQuery.CountAsync();
var pageqry = membersQuery.OrderBy(m => m.MemberId).Skip(page * size).Take(size).ToListAsync();
//wait for them both to complete
Task.WaitAll(countqry, pageqry);
//use the results
var count = countqry.Result;
var page = pageqry.Result;

membersQuery.Count() returns integer not the queryable
you can do
int count = membersQuery.Count();
List<MemberVW> = membersQuery.OrderBy(m => m.MemberId).Skip(page * size).Take(size).ToList();
and you can return with
public class MemberVwWithCount {
public int Count{get;set;}
public List<MemberVW> Members {get; set;}
}

You try to assign the Count Value, which is an Integer, to the variable of your query, which is an IQueryable. That's all there is to it.
If you want to do it in one single query, as you suggest in one of your comments, you can first execute the query to get all Entries, then count the result, and then filter the result with skip/take. This is most probably not the most efficient way to do this, but it should work.
I'd also suggest to use AsNoTracking() if you do not modify any data in this function/api.
EDIT:
I'd suggest this solution for now. The counting is fast, as it actually doesn't fetch any data and just counts the rows. It is still two queries tho, gonna try to combine it & edit my answer later.
var count = await yourContext.YourTable.CountAsync();
var data = await yourContext.YourTable
.OrderBy(x => x.YourProp)
.Skip(10).Take(10)
//.AsNoTracking()
.ToListAsync();
EDIT2:
Okay, so, I couldn't get it to just make on DB-Call yet, however, I could combine it syntactically. However, the approach in my first edit is easier to read and does basically the same. Still, gonna dig deeper into this, there's gotta be a funky way to do this.
var query = yourContext.YourTable.AsQueryable();
var result = await query.OrderBy(x => x.Prop)
.Select(x => new {Data = x, Count = query.Count()} )
.Skip(50).Take(50)
.AsNoTracking()
.ToListAsync();
var count = result.FirstOrDefault()?.Count ?? 0; //If empty/null return 0
var data = result.Select(x => x.Data).ToList();

In membersQuery = membersQuery.Count() line you are assigning integer value to a queryable list, which is incorrect. You can get the list item counts after your query like this i.e.
membersQuery = membersQuery.OrderBy(m => m.MemberId).Skip(page * size).Take(size);
int totalCount = membersQuery.Count();
To get count column in same list, you first need to add Count property in your MembersVW class and then use LINQ projection to add column value.
Solution-1:
memberQuery = membersQuery.Select(p => new MembersVW
{
col1 = p.col1
col2 = p.col2
col3 = p.col3
count = totalCount
});
Solution-2:
With LINQ foreach loop i.e.
membersQuery.ForEach(item =>
{
item.count = totalCount;
});

How to get records from tables in custom order of ids?

I have some Ids store in below variable:
List<int> Ids;
Now I want to get records based on above Ids but with same order as it is in above Ids.
For eg: Records are like this in database:
Employee:
Id
1
2
3
4
5
Now if Ids array holds Ids like this : 4,2,5,3,1 then I am trying to get records in this order order only:
Query:
var data = context.Employee.Where(t => Ids.Contains(t.Id)).ToList();
But above query is giving me output like it is in table:
Id
1
2
3
4
5
Expected output :
Id
4
2
5
3
1
Update:I have already tried this below solution but as this is entity framework it didn't work out:
var data = context.Employee.Where(t => Ids.Contains(t.Id))
.OrderBy(d => Ids.IndexOf(d.Id)).ToList();
For above solution to make it working I have to add to list :
var data = context.Employee.Where(t => Ids.Contains(t.Id)).ToList()
.OrderBy(d => Ids.IndexOf(d.Id)).ToList();
But I don't want to load data in memory and then filter out my record.

Since the order in which the data is returned when you do not specify an ORDER BY is not determined, you have to add an ORDER BY to indicate how you want it sorted. Unfortunately you have to order based on objects/values in-memory, and cannot use that to order in your SQL query.
Therefore, the best you can do is to order in-memory once the data is retrieved from the database.
var data = context.Employee
// Add a criteria that we only want the known ids
.Where(t => Ids.Contains(t.Id))
// Anything after this is done in-memory instead of by the database
.AsEnumerable()
// Sort the results, in-memory
.OrderBy(d => Ids.IndexOf(d.Id))
// Materialize into a list
.ToList();

Without stored procedures you can use Union and ?: that are both canonical functions.
I can't immagine other ways.
?:
You can use it to assign a weigth to each id value then order by the weigth. Also, you have to generate ?: using dynamic linq.
What is the equivalent of "CASE WHEN THEN" (T-SQL) with Entity Framework?
Dynamically generate LINQ queries
Union
I think this is the more simple way to obtain it. In this case you can add a Where/Union for each Id.
EDIT 1
About using Union you can use code similar to this
IQueryable<Foo> query = context.Foos.AsQueryable();
List<int> Ids = new List<int>();
Ids.AddRange(new[] {3,2,1});
bool first = true;
foreach (int id in Ids)
{
if (first)
{
query = query.Where(_ => _.FooId == id);
first = false;
}
else
{
query = query.Union(context.Foos.Where(_ => _.FooId == id));
}
}
var results = query.ToList();
This generate the followiong query
SELECT
[Distinct2].[C1] AS [C1]
FROM ( SELECT DISTINCT
[UnionAll2].[C1] AS [C1]
FROM (SELECT
[Distinct1].[C1] AS [C1]
FROM ( SELECT DISTINCT
[UnionAll1].[FooId] AS [C1]
FROM (SELECT
[Extent1].[FooId] AS [FooId]
FROM [Foos] AS [Extent1]
WHERE [Extent1].[FooId] = #p__linq__0
UNION ALL
SELECT
[Extent2].[FooId] AS [FooId]
FROM [Foos] AS [Extent2]
WHERE [Extent2].[FooId] = #p__linq__1) AS [UnionAll1]
) AS [Distinct1]
UNION ALL
SELECT
[Extent3].[FooId] AS [FooId]
FROM [Foos] AS [Extent3]
WHERE [Extent3].[FooId] = #p__linq__2) AS [UnionAll2]
) AS [Distinct2]
p__linq__0 = 3
p__linq__1 = 2
p__linq__2 = 1
EDIT 2
I think the best approach is in memory approach because it has the same network load, EF does not generate the ugly query that could not work on databases different from SQL Server and code is more readable. In your particular application could be that union/where is better. So, generally I would suggest you to try memory approach then, if you have [performance] issues, you can check if union/where is better.

Select row with max value after group in Linq Nhibernate

I have a table:
Table { Id, Date, Number, Bool }
I need to group it by Number, select the row with max Date inside each group, and retrieve Id for each group. In the end I need to filter that to only have records that are !Bool. I am trying to do this with Linq Nhibernate.
This SQL seems to be doing what I want:
select Id from
(select MAX(Date) as Dt, Number as N from Table group by Number) t, Table table
where table.Date = t.Dt and table.Number = t.N and table.Bool = 0
but turns out NHibernate does not allow for subqueries to be in from. How do I write this with Linq Nhibernate?
It's also quite important for it to be efficient, so I would rather avoid having subqueries in select or where if they iterate over the whole set and (N+1) query problem.
The straightforward approach doesn't work either:
Session.Query<Table>().GroupBy(x => x.Number)
.Select(x => x.Where(y => y.Date == x.Max(z => z.Date)))...

Linq Query to Get Distinct Records from Two Tables

I have two Tables - tblExpenses and tblCategories as follows
tblExpenses
ID (PK),
Place,
DateSpent,
CategoryID (FK)
tblCategory
ID (PK),
Name
I tried various LINQ approaches to get all distinct records from the above two tables but not with much success. I tried using UNION and DISTINCT but it didnt work.
The above two tables are defined in my Model section of my project which in turn will create tables in SQLite. I need to retrieve all the distinct records from both the tables to display values in gridview.
Kindly provide me some inputs to accomplish this task. I did some research to find answer to this question but nothing seemed close to what I wanted. Excuse me if I duplicated this question.
Here is the UNION, DISTINCT approaches I tried:
DISTINCT # ==> Gives me Repetitive values
(from exp in db.Table<tblExpenses >()
from cat in db.Table<tblCategory>()
select new { exp.Id, exp.CategoryId, exp.DateSpent, exp.Expense, exp.Place, cat.Name }).Distinct();
UNION # ==> Got an error while using UNION

I think union already does the distict when you join the two tables you can try somethin like
var query=(from c in db.tblExpenses select c).Concat(from c in
db.tblCategory select c).Distinct().ToList();

You will always get DISTINCT records, since you are selecting the tblExpenses.ID too. (Unless there are multiple categories with the same ID. But that of course would be really, really bad design.)
Remember, when making a JOIN in LINQ, both field names and data types should be the same. Is the field tblExpenses.CategoryID a nullable field?
If so, try this JOIN:
db.Table<tblExpenses>()
.Join(db.Table<tblCategory>(),
exp => new { exp.CategoryId },
cat => new { CategoryId = (int?)cat.ID },
(exp, cat) => new {
exp.Id,
exp.CategoryId,
exp.DateSpent,
exp.Expense,
exp.Place,
cat.Name
})
.Select(j => new {
j.Id,
j.CategoryId,
j.DateSpent,
j.Expense,
j.Place,
j.Name
});

You can try this queries:
A SELECT DISTINCT query like this:
SELECT DISTINCT Name FROM tblCategory INNER JOIN tblExpenses ON tblCategory.categoryID = tblExpenses.categoryID;
limits the results to unique values in the output field. The query results are not updateable.
or
A SELECT DISTINCTROW query like this:
SELECT DISTINCTROW Name FROM tblCategory INNER JOIN tblExpenses ON tblCategory.categoryID = tblExpenses.categoryID;<br/><br/>
looks at the entire underlying tables, not just the output fields, to find unique rows.
reference:http://www.fmsinc.com/microsoftaccess/query/distinct_vs_distinctrow/unique_values_records.asp

securing query authorization in linq-to-sql

I have 3 tables: the User table, the Records table and the UserRecords table.
The columns are so:
UserTable
UserID | OtherUserDataFields
RecordsTable
RecordID | OtherRecordDataFields
UserRecords
UserID | RecordID
The UserRecords table tells me which users have authorization on which record. I have a function that updates the RecordsTable by receiving 2 paramaters: a record TheRecord (which contains a RecordID field) and a UserID.
I'm writing a query in the data context MyDC to fetch the record with the RecordID supplied in the parameter and test if the user is authorized on that record like this:
var RecordToUpdate = (
from r in MyDC.RecordsTable
from u in MyDC.UserRecords
where r.RecordID == TheRecord.RecordID && TheRecord.RecordID == u.RecordID
where u.UserID == TheUserID
select r).SingleOrDefault();
Will this ensure me that only records the user is authorized will be fetched? I want to avoid cases where users maliciously send a record they're not authorized on and make changes to these unauthorized records.
Thanks for your advice.

I agree with Anand, you will need a linq query:
var filterUserRecord = from u in MyDC.UserRecords
where u.UserID == TheUserID
select u;
var q1 = from r in MyDC.RecordsTable
where r.RecordID = TheRecordID
where filterUserRecord.Any(f => f.RecordID == r.RecordID)
select r;
This will be converted to the SQL query like the following:
SELECT * FROM RecordsTable rt WHERE rt.RecordID = TheRecordID AND EXISTS
(SELECT recordId FROM UserRecords ur WHERE ur.userId = TheUserID AND ur.recordID = rt.recordID)
Note that those are IQueryable<T>s and linq queries on it will produce another IQueryable<T> which will contain expressions to be wholly translated to SQL (or whatever the backend is) instead of naively evaluating it at client side.

Well I think your problem could be solved by subquery
sql:
select * from RecordsTable where recordId in
(select recordId from UserRecords where userId = #someUserId)
It could be represented in Linq as following
var filterUserRecord = from u in MyDC.UserRecords
where u.UserID == TheUserID
select u
var q1 = from r in MyDC.RecordsTable
where filterUserRecord.Any(f => f.RecordID == r.RecordID)
Details for subquery in Linq - Read from here

Perhaps read the article here which discusses how SQL Injection (which I assume should be the main security concern here) is handled in LINQ scenarios.
There is also a nice article here regarding Microsoft's security considerations for EF. It's worth a read for anyone developing with these tools!
[Edit] With regards to your last comment, you can use queries similar to those already on this page. To condense a little: If your database is normalised, to the extent that the RecordId is a unique primary key, you can bypass joins to make a query which reads a little better:
var targetRecords =
from userRecords in MyDC.UserRecords
where userRecords.UserTable.UserID == TheUserID
&& userRecords.RecordsTable.RecordID == TheRecord.RecordID
select userRecords;
var targetRecordsResult = targetRecords.SingleOrDefault();
I have separated the query from it's result 'var' here to point out that 'targetRecords' does NOT get evaluated until you call SingleOrDefault on it in order to assign to targetRecordsResult. You can of course wrap this into one statement if you wish.
If, as mentioned, your RecordID is a Unique Primary Key, you will either get the matching Record back, or null. Note that if this isn't the case, i.e. more than on Record could have the same ID, then the SingleOrDefault call can fail. If your database is designed like that, you would have to use a query more like the one which Anand has specified. That is slightly more verbose, but will return you ANY Record with a matching id for that specific user.
In terms of security, note that your SQL statement will get compiled containing the UserID, making it very difficult to tamper with. Hence my point that in this case, the scope and exposure of UserID is your main concern. If, like you have stated, the user (and any potential malitious user) has no access to the variable (via property exposure etc) then this should be more than suitable for your needs.

Try this:
var RecordToUpdate = (from u in MyDC.UserRecords
where u.UserTable.UserID == TheUserID
and u.RecordsTable.RecordID == TheRecord.RecordID).SingleOrDefault();
This will return the result for query which has specified UserID and RecordID.