Develop Reference

Linq tolist() count returns different value from translated sql

I have a situation where the translated sql direct form Visual Studio is returning a different number of records from the Linq. I am expecting 4 items but in the count() of the resulting list I only have 1. This is the case despite creating more records - it always returns 1.
db.DCLVUnknowns.Where(x => x.DCLVUnknownId == Report.DCLid).ToList();
SELECT
[Extent1].[DCLVUnknownId] AS [DCLVUnknownId],
[Extent1].[Gender] AS [Gender],
[Extent1].[Height] AS [Height],
[Extent1].[Weight] AS [Weight],
[Extent1].[Age] AS [Age],
[Extent1].[Race] AS [Race],
[Extent1].[DCLid] AS [DCLid]
FROM [dbo].[DCLVUnknown] AS [Extent1]
Strange thing is I have the same linq expression running fine for other entities and there is no problem. It is consistently happening at the same spot every time.

db.DCLVUnknowns
Is a query of the entire table, not the query for what you want.
If you want to inspect the IQueryable of the full query, try:
var results = db.DCLVUnknowns.Where(x => x.DCLVUnknownId == Report.DCLid);
var theResultSet = results.ToList();
Here results should translate as roughly:
SELECT
[Extent1].[DCLVUnknownId] AS [DCLVUnknownId],
[Extent1].[Gender] AS [Gender],
[Extent1].[Height] AS [Height],
[Extent1].[Weight] AS [Weight],
[Extent1].[Age] AS [Age],
[Extent1].[Race] AS [Race],
[Extent1].[DCLid] AS [DCLid]
FROM [dbo].[DCLVUnknown] AS [Extent1]
WHERE [Extent1].[DCLVUnknownId] = DCLid
Assuming DCLVUnknownId is a PK/ Identity, you should see one result in theResultSet.

Entity Framework Core LINQ trouble creating (select case exists) queries

I'm trying to list all items with an extra column describing whether it's owned by the current user.
So I'm looking for a Linq query that generates something like the following SQL:
SELECT *,
CASE WHEN
EXISTS (
SELECT NULL FROM OwnedItems
WHERE OwnedItems.UserId = #UserId AND OwnedItems.ItemId = Items.Id
)
THEN 'true'
ELSE 'false'
END AS Owned
FROM Items;
According to the internet as well as a successful LinqPad experiment this code should work.
from item in Items
select new
{
Owned = OwnedItems.Any(own => own.UserId == userId && own.ItemId == item.Id),
Item = item
}
In LinqPad this code generates the exact same SQL as I want. But in my project it does something completely different.
My code is a .Net Core 2.1 project using Entity Framework Core 2.1. Since it is a Core project I can't directly test it in LinqPad as it isn't supported yet.
In my project this code results in an unfiltered SELECT statement querying every Item, then for each of them a separate query to check if it exists in the OwnedItems table. Like this:
1 instance of this query runs:
Executed DbCommand (68ms) [Parameters=[], CommandType='Text', CommandTimeout='30']
SELECT *
FROM [Items] AS [item]
Followed by hundreds of these queries taking multiple seconds to run:
Executed DbCommand (32ms) [Parameters=[#__userId_0='?' (DbType = Int32), #_outer_Id='?' (DbType = Int32)], CommandType='Text', CommandTimeout='30']
SELECT CASE
WHEN EXISTS (
SELECT 1
FROM [OwnedItems] AS [ownedItems]
WHERE ([ownedItems].[UserId] = #__userId_0) AND ([ownedItems].[ItemId] = #_outer_Id))
THEN CAST(1 AS BIT) ELSE CAST(0 AS BIT)
END
Some further info, maybe it helps:
If I use the same line as part of a where clause, it works perfectly.
var q = from item in Items
where OwnedItems.Any(o => o.UserId == userId && o.ItemId == item.Id)
select item;
The above linq results in this nice sql:
SELECT *
FROM [Items] AS [item]
WHERE EXISTS (
SELECT 1
FROM [OwnedItems] AS [o]
WHERE ([o].[UserId] = #__userId_0) AND ([o].[ItemId] = [item].[Id]))
Notes:
The above code has been mangled manually so there might be typos in there. Please disregard them.
I understand that this particular query can be done using a left join and checking for nulls but my actual one is more complex and I need (nested) exists clauses.
UPDATE FOR SOLUTION
As #KorsG pointed out if the Item isn't materialized, the proper query is generated.
What I found is that not materializing Item works even if I write the following:
from item in Items
select new
{
Owned = OwnedItems.Any(own => own.UserId == userId && own.ItemId == item.Id),
// Item = item //THIS LINE GENERATES BAD QUERY
Item = new Item {
Id = item.Id,
Name = item.Name,
...
[Literally every single property listed one by one] = item.CorrespondingProperty
...
}
}
So I can actually materialize the full item, I just have to explicitly type every last property. FUN!

You probably need to enable eager loading for the "OwnedItems" navigation property in the query:
https://learn.microsoft.com/en-us/ef/core/querying/related-data#eager-loading
Please post your full linq query if I should give an example.
UPDATE 1
Seems like subqueries have N+1 problems in EF Core and it will maybe be fixed in version 3.
Reference: https://github.com/aspnet/EntityFrameworkCore/issues/10001
UPDATE 2
If you don't need to fully materialize "Items" you should be able to do something like this where you create an anoymous object instead which should "trick" EF into what you want:
from item in Items
select new
{
Owned = OwnedItems.Any(own => own.UserId == userId && own.ItemId == item.Id),
Item = new { Id = item.Id, Name = item.Name }
}
Reference: https://github.com/aspnet/EntityFrameworkCore/issues/11186

You need to tell EF to load the related data, in this case the OwnedItems table.
One way to go about this is to Include the related table. If there's a foreign key that links tables it could be easily done like this:
var dataWithRelatedData = db_context.Items.Include(x => x.OwnedItems).Select ...
Another way to avoid a high number of round trips to the database is to load both datasets in separate queries and then merge them in memory. So you would first make a query to Items then with the data returned another query to OwnedItems and lastly merge them into a single list of objects. This would do just 2 calls to the database, therefore improving performance.

Entity Framework Core; using ORDER BY in query against a (MS) SQL Server

I'm trying to use the following query in combination with Entity Framework Core against a Microsoft SQL Server 2016:
SELECT [a], [b], [c]
FROM [x]
WHERE [a] = {0}
ORDER BY [b]
I use this query like so:
context.MySet.AsNoTracking()
.FromSql(MyQuery, aValue)
.Skip(pageSize * page)
.Take(pageSize)
.Select(x => x.ToJsonDictionary())
.ToList()
I use this in a .NET Core REST API with pagination and I'd like to have the records sorted (alphabetically) to make the pagination more usable.
I get the following error when executing the above statement:
The ORDER BY clause is invalid in views, inline functions, derived
tables, subqueries, and common table expressions, unless TOP, OFFSET
or FOR XML is also specified.Invalid usage of the option NEXT in the
FETCH statement. The ORDER BY clause is invalid in views, inline
functions, derived tables, subqueries, and common table expressions,
unless TOP, OFFSET or FOR XML is also specified.Invalid usage of the
option NEXT in the FETCH statement.
Looking for similar issues I found these some other posts (1, 2, 3) but none of which where used in combination with EF Core and/or they were using it in a different context which does not apply in my case (e.g. subquery).
I tried to use the .OrderBy(..) syntax of EF instead of in the ORDER BY in the query but this doesn't solve the problem. I also tried adding TOP 100 PERCENT after the SELECT in the query in combination with the ORDRE BY; this worked but didn't order the column. It just got ignored. This limitation is described under the EF Limitations. I also found this post that replace the TOP 100 PERCENT... with TOP 99.99 PERCENT... or TOP 9999999... `. This seems like it should work but it doesn't 'feel' right.
This issue in general is further explained here.
Summary: It is not advisable to use ORDER BY in Views. Use ORDER BY
outside the views. In fact, the correct design will imply the same. If
you use TOP along with Views, there is a good chance that View will
not return all the rows of the table or will ignore ORDER BY
completely.
Further I'm confused by the word "view". For me, the term views refers to the usage of the ones created by the CREATE VIEW .. syntax. Is a plain, 'normal' SQL query also considered a view? Or is EF Core wrapping the request in some sort of view and this is the real issue causing this error?
I'm not sure, but so far all the 'solutions' I found seem kind of 'hacky'.
Thoughts?

Let's simplify things a bit. Here's what I came up for testing. I've also added some code for printing the generated sql from EF queries.
class Program
{
static void Main(string[] args)
{
DbClient context = new DbClient();
var rawSql = "select [Id], [Title] from Post order by [Title]";
var query = context.Posts.AsNoTracking()
.FromSql(rawSql)
.Skip(1)
.Take(4)
.OrderBy(x => x.Title);
var generated = query.ToSql();
var results = query.ToList();
}
}
class DbClient : DbContext
{
public DbSet<Post> Posts { get; set; }
protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
{
optionsBuilder.UseSqlServer("conn_string");
}
}
class Post
{
public int Id { get; set; }
public string Title { get; set; }
public override string ToString() => $"{Id} | {Title}";
}
When we look at the value of generated we see what the sql of the query is:
SELECT [t].[Id], [t].[Title]
FROM (
SELECT [p].[Id], [p].[Title]
FROM (
select [Id], [Title] from Post order by [Title]
) AS [p]
ORDER BY (SELECT 1)
OFFSET 1 ROWS FETCH NEXT 4 ROWS ONLY
) AS [t]
ORDER BY [t].[Title]
Notice that there three order by clauses, the inner-most one is the one from rawSql.
We can look at the error message to see why it's not legal:
The ORDER BY clause is invalid in [...] subqueries [...] unless OFFSET [...] is also specified.
The middle order by does include offset, so that's valid even though it's inside a subquery.
To fix this, just simply remove the ordering from your rawSql and keep using the OrderBy() linq method.
var rawSql = "select [Id], [Title] from Post";
var query = context.Posts.AsNoTracking()
.FromSql(rawSql)
.Skip(1)
.Take(4)
.OrderBy(x => x.Title);
This generates:
SELECT [t].[Id], [t].[Title]
FROM (
SELECT [p].[Id], [p].[Title]
FROM (
select [Id], [Title] from Post
) AS [p]
ORDER BY (SELECT 1)
OFFSET 1 ROWS FETCH NEXT 4 ROWS ONLY
) AS [t]
ORDER BY [t].[Title]
Now, all order by clauses are either not in subqueries, or have an offset clause.

Select "IN" in LINQ to SQL

I get a list of entities to update and I have their ids. I want to get the original ones from the database, so I do:
String[] ids = updatedEvents.Select(ue => ue.id).ToArray();
var originalEventsToUpdate = Db.tbl_ffk_event
.Where(e => ids.Contains(e.id))
.ToArray();
But what I get using the log is this generated SQL:
SELECT [t0].[id], [t0].[fs_mapping_id], [t0].[fs_id_value], [t0].[desc]
FROM [dbo].[tbl_ffk_event] AS [t0]
WHERE 0 = 1
-- Context: SqlProvider(Sql2008) Model: AttributedMetaModel Build: 4.0.30319.1
And that SQL means "get the whole table".
How can I generate a "IN" like this:
SELECT [t0].[id], [t0].[fs_mapping_id], [t0].[fs_id_value], [t0].[desc]
FROM [dbo].[tbl_ffk_event] AS [t0]
WHERE [t0].[id] IN ('aaa','bbb','ccc','ddd','eee',)
Thanks in advance.
EDIT:
I feel stupid, I didn't see the WHERE 0 = 1. It's because at that point, there where nothing in the ids collection. I have checked out now ensuring there are items, and the SQL is generated correctly. Sorry.

Actually, due to the clause WHERE 0 = 1 this SQL will return an empty recordset (i.e. correctly mapped in terms of the schema, but with no rows).
The code you give seems correct, but something has convinced the query provider that there can never be a matching row.
Assuming it's not correct in this, I'd look at the column mapping for the id property. Does it match that of the database correctly?

Probably your list is coming empty, this is the normal behavior of Linq.

Try this:
Dim strval As String = ""
Dim strnum(30) As String
strval = "1,2,3,4,5,6,7,9"
strnum = strval.split(",")
originalEventsToUpdate = (from a in Db.tbl_ffk_event where strnum.contains(a.id) select a).tolist

Using IQueryable with Linq

What is the use of IQueryable in the context of LINQ?
Is it used for developing extension methods or any other purpose?

Marc Gravell's answer is very complete, but I thought I'd add something about this from the user's point of view, as well...
The main difference, from a user's perspective, is that, when you use IQueryable<T> (with a provider that supports things correctly), you can save a lot of resources.
For example, if you're working against a remote database, with many ORM systems, you have the option of fetching data from a table in two ways, one which returns IEnumerable<T>, and one which returns an IQueryable<T>. Say, for example, you have a Products table, and you want to get all of the products whose cost is >$25.
If you do:
IEnumerable<Product> products = myORM.GetProducts();
var productsOver25 = products.Where(p => p.Cost >= 25.00);
What happens here, is the database loads all of the products, and passes them across the wire to your program. Your program then filters the data. In essence, the database does a SELECT * FROM Products, and returns EVERY product to you.
With the right IQueryable<T> provider, on the other hand, you can do:
IQueryable<Product> products = myORM.GetQueryableProducts();
var productsOver25 = products.Where(p => p.Cost >= 25.00);
The code looks the same, but the difference here is that the SQL executed will be SELECT * FROM Products WHERE Cost >= 25.
From your POV as a developer, this looks the same. However, from a performance standpoint, you may only return 2 records across the network instead of 20,000....

In essence its job is very similar to IEnumerable<T> - to represent a queryable data source - the difference being that the various LINQ methods (on Queryable) can be more specific, to build the query using Expression trees rather than delegates (which is what Enumerable uses).
The expression trees can be inspected by your chosen LINQ provider and turned into an actual query - although that is a black art in itself.
This is really down to the ElementType, Expression and Provider - but in reality you rarely need to care about this as a user. Only a LINQ implementer needs to know the gory details.
Re comments; I'm not quite sure what you want by way of example, but consider LINQ-to-SQL; the central object here is a DataContext, which represents our database-wrapper. This typically has a property per table (for example, Customers), and a table implements IQueryable<Customer>. But we don't use that much directly; consider:
using(var ctx = new MyDataContext()) {
var qry = from cust in ctx.Customers
where cust.Region == "North"
select new { cust.Id, cust.Name };
foreach(var row in qry) {
Console.WriteLine("{0}: {1}", row.Id, row.Name);
}
}
this becomes (by the C# compiler):
var qry = ctx.Customers.Where(cust => cust.Region == "North")
.Select(cust => new { cust.Id, cust.Name });
which is again interpreted (by the C# compiler) as:
var qry = Queryable.Select(
Queryable.Where(
ctx.Customers,
cust => cust.Region == "North"),
cust => new { cust.Id, cust.Name });
Importantly, the static methods on Queryable take expression trees, which - rather than regular IL, get compiled to an object model. For example - just looking at the "Where", this gives us something comparable to:
var cust = Expression.Parameter(typeof(Customer), "cust");
var lambda = Expression.Lambda<Func<Customer,bool>>(
Expression.Equal(
Expression.Property(cust, "Region"),
Expression.Constant("North")
), cust);
... Queryable.Where(ctx.Customers, lambda) ...
Didn't the compiler do a lot for us? This object model can be torn apart, inspected for what it means, and put back together again by the TSQL generator - giving something like:
SELECT c.Id, c.Name
FROM [dbo].[Customer] c
WHERE c.Region = 'North'
(the string might end up as a parameter; I can't remember)
None of this would be possible if we had just used a delegate. And this is the point of Queryable / IQueryable<T>: it provides the entry-point for using expression trees.
All this is very complex, so it is a good job that the compiler makes it nice and easy for us.
For more information, look at "C# in Depth" or "LINQ in Action", both of which provide coverage of these topics.

Although Reed Copsey and Marc Gravell already described about IQueryable (and also IEnumerable) enough,mI want to add little more here by providing a small example on IQueryable and IEnumerable as many users asked for it
Example: I have created two table in database
CREATE TABLE [dbo].[Employee]([PersonId] [int] NOT NULL PRIMARY KEY,[Gender] [nchar](1) NOT NULL)
CREATE TABLE [dbo].[Person]([PersonId] [int] NOT NULL PRIMARY KEY,[FirstName] [nvarchar](50) NOT NULL,[LastName] [nvarchar](50) NOT NULL)
The Primary key(PersonId) of table Employee is also a forgein key(personid) of table Person
Next i added ado.net entity model in my application and create below service class on that
public class SomeServiceClass
{
public IQueryable<Employee> GetEmployeeAndPersonDetailIQueryable(IEnumerable<int> employeesToCollect)
{
DemoIQueryableEntities db = new DemoIQueryableEntities();
var allDetails = from Employee e in db.Employees
join Person p in db.People on e.PersonId equals p.PersonId
where employeesToCollect.Contains(e.PersonId)
select e;
return allDetails;
}
public IEnumerable<Employee> GetEmployeeAndPersonDetailIEnumerable(IEnumerable<int> employeesToCollect)
{
DemoIQueryableEntities db = new DemoIQueryableEntities();
var allDetails = from Employee e in db.Employees
join Person p in db.People on e.PersonId equals p.PersonId
where employeesToCollect.Contains(e.PersonId)
select e;
return allDetails;
}
}
they contains same linq. It called in program.cs as defined below
class Program
{
static void Main(string[] args)
{
SomeServiceClass s= new SomeServiceClass();
var employeesToCollect= new []{0,1,2,3};
//IQueryable execution part
var IQueryableList = s.GetEmployeeAndPersonDetailIQueryable(employeesToCollect).Where(i => i.Gender=="M");
foreach (var emp in IQueryableList)
{
System.Console.WriteLine("ID:{0}, EName:{1},Gender:{2}", emp.PersonId, emp.Person.FirstName, emp.Gender);
}
System.Console.WriteLine("IQueryable contain {0} row in result set", IQueryableList.Count());
//IEnumerable execution part
var IEnumerableList = s.GetEmployeeAndPersonDetailIEnumerable(employeesToCollect).Where(i => i.Gender == "M");
foreach (var emp in IEnumerableList)
{
System.Console.WriteLine("ID:{0}, EName:{1},Gender:{2}", emp.PersonId, emp.Person.FirstName, emp.Gender);
}
System.Console.WriteLine("IEnumerable contain {0} row in result set", IEnumerableList.Count());
Console.ReadKey();
}
}
The output is same for both obviously
ID:1, EName:Ken,Gender:M
ID:3, EName:Roberto,Gender:M
IQueryable contain 2 row in result set
ID:1, EName:Ken,Gender:M
ID:3, EName:Roberto,Gender:M
IEnumerable contain 2 row in result set
So the question is what/where is the difference? It does not seem to
have any difference right? Really!!
Let's have a look on sql queries generated and executed by entity
framwork 5 during these period
IQueryable execution part
--IQueryableQuery1
SELECT
[Extent1].[PersonId] AS [PersonId],
[Extent1].[Gender] AS [Gender]
FROM [dbo].[Employee] AS [Extent1]
WHERE ([Extent1].[PersonId] IN (0,1,2,3)) AND (N'M' = [Extent1].[Gender])
--IQueryableQuery2
SELECT
[GroupBy1].[A1] AS [C1]
FROM ( SELECT
COUNT(1) AS [A1]
FROM [dbo].[Employee] AS [Extent1]
WHERE ([Extent1].[PersonId] IN (0,1,2,3)) AND (N'M' = [Extent1].[Gender])
) AS [GroupBy1]
IEnumerable execution part
--IEnumerableQuery1
SELECT
[Extent1].[PersonId] AS [PersonId],
[Extent1].[Gender] AS [Gender]
FROM [dbo].[Employee] AS [Extent1]
WHERE [Extent1].[PersonId] IN (0,1,2,3)
--IEnumerableQuery2
SELECT
[Extent1].[PersonId] AS [PersonId],
[Extent1].[Gender] AS [Gender]
FROM [dbo].[Employee] AS [Extent1]
WHERE [Extent1].[PersonId] IN (0,1,2,3)
Common script for both execution part
/* these two query will execute for both IQueryable or IEnumerable to get details from Person table
Ignore these two queries here because it has nothing to do with IQueryable vs IEnumerable
--ICommonQuery1
exec sp_executesql N'SELECT
[Extent1].[PersonId] AS [PersonId],
[Extent1].[FirstName] AS [FirstName],
[Extent1].[LastName] AS [LastName]
FROM [dbo].[Person] AS [Extent1]
WHERE [Extent1].[PersonId] = #EntityKeyValue1',N'#EntityKeyValue1 int',#EntityKeyValue1=1
--ICommonQuery2
exec sp_executesql N'SELECT
[Extent1].[PersonId] AS [PersonId],
[Extent1].[FirstName] AS [FirstName],
[Extent1].[LastName] AS [LastName]
FROM [dbo].[Person] AS [Extent1]
WHERE [Extent1].[PersonId] = #EntityKeyValue1',N'#EntityKeyValue1 int',#EntityKeyValue1=3
*/
So you have few questions now, let me guess those and try to answer them
Why are different scripts generated for same result?
Lets find out some points here,
all queries has one common part
WHERE [Extent1].[PersonId] IN (0,1,2,3)
why? Because both function IQueryable<Employee> GetEmployeeAndPersonDetailIQueryable and
IEnumerable<Employee> GetEmployeeAndPersonDetailIEnumerable of SomeServiceClass contains one common line in linq queries
where employeesToCollect.Contains(e.PersonId)
Than why is the
AND (N'M' = [Extent1].[Gender]) part is missing in IEnumerable execution part, while in both function calling we used Where(i => i.Gender == "M") inprogram.cs`
Now we are in the point where difference came between IQueryable and
IEnumerable
What entity framwork does when an IQueryable method called, it tooks linq statement written inside the method and try to find out if more linq expressions are defined on the resultset, it then gathers all linq queries defined until the result need to fetch and constructs more appropriate sql query to execute.
It provide a lots of benefits like,
only those rows populated by sql server which could be valid by the
whole linq query execution
helps sql server performance by not selecting unnecessary rows
network cost get reduce
like here in example sql server returned to application only two rows after IQueryable execution` but returned THREE rows for IEnumerable query why?
In case of IEnumerable method, entity framework took linq statement written inside the method and constructs sql query when result need to fetch. it does not include rest linq part to constructs the sql query. Like here no filtering is done in sql server on column gender.
But the outputs are same? Because 'IEnumerable filters the result further in application level after retrieving result from sql server
SO, what should someone choose?
I personally prefer to define function result as IQueryable<T> because there are lots of benefit it has over IEnumerable like, you could join two or more IQueryable functions, which generate more specific script to sql server.
Here in example you can see an IQueryable Query(IQueryableQuery2) generates a more specific script than IEnumerable query(IEnumerableQuery2) which is much more acceptable in my point of view.

It allows for further querying further down the line. If this was beyond a service boundary say, then the user of this IQueryable object would be allowed to do more with it.
For instance if you were using lazy loading with nhibernate this might result in graph being loaded when/if needed.