Develop Reference

Entity Framework Core LINQ trouble creating (select case exists) queries

I'm trying to list all items with an extra column describing whether it's owned by the current user.
So I'm looking for a Linq query that generates something like the following SQL:
SELECT *,
CASE WHEN
EXISTS (
SELECT NULL FROM OwnedItems
WHERE OwnedItems.UserId = #UserId AND OwnedItems.ItemId = Items.Id
)
THEN 'true'
ELSE 'false'
END AS Owned
FROM Items;
According to the internet as well as a successful LinqPad experiment this code should work.
from item in Items
select new
{
Owned = OwnedItems.Any(own => own.UserId == userId && own.ItemId == item.Id),
Item = item
}
In LinqPad this code generates the exact same SQL as I want. But in my project it does something completely different.
My code is a .Net Core 2.1 project using Entity Framework Core 2.1. Since it is a Core project I can't directly test it in LinqPad as it isn't supported yet.
In my project this code results in an unfiltered SELECT statement querying every Item, then for each of them a separate query to check if it exists in the OwnedItems table. Like this:
1 instance of this query runs:
Executed DbCommand (68ms) [Parameters=[], CommandType='Text', CommandTimeout='30']
SELECT *
FROM [Items] AS [item]
Followed by hundreds of these queries taking multiple seconds to run:
Executed DbCommand (32ms) [Parameters=[#__userId_0='?' (DbType = Int32), #_outer_Id='?' (DbType = Int32)], CommandType='Text', CommandTimeout='30']
SELECT CASE
WHEN EXISTS (
SELECT 1
FROM [OwnedItems] AS [ownedItems]
WHERE ([ownedItems].[UserId] = #__userId_0) AND ([ownedItems].[ItemId] = #_outer_Id))
THEN CAST(1 AS BIT) ELSE CAST(0 AS BIT)
END
Some further info, maybe it helps:
If I use the same line as part of a where clause, it works perfectly.
var q = from item in Items
where OwnedItems.Any(o => o.UserId == userId && o.ItemId == item.Id)
select item;
The above linq results in this nice sql:
SELECT *
FROM [Items] AS [item]
WHERE EXISTS (
SELECT 1
FROM [OwnedItems] AS [o]
WHERE ([o].[UserId] = #__userId_0) AND ([o].[ItemId] = [item].[Id]))
Notes:
The above code has been mangled manually so there might be typos in there. Please disregard them.
I understand that this particular query can be done using a left join and checking for nulls but my actual one is more complex and I need (nested) exists clauses.
UPDATE FOR SOLUTION
As #KorsG pointed out if the Item isn't materialized, the proper query is generated.
What I found is that not materializing Item works even if I write the following:
from item in Items
select new
{
Owned = OwnedItems.Any(own => own.UserId == userId && own.ItemId == item.Id),
// Item = item //THIS LINE GENERATES BAD QUERY
Item = new Item {
Id = item.Id,
Name = item.Name,
...
[Literally every single property listed one by one] = item.CorrespondingProperty
...
}
}
So I can actually materialize the full item, I just have to explicitly type every last property. FUN!

You probably need to enable eager loading for the "OwnedItems" navigation property in the query:
https://learn.microsoft.com/en-us/ef/core/querying/related-data#eager-loading
Please post your full linq query if I should give an example.
UPDATE 1
Seems like subqueries have N+1 problems in EF Core and it will maybe be fixed in version 3.
Reference: https://github.com/aspnet/EntityFrameworkCore/issues/10001
UPDATE 2
If you don't need to fully materialize "Items" you should be able to do something like this where you create an anoymous object instead which should "trick" EF into what you want:
from item in Items
select new
{
Owned = OwnedItems.Any(own => own.UserId == userId && own.ItemId == item.Id),
Item = new { Id = item.Id, Name = item.Name }
}
Reference: https://github.com/aspnet/EntityFrameworkCore/issues/11186

You need to tell EF to load the related data, in this case the OwnedItems table.
One way to go about this is to Include the related table. If there's a foreign key that links tables it could be easily done like this:
var dataWithRelatedData = db_context.Items.Include(x => x.OwnedItems).Select ...
Another way to avoid a high number of round trips to the database is to load both datasets in separate queries and then merge them in memory. So you would first make a query to Items then with the data returned another query to OwnedItems and lastly merge them into a single list of objects. This would do just 2 calls to the database, therefore improving performance.

Entity Framework Core; using ORDER BY in query against a (MS) SQL Server

I'm trying to use the following query in combination with Entity Framework Core against a Microsoft SQL Server 2016:
SELECT [a], [b], [c]
FROM [x]
WHERE [a] = {0}
ORDER BY [b]
I use this query like so:
context.MySet.AsNoTracking()
.FromSql(MyQuery, aValue)
.Skip(pageSize * page)
.Take(pageSize)
.Select(x => x.ToJsonDictionary())
.ToList()
I use this in a .NET Core REST API with pagination and I'd like to have the records sorted (alphabetically) to make the pagination more usable.
I get the following error when executing the above statement:
The ORDER BY clause is invalid in views, inline functions, derived
tables, subqueries, and common table expressions, unless TOP, OFFSET
or FOR XML is also specified.Invalid usage of the option NEXT in the
FETCH statement. The ORDER BY clause is invalid in views, inline
functions, derived tables, subqueries, and common table expressions,
unless TOP, OFFSET or FOR XML is also specified.Invalid usage of the
option NEXT in the FETCH statement.
Looking for similar issues I found these some other posts (1, 2, 3) but none of which where used in combination with EF Core and/or they were using it in a different context which does not apply in my case (e.g. subquery).
I tried to use the .OrderBy(..) syntax of EF instead of in the ORDER BY in the query but this doesn't solve the problem. I also tried adding TOP 100 PERCENT after the SELECT in the query in combination with the ORDRE BY; this worked but didn't order the column. It just got ignored. This limitation is described under the EF Limitations. I also found this post that replace the TOP 100 PERCENT... with TOP 99.99 PERCENT... or TOP 9999999... `. This seems like it should work but it doesn't 'feel' right.
This issue in general is further explained here.
Summary: It is not advisable to use ORDER BY in Views. Use ORDER BY
outside the views. In fact, the correct design will imply the same. If
you use TOP along with Views, there is a good chance that View will
not return all the rows of the table or will ignore ORDER BY
completely.
Further I'm confused by the word "view". For me, the term views refers to the usage of the ones created by the CREATE VIEW .. syntax. Is a plain, 'normal' SQL query also considered a view? Or is EF Core wrapping the request in some sort of view and this is the real issue causing this error?
I'm not sure, but so far all the 'solutions' I found seem kind of 'hacky'.
Thoughts?

Let's simplify things a bit. Here's what I came up for testing. I've also added some code for printing the generated sql from EF queries.
class Program
{
static void Main(string[] args)
{
DbClient context = new DbClient();
var rawSql = "select [Id], [Title] from Post order by [Title]";
var query = context.Posts.AsNoTracking()
.FromSql(rawSql)
.Skip(1)
.Take(4)
.OrderBy(x => x.Title);
var generated = query.ToSql();
var results = query.ToList();
}
}
class DbClient : DbContext
{
public DbSet<Post> Posts { get; set; }
protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
{
optionsBuilder.UseSqlServer("conn_string");
}
}
class Post
{
public int Id { get; set; }
public string Title { get; set; }
public override string ToString() => $"{Id} | {Title}";
}
When we look at the value of generated we see what the sql of the query is:
SELECT [t].[Id], [t].[Title]
FROM (
SELECT [p].[Id], [p].[Title]
FROM (
select [Id], [Title] from Post order by [Title]
) AS [p]
ORDER BY (SELECT 1)
OFFSET 1 ROWS FETCH NEXT 4 ROWS ONLY
) AS [t]
ORDER BY [t].[Title]
Notice that there three order by clauses, the inner-most one is the one from rawSql.
We can look at the error message to see why it's not legal:
The ORDER BY clause is invalid in [...] subqueries [...] unless OFFSET [...] is also specified.
The middle order by does include offset, so that's valid even though it's inside a subquery.
To fix this, just simply remove the ordering from your rawSql and keep using the OrderBy() linq method.
var rawSql = "select [Id], [Title] from Post";
var query = context.Posts.AsNoTracking()
.FromSql(rawSql)
.Skip(1)
.Take(4)
.OrderBy(x => x.Title);
This generates:
SELECT [t].[Id], [t].[Title]
FROM (
SELECT [p].[Id], [p].[Title]
FROM (
select [Id], [Title] from Post
) AS [p]
ORDER BY (SELECT 1)
OFFSET 1 ROWS FETCH NEXT 4 ROWS ONLY
) AS [t]
ORDER BY [t].[Title]
Now, all order by clauses are either not in subqueries, or have an offset clause.

special select query

I have 3 tables in my sql database like these :
Documents : (DocID, FileName) //list of all docs that were attached to items
Items : (ItemID, ...) //list of all items
DocumentRelation : (DocID, ItemID) //the relation between docs and items
In my winform application I have showed all records of Items table in a grid view and let user to select several rows of it and then if he press EditAll button another grid view should fill by file name of documents that are related to these selected items but not all of them,
Just each of documents which have relation with ALL selected items
Is there any query (sql or linq) to select these documents?

Try something like:
string query;
foreach (Item in SelectedItems)
{
query += "select DocID from DocumentRelation where ItemID =" + Item.Id;
query += "INTERSECT";
}
query -= "INTERSECT";
And exec the Query;

Take one string and keep on adding itemid comma separated in that,like 1,2,3 and then write query like
declare ItemID varchar(50);
set ItemID='1,2,3';
select FileName
from documents
Left Join DocumentRelation on Documents.DocId = DocumentRelation.DocId
where
DocumentRelation.ItemID in (select * from > dbo.SplitString(ItemID))
and then make one function in database like below
ALTER FUNCTION [dbo].[SplitString] (#OrderList varchar(1000))
RETURNS #ParsedList table (OrderID varchar(1000) )
AS BEGIN
IF #OrderList = ''
BEGIN
set #OrderList='Null'
end
DECLARE #OrderID varchar(1000), #Pos int
SET #OrderList = LTRIM(RTRIM(#OrderList))+ ','
SET #Pos = CHARINDEX(',', #OrderList, 1)
IF REPLACE(#OrderList, ',', '') <''
BEGIN
WHILE #Pos 0
BEGIN
SET #OrderID = LTRIM(RTRIM(LEFT(#OrderList, #Pos - 1)))
IF #OrderID < ''
BEGIN
INSERT INTO #ParsedList (OrderID)
VALUES (CAST(#OrderID AS varchar(1000)))
--Use Appropriate conversion
END
SET #OrderList = RIGHT(#OrderList, LEN(#OrderList) - #Pos)
SET #Pos = CHARINDEX(',', #OrderList, 1)
END
END
RETURN
END

Linq
var td =
from s in Items
join r in DocumentRelation on s.ItemID equals r.ItemID
join k in Documents on k.DocID equals r.DocID
where Coll.Contains (s.ItemID) //Here Coll is the collection of ItemID which you can store when the users click on the grid view row
select new
{
FileName=k.FileName,
DocumentID= k.DocId
};
You can loop through td collection and bind to your grid view
SQL
create a stored proc to get the relevant documents for the itemID selected from the grid view and paramterize your in clause
select k.FileName,k.DocId from Items as s inner join
DocumentRelation as r on
s.ItemID=r.ItemID and r.ItemId in (pass the above coll containing selected ItemIds as an input the SP)
inner join Documents as k
on k.DocId=r.DocIk
You can get the information on how to parametrize your sql query

Here's one approach. I'll let you figure out how you want to supply the list of items as arguments. And I also assume that (DocID, ItemID) is a primary key in the relations table. The having condition is what enforces your requirement that all select items are related to the list of documents you're seeking.
;with ItemsSelected as (
select i.ItemID
from Items as i
where i.ItemID in (<list of selected ItemIDs>)
)
select dr.DocID
from DocumentRelation as dr
where dr.ItemID in (select ItemID from ItemsSelected)
group by dr.DocID
having count(dr.ItemID) = (select count(*) from ItemsSelected);
EDIT
As far as I can tell, the accepted answer is equivalent to the solution here despite OP's comment below.
I did some quick tests with a very long series of intersect queries and confirmed that you can indeed expect that approach to become gradually slower with an increasing number of selected items. But a much worse problem was the time taken just to compile the queries. I tried this on a very fast server and found that that step took about eight seconds when roughly one hundred intersects were concatenated.
SQL Fiddle didn't let me do anywhere near as many before producing this error (and taking more than ten seconds in the process): The query processor ran out of internal resources and could not produce a query plan. This is a rare event and only expected for extremely complex queries or queries that reference a very large number of tables or partitions. Please simplify the query. If you believe you have received this message in error, contact Customer Support Services for more information.
There are several possible methods of passing a list of arguments to SQL Server. Assuming that you prefer the dynamic query solution I'd argue that this version is still better while also noting that there is a SQL Server limit on the number of values inside the in.
There are plenty of ways to have this stuff blow up.

EF Pre Compile query and return of a scalar value

I use asp.net 4 c# and ef4.
I have this code, it should compile a query and return a single scalar value (I use anonymous type).
My code does not have apparently errors, but because is the first time I write a compiled query I would like to know if is well written or could be improved for a performance boost.
var query = CompiledQuery.Compile((CmsConnectionStringEntityDataModel ctx)
=> from o in ctx.CmsOptions
where o.OptionId == 7
select new
{
Value = o.Value
});
uxHtmlHead.Text = query(context).FirstOrDefault().Value;// I print the result in a Label
SQL Profile Output:
SELECT TOP (1)
[Extent1].[OptionId] AS [OptionId],
[Extent1].[Value] AS [Value]
FROM [dbo].[CmsOptions] AS [Extent1]
WHERE 7 = [Extent1].[OptionId]
Many Thanks
Result after Wouter advice (please guys have a double check again):
static readonly Func<CmsConnectionStringEntityDataModel, int, string> compiledQueryHtmlHead =
CompiledQuery.Compile<CmsConnectionStringEntityDataModel, int, string>(
(ctx, id) => ctx.CmsOptions.FirstOrDefault(o => o.OptionId == id).Value);
using (var context = new CmsConnectionStringEntityDataModel())
{
int id = 7;
uxHtmlHead.Text = compiledQueryHtmlHead.Invoke(context, id);
}
Resulting SQL (I do not understand why with a LEFT JOIN)
exec sp_executesql N'SELECT
[Project1].[Value] AS [Value]
FROM ( SELECT 1 AS X ) AS [SingleRowTable1]
LEFT OUTER JOIN (SELECT
[Extent1].[Value] AS [Value]
FROM [dbo].[CmsOptions] AS [Extent1]
WHERE [Extent1].[OptionId] = #p__linq__0 ) AS [Project1] ON 1 = 1',N'#p__linq__0 int',#p__linq__0=7

There are 2 things you can improve on.
First, precompiling a query is definitely a good idea but if you have a look at your code you will see that it precompiles the query each and every time instead of only once.
You need to move the precompiled query to a static variable that is initialized only once.
Another thing you need to be careful of is that when precompiling a query you shouldn't modify the query anymore before executing it.
You are building a precompiled query that will select all rows and then you say 'firstordefault' which changes the precompiled query to a SELECT TOP (1) and you lose the benefit of precompiling. You need to move the FirstOrDefault part inside your precompiled query and return only one result.
Have a look at this documentation. If you look at the examples you can see how they use a static field to hold the compiled query and how they specify the return value.

Dynamic LINQ API - SQL Convert Function

I'm trying to use a Dynamic LINQ Query to query a SQL database, and in the Where clause I need to evaluate an '=' condition with a field that is of type TEXT.
Right now, I've got this:
var result = DBCon.PcInValue
.Where(String.Format("InputName = #0 and InputValue) {0} #1", f.Condition), f.Field, f.Value)
.Select("new(OrderNum, OrderLine)");
This doesn't work since you can't use the equal operator on a TEXT data type.
The field that is type TEXT is "InputValue". I tried to convert it like so:
var result = DBCon.PcInValue
.Where(String.Format("InputName = #0 and Convert(nvarchar(100), InputValue) {0} #1", f.Condition), f.Field, f.Value)
.Select("new(OrderNum, OrderLine)");
But it looks like this is not supported.
Anyone have any clues as to how I can do this?
EDIT:
The following SQL Syntax works with no issues, but again I'm not sure if this is possible using the Dynamic LINQ API:
SELECT [t0].[OrderNum], [t0].[OrderLine]
FROM [PcInValue] AS [t0]
WHERE ([t0].[InputName] = 'OpenWidthFt') AND (Convert(nvarchar(100), [t0].[InputValue]) = '10')

I've tested this and it seems to work fine (though it's a bit odd):
var result = DBCon.PcInValue
.Where(String.Format("InputName = #0 and InputValue.ToString() {0} #1", f.Condition), f.Field, f.Value)
.Select("new(OrderNum, OrderLine)");
LinqPad tells me it's translated into something similar to the following (using my own table):
SELECT [t0].[Id], [t0].[Name], [t0].[InputValue]
FROM [People] AS [t0]
WHERE (CONVERT(NVarChar(MAX),[t0].[InputValue])) = #p0