Develop Reference

Entity Framework Core : query improvement

I'm new to EF, I have a database in which a certain table contains keys of a number of other tables, it is the central table from which I go to the other tables (ProductIdentifiers).
The input to the query is not id, but the Name which is not defined as any key.
Here's my Entity Framework query :
public ProductIdentifier? GetFullCedMed(string v)
=> db.ProductIdentifiers.Where(a => a.Name == v)
.Include(ced => ced.Project)
.Include(ced => ced.LimitValues).ThenInclude(l => l.Parameter)
.Include(ced => ced.LimitValues).ThenInclude(l => l.Bin)
.Include(ced => ced.LimitValues).ThenInclude(l => l.Stage)
.Include(ced => ced.LimitValues).ThenInclude(l => l.TestTypeNavigation)
.Include(ced => ced.ConfigValues).ThenInclude(c => c.Parameter)
.Include(ced => ced.ConfigValues).ThenInclude(c => c.Bin)
.Include(ced => ced.ConfigValues).ThenInclude(c => c.Stage)
.Include(ced => ced.ConfigValues).ThenInclude(c => c.TestTypeNavigation)
.Include(ced => ced.FatherCedmed)
.ToList().FirstOrDefault();
The code is converted to a SQL query that looks like this:
exec sp_executesql N'SELECT [p].[id], [p].[FatherCedmedId], [p].[Name], [p].[ProjectId], [m].[id], [m].[IsActive], [m].[Name], [m].[ProductLineName], [p0].[id], [t].[id], [t].[BinId], [t].[CED_MED], [t].[LSL], [t].[ParameterID], [t].[StageID], [t].[TestType], [t].[USL], [t].[id0], [t].[Enabled], [t].[FORMAT], [t].[IsLimit], [t].[ParamID], [t].[Parameter_Name], [t].[Print], [t].[Unit], [t].[id1], [t].[BinDescription], [t].[BinDescriptionOverride], [t].[BinNumber], [t].[BinNumberOverride], [t].[GroupID], [t].[ParamID0], [t].[StageID0], [t].[id2], [t].[Name], [t].[StageNumber], [t].[id3], [t].[LevelId], [t].[Name0], [t].[OrderingId], [t0].[id], [t0].[BinId], [t0].[CED_MED], [t0].[ParameterID], [t0].[StageID], [t0].[TestType], [t0].[Value], [t0].[id0], [t0].[Enabled], [t0].[FORMAT], [t0].[IsLimit], [t0].[ParamID], [t0].[Parameter_Name], [t0].[Print], [t0].[Unit], [t0].[id1], [t0].[BinDescription], [t0].[BinDescriptionOverride], [t0].[BinNumber], [t0].[BinNumberOverride], [t0].[GroupID], [t0].[ParamID0], [t0].[StageID0], [t0].[id2], [t0].[Name], [t0].[StageNumber], [t0].[id3], [t0].[LevelId], [t0].[Name0], [t0].[OrderingId], [p0].[FatherCedmedId], [p0].[Name], [p0].[ProjectId]
FROM [ProductIdentifiers] AS [p]
LEFT JOIN [main_Projects] AS [m] ON [p].[ProjectId] = [m].[id]
LEFT JOIN [ProductIdentifiers] AS [p0] ON [p].[FatherCedmedId] = [p0].[id]
LEFT JOIN (
SELECT [l].[id], [l].[BinId], [l].[CED_MED], [l].[LSL], [l].[ParameterID], [l].[StageID], [l].[TestType], [l].[USL], [p1].[id] AS [id0], [p1].[Enabled], [p1].[FORMAT], [p1].[IsLimit], [p1].[ParamID], [p1].[Parameter_Name], [p1].[Print], [p1].[Unit], [b].[id] AS [id1], [b].[BinDescription], [b].[BinDescriptionOverride], [b].[BinNumber], [b].[BinNumberOverride], [b].[GroupID], [b].[ParamID] AS [ParamID0], [b].[StageID] AS [StageID0], [s].[id] AS [id2], [s].[Name], [s].[StageNumber], [m0].[id] AS [id3], [m0].[LevelId], [m0].[Name] AS [Name0], [m0].[OrderingId]
FROM [LimitValues] AS [l]
INNER JOIN [Parameters] AS [p1] ON [l].[ParameterID] = [p1].[id]
INNER JOIN [Bins] AS [b] ON [l].[BinId] = [b].[id]
LEFT JOIN [Stages] AS [s] ON [l].[StageID] = [s].[id]
LEFT JOIN [main_TestTypes] AS [m0] ON [l].[TestType] = [m0].[id]
) AS [t] ON [p].[id] = [t].[CED_MED]
LEFT JOIN (
SELECT [c].[id], [c].[BinId], [c].[CED_MED], [c].[ParameterID], [c].[StageID], [c].[TestType], [c].[Value], [p2].[id] AS [id0], [p2].[Enabled], [p2].[FORMAT], [p2].[IsLimit], [p2].[ParamID], [p2].[Parameter_Name], [p2].[Print], [p2].[Unit], [b0].[id] AS [id1], [b0].[BinDescription], [b0].[BinDescriptionOverride], [b0].[BinNumber], [b0].[BinNumberOverride], [b0].[GroupID], [b0].[ParamID] AS [ParamID0], [b0].[StageID] AS [StageID0], [s0].[id] AS [id2], [s0].[Name], [s0].[StageNumber], [m1].[id] AS [id3], [m1].[LevelId], [m1].[Name] AS [Name0], [m1].[OrderingId]
FROM [ConfigValues] AS [c]
INNER JOIN [Parameters] AS [p2] ON [c].[ParameterID] = [p2].[id]
INNER JOIN [Bins] AS [b0] ON [c].[BinId] = [b0].[id]
LEFT JOIN [Stages] AS [s0] ON [c].[StageID] = [s0].[id]
LEFT JOIN [main_TestTypes] AS [m1] ON [c].[TestType] = [m1].[id]
) AS [t0] ON [p].[id] = [t0].[CED_MED]
WHERE [p].[Name] = #__v_0
ORDER BY [p].[id], [m].[id], [p0].[id], [t].[id], [t].[id0], [t].[id1], [t].[id2], [t].[id3], [t0].[id], [t0].[id0], [t0].[id1], [t0].[id2]',N'#__v_0 varchar(255)',#__v_0='NAME_OF_RECORD_FROM_ProductIdentifier_TABLE'
Pay attention to the input in the ORDER BY line.
My question is: how can the query be improved?
The tables contain a lot of data and the query takes a lot of time.
Will retrieval by ID make it faster? Indicates that all the data is relevant for me, which means that it is necessary to join all the tables.
Thanks for any other advice.

for creating separated query and getting better performance use .AsSplitQuery()

EF Core Include translates to SQL with sub queries that have no filters

Meetings
.Include(a => a.Document)
.Include(a => a.Plan)
.Include(a => a.User)
.Include(a => a.Topics).ThenInclude(e => e.Extra)
.Include(a => a.Components).ThenInclude(g => g.Extra)
.Include(a => a.Recipients).ThenInclude(i => i.Info)
.Single(a => a.MeetingId == 1)
The above LINQ will translate to SQLs with sub queries that have no filters:
SELECT [blah]
FROM (
SELECT TOP(2) [blah]
FROM [Meeting] AS [m]
LEFT JOIN [Plan] AS [s] ON [m].[PlanId] = [s].[PlanId]
LEFT JOIN [User] AS [u] ON [m].[UserId] = [u].[Id]
WHERE [m].[MeetingId] = 1
) AS [t]
LEFT JOIN [Document] AS [m0] ON [t].[MeetingId] = [m0].[MeetingId]
LEFT JOIN (
SELECT [blah]
FROM [Topic] AS [m1]
LEFT JOIN [Extra] AS [a] ON [m1].[ExtraId] = [a].[ExtraId]
) AS [t0] ON [t].[MeetingId] = [t0].[MeetingId]
LEFT JOIN (
SELECT [blah]
FROM [Component] AS [m2]
LEFT JOIN [Extra] AS [a0] ON [m2].[ExtraId] = [a0].[ExtraId]
) AS [t1] ON [t].[MeetingId] = [t1].[MeetingId]
LEFT JOIN (
SELECT [blah]
FROM [Recipient] AS [m3]
INNER JOIN [Info] AS [l] ON [m3].[InfoId] = [l].[InfoId]
) AS [t2] ON [t].[MeetingId] = [t2].[MeetingId]
If you look at the last 3 joins, there are no filters so they will scan the whole table.
Is there a way to fix this?
This is EF Core 3.1 btw.

LINQ generating Nested/Sub queries

I am using Asp.NET & Entity Framework with SQL Server as Database, somehow I am getting this strange issue
I have this code:
var pricingInfo = (from price in invDB.Pricing.AsNoTracking()
join priceD in invDB.PricingDetail.AsNoTracking() on price.PricingId equals priceDtl.PricingId
join tagD in invDB.PricingTagDetail.AsNoTracking() on priceDtl.PricingDetailId equals tagDtl.PricingDetailId
join it in invDB.Item.AsNoTracking() on tagDtl.ItemId equals item.ItemId
join par in invDB.Party.AsNoTracking() on tagDtl.PartyId equals party.PartyId
join b in invDB.Brand.AsNoTracking() on tagDtl.BrandId equals brd.BrandId into t from brand in t.DefaultIfEmpty()
where tagDtl.AvailableQuantity > 0m && price.PricingNo == printNumber
select new
{
TagNo = tagDtl.TagNo,
SellingRate = tagDtl.SellingRate,
Quantity = tagDtl.AvailableQuantity ?? 0m,
ItemCode = item.Name,
UOMId = priceDtl.UOMId,
Brand = brand.BrandCode,
Supplier = party.PartyCode,
Offer = tagDtl.Offer
}).ToList();
Which generates the below sql query with a sub query, without where condition and it pulls out full records from a large volume data. This results to a heavy memory consumption and performance issues.
SELECT
[Filter1].[PricingId1] AS [PricingId],
[Filter1].[TagNo] AS [TagNo],
[Filter1].[SellingRate1] AS [SellingRate],
CASE WHEN ([Filter1].[AvailableQuantity] IS NULL) THEN cast(0 as decimal(18)) ELSE [Filter1].[AvailableQuantity] END AS [C1],
[Filter1].[Name] AS [Name],
[Filter1].[UOMId 1] AS [UOMId ],
[Extent6].[BrandCode] AS [BrandCode],
[Filter1].[PartyCode] AS [PartyCode],
[Filter1].[Offer] AS [Offer]
FROM
(
SELECT [Extent1].[PricingId] AS [PricingId1], [Extent1].[PricingNo] AS [PricingNo], [Extent2].[UnitOfMeasurementId] AS [UnitOfMeasurementId1], [Extent3].[TagNo] AS [TagNo], [Extent3].[BrandId] AS [BrandId1], [Extent3].[SellingRate] AS [SellingRate1], [Extent3].[AvailableQuantity] AS [AvailableQuantity], [Extent3].[Offer] AS [Offer], [Extent4].[Name] AS [Name], [Extent5].[PartyCode] AS [PartyCode]
FROM [PanERP].[Pricing] AS [Extent1]
INNER JOIN [PanERP].[PricingDetail] AS [Extent2] ON [Extent1].[PricingId] = [Extent2].[PricingId]
INNER JOIN [PanERP].[PricingTagDetail] AS [Extent3] ON [Extent2].[PricingDetailId] = [Extent3].[PricingDetailId]
INNER JOIN [PanERP].[Item] AS [Extent4] ON [Extent3].[ItemId] = [Extent4].[ItemId]
INNER JOIN [PanERP].[Party] AS [Extent5] ON [Extent3].[PartyId] = [Extent5].[PartyId]
WHERE [Extent3].[AvailableQuantity] > cast(0 as decimal(18))
) AS [Filter1]
LEFT OUTER JOIN [PanERP].[Brand] AS [Extent6] ON [Filter1].[BrandId1] = [Extent6].[BrandId]
WHERE ([Filter1].[PricingNo] = #p__linq__0) OR (([Filter1].[PricingNo] IS NULL) AND (#p__linq__0 IS NULL))
But When i change the condition
where tagDtl.AvailableQuantity > 0m
as a variable it creates another SQL query without nested select statement.
Here is the modified code
decimal availableQuantity = 0m;
var pricingInfo = (from price in invDB.Pricing.AsNoTracking()
join priceD in invDB.PricingDetail.AsNoTracking() on price.PricingId equals priceDtl.PricingId
join tagD in invDB.PricingTagDetail.AsNoTracking() on priceDtl.PricingDetailId equals tagDtl.PricingDetailId
join it in invDB.Item.AsNoTracking() on tagDtl.ItemId equals item.ItemId
join par in invDB.Party.AsNoTracking() on tagDtl.PartyId equals party.PartyId
join b in invDB.Brand.AsNoTracking() on tagDtl.BrandId equals brd.BrandId into t from brand in t.DefaultIfEmpty()
where tagDtl.AvailableQuantity > availableQuantity && price.PricingNo == printNumber
select new
{
TagNo = tagDtl.TagNo,
SellingRate = tagDtl.SellingRate,
Quantity = tagDtl.AvailableQuantity ?? availableQuantity,
ItemCode = item.Name,
UOMId = priceDtl.UOMId,
Brand = brand.BrandCode,
Supplier = party.PartyCode,
Offer = tagDtl.Offer
}).ToList();
and here is the SQL query without nested SQL statement.
SELECT
[Extent1].[PricingId] AS [PricingId],
[Extent3].[TagNo] AS [TagNo],
[Extent3].[SellingRate] AS [SellingRate],
CASE WHEN ([Extent3].[AvailableQuantity] IS NULL) THEN cast(0 as decimal(18)) ELSE [Extent3].[AvailableQuantity] END AS [C1],
[Extent4].[Name] AS [Name],
[Extent2].[UOMId ] AS [UOMId ],
[Extent6].[BrandCode] AS [BrandCode],
[Extent5].[PartyCode] AS [PartyCode],
[Extent3].[Offer] AS [Offer]
FROM [PanERP].[Pricing] AS [Extent1]
INNER JOIN [PanERP].[PricingDetail] AS [Extent2] ON [Extent1].[PricingId] = [Extent2].[PricingId]
INNER JOIN [PanERP].[PricingTagDetail] AS [Extent3] ON [Extent2].[PricingDetailId] = [Extent3].[PricingDetailId]
INNER JOIN [PanERP].[Item] AS [Extent4] ON [Extent3].[ItemId] = [Extent4].[ItemId]
INNER JOIN [PanERP].[Party] AS [Extent5] ON [Extent3].[PartyId] = [Extent5].[PartyId]
LEFT OUTER JOIN [PanERP].[Brand] AS [Extent6] ON [Extent3].[BrandId] = [Extent6].[BrandId]
WHERE ([Extent3].[AvailableQuantity] > #p__linq__0) AND (([Extent1].[PricingNo] = #p__linq__1) OR (([Extent1].[PricingNo] IS NULL) AND (#p__linq__1 IS NULL)))
If I move the where condition to the model definition as lambda expression, like this
from price in inventoryDb.Pricing.AsNoTracking().Where(c =>
c.PricingNo == printNumber))
then also it works fine.
Why is LINQ generating a nested Select? How can we avoid this?
Thanks in advance for your answers.

Well, I think you have answered your own question, on your comments. I will just try to clarify what is going on.
When you use a hard-coded constant, like 0m, the framework translates it into SQL keeping the value as a constant:
WHERE [Extent3].[AvailableQuantity] > cast(0 as decimal(18))
When you use a local variable, like “availableQuantity”, the framework creates a parameter:
([Extent3].[AvailableQuantity] > #p__linq__0)
I might be wrong, but, as I see, this is done in order to preserve the programmer’s goal when writing the code (constant = constant, variable = parameter).
And what about the subquery?
This is a query optimization logic (a bad one, probably, at least on this scenario). When you make a query using parameters, you might run it several times, but SQL Server will always use the same execution plan, making the query faster; when you use constants, each query need to be reevaluated (if you check SQL Server Activity Monitor, you will see that queries with parameters are treated as the same query, regardless the parameters values).
This way, in my opinion (sorry, I could not find any documentation about it), Entity Framework is trying to isolate the queries; the outer/generic one, that use parameters, and the inner/specific one, that use constants.
I would be happy if anyone could complement it with some Microsoft documentation about this subject…

LINQ multiple joins with one left join

I am fairly new to LINQ and I am struggling to make a multiple JOIN.
So, this is how my database structure looks like:
Now, how should my query look like, if I have a particular Grade and I want to select
{Student.IndexNo, GradeValue.Value}, but if there is no grade value for a particular grade and particular user, null should be returned (Left join)?

The trick to get a LEFT join is to use the DefaultIfEmpty() method:
var otherValue = 5;
var deps = from tbl1 in Table1
join tbl2 in Table2
on tbl1.Key equals tbl2.Key into joinGroup
from j in joinGroup.DefaultIfEmpty()
where
j.SomeProperty == "Some Value"
&& tbl1.OtherProperty == otherValue
select j;

Deliberately posting this in 2015 for newbies looking for solution on google hits. I managed to hack and slash programming my way into solution.
var projDetails = from r in entities.ProjekRumah
join d in entities.StateDistricts on r.ProjekLocationID equals d.DistrictID
join j in entities.ProjekJenis on r.ProjekTypeID equals j.TypeID
join s in entities.ProjekStatus on r.ProjekStatusID equals s.StatusID
join approvalDetails in entities.ProjekApproval on r.ProjekID equals approvalDetails.ProjekID into approvalDetailsGroup
from a in approvalDetailsGroup.DefaultIfEmpty()
select new ProjectDetailsDTO()
{
ProjekID = r.ProjekID,
ProjekName = r.ProjekName,
ProjekDistrictName = d.DistrictName,
ProjekTypeName = j.TypeName,
ProjekStatusName = s.StatusName,
IsApprovalAccepted = a.IsApprovalAccepted ? "Approved" : "Draft",
ProjekApprovalRemarks = a.ApprovalRemarks
};
Produces following SQL code internally
{SELECT [Extent1].[ProjekID] AS [ProjekID]
,[Extent1].[ProjekName] AS [ProjekName]
,[Extent2].[DistrictName] AS [DistrictName]
,[Extent3].[TypeName] AS [TypeName]
,[Extent4].[StatusName] AS [StatusName]
,CASE
WHEN ([Extent5].[IsApprovalAccepted] = 1)
THEN N'Approved'
ELSE N'Draft'
END AS [C1]
,[Extent5].[ApprovalRemarks] AS [ApprovalRemarks]
FROM [dbo].[ProjekRumah] AS [Extent1]
INNER JOIN [dbo].[StateDistricts] AS [Extent2] ON [Extent1].[ProjekLocationID] = [Extent2].[DistrictID]
INNER JOIN [dbo].[ProjekJenis] AS [Extent3] ON [Extent1].[ProjekTypeID] = [Extent3].[TypeID]
INNER JOIN [dbo].[ProjekStatus] AS [Extent4] ON [Extent1].[ProjekStatusID] = [Extent4].[StatusID]
LEFT JOIN [dbo].[ProjekApproval] AS [Extent5] ON [Extent1].[ProjekID] = [Extent5].[ProjekID]
}

Queries generated by group by vs group join

I have the following group by linq statement
from c in Categories
join p in Products on c equals p.Category into ps
select new { Category = new {c.CategoryID, c.CategoryName}, Products = ps };
However this generates the following left outer join query and returns all categories even if there are no products associated.
SELECT [t0].[CategoryID], [t0].[CategoryName], [t1].[ProductID], [t1].[ProductName], [t1].[SupplierID], [t1].[CategoryID] AS [CategoryID2], [t1].[QuantityPerUnit], [t1].[UnitPrice], [t1].[UnitsInStock], [t1].[UnitsOnOrder], [t1].[ReorderLevel], [t1].[Discontinued], (
SELECT COUNT(*)
FROM [Products] AS [t2]
WHERE [t0].[CategoryID] = [t2].[CategoryID]
) AS [value]
FROM [Categories] AS [t0]
LEFT OUTER JOIN [Products] AS [t1] ON [t0].[CategoryID] = [t1].[CategoryID]
ORDER BY [t0].[CategoryID], [t1].[ProductID]
What I really want is to return only those categories that have associated products. But if I re-write the linq query like so:
from c in Categories
join p in Products on c equals p.Category
group p by new {c.CategoryID, c.CategoryName} into ps
select new { Category = ps.Key, Products = ps };
This gives me the desired result but a query is generated for each category:
SELECT [t0].[CategoryID], [t0].[CategoryName]
FROM [Categories] AS [t0]
INNER JOIN [Products] AS [t1] ON [t0].[CategoryID] = [t1].[CategoryID]
GROUP BY [t0].[CategoryID], [t0].[CategoryName]
GO
-- Region Parameters
DECLARE #x1 Int SET #x1 = 1
DECLARE #x2 NVarChar(9) SET #x2 = 'Beverages'
-- EndRegion
SELECT [t1].[ProductID], [t1].[ProductName], [t1].[SupplierID], [t1].[CategoryID], [t1].[QuantityPerUnit], [t1].[UnitPrice], [t1].[UnitsInStock], [t1].[UnitsOnOrder], [t1].[ReorderLevel], [t1].[Discontinued]
FROM [Categories] AS [t0]
INNER JOIN [Products] AS [t1] ON [t0].[CategoryID] = [t1].[CategoryID]
WHERE (#x1 = [t0].[CategoryID]) AND (#x2 = [t0].[CategoryName])
GO
-- Region Parameters
DECLARE #x1 Int SET #x1 = 2
DECLARE #x2 NVarChar(10) SET #x2 = 'Condiments'
-- EndRegion
SELECT [t1].[ProductID], [t1].[ProductName], [t1].[SupplierID], [t1].[CategoryID], [t1].[QuantityPerUnit], [t1].[UnitPrice], [t1].[UnitsInStock], [t1].[UnitsOnOrder], [t1].[ReorderLevel], [t1].[Discontinued]
FROM [Categories] AS [t0]
INNER JOIN [Products] AS [t1] ON [t0].[CategoryID] = [t1].[CategoryID]
WHERE (#x1 = [t0].[CategoryID]) AND (#x2 = [t0].[CategoryName])
GO
...
Is there a way to do the equivalent of a inner join and group by and still only produce a single query like the group join?

var queryYouWant =
from c in Categories
join p in Products on c equals p.Category
select new {Category = c, Product = p};
var result =
from x in queryYouWant.AsEnumerable()
group x.Product by x.Category into g
select new { Category = g.Key, Products = g };
Is there a way to do the equivalent of a inner join and group by and still only produce a single query like the group join?
No. When you say GroupBy followed by non-aggregated access of the group elements, that's a repeated query with the group key as a filter.

What is the purpose of that join?
Your original query is identical to this:
from c in Categories
select new { Category = new { c.CategoryID, c.CategoryName }, c.Products }
Am I somehow missing something obvious???
If you want only categories with products, then do this:
from c in Categories
where c.Products.Any()
select new { Category = new { c.CategoryID, c.CategoryName }, c.Products }
Or, if you want to flatten the results:
from p in Products
select new { p, p.Category.CategoryID, p.Category.CategoryName }
The latter will translate into an inner or outer join - depending on whether that relationship is nullable. You can force the equivalent of an inner join as follows:
from p in Products
where p.Category != null
select new { p, p.Category.CategoryID, p.Category.CategoryName }