Develop Reference

Improving SQL query generated by LINQ to EF

I have student table and I want to write a Linq query to get multiple counts. The SQL query that Linq generates is too complicated and un-optimized.
Following is definition of my table:
[Id] [int] IDENTITY(1,1) NOT NULL,
[Name] [varchar](100) NULL,
[Age] [int] NULL,
I need to get one count with students with name = test and one count for students with age > 10.
This is one of the query I have tried:
var sql = from st in school.Students
group st by 1 into grp
select new
{
NameCount = grp.Count(k => k.Name == "Test"),
AgeCount = grp.Count(k => k.Age > 5)
};
The SQL query that is generated is:
SELECT
[Limit1].[C1] AS [C1],
[Limit1].[C2] AS [C2],
[Limit1].[C3] AS [C3]
FROM ( SELECT TOP (1)
[Project2].[C1] AS [C1],
[Project2].[C2] AS [C2],
(SELECT
COUNT(1) AS [A1]
FROM [dbo].[Student] AS [Extent3]
WHERE ([Project2].[C1] = 1) AND ([Extent3].[Age] > 5)) AS [C3]
FROM ( SELECT
[Distinct1].[C1] AS [C1],
(SELECT
COUNT(1) AS [A1]
FROM [dbo].[Student] AS [Extent2]
WHERE ([Distinct1].[C1] = 1) AND (N'Test' = [Extent2].[Name])) AS [C2]
FROM ( SELECT DISTINCT
1 AS [C1]
FROM [dbo].[Student] AS [Extent1]
) AS [Distinct1]
) AS [Project2]
) AS [Limit1]
For me this seems to be complex. This can be achieved by following simple query:
select COUNT(CASE WHEN st.Name = 'Test' THEN 1 ELSE 0 END) NameCount,
COUNT(CASE WHEN st.Age > 5 THEN 1 ELSE 0 END) AgeCount from Student st
Is there a way in LINQ with which the SQL query that gets generated will have both the aggregation rather than having it two separate queries joined with nested queries?

From my experience with EF6, conditional Sum (i.e. Sum(condition ? 1 : 0)) is translated much better to SQL than Count with predicate (i.e. Count(condition)):
var query =
from st in school.Students
group st by 1 into grp
select new
{
NameCount = grp.Sum(k => k.Name == "Test" ? 1 : 0),
AgeCount = grp.Sum(k => k.Age > 5 ? 1 : 0)
};
Btw, your SQL example should be using SUM as well. In order to utilize the SQL COUNT which excludes NULLs, it should be ELSE NULL or no ELSE:
select COUNT(CASE WHEN st.Name = 'Test' THEN 1 END) NameCount,
COUNT(CASE WHEN st.Age > 5 THEN 1 END) AgeCount
from Student st
But there is no equivalent LINQ construct for this, hence no way to let EF6 generate such translation. But IMO the SUM is good enough equivalent.

A much simpler query results from not using the unnecessary group by and just querying the table twice in the select:
var sql = from st in school.Students.Take(1)
select new {
NameCount = school.Students.Count(k => k.Name == "Test"),
AgeCount = school.Students.Count(k => k.Age > 5)
};

Output transactions that exceed certain amount within 3 days

In a month, seperate by AccountNo, all transaction that exceeds 10,000 amount within 3 days must be outputted.
This is a sample table:
AccountNo-----Date------------Amount
1-------------5/1/17----------8000
1-------------5/3/17----------1000
1-------------5/4/17----------1000
1-------------5/6/17----------1000
2-------------5/7/17----------3000
2-------------5/10/17---------2000
2-------------5/13/17---------2000
2-------------5/13/17---------3000
3-------------5/14/17---------3000
3-------------5/15/17---------3000
3-------------5/16/17---------9000
4-------------5/17/17---------1000
5-------------5/18/17---------1000
5-------------5/19/17---------1000
5-------------5/20/17---------1000
The result must be:
AccountNo-----Date------------Amount
1-------------5/1/17----------8000
1-------------5/3/17----------1000
1-------------5/4/17----------1000
2-------------5/7/17----------3000
2-------------5/10/17---------2000
2-------------5/13/17---------2000
2-------------5/13/17---------3000
3-------------5/14/17---------3000
3-------------5/15/17---------3000
3-------------5/16/17---------9000
There is a code given to me, but it's not completely working yet.
Select A.*
From YourTable A
Cross Apply (
Select RT1 = sum(case when [Date] <= B2.TstDate then [Amount] else 0 end)
,RT2 = sum(case when [Date] >= B2.TstDate then [Amount] else 0 end)
From YourTable B1
Cross Join (Select TstDate=A.[Date]) B2
Where [Date] between DateAdd(DAY,-2,A.[Date]) and DateAdd(DAY,2,A.[Date])
and Year([Date])=Year(TstDate)
and Month([Date])=Month(TstDate)
) B
Where RT1>=10000 or RT2>=10000

Common table expression: TransInfo
TransInfo is a table with account, year, month, day, amount
The main query:
The [Tmiddle] query adds a row_number based on amount per day per month per accountno
The [Tmiddle] is joined again with TransInfo to limit the results to top 3 of a month AS [Touter] where the top 3 is >= 10.000
The outer most query combines the result with a table again to gain the complete transaction information again
In short:
Filter data on top 3 amounts per day per month per year per account
Check if sum of amount >= 10.000
Show results
Query:
WITH [TransInfo] ([AccountNo], [Year], [Month], [Day], [Amount], [Rownumber])
AS
(
SELECT [AccountNo]
,[Year]
,[Month]
,[Day]
,[Amount]
,ROW_NUMBER() OVER
(
PARTITION BY [AccountNo],
[Year],
[Month]
ORDER BY [Amount] DESC
) AS [Rownumber]
FROM
(
SELECT [AccountNo]
,DATEPART(YEAR, [Date]) AS [Year]
,DATEPART(MONTH, [Date]) AS [Month]
,DATEPART(DAY, [Date]) AS [Day]
,SUM([Amount]) AS [Amount]
FROM [Test].[dbo].[Data]
GROUP BY [AccountNo],
DATEPART(MONTH, [Date]),
DATEPART(YEAR, [Date]),
DATEPART(DAY, [Date])
) AS [Tinner]
)
SELECT [Data].[AccountNo]
,[Data].[Date]
,[Data].[Amount]
FROM [Test].[dbo].[Data]
INNER JOIN
(
SELECT [TransInfo].[AccountNo]
,[TransInfo].[Year]
,[TransInfo].[Month]
,[TransInfo].[Day]
,[TransInfo].[Amount]
FROM [TransInfo]
INNER JOIN
(
SELECT [AccountNo]
,[Year]
,[Month]
FROM [TransInfo]
WHERE [TransInfo].[Rownumber] <= 3
GROUP BY [TransInfo].[AccountNo],
[TransInfo].[Year],
[TransInfo].[Month]
HAVING SUM ([TransInfo].[Amount]) >= 10000
) AS [Tmiddle]
ON [Tmiddle].[AccountNo] = [TransInfo].[AccountNo]
AND [Tmiddle].[Year] = [TransInfo].[Year]
AND [Tmiddle].[Month] = [TransInfo].[Month]
WHERE [TransInfo].[Rownumber] <= 3
) AS [Touter]
ON [Data].[AccountNo] = [TOuter].[AccountNo]
AND DATEPART(YEAR, [Data].[Date]) = [TOuter].[Year]
AND DATEPART(MONTH, [Data].[Date]) = [TOuter].[Month]
AND DATEPART(DAY, [Data].[Date]) = [TOuter].[Day]
Result: Left is query result, right is complete table

Does this work as expected? You may need to change the 2's to 3's depending on what you mean by 'within 3 days'.
SELECT DISTINCT C.*
FROM YourTable C
INNER JOIN (
SELECT a.AccountNo, a.Date
FROM YourTable a
INNER JOIN YourTable b
ON a.AccountNo = b.AccountNo
AND DATEADD(DAY, 2, a.Date) <= b.Date AND b.Date >= a.Date
GROUP BY a.AccountNo, a.Date
HAVING SUM(b.Amount) > 10000
) d
ON C.AccountNo = d.AccountNo
AND DATEADD(DAY, 2, D.Date) <= C.Date AND C.Date >= D.Date

how to compare a string delimited string to a column value in sql without considering sequence

How to compare a string delimited string to a column value in sql without considering sequence?
Suppose I have a value in sql column [fruits] - mango, apple, cherry... I have list in asp.net C# cherry, mango, apple... I want to write sql query such that it can match sql table without order.

I suggest that you look at the fabulous answers in this SO question
How to split a comma-separated value to columns
That said, your solution should be pass each column which contains words to this function and then store it in a table along with a column ID.
So "mango,apple,cherry" becomes a table with values
ColdID Value
_______________
1 mango
1 apple
1 cherry
Now order the tables by ColID ASC, Value ASC and compare both the tables.

This should do it.
DECLARE #str NVARCHAR(MAX)
, #Delim NVARCHAR(255)
SELECT #str = 'cherry,mango,peach,apple'
SELECT #Delim = ','
CREATE TABLE #Fruits ( Fruit VARCHAR(255) )
INSERT INTO #Fruits
( Fruit )
VALUES ( 'cherry' ),
( 'Mango' ),
( 'Apple' ) ,
( 'Banana' )
;WITH lv0 AS (SELECT 0 g UNION ALL SELECT 0)
,lv1 AS (SELECT 0 g FROM lv0 a CROSS JOIN lv0 b) -- 4
,lv2 AS (SELECT 0 g FROM lv1 a CROSS JOIN lv1 b) -- 16
,lv3 AS (SELECT 0 g FROM lv2 a CROSS JOIN lv2 b) -- 256
,lv4 AS (SELECT 0 g FROM lv3 a CROSS JOIN lv3 b) -- 65,536
,lv5 AS (SELECT 0 g FROM lv4 a CROSS JOIN lv4 b) -- 4,294,967,296
,Tally_CTE (n) AS (SELECT ROW_NUMBER() OVER (ORDER BY (SELECT NULL)) FROM lv5)
SELECT SUBSTRING(#str, N, CHARINDEX(#Delim, #str + #Delim, N) - N) AS Item
INTO #StrTable
FROM Tally_CTE
WHERE N BETWEEN 1 AND DATALENGTH(#str) + DATALENGTH(#Delim)
AND SUBSTRING(#Delim + #str, N, LEN(#Delim)) = #Delim;
--#############################################################################
-- in both
--#############################################################################
SELECT *
FROM #Fruits F
JOIN #StrTable ST ON F.Fruit = ST.Item
--#############################################################################
-- in table but not string
--#############################################################################
SELECT *
FROM #Fruits F
LEFT JOIN #StrTable ST ON ST.Item = F.Fruit
WHERE ST.Item IS NULL
--#############################################################################
-- in string but not table
--#############################################################################
SELECT *
FROM #StrTable ST
LEFT JOIN #Fruits F ON ST.Item = F.Fruit
WHERE F.Fruit IS NULL
GO
DROP TABLE #Fruits
DROP TABLE #StrTable

You can use string_split function to do this. I tested this on SQL Server 2017 ctp 2.0 but it should work on 2016 too.
drop table if exists dbo.Fruits;
create table dbo.Fruits (
Fruits varchar(100)
);
insert into dbo.Fruits (Fruits)
values ('cherry,mango,apple'), ('peanut,cherry,mango'),
('apple,cherry,mango')
declare #str varchar(100) = 'apple,mango,cherry';
select
tt.Fruits
, COUNT(tt.value) as Value01
, COUNT(app.value) as Value02
from (
select
*
from dbo.Fruits f
outer apply string_split (f.Fruits, ',') t
) tt
left join string_split (#str, ',') app on tt.value = app.value
group by tt.Fruits

Entity Framework SUM CASE not optimized

I'm trying to write a simple SQL query in LinQ, and no matter how hard I try, I always get a complex query.
Here is the SQL I am trying to achieve (this is not what I'm getting):
SELECT
ClearingAccounts.ID,
SUM(CASE WHEN Payments.StatusID = 1 THEN Payments.TotalAmount ELSE 0 END) AS Sum1,
SUM(CASE WHEN DirectDebits.StatusID = 2 THEN DirectDebits.TotalAmount ELSE 0 END) AS Sum2,
SUM(CASE WHEN Payments.StatusID = 2 THEN Payments.TotalAmount ELSE 0 END) AS Sum3,
SUM(CASE WHEN DirectDebits.StatusID = 1 THEN DirectDebits.TotalAmount ELSE 0 END) AS Sum4
FROM ClearingAccounts
LEFT JOIN Payments ON Payments.ClearingAccountID = ClearingAccounts.ID
LEFT JOIN DirectDebits ON DirectDebits.ClearingAccountID = ClearingAccounts.ID
GROUP BY ClearingAccounts.ID
Here is the code:
from clearingAccount in clearingAccounts
let payments = clearingAccount.Payments
let directDebits = clearingAccount.DirectDebits
select new
{
ID = clearingAccount.ID,
Sum1 = payments.Sum(p => p.StatusID == 1 ? p.TotalAmount : 0),
Sum2 = directDebits.Sum(p => p.StatusID == 2 ? p.TotalAmount : 0),
Sum3 = payments.Sum(p => p.StatusID == 2 ? p.TotalAmount : 0),
Sum4 = directDebits.Sum(p => p.StatusID == 1 ? p.TotalAmount : 0),
}
The generated query gets the data from the respective table for each sum, so four times. I'm not sure if it's even possible to optimize this?
EDIT Here the is generated query:
SELECT
[Project5].[ID] AS [ID],
[Project5].[C1] AS [C1],
[Project5].[C2] AS [C2],
[Project5].[C3] AS [C3],
[Project5].[C4] AS [C4]
FROM ( SELECT
[Project4].[ID] AS [ID],
[Project4].[C1] AS [C1],
[Project4].[C2] AS [C2],
[Project4].[C3] AS [C3],
(SELECT
SUM([Filter5].[A1]) AS [A1]
FROM ( SELECT
CASE WHEN (1 = [Extent5].[StatusID]) THEN [Extent5].[TotalAmount] ELSE cast(0 as decimal(18)) END AS [A1]
FROM [dbo].[DirectDebits] AS [Extent5]
WHERE [Project4].[ID] = [Extent5].[ClearingAccountID]
) AS [Filter5]) AS [C4]
FROM ( SELECT
[Project3].[ID] AS [ID],
[Project3].[C1] AS [C1],
[Project3].[C2] AS [C2],
(SELECT
SUM([Filter4].[A1]) AS [A1]
FROM ( SELECT
CASE WHEN (2 = [Extent4].[StatusID]) THEN [Extent4].[TotalAmount] ELSE cast(0 as decimal(18)) END AS [A1]
FROM [dbo].[Payments] AS [Extent4]
WHERE [Project3].[ID] = [Extent4].[ClearingAccountID]
) AS [Filter4]) AS [C3]
FROM ( SELECT
[Project2].[ID] AS [ID],
[Project2].[C1] AS [C1],
(SELECT
SUM([Filter3].[A1]) AS [A1]
FROM ( SELECT
CASE WHEN (2 = [Extent3].[StatusID]) THEN [Extent3].[TotalAmount] ELSE cast(0 as decimal(18)) END AS [A1]
FROM [dbo].[DirectDebits] AS [Extent3]
WHERE [Project2].[ID] = [Extent3].[ClearingAccountID]
) AS [Filter3]) AS [C2]
FROM ( SELECT
[Project1].[ID] AS [ID],
(SELECT
SUM([Filter2].[A1]) AS [A1]
FROM ( SELECT
CASE WHEN (1 = [Extent2].[StatusID]) THEN [Extent2].[TotalAmount] ELSE cast(0 as decimal(18)) END AS [A1]
FROM [dbo].[Payments] AS [Extent2]
WHERE [Project1].[ID] = [Extent2].[ClearingAccountID]
) AS [Filter2]) AS [C1]
FROM ( SELECT
[Extent1].[ID] AS [ID]
FROM [dbo].[ClearingAccounts] AS [Extent1]
WHERE ([Extent1].[CustomerID] = 3) AND ([Extent1].[Deleted] <> 1)
) AS [Project1]
) AS [Project2]
) AS [Project3]
) AS [Project4]
) AS [Project5]

Edit
Note that as per #usr's comment, that your original Sql Query is broken. By LEFT OUTER joining on two independent tables, and then grouping on the common join key, as soon as one of the DirectDebits or Payments tables returns more than one row, you will erroneously duplicate the TotalAmount value in the 'other' SUMmed colums (and vice versa). e.g. If a given ClearingAccount has 3 DirectDebits and 4 Payments, you will get a total of 12 rows (whereas you should be summing 3 and 4 rows independently for the two tables). A better Sql Query would be:
WITH ctePayments AS
(
SELECT
ClearingAccounts.ID,
-- Note the ELSE 0 projection isn't required as nulls are eliminated from aggregates
SUM(CASE WHEN Payments.StatusID = 1 THEN Payments.TotalAmount END) AS Sum1,
SUM(CASE WHEN Payments.StatusID = 2 THEN Payments.TotalAmount END) AS Sum3
FROM ClearingAccounts
INNER JOIN Payments ON Payments.ClearingAccountID = ClearingAccounts.ID
GROUP BY ClearingAccounts.ID
),
cteDirectDebits AS
(
SELECT
ClearingAccounts.ID,
SUM(CASE WHEN DirectDebits.StatusID = 2 THEN DirectDebits.TotalAmount END) AS Sum2,
SUM(CASE WHEN DirectDebits.StatusID = 1 THEN DirectDebits.TotalAmount END) AS Sum4
FROM ClearingAccounts
INNER JOIN DirectDebits ON DirectDebits.ClearingAccountID = ClearingAccounts.ID
GROUP BY ClearingAccounts.ID
)
SELECT ca.ID, COALESCE(p.Sum1, 0) AS Sum1, COALESCE(d.Sum2, 0) AS Sum2,
COALESCE(p.Sum3, 0) AS Sum3, COALESCE(d.Sum4, 0) AS Sum4
FROM
ClearingAccounts ca
LEFT OUTER JOIN ctePayments p
ON ca.ID = p.ID
LEFT OUTER JOIN cteDirectDebits d
ON ca.ID = d.ID;
-- GROUP BY not required, since we have already guaranteed at most one row
-- per joined table in the CTE's, assuming ClearingAccounts.ID is unique;
You'll want to fix and test this with test cases before you even contemplate conversion to LINQ.
Old Answer(s)
The Sql construct:
SELECT SUM(CASE WHEN ... THEN 1 ELSE 0 END) AS Something
when applied in a SELECT list, is a common hack 'alternative' to pivot data from the 'greater' select into columns which meet the projection criteria (and hence the zero if not matched) . It isn't really a sum at all, its a 'matched' count.
With regards to optimizing the Sql generated, another alternative would be to materialize the data after joining and grouping (and of course, if there is a predicate WHERE clause, apply that in Sql too via IQueryable), and then do the conditional summation in memory:
var result2 = Db.ClearingAccounts
.Include(c => c.Payments)
.Include(c => c.DirectDebits)
.GroupBy(c => c.Id)
.ToList() // or any other means to force materialization here.
.ToDictionary(
grp => grp.Key,
grp => new
{
PaymentsByStatus = grp.SelectMany(x => x.Payments)
.GroupBy(p => p.StatusId),
DirectDebitByStatus = grp.SelectMany(x => x.Payments)
.GroupBy(p => p.StatusId),
})
.Select(ca => new
{
ID = ca.Key,
Sum1 = ca.Value.PaymentsByStatus.Where(pbs => pbs.Key == 1)
.Select(pbs => pbs.Select(x => x.TotalAmount).Sum()),
Sum2 = ca.Value.DirectDebitByStatus.Where(pbs => pbs.Key == 2)
.Select(ddbs => ddbs.Select(x => x.TotalAmount).Sum()),
Sum3 = ca.Value.PaymentsByStatus.Where(pbs => pbs.Key == 2)
.Select(pbs => pbs.Select(x => x.TotalAmount).Sum()),
Sum4 = ca.Value.DirectDebitByStatus.Where(pbs => pbs.Key == 1)
.Select(ddbs => ddbs.Select(x => x.TotalAmount).Sum())
});
However, personally, I would leave this pivot projection directly in Sql, and then use something like SqlQuery to then deserialize the result back from Sql
directly into the final Entity type.

1)
Add AsNoTracking in EF to avoid tracking changes.
Check that you have indexes on the columns you are using for the JOINs. Especially the column that you are using to group by. Profile the query and optimize it. EF has also overhead over a stored procedure.
or
2) If you cannot find a way to make it as fast as you need, create a stored procedure and call it from EF. Even the same query will be faster.

Merging every two rows of data in a column in SQL Server

My table structure is..
Id UserId EventId
1 1 A
2 1 B
3 1 C
4 1 A
5 1 D
The output I need..
UserId EventStart EventEnd
1 A B
1 B C
1 C A
1 A D
I want every two rows to be merged into a row, so if the first row has A and 2nd has B then the first row of result table has A & B..
I have looked into PIVOT but unable to figure out how to get the results I want..
It would be great if I could solve this with sql else if it has to be solved in the middle layer, I'm using C#
Any help is sincerely appreciated..
Thanks..

Assuming that you have have an id column that specifies the ordering, you can get what you want using lead() (in SQL Server 2012+):
select userId, eventid as eventstart,
lead(eventid) over (partition by userid order by id) as eventend
from mytable t;
You are filtering out the last row, which you can do with a subquery (window functions aren't allowed in the where clause):
select t.*
from (select userId, eventid as eventstart,
lead(eventid) over (partition by userid order by id) as eventend
from mytable t
) t
where eventend is null;
In earlier versions of SQL Server, you can get the same effect in other ways, such as a correlated subquery or cross apply. Here is an example:
select t.*
from (select userId, eventid as eventstart,
(select top 1 t2.eventid
from mytable t2
where t2.userid = t.userid and
t2.id > t.id
order by t2.id
) as eventend
from mytable t
) t
where eventend is not null;

An easy approach would be using a CTE with a generated Row_Number() over the ID and joining over UserID and Rownumber.
declare #t Table([ID] [int] IDENTITY(1,1) NOT NULL, UserID int,EventID varchar(10))
insert into #t
Select 1,'A'
UNION ALL Select 1,'B'
UNION ALL Select 1,'C'
UNION ALL Select 1,'A'
UNION ALL Select 1,'D'
UNION ALL Select 2,'B'
UNION ALL Select 2,'C'
UNION ALL Select 2,'A'
UNION ALL Select 2,'D'
;With c as
(
Select UserID,EventID,Row_Number() OVER (Order by UserID,ID ) as RN
from #t
)
Select c1.UserID,c1.EventID as EventStart ,c2.EventID as EventEnd
from c c1
Join c c2 on c2.RN=c1.RN+1 and c2.UserID=c1.UserID