I have designed a recursive SQL CTE expression that sorts a recordset according to its parent Id in a nested hierarchy. How can I execute use this CTE query in my EF6 data context?
I was expecting to find a way to define CTEs in linq statements.
For background, this previous post helped me to identify the CTE:
Order By SQL Query based on Value another Column.
For the purposes of this post I am using a single table in the EF context.
This data model class has been generated from the database using Entity Framework 6 (ADO.NET Entity Data Model)
public partial class Something
{
public int Id{ get; set; }
public string Name{ get; set; }
public string Address { get; set; }
public string Email { get; set; }
public string PhoneNumber { get; set; }
public System.DateTime Date { get; set; }
public int IdParent{ get; set; }
}
And this is the sql query that I want to execute or translate to Linq
with cte (Id, Name, Address, Email, PhoneNumber, Date, IdParent, sort) as
(
select Id, Name, Address, Email, PhoneNumber, Date, IdParent,
cast(right('0000' + cast(row_number() over (order by Id) as varchar(5)), 5) as varchar(1024))
from Something
where Id = IdParent
union all
select t.Id, t.Name, t.Address, t.Email, t.PhoneNumber, t.Date, t.IdParent,
cast(c.sort + right('0000' + cast(row_number() over (order by t.Id) as varchar(5)), 5) as varchar(1024))
from cte c
inner join Something t on c.Id = t.IdParent
where t.Id <> t.IdParent
)
select *
from cte
order by sort
Writing hierachial queries in Linq to SQL is always a mess, it can work in memory but it doesn't translate to efficient SQL queries, this is a good discussion on on SO about some hierarchial Linq techniques
There are a few options:
Don't use Linq at all and query from your CTE directly!
Convert your CTE to a View
Re-write the query so that you don't need the CTE
This is easier if you have a fixed or theoretical limit to the recursion.
Even if you don't want to limit it, if you review the data and find that the highest level of recursion is only 2 or 3, then you could support
How to use a CTE directly in EF 6
DbContext.Database.SqlQuery<TElement>(string sql, params object[] parameters)
Creates a raw SQL query that will return elements of the given type (TElement). The type can be any type that has properties that match the names of the columns returned from the query
Database.SqlQuery on MS Docs
Raw SQL Queries (EF6)
Execute Raw SQL Queries in Entity Framework 6
NOTE: Do NOT use select * for this type (or any) of query, explicitly define the fields that you expect in the output to avoid issues where your query has more columns available than the EF runtime is expecting.
Perhaps of equal importance, if you want or need to apply filtering to this record set, you should implement the filtering in the raw SQL string value. The entire query must be materialized into memory before EF Linq filtering expressions can be applied
.SqlQuery does support passing through parameters, which comes in handy for filter expressions ;)
string cteQuery = #"
with cte (Id, Name, Address, Email, PhoneNumber, Date, IdParent, sort) as
(
select Id, Name, Address, Email, PhoneNumber, Date, IdParent,
cast(right('0000' + cast(row_number() over (order by Id) as varchar(5)), 5) as varchar(1024))
from Something
where Id = IdParent
union all
select t.Id, t.Name, t.Address, t.Email, t.PhoneNumber, t.Date, t.IdParent,
cast(c.sort + right('0000' + cast(row_number() over (order by t.Id) as varchar(5)), 5) as varchar(1024))
from cte c
inner join Something t on c.Id = t.IdParent
where t.Id <> t.IdParent
)
select Id, Name, Address, Email, PhoneNumber, Date, IdParent
from cte
order by sort
";
using (var ctx = new MyDBEntities())
{
var list = ctx.Database
.SqlQuery<Something>(cteQuery)
.ToList();
}
Understanding how and when to use .SqlQuery for executing raw SQL comes in handy when you want to squeeze the most performance out of SQL without writing complex Linq statements.
This comes in handy if you move your CTE into a view or table valued function or a stored procedure, once the results have been materialized into the list in memory, you can treat these records like any other
Convert your CTE to a View
If you are generating your EF model from the database, then you could create a view from your CTE to generate the Something class, however this becomes a bit disconnected if you also want to perform CRUD operations against the same table, having two classes in the model that represent virtually the same structure is a bit redundant IMO, perfectly valid if you want to work that way though.
Views cannot have ORDER BY statements, so you take this statement out of your view definition, but you still include the sort column in the output so that you can sort the results in memory.
Converting your CTE to a view will have the same structure as your current Something class, however it will have an additional column called sort.
How to write the same query without CTE
As I alluded at the start, you can follow this post Hierarchical queries in LINQ to help process the data after bringing the entire list into memory. However in my answer to OPs orginal post, I highlighted how simple self joins on the table can be used to produce the same results, we can easily replicate the self join in EF.
Even when you want to support a theoretically infinitely recursive hierarchy the realty of many datasets is that there is an observable or practical limit to the number of levels. If you can identify that practical limit, and it is a small enough number, then it might be simpler from a C# / Linq perspective to mot bother with the CTE at all
Put it the other way around, ask yourself this question: "If I set a practical limit of X number of levels of recursion, how will that affect my users?"
Put 4 in for X, if the result is that users will not generally be affected, or this data scenario is not likely to occur then lets try it out.
If a limit of 4 is acceptable, then this is your Linq statement:
I've used fluent notation here to demonstrate the relationship to SQL
var list = from child in ctx.Somethings
join parent in ctx.Somethings on child.parentId equals parent.Id
join grandParent in ctx.Somethings on parent.parentId equals grandParent.Id
orderby grandParent.parentId, parent.parentId, child.parentId, child.Id
select child;
I would probably use short hand aliases for this query in production, but the naming convention makes the intended query quickly human relatable.
If you setup a foreign key in the database linking parentId to the Id of the same table, then the Linq side is much simpler
This should generate a navigation property to enable traversing the foreign key through linq, in the following example this property is called Parent
var list = ctx.Somethings
.OrderBy(x => x.Parent.Parent.ParentId)
.ThenBy(x => x.Parent.ParentId)
.ThenBy(x => x.ParentId)
.ThenBy(x => x.Id);
You can see in this way, if we can limit the recusion level, the Linq required for sorting based on the recursive parent is quite simple, and syntactically easy to extrapolate to the number of levels you need.
You could do this for any number of levels, but there is a point where performance might become an issue, or where the number of line of code to achieve this is more than using the SqlQuery option presented first.
I'd recommend you create a view using the SQL provided. Once you create a view, you can map the view to a C# DTO using Entity Framework. After that you can query the view using LINQ.
Don't forget to include the [sort] column in your DTO because you can't include (or at least shouldn't) the sort order in your view definition. You can sort the query using LINQ instead of SQL directly.
Use your CTE With A different ORM
One can call stored procedures from EF and there are a number of posts to that end which you can do... but I recommend that you do a hybrid EF and ADO.Net system which will take advantage of your specialized sql code.
ADO.Net can be used which you will have to write by hand... but there is an ado.net based ORM which uses the principle of returning JSON from SQL Server as models. This ORM can be installed side by side with EF.
It is the Nuget Package SQL-Json (which I am the author) which can use your CTE and provide data as an array of models for your code to use.
Steps
Have your final CTE output return a JSON data by adding for json auto;.
Run the sql and generate json. Take that json to create a C# model using any website which coverts JSON To C# classes. In this example let us call the model class CTEData.
Put your sql into a store procedure.
Include SQL-Json package into your project.
In the model created in step #2 inherit the base class JsonOrmModel.
In the model again add this override public override string GetStoredProcedureName => "[dbo].MyGreatCTE"; with your actual sproc created in step #3.
Get the models:
var connectionStr = #"Data Source=.\Jabberwocky;Initial Catalog=WideWorldImporters";
var jdb = new JsonOrmDatabase(connectionStr);
List<CTEData> ctes = jdb.Get<CTEData>();
Then you can use your cte data as needed.
On the project page it shows how to do what I described with a basic POCO model at SQL-Json.
LINQ for SQL CTE
Here is an approach that applies to many scenarios where LINQ shouldn't/can't/won't/will never produce the SQL you need. This example executes a raw SQL CTE driven by LINQ logic to return the primary key (PK) value's ordinal row number regardless of basic sorts and filters on the entity/table.
The goal is to apply the same constraints to differing requirements. One requirement is the PK row's position w/in those constraints. Another requirement may be the count of rows that satisfy those constraints, etc. Those statistics need to be based on a common constraint broker.
Here, an IQueryable, under the purview of an open DbContext, applies those constraints and is the constraint broker. An alternate approach outside of any DbContext purview is to build expression trees as the constraint broker and return them for evaluation once back under the DbContext umbrella. The shortcut to that is https://github.com/dbelmont/ExpressionBuilder
LINQ could not express Structured Query Language (SQL) Common Table Expressions (CTE) in previous .NET versions. LINQ still can't do that. But...
Hello .NET 5 and my new girlfriend, IQueryable.ToQueryString(). She's the beautiful but potentially lethal kind. Regardless, she gives me all the target row numbers I could ever want.
But, I digress...
/// <summary>
/// Get the ordinal row number of a given primary key value within filter and sort constraints
/// </summary>
/// <param name="TargetCustomerId">The PK value to find across a sorted and filtered record set</param>
/// <returns>The ordinal row number (where the 1st filtered & sorted row is #1 - NOT zero), amongst all other filtered and sorted rows, for further processing - like conversion to page number per a rows-per-page value</returns>
/// <remarks>Doesn't really support fancy ORDER BY clauses here</remarks>
public virtual async Task<int> GetRowNumber(int TargetCustomerId)
{
int rowNumber = -1;
using (MyDbContext context = new MyDbContext())
{
// Always require a record order for row number CTEs
string orderBy = "LastName, FirstName";
// Create a query with a simplistic SELECT but all required Where() criteria
IQueryable<Customer> qrbl = context.Customer
// .Includes are not necessary for filtered row count or the row number CTE
.Include(c => c.SalesTerritory)
.ThenInclude(sr => sr.SalesRegion)
.Where(c => c.AnnualIncome > 30000 && c.SalesTerritory.SalesRegion.SalesRegionName == "South")
.Select(c => c )
;
// The query doesn't necessarily need to be executed...
// ...but for pagination, the filtered row count is valuable for UI stuff - like a "page x of n" pagination control, accurate row sliders or scroll bars, etc.
// int custCount = Convert.ToInt32(await qrbl.CountAsync());
// Extract LINQ's rendered SQL
string sqlCustomer = qrbl.ToQueryString();
// Remove the 1st/outer "SELECT" clause from that extracted sql
int posFrom = sqlCustomer.IndexOf("FROM [schemaname].[Customer] ");
string cteFrom = sqlCustomer.Substring(posFrom);
/*
If you must get a row number from a more complex query, where LINQ nests SELECTs, this approach might be more appropriate.
string[] clauses = sqlCustomer.Split("\r\n", StringSplitOptions.TrimEntries);
int posFrom = clauses
.Select((clause, index) => new { Clause = clause, Index = index })
.First(ci => ci.Clause.StartsWith("FROM "))
.Index
;
string cteFrom = string.Join("\r\n", clauses, posFrom, clauses.Length - posFrom);
*/
// As always w/ all raw sql, prohibit sql injection, etc.
string sqlCte = "WITH cte AS\r\n"
+ $"\t(SELECT [CustomerId], ROW_NUMBER() OVER(ORDER BY {orderBy}) AS RowNumber {cteFrom})\r\n"
+ $"SELECT #RowNumber = RowNumber FROM cte WHERE [CustomerId] = {TargetCustomerId}"
;
SqlParameter paramRow = new SqlParameter("RowNumber", System.Data.SqlDbType.Int);
paramRow.Direction = System.Data.ParameterDirection.Output;
int rows = await context.Database.ExecuteSqlRawAsync(sqlCte, paramRow).ConfigureAwait(false);
if (paramRow.Value != null)
{
rowNumber = (int)paramRow.Value;
}
}
return rowNumber;
}
I have a C# application that uses SQLite as the database and the SQLite Entity Framework 6 provider to generate queries based on user input.
The database contains the following tables and indexes:
CREATE TABLE Lists (
ListRowId INTEGER NOT NULL PRIMARY KEY,
ListId GUID NOT NULL,
ListName TEXT NOT NULL
);
CREATE UNIQUE INDEX [IX_Lists_ListId] ON [Lists] ( [ListId] );
-- One to many relationship: Lists => ListDetails
CREATE TABLE ListDetails (
ListDetailRowId INTEGER NOT NULL PRIMARY KEY,
ListDetailId GUID NOT NULL,
ListId GUID NOT NULL,
Plate TEXT
);
CREATE INDEX [IX_ListDetails_Plate] ON [ListDetails] ( [Plate] ASC );
CREATE TABLE Reads (
ReadRowId INTEGER NOT NULL PPRIMARY KEY,
ReadId GUID NOT NULL,
Plate TEXT
);
-- 1 To many relationship: Reads => Alarms.
-- There may be rows in Reads that have no related rows in Alarms.
CREATE TABLE Alarms (
AlarmRowId INTEGER NOT NULL PPRIMARY KEY,
AlarmId GUID NOT NULL,
ListId GUID NOT NULL,
ListDetailId GUID NOT NULL,
ReadRowId INTEGER NOT NULL
);
CREATE INDEX [IX_Alarms_ListId_ListDetailId] ON [Alarms] ([ListId], [ListDetailId]);
CREATE INDEX [IX_Alarms_ReadId] ON [Alarms] ([ReadRowId]);
Please note that the DDL above only includes the relevant columns and indexes. For reasons of speed and the large number of rows in the ListDetails table, there is no index on the ListDetailId GUID column; nor can I create one. In fact, I cannot change the database's schema at all.
The database does not have any foreign key relationships defined between any of these tables. The reason is internal to our system. I repeat, I cannot change the schema.
Using the SQLite EF6 provider, I've built an entity model from the database. It is a database first model as the application was originally written using a different database and EF 4. We upgraded it to EF 6 and replaced the database with SQLite.
While processing user input, I have to put together a query that joins these tables. Here's the basic EF expression I've built.
from read in context.Reads
join alrm in context.Alarms on read.ReadRowId equals alrm.ReadRowId into alarmJoin
from alarm in alarmJoin.DefaultIfEmpty()
join e in context.ListDetails on alarm.ListPlate equals e.Plate into entryJoin
from entry in entryJoin.DefaultIfEmpty()
join l in context.Lists on alarm.ListId equals l.ListId into listJoin
from list in listJoin.DefaultIfEmpty()
where alarm.ListDetailId = entry.ListDetailId
select new {
alarm,
list.ListName,
read
};
I've used the debugger to take that expression and generate the SQL. I've reduced the output for brevity, as the only part I'm interested in are the join on the ListDetails table:
SELECT *
FROM [Reads] AS [Extent1]
LEFT OUTER JOIN [Alarms] AS [Extent2] ON [Extent1].[ReadRowId] = [Extent2].[ReadRowId]
LEFT OUTER JOIN [ListDetails] AS [Extent3] ON ([Extent2].[ListPlate] = [Extent3].[Plate]) OR (([Extent2].[ListPlate] IS NULL) AND ([Extent3].[Plate] IS NULL))
LEFT OUTER JOIN [Lists] AS [Extent4] ON [Extent2].[ListId] = [Extent4].[ListId]
WHERE ([Extent2].[ListDetailId] = [Extent3].[ListDetailId]) OR (([Extent2].[ListDetailId] IS NULL) AND ([Extent3].[ListDetailId] IS NULL))
Executing EXPLAIN QUERY PLAN on this shows that the query will perform a table scan of the ListDetails table. I do not want that to happen; I want the query to use the index on the Plate column.
If I remove the where clause, the SQL that's generated is different:
SELECT *
FROM [Reads] AS [Extent1]
LEFT OUTER JOIN [Alarms] AS [Extent2] ON [Extent1].[ReadRowId] = [Extent2].[ReadRowId]
LEFT OUTER JOIN [ListDetails] AS [Extent3] ON ([Extent2].[ListPlate] = [Extent3].[Plate]) OR (([Extent2].[ListPlate] IS NULL) AND ([Extent3].[Plate] IS NULL))
LEFT OUTER JOIN [Lists] AS [Extent4] ON [Extent2].[ListId] = [Extent4].[ListId]
EXPLAIN QUERY PLAN on this query shows that the database does indeed use the index on the ListDetails table's Plate column. This is what I want to happen. But, there may be multiple rows in the ListDetails table that have the same Plate; it is not a unique field. I need to return the one and only row that matches the information available to me in the Alarms table.
How do I make my query use the index on the Plate column?
Specifying an index requires a query hint. SqlLite uses the INDEXED BY command. Example:
LEFT OUTER JOIN ListDetails as Extent3 INDEXED BY IX_ListDetails_Plate ON Extent2.ListPlate = Extent3.Plate
LINQ does not provide a method to pass a query hint to the database. LINQ's design philosophy is the developer shouldn't worry about the SQL: that is the DBA's job.
So there probably won't be a .With() LINQ extension coming anytime soon.
However, there are several options / workarounds:
1. The "Proper" Way
The "proper" way per LINQ's design philosophy is for a DBA to create a sproc that uses query hints.
The developer will call the sproc with Entity, and get a strongly typed sproc result.
using(applicationDbContext db = new applicationDbContext())
{
var myStronglyTypedResult = db.Database.MySprocMethod();
}
Easiest way, with Entity handling the translations and class creations. However, you will need permission to create the sproc. And it sounds like you do not have that option.
2. Old School
If LINQ doesn't want to use query hints, then don't use LINQ for the query. Simple enough. Back to DataAdapters and hard coded SQL queries. You already have the query designed, might as well use it.
3. DataContext's Sql Interface
DataContext has a SQL interface built in: The SqlQuery<T>(string sql, object[] params) method. Database.SqlQuery
public class ListDetail
{
public int ListDetailRowId {get; set;}
public Guid ListDetialId {get; set;}
public Guid ListId {get; set;}
public string Plate {get; set;}
}
using(ApplicationDbContext db = new ApplicationDbContext())
{
List<ListDetail> results = db.Database.SqlQuery<ListDetail>("SELECT * FROM ListDetails INDEXED BY IX_my_index WHERE ListDetailRowId = #p0", new object[] {50}).ToList();
return results;
}
This gives you the flexibility of straight SQL, you don't have to mess with connection strings nor DataAdapters, and Entity / DataContext handles the translation from DataTable to Entity for you.
However, you will need to manually create the Entity class. It is the same as any other Entity class, it just won't be automatically created like the sproc method does.
This will probably be your best bet.
While this was a while ago, and I am no longer in that job, I wanted to take a minute to describe how we got around this problem. The problems are:
SQLite does not support Stored Procedures, so there's no way to work around the problem from the database side,
You can't embed the INDEXED BY hint into the LINQ query.
The way we ended up fixing this was by implementing a custom user function in the entity model which added the required INDEXED BY hint to the SQL generated by EF. We also implemented a couple of other user functions for a few other SQL Hints supported by SQLite. This allowed up to put the condition for the join that required the hint inside of our user function and EF did the rest.
As I said, I'm no longer in that position, so I can't include any code, but it's just a matter of adding some XML to the entity model file that defines the user functions and defining a class that has placeholder functions. This is all documented in the EF documents.
I have these two tables
ExpiredAccount Account
-------------- ---------------
ExpiredAccountID AccountID
AccountID (fk) AccountName
... ...
Basically, I want to return a list of ExpiredAccounts displaying the AccountName in the result.
I currently do this using
var expiredAccounts = (from x in ExpiredAccount
join m in Account on x.AccountID equals m.AccountID
select m.AccountName).ToList()
This works fine. However, this takes too long.
There's not a lot of records in expiredAccounts (<200).
The Account table on the otherhand has over 300,000 records.
Is there anyway I could speed up my query, or alternatively, another way to do this more efficiently with or without using LINQ?
Firstly, assuming you are using Entity Framework, you don't need to be using the join at all. You could simply do:
var expiredAccounts = (from x in ExpiredAccount
select x.Account.AccountName).ToList()
However, I don't think they will generate a different query plan on the database. But my guess is that you don't have an index on AccountID in the Account table (although that seems unlikely).
One thing you can do is use ToTraceString (for example: http://social.msdn.microsoft.com/Forums/en-US/adodotnetentityframework/thread/4a17b992-05ca-4e3b-9910-0018e7cc9c8c/) to get the SQL which is being run. Then you can open SQL Management Studio and run that with the execution plan option turned on and it will show you what the execution plan was and what indexes need to be added to make it better.
You can try using Contains method:
var expiredAccounts = (from m in Account where ExpiredAccount.Select(x => x.AccountId)
.Contains(m.AccountId)
select m.AccountName).ToList()
It should generate IN clause in SQL query that will be performed agains database.
I have a large inventory system, and I'm having to re-write part of the I/O portion of it. At its heart, there's a product table and a set of related tables. I need to be able to read pieces of it as efficiently as possible. From C# I construct this query:
select * -- includes productid
into #tt
from products where productClass = 547 -- possibly more conditions
select * from #tt;
select * from productHistory where productid in (select productid from #tt);
select * from productSuppliers where productid in (select productid from #tt);
select * from productSafetyInfo where productid in (select productid from #tt);
select * from productMiscInfo where productid in (select productid from #tt);
drop table #tt;
This query gives me exactly the results I need: 5 result sets each having zero, one or more records (if the first returns zero rows, the others do as well, of course). The program then takes those result sets and crams them into an appropriate DataSet. (Which then gets handed off into a constructor expecting just these records.) This query (with differing conditions) gets run a lot.
My question is, is there a more efficient way to retrieve this data?
Re-working this as a single join won't work because each child might return a variable number of rows.
If you have an index on products.productClass this might yield better performance.
select * from products where productClass = 547 -- includes productid
select productHistory.*
from productHistory
join products
on products.productid = productHistory.productid
and products,productClass = 547;
...
If productID is a clustered index then you will probalbly get better permance with
CREATE TABLE #Temp (productid INT PRIMARY KEY CLUSTERED);
insert into #temp
select productid from products where productClass = 547
order by productid;
go
select productHistory.*
from productHistory
join #Temp
on #Temp.productid = productHistory.productid;
A join on a clustered index seems to give the best performance.
Think about it - SQL can match the first and know it can forget about the rest then move to the second knowing it can move foward (not go back to the top).
With a where in (select ..) SQL cannot take advantage of order.
The more tables you need to join the more reason to #temp as you take about 1/2 second hit creating on populating the #temp.
If you are going to #temp you might as well make it a stuctured temp.
Make sure when you JOIN tables you are joining on indexes. Otherwise you will end up with table scans vs index scans and your code will be very slow specially when joining large tables.
Best practice is to optimize your SQL Queries to avoid table scans.
If you don't have it already, I would strongly suggest making this a stored procedure.
Also, I suspect, but can't prove without testing it, that you will get better performance if you perform joins on the products table for each of your subtables rather than copying into a local table.
Finally, unless you can combine the data, I don't think there is a more efficient way to do this.
Without seeing your schema and knowing a little more about your data and table sizes, it's hard to suggest definitive improvements on the query side.
However, instead of "cramming the results into an appropriate DataSet," since you are using a batched command to return multiple result sets, you could use SqlDataAdapter to do that part for you:
SqlDataAdapter adapter = new SqlDataAdapter(cmd);
DataSet results = new DataSet();
adapter.Fill(results);
After that, the first result set will be in results.Tables[0], the second in results.Tables[1], etc.
I currently have an SQL query which is currently called multiple times like this (pseudo code):
foreach(KeyValuePair kvp in someMapList)
{
select col1 from table where col2 = kvp.key and col3 = kvp.value;
//do some processing on the returned value
}
The above could obviously call the database a large number of times if it is a big list of pairs.
Can anyone think of a more efficient piece of SQL which could essentially return a list of specific values based on a list of two unique pieces of information so that the processing can be done in bulk? One obvious one would be to build up a big piece of SQL with ORs but this would probably be quite inefficient?
Thanks
Carl
Insert KeyValuePair into a table and use a JOIN man
or (if c#)
from record in table
join key in someMap on record.col2 equals key.key && record.col3 equals key.keyvalue
select col1 //or some anonymous type or a DTO I do not know...
That is WAY more efficeint:D
Assuming that you're stuck with the entity-value pair pattern, your best bet is to either create a stored procedure and pass in a delimited string of your pairs which you can then turn into a temporary table and select out using a join, or if you can pass in a table parameter that would be even better.
Another possibility would be to insert into a work table with some kind of unique connection ID for your values and then join against that.
In any case, your goal is to be able to select the data using a set-based method by getting your pairs into the database in some way.