I have designed a recursive SQL CTE expression that sorts a recordset according to its parent Id in a nested hierarchy. How can I execute use this CTE query in my EF6 data context?
I was expecting to find a way to define CTEs in linq statements.
For background, this previous post helped me to identify the CTE:
Order By SQL Query based on Value another Column.
For the purposes of this post I am using a single table in the EF context.
This data model class has been generated from the database using Entity Framework 6 (ADO.NET Entity Data Model)
public partial class Something
{
public int Id{ get; set; }
public string Name{ get; set; }
public string Address { get; set; }
public string Email { get; set; }
public string PhoneNumber { get; set; }
public System.DateTime Date { get; set; }
public int IdParent{ get; set; }
}
And this is the sql query that I want to execute or translate to Linq
with cte (Id, Name, Address, Email, PhoneNumber, Date, IdParent, sort) as
(
select Id, Name, Address, Email, PhoneNumber, Date, IdParent,
cast(right('0000' + cast(row_number() over (order by Id) as varchar(5)), 5) as varchar(1024))
from Something
where Id = IdParent
union all
select t.Id, t.Name, t.Address, t.Email, t.PhoneNumber, t.Date, t.IdParent,
cast(c.sort + right('0000' + cast(row_number() over (order by t.Id) as varchar(5)), 5) as varchar(1024))
from cte c
inner join Something t on c.Id = t.IdParent
where t.Id <> t.IdParent
)
select *
from cte
order by sort
Writing hierachial queries in Linq to SQL is always a mess, it can work in memory but it doesn't translate to efficient SQL queries, this is a good discussion on on SO about some hierarchial Linq techniques
There are a few options:
Don't use Linq at all and query from your CTE directly!
Convert your CTE to a View
Re-write the query so that you don't need the CTE
This is easier if you have a fixed or theoretical limit to the recursion.
Even if you don't want to limit it, if you review the data and find that the highest level of recursion is only 2 or 3, then you could support
How to use a CTE directly in EF 6
DbContext.Database.SqlQuery<TElement>(string sql, params object[] parameters)
Creates a raw SQL query that will return elements of the given type (TElement). The type can be any type that has properties that match the names of the columns returned from the query
Database.SqlQuery on MS Docs
Raw SQL Queries (EF6)
Execute Raw SQL Queries in Entity Framework 6
NOTE: Do NOT use select * for this type (or any) of query, explicitly define the fields that you expect in the output to avoid issues where your query has more columns available than the EF runtime is expecting.
Perhaps of equal importance, if you want or need to apply filtering to this record set, you should implement the filtering in the raw SQL string value. The entire query must be materialized into memory before EF Linq filtering expressions can be applied
.SqlQuery does support passing through parameters, which comes in handy for filter expressions ;)
string cteQuery = #"
with cte (Id, Name, Address, Email, PhoneNumber, Date, IdParent, sort) as
(
select Id, Name, Address, Email, PhoneNumber, Date, IdParent,
cast(right('0000' + cast(row_number() over (order by Id) as varchar(5)), 5) as varchar(1024))
from Something
where Id = IdParent
union all
select t.Id, t.Name, t.Address, t.Email, t.PhoneNumber, t.Date, t.IdParent,
cast(c.sort + right('0000' + cast(row_number() over (order by t.Id) as varchar(5)), 5) as varchar(1024))
from cte c
inner join Something t on c.Id = t.IdParent
where t.Id <> t.IdParent
)
select Id, Name, Address, Email, PhoneNumber, Date, IdParent
from cte
order by sort
";
using (var ctx = new MyDBEntities())
{
var list = ctx.Database
.SqlQuery<Something>(cteQuery)
.ToList();
}
Understanding how and when to use .SqlQuery for executing raw SQL comes in handy when you want to squeeze the most performance out of SQL without writing complex Linq statements.
This comes in handy if you move your CTE into a view or table valued function or a stored procedure, once the results have been materialized into the list in memory, you can treat these records like any other
Convert your CTE to a View
If you are generating your EF model from the database, then you could create a view from your CTE to generate the Something class, however this becomes a bit disconnected if you also want to perform CRUD operations against the same table, having two classes in the model that represent virtually the same structure is a bit redundant IMO, perfectly valid if you want to work that way though.
Views cannot have ORDER BY statements, so you take this statement out of your view definition, but you still include the sort column in the output so that you can sort the results in memory.
Converting your CTE to a view will have the same structure as your current Something class, however it will have an additional column called sort.
How to write the same query without CTE
As I alluded at the start, you can follow this post Hierarchical queries in LINQ to help process the data after bringing the entire list into memory. However in my answer to OPs orginal post, I highlighted how simple self joins on the table can be used to produce the same results, we can easily replicate the self join in EF.
Even when you want to support a theoretically infinitely recursive hierarchy the realty of many datasets is that there is an observable or practical limit to the number of levels. If you can identify that practical limit, and it is a small enough number, then it might be simpler from a C# / Linq perspective to mot bother with the CTE at all
Put it the other way around, ask yourself this question: "If I set a practical limit of X number of levels of recursion, how will that affect my users?"
Put 4 in for X, if the result is that users will not generally be affected, or this data scenario is not likely to occur then lets try it out.
If a limit of 4 is acceptable, then this is your Linq statement:
I've used fluent notation here to demonstrate the relationship to SQL
var list = from child in ctx.Somethings
join parent in ctx.Somethings on child.parentId equals parent.Id
join grandParent in ctx.Somethings on parent.parentId equals grandParent.Id
orderby grandParent.parentId, parent.parentId, child.parentId, child.Id
select child;
I would probably use short hand aliases for this query in production, but the naming convention makes the intended query quickly human relatable.
If you setup a foreign key in the database linking parentId to the Id of the same table, then the Linq side is much simpler
This should generate a navigation property to enable traversing the foreign key through linq, in the following example this property is called Parent
var list = ctx.Somethings
.OrderBy(x => x.Parent.Parent.ParentId)
.ThenBy(x => x.Parent.ParentId)
.ThenBy(x => x.ParentId)
.ThenBy(x => x.Id);
You can see in this way, if we can limit the recusion level, the Linq required for sorting based on the recursive parent is quite simple, and syntactically easy to extrapolate to the number of levels you need.
You could do this for any number of levels, but there is a point where performance might become an issue, or where the number of line of code to achieve this is more than using the SqlQuery option presented first.
I'd recommend you create a view using the SQL provided. Once you create a view, you can map the view to a C# DTO using Entity Framework. After that you can query the view using LINQ.
Don't forget to include the [sort] column in your DTO because you can't include (or at least shouldn't) the sort order in your view definition. You can sort the query using LINQ instead of SQL directly.
Use your CTE With A different ORM
One can call stored procedures from EF and there are a number of posts to that end which you can do... but I recommend that you do a hybrid EF and ADO.Net system which will take advantage of your specialized sql code.
ADO.Net can be used which you will have to write by hand... but there is an ado.net based ORM which uses the principle of returning JSON from SQL Server as models. This ORM can be installed side by side with EF.
It is the Nuget Package SQL-Json (which I am the author) which can use your CTE and provide data as an array of models for your code to use.
Steps
Have your final CTE output return a JSON data by adding for json auto;.
Run the sql and generate json. Take that json to create a C# model using any website which coverts JSON To C# classes. In this example let us call the model class CTEData.
Put your sql into a store procedure.
Include SQL-Json package into your project.
In the model created in step #2 inherit the base class JsonOrmModel.
In the model again add this override public override string GetStoredProcedureName => "[dbo].MyGreatCTE"; with your actual sproc created in step #3.
Get the models:
var connectionStr = #"Data Source=.\Jabberwocky;Initial Catalog=WideWorldImporters";
var jdb = new JsonOrmDatabase(connectionStr);
List<CTEData> ctes = jdb.Get<CTEData>();
Then you can use your cte data as needed.
On the project page it shows how to do what I described with a basic POCO model at SQL-Json.
LINQ for SQL CTE
Here is an approach that applies to many scenarios where LINQ shouldn't/can't/won't/will never produce the SQL you need. This example executes a raw SQL CTE driven by LINQ logic to return the primary key (PK) value's ordinal row number regardless of basic sorts and filters on the entity/table.
The goal is to apply the same constraints to differing requirements. One requirement is the PK row's position w/in those constraints. Another requirement may be the count of rows that satisfy those constraints, etc. Those statistics need to be based on a common constraint broker.
Here, an IQueryable, under the purview of an open DbContext, applies those constraints and is the constraint broker. An alternate approach outside of any DbContext purview is to build expression trees as the constraint broker and return them for evaluation once back under the DbContext umbrella. The shortcut to that is https://github.com/dbelmont/ExpressionBuilder
LINQ could not express Structured Query Language (SQL) Common Table Expressions (CTE) in previous .NET versions. LINQ still can't do that. But...
Hello .NET 5 and my new girlfriend, IQueryable.ToQueryString(). She's the beautiful but potentially lethal kind. Regardless, she gives me all the target row numbers I could ever want.
But, I digress...
/// <summary>
/// Get the ordinal row number of a given primary key value within filter and sort constraints
/// </summary>
/// <param name="TargetCustomerId">The PK value to find across a sorted and filtered record set</param>
/// <returns>The ordinal row number (where the 1st filtered & sorted row is #1 - NOT zero), amongst all other filtered and sorted rows, for further processing - like conversion to page number per a rows-per-page value</returns>
/// <remarks>Doesn't really support fancy ORDER BY clauses here</remarks>
public virtual async Task<int> GetRowNumber(int TargetCustomerId)
{
int rowNumber = -1;
using (MyDbContext context = new MyDbContext())
{
// Always require a record order for row number CTEs
string orderBy = "LastName, FirstName";
// Create a query with a simplistic SELECT but all required Where() criteria
IQueryable<Customer> qrbl = context.Customer
// .Includes are not necessary for filtered row count or the row number CTE
.Include(c => c.SalesTerritory)
.ThenInclude(sr => sr.SalesRegion)
.Where(c => c.AnnualIncome > 30000 && c.SalesTerritory.SalesRegion.SalesRegionName == "South")
.Select(c => c )
;
// The query doesn't necessarily need to be executed...
// ...but for pagination, the filtered row count is valuable for UI stuff - like a "page x of n" pagination control, accurate row sliders or scroll bars, etc.
// int custCount = Convert.ToInt32(await qrbl.CountAsync());
// Extract LINQ's rendered SQL
string sqlCustomer = qrbl.ToQueryString();
// Remove the 1st/outer "SELECT" clause from that extracted sql
int posFrom = sqlCustomer.IndexOf("FROM [schemaname].[Customer] ");
string cteFrom = sqlCustomer.Substring(posFrom);
/*
If you must get a row number from a more complex query, where LINQ nests SELECTs, this approach might be more appropriate.
string[] clauses = sqlCustomer.Split("\r\n", StringSplitOptions.TrimEntries);
int posFrom = clauses
.Select((clause, index) => new { Clause = clause, Index = index })
.First(ci => ci.Clause.StartsWith("FROM "))
.Index
;
string cteFrom = string.Join("\r\n", clauses, posFrom, clauses.Length - posFrom);
*/
// As always w/ all raw sql, prohibit sql injection, etc.
string sqlCte = "WITH cte AS\r\n"
+ $"\t(SELECT [CustomerId], ROW_NUMBER() OVER(ORDER BY {orderBy}) AS RowNumber {cteFrom})\r\n"
+ $"SELECT #RowNumber = RowNumber FROM cte WHERE [CustomerId] = {TargetCustomerId}"
;
SqlParameter paramRow = new SqlParameter("RowNumber", System.Data.SqlDbType.Int);
paramRow.Direction = System.Data.ParameterDirection.Output;
int rows = await context.Database.ExecuteSqlRawAsync(sqlCte, paramRow).ConfigureAwait(false);
if (paramRow.Value != null)
{
rowNumber = (int)paramRow.Value;
}
}
return rowNumber;
}
Related
I may have a slightly naive question, but I have never worked with databases before. I am a .NET engineer and I use Dapper to access the SQL Server database.
The situation is the following: I have a denormalized table for persisting several types of entities. Each has a composite key (type, id, owner_id) and each row of the key is of string type (but it's not important). And, let's say, I'm writing to the database many interests for different users (Bulk post). In order for them to not repeat, I need to make a query and determine which are already present in the database.
So, I have this code in my InterestService class:
private IEnumerable<Interest> GetAlreadyExistingInterestsFor(IEnumerable<Interest> interestsForCreating) =>
_interestRepository.GetInterests(interestsForCreating.Select(interest => interest.Id).ToList(),
interestsForCreating.Select(interest => interest.UserId).ToList());
After that I have some logic and so on. It's not important.
InterestRepository method GetInterests looks like this:
public GetInterests(IList<string> interestIds, IList<string> userIds)
{
var query = #"SELECT type, id, owner_id
FROM entities
WHERE type = 'interest'
AND id IN #InterestIds
AND owner_id IN #UserIds";
return _dbContext.ExecuteQuery(query, new { InterestIds = interestIds, UserIds = userIds });
}
The code may have mistakes because right now I don't have an ability to access a working environment but I think the idea is clear. So, the question is whether this is the best approach to making a query. And if there is a better, then what is it.
Essentially you can simply do exactly what was done in this post, but with two sets instead of 1 for the table valued parameter.
Using Dapper, how do I pass in the values for a sql type as param?
It uses a stored procedure and a sql table valued parameter.
If stored procedure is not an option then you can use one of following methods.
Convert your interestIds and userIds into strings
string interests = "(1, 2, 3, 4)" if lists contains numbers only or ('a1', 'b1', 'c2') if they are strings.
Then just inline them into your query
var query = #"SELECT type, id, owner_id
FROM entities
WHERE type = 'interest'
AND id IN "+ interests
+ " AND owner_id IN " + users;
This method is considered as bad practice and is an invitation to SQL injection attack (in the case of user input). Use it iff you are absolutely sure in your data.
SQL Server 2016+ has a built-in function string_split which can be used here. The function splits a string with separators into a table t(value).
Again convert lists into strings string interests="a1, a2, b3, c4"; (No single quot here)
and query
var query = #"SELECT type, id, owner_id
FROM entities
WHERE type = 'interest'
AND id IN (select value from string_split(#interests,','))
AND owner_id IN (select value from string_split(#users,','))";
For earlier versions you can create UDF with the same functionality.
create function dbo.split_string(#input varchar(max), #separator varchar(2))
returns table as
return
(with cte as(
select cast('<v>'+ REPLACE(#input, #separator, '</v><v>') +'</v>' as xml) x
)
select t.v.value('.[1]', 'varchar(max)') [value]
from cte
cross apply x.nodes('v') t(v))
I have a C# application that uses SQLite as the database and the SQLite Entity Framework 6 provider to generate queries based on user input.
The database contains the following tables and indexes:
CREATE TABLE Lists (
ListRowId INTEGER NOT NULL PRIMARY KEY,
ListId GUID NOT NULL,
ListName TEXT NOT NULL
);
CREATE UNIQUE INDEX [IX_Lists_ListId] ON [Lists] ( [ListId] );
-- One to many relationship: Lists => ListDetails
CREATE TABLE ListDetails (
ListDetailRowId INTEGER NOT NULL PRIMARY KEY,
ListDetailId GUID NOT NULL,
ListId GUID NOT NULL,
Plate TEXT
);
CREATE INDEX [IX_ListDetails_Plate] ON [ListDetails] ( [Plate] ASC );
CREATE TABLE Reads (
ReadRowId INTEGER NOT NULL PPRIMARY KEY,
ReadId GUID NOT NULL,
Plate TEXT
);
-- 1 To many relationship: Reads => Alarms.
-- There may be rows in Reads that have no related rows in Alarms.
CREATE TABLE Alarms (
AlarmRowId INTEGER NOT NULL PPRIMARY KEY,
AlarmId GUID NOT NULL,
ListId GUID NOT NULL,
ListDetailId GUID NOT NULL,
ReadRowId INTEGER NOT NULL
);
CREATE INDEX [IX_Alarms_ListId_ListDetailId] ON [Alarms] ([ListId], [ListDetailId]);
CREATE INDEX [IX_Alarms_ReadId] ON [Alarms] ([ReadRowId]);
Please note that the DDL above only includes the relevant columns and indexes. For reasons of speed and the large number of rows in the ListDetails table, there is no index on the ListDetailId GUID column; nor can I create one. In fact, I cannot change the database's schema at all.
The database does not have any foreign key relationships defined between any of these tables. The reason is internal to our system. I repeat, I cannot change the schema.
Using the SQLite EF6 provider, I've built an entity model from the database. It is a database first model as the application was originally written using a different database and EF 4. We upgraded it to EF 6 and replaced the database with SQLite.
While processing user input, I have to put together a query that joins these tables. Here's the basic EF expression I've built.
from read in context.Reads
join alrm in context.Alarms on read.ReadRowId equals alrm.ReadRowId into alarmJoin
from alarm in alarmJoin.DefaultIfEmpty()
join e in context.ListDetails on alarm.ListPlate equals e.Plate into entryJoin
from entry in entryJoin.DefaultIfEmpty()
join l in context.Lists on alarm.ListId equals l.ListId into listJoin
from list in listJoin.DefaultIfEmpty()
where alarm.ListDetailId = entry.ListDetailId
select new {
alarm,
list.ListName,
read
};
I've used the debugger to take that expression and generate the SQL. I've reduced the output for brevity, as the only part I'm interested in are the join on the ListDetails table:
SELECT *
FROM [Reads] AS [Extent1]
LEFT OUTER JOIN [Alarms] AS [Extent2] ON [Extent1].[ReadRowId] = [Extent2].[ReadRowId]
LEFT OUTER JOIN [ListDetails] AS [Extent3] ON ([Extent2].[ListPlate] = [Extent3].[Plate]) OR (([Extent2].[ListPlate] IS NULL) AND ([Extent3].[Plate] IS NULL))
LEFT OUTER JOIN [Lists] AS [Extent4] ON [Extent2].[ListId] = [Extent4].[ListId]
WHERE ([Extent2].[ListDetailId] = [Extent3].[ListDetailId]) OR (([Extent2].[ListDetailId] IS NULL) AND ([Extent3].[ListDetailId] IS NULL))
Executing EXPLAIN QUERY PLAN on this shows that the query will perform a table scan of the ListDetails table. I do not want that to happen; I want the query to use the index on the Plate column.
If I remove the where clause, the SQL that's generated is different:
SELECT *
FROM [Reads] AS [Extent1]
LEFT OUTER JOIN [Alarms] AS [Extent2] ON [Extent1].[ReadRowId] = [Extent2].[ReadRowId]
LEFT OUTER JOIN [ListDetails] AS [Extent3] ON ([Extent2].[ListPlate] = [Extent3].[Plate]) OR (([Extent2].[ListPlate] IS NULL) AND ([Extent3].[Plate] IS NULL))
LEFT OUTER JOIN [Lists] AS [Extent4] ON [Extent2].[ListId] = [Extent4].[ListId]
EXPLAIN QUERY PLAN on this query shows that the database does indeed use the index on the ListDetails table's Plate column. This is what I want to happen. But, there may be multiple rows in the ListDetails table that have the same Plate; it is not a unique field. I need to return the one and only row that matches the information available to me in the Alarms table.
How do I make my query use the index on the Plate column?
Specifying an index requires a query hint. SqlLite uses the INDEXED BY command. Example:
LEFT OUTER JOIN ListDetails as Extent3 INDEXED BY IX_ListDetails_Plate ON Extent2.ListPlate = Extent3.Plate
LINQ does not provide a method to pass a query hint to the database. LINQ's design philosophy is the developer shouldn't worry about the SQL: that is the DBA's job.
So there probably won't be a .With() LINQ extension coming anytime soon.
However, there are several options / workarounds:
1. The "Proper" Way
The "proper" way per LINQ's design philosophy is for a DBA to create a sproc that uses query hints.
The developer will call the sproc with Entity, and get a strongly typed sproc result.
using(applicationDbContext db = new applicationDbContext())
{
var myStronglyTypedResult = db.Database.MySprocMethod();
}
Easiest way, with Entity handling the translations and class creations. However, you will need permission to create the sproc. And it sounds like you do not have that option.
2. Old School
If LINQ doesn't want to use query hints, then don't use LINQ for the query. Simple enough. Back to DataAdapters and hard coded SQL queries. You already have the query designed, might as well use it.
3. DataContext's Sql Interface
DataContext has a SQL interface built in: The SqlQuery<T>(string sql, object[] params) method. Database.SqlQuery
public class ListDetail
{
public int ListDetailRowId {get; set;}
public Guid ListDetialId {get; set;}
public Guid ListId {get; set;}
public string Plate {get; set;}
}
using(ApplicationDbContext db = new ApplicationDbContext())
{
List<ListDetail> results = db.Database.SqlQuery<ListDetail>("SELECT * FROM ListDetails INDEXED BY IX_my_index WHERE ListDetailRowId = #p0", new object[] {50}).ToList();
return results;
}
This gives you the flexibility of straight SQL, you don't have to mess with connection strings nor DataAdapters, and Entity / DataContext handles the translation from DataTable to Entity for you.
However, you will need to manually create the Entity class. It is the same as any other Entity class, it just won't be automatically created like the sproc method does.
This will probably be your best bet.
While this was a while ago, and I am no longer in that job, I wanted to take a minute to describe how we got around this problem. The problems are:
SQLite does not support Stored Procedures, so there's no way to work around the problem from the database side,
You can't embed the INDEXED BY hint into the LINQ query.
The way we ended up fixing this was by implementing a custom user function in the entity model which added the required INDEXED BY hint to the SQL generated by EF. We also implemented a couple of other user functions for a few other SQL Hints supported by SQLite. This allowed up to put the condition for the join that required the hint inside of our user function and EF did the rest.
As I said, I'm no longer in that position, so I can't include any code, but it's just a matter of adding some XML to the entity model file that defines the user functions and defining a class that has placeholder functions. This is all documented in the EF documents.
I just wondering if I am wasting my time or is there anything I could to improve this query which in turn will improve performance.
Inside a Repository, I am trying to get the 10 most recent items
public List<entity> GetTop10
{
get
{
return m_Context.entity.Where(x => x.bool == false).OrderByDescending(x => x.Created).Take(10).ToList();
}
}
But this is taking a long time as the table its querying has over 11000 rows in it. So my question is, is there anyway I could speed up this kind of query?
I am trying to get my SQL hat on regarding performance, I know the order would slow it down, but how I could I achieve the same result?
Thanks
The particular query you posted is a potential candidate for using a filtered index. Say you have a SQL table:
CREATE TABLE Employees
(
ID INT IDENTITY(1,1) PRIMARY KEY,
Name NVARCHAR(100),
IsAlive BIT
)
You can imagine that generally you only want to query on employees that have not (yet) died so will end up with SQL like this:
SELECT Name FROM Employees WHERE IsAlive = 1
So, why not create a filtered index:
CREATE INDEX IX_Employees_IsAliveTrue
ON Employees(IsAlive)
WHERE IsAlive = 1
So now if you query the table it will use this index which may only be a small portion of your table, especially if you've had a recent zombie invasion and 90% of your staff are now the walking dead.
However, an Entity Framework like this:
var nonZombies = from e in db.Employees
where e.IsAlive == true
select e;
May not be able to use the index (SQL has a problem with filtered indexes and parameterised queries). To get round this, you can create a view in your database:
CREATE VIEW NonZombies
AS
SELECT ID, Name, IsAlive FROM Employees WHERE IsAlive = 1
Now you can add that to your framework (how you do this will vary depending on if you are using code/model/database first) and you will now be able to decide which employees deserve urgent attention (like priority access to food and weapons):
var nonZombies = from e in db.NonZombies
select e;
From your LINQ query will be created SQL SELECT similar to this:
SELECT TOP(10) * FROM entity
WHERE bool = 0
ORDER BY Created DESC
Similar because instead of the '*' will server select concrete columns to map these to entity object.
If this is too slow for you. The error is in the database, not in the EntityFramework.
So try adding some indexes to your table.
I have 2 tables
TableA:
TableAID int,
Col1 varchar(8)
TableB:
TableBID int
Col1 char(8),
Col2 varchar(40)
When I run a SQL query on the 2 tables it returns the following number of rows
SELECT * FROM tableA (7200 rows)
select * FROM tableB (28030 rows)
When joined on col1 and selects the data it returns the following number of rows
select DISTINCT a.Col1,b.Col2 FROM tableA a
join tableB b on a.Col1=b.Col1 (6578 rows)
The above 2 tables on different databases so I created 2 EF models and retried the data separately and tried to join them in the code using linq with the following function. Surprisingly it returns 2886 records instead of 6578 records. Am I doing something wrong?
The individual lists seems to return the correct data but when I join them SQL query and linq query differs in the number of records.
Any help on this greatly appreciated.
// This function is returning 2886 records
public List<tableC_POCO_Object> Get_TableC()
{
IEnumerable<tableC_POCO_Object> result = null;
List<TableA> tableA_POCO_Object = Get_TableA(); // Returns 7200 records
List<TableB> tableB_POCO_Object = Get_TableB(); // Returns 28030 records
result = from tbla in tableA_POCO_Object
join tblb in tableB_POCO_Object on tbla.Col1 equals tblb.Col1
select new tableC_POCO_Object
{
Col1 = tblb.Col1,
Col2 = tbla.Col2
};
return result.Distinct().ToList();
}
The problem lies in the fact that in your POCO world, you're trying to compare two strings using a straight comparison (meaning it's case-sensitive). That might work in the SQL world (unless of course you've enabled case-sensitivity), but doesn't quite work so well when you have "stringA" == "StringA". What you should do is normalize the join columns to be all upper or lower case:
join tblb in tableB_POCO_Object on tbla.Col1.ToUpper() equals tblb.Col1.ToUpper()
Join operator creates a lookup using the specified keys (starts with second collection) and joins the original table/collection back by checking the generated lookup, so if the hashes ever differ they will not join.
Point being, joining OBJECT collections on string data/properties is bad unless you normalize to the same cAsE. For LINQ to some DB provider, if the database is case-insensitive, then this won't matter, but it always matters in the CLR/L2O world.
Edit: Ahh, didn't realize it was CHAR(8) instead of VARCHAR(8), meaning it pads to 8 characters no matter what. In that case, tblb.Col1.Trim() will fix your issue. However, still keep this in mind when dealing with LINQ to Objects queries.
This might happen because you compare a VARCHAR and a CHAR column. In SQL, this depends on the settings of ANSI_PADDING on the sql server, while in C# the string values are read using the DataReader and compared using standard string functions.
Try tblb.Col1.Trim() in your LINQ statement.
As SPFiredrake correctly pointed out this can be caused by case sensitivity, but I also have to ask you why did you write your code in such a way, why not this way:
// This function is returning 2886 records
public List<tableC_POCO_Object> Get_TableC()
{
return from tbla in Get_TableA()
join tblb in Get_TableB() on tbla.Col1 equals tblb.Col1
select new tableC_POCO_Object
{
Col1 = tblb.Col1,
Col2 = tbla.Col2
}.Distinct().ToList();
}
where Get_TableA() and Get_TableB() return IEnumerable instead of List. You have to watch out for that, because when you convert to list the query will be executed instantly. You want to send a single query to the database server.
I'm trying to better utilize the resources of the Entity Sql in the following scenario: I have a table Book which has a Many-To-Many relationship with the Author table. Each book may have from 0 to N authors. I would like to sort the books by the first author name, ie the first record found in this relationship (or null when no authors are linked to a book).
With T-SQL it can be done without difficulty:
SELECT
b.*
FROM
Book AS b
JOIN BookAuthor AS ba ON b.BookId = ba.BookId
JOIN Author AS a ON ba.AuthorId = a.AuthorId
ORDER BY
a.AuthorName;
But I cannot think of how to adapt my code bellow to achieve it. Indeed I don't know how to write something equivalent directly with Entity Sql too.
Entities e = new Entities();
var books = e.Books;
var query = books.Include("Authors");
if (sorting == null)
query = query.OrderBy("it.Title asc");
else
query = query.OrderBy("it.Authors.Name asc"); // This isn't it.
return query.Skip(paging.Skip).Take(paging.Take).ToList();
Could someone explain me how to modify my code to generate the Entity Sql for the desired result? Or even explain me how to write by hand a query using CreateQuery<Book>() to achieve it?
EDIT
Just to elucidate, I'll be working with a very large collection of books (around 100k). Sorting them in memory would be very impactful on the performance. I wish the answers would focus on how to generate the desired ordering using Entity Sql, so the orderby will happens on the database.
The OrderBy method expects you to give it a lambda expression (well, actually a Func delegate, but most people would use lambdas to make them) that can be run to select the field to sort by. Also, OrderBy always orders ascending; if you want descending order there is an OrderByDescending method.
var query = books
.Include("Authors")
.OrderBy(book => book.Authors.Any()
? book.Authors.FirstOrDefault().Name
: string.Empty);
This is basically telling the OrderBy method: "for each book in the sequence, if there are any authors, select the first one's name as my sort key; otherwise, select the empty string. Then return me the books sorted by the sort key."
You could put anything in place of the string.Empty, including for example book.Title or any other property of the book to use in place of the last name for sorting.
EDIT from comments:
As long as the sorting behavior you ask for isn't too complex, the Entity Framework's query provider can usually figure out how to turn it into SQL. It will try really, really hard to do that, and if it can't you'll get a query error. The only time the sorting would be done in client-side objects is if you forced the query to run (e.g. .AsEnumerable()) before the OrderBy was called.
In this case, the EF outputs a select statement that includes the following calculated field:
CASE WHEN ( EXISTS (SELECT
1 AS [C1]
FROM [dbo].[BookAuthor] AS [Extent4]
WHERE [Extent1].[Id] = [Extent4].[Books_Id]
)) THEN [Limit1].[Name] ELSE #p__linq__0 END AS [C1],
Then orders by that.
#p__linq__0 is a parameter, passed in as string.Empty, so you can see it converted the lambda expression into SQL pretty directly. Extent and Limit are just aliases used in the generated SQL for the joined tables etc. Extent1 is [Books] and Limit1 is:
SELECT TOP (1) -- Field list goes here.
FROM [dbo].[BookAuthor] AS [Extent2]
INNER JOIN [dbo].[Authors] AS [Extent3] ON [Extent3].[Id] = [Extent2].[Authors_Id]
WHERE [Extent1].[Id] = [Extent2].[Books_Id]
If you don't care where the sorting is happening (i.e. SQL vs In Code), you can retrieve your result set, and sort it using your own sorting code after the query results have been returned. In my experience, getting specialized sorting like this to work with Entity Framework can be very difficult, frustrating and time consuming.