Develop Reference

.netcore EF linq - this is a BUG? Very strange behavior

I have two table in sql. Document and User. Document have relation to User and I want to get users that I sent document recently.
I need to sort by the date document was sent and get unique (distinct) user with relation to this document
This is my linq queries
var recentClients = documentCaseRepository.Entities
.Where(docCase => docCase.AssignedByAgentId == WC.UserContext.UserId)
.OrderByDescending(userWithDate => userWithDate.LastUpdateDate)
.Take(1000) // I need this because if I comment this line then EF generate completely different sql query.
.Select(doc => new { doc.AssignedToClient.Id, doc.AssignedToClient.FirstName, doc.AssignedToClient.LastName })
.Distinct()
.Take(configuration.MaxRecentClientsResults)
.ToList();
and generated sql query is:
SELECT DISTINCT TOP(5) [t].*
FROM (
SELECT TOP(1000) [docCase.AssignedToClient].[Id]
FROM [DocumentCase] AS [docCase]
INNER JOIN [User] AS [docCase.AssignedToClient]
ON ([docCase].[AssignedToClientId] = [docCase.AssignedToClient].[Id])
WHERE [docCase].[AssignedByAgentId] = 3
ORDER BY [docCase].[LastUpdateDate] DESC
)
AS [t]
Every thing is correct for now. But if I delete this line
.Take(1000) // I need this because...
EF generated completely different query such as:
SELECT DISTINCT TOP(5)
[docCase.AssignedToClient].[Id]
FROM [DocumentCase] AS [docCase]
INNER JOIN [User] AS [docCase.AssignedToClient]
ON ([docCase].[AssignedToClientId] = [docCase.AssignedToClient].[Id])
WHERE [docCase].[AssignedByAgentId] = 3
My question is: why EF not generated orderby clause and subquery with distinct?
This is a BUG EF or I'm doing something wrong? And what I must do to generate in linq this sql query ()
SELECT DISTINCT TOP 5 [t].*
FROM ( SELECT [docCase.AssignedToClient].[Id]
FROM [DocumentCase] AS [docCase]
INNER JOIN [User] AS [docCase.AssignedToClient]
ON [docCase].[AssignedToClientId] = [docCase.AssignedToClient].[Id]
WHERE [docCase].[AssignedByAgentId] = 1
ORDER BY [docCase].[LastUpdateDate] DESC
) AS [t]

OrderBy information not always retained across other operators such as Distinct. Entity Framework does not document (to my knowledge) how exactly OrderBy is propagated.
This kind of makes sense because some operators have undefined output order. The fact that ordering is retained in many situations is a convenience for the developer.
Move the OrderBy to the end of the query (or at least past the Distinct).

The reason for the difference in queries is that Distinct messes up result order. So when you first execute OrderBy and then Distinct, you can just es well not execute OrderBy, because this order is lost anyway. So EF can just optimize it away.
Calling Take in between causes the result set to be semantically different: You first order the items, take the first 1000 items of that order and then call Distinct on them.
What you can change in your query depends mainly on the result you want to achieve. Maybe you want to first make the result set distinct then order by date and finally take the amount of items. Other options are also thinkable based on your requirements.

How to make an algorithm to get all column names from across tables that relations with their primary key in SQL Server and C#?

I have 3 tables (for example 3, but in real over than 30 tables with this conditions) in my SQL Server database: post, user, person.
post: (post_id, post_text, user_id)
user: (user_id, user_name, person_id)
person: (person_id, person_phone, person_email)
Now, in C#, I want an algorithm that creates a query that get result like this:
post.post_id, post.post_text, post.user_id, user.user_id, user.user_name, user.person_id, person.person_id, person.person_email
and I use this method for fill a SqlDataReader in C# for reading and accessing all column values from these records.
I know that the common way to get that result directly and manually using of 'Join' statement, but it is waste time if tables count is very much. So, I want an algorithm that generates this query for manipulate in C# programming.
Thanks a lot.

To query all column names from your tables, you can use:
SELECT obj.Name + '.' + col.Name AS name
FROM sys.columns col
INNER JOIN sys.objects obj
ON obj.object_id = col.object_id
WHERE obj.Name IN ('post', 'user', 'person')
ORDER BY name
Then, for how to call this from C# using SqlDataReader, you can use this documentation: https://msdn.microsoft.com/en-us/library/haa3afyz(v=vs.110).aspx

select post1.post_id, post1.post_text, post1.user_id, user1.user_id, user1.user_name, user1.person_id, person1.person_id, person1.person_email from post post1 inner join user user1 on user1.user_id=post1.user_id inner join person person1 on person1.person_id=user1.person_id

Forcing an Entity Framework 6 query to use the correct index

I have a C# application that uses SQLite as the database and the SQLite Entity Framework 6 provider to generate queries based on user input.
The database contains the following tables and indexes:
CREATE TABLE Lists (
ListRowId INTEGER NOT NULL PRIMARY KEY,
ListId GUID NOT NULL,
ListName TEXT NOT NULL
);
CREATE UNIQUE INDEX [IX_Lists_ListId] ON [Lists] ( [ListId] );
-- One to many relationship: Lists => ListDetails
CREATE TABLE ListDetails (
ListDetailRowId INTEGER NOT NULL PRIMARY KEY,
ListDetailId GUID NOT NULL,
ListId GUID NOT NULL,
Plate TEXT
);
CREATE INDEX [IX_ListDetails_Plate] ON [ListDetails] ( [Plate] ASC );
CREATE TABLE Reads (
ReadRowId INTEGER NOT NULL PPRIMARY KEY,
ReadId GUID NOT NULL,
Plate TEXT
);
-- 1 To many relationship: Reads => Alarms.
-- There may be rows in Reads that have no related rows in Alarms.
CREATE TABLE Alarms (
AlarmRowId INTEGER NOT NULL PPRIMARY KEY,
AlarmId GUID NOT NULL,
ListId GUID NOT NULL,
ListDetailId GUID NOT NULL,
ReadRowId INTEGER NOT NULL
);
CREATE INDEX [IX_Alarms_ListId_ListDetailId] ON [Alarms] ([ListId], [ListDetailId]);
CREATE INDEX [IX_Alarms_ReadId] ON [Alarms] ([ReadRowId]);
Please note that the DDL above only includes the relevant columns and indexes. For reasons of speed and the large number of rows in the ListDetails table, there is no index on the ListDetailId GUID column; nor can I create one. In fact, I cannot change the database's schema at all.
The database does not have any foreign key relationships defined between any of these tables. The reason is internal to our system. I repeat, I cannot change the schema.
Using the SQLite EF6 provider, I've built an entity model from the database. It is a database first model as the application was originally written using a different database and EF 4. We upgraded it to EF 6 and replaced the database with SQLite.
While processing user input, I have to put together a query that joins these tables. Here's the basic EF expression I've built.
from read in context.Reads
join alrm in context.Alarms on read.ReadRowId equals alrm.ReadRowId into alarmJoin
from alarm in alarmJoin.DefaultIfEmpty()
join e in context.ListDetails on alarm.ListPlate equals e.Plate into entryJoin
from entry in entryJoin.DefaultIfEmpty()
join l in context.Lists on alarm.ListId equals l.ListId into listJoin
from list in listJoin.DefaultIfEmpty()
where alarm.ListDetailId = entry.ListDetailId
select new {
alarm,
list.ListName,
read
};
I've used the debugger to take that expression and generate the SQL. I've reduced the output for brevity, as the only part I'm interested in are the join on the ListDetails table:
SELECT *
FROM [Reads] AS [Extent1]
LEFT OUTER JOIN [Alarms] AS [Extent2] ON [Extent1].[ReadRowId] = [Extent2].[ReadRowId]
LEFT OUTER JOIN [ListDetails] AS [Extent3] ON ([Extent2].[ListPlate] = [Extent3].[Plate]) OR (([Extent2].[ListPlate] IS NULL) AND ([Extent3].[Plate] IS NULL))
LEFT OUTER JOIN [Lists] AS [Extent4] ON [Extent2].[ListId] = [Extent4].[ListId]
WHERE ([Extent2].[ListDetailId] = [Extent3].[ListDetailId]) OR (([Extent2].[ListDetailId] IS NULL) AND ([Extent3].[ListDetailId] IS NULL))
Executing EXPLAIN QUERY PLAN on this shows that the query will perform a table scan of the ListDetails table. I do not want that to happen; I want the query to use the index on the Plate column.
If I remove the where clause, the SQL that's generated is different:
SELECT *
FROM [Reads] AS [Extent1]
LEFT OUTER JOIN [Alarms] AS [Extent2] ON [Extent1].[ReadRowId] = [Extent2].[ReadRowId]
LEFT OUTER JOIN [ListDetails] AS [Extent3] ON ([Extent2].[ListPlate] = [Extent3].[Plate]) OR (([Extent2].[ListPlate] IS NULL) AND ([Extent3].[Plate] IS NULL))
LEFT OUTER JOIN [Lists] AS [Extent4] ON [Extent2].[ListId] = [Extent4].[ListId]
EXPLAIN QUERY PLAN on this query shows that the database does indeed use the index on the ListDetails table's Plate column. This is what I want to happen. But, there may be multiple rows in the ListDetails table that have the same Plate; it is not a unique field. I need to return the one and only row that matches the information available to me in the Alarms table.
How do I make my query use the index on the Plate column?

Specifying an index requires a query hint. SqlLite uses the INDEXED BY command. Example:
LEFT OUTER JOIN ListDetails as Extent3 INDEXED BY IX_ListDetails_Plate ON Extent2.ListPlate = Extent3.Plate
LINQ does not provide a method to pass a query hint to the database. LINQ's design philosophy is the developer shouldn't worry about the SQL: that is the DBA's job.
So there probably won't be a .With() LINQ extension coming anytime soon.
However, there are several options / workarounds:
1. The "Proper" Way
The "proper" way per LINQ's design philosophy is for a DBA to create a sproc that uses query hints.
The developer will call the sproc with Entity, and get a strongly typed sproc result.
using(applicationDbContext db = new applicationDbContext())
{
var myStronglyTypedResult = db.Database.MySprocMethod();
}
Easiest way, with Entity handling the translations and class creations. However, you will need permission to create the sproc. And it sounds like you do not have that option.
2. Old School
If LINQ doesn't want to use query hints, then don't use LINQ for the query. Simple enough. Back to DataAdapters and hard coded SQL queries. You already have the query designed, might as well use it.
3. DataContext's Sql Interface
DataContext has a SQL interface built in: The SqlQuery<T>(string sql, object[] params) method. Database.SqlQuery
public class ListDetail
{
public int ListDetailRowId {get; set;}
public Guid ListDetialId {get; set;}
public Guid ListId {get; set;}
public string Plate {get; set;}
}
using(ApplicationDbContext db = new ApplicationDbContext())
{
List<ListDetail> results = db.Database.SqlQuery<ListDetail>("SELECT * FROM ListDetails INDEXED BY IX_my_index WHERE ListDetailRowId = #p0", new object[] {50}).ToList();
return results;
}
This gives you the flexibility of straight SQL, you don't have to mess with connection strings nor DataAdapters, and Entity / DataContext handles the translation from DataTable to Entity for you.
However, you will need to manually create the Entity class. It is the same as any other Entity class, it just won't be automatically created like the sproc method does.
This will probably be your best bet.

While this was a while ago, and I am no longer in that job, I wanted to take a minute to describe how we got around this problem. The problems are:
SQLite does not support Stored Procedures, so there's no way to work around the problem from the database side,
You can't embed the INDEXED BY hint into the LINQ query.
The way we ended up fixing this was by implementing a custom user function in the entity model which added the required INDEXED BY hint to the SQL generated by EF. We also implemented a couple of other user functions for a few other SQL Hints supported by SQLite. This allowed up to put the condition for the join that required the hint inside of our user function and EF did the rest.
As I said, I'm no longer in that position, so I can't include any code, but it's just a matter of adding some XML to the entity model file that defines the user functions and defining a class that has placeholder functions. This is all documented in the EF documents.

Joining Lists using Linq returns different result than corresponding SQL query?

I have 2 tables
TableA:
TableAID int,
Col1 varchar(8)
TableB:
TableBID int
Col1 char(8),
Col2 varchar(40)
When I run a SQL query on the 2 tables it returns the following number of rows
SELECT * FROM tableA (7200 rows)
select * FROM tableB (28030 rows)
When joined on col1 and selects the data it returns the following number of rows
select DISTINCT a.Col1,b.Col2 FROM tableA a
join tableB b on a.Col1=b.Col1 (6578 rows)
The above 2 tables on different databases so I created 2 EF models and retried the data separately and tried to join them in the code using linq with the following function. Surprisingly it returns 2886 records instead of 6578 records. Am I doing something wrong?
The individual lists seems to return the correct data but when I join them SQL query and linq query differs in the number of records.
Any help on this greatly appreciated.
// This function is returning 2886 records
public List<tableC_POCO_Object> Get_TableC()
{
IEnumerable<tableC_POCO_Object> result = null;
List<TableA> tableA_POCO_Object = Get_TableA(); // Returns 7200 records
List<TableB> tableB_POCO_Object = Get_TableB(); // Returns 28030 records
result = from tbla in tableA_POCO_Object
join tblb in tableB_POCO_Object on tbla.Col1 equals tblb.Col1
select new tableC_POCO_Object
{
Col1 = tblb.Col1,
Col2 = tbla.Col2
};
return result.Distinct().ToList();
}

The problem lies in the fact that in your POCO world, you're trying to compare two strings using a straight comparison (meaning it's case-sensitive). That might work in the SQL world (unless of course you've enabled case-sensitivity), but doesn't quite work so well when you have "stringA" == "StringA". What you should do is normalize the join columns to be all upper or lower case:
join tblb in tableB_POCO_Object on tbla.Col1.ToUpper() equals tblb.Col1.ToUpper()
Join operator creates a lookup using the specified keys (starts with second collection) and joins the original table/collection back by checking the generated lookup, so if the hashes ever differ they will not join.
Point being, joining OBJECT collections on string data/properties is bad unless you normalize to the same cAsE. For LINQ to some DB provider, if the database is case-insensitive, then this won't matter, but it always matters in the CLR/L2O world.
Edit: Ahh, didn't realize it was CHAR(8) instead of VARCHAR(8), meaning it pads to 8 characters no matter what. In that case, tblb.Col1.Trim() will fix your issue. However, still keep this in mind when dealing with LINQ to Objects queries.

This might happen because you compare a VARCHAR and a CHAR column. In SQL, this depends on the settings of ANSI_PADDING on the sql server, while in C# the string values are read using the DataReader and compared using standard string functions.
Try tblb.Col1.Trim() in your LINQ statement.

As SPFiredrake correctly pointed out this can be caused by case sensitivity, but I also have to ask you why did you write your code in such a way, why not this way:
// This function is returning 2886 records
public List<tableC_POCO_Object> Get_TableC()
{
return from tbla in Get_TableA()
join tblb in Get_TableB() on tbla.Col1 equals tblb.Col1
select new tableC_POCO_Object
{
Col1 = tblb.Col1,
Col2 = tbla.Col2
}.Distinct().ToList();
}
where Get_TableA() and Get_TableB() return IEnumerable instead of List. You have to watch out for that, because when you convert to list the query will be executed instantly. You want to send a single query to the database server.

OrderBy a Many To Many relationship with Entity Sql

I'm trying to better utilize the resources of the Entity Sql in the following scenario: I have a table Book which has a Many-To-Many relationship with the Author table. Each book may have from 0 to N authors. I would like to sort the books by the first author name, ie the first record found in this relationship (or null when no authors are linked to a book).
With T-SQL it can be done without difficulty:
SELECT
b.*
FROM
Book AS b
JOIN BookAuthor AS ba ON b.BookId = ba.BookId
JOIN Author AS a ON ba.AuthorId = a.AuthorId
ORDER BY
a.AuthorName;
But I cannot think of how to adapt my code bellow to achieve it. Indeed I don't know how to write something equivalent directly with Entity Sql too.
Entities e = new Entities();
var books = e.Books;
var query = books.Include("Authors");
if (sorting == null)
query = query.OrderBy("it.Title asc");
else
query = query.OrderBy("it.Authors.Name asc"); // This isn't it.
return query.Skip(paging.Skip).Take(paging.Take).ToList();
Could someone explain me how to modify my code to generate the Entity Sql for the desired result? Or even explain me how to write by hand a query using CreateQuery<Book>() to achieve it?
EDIT
Just to elucidate, I'll be working with a very large collection of books (around 100k). Sorting them in memory would be very impactful on the performance. I wish the answers would focus on how to generate the desired ordering using Entity Sql, so the orderby will happens on the database.

The OrderBy method expects you to give it a lambda expression (well, actually a Func delegate, but most people would use lambdas to make them) that can be run to select the field to sort by. Also, OrderBy always orders ascending; if you want descending order there is an OrderByDescending method.
var query = books
.Include("Authors")
.OrderBy(book => book.Authors.Any()
? book.Authors.FirstOrDefault().Name
: string.Empty);
This is basically telling the OrderBy method: "for each book in the sequence, if there are any authors, select the first one's name as my sort key; otherwise, select the empty string. Then return me the books sorted by the sort key."
You could put anything in place of the string.Empty, including for example book.Title or any other property of the book to use in place of the last name for sorting.
EDIT from comments:
As long as the sorting behavior you ask for isn't too complex, the Entity Framework's query provider can usually figure out how to turn it into SQL. It will try really, really hard to do that, and if it can't you'll get a query error. The only time the sorting would be done in client-side objects is if you forced the query to run (e.g. .AsEnumerable()) before the OrderBy was called.
In this case, the EF outputs a select statement that includes the following calculated field:
CASE WHEN ( EXISTS (SELECT
1 AS [C1]
FROM [dbo].[BookAuthor] AS [Extent4]
WHERE [Extent1].[Id] = [Extent4].[Books_Id]
)) THEN [Limit1].[Name] ELSE #p__linq__0 END AS [C1],
Then orders by that.
#p__linq__0 is a parameter, passed in as string.Empty, so you can see it converted the lambda expression into SQL pretty directly. Extent and Limit are just aliases used in the generated SQL for the joined tables etc. Extent1 is [Books] and Limit1 is:
SELECT TOP (1) -- Field list goes here.
FROM [dbo].[BookAuthor] AS [Extent2]
INNER JOIN [dbo].[Authors] AS [Extent3] ON [Extent3].[Id] = [Extent2].[Authors_Id]
WHERE [Extent1].[Id] = [Extent2].[Books_Id]

If you don't care where the sorting is happening (i.e. SQL vs In Code), you can retrieve your result set, and sort it using your own sorting code after the query results have been returned. In my experience, getting specialized sorting like this to work with Entity Framework can be very difficult, frustrating and time consuming.