Develop Reference

Fluent Linq and SQL Server

This may be a silly question, but I have fluent linq query that is pulling data off a DB, but, of course, it processes the data in memory, not on the DB.
Is there a means of taking a query (Say table.Where(t => t. id > 22)) and having it run as a db query (i.e. SELECT * FROM TABLE WHERE ID > 22).
I want to use the fluent style because I am building up the query from various places. I know I can make this specific query into query syntax, but my actual queries are more complex.
I have tried standard searches, but I suspect it is not going to be possible, as nothing comes up.
Yes using EF Core.
The code is not necessarily clear:
var getTableData = table.Where(base.whereSelection);
whereSelection.ForEach(w => getTableData = getTableData.Where(w));
At the moment, all of the Where Selections are Funcs - they could be converted to something else (like Expressions) if I knew that would make them run on the DB. The rest of the code is also building up this set of details programatically:
var selectedData = getTableData.Select(Convert);
var sortedTableData = GetSort(selectedData, Ordering);
var sorted = PickDataFrom(sortedTableData);
return CheckAndTake(sorted, pageSize);
Everything seems to work. Just in the wrong place.

Too many parameters were provided in this RPC request. The maximum is 2100 [duplicate]

from f in CUSTOMERS
where depts.Contains(f.DEPT_ID)
select f.NAME
depts is a list (IEnumerable<int>) of department ids
This query works fine until you pass a large list (say around 3000 dept ids) .. then I get this error:
The incoming tabular data stream (TDS) remote procedure call (RPC) protocol stream is incorrect. Too many parameters were provided in this RPC request. The maximum is 2100.
I changed my query to:
var dept_ids = string.Join(" ", depts.ToStringArray());
from f in CUSTOMERS
where dept_ids.IndexOf(Convert.ToString(f.DEPT_id)) != -1
select f.NAME
using IndexOf() fixed the error but made the query slow. Is there any other way to solve this? thanks so much.

My solution (Guids is a list of ids you would like to filter by):
List<MyTestEntity> result = new List<MyTestEntity>();
for(int i = 0; i < Math.Ceiling((double)Guids.Count / 2000); i++)
{
var nextGuids = Guids.Skip(i * 2000).Take(2000);
result.AddRange(db.Tests.Where(x => nextGuids.Contains(x.Id)));
}
this.DataContext = result;

Why not write the query in sql and attach your entity?
It's been awhile since I worked in Linq, but here goes:
IQuery q = Session.CreateQuery(#"
select *
from customerTable f
where f.DEPT_id in (" + string.Join(",", depts.ToStringArray()) + ")");
q.AttachEntity(CUSTOMER);
Of course, you will need to protect against injection, but that shouldn't be too hard.

You will want to check out the LINQKit project since within there somewhere is a technique for batching up such statements to solve this issue. I believe the idea is to use the PredicateBuilder to break the local collection into smaller chuncks but I haven't reviewed the solution in detail because I've instead been looking for a more natural way to handle this.
Unfortunately it appears from Microsoft's response to my suggestion to fix this behavior that there are no plans set to have this addressed for .NET Framework 4.0 or even subsequent service packs.
https://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=475984
UPDATE:
I've opened up some discussion regarding whether this was going to be fixed for LINQ to SQL or the ADO.NET Entity Framework on the MSDN forums. Please see these posts for more information regarding these topics and to see the temporary workaround that I've come up with using XML and a SQL UDF.

I had similar problem, and I got two ways to fix it.
Intersect method
join on IDs
To get values that are NOT in list, I used Except method OR left join.
Update
EntityFramework 6.2 runs the following query successfully:
var employeeIDs = Enumerable.Range(3, 5000);
var orders =
from order in Orders
where employeeIDs.Contains((int)order.EmployeeID)
select order;

Your post was from a while ago, but perhaps someone will benefit from this. Entity Framework does a lot of query caching, every time you send in a different parameter count, that gets added to the cache. Using a "Contains" call will cause SQL to generate a clause like "WHERE x IN (#p1, #p2.... #pn)", and bloat the EF cache.
Recently I looked for a new way to handle this, and I found that you can create an entire table of data as a parameter. Here's how to do it:
First, you'll need to create a custom table type, so run this in SQL Server (in my case I called the custom type "TableId"):
CREATE TYPE [dbo].[TableId] AS TABLE(
Id[int] PRIMARY KEY
)
Then, in C#, you can create a DataTable and load it into a structured parameter that matches the type. You can add as many data rows as you want:
DataTable dt = new DataTable();
dt.Columns.Add("id", typeof(int));
This is an arbitrary list of IDs to search on. You can make the list as large as you want:
dt.Rows.Add(24262);
dt.Rows.Add(24267);
dt.Rows.Add(24264);
Create an SqlParameter using the custom table type and your data table:
SqlParameter tableParameter = new SqlParameter("#id", SqlDbType.Structured);
tableParameter.TypeName = "dbo.TableId";
tableParameter.Value = dt;
Then you can call a bit of SQL from your context that joins your existing table to the values from your table parameter. This will give you all records that match your ID list:
var items = context.Dailies.FromSqlRaw<Dailies>("SELECT * FROM dbo.Dailies d INNER JOIN #id id ON d.Daily_ID = id.id", tableParameter).AsNoTracking().ToList();

You could always partition your list of depts into smaller sets before you pass them as parameters to the IN statement generated by Linq. See here:
Divide a large IEnumerable into smaller IEnumerable of a fix amount of item

Entity Framework Core count does not have optimal performance

I need to get the amount of records with a certain filter.
Theoretically this instruction:
_dbContext.People.Count (w => w.Type == 1);
It should generate SQL like:
Select count (*)
from People
Where Type = 1
However, the generated SQL is:
Select Id, Name, Type, DateCreated, DateLastUpdate, Address
from People
Where Type = 1
The query being generated takes much longer to run in a database with many records.
I need to generate the first query.
If I just do this:
_dbContext.People.Count ();
Entity Framework generates the following query:
Select count (*)
from People
.. which runs very fast.
How to generate this second query passing search criteria to the count?

There is not much to answer here. If your ORM tool does not produce the expected SQL query from a simple LINQ query, there is no way you can let it do that by rewriting the query (and you shouldn't be doing that at the first place).
EF Core has a concept of mixed client/database evaluation in LINQ queries which allows them to release EF Core versions with incomplete/very inefficient query processing like in your case.
Excerpt from Features not in EF Core (note the word not) and Roadmap:
Improved translation to enable more queries to successfully execute, with more logic being evaluated in the database (rather than in-memory).
Shortly, they are planning to improve the query processing, but we don't know when will that happen and what level of degree (remember the mixed mode allows them to consider query "working").
So what are the options?
First, stay away from EF Core until it becomes really useful. Go back to EF6, it's has no such issues.
If you can't use EF6, then stay updated with the latest EF Core version.
For instance, in both v1.0.1 and v1.1.0 you query generates the intended SQL (tested), so you can simply upgrade and the concrete issue will be gone.
But note that along with improvements the new releases introduce bugs/regressions (as you can see here EFCore returning too many columns for a simple LEFT OUTER join for instance), so do that on your own risk (and consider the first option again, i.e. Which One Is Right for You :)

Try to use this lambda expression for execute query faster.
_dbContext.People.select(x=> x.id).Count();

Try this
(from x in _dbContext.People where x.Type == 1 select x).Count();
or you could do the async version of it like:
await (from x in _dbContext.People where x.Type == 1 select x).CountAsync();
and if those don't work out for you, then you could at least make the query more efficient by doing:
(from x in _dbContext.People where x.Type == 1 select x.Id).Count();
or
await (from x in _dbContext.People where x.Type == 1 select x.Id).CountAsync();

If you want to optimize performance and the current EF provider is not not (yet) capable of producing the desired query, you can always rely on raw SQL.
Obviously, this is a trade-off as you are using EF to avoid writing SQL directly, but using raw SQL can be useful if the query you want to perform can't be expressed using LINQ, or if using a LINQ query is resulting in inefficient SQL being sent to the database.
A sample raw SQL query would look like this:
var results = _context.People.FromSql("SELECT Id, Name, Type, " +
"FROM People " +
"WHERE Type = #p0",
1);
As far as I know, raw SQL queries passed to the FromSql extension method currently require that you return a model type, i.e. returning a scalar result may not yet be supported.
You can however always go back to plain ADO.NET queries:
using (var connection = _context.Database.GetDbConnection())
{
connection.Open();
using (var command = connection.CreateCommand())
{
command.CommandText = "SELECT COUNT(*) FROM People WHERE Type = 1";
var result = command.ExecuteScalar().ToString();
}
}

It seems that there has been some problem with one of the early releases of Entity Framework Core. Unfortunately you have not specified exact version so I am not able to dig into EF source code to tell what exactly has gone wrong.
To test this scenario, I have installed the latest EF Core package and managed to get correct result.
Here is my test program:
And here is SQL what gets generated captured by SQL Server Profiler:
As you can see it matches all the expectations.
Here is the excerpt from packages.config file:
...
<package id="Microsoft.EntityFrameworkCore" version="1.1.0" targetFramework="net452" />
...
So, in your situation the only solution is to update to the latest package which is 1.1.0 at the time of writing this.

Does this get what you want:
_dbContext.People.Where(w => w.Type == 1).Count();

I am using EFCore 1.1 here.
This can occur if EFCore cannot translate the entire Where clause to SQL. This can be something as simple as DateTime.Now that might not even think about.
The following statement results in a SQL query that will surprisingly run a SELECT * and then C# .Count() once it has loaded the entire table!
int sentCount = ctx.ScheduledEmail.Where(x => x.template == template &&
x.SendConfirmedDate > DateTime.Now.AddDays(-7)).Count();
But this query will run an SQL SELECT COUNT(*) as you would expect / hope for:
DateTime earliestDate = DateTime.Now.AddDays(-7);
int sentCount = ctx.ScheduledEmail.Where(x => x.template == template
&& x.SendConfirmedDate > earliestDate).Count();
Crazy but true. Fortunately this also works:
DateTime now = DateTime.Now;
int sentCount = ctx.ScheduledEmail.Where(x => x.template == template &&
x.SendConfirmedDate > now.AddDays(-7)).Count();

sorry for the bump, but...
probably the reason the query with the where clause is slow is because you didnt provide your database a fast way to execute it.
in case of the select count(*) from People query we dont need to know the actual data for each field and we can just use a small index that doesnt have all these fields in them so we havent got to spend our slow I/O on. The database software would be clever enough to see that the primary key index requires the least I/O to do the count on. The pk id's require less space than the full row so you get more back to count per I/O block so you can complete faster.
Now in the case of the query with the Type it needs to read the Type to determine it's value. You should create an index on Type if you want your query to be fast or else it will have to do a very slow full table scan, reading all rows. It helps when your values are more discriminating. A column Gender (usually) only has two values and isnt very discriminating, a primary key column where every value is unique is highly dscriminating. Higher discriminating values will result in a shorter index range scan and a faster result to the count.

What I used to count rows using a search query was
_dbContext.People.Where(w => w.Type == 1).Count();
This can also be achieved by
List<People> people = new List<People>();
people = _dbContext.People.Where(w => w.Type == 1);
int count = people.Count();
This way you will get the people list too if you need it further.

How to execute query that Entity Framework doesn't support and get returned data

I am using VS2010, C#, .Net Framework 4, (DB provider Microsoft SQL Server 2008 R2), to write an add-in for Excel. The main idea is to let a user execute a pre-defined query against the DB, so that the data is pulled into Excel.
Say I have hard-coded queries stored in a table [Queries] in the DB. So I can use Entity Framework to get to the table Queries. These queries listed in the table, could return anything from a single value to multiple records.
I am fairly ignorant regarding Entity Framework. Now, I've read that one can execute T-SQL directly against the database here. This has been useful, but I struggle with getting the results back.
using (SYMNHM_DEVEntities dataContext = new SYMNHM_DEVEntities())
{
var query = "Select [Query Name] from [SYM XLS Queries] where [Query ID] = 2";
str = dataContext.ExecuteStoreCommand(query) + "";
}
This gives a result of -1, which is then roughly made into a string. Okay, I know this is not good coding, but just see it as an example. How do I get the actual [Query Name] returned?
In this case, it's just a single value, but if the query would return more values (and I might not even know whether it's strings or not) how do I create an appropriate return type?

Here is an answer for the simple example regarding a string:
using (MyEntities dataContext = new MyEntities())
{
var query = (from q in dataContext.Queries
where q.Query_Name == queryName
select q.Query).Single();
queryResults = dataContext.ExecuteStoreQuery<string>(query);
List<string> list = new List<string>(queryResults.ToArray<string>());
return list; }
I do not yet have a solution for substituting the string type with something else, although I am looking into the matter of returning a Datatable if the query would have more than one string result.

It's not a good pratice use SQL command directly, why you don't use LINQ ?
In your example you can use:
var result = (from q in dataContext.SYMXLSQueries
where q.ID == 2
select q.QueryName).ToArray();
This return an array of strings (If QueryName is a varchar...)

at what point does linq-to-sql or linq send a request to the database

I want to make my queries better but have been un-able to find a resource out there which lays out when a query is shipped of to the db.
DBContext db = new DBContext();
Order _order = (from o in db
where o.OrderID == "qwerty-asdf-xcvb"
select o).FirstOrDefault();
String _custName = _order.Customer.Name +" "+_order.Customer.Surname;
Does the assignment of _custName need to make any request to the database?

Does the assignment of _custName need to make any request to the database?
It depends on whether or not Order.Customer is lazily loaded. If it is lazily loaded, then yes. Otherwise, no.
By the way, you can investigate this easily if you set the DataContext.Log property:
db.Log = Console.Out;
Then you can watch the SQL statements on the console. By stepping through your program you can see exactly when the SQL statement hits the database.
Check out MSDN on Deferred versus Immediate Loading. In particular, you can turn off lazy loading. Watch out for the SELECT N + 1 problem.

Just FYI, besides lazy loading, there is another reason why database activity may not occur when you expect it to when using LINQ. For example, if I change your example code slightly:
DBContext db = new DBContext();
var orders = (from o in db
where o.OrderID == "qwerty-asdf-xcvb"
select o);
var order = orders.FirstOrDefault();
String _custName = _order.Customer.Name +" "+_order.Customer.Surname;
Someone unfamiliar with how LINQ works may expect that all orders are retrieved from the database when the second line of code is executed. In fact, LINQ delays querying the database until the last possible moment, which in this case is the call to FirstOrDefault. Of course, at this point LINQ knows to only retrieve at most one record.