I'm having trouble understanding why the SQL output has a sub-query for a simple query I wrote in LINQ. This is my code:
var list = db.User.Where(u => u.Name == somename).OrderBy(u => u.IdUser).ToList();
where somename is a parameter I'm passing at execution time.
The output SQL is:
SELECT
Project1.IdUser,
Project1.Name
FROM (SELECT
Extent1.IdUser,
Extent1.Name
FROM user AS Extent1
WHERE Extent1.Name = 'John' /* #p__linq__0 */) AS Project1
ORDER BY
Project1.IdUser ASC
Should the output really have a sub-query for something that simple?
I also tried
var list = db.User.Where(u => u.Name.Equals(somename)).OrderBy(u => u.IdUser).ToList();
which generates the same output as above.
If I hard code the parameter, like:
var list = db.User.Where(u => u.Name == "John").OrderBy(u => u.IdUser).ToList();
It works as expected, generating only
SELECT
Extent1.IdUser,
Extent1.Name
FROM user AS Extent1
WHERE 'John' /* #gp1 */ = Extent1.Name
ORDER BY
Extent1.IdUser ASC
A few things I'm using:
EntityFramework 5, .NET 4.5
SQL Server 2012
Glimpse (which uses MiniProfiler) to see the SQL generated
I'm not a LINQ expert, so what am I missing here?
As other pointed, the query results in same execution plan as yours. Entity Framework (and LINQ to Entites) is here to help you avoid writing SQL and bothering about SQL (to some extent). Under normal circumstances you don't care about SQL being generated nor you "debug" it. You just care whether the LINQ query is correct. Entity Framework (should) translates it into correct (sometimes even expected) SQL (and again, execution plan matters).
I'm not saying that you shouldn't look at SQL for performance reasons (or better to say execution plan of that query). But that should be done after you identified performance problems. And you should try to write queries simple first, that's the way to success. Of course if you know SQL you know this world of sets is different from world of object - you can write easily fairly average query in LINQ (thanks to objects world), but this will end up as nasty SQL (sets world) because of "mismatch" between worlds.
Related
This may be a silly question, but I have fluent linq query that is pulling data off a DB, but, of course, it processes the data in memory, not on the DB.
Is there a means of taking a query (Say table.Where(t => t. id > 22)) and having it run as a db query (i.e. SELECT * FROM TABLE WHERE ID > 22).
I want to use the fluent style because I am building up the query from various places. I know I can make this specific query into query syntax, but my actual queries are more complex.
I have tried standard searches, but I suspect it is not going to be possible, as nothing comes up.
Yes using EF Core.
The code is not necessarily clear:
var getTableData = table.Where(base.whereSelection);
whereSelection.ForEach(w => getTableData = getTableData.Where(w));
At the moment, all of the Where Selections are Funcs - they could be converted to something else (like Expressions) if I knew that would make them run on the DB. The rest of the code is also building up this set of details programatically:
var selectedData = getTableData.Select(Convert);
var sortedTableData = GetSort(selectedData, Ordering);
var sorted = PickDataFrom(sortedTableData);
return CheckAndTake(sorted, pageSize);
Everything seems to work. Just in the wrong place.
I am trying to convert my sql queries being run with EF Core's .FromSqlRaw() method into queries to be run with .FromSqlInterpolated() so they are less vulnerable to SQL injection attacks. I've gotten almost everything converted and working fine, but the one thing that is stumping me is how to filter by a list of integers in an or configuration on a single field.
The database is postgres, and the quotes are because I used code first EF Core which means that all of the tables are capitalized like the class. ProjectTypeId is an integer column in my table, and projectTypes is a List<int> type variable.
The code in my where clause I'm trying to replace is:
WHERE ""PartGroups"".""ProjectTypeId"" IN({string.Join(",", projectTypes)})
The closest I've been able to get to getting it working is with this:
""PartGroups"".""ProjectTypeId""::text IN({string.Join(",", projectType)})
or
""PartGroups"".""ProjectTypeId""::text LIKE ANY(ARRAY[{string.Join(",", projectTypes)}])
These will work when there is only one value in projectTypes, but any more than that and it fails. I don't know how to view the resulting query + parameter set, just the query, so I'm not sure what's it's doing to the parameter, so I've been struggling to figure out something that works. Also, the query has ~80 parameters total, so manually setting each one with a raw query isn't really feasible.
One way you could approach it, perhaps, is to leverage EF's ability to compose over the top of a raw
context.SomeTable
.FromSqlInterpolated($"SeLeCt t.* FrOm ...")
.Where(st => idList.Contains(st.id))
EF will put your SQL in as a sub query and write the IN for you in the outer. The DB query optimizer will (probably) then push the IN into the sub query if it can..
SELECT ...
FROM
(
SeLeCt t.* FrOm ...
) x
WHERE x.ID IN (.., ..)
If you want to have a look at the SQL EF made and you're on EF5+ the easiest thing to do is do everything apart from ToList/ToArray, and capture the queryable:
var q = context.Thing.Where(...)
Then pause in the debugger and look at the DebugView property of the q. You'll get the full SQL text and you can slot it into SSMS and inspect the execution plan, check it runs the same as a single level (non hierarchical) query..
The other thing you can do, is create your query in a FormattableString yourself. So long as FromSqlInterpolated receives a FormattableString it will pull it apart and parameterize
var args = new object[] { your, single, arguments, here}.Concat(yourListOfThingsHere.Cast<object>()).ToArray();
var fs = FormattableStringFactory.Create(#"SELECT
some,
columns/{0},
here
FROM
table t
WHERE
someColumn = {1} AND
otherColumn BETWEEN {2} and {3} AND
columnToBeINned IN({" + string.Join("},{", Enumerable.Range(4, yourListOfThingsHere.Count)) + #"})
GROUP BY some, columns/{0}", args);
var qq = context.Table.FromSqlInterpolated(fs).Where(m => ...);
It's, of course, possible to write a helper to do this for you...
So I found a way to actually see how EF Core is interpreting the parameters for the interpolated query and was able to play around with it to get what I wanted out of it. So just in case someone wants to or needs to stick to a pure interpolated query, you can use the following pattern for your needs:
WHERE MyTable.MyIntColumn::text LIKE ANY(Array[{string.Join(",", myIntList).Split(",", StringSplitOptions.None)}])
This also works with text values (just no need to cast to text) and you can use wildcards like this:
WHERE MyTable.MyStringColumn ILIKE ANY(ARRAY[{("%" + string.Join("%,%", myStringList) + "%").Split(",", StringSplitOptions.None)}])
I need to get the amount of records with a certain filter.
Theoretically this instruction:
_dbContext.People.Count (w => w.Type == 1);
It should generate SQL like:
Select count (*)
from People
Where Type = 1
However, the generated SQL is:
Select Id, Name, Type, DateCreated, DateLastUpdate, Address
from People
Where Type = 1
The query being generated takes much longer to run in a database with many records.
I need to generate the first query.
If I just do this:
_dbContext.People.Count ();
Entity Framework generates the following query:
Select count (*)
from People
.. which runs very fast.
How to generate this second query passing search criteria to the count?
There is not much to answer here. If your ORM tool does not produce the expected SQL query from a simple LINQ query, there is no way you can let it do that by rewriting the query (and you shouldn't be doing that at the first place).
EF Core has a concept of mixed client/database evaluation in LINQ queries which allows them to release EF Core versions with incomplete/very inefficient query processing like in your case.
Excerpt from Features not in EF Core (note the word not) and Roadmap:
Improved translation to enable more queries to successfully execute, with more logic being evaluated in the database (rather than in-memory).
Shortly, they are planning to improve the query processing, but we don't know when will that happen and what level of degree (remember the mixed mode allows them to consider query "working").
So what are the options?
First, stay away from EF Core until it becomes really useful. Go back to EF6, it's has no such issues.
If you can't use EF6, then stay updated with the latest EF Core version.
For instance, in both v1.0.1 and v1.1.0 you query generates the intended SQL (tested), so you can simply upgrade and the concrete issue will be gone.
But note that along with improvements the new releases introduce bugs/regressions (as you can see here EFCore returning too many columns for a simple LEFT OUTER join for instance), so do that on your own risk (and consider the first option again, i.e. Which One Is Right for You :)
Try to use this lambda expression for execute query faster.
_dbContext.People.select(x=> x.id).Count();
Try this
(from x in _dbContext.People where x.Type == 1 select x).Count();
or you could do the async version of it like:
await (from x in _dbContext.People where x.Type == 1 select x).CountAsync();
and if those don't work out for you, then you could at least make the query more efficient by doing:
(from x in _dbContext.People where x.Type == 1 select x.Id).Count();
or
await (from x in _dbContext.People where x.Type == 1 select x.Id).CountAsync();
If you want to optimize performance and the current EF provider is not not (yet) capable of producing the desired query, you can always rely on raw SQL.
Obviously, this is a trade-off as you are using EF to avoid writing SQL directly, but using raw SQL can be useful if the query you want to perform can't be expressed using LINQ, or if using a LINQ query is resulting in inefficient SQL being sent to the database.
A sample raw SQL query would look like this:
var results = _context.People.FromSql("SELECT Id, Name, Type, " +
"FROM People " +
"WHERE Type = #p0",
1);
As far as I know, raw SQL queries passed to the FromSql extension method currently require that you return a model type, i.e. returning a scalar result may not yet be supported.
You can however always go back to plain ADO.NET queries:
using (var connection = _context.Database.GetDbConnection())
{
connection.Open();
using (var command = connection.CreateCommand())
{
command.CommandText = "SELECT COUNT(*) FROM People WHERE Type = 1";
var result = command.ExecuteScalar().ToString();
}
}
It seems that there has been some problem with one of the early releases of Entity Framework Core. Unfortunately you have not specified exact version so I am not able to dig into EF source code to tell what exactly has gone wrong.
To test this scenario, I have installed the latest EF Core package and managed to get correct result.
Here is my test program:
And here is SQL what gets generated captured by SQL Server Profiler:
As you can see it matches all the expectations.
Here is the excerpt from packages.config file:
...
<package id="Microsoft.EntityFrameworkCore" version="1.1.0" targetFramework="net452" />
...
So, in your situation the only solution is to update to the latest package which is 1.1.0 at the time of writing this.
Does this get what you want:
_dbContext.People.Where(w => w.Type == 1).Count();
I am using EFCore 1.1 here.
This can occur if EFCore cannot translate the entire Where clause to SQL. This can be something as simple as DateTime.Now that might not even think about.
The following statement results in a SQL query that will surprisingly run a SELECT * and then C# .Count() once it has loaded the entire table!
int sentCount = ctx.ScheduledEmail.Where(x => x.template == template &&
x.SendConfirmedDate > DateTime.Now.AddDays(-7)).Count();
But this query will run an SQL SELECT COUNT(*) as you would expect / hope for:
DateTime earliestDate = DateTime.Now.AddDays(-7);
int sentCount = ctx.ScheduledEmail.Where(x => x.template == template
&& x.SendConfirmedDate > earliestDate).Count();
Crazy but true. Fortunately this also works:
DateTime now = DateTime.Now;
int sentCount = ctx.ScheduledEmail.Where(x => x.template == template &&
x.SendConfirmedDate > now.AddDays(-7)).Count();
sorry for the bump, but...
probably the reason the query with the where clause is slow is because you didnt provide your database a fast way to execute it.
in case of the select count(*) from People query we dont need to know the actual data for each field and we can just use a small index that doesnt have all these fields in them so we havent got to spend our slow I/O on. The database software would be clever enough to see that the primary key index requires the least I/O to do the count on. The pk id's require less space than the full row so you get more back to count per I/O block so you can complete faster.
Now in the case of the query with the Type it needs to read the Type to determine it's value. You should create an index on Type if you want your query to be fast or else it will have to do a very slow full table scan, reading all rows. It helps when your values are more discriminating. A column Gender (usually) only has two values and isnt very discriminating, a primary key column where every value is unique is highly dscriminating. Higher discriminating values will result in a shorter index range scan and a faster result to the count.
What I used to count rows using a search query was
_dbContext.People.Where(w => w.Type == 1).Count();
This can also be achieved by
List<People> people = new List<People>();
people = _dbContext.People.Where(w => w.Type == 1);
int count = people.Count();
This way you will get the people list too if you need it further.
I have an app using Entity Framework. I want to add a tree view listing products, grouped by their categories. I have an old SQL query that will grab all of the products and categories and arrange them into parent nodes and children. I am trying to translate it into LINQ that uses the EF. But the SQL has a WITH sub-query that I am not familiar with using. I have tried using Linqer and LinqPad to sort it out, but they choke on the WITH clause and I am not sure how to fix it. Is this sort of thing possible in LINQ?
Here is the query:
declare #id int
set #id=0
WITH ChildIDs(id,parentid,type,ChildLevel) AS
(
SELECT id,parentid,type,0 AS ChildLevel
FROM dbo.brooks_product
WHERE id = #id
UNION ALL
SELECT e.id,e.parentid,e.type,ChildLevel + 1
FROM dbo.brooks_product AS e
INNER JOIN ChildIDs AS d
ON e.parentid = d.id
WHERE showitem='yes' AND tribflag=1
)
SELECT ID,parentid,type,ChildLevel
FROM ChildIDs
WHERE type in('product','productchild','productgroup','menu')
ORDER BY ChildLevel, type
OPTION (MAXRECURSION 10);
When I run the query, I get data that looks like this (a few thousand rows, truncated here):
ID.....parentid.....type.....ChildLevel
35429..0............menu.....1
49205..0............menu.....1
49206..49205........menu.....2
169999.49206........product..3
160531.169999.......productchild..4
and so on.
The WITH block is a Common Table Expression, and in this case is used to create a recursive query.
This will be VERY difficult in Linq as Linq doesn't play well with recursion. If you need all of the data on one result set that a Stored Procedure would be easier. Another option is to do the recursion in C# (not in Linq but a recursive function) and do multiple round-trips. The performance will not be as good but if you result set is small it may not make much difference (and you will get a better object model).
You may be able to solve this using LINQ to Entities, but it is non-trivial and I suspect it will be very time consuming.
In situations like this, you may prefer to build a SQL View or Table-Valued Function that returns the results for which you're looking. Then import that View or Table-Valued Function into your EF model and you can pull data directly from it using LINQ.
Querying the View in LINQ is no different than querying a table.
To get data from a Table-Valued Function in LINQ, you pass the function's parameters in after the name of the function, like so:
var query = from tvf in _db.MyTableValuedFunction(parameters)
select tvf;
EDIT
As suggested by #thepirat000, Table-Valued Function support is not available in Entity Framework versions prior to version 5. In order to use this functionality, EF must be running with .NET 4.5 or higher.
At the end of the day, I could not get this to work. I ended up writing out a SQL query dynamically and sending that straight to the database. It works fine, and I am not relying on any direct user input so there is no chance of SQL injection. But it seems so old school! For the rest of my program I am using EF and LINQ.
Thanks for the replies!
I'm fairly new to nHibernate having come from an EF background and I'm struggling with the following query :
_patientSearchResultModel = (from patient in _patientRepository.Query(patientSearch.BuildPatientSpecification())
join admission in _admissionRepository.Query(patientSearch.BuildAdmissionSpecification())
on patient.Id equals admission.Patient.Id
orderby admission.AdmissionDate
select new PatientSearchResultModel(patient.Id,
admission.Id,
false,
_phaseTypeMapper.GetPhaseTypeModel(admission.PhaseType),
patient.Last, patient.First,
admission.InPatientLocation,
admission.AdmissionDate,
admission.DischargeDate,
admission.RRI,
null,
admission.CompletionStatus,
admission.FollowupStatus)).ToList();
The intent of this query is to allow users to filter the two queries on parameters built up using the two Build???Specification functions and return the resultset. There could be many admission records and I would only like one PatientSearchResultModel per patient object, with the admission object being the newest one by Admission Date.
These objects are coming from nHibernate and it keeps return a Not Supported exception. There is also an association between Patient and Admissions thus : Patient.Admissions but i couldn't figure out how to then add the query filters return from the function Build???Specifications.
I'd be really grateful if someone could point me in the right direction; am I up against the Linq provider implementation here in nHibernate and need to move to Criteria or is it my Linq query ?
If anyone has any links or suggestions for good books or other learning materials in this area that would also be really helpful too.
I see several potential problems:
If you're using NHibernate 2.x + Linq2NHibernate explicit joins like that are not supported; in other versions they're just considered a smell.
I dont think NHibernate supports calling parameterized constructors in select clauses
I'm very sure NHibernate does not support calling instance methods in the select lambda
I'd suggest using the lambda syntax and SelectMany to alleviate potential join issues. Points #2 & #3 can be solved by projecting into an anonymous type, calling AsEnumerable then projecting into your model type.
Overall I'd suggest restructuring your code like:
var patientSpec = patientSearch.BuildPatientSpecification();
var admissionSpec = patientSearch.BuildAdmissionSpecification();
_patientSearchResultModel = _patientRepository.Where(patientSpec)
.SelectMany(p=>p.Admissions).Where(admissionSpec)
.Select(a=> new {
PatientId = a.Patient.Id,
AdminssionId = a.Id,
a.PhaseType,
a.Patient.Last,
a.Patient.First,
a.InPatientLocation,
a.AdmissionDate,
a.DischargeDate,
a.RRI,
a.CompletionStatus,
a.FollowupStatus
}).AsEnumerable()
.Select(x=> new PatientSearchResultModel(x.PatientId, x.AdmissionId ...))
.ToList();
Divide your query into parts and check which part runs and which doesn't.
My take on this is that select new ... is not supported in Linq to nHibernate.
I would recomend using something else, because it is simply too imature and feature-less to use seriously.
As with most popular LINQ-to-Database query providers, NHibernate will try to translate the whole query into a SQL statement to run against the database. This requires that all elements of your query are possible to express in the SQL flavour you're using.
In your query, the select new statement cannot be expressed in SQL, because you're making a call to the constructor of your PatientSearchResultModel class and are making a call to a GetPhaseTypeModel method.
You should restructure your query to express what you want to execute on the SQL database, then call AsEnumerable() to force the remainder of the query to be evaluated in-memory. After that call, you can call the constructor of your class and any .NET methods, and they will be executed as native code.
This query is too complex to describe it using Linq. It would give wrong result finally (if Patient has more than one admission records, result would have duplicate entries).
I see two steps for solution:
1) At development stage, use in-memory query. So, take Patients using ToList() first (query db at this moment). Some predicates (Patient filter like MRN, First, Last) could be used at this stage.
And then do search in-memory. Not performance, but working solution. Mark it for refactor to optimize later.
2) Finally, use NHibernate IQuery (ISQLQuery) and build sql query manually to make sure it would work as expected and work fast enough on SQL Server side. This is just read-only query and do not require Nhibernate query engine (Linq to Nhibernate) at all.