Develop Reference

Entity Framework gets progressively slow with extra join added even though SQL generated is fast

We have 18 table join which is typical for ERP systems. The join is done via LINQ over Entity Framework.
The join gets progressively slower as more joins are added. The return result set is small(15 records). The LINQ generated query is captured via SQL Profiler and when we run this via Microsoft Management Console it is very fast : 10ms. When we run it via our C# LINQ-over-EntityFramework it takes 4 seconds.
What i guess is happening:
The time it takes to compile expression tree into SQL is 2 seconds out of total 4 seconds, and another 2 seconds i guess is spent internally to convert SQL result set into actually C# classes. Also it is not connected to initialization of entity framework because we run some queries before and repetitive calls to this join produce same 4 seconds.
Is there a way to speed this up. Otherwise we are considering abandoning Entity Framework for being absolutely inefficient...

In case it helps, I had a nasty performance issue, whereby a simple query that took 1-2 seconds in raw SQL took about 11 seconds via EF.
I went from using...
List<GeographicalLocation> geographicalLocations = new SalesTrackerCRMEntities()
.CreateObjectSet<GeographicalLocation>()
.Where(g => g.Active)
.ToList();
which took about 11 seconds via EF, to using...
var geographicalLocations = getContext().CreateObjectSet<GeographicalLocation>()
.AsNoTracking()
.Where(g => g.Active).ToList();
which took less than 200 milliseconds.
The disadvantage to this is that it won't load related entities, so you have to load them manually afterwards, but it gives such an immense performance boost that it was well worth it (in this case at least).
You would have to assess each case individually to see if the extra speed is worth the extra code.

You correctly identified bottlenecks.
If you have quite complex queries, I would suggest you to use compiled queries to overcome expression tree to sql query conversion.
You can refer Compiled Queries in EF from here.
Fo second part if EF is using two much time materialize your object graph then I would suggest to use some other means to retrieve data apart from EF.
One option can be Dapper.NET, You can have your concise sql query and you can directly retrieve its result in concrete model objects using Dapper (or any other tiny ORM)

I suspect your query takes so long to generate becuase you are treating Entity Framework like it is a SQL Query, which is not correct. You have many joins and akward calls in your linq syntax. Generally, your syntax should be similar to the following fictitious modeling query:
var result = (from appointment in appointments
from operation in appointment.Operations
where appointment.Id == 12
select new Model {
Id = appointment.Id,
Name = appointment.Name,
// etc, etc
}).ToList();
There is no use of joins above, the navigation property between Appointment and Operations takes care of the neccessary plumbing. Remember, this is an ORM, there is no concept of a join, only a concept of relationships.
The call to Distinct at the end, also indicates the structure of the db schema may be problematic if it returns too many duplicate results.
If after refactoring the entity model and correctly constructing the query still leaves with underperformance, it is advisable to use a stored procedure and map the result with EF's built in methods for doing so.

It is hard to tell what is going wrong here without seeing how you are using linq, but I suspect this will fix your problem:
var myResult = dataContext.table.Where(x == "Your joins and otherstuff").ToList();
//after converting it to a list use it how you need, but not before.
If this does not help please post your code.

The problem is that you are probably passing it to a data source that is running all sorts of additional queries based on you open result set.
Try this instead:
IEnumerable<SigmaTEK.Web.Models.SchedulerGridModel> tasks = (from appointment in _appointmentRep.Value.Get().Where(a => (a.Start < DbContext.MaxTime && DbContext.MinTime < a.Expiration))
join timeApplink in _timelineAppointmentLink.Value.Get().Where(a => a.AppointmentId != Guid.Empty)
on appointment.Id equals timeApplink.AppointmentId
join timeline in timelineRep.Value.Get().Where(i => timelines.Contains(i.Id))
on timeApplink.TimelineId equals timeline.Id
join repeater in _appointmentRepeaterRep.Value.Get().Where(repeater => (repeater.Start < DbContext.MaxTime && DbContext.MinTime < repeater.Expiration))
on appointment.Id equals repeater.Appointment
into repeaters
from repeater in repeaters.DefaultIfEmpty()
join aInstance in _appointmentInstanceRep.Value.Get()
on appointment.Id equals aInstance.Appointment
into instances
from instance in instances.DefaultIfEmpty()
join opRes in opResRep.Get()
on instance.ResourceOwner equals opRes.Id
into opResources
from op in opResources.DefaultIfEmpty()
join itemResource in _opDocItemResourcelinkRep.Value.Get()
on op.Id equals itemResource.Resource
into itemsResources
from itemresource in itemsResources.DefaultIfEmpty()
join opDocItem in opDocItemRep.Get()
on itemresource.OpDocItem equals opDocItem.Id
into opDocItems
from opdocitem in opDocItems.DefaultIfEmpty()
join opDocSection in opDocOpSecRep.Get()
on opdocitem.SectionId equals opDocSection.Id
into sections
from section in sections.DefaultIfEmpty()
join opDoc in opDocRep.Get()
on section.InternalOperationalDocument equals opDoc.Id
into opdocs
from opdocitem2 in opDocItems.DefaultIfEmpty()
join opDocItemLink in opDocItemStrRep.Get()
on opdocitem2.Id equals opDocItemLink.Parent
into opDocItemLinks
from link in opDocItemLinks.DefaultIfEmpty()
join finItem in finItemsRep.Get()
on link.Child equals finItem.Id
into temp1
from rd1 in temp1.DefaultIfEmpty()
join sec in finSectionRep.Get()
on rd1.SectionId equals sec.Id
into opdocsections
from finopdocsec in opdocsections.DefaultIfEmpty()
join finopdoc in opDocRep.Get().Where(i => i.DocumentType == "Quote")
on finopdocsec.InternalOperationalDocument equals finopdoc.Id
into finOpdocs
from finOpDoc in finOpdocs.DefaultIfEmpty()
join entry in entryRep.Get()
on rd1.Transaction equals entry.Transaction
into entries
from entry2 in entries.DefaultIfEmpty()
join resproduct in resprosductRep.Get()
on entry2.Id equals resproduct.Entry
into resproductlinks
from resprlink in resproductlinks.DefaultIfEmpty()
join res in resRep.Get()
on resprlink.Resource equals res.Id
into rootResource
from finopdoc in finOpdocs.DefaultIfEmpty()
join rel in orgDocIndRep.Get().Where(i => (i.Relationship == "OrderedBy"))
on finopdoc.Id equals rel.OperationalDocument
into orgDocIndLinks
from orgopdoclink in orgDocIndLinks.DefaultIfEmpty()
join org in orgRep.Get()
on orgopdoclink.Organization equals org.Id
into toorgs
from opdoc in opdocs.DefaultIfEmpty()
from rootresource in rootResource.DefaultIfEmpty()
from toorg in toorgs.DefaultIfEmpty()
select new SigmaTEK.Web.Models.SchedulerGridModel()
{
Id = appointment.Id,
Description = appointment.Description,
End = appointment.Expiration,
Start = appointment.Start,
OperationDisplayId = op.DisplayId,
OperationName = op.Name,
AppContextId = _appContext.Id,
TimelineId = timeline.Id,
AssemblyDisplayId = rootresource.DisplayId,
//Duration = SigmaTEK.Models.App.Utils.StringHelpers.TimeSpanToString((appointment.Expiration - appointment.Start)),
WorkOrder = opdoc.DisplayId,
Organization = toorg.Name
}).Distinct().ToList();
//In your UI
MyGrid.DataSource = tasks;
MyGrid.DataBind();
//Do not use an ObjectDataSource! It makes too many extra calls

group by using entity framework

Hi all I am having a sql query which I would like to execute the same using entity framework and bind it to gridview can some one help me. This is my query
SELECT Technology,[Description], Technologies.TechID, COUNT(*) AS 'num_employees'
FROM Technologies
LEFT JOIN Questions
ON Questions.TechID = Technologies.TechID
GROUP BY Technologies.TechID, Technology,[Description]
How can I convert the same to get the results using EF

Personally, I would make this a stored procedure instead, and import it into EF. Then you would just call it in your code and databind the result to your grid.
This way the joining, grouping, etc all happens server side instead of client side. Otherwise, you'd be sending a lot more information than necessary to the client. Plus, the server can theoretically do these operations faster than a client's.

Try this, i didn't have the time to test it both for logic and syntax. You need something similar.
var q = from a in context.Technologies
join b in context.Questions
on a.TechID equal b.TechID into j1
from j2 in j1.DefaultIfEmpty()
group j2 by new { a.TechID, b.Technology, b.Description into grouped
select new { Tecnology = grouped.Technology, Description = grouped.Description, Count = grouped.Count() }

Exception in a CRM LINQ query with joins. Attribute in second table doesn't exist

First of all I'm sorry because this is the second time that I write this question but before was bad explained and now is close.
I'm doing a linq query for a search page for a CRM data base, and wrtiting a normal query like below is not working, I'm getting the exception:
[System.ServiceModel.FaultException<Microsoft.Xrm.Sdk.OrganizationServiceFault>] = {"'Contact' entity doesn't contain attribute with Name = 'title'."}
For a join query, that in the clause Where was something like r.Name == "Me" && j.LastName == "He" I had to did the query with two Where clauses, because I was getting the same exception as above, saying that table 'r' doesn't have 'LastName' attributte.
var cms = from i in aux_pr
join cal in Contact on i.aux_CallerRequestorID.Id equals cal.ContactId.Value
join sub in Subject on i.aux_ClassificationID.Id equals sub.SubjectId
where cal.FullName.Contains(searchTerm) ||
sub.Title.Contains(searchTerm)
In this case, how can I do this query. Thanks in advance!

I want to comment what have I learned and the solution that I have found to my problem hoping could help some one. There are some limitations in CRM LINQ, as explained here
The first that I found, having an entity reference like this:
CrmEntityReference Caller
{
Guid ID;
string name;
}
I can select Caller.name but I CAN'T have Caller.name in the where clause. Solution for this -> Join the table
The second limitation, is when we have joins in the query, we can have different tables in the where if they are an AND predicate, we have to write two clauses where like this:
where cal.FullName.Contains(searchTerm)
where sub.Title.Contains(searchTerm)
But the problem comes when instead of an AND we need use an OR predicate, the only solution we have is do two queries and after do an Union of these queries.
I have four queries for a call that could be done just with one, now in developing stage performance is good due to the amount of records, but we'll see in testing stage how this work.

try to create two different filters..
var cms = from i in aux_pr
join cal in Contact on i.aux_CallerRequestorID.Id equals cal.ContactId.Value
join sub in Subject on i.aux_ClassificationID.Id equals sub.SubjectId
where cal.FullName.Contains(searchTerm) ||
where sub.Title.Contains(searchTerm)

DataTable reader loading is very slow

I need to fetch some data based on a keyword, the query is tested to 100% accurate, but the problem is the the loading of the reader is pretty slow. I have tried replacing this query with one that does not contain inner joins at all and the loading was pretty fast. So I wonder, since I am only selecting one column as a result, why does DataTable.Load() take so much time? Is it the SQLite's ExecuteReader that loads the whole results and not just the one column?
Before using the DataTable, the average time of executing each reader.Read() was 7 seconds.
This is my code:
_database.Connect();
var selectCommand = new SQLiteCommand(
#"SELECT A.ID AS MY_ID FROM MD
INNER JOIN TMD ON MD.ID = TMD.ID_MD
INNER JOIN TR ON TR.ID = TMD.ID_TR
INNER JOIN P ON P.ID = TR.ID_P
INNER JOIN DP ON DP.ID_P = P.ID
INNER JOIN CD ON CD.ID = DP.ID_CD
WHERE CD.DESC = #desc"
);
selectCommand.Parameters.AddWithValue("#desc", value);
using (DbDataReader reader = _database.ExecuteQuery(selectCommand))
{
DataTable data = new DataTable("MyData");
data.Load(reader);
}
_database.Disconnect();

I think this happens due a nature of SQLite and great number of joins.
Try to refactor database scheme, like denormalize data for faster access.

The SQLite Query Planner offers some hints about query optimization for SQLite.
Some items that may apply to your question:
1.) Due to the implementation in SQLite you might try to re-order the multiple joins:
The current implementation of SQLite uses only loop joins. That is to
say, joins are implemented as nested loops. The default order of the
nested loops in a join is for the left-most table in the FROM clause to
form the outer loop and the right-most table to form the inner loop.
So, depending on how the JOINs are constructed there might be a difference in performance.
SQLite tries to optimize this automatically, but as far as I understood the documentation there is no guarantee for success (highlights by me):
However, SQLite will nest the loops in a different order if doing so
will help it to select better indices.
[...]
Join reordering is automatic and usually works well enough that programmers
do not have to think about it, especially if ANALYZE has been used to gather
statistics about the available indices. But occasionally some hints from the
programmer are needed.
2.) Also, please note that INNER JOINS are internally converted into WHERE clauses, so any of the performance tips in the WHERE section of the document might apply, too:
The ON and USING clauses of an inner join are converted into
additional terms of the WHERE clause prior to WHERE clause analysis
described above in paragraph 1.0. Thus with SQLite, there is no
computational advantage to use the newer SQL92 join syntax over the
older SQL89 comma-join syntax. They both end up accomplishing exactly
the same thing on inner joins.
3.) You might consider to select more columns in your statement, if there are any indexes on them:
It is not necessary for every column of an index to appear in a WHERE
clause term in order for that index to be used. But there can not be
gaps in the columns of the index that are used.

Linq query nhibernate; not supported exception

I'm fairly new to nHibernate having come from an EF background and I'm struggling with the following query :
_patientSearchResultModel = (from patient in _patientRepository.Query(patientSearch.BuildPatientSpecification())
join admission in _admissionRepository.Query(patientSearch.BuildAdmissionSpecification())
on patient.Id equals admission.Patient.Id
orderby admission.AdmissionDate
select new PatientSearchResultModel(patient.Id,
admission.Id,
false,
_phaseTypeMapper.GetPhaseTypeModel(admission.PhaseType),
patient.Last, patient.First,
admission.InPatientLocation,
admission.AdmissionDate,
admission.DischargeDate,
admission.RRI,
null,
admission.CompletionStatus,
admission.FollowupStatus)).ToList();
The intent of this query is to allow users to filter the two queries on parameters built up using the two Build???Specification functions and return the resultset. There could be many admission records and I would only like one PatientSearchResultModel per patient object, with the admission object being the newest one by Admission Date.
These objects are coming from nHibernate and it keeps return a Not Supported exception. There is also an association between Patient and Admissions thus : Patient.Admissions but i couldn't figure out how to then add the query filters return from the function Build???Specifications.
I'd be really grateful if someone could point me in the right direction; am I up against the Linq provider implementation here in nHibernate and need to move to Criteria or is it my Linq query ?
If anyone has any links or suggestions for good books or other learning materials in this area that would also be really helpful too.

I see several potential problems:
If you're using NHibernate 2.x + Linq2NHibernate explicit joins like that are not supported; in other versions they're just considered a smell.
I dont think NHibernate supports calling parameterized constructors in select clauses
I'm very sure NHibernate does not support calling instance methods in the select lambda
I'd suggest using the lambda syntax and SelectMany to alleviate potential join issues. Points #2 & #3 can be solved by projecting into an anonymous type, calling AsEnumerable then projecting into your model type.
Overall I'd suggest restructuring your code like:
var patientSpec = patientSearch.BuildPatientSpecification();
var admissionSpec = patientSearch.BuildAdmissionSpecification();
_patientSearchResultModel = _patientRepository.Where(patientSpec)
.SelectMany(p=>p.Admissions).Where(admissionSpec)
.Select(a=> new {
PatientId = a.Patient.Id,
AdminssionId = a.Id,
a.PhaseType,
a.Patient.Last,
a.Patient.First,
a.InPatientLocation,
a.AdmissionDate,
a.DischargeDate,
a.RRI,
a.CompletionStatus,
a.FollowupStatus
}).AsEnumerable()
.Select(x=> new PatientSearchResultModel(x.PatientId, x.AdmissionId ...))
.ToList();

Divide your query into parts and check which part runs and which doesn't.
My take on this is that select new ... is not supported in Linq to nHibernate.
I would recomend using something else, because it is simply too imature and feature-less to use seriously.

As with most popular LINQ-to-Database query providers, NHibernate will try to translate the whole query into a SQL statement to run against the database. This requires that all elements of your query are possible to express in the SQL flavour you're using.
In your query, the select new statement cannot be expressed in SQL, because you're making a call to the constructor of your PatientSearchResultModel class and are making a call to a GetPhaseTypeModel method.
You should restructure your query to express what you want to execute on the SQL database, then call AsEnumerable() to force the remainder of the query to be evaluated in-memory. After that call, you can call the constructor of your class and any .NET methods, and they will be executed as native code.

This query is too complex to describe it using Linq. It would give wrong result finally (if Patient has more than one admission records, result would have duplicate entries).
I see two steps for solution:
1) At development stage, use in-memory query. So, take Patients using ToList() first (query db at this moment). Some predicates (Patient filter like MRN, First, Last) could be used at this stage.
And then do search in-memory. Not performance, but working solution. Mark it for refactor to optimize later.
2) Finally, use NHibernate IQuery (ISQLQuery) and build sql query manually to make sure it would work as expected and work fast enough on SQL Server side. This is just read-only query and do not require Nhibernate query engine (Linq to Nhibernate) at all.