Entity framework LINQ speed converting SQL data into C# objects - c#

I have a fairly complex query written in LINQ which queries my database. I have made sure that I am only pulling back the relevant columns from each entity, so the query is as slim as it can be, and therefore the object which LINQ populates does not contain any unnecessary columns either.
If I run the LINQ query's resulting SQL against the DB directly, the SQL runs in about 4 seconds. However, doing this from my LINQ query takes about 25 seconds. The majority of this 25 seconds is taken up by the returned DB data being written into objects.
Are there any useful tricks which will help LINQ/Entity Framework out when performing this part of the operation?
Edit:
The long delay occurs when I call the ToList() method in the query below:
var q1 = from u in etc.Users
where (u.flags & (int)UserFlags.Student) == (int)UserFlags.Student
&& (ariStudents.Contains(u.id) || ariStudents.Count == 0)
&& (from ua in etc.UserAssociations where ua.datasetId == datasetId && ua.userId == u.id && ua.type == (int)UserAssocTypes.EnrolActive select ua).Any()
orderby u.id
select new ElightsRaw2
{
Student = new ElrStudent{StudentId = u.id},
StudentData = (
from sd in etc.StudentDatas
where sd.userId == u.id
&& sd.datasetId == datasetId
select new ElrStudentData { attP = sd.attP, attA = sd.attA, attL = sd.attL }
).FirstOrDefault(),
ElightOverall = (
from els in etc.eLightsStudents
where els.datasetId == datasetId
&& els.userId == u.id
select els
).FirstOrDefault(),
Groups = (
// get all courses and groups for user
from c in etc.Cours
join g in etc.Groups on c.id equals g.courseId
join ua in etc.UserAssociations on g.id equals ua.fkId
where ua.datasetId == datasetId
&& ua.userId == u.id
&& ua.type == (int)UserAssocTypes.EnrolActive
&& (ua.flags & (int)UserAssociationFlags.AssociationDeleted) == 0
&& (c.flags & (int)CourseFlags.Deleted) == 0
select new ElrGroupHolder
{
CourseName = c.name,
GroupName = g.name,
Breakdown = new ElrGroupHolderBreakdown
{
UnitsBelowScoreThresh1 = (
from uus in etc.UserUnitScores
join un in etc.Units on uus.unitId equals un.id
join cu in etc.CourseUnits on un.id equals cu.unitId
where uus.userId == u.id
&& (un.flags & (int)UnitFlags.Deleted) == 0
&& (cu.flags & (int)CourseUnitFlags.NotStarted) == 0
&& cu.courseId == c.id
&& (uus.performance > -1 && uus.performance <= amberThresh)
&& !(from exc in etc.UserAssociations where exc.userId == u.id && exc.fkId == un.id && exc.type == (int)UserAssocTypes.ExcludedUnit select exc).Any() // not excluded from unit
select uus
).Count(),
UnitsBelowScoreThresh2 = (
from uus in etc.UserUnitScores
join un in etc.Units on uus.unitId equals un.id
join cu in etc.CourseUnits on un.id equals cu.unitId
where uus.userId == u.id
&& (un.flags & (int)UnitFlags.Deleted) == 0
&& (cu.flags & (int)CourseUnitFlags.NotStarted) == 0
&& cu.courseId == c.id
&& (uus.performance > -1 && uus.performance <= redThresh)
&& !(from exc in etc.UserAssociations where exc.userId == u.id && exc.fkId == un.id && exc.type == (int)UserAssocTypes.ExcludedUnit select exc).Any() // not excluded from unit
select uus
).Count(),
CfcCount = (
from cfc in etc.CFCs
where cfc.datasetId == datasetId
&& cfc.studentId == u.id
&& cfc.dt > dCfcCutoff
&& cfc.type == (int)CfcTypes.Concern
&& (cfc.flags & (int)CfcFlags.Deleted) == 0
&& (
// only include attendance and behaviour CFCs
(cfc.flags & (int)CfcFlags.Attendance) == (int)CfcFlags.Attendance
|| (cfc.flags & (int)CfcFlags.Behaviour) == (int)CfcFlags.Behaviour
)
select cfc
).Count(),
SrsScores = (
from srs in etc.SRScores
join sr in etc.SubReviews on srs.srId equals sr.id
where sr.datasetId == datasetId
&& (sr.flags & (int)SrFlags.Deleted) == 0
&& (srs.flags & (int)SrsFlags.Deleted) == 0
&& sr.dt > dCfcCutoff
&& sr.userId == u.id
&& sr.courseId == c.id
select new ElrSrScore
{
attainment = srs.attainment,
attitude = srs.attitude,
motivation = srs.motivation,
studyskill = srs.studyskill
}
).ToList(),
ElightEnrolment = (
from ele in etc.eLightsEnrolments
where ele.datasetId == datasetId
&& ele.groupId == g.id
&& ele.userId == u.id
select ele
).FirstOrDefault(),
},
CourseAttendance = (
from a in etc.Attendances
where a.studentId == u.id
&& a.courseId == c.id
&& a.weekNumberId == null
&& a.datasetId == datasetId
select a
).FirstOrDefault()
}
).ToList(),
};
return q1.ToList();

Yes! I ran into something similar myself not to long ago. What I came up with is using the AsParallel() function within LINQ ... however this comes with a few caveats.
You will be running things in parallel - obvious but if your trying to do things in a non thread safe manner there could be issues
This will only be useful in certain cases - the way I got the most use out of it was when I was converting a lot of database rows into C# objects and doing it repetitively ... think foreach,for,while loops that don't have any decision logic in them but just a straight database row to C# class kind of conversion
It really only makes sense for "large" amounts of data - You're going to have to play around with this and see what the timing difference is but if you are just converting a handful of rows, the overhead for AsParallel() will actually cost you more time, not save you any.
Here is some more reading on AsParallel(), hope this helps!
https://msdn.microsoft.com/en-us/library/vstudio/dd413237%28v=vs.100%29.aspx - AsParallel Method
https://msdn.microsoft.com/en-us/library/dd997425%28v=vs.110%29.aspx - Intro to parallel queries
http://www.dotnetperls.com/asparallel - Good basic walkthrough of AsParallel

Related

Linq Optimizing query with records not matching from another table

I am having following query which pulls data from a table with records that should not be present in another table.
The query is working perfectly but it is taking too much time & performance is affected tremendously.
What changes can i make to this query to get better performance or should i be doing this in another way?
var data = (from A in ctx.tblMachine
where
A.CompanyId == companyId &&
A.InOutDate >= tempDt &&
A.InOutDate <= toDate &&
!(from B in ctx.tblEntry
where
B.CompanyId == companyId &&
A.EmployeeId == B.EmployeeId &&
A.InOutDate == B.EntryDate &&
B.EntryMethod == "M"
select new
{
B.EmployeeId
}).Contains(new { EmployeeId = A.EmployeeId })
orderby
A.EmployeeId, A.InOutDate select new
{
A.EmployeeId,
A.InOutDate,
A.InOutFlag,
A.InOutTime
}).ToList();
You can try with join also instead of inner query... You can use like this make change as per your need..
var data = (from A in ctx.tblMachine
join B in ctx.tblEntry on A.EmployeeId == B.EmployeeId &&
A.InOutDate == B.EntryDate &&
B.CompanyId == companyId &&
B.EntryMethod == "M"
where
A.CompanyId == companyId &&
A.InOutDate >= tempDt &&
A.InOutDate <= toDate &&
!(B.EmployeeId).Contains(new { EmployeeId = A.EmployeeId })
orderby
A.EmployeeId, A.InOutDate
select new
{
A.EmployeeId,
A.InOutDate,
A.InOutFlag,
A.InOutTime
}).ToList();

How to filter data using ternary operator in linq

var PFOpeningList = (from emp in dbContext.EmployeeList
join dsg in dbContext.hrmDesig on emp.HrmDesignationId equals dsg.Id into DsgLeftJoin
from dsgleftjoin in DsgLeftJoin.DefaultIfEmpty()
join opb in dbContext.PfOpeningBalances on emp.Id equals opb.HrmEmployeeId into OpbLeftJoin
from opbleftjoin in OpbLeftJoin.DefaultIfEmpty()
where opbleftjoin.CmnCalendarYearId == clndrId
&& (empId != 0 ? emp.Id == empId
: (dptId != 0 ? emp.HrmDepartmentId == dptId
: (officId != 0 ? emp.HrmOfficeId == officId : emp.CmnCompanyId == CmnId)))
&& emp.CmnCompanyId == CmnId
select new
{
EmployeeId=emp.Id,
EmployeeName=emp.Name,
Designation = dsgleftjoin.Name,
OpeningIncome = (decimal?)opbleftjoin.OpeningIncome,
EmployeeContribution = (decimal?)opbleftjoin.EmployeeContribution,
CompanyContribution = (decimal?)opbleftjoin.CompanyContribution
}).ToList();
I want to achieve all employees with designation(hrmDesig) from EmployeeList. Filtering with Calendar Year is mandatory. But if user select Office/Department/Employee, data should be filtered as well. How can I achieve this?
There's no need to use the conditional operator at all here... I suspect you want a where clause of:
where opbleftjoin.CmnCalendarYearId == clndrId
&& (empId == 0 || emp.Id == empId)
&& (dptId == 0 || emp.HrmDepartmentId == dptId)
&& (officId != 0 ? emp.HrmOfficeId == officId)
&& emp.CmnCompanyId == CmnId;
Better yet, you could add the conditions in steps - just have the mandatory ones to start with:
var query = from emp in dbContext.EmployeeList
join dsg in dbContext.hrmDesig on emp.HrmDesignationId equals dsg.Id into DsgLeftJoin
from dsgleftjoin in DsgLeftJoin.DefaultIfEmpty()
join opb in dbContext.PfOpeningBalances on emp.Id equals opb.HrmEmployeeId into OpbLeftJoin
from opbleftjoin in OpbLeftJoin.DefaultIfEmpty()
where opbleftjoin.CmnCalendarYearId == clndrId
&& emp.CmnCompanyId == CmnId
select new { emp, dsgleftjoin, opbleftjoin };
if (empId != 0)
{
query = query.Where(x => x.emp.Id == empId);
}
// etc
Also note that your query currently seems to assume that dsgleftjoin and opbleftjoin are non-null, when they could easily be null due to the left join.
Your where condition is wrong, you should try with something like this:
var PFOpeningList = (from emp in dbContext.EmployeeList
join dsg in dbContext.hrmDesig on emp.HrmDesignationId equals dsg.Id into DsgLeftJoin
from dsgleftjoin in DsgLeftJoin.DefaultIfEmpty()
join opb in dbContext.PfOpeningBalances on emp.Id equals opb.HrmEmployeeId into OpbLeftJoin
from opbleftjoin in OpbLeftJoin.DefaultIfEmpty()
where opbleftjoin.CmnCalendarYearId == clndrId
&& (empId != 0 ? emp.Id == empId : true)
&& (dptId != 0 ? emp.HrmDepartmentId == dptId : true)
&& (officId != 0 ? emp.HrmOfficeId == officId : true)
&& emp.CmnCompanyId == CmnId
select new
{
EmployeeId=emp.Id,
EmployeeName=emp.Name,
Designation = dsgleftjoin.Name,
OpeningIncome = (decimal?)opbleftjoin.OpeningIncome,
EmployeeContribution = (decimal?)opbleftjoin.EmployeeContribution,
CompanyContribution = (decimal?)opbleftjoin.CompanyContribution
}).ToList();
The idea is that if your filter parameter is 0, you avoid filtering setting the condition to true

Create a dynamic LINQ Query to cater for different conditions

I have a LINQ query to populate an object (which is then the datasource for a grid) with a few joins.
I want the query to be dynamic so it retrieves rows based on parameters passed, but so far it doesn't work as soon as the StatusID is passed in - it brings back all instances (as if a cartesion product is happening)
_viewfetch.POMastStatusID will either be -1 or a value of 1 or above.
QUERY:
var queryforobject = from p in db.POMasts.AsNoTracking()
join pr in db.Profiles.AsNoTracking() on p.ProfileID equals pr.ID
join c in db.CurrencyTypes.AsNoTracking() on p.CurrencyTypeID equals c.ID
join w in db.WHMasts.AsNoTracking() on p.WarehouseID equals w.ID
join t in db.TermCodeTypes.AsNoTracking() on p.TermCodeTypeID equals t.ID
join s in db.POMastStatusTypes.AsNoTracking() on p.StatusID equals s.ID
//Ensure that these are dynamic
where _viewfetch.VendMastID == -1 || p.VendorID == _viewfetch.VendMastID &&
_viewfetch.POMastStatusID == -1 || p.StatusID == _viewfetch.POMastStatusID
orderby p.ID
//Put the query results into the bespoke object
select new POMastObject { ID = p.ID,
OrderNo = p.OrderNo,
RaisedDate = p.RaisedDate,
RaisedBy = pr.Name,
Currency = c.Description,
Warehouse = w.Description,
Terms = t.Description,
LastEditedBy = p.LastEditedBy,
LastEditedDate = p.LastEditedDate,
Status = s.Name };
if (queryforobject.Count() > 0)
_dataobject = queryforobject.ToList();
Does anyone have any suggestions?
Your query is fine, it's just missing parentheses around the two parts of the && operator:
where (_viewfetch.VendMastID == -1 || p.VendorID == _viewfetch.VendMastID) &&
(_viewfetch.POMastStatusID == -1 || p.StatusID == _viewfetch.POMastStatusID)
Since && has higher precedence than ||, your query effectively evaluates with the two conditions in the middle AND-ed together, like this:
_viewfetch.VendMastID == -1 || (p.VendorID == _viewfetch.VendMastID && _viewfetch.POMastStatusID == -1) || p.StatusID == _viewfetch.POMastStatusID
This is not the logic that you are looking for, because when VendMastID is -1 you get all rows.

LINQ where clause is being ignored

I am have an issue with a linq query. I am joining two tables but the where clauses are being completely ignored.
using (var db = new Context())
{
var count = (from c in db.PERSON
join dt in db.DATA_INPUT_CHANGE_LOG
on c.DataInputTypeId equals dt.DataInputTypeId
join x in db.DATA_INPUT_CHANGE_LOG
on c.Id equals x.DataItemId
where c.PersonId == p1Id &&
c.RefPersonId == p2Id &&
c.RelationshipId == rId
where x.Approved
where x.Checked
where x.Hidden == false
select c).Count();
return count > 0;
}
In this particular query the x.Approved, x.Checked and x.Hidden == false where clauses are completely ignored.
Can anyone point me in the right direction?
Your syntax is incorrect. You should only have one where clause. See below:
var count = (from c in db.PERSON
join dt in db.DATA_INPUT_CHANGE_LOG
on c.DataInputTypeId equals dt.DataInputTypeId
join x in db.DATA_INPUT_CHANGE_LOG
on c.Id equals x.DataItemId
where c.PersonId == p1Id &&
c.RefPersonId == p2Id &&
c.RelationshipId == rId &&
x.Approved &&
x.Checked &&
x.Hidden == false
select c).Count();
return count > 0;

Only parameterless constructors and initializers are supported in LINQ to Entities

While trying a sem-complex query to display some ListView content on the page I got stuck on the famous "Only parameterless contstructor and initializers are supported in LINQ to Entities" error.
Here is the code I used ... I can't find a place where I initialized something inside the query with parameters ....
protected void ArtistsList()
{
Guid cat1 = new Guid("916ec8ae-8336-43b1-87c0-8536b2676560");
Guid cat2 = new Guid("92f2a07f-0570-4521-870a-bf898d1e92d6");
var memberOrders = (from o in DataContext.OrderSet
where o.Status == 1 || o.Status == 0
select o.ID);
var memberOrderDetails = (from o in DataContext.OrderDetailSet
where memberOrders.Any(f => f == o.Order.ID)
select o.Product.ID );
var inventoryItems = (from i in DataContext.InventoryItemSet
select i.Inventory.Product.ID);
var products = (from p in DataContext.ProductSet
join m in DataContext.ContactSet on p.ManufacturerID equals m.ID
where p.Active == true
&& p.ShowOnWebSite == true
&& p.Category.ID != cat1
&& p.Category.ID != cat2
&& p.AvailableDate <= DateTime.Today
&& (p.DiscontinuationDate == null || p.DiscontinuationDate >= DateTime.Today)
&& memberOrderDetails.Any(f => f != p.ID)
&& inventoryItems.Any(f => f == p.ID)
select new { ContactID = m.ID, ContactName = m.Name });
artistsRepeater.DataSource = products;
artistsRepeater.DataBind();
Response.Write("PRODUCT COUNT: " + products.Count());
}
The error itself pops on the line artistsRepeater.DataSource = products;
I tried to comment the lines && memberOrderDetails.Any(f => f != p.ID) and && inventoryItems.Any(f => f == p.ID) , still doesn't change anything
Any hints ?
[edit]
With LINQpad, it works with the join but with it is bugging on the commented line
(from p in Products
join m in Members on p.ManufacturerID.Value equals m.ID
where p.Active == true
&& p.ShowOnWebSite == true
&& p.AvailableDate <= DateTime.Today
&& (p.DiscontinuationDate == null || p.DiscontinuationDate >= DateTime.Today)
//&& (from od in MemberOrderDetails where (from mo in MemberOrders where mo.Status == 1 || mo.Status == 0 select mo.ID).Any(f => f == od.ID) select od.Product.ID)
&& (from inv in InventoryItems select inv.Inventory.ProductID).Any(i => i.Value == p.ID)
select m).Distinct()
[edit-2]
It seems that this query in LINQpad is ok :
(from p in Products
join m in Members on p.ManufacturerID.Value equals m.ID
where p.Active == true
&& p.ShowOnWebSite == true
&& p.AvailableDate <= DateTime.Today
&& (p.DiscontinuationDate == null || p.DiscontinuationDate >= DateTime.Today)
&& !(from od in MemberOrderDetails where (from mo in MemberOrders where mo.Status == 1 || mo.Status == 0 select mo).Any(f => f.ID == od.ID) select od.Product.ID).Any(i => i == p.ID)
&& (from inv in InventoryItems select inv.Inventory.ProductID).Any(i => i.Value == p.ID)
select m)
OK, this is subtle, but what if you change your LINQPad query from:
(from p in Products
join m in Members
on p.ManufacturerID.Value equals m.ID
where p.Active == true
&& p.ShowOnWebSite == true
&& p.AvailableDate <= DateTime.Today
&& (p.DiscontinuationDate == null || p.DiscontinuationDate >= DateTime.Today)
&& (from od in MemberOrderDetails
where (from mo in MemberOrders
where mo.Status == 1 || mo.Status == 0
select mo.ID).Any(f => f == od.ID)
select od.Product.ID)
&& (from inv in InventoryItems
select inv.Inventory.ProductID).Any(i => i.Value == p.ID)
...to:
(from p in Products
join m in Members
on p.ManufacturerID.Value equals m.ID
where p.Active == true
&& p.ShowOnWebSite == true
&& p.AvailableDate <= DateTime.Today
&& (p.DiscontinuationDate == null || p.DiscontinuationDate >= DateTime.Today)
&& (from od in MemberOrderDetails
where (from mo in MemberOrders
where mo.Status == 1 || mo.Status == 0
select mo).Any(f => f.ID == od.ID) // NOTE!
select od.Product.ID)
&& (from inv in InventoryItems
select inv.Inventory.ProductID).Any(i => i.Value == p.ID)
Why? I think type inference might be doing you wrong here. I've seen a similar thing with DateTimes.
The most likely culprit is:
select new { ContactID = m.ID, ContactName = m.Name }
This is because anonymous types do not have parameterless constructors. What's odd about that is that anonymous types are de riguer in LINQ to Entities. I just don't see any other line that could be offending.
First try removing that line and see if the error goes away. At least we'll know if it's that line or not. Then we can focus on figuring out why.
Edit: What are the types of OrderSet.ID, Product.ID and Order.ID and ContactSet.ID? Are any of them Guid and implicitly the Guid constructor is being called?
Convert to a list first, then call your select statement:
var res = abc.getEmployess().toList().select(x => new keyvaluepair<int, string>(x.EmpID, x.EmpName.tostring)).tolist();

Categories

Resources