I have the following View defined in my Context class, Entity Framework: I have added .Include() here, with the hopes that this will eliminate calls to database later.
public IQueryable<GeneralReportModel> vwGeneralReportItems
{
get
{
return from p in this.Patients.AsNoTracking().Include(p => p.Age) //Multiple includes added to include all properties on model
join fac in this.Facilities.AsNoTracking() on p.FacilityID equals fac.ID into fJoin
from f in fJoin.DefaultIfEmpty()
join userFac in this.UserFacilities.AsNoTracking() on p.FacilityID equals userFac.FacilityID into ufJoin
from uf in ufJoin.DefaultIfEmpty()
select new GeneralReportModel()
{
Patient = p,
//ReportIndex = ri,
Facility = f,
UserFacility = uf
};
}
}
I also have this function, which does some filtering on my select from the View:
private IQueryable<GeneralReportModel> generalReportQuery(ApplicationTypes.ReportParameterObject repParam)
{
var reportQuery = ReportHelper.GeneralReport_Source(this.DbContext, repParam, false);
reportQuery = ReportHelper.GeneralReport_FilterCriteria(this.DbContext, reportQuery, repParam);
// Lab confirmed - Extra Pulmonary
reportQuery = reportQuery.Where(rq =>
rq.Patient.TypeOfResistantTBConfirmation.Value == ApplicationTypes.ResistantTBConfirmationTypes.LABCONFIRMED.ToString() &&
rq.Patient.SiteOfDiseaseID != repParam.DatabaseSettingObject.SiteOfDiseaseID_ExtraPulmonary &&
rq.Patient.PatientCategoryID != ApplicationGlobal.GLBDATABASESETTINGOBJECT.PatientCategoryID_TransferIn);
return reportQuery;
}
The code below
var reportQueryMdr = reportQuery.Where(rq =>
rq.Patient.TypeOfResistantTB.Value == ApplicationTypes.ResistantTBTypes.MDR.ToString());
var list = reportQueryMdr.ToList();
foreach (var obj in list)
{
obj.ReportIndex = obj.Patient.getRecalculatedReportIndexFields(ApplicationGlobal.GLBDATABASESETTINGOBJECT);
}
reportQueryMdr = list.AsQueryable();
Still calls the Database for each iteration made over the list. How can I make this list entirely reside in memory, with all the Patient properties.
Related
var entity =
from document in db.Context.DocumentEntity
join product in db.Context.ProductEntity on document.ProductId equals product.Id
join partner in db.Context.PartnerEntity on product.PartnerId equals partner.Id
select new
{
document,
product,
partner
} into t1
where request.PartnerFilter.Contains(t1.partner.Name)
group t1 by t1.document.Date into rp
select new
{
PartnerName = rp.FirstOrDefault().partner.Name,
Date = rp.FirstOrDefault().document.Date,
Income = rp.Sum(x => x.document.Income),
Click= rp.Sum(x => x.document.Click)
};
result = ToDataTable(entity.OrderByDescending(d=>d.Date).ToList());
public static DataTable ToDataTable<T>(List<T> items)
{
DataTable dataTable = new DataTable(typeof(T).Name);
PropertyInfo[] Props = typeof(T).GetProperties(BindingFlags.Public | BindingFlags.Instance);
foreach (PropertyInfo prop in Props)
{
var type = (prop.PropertyType.IsGenericType && prop.PropertyType.GetGenericTypeDefinition() == typeof(Nullable<>) ? Nullable.GetUnderlyingType(prop.PropertyType) : prop.PropertyType);
dataTable.Columns.Add(prop.Name, type);
}
foreach (T item in items)
{
var values = new object[Props.Length];
for (int i = 0; i < Props.Length; i++)
{
values[i] = Props[i].GetValue(item, null);
}
dataTable.Rows.Add(values);
}
return dataTable;
}
The problem is on where clause. request.PartnerFilter is a string array and might be null. I need to check if partner.Name is included in it. Kind of Sql Where-In. In the end entity.ToList() throws System.NotSupported Exception. How can I accomplish to filter?
If you want to use Contains inside the EF query expression tree, you need to ensure the variable is not null. And you need to do that (along with the condition if it needs to be applied) outside the query.
For instance:
var partnerFilter = request.PartnerFilter ?? Enumerable.Empty<string>();
bool applyPartnerFilter = partnerFilter.Any();
var entity =
...
where (!applyPartnerFilter || partnerFilter.Contains(t1.partner.Name))
...
But in my opinion it would be much better to apply the optional filter(s) outside the query, for instance:
var partners = db.Context.PartnerEntity.AsQueryable();
if (request.PartnerFilter != null && request.PartnerFilter.Any())
partners = partners.Where(partner => request.PartnerFilter.Contains(partner.Name));
var entity =
...
join partner in partners on product.PartnerId equals partner.Id
...
(no where)
Take this request part out of the equation because Entity Framework doesn't know what to do with a Request object, it can handle strings, string arrays, etc.
string[] strArray=request.PartnerFilter;
var entity =
from document in db.Context.DocumentEntity
join product in db.Context.ProductEntity on document.ProductId equals product.Id
join partner in db.Context.PartnerEntity on product.PartnerId equals partner.Id
select new
{
document,
product,
partner
} into t1
//Check if null
where strArray!=null && strArray.Any() && strArray.Contains(t1.partner.Name)
group t1 by t1.document.Date into rp
select new
{
PartnerName = rp.FirstOrDefault().partner.Name,
Date = rp.FirstOrDefault().document.Date,
Income = rp.Sum(x => x.document.Income),
Click= rp.Sum(x => x.document.Click)
};
Also, use Navigation Properties instead of joins
You use the SQL WHERE IN () Clause with Contains correct. Your only problem is the possible null exception.
What should happen if the array is empty? Would you like to have all the values? Use true if array is null otherwise false
Try this:
string[] partnerNames = request.PartnerFilter;
var entity =
from document in db.Context.DocumentEntity
join product in db.Context.ProductEntity on document.ProductId equals product.Id
join partner in db.Context.PartnerEntity on product.PartnerId equals partner.Id
select new
{
document,
product,
partner
} into t1
where partnerNames?.Contains(t1.partner.Name) ?? true
group t1 by t1.document.Date into rp
select new
{
PartnerName = rp.FirstOrDefault().partner.Name,
Date = rp.FirstOrDefault().document.Date,
Income = rp.Sum(x => x.document.Income),
Click= rp.Sum(x => x.document.Click)
};
WHERE IN - as query Syntax
var selected = from document in Document
where new[] {"Paul", "Peter"}.Contains(document.UserName)
select document
WHERE IN - as method Syntax
var selected = Document
.Where(d => new[] ["Paul","Peter"}.Contains(d.UserName))
I have a ControlMeasure table that holds information on each control measure and a ControlMeasurepeopleExposed Table that holds a record for each person exposed in the control measure this could be 1 record or many records.
I Have a controller that populates a List view
For each item in the list, Control Measure, I would like to create a string that shows all the People at risk
e.g.
PeopleString = "Employees, Public, Others";
Ive added a foreach in the controller to show what I'm trying to do however I'm aware that this wont work.
The controller is this:
public ActionResult ControlMeasureList(int raId)
{
//Populate the list
var hazards = new List<Hazard>(db.Hazards);
var controlMeasures = new List<ControlMeasure>(db.ControlMeasures).Where(x => x.RiskAssessmentId == raId);
var cmcombined = (
from g in hazards
join f in controlMeasures
on new { g.HazardId } equals new { f.HazardId }
select new CMCombined
{
Activity = f.Activity,
ControlMeasureId = f.ControlMeasureId,
ExistingMeasure = f.ExistingMeasure,
HazardName = g.Name,
LikelihoodId = f.LikelihoodId,
Rating = f.Rating,
RiskAssessmentId = f.RiskAssessmentId,
SeverityId = f.SeverityId,
}).OrderBy(x => x.Activity).ToList();
var cmPeopleExp = new List<ControlMeasurePeopleExposed>(db.ControlMeasurePeopleExposeds).Where(x => x.RiskAssessmentId == raId);
var peopleExp = from c in cmPeopleExp
join d in db.PeopleExposeds
on c.PeopleExposedId equals d.PeopleExposedId
orderby d.Name
select new RAPeopleExp
{
RAPeopleExpId = c.PeopleExposedId,
PeopleExpId = c.PeopleExposedId,
PeopleExpName = d.Name,
RiskAssessmentId = c.RiskAssessmentId,
ControlMeasureId = c.ControlMeasureId
};
var model = cmcombined.Select(t => new FullControlMeasureListViewModel
{
ControlMeasureId = t.ControlMeasureId,
HazardName = t.HazardName,
LikelihoodId = t.LikelihoodId,
Rating = t.Rating,
SeverityId = t.SeverityId,
Activity = t.Activity,
ExCM = t.ExistingMeasure,
//This section here is where I'm struggling
var PeopleString = new StringBuilder();
foreach (var p in peopleExp)
{
PeopleString.AppendLine(p.PeopleName);
{
PeopleExposed = PeopleString,
});
return PartialView("_ControlMeasureList", model);
}
I know I cant directly put this code in the controller but it does represent what I want to do.
You can't foreach within an object initializer (which is what you're trying to do when instantiating FullControlMeasureListViewModel). You can, however, use a combination of string.Join and peopleExp.Select:
var model = cmcombined.Select(t => new FullControlMeasureListViewModel
{
//other props
PeopleExposed = string.Join(",", peopleExp
.Where(p => p.ControlMeasureId == t.ControlMeasureId)
.Select(p => p.PeopleExpName));
//other props
});
Right now, I'm trying to write a method for a survey submission program that utilizes a very normalized schema.
I have a method that is meant to generate a survey for a team of people, linking several different EF models together in the process. However, this method runs EXTREMELY slowly for anything but the smallest team sizes (taking 11.2 seconds to execute for a 4-person team, and whopping 103.9 seconds for an 8 person team). After some analysis, I found that 75% of the runtime is taken up in the following block of code:
var TeamMembers = db.TeamMembers.Where(m => m.TeamID == TeamID && m.OnTeam).ToList();
foreach (TeamMember TeamMember in TeamMembers)
{
Employee employee = db.Employees.Find(TeamMember.EmployeeID);
SurveyForm form = new SurveyForm();
form.Submitter = employee;
form.State = "Not Submitted";
form.SurveyGroupID = surveygroup.SurveyGroupID;
db.SurveyForms.Add(form);
db.SaveChanges();
foreach (TeamMember peer in TeamMembers)
{
foreach (SurveySectionDetail SectionDetail in sectionDetails)
{
foreach (SurveyAttributeDetail AttributeDetail in attributeDetails.Where(a => a.SectionDetail.SurveySectionDetailID == SectionDetail.SurveySectionDetailID) )
{
SurveyAnswer answer = new SurveyAnswer();
answer.Reviewee = peer;
answer.SurveyFormID = form.SurveyFormID;
answer.Detail = AttributeDetail;
answer.SectionDetail = SectionDetail;
db.SurveyAnswers.Add(answer);
db.SaveChanges();
}
}
}
}
I'm really at a loss as to how I might go about cutting back the runtime. Is this just the price I pay for having this many related entities? I know that joins are expensive operations, and that I've essentially got 3 Or is there some inefficiency that I'm overlooking?
Thanks for your help!
EDIT: As requested by Xiaoy312, here's how sectionDetails and attributeDetails are defined:
SurveyTemplate template = db.SurveyTemplates.Find(SurveyTemplateID);
List<SurveySectionDetail> sectionDetails = new List<SurveySectionDetail>();
List<SurveyAttributeDetail> attributeDetails = new List<SurveyAttributeDetail>();
foreach (SurveyTemplateSection section in template.SurveyTemplateSections)
{
SurveySectionDetail SectionDetail = new SurveySectionDetail();
SectionDetail.SectionName = section.SectionName;
SectionDetail.SectionOrder = section.SectionOrder;
SectionDetail.Description = section.Description;
SectionDetail.SurveyGroupID = surveygroup.SurveyGroupID;
db.SurveySectionDetails.Add(SectionDetail);
sectionDetails.Add(SectionDetail);
db.SaveChanges();
foreach (SurveyTemplateAttribute attribute in section.SurveyTemplateAttributes)
{
SurveyAttributeDetail AttributeDetail = new SurveyAttributeDetail();
AttributeDetail.AttributeName = attribute.AttributeName;
AttributeDetail.AttributeScale = attribute.AttributeScale;
AttributeDetail.AttributeType = attribute.AttributeType;
AttributeDetail.AttributeOrder = attribute.AttributeOrder;
AttributeDetail.SectionDetail = SectionDetail;
db.SurveyAttributeDetails.Add(AttributeDetail);
attributeDetails.Add(AttributeDetail);
db.SaveChanges();
}
}
There is several points that you can improve :
Do not SaveChanges() on each Add() :
foreach (TeamMember TeamMember in TeamMembers)
{
...
// db.SaveChanges();
foreach (TeamMember peer in TeamMembers)
{
foreach (SurveySectionDetail SectionDetail in sectionDetails)
{
foreach (SurveyAttributeDetail AttributeDetail in attributeDetails.Where(a => a.SectionDetail.SurveySectionDetailID == SectionDetail.SurveySectionDetailID) )
{
...
// db.SaveChanges();
}
}
}
db.SaveChanges();
}
Consider to reduce the numbers of round trips to the database. This can be done by : they are memory-intensive
using Include() to preload your navigation properties; or
cashing the partial or whole table with ToDictionary() or ToLookup()
Instead of Add(), use AddRange() or even BulkInsert() from EntityFramework.BulkInsert if that fits your setup :
db.SurveyAnswers.AddRange(
TeamMembers.SelectMany(p =>
sectionDetails.SelectMany(s =>
attributeDetails.Where(a => a.SectionDetail.SurveySectionDetailID == s.SurveySectionDetailID)
.Select(a => new SurveyAnswer()
{
Reviewee = p,
SurveyFormID = form.SurveyFormID,
Detail = a,
SectionDetail = s,
}))));
Use Include to avoid SELECT N + 1 issue.
SurveyTemplate template = db.SurveyTemplates.Include("SurveyTemplateSections")
.Include("SurveyTemplateSections.SurveyTemplateAttributes")
.First(x=> x.SurveyTemplateID == SurveyTemplateID);
Generate the whole object graph and then save to DB.
List<SurveySectionDetail> sectionDetails = new List<SurveySectionDetail>();
List<SurveyAttributeDetail> attributeDetails = new List<SurveyAttributeDetail>();
foreach (SurveyTemplateSection section in template.SurveyTemplateSections)
{
SurveySectionDetail SectionDetail = new SurveySectionDetail();
//Some code
sectionDetails.Add(SectionDetail);
foreach (SurveyTemplateAttribute attribute in section.SurveyTemplateAttributes)
{
SurveyAttributeDetail AttributeDetail = new SurveyAttributeDetail();
//some code
attributeDetails.Add(AttributeDetail);
}
}
db.SurveySectionDetails.AddRange(sectionDetails);
db.SurveyAttributeDetails.AddRange(attributeDetails);
db.SaveChanges();
Load all employees you want before the loop, this will avoids database query for every team member.
var teamMemberIds = db.TeamMembers.Where(m => m.TeamID == TeamID && m.OnTeam)
.Select(x=>x.TeamMemberId).ToList();
var employees = db.Employees.Where(x => teamMemberIds.Contains(x.EmployeeId));
create a dictionary for attributeDetails based on their sectionDetailId to avoid query the list on every iteration.
var attributeDetailsGroupBySection = attributeDetails.GroupBy(x => x.SectionDetailId)
.ToDictionary(x => x.Key, x => x);
Move saving of SurveyAnswers and SurveyForms to outside of the loops:
List<SurveyForm> forms = new List<SurveyForm>();
List<SurveyAnswer> answers = new List<SurveyAnswer>();
foreach (int teamMemberId in teamMemberIds)
{
var employee = employees.First(x => x.Id == teamMemberId);
SurveyForm form = new SurveyForm();
//some code
forms.Add(form);
foreach (int peer in teamMemberIds)
{
foreach (SurveySectionDetail SectionDetail in sectionDetails)
{
foreach (SurveyAttributeDetail AttributeDetail in
attributeDetailsGroupBySection[SectionDetail.Id])
{
SurveyAnswer answer = new SurveyAnswer();
//some code
answers.Add(answer);
}
}
}
}
db.SurveyAnswers.AddRange(answers);
db.SurveyForms.AddRange(forms);
db.SaveChanges();
Finally if you want faster insertions you can use EntityFramework.BulkInsert. With this extension, you can save the data like this:
db.BulkInsert(answers);
db.BulkInsert(forms);
My intersect in LINQ somehow dont seem to work. I have two excel sheets. I fetch it using LinQToExcel and (LinQToExcel does not support VisitSubQueryExpression i have to do some extra work).
List<BoardSheet> sourceTest = (from t in Boards[0]
where t["Board"] == boardName
select new CircuitSet
{
ID = string.Format(t["ID"]),
Data = string.Format(t["Data"]),
CtrlType = string.Format(t["CtrlType"]),
sys = string.Format(t["sys"]),
code = string.Format(t["code"])
}
).ToList<BoardSheet>();
List<BoardSheet> targetTest = (from t in Boards[0]
where t["Board"] == boardName
select new CircuitSet
{
ID = string.Format(t["ID"]),
Data = string.Format(t["Data"]),
CtrlType = string.Format(t["CtrlType"]),
sys = string.Format(t["sys"]),
code = string.Format(t["code"])
}
).ToList<BoardSheet>();
IEnumerable<BoardSheet> board = sourceTest.Intersect(targetTest);
board's count always returns 0. But when i iterate thro the field values of sourceTest and targetSet i see common field values.
These are instances of reference types. Intersect is using the DefaultComparer for reference types, which is ReferenceEquals. Since sourceTest has no common instances with targetTest, no results are found.
You could create a Comparer, or you could join like this:
List<CircuitSet> results =
(
from s in sourceTest
join t in targetTest
on s.Id equals t.Id
where s.Data == t.Data && s.ControlType == t.ControlType ...
select s
).ToList();
I think an easy way would be to implement IEqualityComparer<CircuitSet>. If CircuitSet's key is ID, then you could do it like this:
public class CircuitSetComparer : IEqualityComparer<CircuitSet>
{
#region IEqualityComparer<CircuitSet> Members
public bool Equals(CircuitSet x, CircuitSet y)
{
return x.ID == y.ID;
}
public int GetHashCode(CircuitSet obj)
{
return obj.ID;
}
#endregion
}
Then in your code:
IEnumerable<BoardSheet> board = sourceTest.Intersect(targetTest, new CircuitSetComparer());
GetHashCode method is tricky though, but it should be alright if my assumptions (ID being the key) are correct.
but i changed the query based on what david suggested
List<BoardSheet> sourceTest =(from s in (from t in Boards[0]
where t["Board"] == boardName
select new CircuitSet
{
ID = string.Format(t["ID"]),
Data = string.Format(t["Data"]),
CtrlType = string.Format(t["CtrlType"]),
sys = string.Format(t["sys"]),
code = string.Format(t["code"])
}
).ToList<BoardSheet>() jon tbl in (from t in Boards[0]
where t["Board"] == boardName
select new CircuitSet
{
ID = string.Format(t["ID"]),
Data = string.Format(t["Data"]),
CtrlType = string.Format(t["CtrlType"]),
sys = string.Format(t["sys"]),
code = string.Format(t["code"])
}
).ToList<BoardSheet>() on s.ID equals tbl.ID select s).ToList<BoardSheet>() ;
The code below is what I currently have and works fine. I feel that I could do more of the work I am doing in Linq instead of C# code.
Is there is anyone out there who can accomplish the same result with more Linq code and less C# code.
public List<Model.Question> GetSurveyQuestions(string type, int typeID)
{
using (eMTADataContext db = DataContextFactory.CreateContext())
{
List<Model.Question> questions = new List<Model.Question>();
List<Linq.Survey_Question> survey_questions;
List<Linq.Survey> surveys = db.Surveys
.Where(s => s.Type.Equals(type) && s.Type_ID.Equals(typeID))
.ToList();
if (surveys.Count > 0)
{
survey_questions = db.Survey_Questions
.Where(sq => sq.Survey_ID == surveys[0].ID).ToList();
foreach (Linq.Survey_Question sq in survey_questions)
{
Model.Question q = Mapper.ToBusinessObject(sq.Question);
q.Status = sq.Status;
questions.Add(q);
}
}
else
{
questions = null;
}
return questions;
}
}
Here is my Mapper function from my Entity to Biz Object
internal static Model.Question ToBusinessObject(Linq.Question q)
{
return new Model.Question
{
ID = q.ID,
Name = q.Name,
Text = q.Text,
Choices = ToBusinessObject(q.Question_Choices.ToList())
};
}
I want my mapper funciton to map the Question Status like so.
internal static Model.Question ToBusinessObject(Linq.Question q)
{
return new Model.Question
{
ID = q.ID,
Name = q.Name,
Text = q.Text,
Choices = ToBusinessObject(q.Question_Choices.ToList()),
Status = q.Survey_Questions[?].Status
};
}
? the issue is this function does not know which survey to pull the status from.
Instead of creating the biz object then setting the Status property in a foreach loop like so
foreach (Linq.Survey_Question sq in survey_questions)
{
Model.Question q = Mapper.ToBusinessObject(sq.Question);
q.Status = sq.Status;
questions.Add(q);
}
I would like to somehow filter the EntitySet<Survey_Question> in the q object above in the calling method, such that there would only be one item in the q.Survey_Questions[?] collection.
below is my database schema and business object schema
What I needed to do was setup a join.
public List<Model.Question> GetSurveyQuestions(string type, int typeID)
{
using (eMTADataContext db = DataContextFactory.CreateContext())
{
return db.Survey_Questions
.Where(s => s.Survey.Type.Equals(type) && s.Survey.Type_ID.Equals(typeID))
.Join(db.Questions,
sq => sq.Question_ID,
q => q.ID,
(sq, q) => Mapper.ToBusinessObject(q, sq.Status)).ToList();
}
}
And then overload my Mapper Function
internal static Model.Question ToBusinessObject(Linq.Question q, string status)
{
return new Model.Question
{
ID = q.ID,
Name = q.Name,
Text = q.Text,
Status = status,
Choices = ToBusinessObject(q.Question_Choices.ToList()),
};
}
from question in db.Survey_Questions
let surveys = (from s in db.Surveys
where string.Equals(s.Type, type, StringComparison.InvariantCultureIgnoreCase) &&
s.Type_ID == typeID)
where surveys.Any() &&
surveys.Contains(s => s.ID == question.ID)
select new Mapper.Question
{
ID = question.Id,
Name = question.Name,
Text = question.Text,
Choices = ToBusinessObject(question.Question_Choices.ToList()),
Status = question.Status
}
Does that get you on the right track?
Why are you duplicating all your classes? You could just extend the LINQ to SQL classes with your business logic - they are partial classes. This is somewhat against the purpose of an OR mapper - persisting business entities.