I have an object that I am populating via mapping and lookup woth Reflection, here is the call
this.SearchResults = (from a in response.postings
select new SearchResponseModel
{
Id = a.id,
TimeStampDate = a.timestampDate,
Body = a.body,
Title = a.heading,
Status = a.status,
State = a.state,
Language = a.language,
Currency = a.currency,
CategoryGroup = a.category_group,
Source = a.source,
ExternalId = a.external_id,
ExternalUrl = a.external_url,
Price = a.price,
Location = PopulateLocation(a.location)
}
).ToList();
And the method that does the mapping.
private static List<LocationLookupModel> PopulateLocation(Location location)
{
List<LocationLookupModel> allLocations = new List<LocationLookupModel>();
if (HttpContext.Current.Session["LocationModel"] == null)
{
HttpContext.Current.Session["LocationModel"] = allLocations = new LocationModel().LocationList;
}
else
{
allLocations = (List<LocationLookupModel>)HttpContext.Current.Session["LocationModel"];
}
List<LocationLookupModel> modelList = new List<LocationLookupModel>();
foreach (PropertyInfo propertyInfo in location.GetType().GetProperties())
{
var value = propertyInfo.GetValue(location);
if (value != null)
{
LocationLookupModel model = (from a in allLocations
where a.Code == propertyInfo.GetValue(location).ToString()
select a).FirstOrDefault();
if (model != null)
{
modelList.Add(model);
}
}
}
return modelList;
}
}
The issue that I run into is that allLocations object has about 70k records (it represents a list of location lookup values for countries, states, zipcodes, etc), and populating about 100 instances of SearchResponseModel takes about 20 seconds. This is far too long for a UI call, and I have not been able to find a way to make it faster. I understand I am basically doing 3 nested loops (calling helper method for each population, looping over reflected properties, and finally the LINQ call over 70k records) so there are some time efficiency issues, but I am a bit lost on what tricks I can use to make this process more efficient.
Related
I have to update one field in the row of the table after fetching two records from the same row. As an easiest practice I have fetched two records individually, created a new value and then updating that particular property through Entity framework. I think there is a better way to do the same thing with less code. If any body can suggest please.
if (objModel.amountpaid==0)
{
using (estatebranchEntities db=new estatebranchEntities())
{
int rentVar = Convert.ToInt32(db.PropertyDetails.Where(m => m.propertyid == objVM.propertyid).Select(m => m.rent).SingleOrDefault());
int balanceVar = Convert.ToInt32(db.PropertyDetails.Where(m => m.propertyid == objVM.propertyid).Select(m => m.balance).SingleOrDefault());
int balanceUpdateVar = (rentVar + balanceVar);
var propInfo = new PropertyDetail() { balance = balanceUpdateVar };
//var result = (from a in db.PropertyDetails
// where a.propertyid == objVM.propertyid
// select new PropertyDetail
// {
// rent = a.rent,
// balance = a.balance
// }).ToList();
db.PropertyDetails.Attach(propInfo);
db.Entry(propInfo).Property(z => z.balance).IsModified = true;
db.SaveChanges();
}
}
Here is what I think you can do.
Fetch the data once and update once.
using (estatebranchEntities db=new estatebranchEntities())
{
var propDetails = db.PropertyDetails.FirstOrDefault(m => m.propertyid == objVM.propertyid);
if (propDetails != null)
{
int rentVar = Convert.ToInt32(propDetails.rent);
int balanceVar = Convert.ToInt32(propDetails.balance);
int balanceUpdateVar = rentVar + balanceVar;
//now do the update
propDetails.balance = balanceUpdateVar;
db.Entry(proDetails).State = EntityState.Modified;
db.SaveChanges();
}
}
if you need to use the rentVar,balanceVar or the balanceUpdateVar, outside of the using statement then declare them outside it.
I need some help to calculate a property inside my Linq query.
I know I need to use "let" somewhere, but I can't figure it out!
So, first I have this method to get my list from Database:
public BindingList<Builders> GetListBuilders()
{
BindingList<Builders> builderList = new BindingList<Builders>();
var ctx = new IWMJEntities();
var query = (from l in ctx.tblBuilders
select new Builders
{
ID = l.BuilderID,
Projeto = l.NomeProjeto,
Status = l.Status,
DataPedido = l.DataPedido,
DataPendente = l.DataPendente,
DataEntregue = l.DataEntregue,
DataAnulado = l.DataAnulado
});
foreach (var list in query)
builderList.Add(list);
return builderList;
}
Then, I have a function to calculate the Days between Dates accordingly to Status:
public int GetDays()
{
int Dias = 0;
foreach (var record in GetListBuilders)
{
if (record.Status == "Recebido")
{
Dias = GetBusinessDays(record.DataPedido, DateTime.Now);
}
else if (record.Status == "Pendente")
{
Dias = GetBusinessDays(record.DataPedido, (DateTime)record.DataPendente);
}
else if (record.Status == "Entregue")
{
Dias = GetBusinessDays(record.DataPedido, (DateTime)record.DataEntregue);
}
else if (record.Status == "Anulado")
{
Dias = GetBusinessDays(record.DataPedido, (DateTime)record.DataAnulado);
}
}
return Dias;
}
I need to call the GetDays in a DataGridView to give the days for each record.
My big problem is, How do I get this? include it in Linq Query? Calling GetDays() (need to pass the ID from each record to GetDays() function)!?
Any help?
Thanks
I think it would be easier to create an extension method:
public static int GetBusinessDays(this Builders builder) // or type of ctx.tblBuilders if not the same
{
if (builder == null) return 0;
switch(builder.status)
{
case "Recebido": return GetBusinessDays(builder.DataPedido, DateTime.Now);
case "Pendente": return GetBusinessDays(builder.DataPedido, (DateTime)builder.DataPendente);
case "Entregue": return GetBusinessDays(builder.DataPedido, (DateTime)builder.DataEntregue);
case "Anulado": GetBusinessDays(builder.DataPedido, (DateTime)builder.DataAnulado);
default: return 0;
}
}
Then, call it like that:
public BindingList<Builders> GetListBuilders()
{
BindingList<Builders> builderList = new BindingList<Builders>();
var ctx = new IWMJEntities();
var query = (from l in ctx.tblBuilders
select new Builders
{
ID = l.BuilderID,
Projeto = l.NomeProjeto,
Status = l.Status,
DataPedido = l.DataPedido,
DataPendente = l.DataPendente,
DataEntregue = l.DataEntregue,
DataAnulado = l.DataAnulado,
Dias = l.GetBusinessDays()
});
foreach (var list in query)
builderList.Add(list);
return builderList;
}
To do better, to convert a object to a new one, you should create a mapper.
Why does it need to be a part of the query? You can't execute C# code on the database. If you want the calculation to be done at the DB you could create a view.
You're query is executed as soon as the IQueryable is enumerated at the foreach loop. Why not just perform the calculation on each item as they are enumerated and set the property when you are adding each item to the list?
Right now, I'm trying to write a method for a survey submission program that utilizes a very normalized schema.
I have a method that is meant to generate a survey for a team of people, linking several different EF models together in the process. However, this method runs EXTREMELY slowly for anything but the smallest team sizes (taking 11.2 seconds to execute for a 4-person team, and whopping 103.9 seconds for an 8 person team). After some analysis, I found that 75% of the runtime is taken up in the following block of code:
var TeamMembers = db.TeamMembers.Where(m => m.TeamID == TeamID && m.OnTeam).ToList();
foreach (TeamMember TeamMember in TeamMembers)
{
Employee employee = db.Employees.Find(TeamMember.EmployeeID);
SurveyForm form = new SurveyForm();
form.Submitter = employee;
form.State = "Not Submitted";
form.SurveyGroupID = surveygroup.SurveyGroupID;
db.SurveyForms.Add(form);
db.SaveChanges();
foreach (TeamMember peer in TeamMembers)
{
foreach (SurveySectionDetail SectionDetail in sectionDetails)
{
foreach (SurveyAttributeDetail AttributeDetail in attributeDetails.Where(a => a.SectionDetail.SurveySectionDetailID == SectionDetail.SurveySectionDetailID) )
{
SurveyAnswer answer = new SurveyAnswer();
answer.Reviewee = peer;
answer.SurveyFormID = form.SurveyFormID;
answer.Detail = AttributeDetail;
answer.SectionDetail = SectionDetail;
db.SurveyAnswers.Add(answer);
db.SaveChanges();
}
}
}
}
I'm really at a loss as to how I might go about cutting back the runtime. Is this just the price I pay for having this many related entities? I know that joins are expensive operations, and that I've essentially got 3 Or is there some inefficiency that I'm overlooking?
Thanks for your help!
EDIT: As requested by Xiaoy312, here's how sectionDetails and attributeDetails are defined:
SurveyTemplate template = db.SurveyTemplates.Find(SurveyTemplateID);
List<SurveySectionDetail> sectionDetails = new List<SurveySectionDetail>();
List<SurveyAttributeDetail> attributeDetails = new List<SurveyAttributeDetail>();
foreach (SurveyTemplateSection section in template.SurveyTemplateSections)
{
SurveySectionDetail SectionDetail = new SurveySectionDetail();
SectionDetail.SectionName = section.SectionName;
SectionDetail.SectionOrder = section.SectionOrder;
SectionDetail.Description = section.Description;
SectionDetail.SurveyGroupID = surveygroup.SurveyGroupID;
db.SurveySectionDetails.Add(SectionDetail);
sectionDetails.Add(SectionDetail);
db.SaveChanges();
foreach (SurveyTemplateAttribute attribute in section.SurveyTemplateAttributes)
{
SurveyAttributeDetail AttributeDetail = new SurveyAttributeDetail();
AttributeDetail.AttributeName = attribute.AttributeName;
AttributeDetail.AttributeScale = attribute.AttributeScale;
AttributeDetail.AttributeType = attribute.AttributeType;
AttributeDetail.AttributeOrder = attribute.AttributeOrder;
AttributeDetail.SectionDetail = SectionDetail;
db.SurveyAttributeDetails.Add(AttributeDetail);
attributeDetails.Add(AttributeDetail);
db.SaveChanges();
}
}
There is several points that you can improve :
Do not SaveChanges() on each Add() :
foreach (TeamMember TeamMember in TeamMembers)
{
...
// db.SaveChanges();
foreach (TeamMember peer in TeamMembers)
{
foreach (SurveySectionDetail SectionDetail in sectionDetails)
{
foreach (SurveyAttributeDetail AttributeDetail in attributeDetails.Where(a => a.SectionDetail.SurveySectionDetailID == SectionDetail.SurveySectionDetailID) )
{
...
// db.SaveChanges();
}
}
}
db.SaveChanges();
}
Consider to reduce the numbers of round trips to the database. This can be done by : they are memory-intensive
using Include() to preload your navigation properties; or
cashing the partial or whole table with ToDictionary() or ToLookup()
Instead of Add(), use AddRange() or even BulkInsert() from EntityFramework.BulkInsert if that fits your setup :
db.SurveyAnswers.AddRange(
TeamMembers.SelectMany(p =>
sectionDetails.SelectMany(s =>
attributeDetails.Where(a => a.SectionDetail.SurveySectionDetailID == s.SurveySectionDetailID)
.Select(a => new SurveyAnswer()
{
Reviewee = p,
SurveyFormID = form.SurveyFormID,
Detail = a,
SectionDetail = s,
}))));
Use Include to avoid SELECT N + 1 issue.
SurveyTemplate template = db.SurveyTemplates.Include("SurveyTemplateSections")
.Include("SurveyTemplateSections.SurveyTemplateAttributes")
.First(x=> x.SurveyTemplateID == SurveyTemplateID);
Generate the whole object graph and then save to DB.
List<SurveySectionDetail> sectionDetails = new List<SurveySectionDetail>();
List<SurveyAttributeDetail> attributeDetails = new List<SurveyAttributeDetail>();
foreach (SurveyTemplateSection section in template.SurveyTemplateSections)
{
SurveySectionDetail SectionDetail = new SurveySectionDetail();
//Some code
sectionDetails.Add(SectionDetail);
foreach (SurveyTemplateAttribute attribute in section.SurveyTemplateAttributes)
{
SurveyAttributeDetail AttributeDetail = new SurveyAttributeDetail();
//some code
attributeDetails.Add(AttributeDetail);
}
}
db.SurveySectionDetails.AddRange(sectionDetails);
db.SurveyAttributeDetails.AddRange(attributeDetails);
db.SaveChanges();
Load all employees you want before the loop, this will avoids database query for every team member.
var teamMemberIds = db.TeamMembers.Where(m => m.TeamID == TeamID && m.OnTeam)
.Select(x=>x.TeamMemberId).ToList();
var employees = db.Employees.Where(x => teamMemberIds.Contains(x.EmployeeId));
create a dictionary for attributeDetails based on their sectionDetailId to avoid query the list on every iteration.
var attributeDetailsGroupBySection = attributeDetails.GroupBy(x => x.SectionDetailId)
.ToDictionary(x => x.Key, x => x);
Move saving of SurveyAnswers and SurveyForms to outside of the loops:
List<SurveyForm> forms = new List<SurveyForm>();
List<SurveyAnswer> answers = new List<SurveyAnswer>();
foreach (int teamMemberId in teamMemberIds)
{
var employee = employees.First(x => x.Id == teamMemberId);
SurveyForm form = new SurveyForm();
//some code
forms.Add(form);
foreach (int peer in teamMemberIds)
{
foreach (SurveySectionDetail SectionDetail in sectionDetails)
{
foreach (SurveyAttributeDetail AttributeDetail in
attributeDetailsGroupBySection[SectionDetail.Id])
{
SurveyAnswer answer = new SurveyAnswer();
//some code
answers.Add(answer);
}
}
}
}
db.SurveyAnswers.AddRange(answers);
db.SurveyForms.AddRange(forms);
db.SaveChanges();
Finally if you want faster insertions you can use EntityFramework.BulkInsert. With this extension, you can save the data like this:
db.BulkInsert(answers);
db.BulkInsert(forms);
I have this query, it selects A LOT of user records from a table. This code block takes 16 seconds from my local/debug machine (more like 5 in production). Anything I do to make this more efficient doubles the amount of time the method takes to return the results. Examples of other things I've tried are below. I don't understand how selecting an anonymous type and having the extra middle section iterating through the anonymous type can possibly be faster than without.
This block takes 16 seconds:
List<BoAssetSecurityUser> userList = new List<BoAssetSecurityUser>();
using (var context = DataObjectFactory.CreateContext())
{
var query = from ui in context.User_Information
where (ui.AssetCustomerID == 1 &&
(ui.GlobalID != "1TPTEMPUSER" ||
ui.GlobalID == null))
select new { ui };
var result =
from q in query
select new
{
UserId = q.ui.UserID,
FirstName = q.ui.FirstName,
LastName = q.ui.LastName,
UserName = q.ui.Username,
Globalid = q.ui.GlobalID
};
foreach (var user in result)
{
BoAssetSecurityUser boAssetSecUser = new BoAssetSecurityUser();
boAssetSecUser.UserId = user.UserId;
boAssetSecUser.FirstName = user.FirstName;
boAssetSecUser.LastName = user.LastName;
boAssetSecUser.UserName = user.UserName;
boAssetSecUser.GlobalId = user.Globalid;
userList.Add(boAssetSecUser);
}
}
return userList;
This takes over 45 seconds to complete:
List<BoAssetSecurityUser> userList = new List<BoAssetSecurityUser>();
using (var context = DataObjectFactory.CreateContext())
{
var query = (from ui in context.User_Information
where (ui.AssetCustomerID == 1 &&
(ui.GlobalID != "1TPTEMPUSER" ||
ui.GlobalID == null))
select ui).ToList();
foreach (var user in query)
{
BoAssetSecurityUser boAssetSecUser = new BoAssetSecurityUser();
boAssetSecUser.UserId = user.UserID;
boAssetSecUser.FirstName = user.FirstName;
boAssetSecUser.LastName = user.LastName;
boAssetSecUser.UserName = user.Username;
boAssetSecUser.GlobalId = user.GlobalID;
userList.Add(boAssetSecUser);
}
}
return userList;
This example also takes over 45 seconds to complete:
List<BoAssetSecurityUser> userList = new List<BoAssetSecurityUser>();
using (var context = DataObjectFactory.CreateContext())
{
var query = from ui in context.User_Information
where (ui.AssetCustomerID == 1 &&
(ui.GlobalID != "1TPTEMPUSER" ||
ui.GlobalID == null))
select new { ui };
foreach (var user in query)
{
BoAssetSecurityUser boAssetSecUser = new BoAssetSecurityUser();
boAssetSecUser.UserId = user.ui.UserID;
boAssetSecUser.FirstName = user.ui.FirstName;
boAssetSecUser.LastName = user.ui.LastName;
boAssetSecUser.UserName = user.ui.Username;
boAssetSecUser.GlobalId = user.ui.GlobalID;
userList.Add(boAssetSecUser);
}
}
return userList;
This is most likely because ui's type has more properties than the 5 you're interested in. The new { ui } anonymous type is unnecessary; your first example is faster because you tell it before you iterate the list (and thus go to the DB) that you're only interested in those 5 fields. In the other examples, you iterate the list, thus pulling the whole ui objects, even though you only use 5 of its properties.
This code should only pull the 5 properties, and so be as fast as your first example, while being more concise:
List<BoAssetSecurityUser> userList = new List<BoAssetSecurityUser>();
using (var context = DataObjectFactory.CreateContext())
{
var query = from ui in context.User_Information
where (ui.AssetCustomerID == 1 && (ui.GlobalID != "1TPTEMPUSER" || ui.GlobalID == null))
select new
{
ui.UserID,
ui.FirstName,
ui.LastName,
ui.Username,
ui.GlobalID
};
foreach (var user in query)
{
BoAssetSecurityUser boAssetSecUser = new BoAssetSecurityUser();
boAssetSecUser.UserId = user.UserID;
boAssetSecUser.FirstName = user.FirstName;
boAssetSecUser.LastName = user.LastName;
boAssetSecUser.UserName = user.Username;
boAssetSecUser.GlobalId = user.GlobalID;
userList.Add(boAssetSecUser);
}
}
return userList;
It's more about the amount of data your bringing from the database. The first query selects only a few columns while the others bring all of them.
Do you have large columns on this table?
It is faster because you only fetch 5 properties per line with your anonymous type. I don't now how many fields you have in User_Information, but they're all fetched when you use .ToList() on your query, you probably get much more data than needed.
I have this simple foreach loop of an anonymous type and I'm wondering if there is a way to make it more efficient.
If it loops through 155 items, it takes about 20 seconds to complete. I've omitted some of the other properties it's setting on the boAsset object, but nothing special - just Strings/Integers.
Any thoughts?
List<BoAsset> assetList = new List<BoAsset>();
foreach (var asset in result)
{
BoAsset boAsset = new BoAsset();
boAsset.Description = asset.Description;
boAsset.DetailedDescription = asset.DetailedDescription;
boAsset.AssetCustomerID = asset.AssetCustomerID;
boAsset.AssetId = asset.AssetId;
boAsset.Keywords = asset.Keywords;
boAsset.Notes = asset.Notes;
boAsset.Photographer = asset.Photographer;
boAsset.PhotographerEmail = asset.PhotographerEmail;
boAsset.Notes = asset.Notes;
boAsset.Author = asset.Author;
boAsset.FileName = asset.FileName;
boAsset.FileExtension = asset.FileExtension;
boAsset.AssetCreateDate = asset.AssetCreateDate;
boAsset.AssetExpireDate = asset.AssetExpireDate;
assetList.Add(boAsset);
}
var query = (from a in context.Assets
join subAf1 in context.AssetFiles on new { aid = a.AssetID, ftid = 1 } equals new { aid = subAf1.AssetID, ftid = subAf1.AssetTypeID } into theAf1
from Af1 in theAf1.DefaultIfEmpty()
join subAf2 in context.AssetFiles on new { aid = a.AssetID, ftid = 2 } equals new { aid = subAf2.AssetID, ftid = subAf2.AssetTypeID } into theAf2
from Af2 in theAf2.DefaultIfEmpty()
join subAf3 in context.AssetFiles on new { aid = a.AssetID, ftid = 3 } equals new { aid = subAf3.AssetID, ftid = subAf3.AssetTypeID } into theAf3
from Af3 in theAf3.DefaultIfEmpty()
join subAf4 in context.AssetFiles on new { aid = a.AssetID, ftid = 4 } equals new { aid = subAf4.AssetID, ftid = subAf4.AssetTypeID } into theAf4
from Af4 in theAf4.DefaultIfEmpty()
join subAf5 in context.AssetFiles on new { aid = a.AssetID, ftid = 5 } equals new { aid = subAf5.AssetID, ftid = subAf5.AssetTypeID } into theAf5
from Af5 in theAf5.DefaultIfEmpty()
join subFp in context.FilePaths on new { fpid = Af1.FilePathID } equals new { fpid = subFp.FilePathID } into theFp1
from fp1 in theFp1.DefaultIfEmpty()
//join fp in context.FilePaths on Af1.FilePathID equals fp.FilePathID
where a.AssetCustomerID == custId && a.AssetID == assetId
select new { a, Af1, Af2, Af3, Af4, Af5, fp1 }).Distinct();
var result = from q in query
select new
{
AssetId = q.a.AssetID,
AssetCustomerId = q.a.AssetCustomerID,
StockImage = q.a.StockImage,
Description = q.a.Description,
DetailedDescription = q.a.DetailedDescription,
Author = q.a.Author,
FileName = q.Af1.FileName, //was 1
FileExtension = q.Af1.FileExtension, //was 1
AssetCreateDate = q.a.AssetCreateDate,
AssetExpireDate = q.a.AssetExpireDate,
AssetActivateDate = q.a.AssetActivateDate,
Notes = q.a.Notes,
Keywords = q.a.Keywords,
Photographer = q.a.Photographer,
PhotographerEmail = q.a.PhotographerEmail
}
The time for creating 155 objects and copying the data into them would be something like a millisecond. There is nothing that you could do to that code to make any significant (or even noticable) improvement.
If you want to save any time, you should look at the query that creates the result. That's what's taking time.
I think that your query is far too complex (too many joins), especially given that there are a few that are not even used.
Note that some LINQ providers (such as LINQ to Entities) can take quite a bit of time in simply processing a query tree to transform it into SQL, if the query is complex. (I have already had some queries take 10 or 15 seconds just for EF to analyze it.)
If, as I suspect, the LINQ implementation that you are using is hitting a database, then use a SQL profiling tool in order to check what actual SQL is being passed to the server, and how performant that SQL is.
Also, note that given the fields you are using, you could actually do away with the foreach loop by materializing results directly into the BoAsset objects.
That is:
var result = from q in query
select new BoAsset
{
AssetId = q.a.AssetID,
AssetCustomerId = q.a.AssetCustomerID,
StockImage = q.a.StockImage,
Description = q.a.Description,
DetailedDescription = q.a.DetailedDescription,
Author = q.a.Author,
FileName = q.Af1.FileName, //was 1
FileExtension = q.Af1.FileExtension, //was 1
AssetCreateDate = q.a.AssetCreateDate,
AssetExpireDate = q.a.AssetExpireDate,
AssetActivateDate = q.a.AssetActivateDate,
Notes = q.a.Notes,
Keywords = q.a.Keywords,
Photographer = q.a.Photographer,
PhotographerEmail = q.a.PhotographerEmail
};
List<BoAsset> assetList = result.ToList();