I'm optimizing a method with a number of Linq queries. So far the execution time is around 3 seconds and I'm trying to reduce it. There is quite a lot of operations and calculations happening in the method, but nothing too complex.
I will appreciate any suggections and ideas how the performance can be improved and code optimized.
The whole code of the method(Below I'll point where I have the biggest delay):
public ActionResult DataRead([DataSourceRequest] DataSourceRequest request)
{
CTX.Configuration.AutoDetectChangesEnabled = false;
var repoKomfortaktion = new KomfortaktionRepository();
var komfortaktionen = CTX.Komfortaktionen.ToList();
var result = new List<AqGeplantViewModel>();
var gruppen = new HashSet<Guid?>(komfortaktionen.Select(c => c.KomfortaktionsGruppeId).ToList());
var hochgeladeneKomplettabzuege = CTX.Komplettabzug.Where(c => gruppen.Contains(c.KomfortaktionsGruppeId)).GroupBy(c => new { c.BetriebId, c.KomfortaktionsGruppeId }).Select(x => new { data = x.Key }).ToList();
var teilnehmendeBetriebe = repoKomfortaktion.GetTeilnehmendeBetriebe(CTX, gruppen);
var hochgeladeneSperrlistenPlz = CTX.SperrlistePlz.Where(c => gruppen.Contains(c.KomfortaktionsGruppeId) && c.AktionsKuerzel != null)
.GroupBy(c => new { c.AktionsKuerzel, c.BetriebId, c.KomfortaktionsGruppeId }).Select(x => new { data = x.Key }).ToList();
var hochgeladeneSperrlistenKdnr = CTX.SperrlisteKdnr.Where(c => gruppen.Contains(c.KomfortaktionsGruppeId) && c.AktionsKuerzel != null)
.GroupBy(c => new { c.AktionsKuerzel, c.BetriebId, c.KomfortaktionsGruppeId }).Select(x => new { data = x.Key }).ToList();
var konfigsProAktion = CTX.Order.GroupBy(c => new { c.Vfnr, c.AktionsId }).Select(c => new { count = c.Count(), c.Key.AktionsId, data = c.Key }).ToList();
foreach (var komfortaktion in komfortaktionen)
{
var item = new AqGeplantViewModel();
var zentraleTeilnehmer = teilnehmendeBetriebe.Where(c => c.TeilnahmeStatus.Any(x => x.KomfortaktionId == komfortaktion.Id && x.AktionsTypeId == 1)).ToList();
var lokaleTeilnehmer = teilnehmendeBetriebe.Where(c => c.TeilnahmeStatus.Any(x => x.KomfortaktionId == komfortaktion.Id && x.AktionsTypeId == 2)).ToList();
var hochgeladeneSperrlistenGesamt =
hochgeladeneSperrlistenPlz.Count(c => c.data.AktionsKuerzel == komfortaktion.Kuerzel && c.data.KomfortaktionsGruppeId == komfortaktion.KomfortaktionsGruppeId) +
hochgeladeneSperrlistenKdnr.Count(c => c.data.AktionsKuerzel == komfortaktion.Kuerzel && c.data.KomfortaktionsGruppeId == komfortaktion.KomfortaktionsGruppeId);
item.KomfortaktionId = komfortaktion.KomfortaktionId;
item.KomfortaktionName = komfortaktion.Aktionsname;
item.Start = komfortaktion.KomfortaktionsGruppe.StartAdressQualifizierung.HasValue ? komfortaktion.KomfortaktionsGruppe.StartAdressQualifizierung.Value.ToString("dd.MM.yyyy") : string.Empty;
item.LokalAngemeldet = lokaleTeilnehmer.Count();
item.ZentralAngemeldet = zentraleTeilnehmer.Count();
var anzHochgelandenerKomplettabzuege = hochgeladeneKomplettabzuege.Count(c => zentraleTeilnehmer.Count(x => x.BetriebId == c.data.BetriebId) == 1) +
hochgeladeneKomplettabzuege.Count(c => lokaleTeilnehmer.Count(x => x.BetriebId == c.data.BetriebId) == 1);
item.KomplettabzugOffen = (zentraleTeilnehmer.Count() + lokaleTeilnehmer.Count()) - anzHochgelandenerKomplettabzuege;
item.SperrlisteOffen = (zentraleTeilnehmer.Count() + lokaleTeilnehmer.Count()) - hochgeladeneSperrlistenGesamt;
item.KonfigurationOffen = zentraleTeilnehmer.Count() - konfigsProAktion.Count(c => c.AktionsId == komfortaktion.KomfortaktionId && zentraleTeilnehmer.Any(x => x.Betrieb.Vfnr == c.data.Vfnr));
item.KomfortaktionsGruppeId = komfortaktion.KomfortaktionsGruppeId;
result.Add(item);
}
return Json(result.ToDataSourceResult(request));
}
The first half (before foreach) takes half a second which is okay. The biggest delay is inside foreach statement in the first iteration and in particular in these lines, execution of zentraleTeilnehmer takes 1.5 second for the first time.
var zentraleTeilnehmer = teilnehmendeBetriebe.Where(c => c.TeilnahmeStatus.Any(x => x.KomfortaktionId == komfortaktion.Id && x.AktionsTypeId == 1)).ToList();
var lokaleTeilnehmer = teilnehmendeBetriebe.Where(c => c.TeilnahmeStatus.Any(x => x.KomfortaktionId == komfortaktion.Id && x.AktionsTypeId == 2)).ToList();
TeilnehmendeBetriebe has over 800 lines, where TeilnahmeStatus property has normally around 4 items. So, maximum 800*4 iterations, which is not a huge number afterall...
Thus, I'm mostly interected in optimizing these lines, hoping to reduce execution time to half a second or so.
What I tried:
Rewrite Linq to foreach: didn't help, same time... probably not surprising, but was worth a try.
foreach (var tb in teilnehmendeBetriebe) //836 items
{
foreach (var ts in tb.TeilnahmeStatus) //3377 items
{
if (ts.KomfortaktionId == komfortaktion.Id && ts.AktionsTypeId == 1)
{
testResult.Add(tb);
break;
}
}
}
Selecting particular columns for teilnehmendeBetriebe with .Select(). Didn't help either.
Neither helped other small manipulations I tried.
What is interesting - while the first iteration of foreach can take up to 2 seconds, the second and further take just milisecons, so .net is capable of optimizing or reusing calculation data.
Any advice on what can be changed in order to improve performance is very welcome!
Edit:
TeilnahmeBetriebKomfortaktion.TeilnahmeStatus is loaded eagerly in the method GetTeilnehmendeBetriebe:
public List<TeilnahmeBetriebKomfortaktion> GetTeilnehmendeBetriebe(Connection ctx, HashSet<Guid?> gruppen)
{
return ctx.TeilnahmeBetriebKomfortaktion.Include(
c => c.TeilnahmeStatus).ToList();
}
Edit2:
The query which is sent when executing GetTeilnehmendeBetriebe:
SELECT
[Extent1].[Id] AS [Id],
[Extent1].[BetriebId] AS [BetriebId],
[Extent1].[MandantenId] AS [MandantenId],
[Extent1].[CreatedUser] AS [CreatedUser],
[Extent1].[UpdatedUser] AS [UpdatedUser],
[Extent1].[CreatedDate] AS [CreatedDate],
[Extent1].[UpdatedDate] AS [UpdatedDate],
[Extent1].[IsDeleted] AS [IsDeleted]
FROM [Semas].[TeilnahmeBetriebKomfortaktion] AS [Extent1]
WHERE [Extent1].[IsDeleted] <> cast(1 as bit)
My assumption is that TeilnahmeBetriebKomfortaktion.TeilnahmeStatus is a lazy loaded collection, resulting in the N + 1 problem. You should eagerly fetch that collection to improve your performance.
The following iterations of the foreach loop are fast, because after the first iteration those objects are no longer requested from the database server but are server from memory.
Related
I've got a search box that I'm providing autocomplete suggestions for but it's really slow, it takes multiple seconds for suggestions to appear. I'm pretty sure my code is inefficient but I'm not sure the best way to improve it, any suggestions?
[HttpPost]
[Route("search")]
public virtual JsonResult Search(string term)
{
var result = new List<SearchResult>();
if (!String.IsNullOrWhiteSpace(term))
{
var searchTerms = term.ToLower().Split(' ');
List<Card> resultList = null;
foreach (var query in searchTerms)
{
if (resultList == null)
{
resultList = CardRepository.FindAll().Where(x => x.Name.ToLower().Contains(query) || x.Set.SetName.ToLower().Contains(query) || x.Variant.ToLower().Contains(query)
|| x.CardNumber.ToLower().Contains(query) || (query == "holo" && x.IsHolo)).ToList();
}
else
{
resultList = resultList.Where(x => x.Name.ToLower().Contains(query) || x.Set.SetName.ToLower().Contains(query) || x.Variant.ToLower().Contains(query)
|| x.CardNumber.ToLower().Contains(query) || (query == "holo" && x.IsHolo)).ToList();
}
}
foreach (var item in resultList.Take(10))
{
result.Add(new SearchResult()
{
label = item.FullCardName,
value = item.CardId.ToString()
});
}
}
return Json(result);
}
EDIT: Added the FindAll() code.
private readonly IDatabase _database;
public IQueryable<Card> FindAll()
{
return _database.CardDataSource.OrderBy(a => a.Name).AsQueryable();
}
SOLUTION: Going on the advice from the comments and with reference to this post Full Text Search with LINQ I moved my searching to the repository as a method and the result is almost instant autocomplete suggestions. I'm not sure how much better I could make the performance but it's easily usable in its current state.
public Card[] Search(string[] searchTerms)
{
IQueryable<Card> cardQuery = _database.CardDataSource;
foreach(var term in searchTerms)
{
var currentTerm = term.Trim();
cardQuery = cardQuery.Where(p => (p.Name.Contains(currentTerm) ||
p.Variant.Contains(currentTerm) ||
p.CardNumber.Contains(currentTerm) ||
p.Set.SetName.Contains(currentTerm) ||
(term == "holo" && p.IsHolo) ||
(term == "reverse" && p.IsHolo))
);
}
return cardQuery.Take(10).ToArray();
}
[HttpPost]
[Route("search")]
public virtual JsonResult Search(string term)
{
var result = new List<SearchResult>();
if (!String.IsNullOrWhiteSpace(term))
{
var searchTerms = term.ToLower().Split(' ');
var resultList = CardRepository.Search(searchTerms);
foreach (var item in resultList)
{
result.Add(new SearchResult()
{
label = item.FullCardName,
value = item.CardId.ToString()
});
}
}
return Json(result);
}
I think that the main problem is that you're using .FindAll() which returns a List<T>.
This means that when you say CardRepository.FindAll() it gets all of the records into an in-memory list and then your subsequent refining queries (e.g. Where(x => x.Name.ToLower().Contains(query)) and so on) are all run against the entire list. So it's right that it's returning really slowly.
You could try rewriting it by simply removing the .FindAll() and see what happens.
Please note, I'm just giving you the main problem, there are other issues, but none is as important as this one.
You could use multi-threading like this (pseudo-C# code):
var allCards = CardRepository.FindAll().ToArray(); // Ensure array.
query = query.ToUpper();
var nameTask = Task.StartNew(() => allCards.Where(x => x.Name.ToUpper().Contains(query)).ToArray());
var setTask = Task.StartNew(() => allCards.Where(x => x.Set.SetName.ToUpper().Contains(query)).ToArray());
var variantTask = Task.StartNew(() => allCards.Where(x => x.Variant.ToUpper().Contains(query)).ToArray());
var cardNumberTask = Task.StartNew(() => allCards.Where(x => x.CardNumber.ToUpper().Contains(query)).ToArray());
var holoTask = Task.StartNew(() => allCards.Where(x => query == "holo" && x.IsHolo).ToArray());
Task.WaitAll(new Task[] {nameTask, setTask, variantTask, cardNumberTask, holoTask});
var result = (nameTask.Result + setTask.Result + variantTask.Result + cardNumberTask.Result + halaTask.Result).Distinct().ToArray();
I'm with EF 6.
I have a list of currents item in the db, which I retrieve with:
ae_alignedPartners_olds = ctx.AE_AlignedPartners.AsNoTracking().ToList(); // list of List<AE_AlignedPartners>
Than, I retry the same objects from a JSON, with:
ae_alignedPartners_news = GetJSONListObjects(); // list of List<AE_AlignedPartners>
Than I'm making some comparisons (to see which I need to update, which I need to delete and which to create. That's the current code:
// intersection
var IDSIntersections = (from itemNew in ae_alignedPartners_news
join itemOld in ae_alignedPartners_olds on itemNew.ObjectID equals itemOld.ObjectID
select itemNew).Select(p => p.ObjectID).ToList();
// to update
IList<AE_AlignedPartners> ae_alignedPartners_toUpdate = new List<AE_AlignedPartners>();
foreach (var item in IDSIntersections)
{
var itemOld = ae_alignedPartners_olds.First(p => p.ObjectID == item);
var itemNew = ae_alignedPartners_news.First(p => p.ObjectID == item);
if (itemOld.Field1 != itemNew.Field1 ||
itemOld.Field2 != itemNew.Field2 ||
itemOld.Field3 != itemNew.Field3 ||
itemOld.Field4 != itemNew.Field4 ||
itemOld.Field5 != itemNew.Field5 ||
itemOld.Field6 != itemNew.Field6 ||
itemOld.Field7 != itemNew.Field7 ||
itemOld.Field8 != itemNew.Field8 ||
itemOld.Field9 != itemNew.Field9)
{
itemOld.Field1 = itemNew.Field1;
itemOld.Field2 = itemNew.Field2;
itemOld.Field3 = itemNew.Field3;
itemOld.Field4 = itemNew.Field4;
itemOld.Field5 = itemNew.Field5;
itemOld.Field6 = itemNew.Field6;
itemOld.Field7 = itemNew.Field7;
itemOld.Field8 = itemNew.Field8;
itemOld.Field9 = itemNew.Field9;
ae_alignedPartners_toUpdate.Add(itemOld);
}
}
// to create
IList<AE_AlignedPartners> ae_alignedPartners_toCreate = ae_alignedPartners_news.Where(p => !IDSIntersections.Contains(p.ObjectID)).ToList();
// to delete
IList<AE_AlignedPartners> ae_alignedPartners_toDelete = ae_alignedPartners_olds.Where(p => !IDSIntersections.Contains(p.ObjectID)).ToList();
Which is faster enough for 1000~ records. Over 50k, it becomes very very slow.
What do you suggest to improve the whole?
If you want to find out what's slow I suggest profiling or simply pausing the debugger 10 times to see where it stops most often (you can try that with your existing code). But here I could spot the problem immediately:
var itemOld = ae_alignedPartners_olds.First(p => p.ObjectID == item);
var itemNew = ae_alignedPartners_news.First(p => p.ObjectID == item);
This is scanning the entire list which is O(N). Together with the outer loop this becomes O(N^2).
The best solution would be to restructure your query so that these lookups are not necessary. It seems to me that the join already outputs the objects you need.
But you can also use a hash table to speed up the lookups.
var dict_ae_alignedPartners_olds = ae_alignedPartners_olds.ToDictionary(p => p.ObjectID);
var dict_ae_alignedPartners_news = ae_alignedPartners_news.ToDictionary(p => p.ObjectID);
foreach (var item in IDSIntersections)
{
var itemOld = dict_ae_alignedPartners_olds[item];
var itemNew = dict_ae_alignedPartners_news[item];
//...
}
Please help me resolve this issue. I still can not understand how it throw exception "system.linq.systemcore_enumerabledebugview evaluation time out" when debuging
var mealDataDetail = ( _dbContext.Meal.Where(m => m.IsDeleted == 0 && m.UserId == userId).ToList()
.GroupJoin(_dbContext.MealImage.Where(i => i.IsDeleted == 0 && i.MealType == (int)mealType).ToList(), p => p.Id, r => r.MealId, (p, rs) => new { p, rs })
.GroupJoin(_dbContext.MealMenu.Where(i => i.IsDeleted == 0 && i.MealType == (int)mealType).ToList(), prs => prs.p.Id, c => c.MealId, (prs, cs) => new MealDataDetail
{
MealId = prs.p.Id,
MealData = new MealData
{
MealImages = prs.rs.Select(im => im.Image),
Description = prs.p.BreakfastDescription,
Time = prs.p.BreakfastTime,
TimeRequired = prs.p.BreakfastTimeRequired,
Appitite = prs.p.BreakfastAppetite,
Status = prs.p.BreakfastStatus == 1,
Calorie = prs.p.BreakfastCalorie,
},
}));
Was it supposed to be a single query?
Keep in mind that your .ToList() is executing the query at that moment.
Try removing the ToList() but I suggest you rewrite it once the database is getting a really monster query. Is better to have a procedure instead
I'm trying to count the Total amount of Weight in a certain column.
I've tried the following coding, but I only seem to get the first row's value and not the rest.
int QuoteId = (from x in db.Quotes where x.Id != null orderby x.Id descending select x.Id).Take(1).SingleOrDefault();
var item = db.QuoteItems.Where(x => x.QuoteId == QuoteId).First();
QuoteItemSectionGroup quoteItemList = new QuoteItemSectionGroup();
foreach (var quoteItem in db.QuoteItemSectionGroups.Where(x => x.QuoteItemId == item.Id).ToList())
{
var total = new QuoteItemSectionGroup
{
Weight = quoteItem.Weight
};
quoteItemList.Weight = total.Weight;
}
So my question is: How can I count the total amount of the Weight column in my table?
You obviously want to add the current number to the Weigth you already obtained, don´t you? Furtheremore you won´t need to create a new instance of QuoteItemSectionGroup only for the sake of setting its Weight-property temporarily.
foreach (var quoteItem in db.QuoteItemSectionGroups.Where(x => x.QuoteItemId == item.Id).ToList())
{
quoteItemList.Weight += quoteItem.Weight; // pay attention on the + before the equality-character
}
The += operator in x += 1 is simply a shortcut for x = x + 1.
Or even simpler using Linq Sum-method
var totalWeight = db.QuoteItemSectionGroups
.Where(x => x.QuoteItemId == item.Id)
.Sum(x => x.Weight);
EDIT: Furthermore you can simplify your code a bit so it finally becomes this:
var item = db.Quotes.Where(x => x.Id != null)
.OrderByDescending(x => x.Id)
.FirstOrDefault();
var totalWeight = db.QuoteItemSectionGroups
.Where(x => x.QuoteItemId == item.Id)
.Sum(x => x.Weight);
I'm trying to run the following query but for some reason MemberTransactionCount and NonMemberTransactionCount are coming back as the exact same values. It seems that the .Where() clauses aren't working as we'd expect them to.
Hoping someone can point out where I might be going wrong.
from trans in transactions
orderby trans.TransactionDate.Year , trans.TransactionDate.Month
group trans by new {trans.TransactionDate.Year, trans.TransactionDate.Month}
into grp
select new MemberTransactions
{
Month = string.Format("{0}/{1}", grp.Key.Month, grp.Key.Year),
MemberTransactionCount =
grp.Where(x => x.Account.Id != Guid.Empty || x.CardNumber != null)
.Sum(x => x.AmountSpent),
NonMemberTransactionCount =
grp.Where(x => x.Account.Id == Guid.Empty && x.CardNumber == null)
.Sum(x => x.AmountSpent)
}
EDIT
I've verified in the database that the results are not what they should be. It seems to be adding everything together and not taking into account the Account criteria that we're looking at.
I ended up solving this with two separate queries. It's not exactly as I wanted, but it does the job and seems to just as quick as I would have hoped.
var memberTrans = from trans in transactions
where trans.Account != null
|| trans.CardNumber != null
orderby trans.TransactionDate.Month
group trans by trans.TransactionDate.Month
into grp
select new
{
Month = grp.Key,
Amount = grp.Sum(x => x.AmountSpent)
};
var nonMemberTrans = (from trans in transactions
where trans.Account == null
&& trans.CardNumber == null
group trans by trans.TransactionDate.Month
into grp
select new
{
Month = grp.Key,
Amount = grp.Sum(x => x.AmountSpent)
}).ToList();
var memberTransactions = new List<MemberTransactions>();
foreach (var trans in memberTrans)
{
var non = (from nt in nonMemberTrans
where nt.Month == trans.Month
select nt).FirstOrDefault();
var date = new DateTime(2012, trans.Month, 1);
memberTransactions.Add(new MemberTransactions
{
Month = date.ToString("MMM"),
MemberTransactionCount = trans.Amount,
NonMemberTransactionCount = non != null ? non.Amount : 0.00m
});
}
I think the main problem here is that you doubt the result, though it might be correct.
Add another property for verification:
TotalAmount = grp.Sum(x => x.AmountSpent)