I've got a search box that I'm providing autocomplete suggestions for but it's really slow, it takes multiple seconds for suggestions to appear. I'm pretty sure my code is inefficient but I'm not sure the best way to improve it, any suggestions?
[HttpPost]
[Route("search")]
public virtual JsonResult Search(string term)
{
var result = new List<SearchResult>();
if (!String.IsNullOrWhiteSpace(term))
{
var searchTerms = term.ToLower().Split(' ');
List<Card> resultList = null;
foreach (var query in searchTerms)
{
if (resultList == null)
{
resultList = CardRepository.FindAll().Where(x => x.Name.ToLower().Contains(query) || x.Set.SetName.ToLower().Contains(query) || x.Variant.ToLower().Contains(query)
|| x.CardNumber.ToLower().Contains(query) || (query == "holo" && x.IsHolo)).ToList();
}
else
{
resultList = resultList.Where(x => x.Name.ToLower().Contains(query) || x.Set.SetName.ToLower().Contains(query) || x.Variant.ToLower().Contains(query)
|| x.CardNumber.ToLower().Contains(query) || (query == "holo" && x.IsHolo)).ToList();
}
}
foreach (var item in resultList.Take(10))
{
result.Add(new SearchResult()
{
label = item.FullCardName,
value = item.CardId.ToString()
});
}
}
return Json(result);
}
EDIT: Added the FindAll() code.
private readonly IDatabase _database;
public IQueryable<Card> FindAll()
{
return _database.CardDataSource.OrderBy(a => a.Name).AsQueryable();
}
SOLUTION: Going on the advice from the comments and with reference to this post Full Text Search with LINQ I moved my searching to the repository as a method and the result is almost instant autocomplete suggestions. I'm not sure how much better I could make the performance but it's easily usable in its current state.
public Card[] Search(string[] searchTerms)
{
IQueryable<Card> cardQuery = _database.CardDataSource;
foreach(var term in searchTerms)
{
var currentTerm = term.Trim();
cardQuery = cardQuery.Where(p => (p.Name.Contains(currentTerm) ||
p.Variant.Contains(currentTerm) ||
p.CardNumber.Contains(currentTerm) ||
p.Set.SetName.Contains(currentTerm) ||
(term == "holo" && p.IsHolo) ||
(term == "reverse" && p.IsHolo))
);
}
return cardQuery.Take(10).ToArray();
}
[HttpPost]
[Route("search")]
public virtual JsonResult Search(string term)
{
var result = new List<SearchResult>();
if (!String.IsNullOrWhiteSpace(term))
{
var searchTerms = term.ToLower().Split(' ');
var resultList = CardRepository.Search(searchTerms);
foreach (var item in resultList)
{
result.Add(new SearchResult()
{
label = item.FullCardName,
value = item.CardId.ToString()
});
}
}
return Json(result);
}
I think that the main problem is that you're using .FindAll() which returns a List<T>.
This means that when you say CardRepository.FindAll() it gets all of the records into an in-memory list and then your subsequent refining queries (e.g. Where(x => x.Name.ToLower().Contains(query)) and so on) are all run against the entire list. So it's right that it's returning really slowly.
You could try rewriting it by simply removing the .FindAll() and see what happens.
Please note, I'm just giving you the main problem, there are other issues, but none is as important as this one.
You could use multi-threading like this (pseudo-C# code):
var allCards = CardRepository.FindAll().ToArray(); // Ensure array.
query = query.ToUpper();
var nameTask = Task.StartNew(() => allCards.Where(x => x.Name.ToUpper().Contains(query)).ToArray());
var setTask = Task.StartNew(() => allCards.Where(x => x.Set.SetName.ToUpper().Contains(query)).ToArray());
var variantTask = Task.StartNew(() => allCards.Where(x => x.Variant.ToUpper().Contains(query)).ToArray());
var cardNumberTask = Task.StartNew(() => allCards.Where(x => x.CardNumber.ToUpper().Contains(query)).ToArray());
var holoTask = Task.StartNew(() => allCards.Where(x => query == "holo" && x.IsHolo).ToArray());
Task.WaitAll(new Task[] {nameTask, setTask, variantTask, cardNumberTask, holoTask});
var result = (nameTask.Result + setTask.Result + variantTask.Result + cardNumberTask.Result + halaTask.Result).Distinct().ToArray();
Related
I have this method with a linq statement below. I'm not a fan of multiple if statement and I'm trying to find what is the best way to not have these if statement and have a private method.
My field values is being set as such:
var fieldValues = await GetFields // then it's being passed to my method.
public static AppraisalContactBorrower BuildCoBorrower(List<LoanFieldValue> fieldValues) {
var coborrower = new AppraisalContactBorrower();
if (fieldValues.Exists(f => f.FieldId == "CX.OS.AO.COBORRNAME")) {
coborrower.Name = fieldValues.First(v => v.FieldId == "CX.OS.AO.COBORRNAME").Value;
}
if (fieldValues.Exists(f => f.FieldId == "CX.OS.AO.BORRCONTACTZIP")) {
borrower.Zip = fieldValues.First(v => v.FieldId == "CX.OS.AO.BORRCONTACTZIP").Value;
}
if (fieldValues.Exists(f => f.FieldId == "CX.OS.AO.BORRCONTACTZIP")) {
borrower.Zip = fieldValues.First(v => v.FieldId == "CX.OS.AO.BORRCONTACTZIP").Value;
}
What I'm trying to do is instead of this:
coborrower.Name = fieldValues.First(v => v.FieldId == "CX.OS.AO.COBORRNAME").Value;
Is having something similar to this.
if (fieldValues.Exists(f => f.FieldId == "CX.OS.AO.BORRCONTACTZIP")) {
coborrower.Name = SETVALUE("CX.OS.AO.BORRCONTACTZIP")}
First, try using Enumerable.ToDictionary to have the field values grouped by FieldId, then use IDictionary.TryGetValue to get the existing values:
public static AppraisalContactBorrower BuildCoBorrower(List<LoanFieldValue> fieldValues) {
var groupedFieldValues = fieldValues.ToDictionary(f => f.FieldId)
var coborrower = new AppraisalContactBorrower();
if (groupedFieldValues.TryGetValue("CX.OS.AO.COBORRNAME", out var name)) {
coborrower.Name = name.Value;
}
if (groupedFieldValues.TryGetValue("CX.OS.AO.BORRCONTACTZIP", out var zip)) {
borrower.Zip = zip.Value;
}
}
Using Dictionary makes it faster to check the appropriate field existence as it is O(1) and with TryGetValue you combine two operations into one (existence check + obtaining the value).
Your two last statements are almost identitical. The equivalent of :
if (groupedFieldValues.TryGetValue("CX.OS.AO.COBORRNAME", out var name)) {
coborrower.Name = name.Value;
}
is:
coborrower.Name = fieldValues.FirstOrDefault(v => v.FieldId == "CX.OS.AO.COBORRNAME")
?? coborrower.Name;
In the original code, coborrower.Name is not updated if the field doesn't exist in the list.
One of my function in C# required me to filter some value.So, I try it by using a lot of loop in it. It works but doesn't look like effecient.Any idea on how to convert code below to LinQ?
Any help is appreciated.
var object1 = JsonConvert.DeserializeObject<List<string>>(object1json);
foreach (var item1 in table1)
{
if (item1.Code == InputCode)
{
for (int i = 0; i < object1.Count(); i++)
{
tempData temp = new tempData();
foreach (var item2 in item1.List)
{
if (item2.Code == object1[i])
{
temp.Code = item2.Code;
temp.Description = item2.Description;
}
}
if(temp.Code != null || temp.Description != null)
final.Add(temp);
}
}
}
If you want your code to be more efficient, as pointed out in the comments, converting it to Linq isn't really going to help. It's still the same logic, just written in a different way. If you're going for readability, it can be improved with just a few changes:
foreach (var item1 in table1.Where(i => i.Code == InputCode))
{
foreach (var code in object)
{
// This could be SingleOrDefault, I don't know if you have duplicates in the list or not
var item2 = item1.List.LastOrDefault(i => i.Code == code);
if(item2 != null)
{
final.Add(new tempData
{
Code = item2.Code,
Description = item2.Description,
});
}
}
}
If you convert the whole thing to Linq:
var final = table1.Where(i => i.Code == InputCode)
.SelectMany(item1 => object.Select(code => item1.List.LastOrDefault(i => i.Code == code))
.Where(item2 => item2 != null)
.Select(item2 => new tempData
{
Code = item2.Code,
Description = item2.Description,
})
.ToList();
Personally, I prefer the first option, as it's a bit easier to read.
I guess what you post is sample code instead of actual code otherwise it would be better to avoid keyword like object in C#. Anyway,
var final = table1.Where(item1 => item1.Code == InputCode)
.SelectMany(item1 => item1.List)
.Where(item2 => #object.Contains(item2.Code))
.Where(temp => temp.Code != null || temp.Description != null)
.Select(item2 => new tempData()
{
Code = item2.Code,
Description = item2.Description
});
I created an extension method to encapsule some where logic like this (this is a very simplified version):
public static IQueryable<Cargo> ReadyToCarry(this IQueryable<Cargo> q)
{
VehicleType[] dontNeedCouple = new VehicleType[] { VehicleType.Sprinter, VehicleType.Van, VehicleType.Truck };
return q.Where(c => c.DriverId > 0 && c.VehicleId > 0)
.Where(c => c.CoupleId > 0 || dontNeedCouple.Contains(c.Vehicle.Type));
}
So I can use it like this:
using (var ctx = new MyNiceContext())
{
var readyCargo = ctx.Cargos.ReadyToCarry().OrderBy(c => c.Id).ToList();
// ...more code
}
Which works nicely, this is translated to SQL and executed by Entity Framework. Now, I have another place I need cargos which are not ready to carry, which means I need exactly the opposite.
My idea was something like this:
public static IQueryable<Cargo> NotReadyToCarry(this IQueryable<Cargo> q)
{
return !q.ReadyToCarry(); // ofc this doesn't work...
}
using (var ctx = new MyNiceContext())
{
var readyCargo = ctx.Cargos.NotReadyToCarry().OrderBy(c => c.Id).ToList();
// OR maybe
var readyCargo = ctx.Cargos.ReadyToCarry(false).OrderBy(c => c.Id).ToList(); // somehow use that bool param to reverse the logic when false
}
I didn't want to recreate the reverse logic from scratch, so if I needed to change it one day, I'd change in one unique place.
I'm accepting alternatives to this approach, since it's a new project.
You can use Except() method:
var readyCargo = ctx.Cargos.ReadyToCarry().OrderBy(c => c.Id);
var notReadyCargo = ctx.Cargos.Except(readyCargo);
OR
you can add some parameter to ReadyToCarry():
public static IQueryable<Cargo> ReadyToCarry(this IQueryable<Cargo> q, bool ready = true)
{
VehicleType[] dontNeedCouple = new VehicleType[] { VehicleType.Sprinter, VehicleType.Van, VehicleType.Truck };
if (ready)
{
return q.Where(c => c.DriverId > 0 && c.VehicleId > 0)
.Where(c => c.CoupleId > 0 || dontNeedCouple.Contains(c.Vehicle.Type));
}
else
{
// logic to get not ready for carrying
}
}
OR
you can combine these two options:
public static IQueryable<Cargo> ReadyToCarry(this IQueryable<Cargo> q, bool ready = true)
{
VehicleType[] dontNeedCouple = new VehicleType[] { VehicleType.Sprinter, VehicleType.Van, VehicleType.Truck };
var readyToCarry = q.Where(c => c.DriverId > 0 && c.VehicleId > 0)
.Where(c => c.CoupleId > 0 || dontNeedCouple.Contains(c.Vehicle.Type));
if (ready)
{
return readyToCarry;
}
else
{
return q.Except(readyToCarry);
}
}
In the last case when you change logic to get ready to carry entities you don't need to change negation of that condition. You should change only one query.
I'm working on a course listing in C# and an course can have up to 5 dates of when they are running. Ideally, the next date after today in the future would be selected, and ordered accordingly in a list.
What i have so far is a course list that gets the next date, and displays it, but it displays all the events without dates first (Null/Blank). I'm trying to show the courses with next dates first, and then those without after this.
C# Code:
public ActionResult FilterList(string role = null, string category = null)
{
return View("~/Views/FilterList.cshtml", GetCourses(role, category));
}
[NonAction]
public List<IEnumerable<Course>> GetCourses(string role = null, string category = null)
{
var collection = new List<IEnumerable<Course>>();
var items = Sitecore.Context.Database.GetItem(SitecoreIDs.Pages.CourseRoot)
.Children.Where(m => m.TemplateID == Course.TemplateID)
.Select(m => (Course)m).ToList();
var dates = new List<FilterDates>();
items.ForEach(m => dates.Add(new FilterDates
{
Dates = new List<DateTime>{ m.Date1, m.Date2, m.Date3, m.Date4, m.Date5 },
Name = m.Name
}));
dates.ForEach(m => m.Dates.RemoveAll(n => n == new DateTime(0001, 01, 01)));
dates.ForEach(m => m.Dates.Sort((a, b) => a.CompareTo(b)));
dates = dates.OrderBy(m => m.Dates.AsQueryable().FirstOrDefault(n => n - DateTime.Now >= TimeSpan.Zero)).ToList();
var model = new List<Course>();
dates.ForEach(m => model.Add(items.AsQueryable().FirstOrDefault(n => n.Name == m.Name)));
if (!string.IsNullOrEmpty(role) || !string.IsNullOrEmpty(category))
{
var currentRole = Sitecore.Context.Database.GetItem(SitecoreIDs.Pages.CategoryRoot)
.Children.AsQueryable().FirstOrDefault(m => m.Fields["Key"].Value == role);
if (!string.IsNullOrEmpty(category))
{
var currentCategory = Sitecore.Context.Database.GetItem(SitecoreIDs.Pages.SeriesRoot)
.Children.AsQueryable().FirstOrDefault(m => m.Fields["Key"].Value == category);
model = model.Where(m => m.Series == currentCategory.Name).ToList();
if (string.IsNullOrEmpty(role))
{
collection.Add(model);
}
}
if (!string.IsNullOrEmpty(role))
{
model = model.Where(m => m.InnerItem.Children.Where(n => n.Fields["Key"].Value == currentRole.Name).Any()).ToList();
List<Course> required = new List<Course>(), recommended = new List<Course>(), refresh = new List<Course>();
foreach (var item in model)
{
foreach (Item inner in item.InnerItem.Children)
{
if (inner.Fields["Key"].Value == currentRole.Name)
{
switch (inner.Fields["Severity"].Value)
{
case "Required":
required.Add(item);
break;
case "Recommended":
recommended.Add(item);
break;
case "Refresh":
refresh.Add(item);
break;
}
}
}
}
collection.Add(required);
collection.Add(recommended);
collection.Add(refresh);
}
}
else
{
collection.Add(model);
}
return collection;
}
I've tried different orderbys, but can't seem to get the ordering right. Any help would be greatly appreciated.
Andy
The code you posted has some extra stuff that seems unrelated to your question about sorting. I am ignoring that and just addressing the question at hand: how to sort your courses so that the ones with the nearest future date are first.
I would create a little method to return the next future date or DateTime.MaxValue as the "null" value.
private DateTime GetNextFutureDate(Course course)
{
var dates =
new[] {course.Date1, course.Date2, course.Date3, course.Date4, course.Date5}.Where(d => d > DateTime.Now).ToArray();
return dates.Length == 0 ? DateTime.MaxValue : dates[0];
}
Then in your GetCourses method you could use it like this:
[NonAction]
public List<IEnumerable<Course>> GetCourses(string role = null, string category = null)
{
var collection = new List<IEnumerable<Course>>();
var model = Sitecore.Context.Database.GetItem(SitecoreIDs.Pages.CourseRoot)
.Children.Where(m => m.TemplateID == Course.TemplateID)
.Select(m => (Course)m).OrderBy(m => GetNextFutureDate(m));
if (!string.IsNullOrEmpty(role) || !string.IsNullOrEmpty(category))
// ... the rest of your code ...
return collection;
}
You might also want to consider making GetNextFutureDate a member or extension method on your Course class.
I'm optimizing a method with a number of Linq queries. So far the execution time is around 3 seconds and I'm trying to reduce it. There is quite a lot of operations and calculations happening in the method, but nothing too complex.
I will appreciate any suggections and ideas how the performance can be improved and code optimized.
The whole code of the method(Below I'll point where I have the biggest delay):
public ActionResult DataRead([DataSourceRequest] DataSourceRequest request)
{
CTX.Configuration.AutoDetectChangesEnabled = false;
var repoKomfortaktion = new KomfortaktionRepository();
var komfortaktionen = CTX.Komfortaktionen.ToList();
var result = new List<AqGeplantViewModel>();
var gruppen = new HashSet<Guid?>(komfortaktionen.Select(c => c.KomfortaktionsGruppeId).ToList());
var hochgeladeneKomplettabzuege = CTX.Komplettabzug.Where(c => gruppen.Contains(c.KomfortaktionsGruppeId)).GroupBy(c => new { c.BetriebId, c.KomfortaktionsGruppeId }).Select(x => new { data = x.Key }).ToList();
var teilnehmendeBetriebe = repoKomfortaktion.GetTeilnehmendeBetriebe(CTX, gruppen);
var hochgeladeneSperrlistenPlz = CTX.SperrlistePlz.Where(c => gruppen.Contains(c.KomfortaktionsGruppeId) && c.AktionsKuerzel != null)
.GroupBy(c => new { c.AktionsKuerzel, c.BetriebId, c.KomfortaktionsGruppeId }).Select(x => new { data = x.Key }).ToList();
var hochgeladeneSperrlistenKdnr = CTX.SperrlisteKdnr.Where(c => gruppen.Contains(c.KomfortaktionsGruppeId) && c.AktionsKuerzel != null)
.GroupBy(c => new { c.AktionsKuerzel, c.BetriebId, c.KomfortaktionsGruppeId }).Select(x => new { data = x.Key }).ToList();
var konfigsProAktion = CTX.Order.GroupBy(c => new { c.Vfnr, c.AktionsId }).Select(c => new { count = c.Count(), c.Key.AktionsId, data = c.Key }).ToList();
foreach (var komfortaktion in komfortaktionen)
{
var item = new AqGeplantViewModel();
var zentraleTeilnehmer = teilnehmendeBetriebe.Where(c => c.TeilnahmeStatus.Any(x => x.KomfortaktionId == komfortaktion.Id && x.AktionsTypeId == 1)).ToList();
var lokaleTeilnehmer = teilnehmendeBetriebe.Where(c => c.TeilnahmeStatus.Any(x => x.KomfortaktionId == komfortaktion.Id && x.AktionsTypeId == 2)).ToList();
var hochgeladeneSperrlistenGesamt =
hochgeladeneSperrlistenPlz.Count(c => c.data.AktionsKuerzel == komfortaktion.Kuerzel && c.data.KomfortaktionsGruppeId == komfortaktion.KomfortaktionsGruppeId) +
hochgeladeneSperrlistenKdnr.Count(c => c.data.AktionsKuerzel == komfortaktion.Kuerzel && c.data.KomfortaktionsGruppeId == komfortaktion.KomfortaktionsGruppeId);
item.KomfortaktionId = komfortaktion.KomfortaktionId;
item.KomfortaktionName = komfortaktion.Aktionsname;
item.Start = komfortaktion.KomfortaktionsGruppe.StartAdressQualifizierung.HasValue ? komfortaktion.KomfortaktionsGruppe.StartAdressQualifizierung.Value.ToString("dd.MM.yyyy") : string.Empty;
item.LokalAngemeldet = lokaleTeilnehmer.Count();
item.ZentralAngemeldet = zentraleTeilnehmer.Count();
var anzHochgelandenerKomplettabzuege = hochgeladeneKomplettabzuege.Count(c => zentraleTeilnehmer.Count(x => x.BetriebId == c.data.BetriebId) == 1) +
hochgeladeneKomplettabzuege.Count(c => lokaleTeilnehmer.Count(x => x.BetriebId == c.data.BetriebId) == 1);
item.KomplettabzugOffen = (zentraleTeilnehmer.Count() + lokaleTeilnehmer.Count()) - anzHochgelandenerKomplettabzuege;
item.SperrlisteOffen = (zentraleTeilnehmer.Count() + lokaleTeilnehmer.Count()) - hochgeladeneSperrlistenGesamt;
item.KonfigurationOffen = zentraleTeilnehmer.Count() - konfigsProAktion.Count(c => c.AktionsId == komfortaktion.KomfortaktionId && zentraleTeilnehmer.Any(x => x.Betrieb.Vfnr == c.data.Vfnr));
item.KomfortaktionsGruppeId = komfortaktion.KomfortaktionsGruppeId;
result.Add(item);
}
return Json(result.ToDataSourceResult(request));
}
The first half (before foreach) takes half a second which is okay. The biggest delay is inside foreach statement in the first iteration and in particular in these lines, execution of zentraleTeilnehmer takes 1.5 second for the first time.
var zentraleTeilnehmer = teilnehmendeBetriebe.Where(c => c.TeilnahmeStatus.Any(x => x.KomfortaktionId == komfortaktion.Id && x.AktionsTypeId == 1)).ToList();
var lokaleTeilnehmer = teilnehmendeBetriebe.Where(c => c.TeilnahmeStatus.Any(x => x.KomfortaktionId == komfortaktion.Id && x.AktionsTypeId == 2)).ToList();
TeilnehmendeBetriebe has over 800 lines, where TeilnahmeStatus property has normally around 4 items. So, maximum 800*4 iterations, which is not a huge number afterall...
Thus, I'm mostly interected in optimizing these lines, hoping to reduce execution time to half a second or so.
What I tried:
Rewrite Linq to foreach: didn't help, same time... probably not surprising, but was worth a try.
foreach (var tb in teilnehmendeBetriebe) //836 items
{
foreach (var ts in tb.TeilnahmeStatus) //3377 items
{
if (ts.KomfortaktionId == komfortaktion.Id && ts.AktionsTypeId == 1)
{
testResult.Add(tb);
break;
}
}
}
Selecting particular columns for teilnehmendeBetriebe with .Select(). Didn't help either.
Neither helped other small manipulations I tried.
What is interesting - while the first iteration of foreach can take up to 2 seconds, the second and further take just milisecons, so .net is capable of optimizing or reusing calculation data.
Any advice on what can be changed in order to improve performance is very welcome!
Edit:
TeilnahmeBetriebKomfortaktion.TeilnahmeStatus is loaded eagerly in the method GetTeilnehmendeBetriebe:
public List<TeilnahmeBetriebKomfortaktion> GetTeilnehmendeBetriebe(Connection ctx, HashSet<Guid?> gruppen)
{
return ctx.TeilnahmeBetriebKomfortaktion.Include(
c => c.TeilnahmeStatus).ToList();
}
Edit2:
The query which is sent when executing GetTeilnehmendeBetriebe:
SELECT
[Extent1].[Id] AS [Id],
[Extent1].[BetriebId] AS [BetriebId],
[Extent1].[MandantenId] AS [MandantenId],
[Extent1].[CreatedUser] AS [CreatedUser],
[Extent1].[UpdatedUser] AS [UpdatedUser],
[Extent1].[CreatedDate] AS [CreatedDate],
[Extent1].[UpdatedDate] AS [UpdatedDate],
[Extent1].[IsDeleted] AS [IsDeleted]
FROM [Semas].[TeilnahmeBetriebKomfortaktion] AS [Extent1]
WHERE [Extent1].[IsDeleted] <> cast(1 as bit)
My assumption is that TeilnahmeBetriebKomfortaktion.TeilnahmeStatus is a lazy loaded collection, resulting in the N + 1 problem. You should eagerly fetch that collection to improve your performance.
The following iterations of the foreach loop are fast, because after the first iteration those objects are no longer requested from the database server but are server from memory.