Develop Reference

How to improve performance when joining List and Linq object

I have one list, read from file:
var lsData = ReadExcelFile<CustomerEntity>(path);
And one Object (loaded into memory):
lsCustomer = await CustomerService.GetAll()
.Where(c => c.isDeleted == null || !c.isDeleted.Value)
.OrderBy(c=> c.Code)
.ToListAsync();
And the join command:
var lsDuplicateEmail =
(from imp in lsData
join cust in lsCustomer
on ImportHelpers.GetPerfectStringWithoutSpace(imp.Email) equals ImportHelpers.GetPerfectStringWithoutSpace(cust.Email)
into gjoin
from g in gjoin.DefaultIfEmpty()
select new
{
ImportItem = imp,
CustomerItem = g,
}
into result
where !string.IsNullOrEmpty(result.ImportItem.Email) && result.CustomerItem != null
&& !ImportHelpers.CompareString(result.ImportItem.Code, result.CustomerItem.Code)
select result);
var lsDuplicateEmailInSystem = lsDuplicateEmail.Select(c => c.ImportItem.Code).Distinct().ToList();
I perform test with lsData list about 2000 records, lsCustomer about 200k records.
The Customer Email field is not indexed in the DB.
The join command executes with about 10s (even though the result is 0 records), too slow.
I've looked around and can't seem to index the email field in lsCustomer. I know the reason for the slowness is because the complexity is O(n*m).
Is there any way to improve performance?

Try the following code. Instead of GroupJoin, which is not needed here I have used Join. Also moved filters up in query.
var lsDuplicateEmail =
from imp in lsData
where !string.IsNullOrEmpty(imp.Email)
join cust in lsCustomer
on ImportHelpers.GetPerfectStringWithoutSpace(imp.Email) equals ImportHelpers.GetPerfectStringWithoutSpace(cust.Email)
where !ImportHelpers.CompareString(imp.Code, cust.Code)
select new
{
ImportItem = imp,
CustomerItem = cust,
};
Also show GetPerfectStringWithoutSpace implementation, maybe it is slow.
Another possible solution is to swap lsData and lsCustomer in query, maybe lookup search is not so fast.

Why does AsEnumerable not work when using FromSqlRaw

I'm currently creating a site that showcases all my patients within a data table and I have to use FromSqlRaw in order to get the data from my database. I have a search funtion that allows me to search the patients within the table but upon entering the page I get this error when I use AsQueryable and no data is displayed in the table. It recommends me to use AsEnumerable but when I do I get an intellisense error. Any ideas on how to fix?
public async Task<IActionResult> Search(StaySearchViewModel model)
{
if (model.Cleared)
{
return Json(new
{
draw = model.Draw,
data = new object[] { },
recordsFiltered = 0,
recordsTotal = 0,
total = 0
});
}
var records = getSearchData(model);
//var records = System.Linq.Enumerable.AsEnumerable(getSearchData(model)); // Hard coding this an enumerable will break line 55, 57, and 64
//Sorting
if (!string.IsNullOrEmpty(model.SortOrder))
records = records.OrderBy(model.SortOrder);
var count = await records.CountAsync().ConfigureAwait(false);
records = records.Skip(model.Start);
if (model.Length != -1) records = records.Take(model.Length);
// Create models
var result = new List<SpStaySearchResultViewModel>();
try
{
await records.ForEachAsync(r =>
{
result.Add(new SpStaySearchResultViewModel()
{
BuildingName = r.BuildingName,
CaseManager = r.CaseManager,
CaseManagerId = r.CaseManagerId,
OccupantFileAs = r.OccupantFileAs,
StayOCFSNumber = r.StayOCFSNumber,
StayId = r.StayId,
MaxOfBillSentDate = r.MaxOfBillSentDate,
CountOfChildren = r.CountOfChildren,
StartDate = r.StartDate,
EndDate = r.EndDate,
OccupantId = r.OccupantId,
IsActive = r.IsActive,
});
}).ConfigureAwait(false);
}
catch (Exception e) { }
return Json(new
{
draw = model.Draw,
data = result,
recordsFiltered = count,
recordsTotal = count,
});
}
private IQueryable<spStaysSearch> getSearchData(StaySearchViewModel model)
{
var records = db.SpStaySearches.FromSqlRaw("dbo.spStaysSearch").AsQueryable();
if (model.OccupantId.HasValue)
records = records.Where(x => x.OccupantId == model.OccupantId);
if (!string.IsNullOrWhiteSpace(model.OccupantFileAs))
records = records.Where(x => x.OccupantFileAs == model.OccupantFileAs);
if (!string.IsNullOrWhiteSpace(model.BuildingName))
records = records.Where(x => x.BuildingName == model.BuildingName);
if (!string.IsNullOrWhiteSpace(model.CaseManager))
records = records.Where(x => x.CaseManager == model.CaseManager);
if (!string.IsNullOrWhiteSpace(model.BuildingName))
records = records.Where(x => x.BuildingName == model.BuildingName);
if (model.IntakeDateStart.HasValue && model.IntakeDateEnd.HasValue)
{
records = records.Where(x => x.StartDate >= model.IntakeDateStart && x.StartDate <= model.IntakeDateEnd);
}
else
{
if (model.IntakeDateStart.HasValue)
records = records.Where(x => x.StartDate >= model.IntakeDateStart);
if (model.IntakeDateEnd.HasValue)
records = records.Where(x => x.StartDate <= model.IntakeDateEnd);
}
if (model.ExitDateStart.HasValue && model.ExitDateEnd.HasValue)
{
records = records.Where(x => x.EndDate >= model.ExitDateStart && x.EndDate <= model.ExitDateEnd);
}
else
{
if (model.ExitDateStart.HasValue)
records = records.Where(x => x.EndDate >= model.ExitDateStart);
if (model.ExitDateEnd.HasValue)
records = records.Where(x => x.EndDate <= model.ExitDateEnd);
}
if (model.IsActive.HasValue)
records = records.Where(x => x.IsActive == model.IsActive);
return records;
}

Try this
var records = getSearchData(model).ToList();
var count = records.Count;
.....
You can't order records by model.SortOrder since it has nothing to do with records.
You can only do something like this
if (!string.IsNullOrEmpty(model.SortOrder)) records = records.OrderBy(r=> r.Id);

because your source data is a Stored Procedure, you cannot compose additional query expressions over the top of it. Instead you must load it into memory, as the error suggests, by enumerating the result set.
Including Related Data
SQL Server doesn't allow composing over stored procedure calls, so any attempt to apply additional query operators to such a call will result in invalid SQL. Use AsEnumerable or AsAsyncEnumerable method right after FromSqlRaw or FromSqlInterpolated methods to make sure that EF Core doesn't try to compose over a stored procedure.
The obvious way to interpret this in the code is to call .ToList() on the results from the SP, then to match your existing code pattern you can cast that result back to IQueryable:
var records = db.SpStaySearches.FromSqlRaw("dbo.spStaysSearch")
.ToList()
.AsQueryable()
Using AsEnumerable() is sometimes problematic as there are many different libraries that you may have implemented that might all provide an AsEnumerable() extension method.
We have to do this because even in SQL you cannot simply select from an SP and then add where clauses to it, you first have to read the results into a temporary table or a table variable, then you can re-query from the result set, that is what we are effectively doing now, we are reading the results into a C# variable (.ToList()) and then composing an in-memory query over the top of that result.
If your search logic must be encapsulated in a stored procedure, then given the technical limitations, the usual expectation is that you would add the search arguments as optional parameters to the stored procedure, rather then tring to add filter clauses on top of the results in C#.
We can help with how to move your filter logic into dbo.spStaysSearch but you'll have to post the content of that SP, ideally as a new question.
Instead of using an SP at all, where we lose practically all the goodness that EF can offer us, an alternative approach is to replace your SP entirely with a raw SQL then the rest of your logic will work as expected.
var sql = #"
SELECT
tblStays.*, tblOccupant.OccupantID,
tblOccupant.FileAs AS OccupantFileAs,
IIF(tblStays.BuildingName LIKE 'Main Shelter',
tblOccupant.OCFSMainNumber,
tblOccupant.OCFSNorthNumber) AS StayOCFSNumber,
COALESCE([CountOfOccupantStayID], 0) AS CountOfChildren,
tblCaseManager.FileAs AS CaseManager,
StaysMaxBillSentDate.MaxOfBillSentDate
FROM tblStays
LEFT JOIN tblOccupantStays ON tblStays.StayID = tblOccupantStays.StayID
LEFT JOIN tblOccupant ON tblOccupantStays.OccupantID = tblOccupant.OccupantID
LEFT JOIN (
SELECT lkpOccStays.StayID
, COUNT(tblOccupantStays.OccupantStayID) AS CountOfOccupantStayID
FROM tblOccupantStays lkpOccStays
INNER JOIN tblOccupant lkpChild ON lkpOccStays.OccupantID = lkpChild.OccupantID
WHERE lkpChild.OccupantType LIKE 'Child'
GROUP BY lkpOccStays.StayID
) OccupantStays_CountOfChildren ON tblStays.StayID = OccupantStays_CountOfChildren.StayID
LEFT JOIN tblCaseManager ON tblStays.CaseManagerID = tblCaseManager.CaseManagerID
LEFT JOIN (SELECT tblStayBillingHx.StayID
, MAX(tblStayBillingHx.BillSentDate) AS MaxOfBillSentDate
FROM tblStayBillingHx
GROUP BY tblStayBillingHx.StayID
) StaysMaxBillSentDate ON tblStays.StayID = StaysMaxBillSentDate.StayID
";
var records = db.SpStaySearches.FromSqlRaw(sql);
In this way the SP is providing the structure of the resultset, which might be necessary if you are using the Database-First approach but you are no longer executing the SP at all.
The SQL in this answer is provided as a guide to the syntax only, there is not enough information available to determine the validity of the query or that the results conform to your business requirements.

C# Entity Framework 7 Linq to Entity check if data exist in multiple table

I am trying to identify if a record exists in three tables.
TableMapping:
RbMapping:
RgMapping:
I am using EF 7 dbContext to access these tables.
I have written two samples of LINQ but not sure if it will perform against large volume of data.
Method 1:
bool isExist = dbContext.MappingTable
.AsNoTracking()
.Where(x => x.mappingCode == 1234)
.Any(x => x.RbMappings.Any(y => y.MappingId == x.MappingId && y.RbId == 3)
&& x.RgMapping.Any(z=> z.MappingId == x.MappingId && z.RgId == 2);
Method 2:
bool IsExist = (from mp in dbContext.MappingTable
join rb in dbContext.RbMapping on new
{
JoinProperty1 = mp.MappingId,
JoinProperty2 = rbId // Will be input field = 3
}
equals new
{
JoinProperty1 = rb.MappingId,,
JoinProperty2 = rb.RbId
}
join rg in dbCContext.RgMappings on new
{
JoinProperty1 = rb.MappingId,
}
equals new
{
JoinProperty1 = rb.MappingId,
}
where hs.MappingCode == 1234
select mp).AnyAsync();
Can anyone advise which one will work better for large volumes of data? I'll be running these Linq queries several times.
And these tables are also huge around 4 million records.

Performance tuning Entity Framework queries

So I've developed a dashboard which queries a database. The database has data stored in it from google analytics for a website we have.
I'm using ASP.NET MVC 5, EF, Linq with Telerik controls/widgets. The controller instantiates a service layer where I have my db context and business logic. Each svc.method() pertains to a specific result set I'm after that I package up in the VM for unpackaging into a widget within the view.
Currently, the response time in the network tab of Google Chrome is 5.6 seconds. I've illustrated one of the 8 methods to show you my approach.
My question is; how can I improve performance so that the page loads faster? Would making each method async improve it?
Thanks in advance for any advice you can provide.
Controller:
public ActionResult WebStats()
{
//other code removed for brevity
//Service layer where the db is queried and the business logic is performend
WebStatsService svc = new WebStatsService();
//view model
WebStatsViewModel vm = new WebStatsViewModel();
vm.PageViews = svc.GetPageViews(vm);
vm.UniquePageViews = svc.GetUniquePageViews(vm);
vm.UserRatioByCountry = svc.GetUserRatioByCountry(vm);
vm.PageViewsByCountry = svc.GetPageViewsByCountry(vm);
vm.TopTenHealthCenters = svc.GetTopTenHealthCenters(vm);
vm.UserTypeRatio = svc.GetUserTypeRatio(vm);
vm.TopTenHealthCentersByDateRange = svc.GetTopTenHealthCentersByDateRange(vm);
vm.ReferralSources = svc.GetTopTenReferralSources(vm);//Get top 10 referral paths
return View(vm);
}
Service:
public List<PageViews> GetPageViews(WebStatsViewModel vm)
{
using (ApplicationDbContext db = new ApplicationDbContext())
{
List<PageViews> pageViewStats = new List<PageViews>();
var results = db.PageStats.Where(x => (vm.CMS.Equals("All") || x.Source.Equals(vm.CMS))
&& (vm.HealthCenter.Equals("All") || x.HealthSectionName.Equals(vm.HealthCenter))
&& (vm.Country.Equals("All") || x.Country.Equals(vm.Country))
&& (vm.City.Equals("All") || x.City.Equals(vm.City))
&& (x.Date >= vm.StartDate)
&& (x.Date <= vm.EndDate)
).Select(x => new
{
Date = x.Date,
Total = x.PageViews
}).ToList();
var distinctDate = results.OrderBy(x => x.Date).Select(x => x.Date).Distinct();
foreach (var date in distinctDate)
{
PageViews pageViewStat = new PageViews();
pageViewStat.Date = date.Value.ToShortDateString();
pageViewStat.Total = results.Where(x => x.Date == date).Sum(x => x.Total);
pageViewStats.Add(pageViewStat);
}
return pageViewStats;
}
}

Here are some tips for EF queries:
(1) Avoid mixing constant and actual predicate in dynamic filters like this:
(vm.CMS.Equals("All") || x.Source.Equals(vm.CMS))
It might look concise, but generates awful and inefficient SQL. Instead, use if statements and chained Where:
// Base query including static filters
var query = db.PageStats.AsQueryable();
// Apply dynamic filters
if (!vm.CMS.Equals("All"))
query = query.Where(x => x.Source.Equals(vm.CMS));
// ...
// The rest of the query
query = query.Select(...
(2) Try returning as less data as possible from the SQL query.
For instance, your query is populating a list with (Date, Total) pairs, which you then manually (and not very efficiently) group by Date and take Sum(Total). Instead, you can make the EF query directly return that grouped/aggregated data.
Applying all that to your example would lead to something like this:
using (ApplicationDbContext db = new ApplicationDbContext())
{
var query = db.PageStats
.Where(x => x.Date >= vm.StartDate && x.Date <= vm.EndDate);
if (!vm.CMS.Equals("All"))
query = query.Where(x => x.Source.Equals(vm.CMS));
if (!vm.HealthCenter.Equals("All"))
query = query.Where(x => x.HealthSectionName.Equals(vm.HealthCenter));
if (!vm.Country.Equals("All"))
query = query.Where(x => x.Country.Equals(vm.Country));
if (!vm.City.Equals("All"))
query = query.Where(x => x.City.Equals(vm.City));
query = query
.GroupBy(x => x.Date)
.Select(g => new
{
Date = g.Key,
Total = g.Sum(x => x.PageViews)
})
.OrderBy(x => x.Date);
var pageViewStats = query
.AsEnumerable() // SQL query ends here
.Select(x => new PageViews
{
Date = x.Date.Value.ToShortDateString(),
Total = x.Total
})
.ToList();
return pageViewStats;
}
You can try and compare the performance with the original.
(Note: for this specific query we need to use two projections - one temporary in SQL query and one final in the in memory query. This is because of the need of ToShortDateString() method which is not supported for the SQL query. In most of the cases a single final projection in the SQL query would be sufficient.)

Some tips:
Indexes - index columns that appear in the where clause of select operations, use SQL profiler to detect 'table scan' operations and add indexes to avoid them (replace them with index search or clustered index search)
Caching - store the trace from SQL profiler above to a table in DB (SQL Profiler can do it) and group SQL commands by sql text, this may show some repeating selects that can be avoided by caching
Glimpse - can count SQL commands per web request, the number can be suprising if the web application has not been optimized yet. Glimpse can tell much more, for example how much time of the total time of a web request is spent on the server and how much time in the web browser rendering the page.
as the last resort, write your own SQL for the most exposed queries

c# linq syntax slow due to multiple queries in single query

I am wondering if there is a better, more efficient way to re-code the linq syntax below to make the query run faster i.e. with a single call to the database. My database is located remotely which causes this to be quite slow:
var query = (from ticket in dataClassesDataContext.Tickets.Where(TicketsToShow.And(SearchVals))
select new
{
Priority = ticket.TicketPriority.TicketPriorityName,
Ticket = string.Format(TicketFormat, ticket.TicketID),
AssetId = ticket.Asset.Serial,
OpenDate = ticket.CheckedInDate,
OpenFor = CalculateOpenDaysAndHours(ticket.CheckedInDate, ticket.ClosedDate),
Account = ticket.Account.Customer.Name,
Description = ticket.Description.Replace("\n", ", "),
Status = ticket.TicketStatus.TicketStatusName,
Closed = ticket.ClosedDate,
THIS IS THE CAUSE ====>>> Amount = GetOutstandingBalanceForTicket(ticket.TicketID),
Paid = ticket.Paid,
Warranty = ticket.WarrantyRepair,
AssetLocation = GetAssetLocationNameFromID(ticket.Asset.LocationID, AssLocNames)
}).Skip(totalToDisplay * page).Take(totalToDisplay);
if (SortOrder.ToLower().Contains("Asc".ToLower()))
{
query = query.OrderBy(p => p.OpenDate);
}
else
{
query = query.OrderByDescending(p => p.OpenDate);
}//ENDIF
The main cause for the poor performance is the code in the function GetOutstandingBalanceForTicket below which calculates the sum of all items in an invoice and returns this as a total in a string:
public static string GetOutstandingBalanceForTicket(int TicketID)
{
string result = string.Empty;
decimal total = 0;
try
{
using (DataClassesDataContext dataClassesDataContext = new DataClassesDataContext(cDbConnection.GetConnectionString()))
{
var queryCustomerTickets = from ticket in dataClassesDataContext.Tickets
where
(ticket.TicketID == TicketID)
select ticket;
if (queryCustomerTickets != null)
{
foreach (var ticket in queryCustomerTickets)
{
var queryTicketChargeItems = from chargeItem in dataClassesDataContext.ProductChargeItems
where chargeItem.ChargeID == ticket.ChargeID &&
chargeItem.Deleted == null
select chargeItem;
foreach (var chargeItem in queryTicketChargeItems)
{
total += (chargeItem.Qty * chargeItem.Price);
}
}
}
}
}
catch (Exception ex)
{
}
return total.ToString("0.##");
}
Thank you in advance.

As you pointed out this code is quite slow as a query will be required for each ticket.
to eliminate the need for multiple queries you should look at applying an inner join between the ticketsToShow and the tickets entity (on the ticketid), using groupby to provide the sum of the charges for each ticket.
This is well illustrated in the answers to LINQ: Using INNER JOIN, Group and SUM

Ideally you would probably approach it more as an eager loading all at once type of setup. However, I do not think linq2sql supports that (I know EF does). One thing you can do is avoid the nested query though. Since you already have access to the ticket table, perhaps you should just issue a Sum() on it from your select statement. Hard for me to verify if any of this is an improvement so this code is kind of on the fly if you will.
//(from ticket in dataClassesDataContext.Tickets.Where(TicketsToShow.And(SearchVals))
(from ticket in dataClassesDataContext.Tickets
//this would be where you could eager load if possible (not entirely required)
//.Include is an EF method used only as example
/*.Include(t => t.TicketPriority)//eager load required entities
.Include(t => t.Asset)//eager load required entities
.Include(t => t.Account.Customer)//eager load required entities
.Include(t => t.TicketStatus)//eager load required entities
.Include(t => t.ProductChargeItems)//eager load required entities
*/
.Where(TicketsToShow.And(SearchVals))
select new
{
Priority = ticket.TicketPriority.TicketPriorityName,
Ticket = string.Format(TicketFormat, ticket.TicketID),
AssetId = ticket.Asset.Serial,
OpenDate = ticket.CheckedInDate,
OpenFor = CalculateOpenDaysAndHours(ticket.CheckedInDate, ticket.ClosedDate),
Account = ticket.Account.Customer.Name,
Description = ticket.Description.Replace("\n", ", "),
Status = ticket.TicketStatus.TicketStatusName,
Closed = ticket.ClosedDate,
//Use Sum and the foreign relation instead of a nested query
Amount = ticket.ProductChargeItems.Where(pci => pci.Deleted == null).Sum(pci => pci.Qty * pci.Price),
Paid = ticket.Paid,
Warranty = ticket.WarrantyRepair,
AssetLocation = GetAssetLocationNameFromID(ticket.Asset.LocationID, AssLocNames)
}).Skip(totalToDisplay * page).Take(totalToDisplay);
if (SortOrder.ToLower().Contains("Asc".ToLower()))
{
query = query.OrderBy(p => p.OpenDate);
}
else
{
query = query.OrderByDescending(p => p.OpenDate);
}

I think, you can make this query simplier. Somethink like this:
public static string GetOutstandingBalanceForTicket(DataClassesDataContext context, int TicketID)
{
decimal total = 0;
var total = (from ticket in context.Tickets
join chargeItem from context.ProductChargeItems on chargeItem.ChargeID == ticket.ChargeID
where (ticket.TicketID == TicketID && chargeItem.Deleted == null)
select chargeItem).Sum(chargeItem => chargeItem.Qty * chargeItem.Price);
return total.ToString("0.##");
}
/*...*/
Amount = GetOutstandingBalanceForTicket(dataClassesDataContext, ticket.TicketID),
Now, you can inline this methos in your query.
It can contains syntax errors, because I wrote it in notepad.