Develop Reference

How to factorize a Linq request with multiple includes?

My model has:
Several DeviceStatus attached to one mandatory Device
SeveralDevice attached to one mandatory Panel
When I query DeviceStatus, I need to have Device and Panel attached to it in the query result.
... DeviceStatus.Device is null in the query result.
Here is the Linq Query:
using (var actiContext = new ActigraphyContext())
{
var todayStatus =
from s in actiContext.DeviceStatus.Include(s1 => s1.Device.Panel)
where DbFunctions.TruncateTime(s.TimeStamp) == DbFunctions.TruncateTime( DateTimeOffset.Now)
&& s.Device.Panel.Mac == mac
&& (s.Device.Ty == 4 || s.Device.Ty == 9)
select s;
// var tempList = todayStatus.toList();
var todayLastStatus =
from s in todayStatus.Include(s1 => s1.Device.Panel)
let lastTimeStamp = todayStatus.Max(s1 => s1.TimeStamp)
where s.TimeStamp == lastTimeStamp
select s;
var requestResult = todayLastStatus.FirstOrDefault();
return requestResult;
}
If I uncomment the line // var tempList = todayStatus.toList();, where tempList is not used, it works: requestResult.Device is set!
But the bad side is todayStatus.toList triggers a request that brings a huge amount of data.
So how to get the DeviceStatus with its relative objects ?
Note: the database behind is SQL Server 2012

When you call an Include() over a LINQ query, it performs Eagerly Loading.
As documented in MSDN:
Eager loading is the process whereby a query for one type of entity also loads related entities as part of the query. Eager loading is achieved by use of the Include method.
When the entity is read, related data is retrieved along with it. This typically results in a single join query that retrieves all of the data that's needed. You specify eager loading by using the Include method.
So you need to call the .toList() to complete the query execution.
Since the data is huge, you can pickup relative specific columns as per your requirement by using the Select clause.
var todayStatus =
from s in actiContext.DeviceStatus
.Include(s1 => s1.Device.Panel.Select(d => new
{
d.DeviceId,
d.DeviceName,
d.PanelID
}))
where DbFunctions.TruncateTime(s.TimeStamp) == DbFunctions.TruncateTime( DateTimeOffset.Now)
&& s.Device.Panel.Mac == mac
&& (s.Device.Ty == 4 || s.Device.Ty == 9)
select s;
var tempList = todayStatus.toList();

The query doesn't actually run until you do a call like ToList(), which is why uncommenting that line works. If the query is bringing back too much data, then you need to change the query to narrow down the amount of data you're bringing back.

Ok this request is a more simple way to achieve this:
using (var actiContext = new ActigraphyContext())
{
var todayLastStatus =
from s in actiContext.DeviceStatus.Include(s1 => s1.Device.Panel)
where DbFunctions.TruncateTime(s.TimeStamp) == DbFunctions.TruncateTime( DateTimeOffset.Now)
&& s.Device.Panel.Mac == mac
&& (s.Device.Ty == 4 || s.Device.Ty == 9)
orderby s.TimeStamp descending
select s;
var requestResult = todayLastStatus.Take(1).FirstOrDefault();
return requestResult;
}
But the question remains: why didn't I get the relative object in my first request ?

Search method joing multiple tables

I need help with a search method for searching the tables for a matching text.
This works, except that the join needs to be LEFT OUTER JOIN otherwise I dont get any results if the pageId is missing in any of the tables.
This solution takes to long time to run, I would appreciate if someone can help me out with a better solution to handle this task.
public async Task<IEnumerable<Result>> Search(string query)
{
var temp = await (from page in _context.Pages
join pageLocation in _context.PageLocations on page.Id equals pageLocation.PageId
join location in _context.Locations on pageLocation.LocationId equals location.Id
join pageSpecialty in _context.PageSpecialties on page.Id equals pageSpecialty.PageId
join specialty in _context.Specialties on pageSpecialty.SpecialtyId equals specialty.Id
where
page.Name.ToLower().Contains(query)
|| location.Name.ToLower().Contains(query)
|| specialty.Name.ToLower().Contains(query)
select new Result
{
PageId = page.Id,
Name = page.Name,
Presentation = page.Presentation,
Rating = page.Rating
}).ToListAsync();
var results = new List<Result>();
foreach (var t in temp)
{
if (!results.Exists(p => p.PageId == t.PageId))
{
t.Locations = GetLocations(t.PageId);
t.Specialties = GetSpecialties(t.PageId);
results.Add(t);
}
}
return results;
}

Using navigation properties, the query could look like:
var temp = await (from page in _context.Pages
where Name.Contains(query)
|| page.PageLocation.Any(pl => pl.Location.Name.Contains(query))
|| page.PageSpecialties.Any(pl => pl.Specialty.Name.Contains(query))
select new Result
{
PageId = page.Id,
Name = page.Name,
Presentation = page.Presentation,
Rating = page.Rating,
Locations = page.PageLocation.Select(pl => pl.Location),
Specialties = page.PageSpecialties.Select(pl => pl.Specialty)
}).ToListAsync();
This has several benefits:
By the absence of joins, The query returns unique Result objects right away, so you don't need to deduplicate them afterwards.
The locations and specialties are loaded in the same query instead of two queries per Result (aka n+1 problem).
(Likely) ToLower is removed because the search is probably not case sensitive anyway. The query is executed as SQL and most of the times, SQL databases have case-insensitive collations. Removing ToLower makes the query sargable again.

Performance tuning Entity Framework queries

So I've developed a dashboard which queries a database. The database has data stored in it from google analytics for a website we have.
I'm using ASP.NET MVC 5, EF, Linq with Telerik controls/widgets. The controller instantiates a service layer where I have my db context and business logic. Each svc.method() pertains to a specific result set I'm after that I package up in the VM for unpackaging into a widget within the view.
Currently, the response time in the network tab of Google Chrome is 5.6 seconds. I've illustrated one of the 8 methods to show you my approach.
My question is; how can I improve performance so that the page loads faster? Would making each method async improve it?
Thanks in advance for any advice you can provide.
Controller:
public ActionResult WebStats()
{
//other code removed for brevity
//Service layer where the db is queried and the business logic is performend
WebStatsService svc = new WebStatsService();
//view model
WebStatsViewModel vm = new WebStatsViewModel();
vm.PageViews = svc.GetPageViews(vm);
vm.UniquePageViews = svc.GetUniquePageViews(vm);
vm.UserRatioByCountry = svc.GetUserRatioByCountry(vm);
vm.PageViewsByCountry = svc.GetPageViewsByCountry(vm);
vm.TopTenHealthCenters = svc.GetTopTenHealthCenters(vm);
vm.UserTypeRatio = svc.GetUserTypeRatio(vm);
vm.TopTenHealthCentersByDateRange = svc.GetTopTenHealthCentersByDateRange(vm);
vm.ReferralSources = svc.GetTopTenReferralSources(vm);//Get top 10 referral paths
return View(vm);
}
Service:
public List<PageViews> GetPageViews(WebStatsViewModel vm)
{
using (ApplicationDbContext db = new ApplicationDbContext())
{
List<PageViews> pageViewStats = new List<PageViews>();
var results = db.PageStats.Where(x => (vm.CMS.Equals("All") || x.Source.Equals(vm.CMS))
&& (vm.HealthCenter.Equals("All") || x.HealthSectionName.Equals(vm.HealthCenter))
&& (vm.Country.Equals("All") || x.Country.Equals(vm.Country))
&& (vm.City.Equals("All") || x.City.Equals(vm.City))
&& (x.Date >= vm.StartDate)
&& (x.Date <= vm.EndDate)
).Select(x => new
{
Date = x.Date,
Total = x.PageViews
}).ToList();
var distinctDate = results.OrderBy(x => x.Date).Select(x => x.Date).Distinct();
foreach (var date in distinctDate)
{
PageViews pageViewStat = new PageViews();
pageViewStat.Date = date.Value.ToShortDateString();
pageViewStat.Total = results.Where(x => x.Date == date).Sum(x => x.Total);
pageViewStats.Add(pageViewStat);
}
return pageViewStats;
}
}

Here are some tips for EF queries:
(1) Avoid mixing constant and actual predicate in dynamic filters like this:
(vm.CMS.Equals("All") || x.Source.Equals(vm.CMS))
It might look concise, but generates awful and inefficient SQL. Instead, use if statements and chained Where:
// Base query including static filters
var query = db.PageStats.AsQueryable();
// Apply dynamic filters
if (!vm.CMS.Equals("All"))
query = query.Where(x => x.Source.Equals(vm.CMS));
// ...
// The rest of the query
query = query.Select(...
(2) Try returning as less data as possible from the SQL query.
For instance, your query is populating a list with (Date, Total) pairs, which you then manually (and not very efficiently) group by Date and take Sum(Total). Instead, you can make the EF query directly return that grouped/aggregated data.
Applying all that to your example would lead to something like this:
using (ApplicationDbContext db = new ApplicationDbContext())
{
var query = db.PageStats
.Where(x => x.Date >= vm.StartDate && x.Date <= vm.EndDate);
if (!vm.CMS.Equals("All"))
query = query.Where(x => x.Source.Equals(vm.CMS));
if (!vm.HealthCenter.Equals("All"))
query = query.Where(x => x.HealthSectionName.Equals(vm.HealthCenter));
if (!vm.Country.Equals("All"))
query = query.Where(x => x.Country.Equals(vm.Country));
if (!vm.City.Equals("All"))
query = query.Where(x => x.City.Equals(vm.City));
query = query
.GroupBy(x => x.Date)
.Select(g => new
{
Date = g.Key,
Total = g.Sum(x => x.PageViews)
})
.OrderBy(x => x.Date);
var pageViewStats = query
.AsEnumerable() // SQL query ends here
.Select(x => new PageViews
{
Date = x.Date.Value.ToShortDateString(),
Total = x.Total
})
.ToList();
return pageViewStats;
}
You can try and compare the performance with the original.
(Note: for this specific query we need to use two projections - one temporary in SQL query and one final in the in memory query. This is because of the need of ToShortDateString() method which is not supported for the SQL query. In most of the cases a single final projection in the SQL query would be sufficient.)

Some tips:
Indexes - index columns that appear in the where clause of select operations, use SQL profiler to detect 'table scan' operations and add indexes to avoid them (replace them with index search or clustered index search)
Caching - store the trace from SQL profiler above to a table in DB (SQL Profiler can do it) and group SQL commands by sql text, this may show some repeating selects that can be avoided by caching
Glimpse - can count SQL commands per web request, the number can be suprising if the web application has not been optimized yet. Glimpse can tell much more, for example how much time of the total time of a web request is spent on the server and how much time in the web browser rendering the page.
as the last resort, write your own SQL for the most exposed queries

c# linq syntax slow due to multiple queries in single query

I am wondering if there is a better, more efficient way to re-code the linq syntax below to make the query run faster i.e. with a single call to the database. My database is located remotely which causes this to be quite slow:
var query = (from ticket in dataClassesDataContext.Tickets.Where(TicketsToShow.And(SearchVals))
select new
{
Priority = ticket.TicketPriority.TicketPriorityName,
Ticket = string.Format(TicketFormat, ticket.TicketID),
AssetId = ticket.Asset.Serial,
OpenDate = ticket.CheckedInDate,
OpenFor = CalculateOpenDaysAndHours(ticket.CheckedInDate, ticket.ClosedDate),
Account = ticket.Account.Customer.Name,
Description = ticket.Description.Replace("\n", ", "),
Status = ticket.TicketStatus.TicketStatusName,
Closed = ticket.ClosedDate,
THIS IS THE CAUSE ====>>> Amount = GetOutstandingBalanceForTicket(ticket.TicketID),
Paid = ticket.Paid,
Warranty = ticket.WarrantyRepair,
AssetLocation = GetAssetLocationNameFromID(ticket.Asset.LocationID, AssLocNames)
}).Skip(totalToDisplay * page).Take(totalToDisplay);
if (SortOrder.ToLower().Contains("Asc".ToLower()))
{
query = query.OrderBy(p => p.OpenDate);
}
else
{
query = query.OrderByDescending(p => p.OpenDate);
}//ENDIF
The main cause for the poor performance is the code in the function GetOutstandingBalanceForTicket below which calculates the sum of all items in an invoice and returns this as a total in a string:
public static string GetOutstandingBalanceForTicket(int TicketID)
{
string result = string.Empty;
decimal total = 0;
try
{
using (DataClassesDataContext dataClassesDataContext = new DataClassesDataContext(cDbConnection.GetConnectionString()))
{
var queryCustomerTickets = from ticket in dataClassesDataContext.Tickets
where
(ticket.TicketID == TicketID)
select ticket;
if (queryCustomerTickets != null)
{
foreach (var ticket in queryCustomerTickets)
{
var queryTicketChargeItems = from chargeItem in dataClassesDataContext.ProductChargeItems
where chargeItem.ChargeID == ticket.ChargeID &&
chargeItem.Deleted == null
select chargeItem;
foreach (var chargeItem in queryTicketChargeItems)
{
total += (chargeItem.Qty * chargeItem.Price);
}
}
}
}
}
catch (Exception ex)
{
}
return total.ToString("0.##");
}
Thank you in advance.

As you pointed out this code is quite slow as a query will be required for each ticket.
to eliminate the need for multiple queries you should look at applying an inner join between the ticketsToShow and the tickets entity (on the ticketid), using groupby to provide the sum of the charges for each ticket.
This is well illustrated in the answers to LINQ: Using INNER JOIN, Group and SUM

Ideally you would probably approach it more as an eager loading all at once type of setup. However, I do not think linq2sql supports that (I know EF does). One thing you can do is avoid the nested query though. Since you already have access to the ticket table, perhaps you should just issue a Sum() on it from your select statement. Hard for me to verify if any of this is an improvement so this code is kind of on the fly if you will.
//(from ticket in dataClassesDataContext.Tickets.Where(TicketsToShow.And(SearchVals))
(from ticket in dataClassesDataContext.Tickets
//this would be where you could eager load if possible (not entirely required)
//.Include is an EF method used only as example
/*.Include(t => t.TicketPriority)//eager load required entities
.Include(t => t.Asset)//eager load required entities
.Include(t => t.Account.Customer)//eager load required entities
.Include(t => t.TicketStatus)//eager load required entities
.Include(t => t.ProductChargeItems)//eager load required entities
*/
.Where(TicketsToShow.And(SearchVals))
select new
{
Priority = ticket.TicketPriority.TicketPriorityName,
Ticket = string.Format(TicketFormat, ticket.TicketID),
AssetId = ticket.Asset.Serial,
OpenDate = ticket.CheckedInDate,
OpenFor = CalculateOpenDaysAndHours(ticket.CheckedInDate, ticket.ClosedDate),
Account = ticket.Account.Customer.Name,
Description = ticket.Description.Replace("\n", ", "),
Status = ticket.TicketStatus.TicketStatusName,
Closed = ticket.ClosedDate,
//Use Sum and the foreign relation instead of a nested query
Amount = ticket.ProductChargeItems.Where(pci => pci.Deleted == null).Sum(pci => pci.Qty * pci.Price),
Paid = ticket.Paid,
Warranty = ticket.WarrantyRepair,
AssetLocation = GetAssetLocationNameFromID(ticket.Asset.LocationID, AssLocNames)
}).Skip(totalToDisplay * page).Take(totalToDisplay);
if (SortOrder.ToLower().Contains("Asc".ToLower()))
{
query = query.OrderBy(p => p.OpenDate);
}
else
{
query = query.OrderByDescending(p => p.OpenDate);
}

I think, you can make this query simplier. Somethink like this:
public static string GetOutstandingBalanceForTicket(DataClassesDataContext context, int TicketID)
{
decimal total = 0;
var total = (from ticket in context.Tickets
join chargeItem from context.ProductChargeItems on chargeItem.ChargeID == ticket.ChargeID
where (ticket.TicketID == TicketID && chargeItem.Deleted == null)
select chargeItem).Sum(chargeItem => chargeItem.Qty * chargeItem.Price);
return total.ToString("0.##");
}
/*...*/
Amount = GetOutstandingBalanceForTicket(dataClassesDataContext, ticket.TicketID),
Now, you can inline this methos in your query.
It can contains syntax errors, because I wrote it in notepad.

Can I clone an IQueryable in linq? For UNION purposes?

I have a table of WorkOrders. The table has a PrimaryWorker & PrimaryPay field. It also has a SecondaryWorker & SecondaryPay field (which can be null).
I wish to run 2 very similar queries & union them so that it will return a Worker Field & Pay field. So if a single WorkOrder record had both the PrimaryWorker and SecondaryWorker field populated I would get 2 records back.
The "where clause" part of these 2 queries is very similar and long to construct. Here's a dummy example
var q = ctx.WorkOrder.Where(w => w.WorkDate >= StartDt && w.WorkDate <= EndDt);
if(showApprovedOnly)
{
q = q.Where(w => w.IsApproved);
}
//...more filters applied
Now I also have a search flag called hideZeroPay. If that's true I don't want to include the record if the worker was payed $0. But obviously for 1 query I need to compare the PrimaryPay field and in the other I need to compare the SecondaryPay field.
So I'm wondering how to do this.
Can I clone my base query q and make a primary & secondary worker query out of it and then union those 2 queries together?

Hmm, I'm not sure that I understand you intention. But I think cloning is not neccessary. Why don't you split two new queries from your base query?
var baseQuery = ctx.WorkOrder.Where(w => w.WorkDate >= StartDt && w.WorkDate <= EndDt);
IQueryable<WorkOrder> query1;
if (showApprovedOnly)
{
query1 = baseQuery.Where(w => w.IsApproved);
}
//more filters on query1
...
IQueryable<WorkOrder> query2;
if (/*something*/)
query2 = baseQuery.Where(w => w.SomeThing);
After defining your queries you can interpret them (per enumeration) and retrieve your different results.
var res1 = query1.ToList();
var res2 = query2.ToList();

When you do your second Where you are actually cloning your query.
Here you create your initial queryable object.
var q = ctx.WorkOrder.Where(w => w.WorkDate >= StartDt && w.WorkDate <= EndDt);
Here you create a new queryable with the where associated
if(showApprovedOnly)
{
q = q.Where(w => w.IsApproved);
}
//...more filters applied
All you need to do is create a new variable to store the ammended query.
var qw = q.Where(w=> w.IsApproved);
This works because the queryable is created as an object and the query itself is only run once you enumerate it.