So I've developed a dashboard which queries a database. The database has data stored in it from google analytics for a website we have.
I'm using ASP.NET MVC 5, EF, Linq with Telerik controls/widgets. The controller instantiates a service layer where I have my db context and business logic. Each svc.method() pertains to a specific result set I'm after that I package up in the VM for unpackaging into a widget within the view.
Currently, the response time in the network tab of Google Chrome is 5.6 seconds. I've illustrated one of the 8 methods to show you my approach.
My question is; how can I improve performance so that the page loads faster? Would making each method async improve it?
Thanks in advance for any advice you can provide.
Controller:
public ActionResult WebStats()
{
//other code removed for brevity
//Service layer where the db is queried and the business logic is performend
WebStatsService svc = new WebStatsService();
//view model
WebStatsViewModel vm = new WebStatsViewModel();
vm.PageViews = svc.GetPageViews(vm);
vm.UniquePageViews = svc.GetUniquePageViews(vm);
vm.UserRatioByCountry = svc.GetUserRatioByCountry(vm);
vm.PageViewsByCountry = svc.GetPageViewsByCountry(vm);
vm.TopTenHealthCenters = svc.GetTopTenHealthCenters(vm);
vm.UserTypeRatio = svc.GetUserTypeRatio(vm);
vm.TopTenHealthCentersByDateRange = svc.GetTopTenHealthCentersByDateRange(vm);
vm.ReferralSources = svc.GetTopTenReferralSources(vm);//Get top 10 referral paths
return View(vm);
}
Service:
public List<PageViews> GetPageViews(WebStatsViewModel vm)
{
using (ApplicationDbContext db = new ApplicationDbContext())
{
List<PageViews> pageViewStats = new List<PageViews>();
var results = db.PageStats.Where(x => (vm.CMS.Equals("All") || x.Source.Equals(vm.CMS))
&& (vm.HealthCenter.Equals("All") || x.HealthSectionName.Equals(vm.HealthCenter))
&& (vm.Country.Equals("All") || x.Country.Equals(vm.Country))
&& (vm.City.Equals("All") || x.City.Equals(vm.City))
&& (x.Date >= vm.StartDate)
&& (x.Date <= vm.EndDate)
).Select(x => new
{
Date = x.Date,
Total = x.PageViews
}).ToList();
var distinctDate = results.OrderBy(x => x.Date).Select(x => x.Date).Distinct();
foreach (var date in distinctDate)
{
PageViews pageViewStat = new PageViews();
pageViewStat.Date = date.Value.ToShortDateString();
pageViewStat.Total = results.Where(x => x.Date == date).Sum(x => x.Total);
pageViewStats.Add(pageViewStat);
}
return pageViewStats;
}
}
Here are some tips for EF queries:
(1) Avoid mixing constant and actual predicate in dynamic filters like this:
(vm.CMS.Equals("All") || x.Source.Equals(vm.CMS))
It might look concise, but generates awful and inefficient SQL. Instead, use if statements and chained Where:
// Base query including static filters
var query = db.PageStats.AsQueryable();
// Apply dynamic filters
if (!vm.CMS.Equals("All"))
query = query.Where(x => x.Source.Equals(vm.CMS));
// ...
// The rest of the query
query = query.Select(...
(2) Try returning as less data as possible from the SQL query.
For instance, your query is populating a list with (Date, Total) pairs, which you then manually (and not very efficiently) group by Date and take Sum(Total). Instead, you can make the EF query directly return that grouped/aggregated data.
Applying all that to your example would lead to something like this:
using (ApplicationDbContext db = new ApplicationDbContext())
{
var query = db.PageStats
.Where(x => x.Date >= vm.StartDate && x.Date <= vm.EndDate);
if (!vm.CMS.Equals("All"))
query = query.Where(x => x.Source.Equals(vm.CMS));
if (!vm.HealthCenter.Equals("All"))
query = query.Where(x => x.HealthSectionName.Equals(vm.HealthCenter));
if (!vm.Country.Equals("All"))
query = query.Where(x => x.Country.Equals(vm.Country));
if (!vm.City.Equals("All"))
query = query.Where(x => x.City.Equals(vm.City));
query = query
.GroupBy(x => x.Date)
.Select(g => new
{
Date = g.Key,
Total = g.Sum(x => x.PageViews)
})
.OrderBy(x => x.Date);
var pageViewStats = query
.AsEnumerable() // SQL query ends here
.Select(x => new PageViews
{
Date = x.Date.Value.ToShortDateString(),
Total = x.Total
})
.ToList();
return pageViewStats;
}
You can try and compare the performance with the original.
(Note: for this specific query we need to use two projections - one temporary in SQL query and one final in the in memory query. This is because of the need of ToShortDateString() method which is not supported for the SQL query. In most of the cases a single final projection in the SQL query would be sufficient.)
Some tips:
Indexes - index columns that appear in the where clause of select operations, use SQL profiler to detect 'table scan' operations and add indexes to avoid them (replace them with index search or clustered index search)
Caching - store the trace from SQL profiler above to a table in DB (SQL Profiler can do it) and group SQL commands by sql text, this may show some repeating selects that can be avoided by caching
Glimpse - can count SQL commands per web request, the number can be suprising if the web application has not been optimized yet. Glimpse can tell much more, for example how much time of the total time of a web request is spent on the server and how much time in the web browser rendering the page.
as the last resort, write your own SQL for the most exposed queries
Related
I'm currently creating a site that showcases all my patients within a data table and I have to use FromSqlRaw in order to get the data from my database. I have a search funtion that allows me to search the patients within the table but upon entering the page I get this error when I use AsQueryable and no data is displayed in the table. It recommends me to use AsEnumerable but when I do I get an intellisense error. Any ideas on how to fix?
public async Task<IActionResult> Search(StaySearchViewModel model)
{
if (model.Cleared)
{
return Json(new
{
draw = model.Draw,
data = new object[] { },
recordsFiltered = 0,
recordsTotal = 0,
total = 0
});
}
var records = getSearchData(model);
//var records = System.Linq.Enumerable.AsEnumerable(getSearchData(model)); // Hard coding this an enumerable will break line 55, 57, and 64
//Sorting
if (!string.IsNullOrEmpty(model.SortOrder))
records = records.OrderBy(model.SortOrder);
var count = await records.CountAsync().ConfigureAwait(false);
records = records.Skip(model.Start);
if (model.Length != -1) records = records.Take(model.Length);
// Create models
var result = new List<SpStaySearchResultViewModel>();
try
{
await records.ForEachAsync(r =>
{
result.Add(new SpStaySearchResultViewModel()
{
BuildingName = r.BuildingName,
CaseManager = r.CaseManager,
CaseManagerId = r.CaseManagerId,
OccupantFileAs = r.OccupantFileAs,
StayOCFSNumber = r.StayOCFSNumber,
StayId = r.StayId,
MaxOfBillSentDate = r.MaxOfBillSentDate,
CountOfChildren = r.CountOfChildren,
StartDate = r.StartDate,
EndDate = r.EndDate,
OccupantId = r.OccupantId,
IsActive = r.IsActive,
});
}).ConfigureAwait(false);
}
catch (Exception e) { }
return Json(new
{
draw = model.Draw,
data = result,
recordsFiltered = count,
recordsTotal = count,
});
}
private IQueryable<spStaysSearch> getSearchData(StaySearchViewModel model)
{
var records = db.SpStaySearches.FromSqlRaw("dbo.spStaysSearch").AsQueryable();
if (model.OccupantId.HasValue)
records = records.Where(x => x.OccupantId == model.OccupantId);
if (!string.IsNullOrWhiteSpace(model.OccupantFileAs))
records = records.Where(x => x.OccupantFileAs == model.OccupantFileAs);
if (!string.IsNullOrWhiteSpace(model.BuildingName))
records = records.Where(x => x.BuildingName == model.BuildingName);
if (!string.IsNullOrWhiteSpace(model.CaseManager))
records = records.Where(x => x.CaseManager == model.CaseManager);
if (!string.IsNullOrWhiteSpace(model.BuildingName))
records = records.Where(x => x.BuildingName == model.BuildingName);
if (model.IntakeDateStart.HasValue && model.IntakeDateEnd.HasValue)
{
records = records.Where(x => x.StartDate >= model.IntakeDateStart && x.StartDate <= model.IntakeDateEnd);
}
else
{
if (model.IntakeDateStart.HasValue)
records = records.Where(x => x.StartDate >= model.IntakeDateStart);
if (model.IntakeDateEnd.HasValue)
records = records.Where(x => x.StartDate <= model.IntakeDateEnd);
}
if (model.ExitDateStart.HasValue && model.ExitDateEnd.HasValue)
{
records = records.Where(x => x.EndDate >= model.ExitDateStart && x.EndDate <= model.ExitDateEnd);
}
else
{
if (model.ExitDateStart.HasValue)
records = records.Where(x => x.EndDate >= model.ExitDateStart);
if (model.ExitDateEnd.HasValue)
records = records.Where(x => x.EndDate <= model.ExitDateEnd);
}
if (model.IsActive.HasValue)
records = records.Where(x => x.IsActive == model.IsActive);
return records;
}
Try this
var records = getSearchData(model).ToList();
var count = records.Count;
.....
You can't order records by model.SortOrder since it has nothing to do with records.
You can only do something like this
if (!string.IsNullOrEmpty(model.SortOrder)) records = records.OrderBy(r=> r.Id);
because your source data is a Stored Procedure, you cannot compose additional query expressions over the top of it. Instead you must load it into memory, as the error suggests, by enumerating the result set.
Including Related Data
SQL Server doesn't allow composing over stored procedure calls, so any attempt to apply additional query operators to such a call will result in invalid SQL. Use AsEnumerable or AsAsyncEnumerable method right after FromSqlRaw or FromSqlInterpolated methods to make sure that EF Core doesn't try to compose over a stored procedure.
The obvious way to interpret this in the code is to call .ToList() on the results from the SP, then to match your existing code pattern you can cast that result back to IQueryable:
var records = db.SpStaySearches.FromSqlRaw("dbo.spStaysSearch")
.ToList()
.AsQueryable()
Using AsEnumerable() is sometimes problematic as there are many different libraries that you may have implemented that might all provide an AsEnumerable() extension method.
We have to do this because even in SQL you cannot simply select from an SP and then add where clauses to it, you first have to read the results into a temporary table or a table variable, then you can re-query from the result set, that is what we are effectively doing now, we are reading the results into a C# variable (.ToList()) and then composing an in-memory query over the top of that result.
If your search logic must be encapsulated in a stored procedure, then given the technical limitations, the usual expectation is that you would add the search arguments as optional parameters to the stored procedure, rather then tring to add filter clauses on top of the results in C#.
We can help with how to move your filter logic into dbo.spStaysSearch but you'll have to post the content of that SP, ideally as a new question.
Instead of using an SP at all, where we lose practically all the goodness that EF can offer us, an alternative approach is to replace your SP entirely with a raw SQL then the rest of your logic will work as expected.
var sql = #"
SELECT
tblStays.*, tblOccupant.OccupantID,
tblOccupant.FileAs AS OccupantFileAs,
IIF(tblStays.BuildingName LIKE 'Main Shelter',
tblOccupant.OCFSMainNumber,
tblOccupant.OCFSNorthNumber) AS StayOCFSNumber,
COALESCE([CountOfOccupantStayID], 0) AS CountOfChildren,
tblCaseManager.FileAs AS CaseManager,
StaysMaxBillSentDate.MaxOfBillSentDate
FROM tblStays
LEFT JOIN tblOccupantStays ON tblStays.StayID = tblOccupantStays.StayID
LEFT JOIN tblOccupant ON tblOccupantStays.OccupantID = tblOccupant.OccupantID
LEFT JOIN (
SELECT lkpOccStays.StayID
, COUNT(tblOccupantStays.OccupantStayID) AS CountOfOccupantStayID
FROM tblOccupantStays lkpOccStays
INNER JOIN tblOccupant lkpChild ON lkpOccStays.OccupantID = lkpChild.OccupantID
WHERE lkpChild.OccupantType LIKE 'Child'
GROUP BY lkpOccStays.StayID
) OccupantStays_CountOfChildren ON tblStays.StayID = OccupantStays_CountOfChildren.StayID
LEFT JOIN tblCaseManager ON tblStays.CaseManagerID = tblCaseManager.CaseManagerID
LEFT JOIN (SELECT tblStayBillingHx.StayID
, MAX(tblStayBillingHx.BillSentDate) AS MaxOfBillSentDate
FROM tblStayBillingHx
GROUP BY tblStayBillingHx.StayID
) StaysMaxBillSentDate ON tblStays.StayID = StaysMaxBillSentDate.StayID
";
var records = db.SpStaySearches.FromSqlRaw(sql);
In this way the SP is providing the structure of the resultset, which might be necessary if you are using the Database-First approach but you are no longer executing the SP at all.
The SQL in this answer is provided as a guide to the syntax only, there is not enough information available to determine the validity of the query or that the results conform to your business requirements.
In a .NET Core Project I'm using EntityFramework and I have a DbContext (shopContext) injected in my class repository. I have the next query:
var res = shopContext.Orders.Where(x => x.Status == 1).Sum(p => p.Total);
Occasionally, the Orders table is doing maintenance tasks and the table is locked. For this query, I can't wait to the maintenance tasks and I need access to the table with the IsolationLevel.ReadUncommited option in the transaction:
using (var transaction = mutuaContext.Database.BeginTransaction(IsolationLevel.ReadUncommitted))
{
var res = shopContext.Orders.Where(x => x.Status == 1).Sum(p => p.Total);
}
The problem is that I only want that the context execute the query with this IsolationLevel configuration in these query, but the next queries continue executing although the table is locked yet.
Why are the following queries not waiting for the Table to be unlocked?
Example of my code:
using (var transaction = mutuaContext.Database.BeginTransaction(IsolationLevel.ReadUncommitted))
{
var res = shopContext.Orders.Where(x => x.Status == 1).Sum(p => p.Total); // this code would be executed
}
var total = shopContext.Orders.Where(x => x.Status == 0).Sum(p => p.Total); // this code would NOT be executed but is executed
I don't understand how the context get the transaction configuration. I would like that someone explain it to me.
I tried call to transaction.Commit() after fist query, but still not working.
use
yourContext.Database.BeginTransaction(IsolationLevel.ReadUncommitted)
// your normal queries via yourContext goes here
// do not forget to end the transaction
You can use raw SQL query (there is a similar SqlQuery() method for EF6, as well) and specify with (nolock) table hint. Something like this:
var res = shopContext.Orders.FromSqlRaw("select sum(Total) from dbo.Orders with (nolock) where Status = 1").ToList();
However, once you will deploy this into production environment and put your code under a decent concurrent load, most probably you will not like the outcome.
UPD: For EF Core 2.2, the syntax is a bit different:
var res = shopContext.Orders.FromSql("select * from Orders with(nolock)")
.Where(x => x.Status == 1).Sum(p => p.Total);
Good morning,
I'm having trouble with a EF query. This is what i am trying to do.
First i am pulling a list of ID's like so (List of IDs are found in the included x.MappingAccts entity):
Entities.DB1.Mapping mapping = null;
using (var db = new Entities.DB1.DB1Conn())
{
mapping = db.Mappings.Where(x => x.Code == code).Include(x => x.MappingAccts).FirstOrDefault();
}
Later, i'm trying to do a query on a different DB against the list of Id's i pulled above (essentially a IN clause):
using (var db = new Entities.DB2.DB2Conn())
{
var accounts = db.Accounts.Where(mapping.MappingAccts.Any(y => y.Id == ?????????)).ToList();
}
As you can see i only got part way with this.
Basically what i need to do is query the Accounts table against it's ID column and pull all records that match mapping.MappingAccts.Id column.
Most of the examples i am finding explain nicely how to do this against a single dimension array but i'm looking to compare specific columns.
Any assist would be awesome.
Nugs
An IN clause is generated using a IEnumerable.Contains.
From the first DB1 context, materialize the list of Id's
var idList = mapping.MappingAccts.Select(m => m.Id).ToList();
Then in the second context query against the materialized list of id's
var accounts = db.Accounts
.Where(a => idList.Contains(a.Id))
.ToList();
The only problem you may have is with the amount of id's you are getting in the first list. You may hit a limit with the SQL query.
This will give the list of Accounts which have the Ids contained by MappingAccts
using (var db = new Entities.DB2.DB2Conn())
{
var accounts = db.Accounts.Where(s => mapping.MappingAccts.Any(y => y.Id == s.Id)).ToList();
}
My model has:
Several DeviceStatus attached to one mandatory Device
SeveralDevice attached to one mandatory Panel
When I query DeviceStatus, I need to have Device and Panel attached to it in the query result.
... DeviceStatus.Device is null in the query result.
Here is the Linq Query:
using (var actiContext = new ActigraphyContext())
{
var todayStatus =
from s in actiContext.DeviceStatus.Include(s1 => s1.Device.Panel)
where DbFunctions.TruncateTime(s.TimeStamp) == DbFunctions.TruncateTime( DateTimeOffset.Now)
&& s.Device.Panel.Mac == mac
&& (s.Device.Ty == 4 || s.Device.Ty == 9)
select s;
// var tempList = todayStatus.toList();
var todayLastStatus =
from s in todayStatus.Include(s1 => s1.Device.Panel)
let lastTimeStamp = todayStatus.Max(s1 => s1.TimeStamp)
where s.TimeStamp == lastTimeStamp
select s;
var requestResult = todayLastStatus.FirstOrDefault();
return requestResult;
}
If I uncomment the line // var tempList = todayStatus.toList();, where tempList is not used, it works: requestResult.Device is set!
But the bad side is todayStatus.toList triggers a request that brings a huge amount of data.
So how to get the DeviceStatus with its relative objects ?
Note: the database behind is SQL Server 2012
When you call an Include() over a LINQ query, it performs Eagerly Loading.
As documented in MSDN:
Eager loading is the process whereby a query for one type of entity also loads related entities as part of the query. Eager loading is achieved by use of the Include method.
When the entity is read, related data is retrieved along with it. This typically results in a single join query that retrieves all of the data that's needed. You specify eager loading by using the Include method.
So you need to call the .toList() to complete the query execution.
Since the data is huge, you can pickup relative specific columns as per your requirement by using the Select clause.
var todayStatus =
from s in actiContext.DeviceStatus
.Include(s1 => s1.Device.Panel.Select(d => new
{
d.DeviceId,
d.DeviceName,
d.PanelID
}))
where DbFunctions.TruncateTime(s.TimeStamp) == DbFunctions.TruncateTime( DateTimeOffset.Now)
&& s.Device.Panel.Mac == mac
&& (s.Device.Ty == 4 || s.Device.Ty == 9)
select s;
var tempList = todayStatus.toList();
The query doesn't actually run until you do a call like ToList(), which is why uncommenting that line works. If the query is bringing back too much data, then you need to change the query to narrow down the amount of data you're bringing back.
Ok this request is a more simple way to achieve this:
using (var actiContext = new ActigraphyContext())
{
var todayLastStatus =
from s in actiContext.DeviceStatus.Include(s1 => s1.Device.Panel)
where DbFunctions.TruncateTime(s.TimeStamp) == DbFunctions.TruncateTime( DateTimeOffset.Now)
&& s.Device.Panel.Mac == mac
&& (s.Device.Ty == 4 || s.Device.Ty == 9)
orderby s.TimeStamp descending
select s;
var requestResult = todayLastStatus.Take(1).FirstOrDefault();
return requestResult;
}
But the question remains: why didn't I get the relative object in my first request ?
I have a log table in my db and wants to fetch only those records which are added most recently based on the column name RowCreateDate, this is how I am trying to achieve the records which is bringing the rows from the db but I feel may be there is a better way to achieve the same.
using (var context = new DbEntities())
{
// get date
var latestDate = context.Logs.Max(o => o.RowCreateDate);
if(latestDate!=null)
{
lastDate = new DateTime(latestDate.Value.Year, latestDate.Value.Month, latestDate.Value.Day,00,00,00);
logs = context.Logs.Where( o.RowCreateDate >= lastDate).ToList();
}
}
What i need to know I am doing right or there would another better way?
Yet another option:
context.Logs.Where(c => DbFunctions.TruncateTime(c.RowCreateDate) == DbFunctions.TruncateTime(context.Logs.Max(o => o.RowCreateDate)))
This reads explicitly like what you want (get all rows with date equals max date) and will also result in one query (not two as you might have expected).
You can't simplify this code because LINQ to Entities does not support TakeWhile method.
You can use
using (var context = new DbEntities())
{
// get date
var latestDate = context.Logs.Max(o => o.RowCreateDate);
if(latestDate!=null)
{
lastDate = new DateTime(latestDate.Value.Year, latestDate.Value.Month, latestDate.Value.Day,00,00,00);
logs = context.Logs
.OrderBy(o => o.RowCreateDate)
.AsEnumerable()
.TakeWhile(o => o.RowCreateDate >= lastDate);
}
}
BUT it takes all your data from DB, which is not very good and I do not recommend it.
I think this will do (if we assume you want to get top 3 most recent record):
var topDates = context.Logs.OrderByDescending(x=>x.RowCreateDate).Take(3)
First, I think that your code is fine. I don't see the problem with the two queries. But if you want to simplify it you use TruncateTime, like this:
IGrouping<DateTime?, Logs> log =
context.Logs.GroupBy(x => DbFunctions.TruncateTime(x.RowCreateDate))
.OrderByDescending(x => x.Key).FirstOrDefault();
It will return a grouped result with the logs created during the last day for RowCreateDate.