In a .NET Core Project I'm using EntityFramework and I have a DbContext (shopContext) injected in my class repository. I have the next query:
var res = shopContext.Orders.Where(x => x.Status == 1).Sum(p => p.Total);
Occasionally, the Orders table is doing maintenance tasks and the table is locked. For this query, I can't wait to the maintenance tasks and I need access to the table with the IsolationLevel.ReadUncommited option in the transaction:
using (var transaction = mutuaContext.Database.BeginTransaction(IsolationLevel.ReadUncommitted))
{
var res = shopContext.Orders.Where(x => x.Status == 1).Sum(p => p.Total);
}
The problem is that I only want that the context execute the query with this IsolationLevel configuration in these query, but the next queries continue executing although the table is locked yet.
Why are the following queries not waiting for the Table to be unlocked?
Example of my code:
using (var transaction = mutuaContext.Database.BeginTransaction(IsolationLevel.ReadUncommitted))
{
var res = shopContext.Orders.Where(x => x.Status == 1).Sum(p => p.Total); // this code would be executed
}
var total = shopContext.Orders.Where(x => x.Status == 0).Sum(p => p.Total); // this code would NOT be executed but is executed
I don't understand how the context get the transaction configuration. I would like that someone explain it to me.
I tried call to transaction.Commit() after fist query, but still not working.
use
yourContext.Database.BeginTransaction(IsolationLevel.ReadUncommitted)
// your normal queries via yourContext goes here
// do not forget to end the transaction
You can use raw SQL query (there is a similar SqlQuery() method for EF6, as well) and specify with (nolock) table hint. Something like this:
var res = shopContext.Orders.FromSqlRaw("select sum(Total) from dbo.Orders with (nolock) where Status = 1").ToList();
However, once you will deploy this into production environment and put your code under a decent concurrent load, most probably you will not like the outcome.
UPD: For EF Core 2.2, the syntax is a bit different:
var res = shopContext.Orders.FromSql("select * from Orders with(nolock)")
.Where(x => x.Status == 1).Sum(p => p.Total);
Related
I am trying to count the amount of rows for a specific Account. I configure entityframework to use lazy loading.
I load the account like this:
Account? account = FindSingleOrDefault(x => x.IdCompanyOrUser == id & x.AccountType == accountType);
and later on if necessary I try to get the count of the virtual property loginattempts. I do this by executing this:
account.LoginAttempts.Count();
I read everywhere that this should generate a select count(*) but instead my entityframework is generating the following query:
Executed DbCommand (4ms) [Parameters=[#__p_0='?' (DbType = Int64), #__p_1='?' (DbType = Int32)], CommandType='Text', CommandTimeout='30']
SELECT l."Id", l."AccountIdCompanyOrUser", l."AccountType", l."FailReason", l."Ip", l."Success", l."Time"
FROM "LoginAttempt" AS l
WHERE (l."AccountIdCompanyOrUser" = #__p_0) AND (l."AccountType" = #__p_1) <s:Microsoft.EntityFrameworkCore.Database.Command>
The count result is correct but with this query, it will be loading all the entities first. This could have a serious performance impact when my application is filled with millions of records. For this reason, I really need it to use the select count.
my configuration of entity framework is the following:
builder.Services.AddDbContext<AuthDbContext>(options =>
{
options.UseLazyLoadingProxies();
});
modelBuilder.Entity<Account>()
.HasKey(table => new
{
table.IdCompanyOrUser,
table.AccountType
});
modelBuilder.Entity<Account>()
.Navigation(e => e.LoginAttempts);
the login attempt is a virtual list of the model LoginAttempt which also has its own table in the database.
I also tried this other options and it didn't work:
account.LoginAttempts.Count();
account.LoginAttempts.AsQueryable().Count();
account.LoginAttempts.AsQueryable().ToList().Count();
Additional information:
Version Microsoft.AspNetCore.Identity.EntityFrameworkCore 6.0.2
Version Microsoft.EntityFrameworkCore.Proxies 6.0.5
.Net 6
It is expected behavior. When you access Lazy Loading navigation property - EF loads property, in your case whole collection. Then Count is performed on loaded collection.
To get just count of items you have to run query again:
var attemptsCount = _context.Accounts
.Where(a => a.Id == account.Id)
.Select(a => a.LoginAttempts.Count())
.First();
Or faster variant
var attemptsCount = _context.LoginAttempts
.Where(a => a.AccountId == account.Id)
.Count();
You can use 'where' and 'select' combined
db.Table
.Where(x => x.IdCompanyOrUser == id & x.AccountType == accountType)
.Select(x => x.LoginAttemps.Count()).FirstOrDefault()
Edit:
As gert arnolt pointed out below, using Include is unnecessary so I removed it.
Include is needed ONLY for loading related entities. It has zero
influence on filters. LINQ translator will still create SQL because it
uses just metadata information - what is needed for filter
So I've developed a dashboard which queries a database. The database has data stored in it from google analytics for a website we have.
I'm using ASP.NET MVC 5, EF, Linq with Telerik controls/widgets. The controller instantiates a service layer where I have my db context and business logic. Each svc.method() pertains to a specific result set I'm after that I package up in the VM for unpackaging into a widget within the view.
Currently, the response time in the network tab of Google Chrome is 5.6 seconds. I've illustrated one of the 8 methods to show you my approach.
My question is; how can I improve performance so that the page loads faster? Would making each method async improve it?
Thanks in advance for any advice you can provide.
Controller:
public ActionResult WebStats()
{
//other code removed for brevity
//Service layer where the db is queried and the business logic is performend
WebStatsService svc = new WebStatsService();
//view model
WebStatsViewModel vm = new WebStatsViewModel();
vm.PageViews = svc.GetPageViews(vm);
vm.UniquePageViews = svc.GetUniquePageViews(vm);
vm.UserRatioByCountry = svc.GetUserRatioByCountry(vm);
vm.PageViewsByCountry = svc.GetPageViewsByCountry(vm);
vm.TopTenHealthCenters = svc.GetTopTenHealthCenters(vm);
vm.UserTypeRatio = svc.GetUserTypeRatio(vm);
vm.TopTenHealthCentersByDateRange = svc.GetTopTenHealthCentersByDateRange(vm);
vm.ReferralSources = svc.GetTopTenReferralSources(vm);//Get top 10 referral paths
return View(vm);
}
Service:
public List<PageViews> GetPageViews(WebStatsViewModel vm)
{
using (ApplicationDbContext db = new ApplicationDbContext())
{
List<PageViews> pageViewStats = new List<PageViews>();
var results = db.PageStats.Where(x => (vm.CMS.Equals("All") || x.Source.Equals(vm.CMS))
&& (vm.HealthCenter.Equals("All") || x.HealthSectionName.Equals(vm.HealthCenter))
&& (vm.Country.Equals("All") || x.Country.Equals(vm.Country))
&& (vm.City.Equals("All") || x.City.Equals(vm.City))
&& (x.Date >= vm.StartDate)
&& (x.Date <= vm.EndDate)
).Select(x => new
{
Date = x.Date,
Total = x.PageViews
}).ToList();
var distinctDate = results.OrderBy(x => x.Date).Select(x => x.Date).Distinct();
foreach (var date in distinctDate)
{
PageViews pageViewStat = new PageViews();
pageViewStat.Date = date.Value.ToShortDateString();
pageViewStat.Total = results.Where(x => x.Date == date).Sum(x => x.Total);
pageViewStats.Add(pageViewStat);
}
return pageViewStats;
}
}
Here are some tips for EF queries:
(1) Avoid mixing constant and actual predicate in dynamic filters like this:
(vm.CMS.Equals("All") || x.Source.Equals(vm.CMS))
It might look concise, but generates awful and inefficient SQL. Instead, use if statements and chained Where:
// Base query including static filters
var query = db.PageStats.AsQueryable();
// Apply dynamic filters
if (!vm.CMS.Equals("All"))
query = query.Where(x => x.Source.Equals(vm.CMS));
// ...
// The rest of the query
query = query.Select(...
(2) Try returning as less data as possible from the SQL query.
For instance, your query is populating a list with (Date, Total) pairs, which you then manually (and not very efficiently) group by Date and take Sum(Total). Instead, you can make the EF query directly return that grouped/aggregated data.
Applying all that to your example would lead to something like this:
using (ApplicationDbContext db = new ApplicationDbContext())
{
var query = db.PageStats
.Where(x => x.Date >= vm.StartDate && x.Date <= vm.EndDate);
if (!vm.CMS.Equals("All"))
query = query.Where(x => x.Source.Equals(vm.CMS));
if (!vm.HealthCenter.Equals("All"))
query = query.Where(x => x.HealthSectionName.Equals(vm.HealthCenter));
if (!vm.Country.Equals("All"))
query = query.Where(x => x.Country.Equals(vm.Country));
if (!vm.City.Equals("All"))
query = query.Where(x => x.City.Equals(vm.City));
query = query
.GroupBy(x => x.Date)
.Select(g => new
{
Date = g.Key,
Total = g.Sum(x => x.PageViews)
})
.OrderBy(x => x.Date);
var pageViewStats = query
.AsEnumerable() // SQL query ends here
.Select(x => new PageViews
{
Date = x.Date.Value.ToShortDateString(),
Total = x.Total
})
.ToList();
return pageViewStats;
}
You can try and compare the performance with the original.
(Note: for this specific query we need to use two projections - one temporary in SQL query and one final in the in memory query. This is because of the need of ToShortDateString() method which is not supported for the SQL query. In most of the cases a single final projection in the SQL query would be sufficient.)
Some tips:
Indexes - index columns that appear in the where clause of select operations, use SQL profiler to detect 'table scan' operations and add indexes to avoid them (replace them with index search or clustered index search)
Caching - store the trace from SQL profiler above to a table in DB (SQL Profiler can do it) and group SQL commands by sql text, this may show some repeating selects that can be avoided by caching
Glimpse - can count SQL commands per web request, the number can be suprising if the web application has not been optimized yet. Glimpse can tell much more, for example how much time of the total time of a web request is spent on the server and how much time in the web browser rendering the page.
as the last resort, write your own SQL for the most exposed queries
I am trying to create a query for entity framework that will allow me to take a list of ids and update a field associated with them.
Example in SQL:
UPDATE Friends
SET msgSentBy = '1234'
WHERE id IN (1, 2, 3, 4)
How do I convert the above into entity framework?
something like below
var idList=new int[]{1, 2, 3, 4};
using (var db=new SomeDatabaseContext())
{
var friends= db.Friends.Where(f=>idList.Contains(f.ID)).ToList();
friends.ForEach(a=>a.msgSentBy='1234');
db.SaveChanges();
}
UPDATE:
you can update multiple fields as below
friends.ForEach(a =>
{
a.property1 = value1;
a.property2 = value2;
});
The IQueryable.ToQueryString method introduced in Entity Framework Core 5.0 may help with this scenario, if you are willing to have some raw SQL appearing in your code. This method will generate SQL that can be included in a raw SQL query to perform a bulk update of records identified by that query.
For example:
using var context = new DbContext();
var ids = new List<int>() { 1, 2, 3, 4 };
var query = context.Friends.Where(_ => ids.Contains(_.id)).Select(_ => _.id);
var sql = $"UPDATE Friends SET msgSentBy = {{0}} WHERE id IN ({query.ToQueryString()})";
context.Database.ExecuteSqlRaw(sql, "1234");
The major drawback of this approach is the use of raw SQL. However I don't know of any reasonable way to avoid that with current Entity Framework Core capabilities - you're stuck with this caveat, or the caveats of other answers posted here such as:
Introducing a dependency on another library like https://github.com/yangzhongke/Zack.EFCore.Batch.
Using DbContext.SaveChanges() which will update records one at a time rather than doing a bulk update.
If (when) the following issue is addressed in the future then we are likely to get a better answer here: Bulk (i.e. set-based) CUD operations (without loading data into memory) #795
var idList=new int[]{1, 2, 3, 4};
var friendsToUpdate = await Context.Friends.Where(f =>
idList.Contains(f.Id).ToListAsync();
foreach(var item in previousEReceipts)
{
item.msgSentBy = "1234";
}
You can use foreach to update each element that meets your condition.
Here is an example in a more generic way:
var itemsToUpdate = await Context.friends.Where(f => f.Id == <someCondition>).ToListAsync();
foreach(var item in itemsToUpdate)
{
item.property = updatedValue;
}
Context.SaveChanges()
In general you will most probably use async methods with await for db queries.
I have created a library to batch delete or update records with a round trip on EF Core 5.
Sample code as follows:
await ctx.DeleteRangeAsync(b => b.Price > n || b.AuthorName == "zack yang");
await ctx.BatchUpdate()
.Set(b => b.Price, b => b.Price + 3)
.Set(b=>b.AuthorName,b=>b.Title.Substring(3,2)+b.AuthorName.ToUpper())
.Set(b => b.PubTime, b => DateTime.Now)
.Where(b => b.Id > n || b.AuthorName.StartsWith("Zack"))
.ExecuteAsync();
Github repository: https://github.com/yangzhongke/Zack.EFCore.Batch
Report: https://www.reddit.com/r/dotnetcore/comments/k1esra/how_to_batch_delete_or_update_in_entity_framework/
The best way to do a masive update with Entity Framework 7 is like this:
context.Friends
.Where(f => f.Id <= 1_000)
.ExecuteUpdate(f => f.SetProperty(x => x.Name, x => $"Updated {x.Name}"));
Referenece: https://learn.microsoft.com/en-us/ef/core/what-is-new/ef-core-7.0/whatsnew#executeupdate-and-executedelete-bulk-updates
I have a following foreach statement:
foreach (var articleId in cleanArticlesIds)
{
var countArt = context.TrackingInformations.Where(x => x.ArticleId == articleId).Count();
articleDictionary.Add(articleId, countArt);
}
Database looks like this
TrackingInformation(Id, ArticleId --some stuff
Article(Id, --some stuff
what I want to do is to get all the article ids count from TrackingInformations Table.
For example:
ArticleId:1 Count:1
ArticleId:2 Count:8
ArticleId:3 Count:5
ArticleId:4 Count:0
so I can have a dictionary<articleId, count>
Context is the Entity Framework DbContext. The problem is that this solution works very slow (there are > 10k articles in db and they should rapidly grow)
Try next query to gather grouped data and them add missing information. You can try to skip Select clause, I don't know if EF can handle ToDictionary in good manner.
If you encounter Select n + 1 problem (huge amount of database requests), you can add ToList() step between Select and ToDictionary, so that all required information will be brought into memory.
This depends all your mapping configuration, environment, so in order to get good performance, you need to play a little bit with different queries. Main approach is to aggregate as much data as possible at database level with few queries.
var articleDictionary =
context.TrackingInformations.Where(trackInfo => cleanArticlesIds.Contains(trackInfo.ArticleId))
.GroupBy(trackInfo => trackInfo.ArticleId)
.Select(grp => new{grp.Key, Count = grp.Count()})
.ToDictionary(info => "ArticleId:" + info.Key,
info => info.Count);
foreach (var missingArticleId in cleanArticlesIds)
{
if(!articleDictionary.ContainsKey(missingArticleId))
articleDictionary.add(missingArticleId, 0);
}
If TrackingInformation is a navigatable property of Article, then you can do this:
var result=context.Article.Select(a=>new {a.id,Count=a.TrackingInformation.Count()});
Putting it into a dictionary is simple as well:
var result=context.Article
.Select(a=>new {a.id,Count=a.TrackingInformation.Count()})
.ToDictionary(a=>a.id,a=>a.Count);
If TrackingInforation isn't a navigatable property, then you can do:
var result=context.Article.GroupJoin(
context.TrackingInformation,
foo => foo.id,
bar => bar.id,
(x,y) => new { id = x.id, Count = y.Count() })
.ToDictionary(a=>a.id,a=>a.Count);
I get the following error when trying to join an array to the Linq-to-EF query
An error occurred while executing the command definition. See the inner exception for details. Some part of your SQL statement is nested too deeply. Rewrite the query or break it up into smaller queries.
The code is as follows:
var vids = new List<string>();
using (var ctx = new MyDbContext())
{
var qry = ctx.Pickups.Where(p => p.UserName == User.Identity.Name);
if (someBoolean)
{
var v = GetVids(); // get the list of Ids from a web service
vids.AddRange(v);
}
if (vids.Count() > 0)
{
qry = qry.Join(vids, p => p.VId, v => v, (v, p) => p);
}
var data = qry
.Select(p => new
{
// etc.
});
}
The problem is that the web service is not associated with the DB I'm using EF with, otherwise, I'd just do the join against the tables in the DB. The number of Id's I get back from the web service could be upwards of a hundred or so (usually 5-10). If I comment out the Join, the code works fine, so I know the error is in the Join. With just a few (up to about 30) ids in vids, the join works perfectly.
What do you recommend to remedy this problem? The only thing I could think of was to insert the list of IDs into the DB and do the join that way. That doesn't seem too appealing to me.
Try to replace if (vids.Count() > 0) with:
if (vids.Count > 0)
{
qry = qry.Where(arg => vids.Contains(arg.VId));
}
This will work only if vids is less then 2100 elements, as this will be translated to IN (x, y, .., n) condition.
If you use entity framework 3.5, then Contains will not work. You can find possible solution here: 'Contains()' workaround using Linq to Entities?