I have started using performance wizard in visual studio 2012 because there was a slow method which is basically used to get all users from the datacontext. I fixed the initial problem but I am now curious if I can make it faster.
Currently I am doing this:
public void GetUsers(UserManagerDashboard UserManagerDashboard)
{
try
{
using (GenesisOnlineEnties = new GenesisOnlineEntities())
{
var q = from u in GenesisOnlineEnties.vw_UserManager_Users
select u;
foreach (var user in q)
{
User u = new User();
u.UserID = user.UserId;
u.ApplicationID = user.ApplicationId;
u.UserName = user.UserName;
u.Salutation = user.Salutation;
u.EmailAddress = user.Email;
u.Password = user.Password;
u.FirstName = user.FirstName;
u.Surname = user.LastName;
u.CompanyID = user.CompanyId;
u.CompanyName = user.CompanyName;
u.GroupID = user.GroupId;
u.GroupName = user.GroupName;
u.IsActive = user.IsActive;
u.RowType = user.UserType;
u.MaximumConcurrentUsers = user.MaxConcurrentUsers;
u.ModuleID = user.ModuleId;
u.ModuleName = user.ModuleName;
UserManagerDashboard.GridUsers.users.Add(u);
}
}
}
catch (Exception ex)
{
}
}
It's a very straight forward method. Connect to the database using entity framework, get all users from the view "vw_usermanager_users" and populate the object which is part of a collection.
I was casting ?int to int and I changed the property to ?int so no cast is needed. I know that it is going to take longer because I am looping through records. But is it possible to speed this query up?
Ok, first things first, what does your vw_UserManager_Users object look like? If any of those properties you're referencing are navigational properties:-
public partial class UserManager_User
{
public string GroupName { get { return this.Group.Name; } }
// See how the getter traverses across the "Group" relationship
// to get the name?
}
then you're likely running face-first into this issue - basically you'll be querying your database once for the list of users, and then once (or more) for each user to load the relationships. Some people, when faced with a problem, think "I know, I'll use an O/RM". Now they have N+1 problems.
You're better to use query projection:-
var q = from u in GenesisOnlineEnties.vw_UserManager_Users
select new User()
{
UserID = u.UserId,
ApplicationId = u.ApplicationID,
GroupName = u.Group.Name, // Does the join on the database instead
...
};
That way, the data is already in the right shape, and you only send the columns you actually need across the wire.
If you want to get fancy, you can use AutoMapper to do the query projection for you; saves on some verbosity - especially if you're doing the projection in multiple places:-
var q = GenesisOnlineEnties.vw_UserManager_Users.Project().To<User>();
Next up, what grid are you using? Can you use databinding (or simply replace the Grid's collection) rather than populating it one-by-one with the results from your query?:-
UserManagerDashboard.GridUsers.users = q.ToList();
or:-
UserManagerDashboard.GridUsers.DataSource = q.ToList();
or maybe:-
UserManagerDashboard.GridUsers = new MyGrid(q.ToList());
The way you're adding the users to the grid right now is like moving sand from one bucket to another one grain at a time. If you're making a desktop app it's even worse because adding an item to the grid will probably trigger a redraw of the UI (i.e. one grain at a time and, describing every grain in the bucket to your buddy after each one). Either way you're doing unnecessary work, see what methods your grid gives you to avoid this.
How many users are in the table? If the number is very large, then you'll want to page your results. Make sure that the paging happens on the database rather than after you've got all the data - otherwise it kind of defeats the purpose:-
q = q.Skip(index).Take(pageSize);
though bear in mind that some grids interact with IQueryable to do paging out-of-the-box, in that case you'd just pass q to the grid directly.
Those are the obvious ones. If that doesn't fix your problem, post more code and I'll take a deeper look.
Yes, by turning off change tracking:
var q = from u in GenesisOnlineEnties.vw_UserManager_Users.AsNoTracking()
select u;
Unless you are using all the properties on the entity you can also select only the columns you want.
var q = from u in GenesisOnlineEnties.vw_UserManager_Users.AsNoTracking()
select new User
{
UserId = u.UserId,
...
}
Related
I've been dealing with an issue lately, and although i have some solutions in mind, i'd like to find the best one from every point of view.
Let's say i have a WPF app with EF Core. There are about 3000 customers in my database (SQLite in my case, but in the future this should also work with slower ones). When the user opens the customer's list, i'm loading only some of them (quantity = 50, page = 0), in alphabetical order. As soon as the user scrolls down to the bottom, 50 more are loaded (quantity = 50, page = 1).
CustomerRepository.GetQueryableAll().Skip(page * quantity).Take(quantity).ToList();
Everything works fine. Here comes the problem though: there's a button to create a new customer, which opens a modal window. Let's say the user creates a customer with starting letter W. As soon as he/she hits SAVE, the new customer is saved to the database, the window is closed, and the list must be reloaded. But loading the whole list until W is, of course, really slow.
So far, i've tried to query the database in a background task and store how many customers start with each letter of the database in a static Dictionary: as soon as SAVE is hit, i can guess more or less how many "pages" to Skip() in the database and get the group of 50 in which the new customer will be. It works, it's quite fast, but i'm worried that it won't work in countries with non Latin alphabets:
public async Task<Dictionary<char, int>> GetCustomersByInitialsCount()
{
return await Task.Run(async delegate
{
var dictionary = new Dictionary<char, int>();
for (char c = 'A'; c <= 'Z'; c++)
{
var count = await CustomerRepository.GetCustomerCountStartingWith(c.ToString());
dictionary.Add(c, count);
}
return dictionary;
});
}
[... and in the repository:]
public async Task<int> GetCustomerCountStartingWith(string startingLetter)
{
using (var dbContext = new MyDbContext())
{
return await dbContext.Set<Customer>().CountAsync(p => p.LastName.ToUpper().StartsWith(startingLetter.ToUpper()));
}
}
Otherwise, instead of this background query, i could also try to "guess" the right page depending on the starting char, but i'm still puzzled by the unexpected outcomes i could have with non latin languages.
If anybody knows better tools or have any other useful ideas, i'll gladly consider them!
Thank you very much in advance and happy coding.
What if you add a request to get all the first "letters" in your table ?
public async Task<List<string>> GetCustomerFirstLetter()
{
using (var dbContext = new MyDbContext())
{
return await dbContext.Set<Customer>().Select(x => x.lastName.Substring(0, 1)).Distinct().ToList();
}
}
and then
public async Task<Dictionary<char, int>> GetCustomersByInitialsCount()
{
return await Task.Run(async delegate
{
var dictionary = new Dictionary<char, int>();
var letters = GetCustomerFirstLetter();
foreach(letter in letters)
{
var count = await CustomerRepository.GetCustomerCountStartingWith(letter);
dictionary.Add(letter, count);
}
return dictionary;
});
}
Alternative solution. A little bit more efficient from my point of view
Your problem boils down to how to get new customer's row number in whole dataset ordered by customer's name.
First of all, in plain SQL for SQLite or MSSQL you may solve your problem of getting right page number with ROW_NUMBER function. Query example:
SELECT TOP 1 rnd.rownum, rnd.LastName
from (SELECT ROW_NUMBER() OVER( ORDER BY c.LastName) AS rownum, c.LastName
FROM [Customer] c) rnd
WHERE rnd.LastName = '<your new customers name here>'
So, after getting exact rownumber value and having already page count param you can easily calculate needed page.
Getting back to your code. This feature can be implemented in EF with overloaded version of Select method, but unfortunately, it has not been implemented in EF Core for IQueryable yet (see this).
But you can still pass exact query right to db using FromSql method.
Solution consists of two steps:
To get required data you need to define Query for model builder this way (additional fields just for example, youl need RowNum only):
protected override void OnModelCreating(ModelBuilder modelBuilder)
{
modelBuilder.Query<CustomerRownNum>();
}
public class CustomerRownNum
{
public long RowNum { get; set; }
public Guid Id { get; set; }
public string LastName { get; set; }
}
Then you need to pass mentioned above SQL query to context's Query method this way:
string customerLastName = "<your customer's last name>";
var result = dbContext.Query<CustomerRownNum>().FromSql(
#"select top 1 rnd.RowNum, rnd.Id, rnd.LastName
from
(SELECT ROW_NUMBER() OVER( ORDER BY c.LastName) AS RowNum
, c.Id, c.LastName
FROM [Customer] c) rnd
WHERE rnd.LastName = {0}", customerLastName).FirstOrDefault();
Finally you'll get data you needed right in result variable.
Hope that helps!
In my service, first I generate 40,000 possible combinations of home and host countries, like so (clientLocations contains 200 records, so 200 x 200 is 40,000):
foreach (var homeLocation in clientLocations)
{
foreach (var hostLocation in clientLocations)
{
allLocationCombinations.Add(new AirShipmentRate
{
HomeCountryId = homeLocation.CountryId,
HomeCountry = homeLocation.CountryName,
HostCountryId = hostLocation.CountryId,
HostCountry = hostLocation.CountryName,
HomeLocationId = homeLocation.LocationId,
HomeLocation = homeLocation.LocationName,
HostLocationId = hostLocation.LocationId,
HostLocation = hostLocation.LocationName,
});
}
}
Then, I run the following query to find existing rates for the locations above, but also include empty the missing rates; resulting in a complete recordset of 40,000 rows.
var allLocationRates = (from l in allLocationCombinations
join r in Db.PaymentRates_AirShipment
on new { home = l.HomeLocationId, host = l.HostLocationId }
equals new { home = r.HomeLocationId, host = (Guid?)r.HostLocationId }
into matches
from rate in matches.DefaultIfEmpty(new PaymentRates_AirShipment
{
Id = Guid.NewGuid()
})
select new AirShipmentRate
{
Id = rate.Id,
HomeCountry = l.HomeCountry,
HomeCountryId = l.HomeCountryId,
HomeLocation = l.HomeLocation,
HomeLocationId = l.HomeLocationId,
HostCountry = l.HostCountry,
HostCountryId = l.HostCountryId,
HostLocation = l.HostLocation,
HostLocationId = l.HostLocationId,
AssigneeAirShipmentPlusInsurance = rate.AssigneeAirShipmentPlusInsurance,
DependentAirShipmentPlusInsurance = rate.DependentAirShipmentPlusInsurance,
SmallContainerPlusInsurance = rate.SmallContainerPlusInsurance,
LargeContainerPlusInsurance = rate.LargeContainerPlusInsurance,
CurrencyId = rate.RateCurrencyId
});
I have tried using .AsEnumerable() and .AsNoTracking() and that has sped things up quite a bit. The following code shaves several seconds off of my query:
var allLocationRates = (from l in allLocationCombinations.AsEnumerable()
join r in Db.PaymentRates_AirShipment.AsNoTracking()
But, I am wondering: How can I speed this up even more?
Edit: Can't replicate foreach functionality in linq.
allLocationCombinations = (from homeLocation in clientLocations
from hostLocation in clientLocations
select new AirShipmentRate
{
HomeCountryId = homeLocation.CountryId,
HomeCountry = homeLocation.CountryName,
HostCountryId = hostLocation.CountryId,
HostCountry = hostLocation.CountryName,
HomeLocationId = homeLocation.LocationId,
HomeLocation = homeLocation.LocationName,
HostLocationId = hostLocation.LocationId,
HostLocation = hostLocation.LocationName
});
I get an error on from hostLocation in clientLocations which says "cannot convert type IEnumerable to Generic.List."
The fastest way to query a database is to use the power of the database engine itself.
While Linq is a fantastic technology to use, it still generates a select statement out of the Linq query, and runs this query against the database.
Your best bet is to create a database View, or a stored procedure.
Views and stored procedures can easily be integrated into Linq.
Material Views ( in MS SQL ) can further speed up execution, and missing indexes are by far the most effective tool in speeding up database queries.
How can I speed this up even more?
Optimizing is a bitch.
Your code looks fine to me. Make sure to set the index on your DB schema where it's appropriate. And as already mentioned: Run your Linq against SQL to get a better idea of the performance.
Well, but how to improve performance anyway?
You may want to have a glance at the following link:
10 tips to improve LINQ to SQL Performance
To me, probably the most important points listed (in the link above):
Retrieve Only the Number of Records You Need
Turn off ObjectTrackingEnabled Property of Data Context If Not
Necessary
Filter Data Down to What You Need Using DataLoadOptions.AssociateWith
Use compiled queries when it's needed (please be careful with that one...)
I am returning a list of restaurants that pulls information from the RESTAURANT, CUISINE, CITY, and STARRATING tables. I want to get a list of each restaurant with its associated city and cuisine along with the average rating in the STARRATING table. This is what I have, so far ... Thanks in advance.
RestaurantsEntities db = new RestaurantsEntities();
public List<RESTAURANT> getRestaurantsWRating(string cuisineName, string cityName, string priceName, string ratingName)
{
var cuisineID = db.CUISINEs.First(s => s.CUISINE_NAME == cuisineName).CUISINE_ID;
List<RESTAURANT> result = (from RESTAURANT in db.RESTAURANTs.Include("CITY").Include("CUISINE").Include("STARRATING")
where RESTAURANT.CUISINE_ID == cuisineID
orderby RESTAURANT.REST_NAME ascending
select RESTAURANT).ToList();
return result;
}
From what you have it looks like Restaurant has a STARRATING collection. If so, this is what you can do:
from r in db.Restaurants
where r.CUISINE_ID == cuisineID
orderby r.REST_NAME ascending
select new {
Restaurant = r,
City = r.CITY,
Cuisine = r.CUISINE,
AvgRating = r.STARRATING.Average(rt => rt.Rating)
}
You'd need to give more informations about your classes and associations (preferably a class diagram) if this is not right.
(BTW using capitals for class and property names is not conventional).
First I would wrap your whole code block above in a using statement:
using(RestaurantEntities db = new RestaurantEntities())
{
...
}
This will help with cleanup for the EF context.
The way I would typically do this is if you have control of your database, I would create a view in the database that does this work, add the view to your entity model and query the view. This simplifies the whole process and offloads the work of the aggregation to the database.
If you don't have control over the database or don't prefer the view technique then I would query using the include technique as you have done and then add a partial class to RESTAURANT (if using model-first) in order to add an AverageRating property and then manually calculate the average for each related STARRATING set of related rows and apply the resultant value to the added property. You could do this through linq to objects once you have all the data back. This technique would not scale very well as more data is accumulated unless you are confident you never return but one or a few RESTAURANT instances. You could use something like:
//query data as you have done above...
foreach(RESTAURANT r in result)
{
if(r.STARRATING.Count() > 0)
{
r.AverageRating = r.STARRATING.Average(rating => rating.Value); //.Value is your field name
}
else
{
r.AverageRating = 0; // or whatever default you prefer...
}
}
Hope this helps.
I'm new to Entify Framework so this is probably a very basic question. In a WinForms application I have a data entry page that works fine until I add a listbox and try to update the database with the selections that have been made.
On the form the user selects a file to upload and specifies one or more departments that can access the file. Here's how I thought it would work:
using (var ctx = new FCEntities())
{
var batch = new Batch() { Description = txtDescription.Text, Filename = filename, Departments = (System.Data.Objects.DataClasses.EntityCollection<Department>)lstDepartments.SelectedItems };
ctx.AddToBatches(batch);
ctx.SaveChanges();
}
But when this didn't work I did some research and learned that I can't cast the SelectedItems to EntityCollection so I decided to copy the items from the original collection into a new collection and then use the new collection as follows:
using (var ctx = new FCEntities())
{
var departments = new System.Data.Objects.DataClasses.EntityCollection<Department>();
foreach (var department in lstDepartments.SelectedItems)
{
departments.Add((Department)department);
}
var batch = new Batch() {Description = txtDescription.Text, Filename = filename, Departments=departments };
ctx.AddToBatches(batch);
ctx.SaveChanges();
}
This didn't work either and gave this error on the departments.Add line:
"An object that is attached to an ObjectContext cannot be added to an
EntityCollection or EntityReference that is not associated with a
source object."
I don't understand because it doesn't appear to me that the department object is attached to the ObjectContext? I'm obviously missing something fundamental, so any advice and/or links to examples of how others do this would be appreciated.
I wanted to leave an answer to this in case someone else runs into this someday. The comments left by Wiktor were helpful in getting me in the right direction. I decided I had a lack of fundamental understanding so I did some reading on MSDN and was able to resolve my issue.
The datamodel behind this existed of three tables: Batches, Departments, and Batches_Departments which allowed for a many to many relationship between Batches and Departments.
The problem with my original code/logic, in a nutshell, was that the Department objects in the ListBox were associated with a different context than the one I was using in my Save method. EF didn't like this for obvious reasons (at least now they are obvious), so in the save method I used the ID from the selected Departments to get a reference to the same Department in the current context. I could then add this Department to the newly created batch.
Here's what the code now looks like:
using (var ctx = new FCEntities())
{
var batch = new Batch() { Description = txtDescription.Text, Filename = filename};
foreach (var department in lstDepartments.CheckedItems)
{
var dept = (from d in ctx.Departments where d.DepartmentID == ((Department)department).DepartmentID select d).First();
batch.Departments.Add(dept);
}
ctx.Batches.AddObject(batch);
ctx.SaveChanges();
}
Hopefully this helps someone else who is dealing with the same issue.
This section simply reads from an excel spreadsheet. This part works fine with no performance issues.
IEnumerable<ImportViewModel> so=data.Select(row=>new ImportViewModel{
PersonId=(row.Field<string>("person_id")),
ValidationResult = ""
}).ToList();
Before I pass to a View I want to set ValidationResult so I have this piece of code. If I comment this out the model is passed to the view quickly. When I use the foreach it will take over a minute. If I hardcode a value for item.PersonId then it runs quickly. I know I'm doing something wrong, just not sure where to start and what the best practice is that I should be following.
foreach (var item in so)
{
if (db.Entity.Any(w => w.ID == item.PersonId))
{
item.ValidationResult = "Successful";
}
else
{
item.ValidationResult = "Error: ";
}
}
return View(so.ToList());
You are now performing a database call per item in your list. This is really hard on your database and thus your performance. Try to itterate trough your excel result, gather all users and select them in one query. Make a list from this query result (else the query call is performed every time you access the list). Then perform a match between the result list and your excel.
You need to do something like this :
var ids = so.Select(i=>i.PersonId).Distinct().ToList();
// Hitting Database just for this time to get all Users Ids
var usersIds = db.Entity.Where(u=>ids.Contains(u.ID)).Select(u=>u.ID).ToList();
foreach (var item in so)
{
if (usersIds.Contains(item.PersonId))
{
item.ValidationResult = "Successful";
}
else
{
item.ValidationResult = "Error: ";
}
}
return View(so.ToList());