This one has been a head-scratcher for me. Still kind of new to certain things and EFCore, but I believe I have been doing well. I want to know what could have possibly changed.
I have created an endpoint that gets a specific number of list of Deals, it had been working since it's inception (DotNet Core 2.2), but after updating to 3.1, it doesn't seem to be working correctly.
I know I did this in a weird way, I found it as a solution somewhere else and tried it, but hear me out. I have a Deal, that Deal has a SubCategory (which has an ID, name, etc...), then within the SubCategory, I have a Category (Which has an ID, name, etc...).
To get my specific list of Deals, I called this within the endpoint:
List<Deal> deals = new List<Deal>();
using (var context = new DbContext())
{
deals = await context.Deals
Where(x => x.ApplicationUserId == user.Id)
.Skip(skip)
.Take(take)
.Include(x => x.SubCategory)
.ThenInclude(x => x.Category)
.ToListAsync();
}
There are some other things here, but they are still working without issue that are done the same way.
Now, here's the issue. Say I have 20 deals, I load those deals up, 5 of them can't seem to get the category. No matter how many times I load the endpoint, it returns the category of null for those same 5.
Even weirder, let's say I upload a deal, now I have 21 deals, a different set of deals now seem to return category of null, not the same 5 from before. It's like it's random only when the sequence of deals has been altered in some way, so the deals that had category null before, now have the category back, and a random set of a number of deals are returning category null until a new deal is added or one removed.
I have a fix for this, in terms of deciding to use a Linq Query instead of using this, and this is working without fail, but I want to know what could possibly be causing this weird issue.
FYI... I get 0 errors when this happens, just null categories.
Related
I am new to working with Entity framework, and cannot seem to find any information solving this; or if it is even possible. I am writing code first and trying to get an object from the db.
My structure is that it is possible to make a mainThing. Each mainThing can contain multiple thing1s, with multiple thing2s. However each thing1 can also contain a new thing1 and so on.
Now to the issue, when I am getting this mainThing from the db, I do not know how many levels of thing1s to go into.
My query is like this:
var copy = _db.Mainthing
.Include(x=>x.thing1).ThenInclude(x => x.thing1).ThenInclude(x =>x.thing1).XXX.ThenInclude(x =>x.thing2)
.AsSplitQuery()
.AsNoTrackingWithIdentityResolution()
.FirstOrDefault(x => x.Id == memainthingId);```
where the XXX is is where I do not know how many levels to go down, bc there is no way of knowing how many levels of thing1s there are. Is there a way to find out how many levels it is possible to go down/dynamically get them?
We have the following method that allows us to search a table of Projects for a DataGrid:
public async Task<IEnumerable<Project>> GetFilteredProjects(string searchString)
{
var projects = _context.Projects.Where(p => p.Current);
projects.Include(p => p.Client);
projects.Include(p => p.Architect);
projects.Include(p => p.ProjectManager);
if (!string.IsNullOrEmpty(searchString))
{
projects = projects
.Where(p => p.NormalizedFullProjectName.Contains(searchString)
|| p.Client.NormalizedName.Contains(searchString)
|| p.Architect.NormalizedFullName.Contains(searchString)
|| p.ProjectManager.NormalizedFullName.Contains(searchString));
}
projects = projects.OrderBy(p => p.Name).Take(10);
return await projects.ToListAsync();
}
If we do not use the Include on the projects then the searching is instantaneous. However, after adding them in the search can take over 3 seconds.
We need to include the other Entities to allow the Users to search on them should they want to.
How are we able to improve performance but still keep the Include to allow searching on them?
Without Incldue the method looks like so:
public async Task<IEnumerable<Project>> GetFilteredProjects(string searchString)
{
var projects = _context.Projects.Where(p => p.Current);
if (!string.IsNullOrEmpty(searchString))
{
projects = projects
.Where(p => p.Name.Contains(searchString));
}
projects = projects.OrderBy(p => p.Name).Take(10);
return await projects.ToListAsync();
}
Without Include the performance looks like so:
With Include:
The short answer is that including all the extra entities takes time and effort, thus increasing the load times.
However, there is a flaw in your assumption:
We need to include the other Entities to allow the Users to search on them should they want to.
That is not (necessarily) correct. Filtering happens on the database level. Include tells Entity Framework to load the records from the database. These are two separate things.
Look at the following examples:
_context.Projects
.Include(p => p.Architect)
.Where(p => p.Architect.Name == "Bob")
.ToList()
This will give you a list of projects (and their architects) who have an architect named Bob.
_context.Projects
.Where(p => p.Architect.Name == "Bob")
.ToList()
This will give you a list of projects (without architects) who have an architect named Bob; but it does not actually load the Architect object into memory.
_context.Projects
.Include(p => p.Architect)
.ToList()
This will give you a list of projects (and their architects). It will contain every project, the list is not filtered.
You only need to use Include when you want to do in-memory filtering, i.e. on a collection that was already loaded from the database.
Whether that is the case for you depends on this part:
projects = projects
.Where(p => p.NormalizedFullProjectName.Contains(searchString)
|| p.Client.NormalizedName.Contains(searchString)
|| p.Architect.NormalizedFullName.Contains(searchString)
|| p.ProjectManager.NormalizedFullName.Contains(searchString));
If NormalizedFullProjectName (and the other properties) are database columns, then EF is able to perform the filtering at the database level. You do not need the Include for filtering the items.
If NormalizedFullProjectName (and the other properties) are not database columns, then EF will first have to load the items in memory before it can apply the filter. In this case, you do need the Include, because the architects (and others) need to be loaded in memory.
If you are only loading the related entities for filtering purposes (not display purposes), and you are doing the filtering on the database level; then you can simply remove the include statements.
If you need those related entities to be loaded (for in-memory filtering, or for display purposes), then you can't easily remove the Include statements, unless you write a Select that specifies the fields you need.
For example:
_context.Projects
.Select(p => new { Project = p, ArchitectName = p.Architect.Name })
.ToList()
This will load the project entities (in their entirety) but only the name of the architect and none of the other properties. This can be a significant performance boost if your related entities have many properties that you currently do not need.
Note: The current example uses an anonymous type. I generally advocate creating a custom type for this; but that's unrelated to the performance issue we're addressing here.
Update
Based on your update, you seemingly imply that the intended filtering happens after the objects have been loaded from the database.
This is the source of your performance problems. You are fetching a lot of data but only show part of it. The data that does not get shown still needs to be loaded, which is wasted effort.
There are separate arguments for performance here:
Load everything once - Load all the data once (which might take a long time), but then allow the user to filter the loaded data (which is very fast)
Load chunks - Only load the data that matches the applied filters. If the user changes the filters, you load the data again. The first load will be much faster, but the subsequent filtering actions will take longer compared to in-memory filtering.
What you should do here is not my decision. It's a matter of priorities. Some customers prefer one over the other. I would say that in most cases, the second option (loading chunks) is the better option here, as it prevents needlessly loading a massive dataset if the user never looks through 90% of it. That's a waste of performance and network load.
The answer I gave applies to the "load chunks" approach.
If you decide to take the "load everything once" approach, then you will have to accept the performance hit of that initial load. The best you can do is severely limit the returned data columns (like I showed with the Select) in order to minimize the performance/network cost.
I see no reasonable argument to mix these two approaches. You'll end up with both drawbacks.
I am using EF to get data from my MySQL database.
Now, I have two tables, the first is customers and project_codes.
The table project_codes have a FK to customers named id_customers. This way I am able to query which projects belong to a customer.
On my DAL, I got the following:
public List<customer> SelectAll()
{
using (ubmmsEntities db = new ubmmsEntities())
{
var result = db.customers
.Include(p => p.project_codes)
.OrderBy(c=>c.customer_name)
.ToList();
return result;
}
}
That outputs to me all my customer and their respective project_codes.
Recently I needed to only get the project codes, so I created a DAL to get all the project codes from my database. But then I got myself thinking: "Should I have done that? Wouldn't be best to use my SelectAll() method and from it use Linq to fetch me the list of project_codes off all customers?
So this that was my first question. I mean, re-utilizing methods as much as possible is a good thing from a maintainability perspective, right?
The second question would be, how to get all the project_codes to a List? Doing it directly is easy, but I failed to achieve that using the SelectAll() as a base.
It worked alright if I had the customer_id using:
ddlProject.DataSource = CustomerList.Where(x => x.id.Equals(customer_id))
.Select(p => p.project_codes).FirstOrDefault();
That outputed me the project codes of that customer, but I tried different approaches (foreach, where within whhere and some others at random) but they either the syntax fail or don't output me a list with all the project_codes. So that is another reason for me going with a specific method to get me the project codes.
Even if "common sense" or best practices says it is a bad idea to re-utilize the method as mentioned above, I would like some directions on how to achieve a list of project_codes using the return of SelectAll()... never know when it can come in hand.
Let me know your thoughts.
There's a trade-off here; you are either iterating a larger collection (and doing selects, etc) on an in-memory collection, or iterating a smaller collection but having to go to a database to do it.
You will need to profile your setup to determine which is faster, but its entirely possible that the in-memory approach will be better (though stale if your data could have changed!).
To get all the project_codes, you should just need:
List<customer> customers; //Fill from DAL
List<project_code> allProjects = customers.SelectMany(c => c.project_codes).ToList();
Note that I used SelectMany to flatten the hierarchy of collections, I don't think SelectAll is actually a LINQ method.
I have started using the Entity Framework quite recently and think it is very good but I am a bit confused over a couple of things.
I'm writing a winforms-application where you are presented a list of persons in a list and if you click a particular person more information about that person appears in textboxes. Nothing fancy so far, but you are supposed to be able to edit the data about the person so data binding is nice here.
It works just fine but I am bit confused about what is the correct way to do it. First I did like this:
var query = _context.Person.Where(c => c.Id == chosenId);
this.personBindingSource.DataSource = query.ToList();
Then I read a bit and tried:
var local = _context.Person.Local;
IEnumerable<Customer> enumerable = local.Where(c => c.Id == chosenId);
this.personBindingSource.DataSource = enumerable.ToList();
This one seems to work fine as well.
Then I saw that someone suggested something like:
_context.Person.Load();
this.personBindingSource.DataSource = _context.Person.Local.ToBindingList();
I am a bit confused right now on what approach is the correct one and what is the difference between these three? Anyone can help me?
Depends on What You Want to do
I honestly never liked getting this answer because it seems to be the answer to everything but in this case it is really the only answer I can give
What the Local Property Does
Local gives you a reference to the current elements being tracked by the data context that hasn't been marked by delete, essentially you are asking for the current object that have already been loaded into memory by the context, you can read more about it here DbSet(TEntity).Local Property
What The Load Method Does
The Load method eagerly loads the targeted context, you can read more about it here.
DbExtensions.Load Method
What ToBindingList Does
Basically this is creating a two way binding between whatever entity you have created and the UI when you use a collection created using this method. That is that any changes in the UI should be automatically reflected in the related entities within this collection. You can read more about it using the following links
BindingList(T) Class DbExtensions.ToBindingList()
What Each Of Your Examples Do
First Example
var query = _context.Person.Where(c => c.Id == chosenId);
this.personBindingSource.DataSource = query.ToList();
Under the covers the following is going on
Creating a Query to be processed by the server with your linq expressionGetting the content from the database and creating a list around it
Here you are grabbing any of the people with Id of chosen Id from the database and then loading them into your application and creating a list
Second Example
var local = _context.Person.Local;
IEnumerable<Customer> enumerable = local.Where(c => c.Id == chosenId);
this.personBindingSource.DataSource = enumerable.ToList();
Under the covers the following is going on
Getting all of the currently tracked objects by the context object that have been hit by a query but have not been marked as deletedgetting all of the elements in local memory that have the Id chosen Id
Here you are grabbing any people that have already been loaded into the context this is not going to get all of your persisted data items, you must have hit them in other queries
Third Example
_context.Person.Load();
this.personBindingSource.DataSource = _context.Person.Local.ToBindingList();
Under the covers the following is going on
You are loading all of the people into local memoryYou create binding list (allows two way data binding between the objects and the UI) You bind the list to the UI element personBindingSource
Unless you want to load all of the items into memory this is probably not what you want to do, if the dataset ever grows large enough it will slow your program down and could possibly cause it to not work correctly (unlikely with case of person in most scenarios but is possible)
When You Should Use Them
FirstWhen you want to just get the data that matches your query into local memory and don't need a link between UI and the entitiesSecondWhen you have already run a query in the context and need to use it someplace else but don't need to rerun the query since it is currently in memoryThirdWhen you want to load all of the elements of an entity set into memory and create a two way databinding between them and a control
The only difference between the first and second example you have there is one is deferred and one is not. The third one is a little different as it creates a two way binding between your database and control datasource, effectively making the datacontext track changes to the list for you (add, & deletes).
In all your examples, so long as you keep you datacontext open changes to the objects themselves would be tracked.
As far as which way is correct, that depends on your application. Pick the approach that works best for you based on what you are trying to accomplish
I am playing around with RavenDb I have a very simple class that contains a collection, I am trying to return all the objects where the contained collection has more than 1 record, cannot seem to make it work.
Note: I am using an In-Memory Embedded document store in LinqPad, reading some data from a RDBMS and inserting into the In-Memory store (this works, if I just Query<Agency>().Take(100) I see my records...
Any Idea's ?
below image just to show that the db does contain my data...
ok, I have figured it out, can't say I fully understand it...but...
PopulateRavenInMemory();
DatabaseCommands.PutIndex("MultipleAddresses",
new IndexDefinitionBuilder<Agency>
{
Map = agencies => from a in agencies
where a.Addresses.Count() > 1
select new {}
});
Query<Agency>("MultipleAddresses").Customize(x => x.WaitForNonStaleResultsAsOfNow()).Dump();
I understand the WaitForNonStaleResults call, that makes sense, but I don't really understand why my Map function cannot select the class, it seems to demand a projection, I can move on, but I hate not knowing why this is so.