Develop Reference

AspNet Core Web API high memory usage with EF Core

I'm seeing my memory usage getting too hign on Diagnostic Tools after I call some endpoints.
I tried to isolate the problem in the smallest chunk I could, so I could eliminate all other factors, and I ended up with the following:
[HttpGet("test")]
public ActionResult Test()
{
var results = _context.Products
.Include(x => x.Images)
.Include(x => x.Options)
.ThenInclude(x => x.Lists)
.ThenInclude(x => x.PriceChangeRule)
.Include(x => x.Options)
.ThenInclude(x => x.Lists)
.ThenInclude(x => x.Items)
.ThenInclude(x => x.PriceChangeRule)
.Include(x => x.Options)
.ThenInclude(x => x.Lists)
.ThenInclude(x => x.Items)
.ThenInclude(x => x.SupplierFinishingItem)
.ThenInclude(x => x.Parent)
.Include(x => x.Category)
.ThenInclude(x => x.PriceFormation)
.ThenInclude(x => x.Rules)
.Include(x => x.Supplier)
.ThenInclude(x => x.PriceFormation)
.ThenInclude(x => x.Rules)
.Include(x => x.PriceFormation)
.ThenInclude(x => x.Rules)
.AsNoTracking().ToList();
return Ok(_mapper.Map<List<AbstractProductListItemDto>>(results));
}
It is a big query, with lots of includes, but the amount of data returned from the database is not huge, it's ~10.000 items. When I serialize this result it has only 3.5Mb.
My API is using around 300Mb of memory, then when I call this test endpoint, this value goes to about 1.2Gb. I think this is too much for only 3.5Mb of data, but I don't know how EF Core works internally, so I'll just ignore it.
My problem is that, as far as I understand, the DbContext is added as a scoped service, so it's created when the request starts and then killed when it finishes. Here's how I'm registering it:
services.AddDbContext<DatabaseContext>(options =>
options.UseSqlServer(
Configuration.GetConnectionString("DefaultConnection")));
If my understanding is correct, when the request is finished, that huge amount of memory should be disposed, right?
The problem is that my memory usage never goes back again, I tried disposing the context manually, calling the garbage collector manually too, but the memory stays at 1.2Gb.
Am I missing something here?

A potential area I can see is that you may be loading far more data out of the database than is required by the serialization to AbstractProductListItemDto. For example, your Product might have fields like below:
public class Product
{
public string Id { get; set; }
public string Name { get; set; }
public string Description { get; set; }
}
However, your final DTO may only have one or two of those properties, for example:
public class AbstractProductListItemDto
{
public string ProductId { get; set; }
public string ProductName { get; set; }
}
This may also be true for the other tables which you are including (Options, Lists, Rules, etc), especially tables which are one-to-many which could easily explode the number rows/columns being queried.
A potential way to optimize this is to do the projection yourself as part of the LINQ query. This would take advantage of a feature of EF Core where it only selects the columns from the database which you have specified. For example:
This would select all columns from the product table
var results = _context.Products.ToList();
This would select only the Id and Name columns from the product table, resulting in less memory usage
var results = _context.Products.Select(x => new ProductDto {
Id = x.Id,
Name = x.Name,
}
From the question I do not know all of the properties on all of the items which you are mapping, so it would be up to you if you wanted to do that mapping manually. The critical part is that you would need to do that in a call to Select() before your call to ToList() on your query.
However, there is a potential shortcut if you are using Automapper
Automapper includes a shortcut which attempts to write these query projections for you. It may not work depending on how much additional logic is happening within Automapper, but it might be worth a try. You would want to read up on the ProjectTo<>() method. If you were using projection, the code would probably look something like this:
Edit: It was correctly pointed out in comments that the Include() calls are not needed when using ProjectTo<>(). Here is a shorter sample with the original included below it
Updated:
using AutoMapper.QueryableExtensions;
// ^^^ Added to your usings
//
[HttpGet("test")]
public ActionResult Test()
{
var projection = _context.Products.ProjectTo<AbstractProductListItemDto>(_mapper.ConfigurationProvider);
return Ok(projection.ToList());
}
Original:
using AutoMapper.QueryableExtensions;
// ^^^ Added to your usings
//
[HttpGet("test")]
public ActionResult Test()
{
var results = _context.Products
.Include(x => x.Images)
.Include(x => x.Options)
.ThenInclude(x => x.Lists)
.ThenInclude(x => x.PriceChangeRule)
.Include(x => x.Options)
.ThenInclude(x => x.Lists)
.ThenInclude(x => x.Items)
.ThenInclude(x => x.PriceChangeRule)
.Include(x => x.Options)
.ThenInclude(x => x.Lists)
.ThenInclude(x => x.Items)
.ThenInclude(x => x.SupplierFinishingItem)
.ThenInclude(x => x.Parent)
.Include(x => x.Category)
.ThenInclude(x => x.PriceFormation)
.ThenInclude(x => x.Rules)
.Include(x => x.Supplier)
.ThenInclude(x => x.PriceFormation)
.ThenInclude(x => x.Rules)
.Include(x => x.PriceFormation)
.ThenInclude(x => x.Rules)
.AsNoTracking(); // Removed call to ToList() to keep it as IQueryable<>
var projection = results.ProjectTo<AbstractProductListItemDto>(_mapper.ConfigurationProvider);
return Ok(projection.ToList());
}

AutoMapper and Entity Framework Include with circular relationships with No Tracking

Using Entity Framework 6, I'm trying to eagerly load my Caller models from the database using .AsNoTracking(), but I'm hitting a snag when I try to map these models to their ViewModels using AutoMapper 6.
The Caller has an Address, which is a many-to-one relationship (caller's can have one address, address can have multiple callers).
Here are the (reduced) model classes (ViewModels are nearly identical)
public class Caller
{
public Guid Id { get; set; }
public string FirstName { get; set; }
public Address Address { get; set; }
}
public class Address
{
public Guid Id { get; set; }
public string City { get; set; }
public virtual ICollection<Caller> Callers { get; set; }
}
Here is how I am mapping them
// Address
CreateMap<Address, AddressViewModel>()
.ForMember(vm => vm.Id, map => map.MapFrom(m => m.Id))
.ForMember(vm => vm.CallerViewModels, map => map.MapFrom(m => m.Callers))
.ForMember(vm => vm.City, map => map.MapFrom(m => m.City))
.ReverseMap();
// Caller
CreateMap<Caller, CallerViewModel>()
.ForMember(vm => vm.Id, map => map.MapFrom(m => m.Id))
.ForMember(vm => vm.AddressViewModel, map => map.MapFrom(m => m.Address))
.ForMember(vm => vm.FirstName, map => map.MapFrom(m => m.FirstName))
.ReverseMap();
In my CallerRepository I am using this function:
public async Task<Caller> GetFullCallerAsNoTrackingAsync(Guid id)
{
return await _context.Callers
.AsNoTracking()
.Include(c => c.Address)
.FirstOrDefaultAsync(c => c.Id == id);
}
My problem happens here:
// Map Caller to a CallerViewModel
Caller caller = await unitOfWork.CallerRepo.GetFullCallerAsNoTrackingAsync(Guid.Parse(callerId));
CallerViewModel callerViewModel = Mapper.Map<CallerViewModel>(caller); // Throws exception
The exception that gets thrown says
Error Mapping Types ... Caller.Address -> CallerViewModel.Address ... When an object is returned with a NoTracking merge option, Load can only be called when the EntityCollection or EntityReference does not contain objects.
This works just fine when I remove the .AsNoTracking(), but for performance reasons I'm trying to keep that in.
I don't need to know Caller -> Address -> Callers, I just need Caller -> Address
Any suggestions on how I can achieve this?
Edit / Update:
Thanks to FoundNil's answer I was able to get this done.
I changed my Address Map to:
CreateMap<Address, AddressViewModel>()
.ForMember(vm => vm.Id, map => map.MapFrom(m => m.Id))
.ForMember(vm => vm.CallerViewModels, map => map.MapFrom(m => m.Callers).Ignore())
.ForMember(vm => vm.City, map => map.MapFrom(m => m.City))
.ReverseMap();
And I did the same to a different property, CallDetailViewModels, on my Caller -> CallerViewModel map
CreateMap<Caller, CallerViewModel>()
.ForMember(vm => vm.Id, map => map.MapFrom(m => m.Id))
.ForMember(vm => vm.AddressViewModel, map => map.MapFrom(m => m.Address))
.ForMember(vm => vm.CallDetailViewModels, map => map.MapFrom(m => m.CallDetails).Ignore())
The similarities I see between this and Address is that Caller is the parent object of Address, and CallDetail is the parent object of Caller
Both of these parents were navigation properties in their respective Model class:
Caller -> public virtual ICollection<CallDetail> CallDetails { get; set; }
Address -> public virtual ICollection<Caller> Callers { get; set; }
Perhaps this might give a useful flag to others of where they might encounter this problem.
Note: My CallDetail has a many-to-many relationship with Caller, so it also has a navigation property of Callers, and I'm not ignoring that in my CallDetail Map.

I'm not entirely sure why its happening, but I would guess the problem is that when you use .AsNoTracking() something happens between Address -> Callers in the context, so there is no longer a way to map ICollection<Caller> and its view model.
And since you mentioned you only want Caller -> Address you should try this map:
// Address
CreateMap<Address, AddressViewModel>()
.ForMember(x => x.Callers, opt => opt.Ignore())
.ReverseMap();
// Caller
CreateMap<Caller, CallerViewModel>()
.ForMember(vm => vm.AddressViewModel, map => map.MapFrom(m => m.Address))
.ReverseMap();

Unable to convert IEnumerable<int> to List<T> in Entity Framework

I have a query like this
IEnumerable<int> companies = Employers
.Where(p => p.Id == User.ID
.SelectMany(p => p.companies)
.Select(p => p.Id);
return Employers
.Where(p => companies.Contains(p.Companies)
.Include(p => p.FirstName)
.ToList();
Here p.Companies refers to
Public Virtual List<Company>
under the class Employers.
This error occurs:
'IEnumerable' does not contain a definition for 'Contains' and the best extension method overload 'Queryable.Contains>(IQueryable>,List)' requires a receiver of type 'IQueryable>'

Doing this:
Where(p => companies.Contains(p.Companies)
you try to check if companies (IEnumerable<int> from previous query) contains some companies (object), not int value.
You can't check if collection of int values contains some value which is not int(Company is some object).
If you want to query all employers which have company with Id that is in companies variable use this Linq query:
return Employers
.Where(p => p.Companies.Any(c => companies.Contains(c.Id)))
.Include(p => p.FirstName)
.ToList();

Where on child's parent object

I am trying to get all object where the childs parentId equals the page id.
This is what I am trying to do:
public IEnumerable<Route> GetAll(int locationId)
{
_db.Configuration.ProxyCreationEnabled = false;
return _db.Routes.Include(o => o.ConnectionPointRoutes.Select(s => s.Segment))
.Include(o => o.ConnectionPointRoutes.Select(c => c.ConnectionPoint)
.Where( c => c.Location.LocationId == locationId)).ToList();
}
but I keep getting this error :
An exception of type 'System.ArgumentException' occurred in EntityFramework.dll but was not handled in user code
Additional information: The Include path expression must refer to a
navigation property defined on the type. Use dotted paths for
reference navigation properties and the Select operator for collection
navigation properties.
Any thoughts?

I ended up doing this :
public IEnumerable<Route> GetAll(int locationId)
{
return _db.Routes
.Include(o => o.ConnectionPointRoutes.Select(s => s.Segment))
.Include(o => o.ConnectionPointRoutes.Select(c => c.ConnectionPoint))
.Where(c => c.ConnectionPointRoutes.Select(s => s.ConnectionPoint.Location)
.FirstOrDefault(q => q.LocationId == locationId).LocationId == locationId)
.ToList();
}

Conditional Include() in Entity Framework [duplicate]

This question already has answers here:
EF: Include with where clause [duplicate]
(5 answers)
Closed 2 years ago.
I have seen a few answers to similar questions, however I cannot seem to work out how to apply the answer to my issue.
var allposts = _context.Posts
.Include(p => p.Comments)
.Include(aa => aa.Attachments)
.Include(a => a.PostAuthor)
.Where(t => t.PostAuthor.Id == postAuthorId).ToList();
Attachments can be uploaded by the Author (type Author) or Contributor (type Contributor). What I want to do, is only get the Attachments where the owner of the attachment is of type Author.
I know this doesn't work and gives an error:
.Include(s=>aa.Attachments.Where(o=>o.Owner is Author))
I've read about Filtered Projection here
EDIT - link to article:
: http://blogs.msdn.com/b/alexj/archive/2009/10/13/tip-37-how-to-do-a-conditional-include.aspx,
but I just can't get my head around it.
I don't want to include the filter in the final where clause as I want ALL posts, but I only want to retrieve the attachments for those posts that belong to the Author.
EDIT 2: - Post schema requested
public abstract class Post : IPostable
{
[Key]
public int Id { get; set; }
[Required]
public DateTime PublishDate { get; set; }
[Required]
public String Title { get; set; }
[Required]
public String Description { get; set; }
public Person PostAuthor { get; set; }
public virtual ICollection<Attachment> Attachments { get; set; }
public List<Comment> Comments { get; set; }
}

EF Core 5.0 is introducing Filtered Include soon.
var blogs = context.Blogs
.Include(e => e.Posts.Where(p => p.Title.Contains("Cheese")))
.ToList();
Reference: https://learn.microsoft.com/en-us/ef/core/what-is-new/ef-core-5.0/whatsnew#filtered-include

From the link you posted I can confirm that trick works but for one-many (or many-one) relationship only. In this case your Post-Attachment should be one-many relationship, so it's totally applicable. Here is the query you should have:
//this should be disabled temporarily
_context.Configuration.LazyLoadingEnabled = false;
var allposts = _context.Posts.Where(t => t.PostAuthor.Id == postAuthorId)
.Select(e => new {
e,//for later projection
e.Comments,//cache Comments
//cache filtered Attachments
Attachments = e.Attachments.Where(a => a.Owner is Author),
e.PostAuthor//cache PostAuthor
})
.AsEnumerable()
.Select(e => e.e).ToList();

Remove the virtual keyword from your Attachments navigation property to prevent lazy loading:
public ICollection<Attachment> Attachments { get; set; }
First method: Issue two separate queries: one for the Posts, one for the Attachments, and let relationship fix-up do the rest:
List<Post> postsWithAuthoredAttachments = _context.Posts
.Include(p => p.Comments)
.Include(p => p.PostAuthor)
.Where(p => p.PostAuthor.Id == postAuthorId)
.ToList();
List<Attachment> filteredAttachments = _context.Attachments
.Where(a => a.Post.PostAuthor.Id == postAuthorId)
.Where(a => a.Owner is Author)
.ToList()
Relationship fixup means that you can access these filtered Attachments via a Post's navigation property
Second method: one query to the database followed by an in-memory query:
var query = _context.Posts
.Include(p => p.Comments)
.Include(p => p.PostAuthor)
.Where(p => p.PostAuthor.Id == postAuthorId)
.Select(p => new
{
Post = p,
AuthoredAttachments = p.Attachments
Where(a => a.Owner is Author)
}
);
I would just use the anonymous type here
var postsWithAuthoredAttachments = query.ToList()
or I would create a ViewModel class to avoid the anonymous type:
List<MyDisplayTemplate> postsWithAuthoredAttachments =
//query as above but use new PostWithAuthoredAttachments in the Select
Or, if you really want to unwrap the Posts:
List<Post> postsWithAuthoredAttachments = query.//you could "inline" this variable
.AsEnumerable() //force the database query to run as is - pulling data into memory
.Select(p => p) //unwrap the Posts from the in-memory results
.ToList()

You can use this implementation of an extension method (eg.) Include2(). After that, you can call:
_context.Posts.Include2(post => post.Attachments.Where(a => a.OwnerId == 1))
The code above includes only attachments where Attachment.OwnerId == 1.

try this
var allposts = _context.Posts
.Include(p => p.Comments)
.Include(a => a.PostAuthor)
.Where(t => t.PostAuthor.Id == postAuthorId).ToList();
_context.Attachments.Where(o=>o.Owner is Author).ToList();

For net core
https://learn.microsoft.com/ru-ru/ef/core/querying/related-data/explicit
var allposts = _context.Posts
.Include(p => p.Comments)
.Include(a => a.PostAuthor)
.Where(t => t.PostAuthor.Id == postAuthorId).ToList();
_context.Entry(allposts)
.Collection(e => e.Attachments)
.Query()
.Where(e=> e.Owner is Author)
.Load();
it makes 2 query to sql.

Lambda in Include() may only point to a property:
.Include(a => a.Attachments)
.Include(a => a.Attachments.Owner);
Your condition doesn't makes sense for me because Include() means join and you either do it or not. And not conditionally.
How would you write this in raw SQL?
Why not just this:
context.Attachments
.Where(a => a.Owner.Id == postAuthorId &&
a.Owner.Type == authorType);
?

Assuming "a" being of type "YourType", a conditonal include could be solved by using a method extension, e.g.
public static class QueryableExtensions
{
public static IQueryable<T> ConditionalInclude<T>(this IQueryable<T> source, bool include) where T : YourType
{
if (include)
{
return source
.Include(a => a.Attachments)
.Include(a => a.Attachments.Owner));
}
return source;
}
}
... then just use this like you are using .Include, e.g.
bool yourCondition;
.ConditionalInclude(yourCondition)