Background
ArticleService is a class that provides methods for front-end layers to facilitate business with the back-end.
Two of its base responsibilities are to convert ViewModels (ArticleViewModel) to the appropriate Models (Article) when persisting data, and the reverse, convert Models to ViewModels when fetching data... so often that I created a private method building ViewModel objects:
private ArticleViewModel BuildViewModel(Article a)
{
return new ArticleViewModel { Title = a.Title /* all properties */ }
}
Moving along, the ArticleService provides a method to fetch all articles from the data store, returning them as ViewModels: public IEnumerable<ArticleViewModel> All()
A calling class uses it like this: var articleViewModels = _articleService.All();
Simple, right?
Problem:
I originally wrote All() lazily with a classic foreach loop:
private IEnumerable<ArticleViewModel> All()
{
var viewModels = new List<ArticleViewModel>();
foreach (var article in _db.Articles)
viewModels.Add(BuildViewModel(article));
return viewModels;
}
It worked fine - articleViewModels was an instantiate list of all view models.
Next, I used ReSharper to convert this loop to a LINQ statement for performance and prettiness, and then combined the assignment statement with the return statement. Result:
private IEnumerable<ArticleViewModel> All()
{
return _db.Articles.Select(article => BuildViewModel(article)).ToList();
}
I debugged the LINQ statement and the beast awakened:
LINQ to Entities does not recognize the method 'ArticleViewModel
BuildViewModel(Article)' and this method cannot be translated into a store expression.
Question - Why does this LINQ statement break my code?
Note: Stepping back to an explicit declaration, assignment, return worked with the LINQ statement, so I'm almost certain it's something to do with the lambda logic.
Question - Why does this LINQ statement break my code?
Because LINQ to Entities is trying to translate BuildViewModel into SQL. It doesn't know how, so it dies miserably.
In your original version, you were streaming the entity from the database to your local box, and then doing the projection using BuildViewModel client-side. That's fine.
so I'm almost certain it's something to do with the lambda logic.
Nope. It's because LINQ to Entities can't translate BuildViewModel into SQL. It doesn't matter if you use a lambda expression to express the projection or not.
You can rewrite the code like this:
return _db.Articles.
.AsEnumerable()
.Select(article => BuildViewModel(article)).ToList();
This causes _db.Articles to be treated as a plain old enumerable, and then does the projection client side. Now LINQ to Entities doesn't have to figure out what to do with BuildViewModel.
The answer by Jason explains the error, but doesn't [didn't] list the simple fix (if you still want to use LINQ). Just add a call to .AsEnumerable() right after "_db.Articles". This will force any future LINQ statements to be performed using LINQ to Objects (in memory) rather than trying to use the IQueryable to perform the statements against the database.
Simply your function BuildViewModel is not traslable to raw SQL. That is.
Related
I am making a call to a function in .net from angular js, the time it takes to get the response to angular from .net is more than 5 seconds. How can I make the mapping of the result of a sql query decrease in time, I have the following code.
List<CarDTO> result = new List<CarDTO>();
var cars = await _carsUnitOfWork.CarRepository.GetCarDefault(carFilter,true,_options.Value.priorityLabel);
result = cars.Select(car => _mapper.Map<Car, CarDTO>(car)).ToList();
The code you have provided isn't expanded enough to identify a cause, but there are a number of clues:
Check / post the code for CareRepository.GetCarDefault(). The call implies that this is returning an IEnumerable given it is Awaited. It isn't clear what all of the parameters are and how they affect the query. As your database grows, this appears to return all cars, rather than supporting pagination. (What happens when you have 10,000 Car records, or 1,000,000?)
Next would be the use of Automapper's Map method. Combined with IEnumerable this means that your repository is going through the hassle of loading all Care entities into memory, then Automapper is allocating a duplicate set of DTOs into memory copying across data from the entities.
Lazy loading is a distinct risk with an approach like this. If the CarDTO pulls any fields from entities referenced by a Car, this will trip off additional queries for each individual car.
For best performance, I highly recommend adopting an IQueryable return type on Repository methods and leveraging Automapper's ProjectTo method rather than Map. This is equivalent to using Select in Linq, as ProjectTo will bubble down into the SQL generation to build efficient queries and return the collection of DTOs. This removes the risk of lazy loading calls as well as the double memory allocation for entities then DTOs.
Implementing this with your Unit of Work pattern is a bit of an unknown without seeing the code. However it would look something more like:
var result = await _carsUnitOfWork.CarRepository
.GetCarDefault(carFilter,true,_options.Value.priorityLabel)
.ProjectTo<CarDto>(mapperConfig)
.ToListAsync(); // Add Skip() and Take() to support pagination.
The repository method would be changed from being something like:
public async IEnumerable<Car> GetCarDefault( ... )
to
public IQueryable<Car> GetCarDefault( ... )
Rather than the method returning something like .ToListAsync(), you return the built Linq expression.
I.e. change from something like:
var result = _context.Cars.Include().Where(x => ...).ToListAsync();
return result;
to
var query = _context.Cars.Where(x => ....);
return query;
The key differences is that where the existing method likely returns ToListAsync() we remove that and return the unmaterialized IQueryable that Linq is building. Also, if the current implementation is Eager Loading any relations /w .Include() we exclude those. The caller performing projection doesn't need that. If the caller does need Car entity graphs, (such as when updating data) the caller can append .Include() statements.
It is also worth running an SQL Profiler to look at what queries are being run against the database server. This can give you the queries to inspect and test, as well as highlight any unexpected queries being triggered. (I.e. caused by lazy loading calls)
That should give you some ideas on where to start.
I have a Ling Query for calculate percentage.
var total = db.Schema_Details.Where(x => x.RID == sName).Select(x => new{
Rid = x.RID,
Amount = Convert.ToDouble(x.Lum_Sum_Amount),
Allocation = Convert.ToDouble(x.Allocation),
total = ((x.Allocation / 100) * x.Lum_Sum_Amount)}).ToList();
But exception occurred.
How can I solve this?
Since you use the identifier db, I assume db is a DbContext. This means that your linq statements will be executed in SQL on the database server side and not in your memory (SQL or something similar, depending on the type of DB your are using).
You can see this if you check the type of your statement in the debugger: is it an IQueryable or an IEnumerable? IQueryables are executed by the database server, IEnumerable are usually executed in your memory. Usually your database server can execute your linq statements faster and more efficient than you, so it is best to make sure your linq statements are IQueryable.
However, the disadvantage is, that you database server does not know any of your classes, functions, nor .NET. So in IQueryables you can only use fairly simple calculations.
If you really need to call some local functions somewhere halfway your linq statement, add AsEnumerable()to your statement:
var myLinqResult = db. ... // something IQueryable
.Where(item => ... ) // simple statement: still IQueryable
.Select(item => ...) // still simple, still IQueryable
// now you want to call one of your own functions:
.AsEnumerable()
.Select(item => myOwnLocalFunctions(item));
See stackoverflow: Difference between IQueryable and IEnumerable
The local function you want to perform is Convert.ToDouble. IQueryable does not know the Convert class, and thus can't perform your function as IQueryable. AsEnumerable will solve this problem.
However, in this case I would not advise to use AsEnumerable. When executing linq statements on a database try to avoid using local classes and function calls as much as possible. Instead of Convert.ToDoubleuse (double)x.Sum_Amount. Your linq-to-sql translator (or similar database language converter) will know how to translate this as IQueryable.
There are some calculations of which there are SQL equivalents, like DateTime calculations, or string reverse. It would be a shame if you had to do things like DateTime.AddDays(1) in local memory while there are perfect SQL equivalents for it. This would especially be a problem if you need to Join, Select or GroupBy after your local functions. Your database server can do these things far more efficiently than you. Luckily there are some IQueryable extension functions for them. They can be found in System.Data.Entity.DbFunctions
Usually the distinction between LINQ to SQL and LINQ to Objects isn't much of an issue, but how can I determine which is happening?
It would be useful to know when writing the code, but I fear one can only be sure at run time sometimes.
It's not micro optimization to make the distinction between Linq-To-Sql and Linq-To-Objects. The latter requires all data to be loaded into memory before you start filtering it. Of course, that can be a major issue.
Most LINQ methods are using deferred execution, which means that it's just building the query but it's not yet executed (like Select or Where). Few others are executing the query and materialize the result into an in-memory collection (like ToLIst or ToArray). If you use AsEnumerable you are also using Linq-To-Objects and no SQL is generated for the parts after it, which means that the data must be loaded into memory (yet still using deferred execution).
So consider the following two queries. The first selects and filters in the database:
var queryLondonCustomers = from cust in db.customers
where cust.City == "London"
select cust;
whereas the second selects all and filters via Linq-To-Objects:
var queryLondonCustomers = from cust in db.customers.AsEnumerable()
where cust.City == "London"
select cust;
The latter has one advantage: you can use any .NET method since it doesn't need to be translated to SQL (e.g. !String.IsNullOrWhiteSpace(cust.City)).
If you just get something that is an IEnumerable<T>, you can't be sure if it's actually a query or already an in-memory object. Even the try-cast to IQueryable<T> will not tell you for sure what it actually is because of the AsQueryable-method. Maybe you could try-cast it to a collection type. If the cast succeeds you can be sure that it's already materialized but otherwise it doesn't tell you if it's using Linq-To-Sql or Linq-To-Objects:
bool isMaterialized = queryLondonCustomers as ICollection<Customer> != null;
Related: EF ICollection Vs List Vs IEnumerable Vs IQueryable
The first solution comes into my mind is checking the query provider.
If the query is materialized, which means the data is loaded into memory, EnumerableQuery(T) is used. Otherwise, a special query provider is used, for example, System.Data.Entity.Internal.Linq.DbQueryProvider for entityframework.
var materialized = query
.AsQueryable()
.Provider
.GetType()
.GetGenericTypeDefinition() == typeof(EnumerableQuery<>);
However the above are ideal cases because someone can implement a custom query provider behaves like EnumerableQuery.
I had the same question, for different reasons.
Judging purely on your title & initial description (which is why google search brought me here).
Pre compilation, given an instance that implements IQueryable, there's no way to know the implementation behind the interface.
At runtime, you need to check the instance's Provider property like #Danny Chen mentioned.
public enum LinqProvider
{
Linq2SQL, Linq2Objects
}
public static class LinqProviderExtensions
{
public static LinqProvider LinqProvider(this IQueryable query)
{
if (query.Provider.GetType().IsGenericType && query.Provider.GetType().GetGenericTypeDefinition() == typeof(EnumerableQuery<>))
return LinqProvider.Linq2Objects;
if (typeof(ICollection<>).MakeGenericType(query.ElementType).IsAssignableFrom(query.GetType()))
return LinqProvider.Linq2Objects;
return LinqProvider.Linq2SQL;
}
}
In our case, we are adding additional filters dynamically, but ran into issues with different handling of case-sensitivity/nullreference handling on different providers.
Hence, at runtime we had to tweak the filters that we add based on the type of provider, and ended up adding this extension method:
Using EF core in net core 6
To see if the provider is an EF provider, use the following code:
if (queryable.Provider is Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryProvider)
{
// Queryable is backed by EF and is not an in-memory/client-side queryable.
}
One could get the opposite by testing the provider against System.Linq.EnumerableQuery (base type of EnumerableQuery<T> - so you don't have to test generics).
This is useful if you have methods like EF.Functions.Like(...) which can only be executed in the database - and you want to branch to something else in case of client-side execution.
I have the following method in a data access class which uses entity framework:
public static IEnumerable<entityType> GetWhere(Func<entityType, bool> wherePredicate)
{
using (DataEntities db = new DataEntities())
{
var query = (wherePredicate != null)
? db.Set<entityType>().Where(wherePredicate).ToList()
: db.Set<entityType>().ToList();
return query;
}
}
This works fine when I use the entities across all layers... however I am trying to move to using a DTO class and I would like to do something like the following:
public static IEnumerable<EntityTypeDTO> GetWhere(Func<EntityTypeDTO, bool> wherePredicate)
{
//call a method here which will convert Func<EntityTypeDTO,bool> to
// Func<EntityType,bool>
using (DataEntities db = new DataEntities())
{
var query = new List<EntityType>();
if (wherePredicate == null)
{
query = db.Set<EntityType>().ToList();
}
else
{
query = (wherePredicate != null)
? db.Set<EntityType>().Where(wherePredicate).AsQueryable<EntityType>().ToList()
: db.Set<EntityType>().ToList();
}
List<EntityTypeDTO> result = new List<EntityTypeDTO>();
foreach(EntityType item in query)
{
result.Add(item.ToDTO());
}
return result;
}
}
Essentially I want a method which will convert Func to Func.
I think I have to break down the Func into an expression tree and then rebuild it somehow in the entityType?
I want to do this to allow the Presentation Layer to just pass the Expression queries?
Am I missing something basic or is there an easier design pattern that can pass a query from a DTO to a data access class without knowing the details of the query?
I have tried making the DTO inherit from the entity which doesn't seem to work either?
If there is a better design pattern that I am missing I would love a pointer and I can investigate from there...
Firstly I would suggest that you put a querying layer of your own in front of Entity Framework rather than allowing any arbitrary Func to be passed in because it will be very easy in the future to pass a Func that Entity Framework can not translate into a SQL statement (it can only translate some expressions - the basics are fine but if your expression calls a C# method, for example, then Entity Framework will probably fail).
So your search layer could have classes that you build up as criteria (eg. a "ContainsName" search class or a "ProductHasId" class) that are then translated into expressions in your search layer. This separates your app entirely from the ORM, which means that ORM details (like the entities or like the limitations of what Funcs can and can't be translated) don't leak out. There's lots out there that's been written about this some of arrangement.
One final note, though, if you are working close to the ORM layer, Entity Framework is very clever and you could probably get a long way without trying to translate your Func<dto, bool> to a Func<entity, bool>. For example, in the below code, accessing "context.Products" returns a "DbSet" and calling Select on it returns an IQueryable and calling Where on that also returns an IQueryable. Entity Framework will translate all of that into a single SQL statement so it won't pull all other Products into memory and then filter the ID on that memory set, it will actually perform the filtering in SQL even though the filter is operating on a projected type (which is equivalent to the DTO in your case) and not the Entity Framework entity -
var results = context.Products
.Select(p => new { ID = p.ProductID, Name = p.ProductName })
.Where(p => p.ID < 10)
.ToList();
The SQL executed is:
SELECT
[Extent1].[ProductID] AS [ProductID],
[Extent1].[ProductName] AS [ProductName]
FROM [dbo].[Products] AS [Extent1]
WHERE [Extent1].[ProductID] < 10
So, if you changed your code to get something like..
return context.Products
.Map<Product, ProductDTO()>()
.Where(productDtoWherePredicate)
.ToList();
.. then you might be just fine with the Funcs that you already have. I presume that you already have some sort of mapping functions to get from EF entities to DTOs (but if not then you might want to look into AutoMapper to help you out - which has support for "projections", which are basically IQueryable maps).
I am going to put this up as an answer.Thanks to Dan for the quick answer. Looking at what you are saying I can write a query/filter set of classes. for example, take the following code:
GetProducts().GetProductsInCategory().GetProductsWithinPriceRange(minPrice, maxPrice);
This code would run like so: Get Products would get all products in the table and the remaining functions would filter the results. if all queries run like this it may put a significant load on the Data Access Layer/ DB Server Connections... not sure.
or
An Alternate I will work on also is:
If each function creates a Linq expression, I could combine them like this: How do I combine multiple linq queries into one results set?
this may allow me to do this in a manner where I can return the filtered results set from the database.
Either way I am marking this as answered. I will update when I have more details.
Is there a way to create functions for linq to sql that act different when used in linq to entities?
I do not whant to use a AsQueriable. I becouse I whant to do as much calculations on the sql server as possible.
example:
var UserIDs = Users.Select(x=> ConvertToString(x.UserID));
For linq to sql the code has to be:
public string ConvertToString(int id)
{
return SqlFunctions.StringConvert((double?)id);
}
For linq to entities the code has to be:
public string ConvertToString(int id)
{
return id.ToString();
}
You can use SqlFunctions in Entity Framework 4 and above. In your example, you do not have to do the conversion in the application layer.
http://blogs.microsoft.co.il/blogs/gilf/archive/2009/05/23/calling-database-functions-in-linq-to-entities-in-entity-framework-4.aspx
In Linq-to-Entites you won't be allowed to use the ConvertToString call in a query anyway, so this is a brave, but fruitless effort. The reason is that every statement in an L2E query must be an Expression that must be translatable to SQL by the query provider. And (of course) there is no built-in translation for any arbitrary method. L2E is very limited here. It is not even possible to use most native .Net methods like
Users.Select(x=> Convert.ToString(x.UserID)); // NotSupportedException
Linq-to-Sql is a bit more lenient in this. It will run the statement decently. But that does not help you, because you are looking for a common denominator. The bottleneck is L2E and it is a narrow bottleneck.