I'm trying to use predicates in my EF filtering code.
This works:
IQueryable<Customer> filtered = customers.Where(x => x.HasMoney && x.WantsProduct);
But this:
Predicate<T> hasMoney = x => x.HasMoney;
Predicate<T> wantsProduct = x => x.WantsProduct;
IQueryable<Customer> filtered = customers.Where(x => hasMoney(x) && wantsProduct(x));
fails at runtime:
The LINQ expression node type 'Invoke' is not supported in LINQ to Entities.
I can't use the first option, as this is a simple example, and in reality, I'm trying to combine a bunch of predicates together (with and, not, or, etc.) to achieve what I want.
How can I get the EF Linq provider to "understand" my predicate(s)?
I get the same result if I use a Func<T, bool>. It works with an Expression<Func<T>>, but I can't combine expressions together for complex filtering. I'd prefer to avoid external libraries if possible.
UPDATE:
If this cannot be done, what options do I have? Perhaps expressions be combined / or'ed / and'ed somehow to achieve the same effect?
Expression<Func<Customer, bool>> hasMoney = x => x.HasMoney;
Expression<Func<Customer, bool>> wantsProduct = x => x.WantsProduct;
IQueryable<Customer> filtered = customers.Where(hasMoney).Where(wantsProduct);
Use Expression<T> to leave x => x.HasMoney as an expression tree, and not compile it to a .NET method
Use Expression<Func<Customer, bool>> rather than Expression<Predicate<Customer>>, because that's what Queryable.Where expects
Pass it directly in .Where, combining them using multiple .Where calls instead of &&.
It's possible to get more complex conditions (including not, or, etc.) working by rewriting them using .Union, .Except, etc.
An alternative is to use LINQKit's AsExpandable:
Expression<Func<Customer, bool>> hasMoney = x => x.HasMoney;
Expression<Func<Customer, bool>> wantsProduct = x => x.WantsProduct;
IQueryable<Customer> filtered = customers.AsExpandable().Where(x => hasMoney.Invoke(x) && wantsProduct.Invoke(x));
Unfortunately there's no way to use Predicate<T> in EF linq since it's impossible to map it on SQL query. This can be done with Expressions only because they can be parsed and converted to SQL.
In fact there are 4 language features that made linq possible:
Extension methods
Type inference
Closures
and for linq2sql especially
Expressions
UPDATE:
The possible solution is building expressions programmatically. How to: Use Expression Trees to Build Dynamic Queries
Related
I'm using EF Core 7. I want to perform server evaluation and project to a named tuple.
I tried:
var products = await _context.Products.Select(x => (x.Id, x.Name)).ToListAsync();
Which gives:
An expression tree may not contain a tuple literal.
I could do this with a regular tuple (Tuple.Create()) or an anonymous object (new {}), but I'd like to use a named tuple, if possible.
Can this be done somehow?
No, currently it is not possible because it is not possible to use value tuples in expression trees. You can monitor following issues/discussions at the github:
[Proposal]: Expression tree evolution
Expression trees support for tuples.
If you really-really want to use value tuples the only way is to map to them after the materialization of the query, but I would argue that it is rather pointless in most cases. Something like the following:
var products = (await _context.Products
.Select(x => new {x.Id, x.Name})
.ToListAsync())
.Select(ac => (ac.Id, ac.Name))
.ToList();
I was fiddling around with the linq its Func parameter(on entity framework). Then I found out this behaviour
var idMatchQuery = new Func<MyClass, bool>(x => x.Id == someId);
var statusMatchQuery = new Func<MyClass, bool>(x => x.Status == someStatus);
/// works
var a = myClassEntity.FirstOrDefault(idMatchQuery);
/// doesn't work
var b = myClassEntity.FirstOrDefault(p => idMatchQuery(p) && statusMatchQuery(p));
/// doesn't work
var c = myClassEntity.FirstOrDefault(p => idMatchQuery(p) && p.Status == 1);
It throws UnsupportedOperationException since the EF does not recognize those queries. I could have accept it much easier if none of the above were not working. But it bugs me when it works with one and only one Func query, but not with combinations with other queries.
I'm sure there exist an explanation for it, but I guess my search terms were too naive for the answer I'm looking for.
What is the explanation for this behaviour?
It is caused that, EF should translate your predicate to TSQL language. You can check, that parameter of FirstOrDefault method is not Func<T, bool>, but instead Expression<Func<T, bool>>, because last one give us oppotunity to parse it and translate to TSQL. When you use two Func<T, bool> or Func<T, bool> with simple condition EF can't translate and parse it to TSQL due to Expression<Func<T, bool>> inner stuff and features complexity, that is why EF keep this predicates at origin state and send to server as it was written at first, as a result - UnsupportedOperationException. So, for EF - parse first predicate much more easily than other two.
Conclusion: it is caused by features and methodology of translation C# predicates from Expression<Func<T,bool>> to TSQL, because of it's sometimes enough high complexity.
I have this method below that queries over database to get for instance User, Orders or any entity for that matter. I want to know if the below query is optimal or do i have to tweak that.
Does the query do where, only after storing all records in the memory ?
Is there a way i can specify Expression for the filtering rather than Func<> ?
What it the optimal way to get Count, specifying a condition/where without much memory consumption?
C# Sample Code
public IList<TEntity> Find(Func<TEntity, bool> predicate)
{
//Criteria for creating
ICriteria filterCriterea = _unitOfWork.CurrentSession.CreateCriteria(typeof(TEntity));
//filtered result
return filterCriterea.Future<TEntity>().Where(predicate).ToList();
}
The are several things that you need to fix:
1. Ask for an Expression, not a Func
Your Find method specifies that it wants a Func. If you want an Expression, you have to say so:
public IList<TEntity> Find(Expression<Func<TEntity, bool>> predicate)
2. Use LINQ or QueryOver instead of ICriteria
Assuming you're not using an ancient version of NHibernate that doesn't have LINQ or QueryOver (oddly, there are a LOT of people still using NHibernate 1.2.1 - I can't figure out why)...
ICriteria doesn't understand Expressions. Replace CreateCriteria(typeof(TEntity)) with Query<TEntity>() or QueryOver<TEntity>() depending on which query syntax you prefer.
3. Where should go before Future
You need to move the Where before the Future. Future is used to batch several queries together so that they all get executed in one round trip to the database as soon as your code tries to evaluate the results of one of the "Future" queries. At the point your above code calls Future, the query only consists of CreateCriteria(typeof(TEntity)), which just tells it which table to query against, like so:
select * from TEntity
If you want a where clause in there, you have to switch those method calls around:
filterCriterea.Where(predicate).Future<TEntity>()
This should give you a SQL query like:
select * from TEntity where ...
It is very important to learn the difference between IQueryable and IEnumerable. You should notice that Query returns an IQueryable, whereas Future returns an IEnumerable. Manipulating an IQueryable using Expressions will result in changes to the SQL that gets executed. Manipulating an IEnumerable using a Func just changes the way you're looking at the in-memory data.
4. ToList immediately after Future negates benefit of Future
ToList iterates over the collection that is passed to it and puts each element in a list. The act of iterating over the collection will cause the Future query to be immediately executed, not giving you any chance to batch it together with other queries. If you must have a list, you should just omit the Future. I think however, that it would be a better idea to change your method to return an IEnumerable, which would give you the ability to batch this query together with other queries. Like so...
public IEnumerable<TEntity> Find(Expression<Func<TEntity, bool>> predicate)
{
var query = _unitOfWork.CurrentSession.Query<TEntity>();
return query.Where(predicate).Future<TEntity>();
}
In order to take advantage of the query batching, you are probably going to have to rearrange the code that calls this Find method. For example, instead of...
foreach (var openThing in thingRepository.Find(x => x.Status == Status.Open))
CloseIt(openThing);
foreach (var negativeWhatsit in whatsitRepository.Find(x => x.Amount < 0))
BePositive(negativeWhatsit);
... you should do this:
var openThings = thingRepository.Find(x => x.Status == Status.Open);
var negativeWhatsits = whatsitRepository.Find(x => x.Amount < 0);
// both queries will be executed here in one round-trip to database
foreach (var openThing in openThings)
CloseIt(openThing);
foreach (var negativeWhatsit in negativeWhatsits)
BePositive(negativeWhatsit);
I write some Parameterize Lambda Queries
//Method 1:
Func<SalesOrderLineEntity, bool> func01 = (o => o.SOLNumber == "123456");
var q01 = _context.SalesOrderLineEntities.Where(func01).ToList();
//Got the result, but SQLServer Read All Records to memory before "where"
//Method 2:
Expression<Func<SalesOrderLineEntity, bool>> exp02 = (o => o.SOLNumber == "123456");
var q02 = _context.SalesOrderLineEntities.Where(exp02).ToList();
//Got the result,Exec "Where" in SQLServer
//Method 3:
Expression<Func<SalesOrderLineEntity, bool>> exp03 = (o => func01(o));
var q03 = _context.SalesOrderLineEntities.Where(exp03.Compile()).ToList();
//Same to Method 1,because Compile() result is Func<SalesOrderLineEntity, bool>
//Method 4:
var q04 = _context.SalesOrderLineEntities.Where(exp03).ToList();
//Error:The LINQ expression node type 'Invoke' is not supported in LINQ to Entities
Method 1 and 3：Efficiency is very low
Method 4:Error
Method 2:Need I Build a Expression through the Lambda. I feel it is very difficult, because i will use many "if,else".it easier to create a function.
What is the correct way to do that?
Variations
Method 1: EF reads all the records from the DB because you pass a Func into the Where clause, which is not the right candidate: EF cannot extract the needed information from it to build the query, it can only use that function on an in-memory collection.
Method 2: this is the correct way to do EF queries because EF builds up the actual query based on the Expression tree. It may look like the same as Method 1 when you write .Where but this is different.
IQueryable extension methods are using Expression trees, so you can (or EF can) evaluate that information at runtime.
Method 3: this is essentially the same as Method 1, because you compile the expression. This is a key difference while you use them: an expression contains the informations to build the actual operation but that's not the operation itself. You need to compile it before (or for example you can build SQL queries based on them, that's how EF works).
Method 4: EF cannot translate your func01() call to any SQL function. It cannot translate any kind of code because it needs an equivalent SQL operation. You can try to use a general method, you will get the same result, it's not about the Func.
What happens here?
If we simplify the underlying process then the answer above might by more clear.
//Method 2:
Expression<Func<SalesOrderLineEntity, bool>> exp02 = (o => o.SOLNumber == "123456");
var q02 = _context.SalesOrderLineEntities.Where(exp02).ToList();
//Got the result,Exec "Where" in SQLServer
EF can read the following (via the expressions):
the user want to filter with Where
here is an expression for that, let's get some information
well, it needs SalesOrderLineEntity and I have a mapping for that type
the expression tells that the property SOLNumber must be equal to "123456"
ok, I have a mapping for SOLNumber so it's good
and I can translate the equal operator to an equivalent SQL operator
everything okay, so we can build the SQL query
Of course, you cannot do this with a Func for example because that object doesn't contain these informations.
Not sure if this is applicable but have you look at compiled queries: Compiled Queries (LINQ to Entities) which should result in a more efficient SQL statement
I'm using Microsoft's Entity Framework as an ORM and am wondering how to solve the following problem.
I want to get a number of Product objects from the Products collection where the Product.StartDate is greater than today. (This is a simplified version of the whole problem.)
I currently use:
var query = dbContext.Products.Where(p => p.StartDate > DateTime.Now);
When this is executed, after using ToList() for example on query, it works and the SQL created is effectively:
SELECT * FROM Product WHERE StartDate > (GetDate());
However, I want to move the predicate to a function for better maintainability, so I tried this:
private Func<Product, bool> GetFilter()
{
Func<Product, bool> filter = p => p.StartDate > DateTime.Now;
return filter;
}
var query = dbContext.Products.Where(GetFilter());
This also works from a code point of view insofar as it returns the same Product set but this time the SQL created is analogous to:
SELECT * FROM Product;
The filter is moved from the SQL Server to the client making it much less efficient.
So my questions are:
Why is this happening, why does the LINQ parser treat these two formats so differently?
What can I do to take advantage of having the filter separate but having it executed on the server?
You need to use an Expression<Func<Product, bool>> in order for it to work like you intend. A plain Func<Product, bool> tells LINQ that you want it to run the Where in MSIL in your program, not in SQL. That's why the SQL is pulling in the whole table, then your .NET code is running the predicate on the entire table.
You are returning a Func, but to inject the predicate into the SQL, LINQ requires an expression tree. It should work if you change the return type of your method (and of your local variable, of course) to Expression<Func<Product, bool>>.
Since in second case filter func maybe arbitrary LINQ to EF can't parse you filter to SQL and has to resolve it on client side.