Develop Reference

Add two expressions to create a predicate in Entity Framework Core 3 does not work

I am trying to build an "And" predicate method using C# with Entity Framework Core 3 in a .NET Core application.
The function adds two expressions to each other and passes it to an IQueryable code:
public Expression<Func<T, bool>> AndExpression<T>
(Expression<Func<T, bool>> left, Expression<Func<T, bool>> right)
{
var andExpression = Expression.AndAlso(
left.Body, Expression.Invoke(right,
left.Parameters.Single()));
return Expression.Lambda<Func<T, bool>>(andExpression, left.Parameters);
}
The call for of the function
Expression<Func<Entity, bool>> left = t => t.Id == "id1";
Expression<Func<Entity, bool>> right = t => t.Id == "id2";
var exp = AndExpression(left, right);
this.dbContext.Set<Entity>().source.Where(exp).ToList();
My code works fine in version EF Core 2, but after I updated the version to version 3 it throws the following exception
The LINQ expression 'Where(
source: DbSet,
predicate: (s) => (t => t.Id == "id1") && Invoke(t => t.Id == "id2")
)' could not be translated. Either rewrite the query in a form that can be translated, or switch to client evaluation explicitly by
inserting a call to either AsEnumerable(), AsAsyncEnumerable(),
ToList(), or ToListAsync().
I cannot translate the query to Enumerable due to memory issues.
I understand the problem but I don't know if there is a way to avoid it.
If anyone has a tip for me, I would appreciate that. Thanks a lot!

You need to unwrap the bodies of your lambdas, rather than using Expression.Invoke. You'll also have to rewrite at least one of the lambdas, so they both use the same parameters.
We'll use an ExpressionVisitor to replace right's parameter with the corresponding parameter from left. Then we'll construct the AndExpression using left's body and the rewritten body from right, and finally create a new lambda.
public class ParameterReplaceVisitor : ExpressionVisitor
{
public ParameterExpression Target { get; set; }
public ParameterExpression Replacement { get; set; }
protected override Expression VisitParameter(ParameterExpression node)
{
return node == Target ? Replacement : base.VisitParameter(node);
}
}
public static Expression<Func<T, bool>> AndExpression<T>(
Expression<Func<T, bool>> left, Expression<Func<T, bool>> right)
{
var visitor = new ParameterReplaceVisitor()
{
Target = right.Parameters[0],
Replacement = left.Parameters[0],
};
var rewrittenRight = visitor.Visit(right.Body);
var andExpression = Expression.AndAlso(left.Body, rewrittenRight);
return Expression.Lambda<Func<T, bool>>(andExpression, left.Parameters);
}
This results in a lambda with the following DebugView:
.Lambda #Lambda1<System.Func`2[System.String,System.Boolean]>(Entity $t) {
$t.Id == "id1" && $t.Id == "id2"
}
Whereas your code results in a lambda with the following DebugView:
.Lambda #Lambda1<System.Func`2[System.String,System.Boolean]>(System.String $t) {
$t == "id1" && .Invoke (.Lambda #Lambda2<System.Func`2[System.String,System.Boolean]>)($t)
}
.Lambda #Lambda2<System.Func`2[System.String,System.Boolean]>(System.String $t) {
$t == "id2"
}
See how yours is calling a lambda from within a lambda (something that EF can't handle), whereas mine just has a single lambda.

The issue here is that your query didn't work, or at least not how you intended in EF Core 2 either - In EF3 it will throw an exception for things like this due to a behavioural change, within EF Core 2 this would have silently pulled into memory and done the operation there, so in essence your query would have always been pulling the data into memory and performing the operation there. However now its telling you that it would do that and wanting you to explicitly set it to, or modify your expression to something that it can translate into a sql statement correctly.
This is actually a pretty good example case of why the team did this, seemingly you were unaware that this was previously pulling all of the data into memory and performing the operation there - now that you are aware you can work on a fix to get it performing the operation on the sql server
For more info feel free to read up Here
A modification to your code to be somewhere along the lines of
public static Expression<Func<T, bool>> AndExpression
(this Expression<Func<T, bool>> left, Expression<Func<T, bool>> right)
{
var invoked = var invokedExpr = Expression.Invoke (right, left.Parameters.Cast<Expression> ());
return Expression.Lambda<Func<T, bool>>
(Expression.AndAlso (left.Body, invoked), left.Parameters);
}
working from memory so code above may not be perfect - at work so I cant fully test

I took canton7's solution and made it into an extension that checks for null values:
private static Expression<Func<T, bool>> _AndExpression<T>(Expression<Func<T, bool>> left, Expression<Func<T, bool>> right)
{
ParameterReplaceVisitor visitor = new ParameterReplaceVisitor()
{
Target = right.Parameters[0],
Replacement = left.Parameters[0],
};
Expression rewrittenRight = visitor.Visit(right.Body);
Expression andExpression = Expression.AndAlso(left.Body, rewrittenRight);
return Expression.Lambda<Func<T, bool>>(andExpression, left.Parameters);
}
public static Expression<Func<T, bool>> AndExpression<T>(this Expression<Func<T, bool>> left, Expression<Func<T, bool>> right)
{
bool leftNotNull = (left != null);
bool rightNotNull = (right != null);
return (leftNotNull && rightNotNull)
? _AndExpression(left, right)
: left;
}
Usage:
expression1 = expression1.AndExpression<T>(expression2);

Creating a dynamic EF filter that builds a LINQ Where equals/contains statement for any string entity property

In short, I'm looking to do what this guy did, but with Entity Framework 6.
Implementing the proposed solution results in the error "The LINQ expression node type 'Invoke' is not supported in LINQ to Entities." Since the proposed solution uses Invoke, this is obviously an issue.
I understand that there's a way to harness a custom Compose method to rewrite the expression tree without using Invoke, but I can't seem to wrap my head around it.
Here's what I'm trying to write.
I build an IQueryable<TEntity> dynamically using a QueryParameters object that's just a bag of properties to use for the WHERE clauses. TEntity is a standard code-first EF entity with data annotations all over the place. The query contruction looks something like this:
IQueryable<TEntity> query = Context.Set<TEntity>();
if (queryParams == null)
return query;
if (!string.IsNullOrWhiteSpace(queryParams.FirstName))
{
if (queryParams.ExactSearch)
{
query = query.Where(x => x.FirstName == queryParams.FirstName);
}
else
{
if (queryParams.PreferStartsWith)
{
query = query.Where(
x => x.FirstName.ToLower()
.StartsWith(
queryParams.FirstName
.ToLower()));
}
else
{
query = query.Where(
x => x.FirstName.ToLower()
.Contains(
queryParams.FirstName
.ToLower()));
}
}
}
// ... repeat for all of queryParams' string props.
// DateTime, int, bool, etc have their own filters.
This gets repeated for every query parameter for a string field to be queried. Obviously, this results in a lot of repeated code. I would love to be able to write a filter with a signature like this:
public static IQueryable<TEntity> Search<TEntity>(
this IQueryable<TEntity> query,
Expression<Func<TEntity, string>> fieldExpression,
string searchValue,
bool exactSearch = true,
bool useStartsWithOverContains = false) {...}
Which I can then consume like this:
if (!string.IsNullOrWhiteSpace(queryParams.FirstName))
{
query = query.Search(
x => x.FirstName,
queryParams.FirstName,
queryParams.ExactSearch,
queryParams.PreferStartsWith);
}
The closest I've come a definition for that extension method is the below, but as mentioned, it produces that "'Invoke' is not supported in LINQ to Entities" error:
public static IQueryable<TEntity> Search<TEntity>(
this IQueryable<TEntity> query,
Expression<Func<TEntity, string>> fieldExpression,
string searchValue,
bool exactSearch = true,
bool useStartsWithOverContains = false)
{
if (string.IsNullOrWhiteSpace(searchValue))
return query;
searchValue = searchValue.Trim();
Expression<Func<TEntity, bool>> expression;
if (exactSearch)
{
var x = Expression.Parameter(typeof(TEntity), "x");
var left = Expression.Invoke(fieldExpression, x);
var right = Expression.Constant(searchValue);
var equalityExpression = Expression.Equal(left, right);
expression = Expression.Lambda<Func<TEntity, bool>>(
equalityExpression,
x);
}
else
{
searchValue = searchValue.ToLower();
var x = Expression.Parameter(typeof(TEntity), "x");
var fieldToLower = Expression.Call(
Expression.Invoke(fieldExpression, x),
typeof(string).GetMethod(
"ToLower",
Type.EmptyTypes));
var searchValueExpression =
Expression.Constant(searchValue);
var body = Expression.Call(
fieldToLower,
typeof(string).GetMethod(
useStartsWithOverContains ? "StartsWith" : "Contains",
new[] { typeof(string) }),
searchValueExpression);
expression = Expression.Lambda<Func<TEntity, bool>>(
body,
x);
}
return query.Where(expression);
}
I started to include the Compose method I mentioned, but I got lost really quickly, and thus removed it.
Open to any guidance! Thank you!

This is much easier to do by composing expressions than it is by trying to manually construct the expressions every single time. It's faster to write, so much less error prone, and actually ends up with code you can actually read at the end of it. All you need to do is write the code for how you use the value in the composed expression, which you already have from your original code.
public static IQueryable<TEntity> Search<TEntity>(
this IQueryable<TEntity> query,
Expression<Func<TEntity, string>> fieldExpression,
string searchValue,
bool exactSearch = true,
bool useStartsWithOverContains = false)
{
if (string.IsNullOrWhiteSpace(searchValue))
return query;
searchValue = searchValue.Trim();
if (exactSearch)
{
return query.Where(fieldExpression.Compose(field => field == searchValue));
}
else if (useStartsWithOverContains)
{
return query.Where(fieldExpression.Compose(field => field.StartsWith(searchValue.ToLower())));
}
else
{
return query.Where(fieldExpression.Compose(field => field.Contains(searchValue.ToLower())));
}
}
Note you should probably go with an enum for "Comparison" or something like that, rather than having two booleans. For example, right now someone can say that they don't want an exact sure but that they do want to use starts with. Just have one parameter with the three options.

How to build a simple property selector expression in ef6

How can I create a property selector for entity framework like this?
public static List<T> StandardSearchAlgorithm<T>(this IQueryable<T> queryable, Func<T, string> property, string query)
{
return queryable.Where(e => property(e).ToLower().IndexOf(query) > -1).ToList();
}
I want the calling code to be able to be clean and simple like this:
var usernameResults = _db.Users.StandardSearchAlgorithm(u => u.Username, query);
I get a "The LINQ expression node type 'Invoke' is not supported in LINQ to Entities." error. I cannot work out how to get the expression built.
UPDATE:
Based on the answer by MBoros here is the code I ended up with. It works great.
The key to expression trees is to understand expression trees are all about breaking up what you normally write in code (like "e => e.Username.IndexOf(query)") into a series of objects: "e" gets its own object, "Username" its own object, "IndexOf()" its own object, the "query" constant its own object, and so on. The second key is to know that you can use a series of static methods on the Expression class to create various kinds of these objects, as shown below.
PropertyInfo pinfo = (PropertyInfo)((MemberExpression)property.Body).Member;
ParameterExpression parameter = Expression.Parameter(typeof(T), "e");
MemberExpression accessor = Expression.Property(parameter, pinfo);
ConstantExpression queryString = Expression.Constant(query, typeof(string));
ConstantExpression minusOne = Expression.Constant(-1, typeof(int));
MethodInfo indexOfInfo = typeof(string).GetMethod("IndexOf", new[] { typeof(string) }); // easiest way to do this
Expression indexOf = Expression.Call(accessor, indexOfInfo, queryString);
Expression expression = Expression.GreaterThan(indexOf, minusOne);
Expression<Func<T, bool>> predicate = Expression.Lambda<Func<T, bool>>(expression, parameter);
//return predicate.Body.ToString(); // returns "e => e.Username.IndexOf(query) > -1" which is exactly what we want.
var results = queryable.Where(predicate).ToList();
return results;
Now I have a real problem, but I will ask it in a separate question. My real query looks like this:
public static List<T> StandardSearchAlgorithm<T>(this IQueryable<T> queryable, Func<T, string> property, string query)
{
return queryable.Where(e => property(e).IndexOf(query) > -1).Select(e=> new { Priority = property(e).IndexOf(query), Entity = e } ).ToList();
}
So I need to build an expression that returns an Anonymous Type!! Or even if I create a class to help, I need to write an expression that returns a new object. But I will include this in a separate question.

You cannot invoke CLR delegates so simply in sql. But you can pass in the property selector as an Expression tree., so your signature would be:
public static List<T> StandardSearchAlgorithm<T>(this IQueryable<T> queryable, Expression<Func<T, string>> property, string query)
Calling would look the same. But now that you have an expression in your hand, you can have a look at this answer:
Pass expression parameter as argument to another expression
It gives you the tools to simply put an expression tree inside another one. In your case it would look like:
Expression<Func<T, bool>> predicate = e => property.AsQuote()(e).Contains(query);
predicate = predicate.ResolveQuotes();
return queryable.Where(predicate).ToList();
Once you are there, you still have the .ToLower().Contains() calls (use .Contains instead of the .IndexOf()> 1). This is actually tricky. Normally the db uses its default collation, so if it set to CI (case insensitive), then it will do the compare that way. If you don't have any constraints, and can adjust the db collation, I would go for that. In this case you can omit the .ToLower() call.
Otherwise check out this anser: https://stackoverflow.com/a/2433217/280562

Creating a linq expression dynamically or parsing the string value of the expression

Got an problem where I need to crate an linq expression dynamically. But I cant seem to wrap my head around it.
So first off I got an check list box. Where I input values from a file that is selected, hence I will not know how many parameters there will be there before I run the code. Then the plan is to construct a linq sentence based on the name of the elements in the checkbox. Let me illustrate:
I want to create an linq statement that looks like this:
var result = from n in data where n.Item1 == "someValueFromCheckBox" select n;
Where the data, is an tupled list like :
List<Tuple<string, string>>
And that is fine as long as the user only selects one item from the checkbox. But when the user for example selects two items, I need to create a linqu expression that looks like this:
var result = from n in data where n.Item1 == "someValueFromCheckBox" || n.Item1 == "someValueFromCheckBox1" select n;
So my first thought was to build the linq statement as a string. In the following manner:
var selectedItems = checkedListBoxSelectedTerms.CheckedItems;
var linqStatement = "";
for (int i = 0; i < selectedItems.Count; i++ )
{
linqStatement += selectedItems[i].ToString() + "|| n.item1 ==" ;
}
//this is simply to remove the || "n.Item1 ==", at the end because it
is not needed after the last selecteditem.
linqStatement = linqStatement.Remove(linqStatement.Length - 13, 13);
But that did not work as planned, because when I put it in like this:
var result = from n in data where n.Item1 == linqStatement select n;
The whole thing becomes a string. And I cant have the "n.Item1 ==" to be a string. In theory that solution would work great for my use, because I would not have to worry about how manay elements that are in the checkbox and how many elements the user selects, but I cannot pass linqStatement whole as a string. So any help would be appriciated, because I cant seem to find an good example to do something like this.
Thanks in advance!

For such a simple example, you should simply have a list or set of strings, and then do Contains. E.g.
var selectedItems = checkedListBoxSelectedTerms.CheckedItems
.Select(x => x.ToString()).ToList();
var result = from n in data where selectedItems.Contains(n.Item1) select n;
If you need more flexible dynamic building capabilities, try something like Dynamic LINQ or PredicateBuilder.

The PredicateBuilder is what I use for dynamic queries.
Please see here for more info:
http://www.albahari.com/nutshell/predicatebuilder.aspx

So what you're looking for is an implementation of a predicate builder that can take any number of expressions and OR them together. Here is an implementation that will work with any query provider:
public static class PredicateBuilder
{
public static Expression<Func<T, bool>> True<T>() { return f => true; }
public static Expression<Func<T, bool>> False<T>() { return f => false; }
public static Expression<Func<T, bool>> Or<T>(
this Expression<Func<T, bool>> expr1,
Expression<Func<T, bool>> expr2)
{
var secondBody = expr2.Body.Replace(expr2.Parameters[0],
expr1.Parameters[0]);
return Expression.Lambda<Func<T, bool>>
(Expression.OrElse(expr1.Body, secondBody), expr1.Parameters);
}
public static Expression<Func<T, bool>> And<T>(
this Expression<Func<T, bool>> expr1,
Expression<Func<T, bool>> expr2)
{
var secondBody = expr2.Body.Replace(expr2.Parameters[0],
expr1.Parameters[0]);
return Expression.Lambda<Func<T, bool>>
(Expression.AndAlso(expr1.Body, secondBody), expr1.Parameters);
}
}
which uses:
internal class ReplaceVisitor : ExpressionVisitor
{
private readonly Expression from, to;
public ReplaceVisitor(Expression from, Expression to)
{
this.from = from;
this.to = to;
}
public override Expression Visit(Expression node)
{
return node == from ? to : base.Visit(node);
}
}
public static Expression Replace(this Expression expression,
Expression searchEx, Expression replaceEx)
{
return new ReplaceVisitor(searchEx, replaceEx).Visit(expression);
}
This allows you to take any two expressions and create a new expression that is the result of ORing them together.
List<Tuple<string, string>> list = CreateList();
var finalExpression = list.Aggregate(PredicateBuilder.False<Data>(),
(predicate, pair) => predicate.Or(n => n.Item1 == pair.Item2));
var query = data.Where(finalExpression);

How do I dynamically create an Expression<Func<MyClass, bool>> predicate?

How would I go about using an Expression Tree to dynamically create a predicate that looks something like...
(p.Length== 5) && (p.SomeOtherProperty == "hello")
So that I can stick the predicate into a lambda expression like so...
q.Where(myDynamicExpression)...
I just need to be pointed in the right direction.
Update: Sorry folks, I left out the fact that I want the predicate to have multiple conditions as above. Sorry for the confusion.

Original
Like so:
var param = Expression.Parameter(typeof(string), "p");
var len = Expression.PropertyOrField(param, "Length");
var body = Expression.Equal(
len, Expression.Constant(5));
var lambda = Expression.Lambda<Func<string, bool>>(
body, param);
Updated
re (p.Length== 5) && (p.SomeOtherProperty == "hello"):
var param = Expression.Parameter(typeof(SomeType), "p");
var body = Expression.AndAlso(
Expression.Equal(
Expression.PropertyOrField(param, "Length"),
Expression.Constant(5)
),
Expression.Equal(
Expression.PropertyOrField(param, "SomeOtherProperty"),
Expression.Constant("hello")
));
var lambda = Expression.Lambda<Func<SomeType, bool>>(body, param);

Use the predicate builder.
http://www.albahari.com/nutshell/predicatebuilder.aspx
Its pretty easy!

To combine several predicates with the && operator, you join them together two at a time.
So if you have a list of Expression objects called predicates, do this:
Expression combined = predicates.Aggregate((l, r) => Expression.AndAlso(l, r));

To associate Lambda expression each other:
An other way is to use the following code. It s more flexible than the Schotime answer in my advice and work perfectly. No external Nuggets needed
Framework 4.0
// Usage first.Compose(second, Expression.And)
public static Expression<T> Compose<T>(this Expression<T> First, Expression<T> Second, Func<Expression, Expression, Expression> Merge)
{
// build parameter map (from parameters of second to parameters of first)
Dictionary<ParameterExpression,ParameterExpression> map = First.Parameters.Select((f, i) => new { f, s = Second.Parameters[i] }).ToDictionary(p => p.s, p => p.f);
// replace parameters in the second lambda expression with parameters from the first
Expression secondBody = ParameterRebinder.ReplaceParameters(map, Second.Body);
// apply composition of lambda expression bodies to parameters from the first expression
return Expression.Lambda<T>(Merge(First.Body, secondBody), First.Parameters);
}
public static Expression<Func<T, bool>> And<T>(this Expression<Func<T, bool>> First, Expression<Func<T, bool>> Second)
{
return First.Compose(Second, Expression.And);
}
public static Expression<Func<T, bool>> Or<T>(this Expression<Func<T, bool>> First, Expression<Func<T, bool>> second)
{
return First.Compose(second, Expression.Or);
}
public class ParameterRebinder : ExpressionVisitor
{
private readonly Dictionary<ParameterExpression, ParameterExpression> map;
public ParameterRebinder(Dictionary<ParameterExpression, ParameterExpression> map)
{
this.map = map ?? new Dictionary<ParameterExpression, ParameterExpression>();
}
public static Expression ReplaceParameters(Dictionary<ParameterExpression, ParameterExpression> map, Expression exp)
{
return new ParameterRebinder(map).Visit(exp);
}
protected override Expression VisitParameter(ParameterExpression p)
{
ParameterExpression replacement;
if (map.TryGetValue(p, out replacement))
{
p = replacement;
}
return base.VisitParameter(p);
}
}

I have a open-source project called Exprelsior that provides very simple ways to create dynamic predicates:
Based on your example:
var exp1 = ExpressionBuilder.CreateBinary<YourClass>("MyProperty.Length", 5, ExpressionOperator.Equals);
var exp2 = ExpressionBuilder.CreateBinary<YourClass>("SomeOtherProperty", "hello", ExpressionOperator.Equals);
var fullExp = exp1.And(exp2);
// Use it normally...
q.Where(fullExp)
It even supports full text predicate generation, so you can receive any dynamic query from an HTTP GET method, for example:
var exp1 = ExpressionBuilder.CreateBinaryFromQuery<YourClass>("eq('MyProperty.Length', '5')");
var exp2 = ExpressionBuilder.CreateBinaryFromQuery<YourClass>("eq('SomeOtherProperty', 'hello')");
var fullExp = exp1.And(exp2);
// Use it normally...
q.Where(fullExp)
It supports a lot more data types and operators.
Link: https://github.com/alexmurari/Exprelsior