Develop Reference

How to build dynamic query with Where and OR using Expression

I hope somebody can guide and help me with this. We have an inherited project that uses ExpressionHelper class. Basically, this Expression Helper will return an IQueryable that build a dynamic query base on the search term that the user provided.
For example, I have the below code where I pass 2 search terms.
IQueryable<UserEntity> modifiedQuery = _uow.UserRepository.GetAll();;
var searchTerms = new List<SearchTerm>
{
new SearchTerm { Name = "FirstName", Operator = "eq", Value = "Bob" },
new SearchTerm { Name = "FirstName", Operator = "eq", Value = "John" }
};
foreach (var searchTerm in searchTerms)
{
var propertyInfo = ExpressionHelper
.GetPropertyInfo<TEntity>(searchTerm.EntityName ?? searchTerm.Name);
var obj = ExpressionHelper.Parameter<TEntity>();
var left = ExpressionHelper.GetPropertyExpression(obj, propertyInfo);
var right = searchTerm.ExpressionProvider.GetValue(searchTerm.Value);
var comparisonExpression = searchTerm.ExpressionProvider
.GetComparison(left, searchTerm.Operator, right);
// x => x.Property == "Value"
var lambdaExpression = ExpressionHelper
.GetLambda<TEntity, bool>(obj, comparisonExpression);
// query = query.Where...
modifiedQuery = ExpressionHelper.CallWhere(modifiedQuery, lambdaExpression);
}
With the code above and using the below ExpressionHelper class, this generate the below SQL query when I check using SQLProfiler. Please notice the AND in the query. What I actually what is OR.
Constructed QUERY in SQL Profiler
SELECT
[Extent1].[FirstName] AS [FirstName],
FROM [dbo].[tblUser] AS [Extent1]
WHERE ([Extent1].[Conatact1] = N'Bob') AND ([Extent1].[Contact2] = N'John')
ExpressionHelper.cs
public static class ExpressionHelper
{
private static readonly MethodInfo LambdaMethod = typeof(Expression)
.GetMethods()
.First(x => x.Name == "Lambda" && x.ContainsGenericParameters && x.GetParameters().Length == 2);
private static MethodInfo[] QueryableMethods = typeof(Queryable)
.GetMethods()
.ToArray();
private static MethodInfo GetLambdaFuncBuilder(Type source, Type dest)
{
var predicateType = typeof(Func<,>).MakeGenericType(source, dest);
return LambdaMethod.MakeGenericMethod(predicateType);
}
public static PropertyInfo GetPropertyInfo<T>(string name)
=> typeof(T).GetProperties()
.Single(p => p.Name == name);
public static ParameterExpression Parameter<T>()
=> Expression.Parameter(typeof(T));
public static MemberExpression GetPropertyExpression(ParameterExpression obj, PropertyInfo property)
=> Expression.Property(obj, property);
public static LambdaExpression GetLambda<TSource, TDest>(ParameterExpression obj, Expression arg)
=> GetLambda(typeof(TSource), typeof(TDest), obj, arg);
public static LambdaExpression GetLambda(Type source, Type dest, ParameterExpression obj, Expression arg)
{
var lambdaBuilder = GetLambdaFuncBuilder(source, dest);
return (LambdaExpression)lambdaBuilder.Invoke(null, new object[] { arg, new[] { obj } });
}
public static IQueryable<T> CallWhere<T>(IQueryable<T> query, LambdaExpression predicate)
{
var whereMethodBuilder = QueryableMethods
.First(x => x.Name == "Where" && x.GetParameters().Length == 2)
.MakeGenericMethod(new[] { typeof(T) });
return (IQueryable<T>)whereMethodBuilder
.Invoke(null, new object[] { query, predicate });
}
public static IQueryable<TEntity> CallOrderByOrThenBy<TEntity>(
IQueryable<TEntity> modifiedQuery,
bool useThenBy,
bool descending,
Type propertyType,
LambdaExpression keySelector)
{
var methodName = "OrderBy";
if (useThenBy) methodName = "ThenBy";
if (descending) methodName += "Descending";
var method = QueryableMethods
.First(x => x.Name == methodName && x.GetParameters().Length == 2)
.MakeGenericMethod(new[] { typeof(TEntity), propertyType });
return (IQueryable<TEntity>)method.Invoke(null, new object[] { modifiedQuery, keySelector });
}
}
I have hard time understanding on how the query was created and how do I change it to become OR in the created query.
Hope someone can guide me and point to the right direction. Thank you!

Add to SearchTerm a new property (C# 6.0 syntax here):
// This is quite wrong. We should have an enum here, but Operator is
// done as a string, so I'm maintaining the "style"
// Supported LogicalConnector: and, or
public string LogicalConnector { get; set; } = "and";
}
Then:
private static IQueryable<TEntity> BuildQuery<TEntity>(IQueryable<TEntity> modifiedQuery, List<SearchTerm> searchTerms)
{
Expression comparisonExpressions = null;
var obj = ExpressionHelper.Parameter<TEntity>();
foreach (var searchTerm in searchTerms)
{
var propertyInfo = ExpressionHelper
.GetPropertyInfo<TEntity>(searchTerm.EntityName ?? searchTerm.Name);
var left = ExpressionHelper.GetPropertyExpression(obj, propertyInfo);
var right = searchTerm.ExpressionProvider.GetValue(searchTerm.Value);
var comparisonExpression = searchTerm.ExpressionProvider.GetComparison(left, searchTerm.Operator, right);
if (comparisonExpressions == null)
{
comparisonExpressions = comparisonExpression;
}
else if (searchTerm.LogicalConnector == "and")
{
comparisonExpressions = Expression.AndAlso(comparisonExpressions, comparisonExpression);
}
else if (searchTerm.LogicalConnector == "or")
{
comparisonExpressions = Expression.OrElse(comparisonExpressions, comparisonExpression);
}
else
{
throw new NotSupportedException(searchTerm.LogicalConnector);
}
}
if (comparisonExpressions != null)
{
// x => x.Property == "Value"
var lambdaExpression = ExpressionHelper.GetLambda<TEntity, bool>(obj, comparisonExpressions);
// query = query.Where...
modifiedQuery = ExpressionHelper.CallWhere(modifiedQuery, lambdaExpression);
}
return modifiedQuery;
}
Use it like:
var searchTerms = new List<SearchTerm>
{
new SearchTerm { Name = "PrimaryContact", Operator = "eq", Value = "Bob" },
new SearchTerm { Name = "SecondaryContact", Operator = "eq", Value = "Bob" },
new SearchTerm { Name = "PrimaryContact", Operator = "eq", Value = "John", LogicalConnector = "or", }
};
IQueryable<UserEntity> query = BuildQuery<UserEntity>(modifiedQuery, searchTerms);
Note that there is no way in this code to explicitly set brackets, that will be implicitly set as:
(((A opB b) opC C) opD D)
Where A, B, C, D are the SearchTerm[0], SearchTerm[1], SearchTerm[2], SearchTerm[3] and opB, opC, opD are the operators defined in SearchTerm[1].LogicalConnector, SearchTerm[2].LogicalConnector, SearchTerm[3].LogicalConnector.
While putting brackets is easy, choosing how to "describe" them is complex, unless you change significantly your SearchTerm collection (it couldn't be a "linear" array but it would need to be a tree).
P.S. I was wrong, you don't need an ExpressionVisitor. You need an ExpressionVisitor when you are trying to "merge" multiple LambdaExpressions that have distinct ParameterExpression. In this code we are able to have a single var obj = ExpressionHelper.Parameter<TEntity>() for all the query, so no problems merging the conditions. To make it clear: if you want to "merge" x1 => x1.Foo == "Foo1" with x2 => x2.Foo == "Foo2" then you need an ExpressionVisitor that replaces x2 with x1, otherwise you would get a wrong query like x1 => x1.Foo == "Foo1" || x2.Foo == "Foo2". In the code given we have only x1 (that is var obj = ExpressionHelper.Parameter<TEntity>()), so no problem.

Dynamic extraction optimization of nested properties values

I have small piece of code responsible for dynamic extraction of properties values from objects instances through reflection:
public static object ExtractValue(object source, string property)
{
var props = property.Split('.');
var type = source.GetType();
var arg = Expression.Parameter(type, "x");
Expression expr = arg;
foreach (var prop in props)
{
var pi = type.GetProperty(prop);
if (pi == null)
throw new ArgumentException(string.Format("Field {0} not found.", prop));
expr = Expression.Property(expr, pi);
type = pi.PropertyType;
}
var delegateType = typeof(Func<,>).MakeGenericType(source.GetType(), type);
var lambda = Expression.Lambda(delegateType, expr, arg);
var compiledLambda = lambda.Compile();
var value = compiledLambda.DynamicInvoke(source);
return value;
}
It can extract values of nested properties, like: ExtractValue(instance, "PropA.PropB.PropC").
Despite the fact I like this method and its implementation, when, say, PropB is null, DynamicInvoke() just throws NullReferenceException (wrapped by TargetInvocationException). Because I needed to know which exact property is null is such case, I modified its body a bit (standard step-by-step extraction chain):
public static object ExtractValue(object source, string property)
{
var props = property.Split('.');
for (var i = 0; i < props.Length; i++)
{
var type = source.GetType();
var prop = props[i];
var pi = type.GetProperty(prop);
if (pi == null)
throw new ArgumentException(string.Format("Field {0} not found.", prop));
source = pi.GetValue(source, null);
if (source == null && i < props.Length - 1)
throw new ArgumentNullException(pi.Name, "Extraction interrupted.");
}
return source;
}
Now it looks a bit worse (I like lambdas) but behaves much better, not only because it gives more meaningful information of what has failed, but also because this version is about 66 times faster than the first one (coarse test below):
var model = new ModelA
{
PropB = new ModelB {PropC = new ModelC {PropD = new ModelD {PropE = new ModelE {PropF = "hey"}}}}
};
const int times = 1000000;
var start = DateTime.Now;
for (var i = 0; i < times; i++)
ExtractValueFirst(model, "PropB.PropC.PropD.PropE.PropF");
var ticks_first = (DateTime.Now - start).Ticks;
Console.WriteLine(":: first - {0} iters tooks {1} ticks", times, ticks_first);
start = DateTime.Now;
for (var i = 0; i < times; i++)
ExtractValueSecond(model, "PropB.PropC.PropD.PropE.PropF");
var ticks_second= (DateTime.Now - start).Ticks;
Console.WriteLine(":: second - {0} iters tooks {1} ticks", times, ticks_second);
Console.WriteLine("ticks_first/ticks_second: {0}", (float)ticks_first / ticks_second);
Console.ReadLine();
How can this code be optimized in .NET to perform even faster (caching, direct IL maybe, etc)?

You can increase performance significantly by caching the compiled delegates:
static readonly ConcurrentDictionary<Tuple<Type,string>,Delegate> _delegateCache = new ConcurrentDictionary<Tuple<Type,string>,Delegate>();
public static object ExtractValue(object source, string expression)
{
Type type = source.GetType();
Delegate del = _delegateCache.GetOrAdd(new Tuple<Type,string>(type,expression),key => _getCompiledDelegate(key.Item1,key.Item2));
return del.DynamicInvoke(source);
}
// if you want to acces static aswell...
public static object ExtractStaticValue(Type type, string expression)
{
Delegate del = _delegateCache.GetOrAdd(new Tuple<Type,string>(type,expression),key => _getCompiledDelegate(key.Item1,key.Item2));
return del.DynamicInvoke(null);
}
private static Delegate _getCompiledDelegate(Type type, string expression)
{
var arg = Expression.Parameter(type, "x");
Expression expr = arg;
foreach (var prop in property.Split('.'))
{
var pi = type.GetProperty(prop);
if (pi == null) throw new ArgumentException(string.Format("Field {0} not found.", prop));
expr = Expression.Property(expr, pi);
type = pi.PropertyType;
}
var delegateType = typeof(Func<,>).MakeGenericType(source.GetType(), type);
var lambda = Expression.Lambda(delegateType, expr, arg);
return lambda.Compile();
}

I've done some execution time measurements, which are presented below:
private static Func<object, object> _cachedFunc;
private static Delegate _cachedDel;
static void Main(string[] args)
{
var model = new ModelA
{
PropB = new ModelB {PropC = new ModelC {PropD = new ModelD {PropE = new ModelE {PropF = "hey"}}}}
};
const string property = "PropB.PropC.PropD.PropE.PropF";
var watch = new Stopwatch();
var t1 = MeasureTime(watch, () => ExtractValueDelegate(model, property), "compiled delegate dynamic invoke");
var t2 = MeasureTime(watch, () => ExtractValueCachedDelegate(model, property), "compiled delegate dynamic invoke / cached");
var t3 = MeasureTime(watch, () => ExtractValueFunc(model, property), "compiled func invoke");
var t4 = MeasureTime(watch, () => ExtractValueCachedFunc(model, property), "compiled func invoke / cached");
var t5 = MeasureTime(watch, () => ExtractValueStepByStep(model, property), "step-by-step reflection");
var t6 = MeasureTime(watch, () => ExtractValueStandard(model), "standard access (model.prop.prop...)");
Console.ReadLine();
}
public static long MeasureTime<T>(Stopwatch sw, Func<T> funcToMeasure, string funcName)
{
const int times = 100000;
sw.Reset();
sw.Start();
for (var i = 0; i < times; i++)
funcToMeasure();
sw.Stop();
Console.WriteLine(":: {0, -45} - {1} iters tooks {2, 10} ticks", funcName, times, sw.ElapsedTicks);
return sw.ElapsedTicks;
}
public static object ExtractValueDelegate(object source, string property)
{
var ptr = GetCompiledDelegate(source.GetType(), property);
return ptr.DynamicInvoke(source);
}
public static object ExtractValueCachedDelegate(object source, string property)
{
var ptr = _cachedDel ?? (_cachedDel = GetCompiledDelegate(source.GetType(), property));
return ptr.DynamicInvoke(source);
}
public static object ExtractValueFunc(object source, string property)
{
var ptr = GetCompiledFunc(source.GetType(), property);
return ptr(source); //return ptr.Invoke(source);
}
public static object ExtractValueCachedFunc(object source, string property)
{
var ptr = _cachedFunc ?? (_cachedFunc = GetCompiledFunc(source.GetType(), property));
return ptr(source); //return ptr.Invoke(source);
}
public static object ExtractValueStepByStep(object source, string property)
{
var props = property.Split('.');
for (var i = 0; i < props.Length; i++)
{
var type = source.GetType();
var prop = props[i];
var pi = type.GetProperty(prop);
if (pi == null)
throw new ArgumentException(string.Format("Field {0} not found.", prop));
source = pi.GetValue(source, null);
if (source == null && i < props.Length - 1)
throw new ArgumentNullException(pi.Name, "Extraction interrupted.");
}
return source;
}
public static object ExtractValueStandard(ModelA source)
{
return source.PropB.PropC.PropD.PropE.PropF;
}
private static Func<object, object> GetCompiledFunc(Type type, string property)
{
var arg = Expression.Parameter(typeof(object), "x");
Expression expr = Expression.Convert(arg, type);
var propType = type;
foreach (var prop in property.Split('.'))
{
var pi = propType.GetProperty(prop);
if (pi == null) throw new ArgumentException(string.Format("Field {0} not found.", prop));
expr = Expression.Property(expr, pi);
propType = pi.PropertyType;
}
expr = Expression.Convert(expr, typeof(object));
var lambda = Expression.Lambda<Func<object, object>>(expr, arg);
return lambda.Compile();
}
private static Delegate GetCompiledDelegate(Type type, string property)
{
var arg = Expression.Parameter(type, "x");
Expression expr = arg;
var propType = type;
foreach (var prop in property.Split('.'))
{
var pi = propType.GetProperty(prop);
if (pi == null) throw new ArgumentException(string.Format("Field {0} not found.", prop));
expr = Expression.Property(expr, pi);
propType = pi.PropertyType;
}
var delegateType = typeof(Func<,>).MakeGenericType(type, propType);
var lambda = Expression.Lambda(delegateType, expr, arg);
return lambda.Compile();
}
Btw: As you can see I've omitted storing compiled lambdas inside dictionary (like in the answer qiven by CSharpie), because dictionary lookup is time consuming when you compare it to compiled lambdas execution time.

Dealing with nulls indynamically created linq expression

I am dynamically creating a Linq expression based on string arrays and I'm running into a problem. The way that the expression is created and parenthesized it is causing this to throw an object null reference on Id 3. It creates this expression and If it were parenthesized correctly it wouldn't evaluate the second half of the expression and wouldn't throw an error I assume. Anyone have a way of creating the expression so it doesn't end up parenthesized like this?
{x => ((True And x.Id.ToString().ToLower().Contains("John")) Or ((x.Name != null) And x.Name.ToString().ToLower().Contains("John")))}
class Person
{
public int Id { get; set; }
public string Name { get; set; }
}
public class Test
{
public void someMethod()
{
var x = new List<Person>(new Person[] {
new Person { Id = 1, Name = "Jerry" },
new Person { Id = 2, Name = "Mary" },
new Person { Id = 3, Name = null },
new Person { Id = 4, Name = "John" },
new Person { Id = 5, Name = "Amy" }
});
var columns = new List<string>(new string[] {
"Name",
"Id"
});
var searchTerm = "John";
var searchColumns = columns.Select(a => new { ColName = a });
var type = typeof(Person);
ParameterExpression paramExpr = Expression.Parameter(type, "x");
Expression body = null;
var piList = new List<PropertyInfo>();
foreach (var s in searchColumns)
piList.Add(type.GetProperty(s.ColName));
if (piList[0].PropertyType.IsPrimitive || piList[0].PropertyType.Equals(typeof(DateTime)))
body = Expression.Constant(true);
else
body = Expression.NotEqual(Expression.Property(paramExpr, piList[0]), Expression.Constant(null, piList[0].PropertyType));
body = Expression.And(body,
Expression.Call(
Expression.Call(
Expression.Call(
Expression.Property(paramExpr, piList[0]),
typeof(Convert).GetMethod("ToString", Type.EmptyTypes)
),
typeof(string).GetMethod("ToLower", new Type[0])
),
typeof(string).GetMethod("Contains"),
Expression.Constant(searchTerm.ToLower())
));
for (int i = 1; i < piList.Count; i++)
{
Expression body1 = null;
if (piList[i].PropertyType.IsPrimitive || piList[i].PropertyType.Equals(typeof(DateTime)))
body1 = Expression.Constant(true);
else
body1 = Expression.NotEqual(Expression.Property(paramExpr, piList[i]), Expression.Constant(null, piList[i].PropertyType));
body =
Expression.Or(body,
Expression.And(body1,
Expression.Call(
Expression.Call(
Expression.Call(
Expression.Property(paramExpr, piList[i]),
typeof(Convert).GetMethod("ToString", Type.EmptyTypes)
),
typeof(string).GetMethod("ToLower", new Type[0])
),
typeof(string).GetMethod("Contains"),
Expression.Constant(searchTerm.ToLower())
)
));
}
var lambda = Expression.Lambda<Func<Person, bool>>(body, paramExpr);
}
}

Why not just build one of these. They both avoid the issue with null values for Name.
(x.Name ?? "").IndexOf("John", StringComparison.CurrentCultureIgnoreCase) >= 0
or, if you really want equality, not contains it as a substring
string.Equals(x.Name, "John", StringComparison.CurrentCultureIgnoreCase)
BTW - x.Id will never contain "John" and neither will lowercase strings.
Also, you might want to consider using a PredicateBuilder instead of building the expression directly.

How to create a LambdaExpression that copies all matching properties

I have made the basic stuff but I am stuck at creating the actually LambaExpresion:
Anyone have some pointers for what I want to write in the line with ? var COPYEXPRESSION = ...
public Expression<Func<TSource, TDestination>> GetOrCreateMapExpression<TSource, TDestination>()
{
return (Expression<Func<TSource, TDestination>>)
_expressionCache.GetOrAdd(new TypePair(typeof(TSource), typeof(TDestination)), tp =>
{
return CreateMapExpression(tp.SourceType, tp.DestinationType);
});
}
private LambdaExpression CreateMapExpression(Type source, Type destination)
{
ParameterExpression instanceParameter = Expression.Parameter(source);
var sourceMembers = source.GetProperties();
var destMembers = destination.GetProperties();
var matchingMembers = sourceMembers.Select(s =>
new
{
Source = s,
Dest = destMembers.FirstOrDefault(d =>
d.Name.Equals(s.Name) && d.PropertyType == s.PropertyType)
}).Where(map => map.Dest != null).ToArray();
var COPYEXPRESSION = ...
return Expression.Lambda(COPYEXPRESSION , instanceParameter);
}
Update
I have gotten the return type correct, but when unit testing this, the mapped classes are having null on the properties.
private LambdaExpression CreateMapExpression(Type source, Type destination)
{
ParameterExpression instanceParameter = Expression.Parameter(source);
var instance2Parameter = Expression.New(destination);
LabelTarget returnTarget = Expression.Label(destination);
var sourceMembers = source.GetProperties().Where(p => p.GetMethod.IsPublic);
var destMembers = destination.GetProperties().Where(p => p.SetMethod.IsPublic);
var matchingMembers = sourceMembers.Select(s =>
new
{
Source = s,
Dest = destMembers.FirstOrDefault(d =>
d.Name.Equals(s.Name) && d.PropertyType == s.PropertyType)
}).Where(map => map.Dest != null).ToArray();
var block = Expression.Block(Expression.Block(
matchingMembers.Select(p =>
Expression.Assign(
Expression.Property(instance2Parameter, p.Dest),
Expression.Property(instanceParameter, p.Source)))),
Expression.Label(returnTarget, instance2Parameter));
return Expression.Lambda(block, instanceParameter);
}
Solution
This worked for me:
return Expression.Lambda( Expression.MemberInit(Expression.New(destination),
matchingMembers.Select(p =>
Expression.Bind(p.Dest, Expression.Property(instanceParameter, p.Source)))),
instanceParameter);

Given
ParameterExpression instanceParameter = Expression.Parameter(source);
ParameterExpression instance2Parameter = Expression.Parameter(destination);
you need TWO parameters, one for the source and one for the destination...
UNLESS you are building a new destination, then you need an Expression.Variable for instance2Parameter where you'll put the Expression.New
var block = Expression.Block(
matchingMembers.Select(p =>
Expression.Assign(
Expression.Property(instance2Parameter, p.Dest),
Expression.Property(instanceParameter, p.Source)))
This is a block containing all the Expression.Assign
Note that you should check for the presence of a setter in Dest and for a getter in Source (but it's probably better to do it in the sourceMembers.Select.)

OrderBy with variable TKey

I have a list of FileInfo which I have to sort by different properties, e.g.
List<FileInfo> infoListOrdered = infoList.OrderBy(x => x.CreationTime).ToList();
Instead of writing the expression for every FileInfo property, would it be possible to pass the TKey as a parameter?

No, TKey is a type; the lamba x=>x.CreationTime is the value selector returning a value of type TKey. You can't select something based on its type only, for one there may be multiple property instances of that type in your class.

Here's a solution based on my (mis)understanding of the question. It gives OrderBy for each property prop that you might want to order by:
static void Main()
{
IEnumerable<FileInfo> infoList = XXX; // your source to sort
var orderByMeth = typeof(Enumerable).GetMethods().Single(m => m.Name == "OrderBy" && m.GetParameters().Length == 2);
var tFileInfo = typeof(FileInfo);
foreach (var prop in tFileInfo.GetProperties(BindingFlags.DeclaredOnly | BindingFlags.Public | BindingFlags.Instance))
{
var tKey = prop.PropertyType;
var xParam = Expression.Parameter(tFileInfo);
var propBody = Expression.Property(xParam, prop.GetMethod);
var lambda = Expression.Lambda(propBody, xParam);
var func = lambda.Compile();
var orderByMethConstr = orderByMeth.MakeGenericMethod(tFileInfo, tKey);
var result = orderByMethConstr.Invoke(null, new object[] { infoList, func, });
var infoListOrdered = (IOrderedEnumerable<FileInfo>)result;
// keep infoListOrdered; foreach through it to get that particular ordering
}
}