I have recently joined a project where a Sort method was conditionally passing in a lambda expression to a LINQ query to define which property should be sorted on. The problem was that the lambda expression was being passed in a Func<TEntity, Object> and not in an Expression<Func<TEntity, Object>> so that the sorting was taking place in memory and not on the database (because the overload of OrderBy that takes an IEnumerable is called). This is the version in SortWithDelegate (see below).
When I use an Expression<Func<TEntity, Object>> (see SortWithExpression below) then, when a string property is passed in the orderBy parameter, the ordering is done correctly in the database. However, when I try to sort on an integer (or datetime) using Expression<Func<TEntity, Object>> I get the following error:
Unable to cast the type 'System.Int32' to type 'System.Object'. LINQ to Entities only supports casting EDM primitive or enumeration types.
To avoid this I have to wrap the integer or datetime field to be sorted on inside an anonymous type like so: orderByFunc = sl => new {sl.ParentUnit.Id};. I understand I need to do this as the return type of the Func is Object. However, what I do not understand is why I need to do this when working with the LINQ to Entities provider but not the LINQ to Objects provider?
void Main()
{
var _context = new MyContext();
string sortProperty = "Id";
bool sortAscending = false;
IQueryable<Qualification> qualifications = _context.Qualifications.Include(q => q.ParentUnit);
qualifications = SortWithExpression(sortProperty, sortAscending, qualifications);
qualifications.Dump();
}
private static IQueryable<Qualification> SortWithDelegate(string orderBy, bool sortAscending, IQueryable<Qualification> qualificationsQuery)
{
Func<Qualification, Object> orderByFunc;
switch (orderBy)
{
case "Name":
orderByFunc = sl => sl.Name;
break;
case "ParentUnit":
orderByFunc = sl => sl.ParentUnit.Name;
break;
case "Id":
orderByFunc = sl => sl.ParentUnit.Id;
break;
case "Created":
orderByFunc = sl => sl.Created;
break;
default:
orderByFunc = sl => sl.Name;
break;
}
qualificationsQuery = sortAscending
? qualificationsQuery.OrderBy(orderByFunc).AsQueryable()
: qualificationsQuery.OrderByDescending(orderByFunc).AsQueryable();
return qualificationsQuery;
}
private static IQueryable<Qualification> SortWithExpression(string orderBy, bool sortAscending, IQueryable<Qualification> qualificationsQuery)
{
Expression<Func<Qualification, Object>> orderByFunc;
switch (orderBy)
{
case "Name":
orderByFunc = sl => sl.Name;
break;
case "ParentUnit":
orderByFunc = sl => sl.ParentUnit.Name;
break;
case "Id":
orderByFunc = sl => new {sl.ParentUnit.Id};
break;
case "Created":
orderByFunc = sl => new {sl.Created};
break;
default:
orderByFunc = sl => sl.Name;
break;
}
qualificationsQuery = sortAscending
? qualificationsQuery.OrderBy(orderByFunc)
: qualificationsQuery.OrderByDescending(orderByFunc);
return qualificationsQuery;
}
Added
Just thought I'd add my own solution to this problem. To avoid boxing int and datetime I've created a generic extension method on IQueryable<T> to which I pass in the lambda expression to indicate the sort field and a boolean indicating whether the sort order should be ascending or not:
public static IQueryable<TSource> OrderBy<TSource, TResult>(this IQueryable<TSource> query, Expression<Func<TSource, TResult>> func, bool sortAscending)
{
return sortAscending ?
query.OrderBy(func) :
query.OrderByDescending(func);
}
private static IQueryable<Qualification> Sort(string orderBy, bool sortAscending, IQueryable<Qualification> qualificationsQuery)
{
switch (orderBy)
{
case "Name":
return qualificationsQuery.OrderBy(sl => sl.Name, sortAscending);
case "ParentUnit":
return qualificationsQuery.OrderBy(s1 => s1.ParentUnit.Name, sortAscending);
default:
return qualificationsQuery.OrderBy(sl => sl.Name, sortAscending);
}
}
Expression trees as their name implies are expression about doing something.you can visit the expressions and interpret them to your own business or like lambda expressions you can compile them and invoke as a delegate.
When you pass the expression to orderby method in Linq to Entities, it will be visited by Linq to Entities and in your case that "Int32 to Object" exception will be generated because the way it is interpreted as a MemberInfo that turn into a column name for the database query. But when you use it as a Func delegate, it can not be translated and it will be invoked as a delegate for comparison in orderby method's sort algorithm.
Related
I am working on a dynamic query solution for a project. I want to avoid a bunch of if/else or switch statements just to change the [DynamicFieldName] part of these queries.
IQueryable<MyDataType> allItems = (from item in Context.MyDataTypes select item);
foreach (QueryEntry currentEntry in query.Fields)
{
allItems = allItems.Where(item => item.[DynamicFieldName] == currentEntry.Value);
}
The user gets to build the query via the GUI that has a variable number of fields. In the end, they will also have a variety of comparisons to choose from (Less than, greater than, equal, contains, etc.) that vary by data type.
What method can I use to build this programatically in a nice reusable fashion?
Have a look at this code:
public static class CustomQueryBuilder
{
//todo: add more operations
public enum Operator
{
Equal = 0,
GreaterThan = 1,
LesserThan = 2
}
public static IQueryable<T> Where<T>(this IQueryable<T> query, string property, Operator operation, object value)
{
//it's an item which property we are referring to
ParameterExpression parameter = Expression.Parameter(typeof(T));
//this stands for "item.property"
Expression prop = Expression.Property(parameter, property);
//wrapping our value to use it in lambda
ConstantExpression constant = Expression.Constant(value);
Expression expression;
//creating the operation
//todo: add more cases
switch (operation)
{
case Operator.Equal:
expression = Expression.Equal(prop, constant);
break;
case Operator.GreaterThan:
expression = Expression.GreaterThan(prop, constant);
break;
case Operator.LesserThan:
expression = Expression.LessThan(prop, constant);
break;
default:
throw new ArgumentException("Invalid operation specified");
}
//create lambda ready to use in queries
var lambda = Expression.Lambda<Func<T, bool>>(expression, parameter);
return query.Where(lambda);
}
}
Usage
var users = context
.Users
.Where("Name", CustomQueryBuilder.Operator.Equal, "User")
.ToList();
Which is equal to
var users = context
.Users
.Where(u => u.Name == "User")
.ToList();
In short, I'm looking to do what this guy did, but with Entity Framework 6.
Implementing the proposed solution results in the error "The LINQ expression node type 'Invoke' is not supported in LINQ to Entities." Since the proposed solution uses Invoke, this is obviously an issue.
I understand that there's a way to harness a custom Compose method to rewrite the expression tree without using Invoke, but I can't seem to wrap my head around it.
Here's what I'm trying to write.
I build an IQueryable<TEntity> dynamically using a QueryParameters object that's just a bag of properties to use for the WHERE clauses. TEntity is a standard code-first EF entity with data annotations all over the place. The query contruction looks something like this:
IQueryable<TEntity> query = Context.Set<TEntity>();
if (queryParams == null)
return query;
if (!string.IsNullOrWhiteSpace(queryParams.FirstName))
{
if (queryParams.ExactSearch)
{
query = query.Where(x => x.FirstName == queryParams.FirstName);
}
else
{
if (queryParams.PreferStartsWith)
{
query = query.Where(
x => x.FirstName.ToLower()
.StartsWith(
queryParams.FirstName
.ToLower()));
}
else
{
query = query.Where(
x => x.FirstName.ToLower()
.Contains(
queryParams.FirstName
.ToLower()));
}
}
}
// ... repeat for all of queryParams' string props.
// DateTime, int, bool, etc have their own filters.
This gets repeated for every query parameter for a string field to be queried. Obviously, this results in a lot of repeated code. I would love to be able to write a filter with a signature like this:
public static IQueryable<TEntity> Search<TEntity>(
this IQueryable<TEntity> query,
Expression<Func<TEntity, string>> fieldExpression,
string searchValue,
bool exactSearch = true,
bool useStartsWithOverContains = false) {...}
Which I can then consume like this:
if (!string.IsNullOrWhiteSpace(queryParams.FirstName))
{
query = query.Search(
x => x.FirstName,
queryParams.FirstName,
queryParams.ExactSearch,
queryParams.PreferStartsWith);
}
The closest I've come a definition for that extension method is the below, but as mentioned, it produces that "'Invoke' is not supported in LINQ to Entities" error:
public static IQueryable<TEntity> Search<TEntity>(
this IQueryable<TEntity> query,
Expression<Func<TEntity, string>> fieldExpression,
string searchValue,
bool exactSearch = true,
bool useStartsWithOverContains = false)
{
if (string.IsNullOrWhiteSpace(searchValue))
return query;
searchValue = searchValue.Trim();
Expression<Func<TEntity, bool>> expression;
if (exactSearch)
{
var x = Expression.Parameter(typeof(TEntity), "x");
var left = Expression.Invoke(fieldExpression, x);
var right = Expression.Constant(searchValue);
var equalityExpression = Expression.Equal(left, right);
expression = Expression.Lambda<Func<TEntity, bool>>(
equalityExpression,
x);
}
else
{
searchValue = searchValue.ToLower();
var x = Expression.Parameter(typeof(TEntity), "x");
var fieldToLower = Expression.Call(
Expression.Invoke(fieldExpression, x),
typeof(string).GetMethod(
"ToLower",
Type.EmptyTypes));
var searchValueExpression =
Expression.Constant(searchValue);
var body = Expression.Call(
fieldToLower,
typeof(string).GetMethod(
useStartsWithOverContains ? "StartsWith" : "Contains",
new[] { typeof(string) }),
searchValueExpression);
expression = Expression.Lambda<Func<TEntity, bool>>(
body,
x);
}
return query.Where(expression);
}
I started to include the Compose method I mentioned, but I got lost really quickly, and thus removed it.
Open to any guidance! Thank you!
This is much easier to do by composing expressions than it is by trying to manually construct the expressions every single time. It's faster to write, so much less error prone, and actually ends up with code you can actually read at the end of it. All you need to do is write the code for how you use the value in the composed expression, which you already have from your original code.
public static IQueryable<TEntity> Search<TEntity>(
this IQueryable<TEntity> query,
Expression<Func<TEntity, string>> fieldExpression,
string searchValue,
bool exactSearch = true,
bool useStartsWithOverContains = false)
{
if (string.IsNullOrWhiteSpace(searchValue))
return query;
searchValue = searchValue.Trim();
if (exactSearch)
{
return query.Where(fieldExpression.Compose(field => field == searchValue));
}
else if (useStartsWithOverContains)
{
return query.Where(fieldExpression.Compose(field => field.StartsWith(searchValue.ToLower())));
}
else
{
return query.Where(fieldExpression.Compose(field => field.Contains(searchValue.ToLower())));
}
}
Note you should probably go with an enum for "Comparison" or something like that, rather than having two booleans. For example, right now someone can say that they don't want an exact sure but that they do want to use starts with. Just have one parameter with the three options.
For example, I want to try something like this. The sorting par might have none, 1 or several sorting columns by different order. But I can't use the ThenBy method as it's only available after an OrderBy. The code below will keep resetting the order to the last item in the sorting list. I don't want to change the method signature either. Help would be greatly appreciated, thanks.
public IQueryable<Person> FilterList(string forename, List<Sorting> sorting)
{
IQueryable<Person> query = dc.Set<Person>();
if(!string.IsNullOrEmpty(forename)){
query = query.Where(w=>w.Forename.Contains(forename));
foreach(var sort in sorting)
{
switch(sort.By)
{
case "asc":
switch(sort.Sort)
{
case "forename":
query = query.OrderBy(o=>o.Forename);
break;
case "surname":
query = query.OrderBy(o=>o.Surname);
break;
}
break;
case "desc":
switch(sort.Sort)
{
case "forename":
query = query.OrderByDescending(o=>o.Forename);
break;
case "surname":
query = query.OrderByDescending(o=>o.Surname);
break;
}
break;
}
}
return query;
}
public class Sorting
{
public string Sort{get;set;}
public string By{get;set;}
}
AgentFire's answer is effectively appropriate, but a bit long-winded to special-case in each situation. I'd add an extension method:
EDIT: Apparently, the code below doesn't correctly determine whether the query is already ordered; even the unordered version implements IOrderedQueryable<T>, apparently. Using source.Expression.Type to check for IOrderedQueryable<T> apparently works, but I've left this code here as the more logical approach - and as this "fix" sounds more brittle than I'd like for an answer with my name on it :)
public static IOrderedQueryable<T> AddOrdering<T, TKey>(
this IQueryable<T> source,
Expression<Func<T, TKey>> keySelector,
bool descending)
{
IOrderedQueryable<T> ordered = source as IOrderedQueryable<T>;
// If it's not ordered yet, use OrderBy/OrderByDescending.
if (ordered == null)
{
return descending ? source.OrderByDescending(keySelector)
: source.OrderBy(keySelector);
}
// Already ordered, so use ThenBy/ThenByDescending
return descending ? ordered.ThenByDescending(keySelector)
: ordered.ThenBy(keySelector);
}
Then your calling code becomes:
foreach(var sort in sorting)
{
bool descending = sort.By == "desc";
switch (sort.Sort)
{
case "forename":
query = query.AddOrdering(o => o.Forename, descending);
break;
case "surname":
query = query.AddOrdering(o => o.Surname, descending);
break;
}
}
This way each additional sort option only adds a tiny code overhead.
This is the code I'm using, tried and tested. Based on Jon Skeets solution, credit to him I merely tweaked for my purposes.
public static IOrderedQueryable<T> AddOrdering<T, TKey>(
this IQueryable<T> source,
Expression<Func<T, TKey>> keySelector,
bool descending)
{
// If it's not ordered yet, use OrderBy/OrderByDescending.
if(source.Expression.Type != typeof(IOrderedQueryable<T>))
{
return descending ? source.OrderByDescending(keySelector)
: source.OrderBy(keySelector);
}
// Already ordered, so use ThenBy/ThenByDescending
return descending ? ((IOrderedQueryable<T>)source).ThenByDescending(keySelector)
: ((IOrderedQueryable<T>)source).ThenBy(keySelector);
}
Then your calling code becomes:
foreach(var sort in sorting)
{
bool descending = sort.By == "desc";
switch (sort.Sort)
{
case "forename":
query = query.AddOrdering(o => o.Forename, descending);
break;
case "surname":
query = query.AddOrdering(o => o.Surname, descending);
break;
}
}
You might want to use cast:
var sortedQueryable = query as IOrderedQueryable<Person>;
if (sortedQueryable != null)
{
query = sortedQueryable.ThenBy(o => o.Forename);
}
else
{
query = query.OrderBy(o => o.Forename);
}
In each iteration it will do.
Also, I would change magic strings "asc" and "desc" to enum (like enum SortMode), and List<Sorting> parameter to this:
List<Tuple<Expression<Func<Person, object>>, SortMode>> parameter;
So in your foreach loop you would only need to if for SortMode and pass the Item2 of a tuple to orderers.
First order by a constant so you have an IOrderedQueryable<T>, then you can do all your custom ordering through the ThenBy method.
public IQueryable<Person> FilterList(string forename, List<Sorting> sorting) {
IQueryable<Person> query = dc.Set<Person>();
if(!string.IsNullOrEmpty(forename)){
query = query.Where(w=>w.Forename.Contains(forename));
var orderedQuery = query.OrderBy(x => 1);
foreach(var sort in sorting) {
switch(sort.By) {
case "asc":
switch(sort.Sort) {
case "forename":
orderedQuery = orderedQuery.ThenBy(o=>o.Forename);
break;
case "surname":
orderedQuery = orderedQuery.ThenBy(o=>o.Surname);
break;
}
break;
case "desc":
switch(sort.Sort) {
case "forename":
orderedQuery = orderedQuery.ThenByDescending(o=>o.Forename);
break;
case "surname":
orderedQuery = orderedQuery.ThenByDescending(o=>o.Surname);
break;
}
break;
}
}
return orderedQuery;
}
There are a lot of options to do it. For example:
bool isSorted = false;
foreach(var sort in sorting)
{
switch(sort.By)
{
case "asc":
switch(sort.Sort)
{
case "forename":
if (!isSorted)
{
query = query .OrderBy(o => o.Forename);
isSorted = true;
}
else
{
query = ((IOrderedQueryable<Person>)query).ThenBy(o => o.Forename);
}
...
}
break;
case "desc":
...
}
}
Edit
Thanks to AgentFire for pointing out my error.
I am using Entity Framework and am building up a IQueryable<T>
IQueryable<Message> query = db.Messages;
query = query.OrderByDescending(m => m.Created);
query = query.Where(m => m.Deleted == false);
if (lastTime != null)
query = query.Where(m => m.Created < lastTime);
var func = ExpressionHelper.GetPredicate(stream);
query = query.Where(func).AsQueryable; *** ISSUE HERE?? ***
query = query.Skip(skip)
.Take(count)
.Include(m => m.Tags)
.Include(m => m.Properties);
var results = query.ToList();
The issue is the Tags and Properties are not populated in the final list.
I believe it has something to do with the Func<> I am passing in and also believe that after it although AsQueryable is used it doesn't represent IQueryable or does no longer connect to the database.
Is there a way to get Tags and Properties populated?
I'm not sure if making the Func<> to be Expression<Func<>> would help and if so is there a way to convert the below to be Expression<Func<>>
UPDATE:
public static Func<Message, bool> GetPredicate(string expression)
{
Func<Message, bool> result = null;
try
{
ParameterExpression parameter = Expression.Parameter(typeof(Classes.Message), "Message");
var lambda = DynamicExpression.ParseLambda(new[] { parameter }, null, expression);
result = lambda.Compile() as Func<Message, bool>;
}
catch (Exception e)
{
Log.Fatal(e);
}
return result;
}
At this point here:
query.Where(func) // where func is Func<...>
you have switched into LINQ-to-Objects. Anything you do after than (in terms of .Include etc) is irrelevant - you are no longer composing an EF query. You have a thin IQueryable<T> wrapper over a LINQ-to-Objects version of the sequence as it was at that line, i.e.
query = query.Where(func).AsQueryable(); // this is just a thing veneer over L2O
Switching to Expression<Func<...>> is likely to help.
If you absolutely positively can't generate an Expression, you could move the Include etc above this point.
I haven't actually wired this in so it may be slightly off, but as Marc says you've swapped into Linq to Objects at the AsQueryable.
If you remove that call, and then switch the GetPredicate to below it should work. It's the same except that we don't need to compile the lambda, this will happen when the query reaches entity framework.
public static Expression<Func<Message, bool>> GetPredicate(string expression)
{
Expression<Func<Message, bool>> result = null;
try
{
ParameterExpression parameter = Expression.Parameter(typeof(Classes.Message), "Message");
var lambda = DynamicExpression.ParseLambda(new[] { parameter }, null, expression);
result = (Expression<Func<Message, bool>>)lambda;
}
catch (Exception e)
{
Log.Fatal(e);
}
return result;
}
where can I find a good example of using linq & lambda expressions to generate dynamic sql?
For example, I need a method to take these parameters
GoupOperator ('And' or 'OR')
A list of objects each with the following parameters:
SearchColumn.
SearchValue.
SearchOperator (equals, contains, does not equal ...)
And another method to orderby any particular column ascending or descending
If this question has been properly answered before , I will gladly delete it - none of previous answers Ive seen are comprehensive enough for a person new to linq expressions to plug into an existing application with little trouble --thanks
I found a couple of linq extension methods (generic Where and OrderBy methods written by Ilya Builuk that take the columnname, search value and search operations plus a grouping operator) on codeproject here that shows how to do this using asp.net mvc.
the methods construct a dynamic expression tree -very elegant solution.
Since I had started using a traditional asmx web service, I used his helpers in my project and just made a few changes to get it running here -
Here are the 2 methods
public static class LinqExtensions
{
/// <summary>Orders the sequence by specific column and direction.</summary>
/// <param name="query">The query.</param>
/// <param name="sortColumn">The sort column.</param>
/// <param name="ascending">if set to true [ascending].</param>
public static IQueryable<T> OrderBy<T>(this IQueryable<T> query, string sortColumn, string direction)
{
string methodName = string.Format("OrderBy{0}",
direction.ToLower() == "asc" ? "" : "descending");
ParameterExpression parameter = Expression.Parameter(query.ElementType, "p");
MemberExpression memberAccess = null;
foreach (var property in sortColumn.Split('.'))
memberAccess = MemberExpression.Property
(memberAccess ?? (parameter as Expression), property);
LambdaExpression orderByLambda = Expression.Lambda(memberAccess, parameter);
MethodCallExpression result = Expression.Call(
typeof(Queryable),
methodName,
new[] { query.ElementType, memberAccess.Type },
query.Expression,
Expression.Quote(orderByLambda));
return query.Provider.CreateQuery<T>(result);
}
public static IQueryable<T> Where<T>(this IQueryable<T> query,
string column, object value, WhereOperation operation)
{
if (string.IsNullOrEmpty(column))
return query;
ParameterExpression parameter = Expression.Parameter(query.ElementType, "p");
MemberExpression memberAccess = null;
foreach (var property in column.Split('.'))
memberAccess = MemberExpression.Property
(memberAccess ?? (parameter as Expression), property);
//change param value type
//necessary to getting bool from string
ConstantExpression filter = Expression.Constant
(
Convert.ChangeType(value, memberAccess.Type)
);
//switch operation
Expression condition = null;
LambdaExpression lambda = null;
switch (operation)
{
//equal ==
case WhereOperation.Equal:
condition = Expression.Equal(memberAccess, filter);
lambda = Expression.Lambda(condition, parameter);
break;
//not equal !=
case WhereOperation.NotEqual:
condition = Expression.NotEqual(memberAccess, filter);
lambda = Expression.Lambda(condition, parameter);
break;
//string.Contains()
case WhereOperation.Contains:
condition = Expression.Call(memberAccess,
typeof(string).GetMethod("Contains"),
Expression.Constant(value));
lambda = Expression.Lambda(condition, parameter);
break;
}
MethodCallExpression result = Expression.Call(
typeof(Queryable), "Where",
new[] { query.ElementType },
query.Expression,
lambda);
return query.Provider.CreateQuery<T>(result);
}
}
Below is how I used these methods, the return object is simply a custom object that supplies data to a client side grid
public class Service1 : System.Web.Services.WebService
{
[WebMethod]
[ScriptMethod(ResponseFormat = ResponseFormat.Json)]
public JgGrid SearchGrid(int rows, int page, string sidx, string sord,string filters)
{
AdvWorksDataContext dc = new AdvWorksDataContext();
JavaScriptSerializer serializer = new JavaScriptSerializer();
filters f = serializer.Deserialize<filters>(filters);
var p = dc.vProductAndDescriptions.AsQueryable();
if (f.groupOp == "AND")
foreach (var rule in f.rules)
p = p.Where<vProductAndDescription>(
rule.field, rule.data,
(WhereOperation)StringEnum.Parse(typeof(WhereOperation), rule.op)
);
else
{
//Or
var temp = (new List<vProductAndDescription>()).AsQueryable();
foreach (var rule in f.rules)
{
var t = p.Where<vProductAndDescription>(
rule.field, rule.data,
(WhereOperation)StringEnum.Parse(typeof(WhereOperation), rule.op)
);
temp = temp.Concat<vProductAndDescription>(t);
}
p = temp;
}
p = p.OrderBy<vProductAndDescription>(sidx, sord);
return new JgGrid(page, p, rows);
}
}
For cases where I have many columns that need dynamic query composition I use Dynamic Linq. This is a library written as an example for .net 3.5 and it illustrates how you can write linq extensions that operate on the expression tree.
It can also be used for composing dynamic queries based on strings received from the client, such as column names, sorting, etc.
Here's a link a to an article posted by Scott Guthrie
http://weblogs.asp.net/scottgu/archive/2008/01/07/dynamic-linq-part-1-using-the-linq-dynamic-query-library.aspx
In the article you will find the links to the examples library which contains the source code for the Dynamic Linq library.
It seems to me that you're trying to build a linq provider...
Try to check this tutorial serie on how to implement a custom Linq to SQL provider : LINQ: Building an IQueryable provider series