Develop Reference

Complex expression in where clause in Entity Framework in repository

I have got the following expression that works with mockup data - hereby not using Entity Framework:
public static Expression<Func<Resource, bool>> FilterResourcesByUserCriteria(IEnumerable<FilterValue> filterValuesForUser)
{
Expression<Func<Resource, bool>> filter = (resource) =>
// Get filter values for the current resource in the loop
resource.ResourceFilterValues
// Group filter values for user
.GroupBy(filterValue => filterValue.FilterValue.FilterGroup.Id)
// Each group must fulfill the following logic
.All(filterGroup =>
// For each filter group, only select the user values from the same group
filterValuesForUser
.Where(filterValueForUser => filterValueForUser.FilterGroup.Id == filterGroup.Key)
.Select(filterValueForUser => filterValueForUser.FilterValue1)
// Each group must at least one value in the sublist of filter values of the current user
.Any(filterValueForUser => filterGroup
.Select(resourceFilterValue => resourceFilterValue.FilterValue.FilterValue1)
.Any(x => x == filterValueForUser))
);
}
However, I get this famous exception when I try to insert this expression in the where clause of my repository method (using Entity Framework):
Unable to create a constant value of type. Only primitive types or enumeration types are supported in this context.
I suspect this has something to do with a parameter called filterValuesForUser, which is a collection of a complex (i.e. custom) type.
Is this behavior even possible in Entity Framework where I do a subquery that is not directly related to Entity Framework? What I want to achieve here is to query on a subset of a custom list for each group in the query.
Any solutions for this or other workarounds? I'd like to minimize the amount of database calls, preferrably limit it to just one.

The exact query you are asking for is impossible with LinqToEF (due to limitation of SQL). But fear not. It is possible to salvage your problem with a slight tweaking.
public static Expression<Func<Resource, bool>> FilterResourcesByUserCriteria(FilterValue filterValueForUser)
{
//I assume you can write this part yourself.
}
public IQueryable<Resource> GetResources()
{
IQueryable<Resource> resources = _context.Resources;
IEnumerable<FilterValue> filterValuesForUser = GetFilterValues();
IEnumerable<IQueryable<Resource>> queries = from filter in filterValuesForUser
let filterExp = FilterResourcesByUserCriteria(filter)
select resources.Where(filterExp);
return Enumerable.Aggregate(queries, (l, r) => Queryable.Concat(l, r));
}
Types and Extension methods expanded for clarity.

In addition to Aron's answer, I used the PredicateBuilder utility in the LinqKit assembly to generate 1 expression rather than multiple and separate expresssions. This also avoids doing multiple database calls.
Here is how you can achieve this (pseudo-code):
public IQueryable<Resource> GetResources()
{
MyContext ctx = new MyContext ();
IEnumerable<Expression<Func<Resource, bool>>> queries =
filterValuesForUser.GroupBy(x => x.FilterGroup)
.Select(filter => SecurityFilters.FilterResourcesByUserCriteriaEF(filter.Select(y => y.FilterValue1)))
.Select(filterExpression => { return filterExpression; });
Expression<Func<Resource, bool>> query = PredicateBuilder.True<Resource>();
foreach (Expression<Func<Resource, bool>> filter in queries)
{
query = query.And(filter);
}
return ctx.Resources.AsExpandable().Where(query);
}
public static Expression<Func<Resource, bool>> FilterResourcesByUserCriteriaEF(IEnumerable<string> filterValuesForUser)
{
// From the resource's filter values, check if there are any present in the user's filter values
return (x) => x.ResourceFilterValues.Any(y => filterValuesForUser.Contains(y.FilterValue.FilterValue1));
}
I'm still having issues with getting this working in my repository but that has something do with something blocking AsExpandable() from working properly.

Effectively Using LINQ in C#

I have an app written with C#. I have a collection of objects and I need to perform tasks like sorting and jumping to a page of the objects. Currently, I have something that looks like this:
this.Items = model.Items
.Where(x => x.Id > 0)
.ToList();
if (SortBy == "Name")
{
if (Direction == "Descending")
this.Items = (this.Items.OrderByDescending(x => x.Name)).ToList();
else
this.Items = (this.Items.OrderBy(x => x.Name)).ToList();
}
else if (SortBy == "Description")
{
if (Direction == "Descending")
this.Items = (this.Items.OrderByDescending(x => x.Description)).ToList();
else
this.Items = (this.Items.OrderBy(x => x.Description)).ToList();
}
// I want to jump to the second page
this.Items = this.Items.Skip(26).ToList();
Is there a better way to do this? All of the ToList calls seem like that would be performance draining. Maybe I'm wrong though.
Thanks!

You can defer calling .ToList until your last line since based on your logic it always runs. Interim logic builds upon an IQueryable. Whether or not ToList has a significant impact largerly depends on whether this is Linq2Objects or it is Linq2SQL/Entitity Framework/Some other Database ORM. With database ORMs typically each .ToList hydrates the list and results in a database query executed. For Linq2Objects there is still a performance cost, but no where near the magnitude involved in a database call.
// itemsQuery will be of type IQueryable or IEnumerable depending on type of Items
var itemsQuery = model.Items.Where(x => x.Id > 0);
if (SortBy == "Description")
{
// we'll chain additional criteria or sorts to the IQueryable,
// and update the query reference by assigning over the previous itemsQuery variable
if (Direction == "Descending")
itemsQuery = itemsQuery.OrderByDescending(x => x.Description);
else
itemsQuery = itemsQuery.OrderBy(x => x.Description);
}
// I want to jump to the second page
this.Items = itemsQuery.Skip(26).ToList();
Essentially you are dynamically building up a query, but not actually running the query until the end where you call ToList, which will essentially build an SQL query from the expression tree and execute the entire chained sequence of .Where(...).OrderBy(...).Skip(26).ToList

You're quite right that all of the ToList calls are a major problem. Not only are they forcing all of the work to be done in the client if the underlying data source represents a database query provider, but even if it is inherently a LINQ to objects query you're needlessly creating, populating, and discarding lists. You're also removing any ability for deferred execution. Since you're not actually getting anything out of those calls, you can simply remove them all.
Next, you can write a method to create a selector, so that you don't need to duplicate all of that code for each column you sort on:
public static Expression<Func<T, TResult>> CreateSelector<T, TResult>(
string memberName)
{
var param = Expression.Parameter(typeof(T));
var body = Expression.PropertyOrField(param, memberName);
return Expression.Lambda<Func<T, TResult>>(body, param);
}
This allows you to write:
IQueryable<Item> items = model.Items;
if (Direction == "Descending")
items = items.OrderByDescending(CreateSelector<Item, string>(SortBy));
else
items = items.OrderBy(CreateSelector<Item, string>(SortBy));
items = items.Skip(26);
If the underlying data source that you have is an IEnumerable, not an IQueryable, just change the type of Items and Compile the selector expression.

Odata Error translating Linq expression at call Contains

Is it possible using ASP.NET Web APi OData make similar:
List<string> customersTitles = Odata.OrdersService.Select(o=>o.CustomerTitle).Distinct().ToList();
List<Customer> customers = Odata.CustomerService.Where(m => customersTitles .Contains(m.CustomerTitle))
Get error:
Error translating Linq expression to URI: The expression
value(System.Collections.Generic.List`1[System.String]).Contains([10007].CustomerTitle)
is not supported.}
API:
public class CustomerController : EntitySetController<Customer, int>
{
[Queryable]
public override IQueryable<Customer> Get()
{
Expression filter = this.QueryOptions.Filter.ToExpression<Customer>();
return db.Query<Customer>(filter as Expression<Func<Customer, bool>>);
}
}

The Contains construct is not supported via URIs since the list of strings existing at the client side is not a Server side resource.
Linq2Sql Provider has an innate translation for Contains, which gets translated to IN clause for SQL.
With OData, such a translation is not supported. What you need to build is an expanded query list for your where clause using all the Title values:
Because this does not work:
List<Customer> customers = Odata.CustomerService.Where(m => customersTitles .Contains(m.CustomerTitle))
the expanded query option helps us in building a query like:
List<Customer> customers = Odata.CustomerService.Where(m => m.CustomerTitle == customerTitles[0] || m.CustomerTitle == customerTitles[1]); // and so on
Here is the code for the filter building:
var titleFilterList = customerTitles.Select(title => String.Format("(CustomerTitle eq {0})", title));
var titleFilter = String.Join(" or ", titleFilterList);
var customers = Odata.CustomerService.AddQueryOption("$filter", titleFilter).Execute().ToList(); // you may have to cast this.
There is another option to do the same in a strongly typed manner using a nice extension method and building a dynamic Expression based predicate. Follow the steps from here:
http://blogs.msdn.com/b/phaniraj/archive/2008/07/17/set-based-operations-in-ado-net-data-services.aspx

The following extension method can be used LINQ to perform OData queries that test if the value of a property is contained in a set similar to how Contains works with LINQ to EF. It is based on the link provided in Raja Nadar's answer, specifically the comment at the end by Nick.
public static IQueryable<T> WherePropertyIsIn<T, TSet>(
this IQueryable<T> query,
IEnumerable<TSet> set,
Expression<Func<T, TSet>> propertyExpression
) {
var filterPredicate = set.Select(value => Expression.Equal(propertyExpression.Body, Expression.Constant(value)))
.Aggregate<Expression, Expression>(Expression.Constant(false), Expression.Or);
var filterLambdaExpression = Expression.Lambda<Func<T, bool>>(filterPredicate, propertyExpression.Parameters.Single());
return query.Where(filterLambdaExpression);
}
Usage:
var allowed_states = getAllowedStates();
var maxPopulation = getMaxPopulation();
// Instead of...
var cities = context.Cities.Where(c => allowed_states.Contains(c.State) && c.Population <= maxPopulation);
// Use...
var cities = context.Cities.Where(c => c.Population <= maxPopulation).WherePropertyIsIn(allowed_states, c => c.Cities);
Note that you must have a separate Where call (as shown above) if you wish to filter by more than WherePropertyIsIn allows. It would be nice if this could be combined into a single Where but I couldn't figure out how.

I have run into issues with Timothy's solution when using it against Configuration Manager's OData service. CM utilises WML internally and therefore is even more limited in what can be parsed.
I have changed the code to generate a bit 'cleaner' query, which avoid using Expression.Constant(false) as aggregate base -> it produces 0 that is not accepted by WML
public static IQueryable<T> WherePropertyIsIn<T, TSet>(this IQueryable<T> query,
IEnumerable<TSet> valuesList, Expression<Func<T, TSet>> propertySelector)
{
if (valuesList == null) throw new ArgumentNullException(nameof(valuesList));
//if there are no values, no entities can fullfil the condition -> return empty
if (!valuesList.Any())
return Enumerable.Empty<T>().AsQueryable();
//create a check for each value
var filters = valuesList.Select(value => Expression.Equal(propertySelector.Body, Expression.Constant(value)));
//build an expression aggregating checks with OR, use first check as starter (could be '0', but doesn't get mapped to WML)
var firstCheck = filters.First();
//we could duplicate first check, but why not just skip it
var filterPredicate = filters.Skip(1).Aggregate(firstCheck, (Func<Expression, Expression, Expression>)Expression.Or);
var filterLambdaExpression = Expression.Lambda<Func<T, bool>>(filterPredicate, propertySelector.Parameters.Single());
return query.Where(filterLambdaExpression);
}
}
A side effect is that when there are no values passed as argument, the query will immediately return an empty result, which may save time comparing to processing it by external service. At the same time it may be undesired if query is meant to actually run on server.

How to use LINQ Contains() to find a list of enums?

I have an enum called OrderStatus, and it contains various statuses that an Order can be in:
Created
Pending
Waiting
Valid
Active
Processed
Completed
What I want to do is create a LINQ statement that will tell me if the OrderStaus is Valid, Active, Processed or Completed.
Right now I have something like:
var status in Order.Status.WHERE(status =>
status.OrderStatus == OrderStatus.Valid ||
status.OrderStatus == OrderStatus.Active||
status.OrderStatus == OrderStatus.Processed||
status.OrderStatus == OrderStatus.Completed)
That works, but it's very "wordy". Is there a way to convert this to a Contains() statement and shorten it up a bit?

Sure:
var status in Order.Status.Where(status => new [] {
OrderStatus.Valid,
OrderStatus.Active,
OrderStatus.Processed,
OrderStatus.Completed
}.Contains(status.OrderStatus));
You could also define an extension method In() that would accept an object and a params array, and basically wraps the Contains function:
public static bool In<T>(this T theObject, params T[] collection)
{
return collection.Contains(theObject);
}
This allows you to specify the condition in a more SQL-ish way:
var status in Order.Status.Where(status =>
status.OrderCode.In(
OrderStatus.Valid,
OrderStatus.Active,
OrderStatus.Processed,
OrderStatus.Completed));
Understand that not all Linq providers like custom extension methods in their lambdas. NHibernate, for instance, won't correctly translate the In() function without additional coding to extend the expression parser, but Contains() works just fine. For Linq 2 Objects, no problems.

I have used this extension:
public static bool IsIn<T>(this T value, params T[] list)
{
return list.Contains(value);
}
You may use this as the condition:
Where(x => x.IsIn(OrderStatus.Valid, ... )

If that set of statuses has some meaning, for example those are statuses for accepted orders, you can define an extension method on your enum and use that in your linq query.
public static class OrderStatusExtensions
{
public static bool IsAccepted(this OrderStatuses status)
{
return status == OrderStatuses.Valid
|| status == OrderStatuses.Active
|| status == OrderStatuses.Processed
|| status == OrderStatuses.Completed;
}
}
var acceptedOrders = from o in orders
where o.Status.IsAccepted()
select o;
Even if you could not give the method a simple name, you could still use something like IsValidThroughCompleted. In either case, it seems to convey a little more meaning this way.

Assumnig that the enum is defined in the order you specified in the question, you could shorten this by using an integer comparison.
var result =
Order.Status.Where(x =>
(int)x >= (int)OrderStatus.Valid &
& (int)x <= (int)OrderStatus.Completed);
This type of comparison though can be considered flaky. A simply re-ordering of enumeration values would silently break this comparison. I would prefer to stick with the more wordy version and probably clean up it up by refactoring out the comparison to a separate method.

You could put these in a collection, and use:
OrderStatus searchStatus = new[] {
OrderStatus.Valid,
OrderStatus.Active,
OrderStatus.Processed,
OrderStatus.Completed };
var results = Order.Status.Where(status => searchStatus.Contains(status));

Why does this LINQ-to-SQL query get a NotSupportedException?

The following LINQ statement:
public override List<Item> SearchListWithSearchPhrase(string searchPhrase)
{
List<string> searchTerms = StringHelpers.GetSearchTerms(searchPhrase);
using (var db = Datasource.GetContext())
{
return (from t in db.Tasks
where searchTerms.All(term =>
t.Title.ToUpper().Contains(term.ToUpper()) &&
t.Description.ToUpper().Contains(term.ToUpper()))
select t).Cast<Item>().ToList();
}
}
gives me this error:
System.NotSupportedException: Local
sequence cannot be used in LINQ to SQL
implementation of query operators
except the Contains() operator.
Looking around it seems my only option is to get all my items first into a generic List, then do a LINQ query on that.
Or is there a clever way to rephrase the above LINQ-to-SQL statement to avoid the error?
ANSWER:
Thanks Randy, your idea helped me to build the following solution. It is not elegant but it solves the problem and since this will be code generated, I can handle up to e.g. 20 search terms without any extra work:
public override List<Item> SearchListWithSearchPhrase(string searchPhrase)
{
List<string> searchTerms = StringHelpers.GetSearchTerms(searchPhrase);
using (var db = Datasource.GetContext())
{
switch (searchTerms.Count())
{
case 1:
return (db.Tasks
.Where(t =>
t.Title.Contains(searchTerms[0])
|| t.Description.Contains(searchTerms[0])
)
.Select(t => t)).Cast<Item>().ToList();
case 2:
return (db.Tasks
.Where(t =>
(t.Title.Contains(searchTerms[0])
|| t.Description.Contains(searchTerms[0]))
&&
(t.Title.Contains(searchTerms[1])
|| t.Description.Contains(searchTerms[1]))
)
.Select(t => t)).Cast<Item>().ToList();
case 3:
return (db.Tasks
.Where(t =>
(t.Title.Contains(searchTerms[0])
|| t.Description.Contains(searchTerms[0]))
&&
(t.Title.Contains(searchTerms[1])
|| t.Description.Contains(searchTerms[1]))
&&
(t.Title.Contains(searchTerms[2])
|| t.Description.Contains(searchTerms[2]))
)
.Select(t => t)).Cast<Item>().ToList();
default:
return null;
}
}
}

Ed, I've run into a similiar situation. The code is below. The important line of code is where I set the memberList variable. See if this fits your situation. Sorry if the formatting didn't come out to well.
Randy
// Get all the members that have an ActiveDirectorySecurityId matching one in the list.
IEnumerable<Member> members = database.Members
.Where(member => activeDirectoryIds.Contains(member.ActiveDirectorySecurityId))
.Select(member => member);
// This is necessary to avoid getting a "Queries with local collections are not supported"
//error in the next query.
memberList = members.ToList<Member>();
// Now get all the roles associated with the members retrieved in the first step.
IEnumerable<Role> roles = from i in database.MemberRoles
where memberList.Contains(i.Member)
select i.Role;

Since you cannot join local sequence with linq table, the only way to translate the above query into SQL woluld be to create WHERE clause with as many LIKE conditions as there are elements in searchTerms list (concatenated with AND operators). Apparently linq doesn't do that automatically and throws an expception instead.
But it can be done manually by iterating through the sequence:
public override List<Item> SearchListWithSearchPhrase(string searchPhrase)
{
List<string> searchTerms = StringHelpers.GetSearchTerms(searchPhrase);
using (var db = Datasource.GetContext())
{
IQueryable<Task> taskQuery = db.Tasks.AsQueryable();
foreach(var term in searchTerms)
{
taskQuery = taskQuery.Where(t=>t.Title.ToUpper().Contains(term.ToUpper()) && t.Description.ToUpper().Contains(term.ToUpper()))
}
return taskQuery.ToList();
}
}
Mind that the query is still executed by DBMS as a SQL statement. The only drawback is that searchTerms list shouldn't be to long - otherwise the produced SQL statement won'tbe efficient.