Develop Reference

Linq to entities extension method inner query (EF6)

Can someone explain to me why the EF Engine is failing in the following scenario?
It works fine with the following expression:
var data = context.Programs
.Select(d => new MyDataDto
{
ProgramId = d.ProgramId,
ProgramName = d.ProgramName,
ClientId = d.ClientId,
Protocols = d.Protocols.Where(p => p.UserProtocols.Any(u => u.UserId == userId))
.Count(pr => pr.Programs.Any(pg => pg.ProgramId == d.ProgramId))
})
.ToList();
But if I encapsulate some into an extension method:
public static IQueryable<Protocol> ForUser(this IQueryable<Protocol> protocols, int userId)
{
return protocols.Where(p => p.UserProtocols.Any(u => u.UserId == userId));
}
The resulting query:
var data = context.Programs
.Select(d => new MyDataDto
{
ProgramId = d.ProgramId,
ProgramName = d.ProgramName,
ClientId = d.ClientId,
Protocols = d.Protocols.ForUser(userId)
.Count(pr => pr.Programs.Any(pg => pg.ProgramId == d.ProgramId))
})
.ToList();
Fails with the exception: LINQ to Entities does not recognize the method 'System.Linq.IQueryable1[DAL.Protocol] ForUser(System.Linq.IQueryable1[DAL.Protocol], Int32)' method, and this method cannot be translated into a store expression.
I would expect the EF Engine to build the entire expression tree, chaining the necessary expressions and then generate the SQL. Why doesn't it do that?

This is happening because the call to ForUser() is being made inside of the expression tree that the C# compiler builds when it sees the lambda you pass into Select. Entity Framework tries to figure out how to convert that function into SQL, but it can't invoke the function for a few reasons (e.g. d.Protocols does not exist at the moment).
The simplest approach that works for a case like this is to have your helper return a criteria lambda expression, and then pass that into the .Where() method yourself:
public static Expression<Func<Protocol, true>> ProtocolIsForUser(int userId)
{
return p => p.UserProtocols.Any(u => u.UserId == userId);
}
...
var protocolCriteria = Helpers.ProtocolIsForUser(userId);
var data = context.Programs
.Select(d => new MyDataDto
{
ProgramId = d.ProgramId,
ProgramName = d.ProgramName,
ClientId = d.ClientId,
Protocols = d.Protocols.Count(protocolCriteria)
})
.ToList();
More information
When you invoke a LINQ method outside of an expression tree (like you do with context.Programs.Select(...)), the Queryable.Select() extension method actually gets invoked, and its implementation returns an IQueryable<> that represents the extension method getting called on the original IQueryable<>. Here's the implementation of Select, for instance:
public static IQueryable<TResult> Select<TSource,TResult>(this IQueryable<TSource> source, Expression<Func<TSource, TResult>> selector) {
if (source == null)
throw Error.ArgumentNull("source");
if (selector == null)
throw Error.ArgumentNull("selector");
return source.Provider.CreateQuery<TResult>(
Expression.Call(
null,
GetMethodInfo(Queryable.Select, source, selector),
new Expression[] { source.Expression, Expression.Quote(selector) }
));
}
When the queryable's Provider has to generate actual data from the IQueryable<>, it analyzes the expression tree and tries to figure out how to interpret those method calls. Entity Framework has built-in knowledge of many LINQ-related functions like .Where() and .Select(), so it knows how to translate those method calls into SQL. However, it doesn't know what to do for methods that you write.
So why does this work?
var data = context.Programs.ForUser(userId);
The answer is that your ForUser method is not implemented like the Select method above: you are not adding an expression to the queryable to represent calling ForUser. Instead, you are returning the result of a .Where() call. From the IQueryable<>'s perspective, it's as if Where() was called directly, and the call to ForUser() never happened.
You can prove this by capturing the Expression property on the IQueryable<>:
Console.WriteLine(data.Expression.ToString());
... which will produce something like this:
Programs.Where(u => (u.UserId == value(Helpers<>c__DisplayClass1_0).userId))
There's no call to ForUser() anywhere in that expression.
On the other hand, if you include the ForUser() call inside of an expression tree like this:
var data = context.Programs.Select(d => d.Protocols.ForUser(id));
... then the .ForUser() method never actually gets invoked, so it never returns an IQueryable<> that knows the .Where() method got called. Instead, the expression tree for the queryable shows .ForUser() getting invoked. Outputting its expression tree would look something like this:
Programs.Select(d => d.Protocols.ForUser(value(Repository<>c__DisplayClass1_0).userId))
Entity Framework has no idea what ForUser() is supposed to do. As far as it's concerned, you could have written ForUser() to do something that's impossible to do in SQL. So it tells you that's not a supported method.

As I mentioned in my comment above, I can't tell why the EF Engine is working the way it is. Therefore, I've tried to find a way to re-write the query so I'll be able to make use of my extension methods.
The tables are:
Program -> 1..m -> ProgramProtocol -> m..1 -> Protocol
ProgramProtocol is just a join table and is not mapped in the model by Entity Framework.
The idea is simple: select "from left", select "from right" and then join the resulted sets for proper filtering:
var data = context.Programs.ForUser(userId)
.SelectMany(pm => pm.Protocols,
(pm, pt) => new {pm.ProgramId, pm.ProgramName, pm.ClientId, pt.ProtocolId})
.Join(context.Protocols.ForUser(userId), pm => pm.ProtocolId,
pt => pt.ProtocolId, (pm, pt) => pm)
.GroupBy(pm => new {pm.ProgramId, pm.ProgramName, pm.ClientId})
.Select(d => new MyDataDto
{
ProgramName = d.Key.ProgramName,
ProgramId = d.Key.ProgramId,
ClientId = d.Key.ClientId,
Protocols = d.Count()
})
.ToList();

Combining expression trees

I have the following expression:
public Expression<Func<T, bool>> UserAccessCheckExpression<T>(int userId) where T : class
{
return x => (IsAdmin || userId == CurrentUserId || userId == 0);
}
Then I want to apply this filter to several collections (IQueryable) like this one:
return tasks
.Where(t => t.TaskUsers
.Any(x => UserAccessCheckExpression<TaskUser>(x.User) && x.SomeBool == true));
I'm getting the following error while doing so:
Error 40 Cannot implicitly convert type System.Linq.Expressions.Expression<System.Func<TaskUser,bool>> to bool
I can't use workaround with interface inheritance (like TaskUser inherits interface with int UserId property (where T : IHasUserId)) since I want to combine logic.

The problem is that your UserAccessCheckExpression() method is returning an Expression while the Any() method is expecting a boolean.
Now, you can get your code to compile by compiling the Expression and invoking the method (using UserAccessCheckExpression<TaskUser>(x.User).Compile().Invoke(x.User)) but that would obviously fail on runtime because Linq-to-Entities wouldn't be able to translate your Any() to a store query as it no longer contains an Expression.
LinqKit is aiming to solve this problem using its own Invoke extension method that while letting your code compile, will make sure your Expression will get replaced back to its original form using another extension method named AsExpandable() that is extending the entity set.
Try this:
using LinqKit.Extensions;
return tasks
.AsExpandable()
.Where(t => t.TaskUsers.Any(
x => UserAccessCheckExpression<TaskUser>(x.User).Invoke(x)
&& x.SomeBool == true));
More on LinqKit

Yeah, so, you can't do that. There's a difference between an Expression<> and a Func<>. You're trying to use the UserAccessCheckExpression as a func. I'm not sure what you're trying to do, but you can compile it to a func and then use it sorta like you are:
var expr = UserAccessCheckExpression<TaskUser>(x.User);
var func = expr.Compile();
// Later use it like ...
var result = func();
But I expect you're using this with EF or Linq2Sql? That being the case you'll need to rewrite the expression. It can be done by hand (not easy) or, better, use a tool like PredicateBuilder.

Complex expression in where clause in Entity Framework in repository

I have got the following expression that works with mockup data - hereby not using Entity Framework:
public static Expression<Func<Resource, bool>> FilterResourcesByUserCriteria(IEnumerable<FilterValue> filterValuesForUser)
{
Expression<Func<Resource, bool>> filter = (resource) =>
// Get filter values for the current resource in the loop
resource.ResourceFilterValues
// Group filter values for user
.GroupBy(filterValue => filterValue.FilterValue.FilterGroup.Id)
// Each group must fulfill the following logic
.All(filterGroup =>
// For each filter group, only select the user values from the same group
filterValuesForUser
.Where(filterValueForUser => filterValueForUser.FilterGroup.Id == filterGroup.Key)
.Select(filterValueForUser => filterValueForUser.FilterValue1)
// Each group must at least one value in the sublist of filter values of the current user
.Any(filterValueForUser => filterGroup
.Select(resourceFilterValue => resourceFilterValue.FilterValue.FilterValue1)
.Any(x => x == filterValueForUser))
);
}
However, I get this famous exception when I try to insert this expression in the where clause of my repository method (using Entity Framework):
Unable to create a constant value of type. Only primitive types or enumeration types are supported in this context.
I suspect this has something to do with a parameter called filterValuesForUser, which is a collection of a complex (i.e. custom) type.
Is this behavior even possible in Entity Framework where I do a subquery that is not directly related to Entity Framework? What I want to achieve here is to query on a subset of a custom list for each group in the query.
Any solutions for this or other workarounds? I'd like to minimize the amount of database calls, preferrably limit it to just one.

The exact query you are asking for is impossible with LinqToEF (due to limitation of SQL). But fear not. It is possible to salvage your problem with a slight tweaking.
public static Expression<Func<Resource, bool>> FilterResourcesByUserCriteria(FilterValue filterValueForUser)
{
//I assume you can write this part yourself.
}
public IQueryable<Resource> GetResources()
{
IQueryable<Resource> resources = _context.Resources;
IEnumerable<FilterValue> filterValuesForUser = GetFilterValues();
IEnumerable<IQueryable<Resource>> queries = from filter in filterValuesForUser
let filterExp = FilterResourcesByUserCriteria(filter)
select resources.Where(filterExp);
return Enumerable.Aggregate(queries, (l, r) => Queryable.Concat(l, r));
}
Types and Extension methods expanded for clarity.

In addition to Aron's answer, I used the PredicateBuilder utility in the LinqKit assembly to generate 1 expression rather than multiple and separate expresssions. This also avoids doing multiple database calls.
Here is how you can achieve this (pseudo-code):
public IQueryable<Resource> GetResources()
{
MyContext ctx = new MyContext ();
IEnumerable<Expression<Func<Resource, bool>>> queries =
filterValuesForUser.GroupBy(x => x.FilterGroup)
.Select(filter => SecurityFilters.FilterResourcesByUserCriteriaEF(filter.Select(y => y.FilterValue1)))
.Select(filterExpression => { return filterExpression; });
Expression<Func<Resource, bool>> query = PredicateBuilder.True<Resource>();
foreach (Expression<Func<Resource, bool>> filter in queries)
{
query = query.And(filter);
}
return ctx.Resources.AsExpandable().Where(query);
}
public static Expression<Func<Resource, bool>> FilterResourcesByUserCriteriaEF(IEnumerable<string> filterValuesForUser)
{
// From the resource's filter values, check if there are any present in the user's filter values
return (x) => x.ResourceFilterValues.Any(y => filterValuesForUser.Contains(y.FilterValue.FilterValue1));
}
I'm still having issues with getting this working in my repository but that has something do with something blocking AsExpandable() from working properly.

Odata Error translating Linq expression at call Contains

Is it possible using ASP.NET Web APi OData make similar:
List<string> customersTitles = Odata.OrdersService.Select(o=>o.CustomerTitle).Distinct().ToList();
List<Customer> customers = Odata.CustomerService.Where(m => customersTitles .Contains(m.CustomerTitle))
Get error:
Error translating Linq expression to URI: The expression
value(System.Collections.Generic.List`1[System.String]).Contains([10007].CustomerTitle)
is not supported.}
API:
public class CustomerController : EntitySetController<Customer, int>
{
[Queryable]
public override IQueryable<Customer> Get()
{
Expression filter = this.QueryOptions.Filter.ToExpression<Customer>();
return db.Query<Customer>(filter as Expression<Func<Customer, bool>>);
}
}

The Contains construct is not supported via URIs since the list of strings existing at the client side is not a Server side resource.
Linq2Sql Provider has an innate translation for Contains, which gets translated to IN clause for SQL.
With OData, such a translation is not supported. What you need to build is an expanded query list for your where clause using all the Title values:
Because this does not work:
List<Customer> customers = Odata.CustomerService.Where(m => customersTitles .Contains(m.CustomerTitle))
the expanded query option helps us in building a query like:
List<Customer> customers = Odata.CustomerService.Where(m => m.CustomerTitle == customerTitles[0] || m.CustomerTitle == customerTitles[1]); // and so on
Here is the code for the filter building:
var titleFilterList = customerTitles.Select(title => String.Format("(CustomerTitle eq {0})", title));
var titleFilter = String.Join(" or ", titleFilterList);
var customers = Odata.CustomerService.AddQueryOption("$filter", titleFilter).Execute().ToList(); // you may have to cast this.
There is another option to do the same in a strongly typed manner using a nice extension method and building a dynamic Expression based predicate. Follow the steps from here:
http://blogs.msdn.com/b/phaniraj/archive/2008/07/17/set-based-operations-in-ado-net-data-services.aspx

The following extension method can be used LINQ to perform OData queries that test if the value of a property is contained in a set similar to how Contains works with LINQ to EF. It is based on the link provided in Raja Nadar's answer, specifically the comment at the end by Nick.
public static IQueryable<T> WherePropertyIsIn<T, TSet>(
this IQueryable<T> query,
IEnumerable<TSet> set,
Expression<Func<T, TSet>> propertyExpression
) {
var filterPredicate = set.Select(value => Expression.Equal(propertyExpression.Body, Expression.Constant(value)))
.Aggregate<Expression, Expression>(Expression.Constant(false), Expression.Or);
var filterLambdaExpression = Expression.Lambda<Func<T, bool>>(filterPredicate, propertyExpression.Parameters.Single());
return query.Where(filterLambdaExpression);
}
Usage:
var allowed_states = getAllowedStates();
var maxPopulation = getMaxPopulation();
// Instead of...
var cities = context.Cities.Where(c => allowed_states.Contains(c.State) && c.Population <= maxPopulation);
// Use...
var cities = context.Cities.Where(c => c.Population <= maxPopulation).WherePropertyIsIn(allowed_states, c => c.Cities);
Note that you must have a separate Where call (as shown above) if you wish to filter by more than WherePropertyIsIn allows. It would be nice if this could be combined into a single Where but I couldn't figure out how.

I have run into issues with Timothy's solution when using it against Configuration Manager's OData service. CM utilises WML internally and therefore is even more limited in what can be parsed.
I have changed the code to generate a bit 'cleaner' query, which avoid using Expression.Constant(false) as aggregate base -> it produces 0 that is not accepted by WML
public static IQueryable<T> WherePropertyIsIn<T, TSet>(this IQueryable<T> query,
IEnumerable<TSet> valuesList, Expression<Func<T, TSet>> propertySelector)
{
if (valuesList == null) throw new ArgumentNullException(nameof(valuesList));
//if there are no values, no entities can fullfil the condition -> return empty
if (!valuesList.Any())
return Enumerable.Empty<T>().AsQueryable();
//create a check for each value
var filters = valuesList.Select(value => Expression.Equal(propertySelector.Body, Expression.Constant(value)));
//build an expression aggregating checks with OR, use first check as starter (could be '0', but doesn't get mapped to WML)
var firstCheck = filters.First();
//we could duplicate first check, but why not just skip it
var filterPredicate = filters.Skip(1).Aggregate(firstCheck, (Func<Expression, Expression, Expression>)Expression.Or);
var filterLambdaExpression = Expression.Lambda<Func<T, bool>>(filterPredicate, propertySelector.Parameters.Single());
return query.Where(filterLambdaExpression);
}
}
A side effect is that when there are no values passed as argument, the query will immediately return an empty result, which may save time comparing to processing it by external service. At the same time it may be undesired if query is meant to actually run on server.

How to reuse where clauses in Linq To Sql queries

I have users searching records of type Record. They type a search term in a textbox and then I search records by matching several fields with the search term.
My query looks like:
var results = from record in DataContext.Records
where
record.Field1.ToLower().Contains(term) ||
record.Field2.ToLower().Contains(term) ||
record.Field3.ToLower().Contains(term)
select record;
I have a number of queries that all use the same filter and thus I would like to extract the filtering so it can be reused. Something like:
var filter = new Func<Record, string, bool>(
(record, term) =>
record.Field1.ToLower().Contains(term) ||
record.Field2.ToLower().Contains(term) ||
record.Field3.ToLower().Contains(term)
);
var results = from record in DataContext.Records
where filter(record, term)
select record;
However, it does not work because:
Method 'System.Object DynamicInvoke(System.Object[])' has no supported translation to SQL.
How can I reuse my where condition across queries?

You need to build an expression instead of a function:
Expression<Func<Record, bool>> filter =
record => record.Field1.ToLower().Contains(term); // rest omitted
The lambda expression remains the same, but you need to return it into a variable of type Expression<Func<Record, bool>> -- that will make the C# compiler compile it as an expression instead of a delegate, allowing it to be passed to LINQ to SQL.
However, you won't be able to use an expression variable with a C#-syntax where clause: you'll need to use the Where extension method:
var results = DataContext.Records.Where(filter);
Edited to add: If you want to be able to create filters on different terms, you just need a method to produce an expression from a term:
private static Expression<Func<Record, bool>> Filter(string term)
{
return r => r.Field1.ToLower().Contains(term);
}
var results = DataContext.Records.Where(Filter(term));
If you prefer to keep filter as a lambda as you have at the moment, you can do so, but the generics get a bit nested:
Func<string, Expression<Func<Record, bool>>> filter =
term => (r => r.Field1.ToLower().Contains(term));
var results = DataContext.Records.Where(filter(term));
Regardless, the important thing is that what goes in the Where clause must be an Expression<Func<Record, bool>> -- but as shown above you can make the expression depend on term by building a suitable expression on the fly. Which is exactly what LINQ to SQL would be doing if you spelled out the filter longhand in the Where clause.

Use a CompiledQuery!
var filter = CompiledQuery.Compile(
(DatabaseDataContext dc, Record record, string term) =>
record.Field1.ToLower().Contains(term) ||
record.Field2.ToLower().Contains(term) ||
record.Field3.ToLower().Contains(term)
);
var results = from record in DataContext.Records
where filter(DataContext, record, term)
select record;
For more information, see How to: Store and Reuse Queries.

In addition to the Expression<Func<Record, bool>> issue that others have pointed out, I suggest looking into PredicateBuilder. It's very good for dynamically combining lambda expressions.

I think you need to make it an Expression<Func<Record, bool>>. Otherwise it's trying to translate the actual C# method call to SQL rather than the description of it. This is not a guarantee that this version will work; I'm not sure which string functions are translatable to SQL.