Develop Reference

Custom Extension Method in Linq-EF Equivalent to SQL "IN" Clause

I am trying to create a custom Linq to Entities extension method which takes a comma-delimited string, converts it to an array, then using IEnumerable<string>.Contains to generate the equivalent of a SQL IN clause.
Easy enough when you always know the table/entity and its column/property that you want to apply this filter to. The challenge is that I want to be able to use this extension method on any entity or property.
This is how far I've come:
public static IQueryable<TSource> CustomInClause<TSource>(this IQueryable<TSource> myQuery, Expression<Func<TSource, string>> colExpression, string filterCriteria)
{
string[] myArray = filterCriteria.Split(",", StringSplitOptions.RemoveEmptyEntries);
//Various other operations here..............
if (myArray.Length > 0)
{
myQuery = myQuery.Where(b => myArray.Contains(colExpression));
}
return myQuery;
}
As you can see, I am trying to use colExpression as a dynamic expression which will be the equivalent of x => x.SomeColumn where SomeColumn could be any string/varchar column.
I would then implement this extension like this:
var q = context.SomeTable.CustomInClause(f => f.SomeColumn, someString);
var q2 = context.OtherTable.CustomInCluse(f => f.OtherColumn, otherString);
Right now I get this error:
'string[]' does not contain a definition for 'Contains' and the best
extension method overload
'ParallelEnumerable.Contains>>(ParallelQuery>>,
Expression>)' requires a receiver of type
'ParallelQuery>>'
I'm not quite sure how to use a parallel query in this instance, or if there is another solution. Any ideas?

You have to build Contains call as part of expression in where clause
var myArray = filterCriteria.Split(",".ToCharArray(), StringSplitOptions.RemoveEmptyEntries)
.ToList();
var containsExp = Expression.Call(Expression.Constant(myArray),
"Contains", null, colExpression.Body);
if (myArray.Count > 0)
{
myQuery = myQuery.Where(Expression.Lambda<Func<TSource, bool>>
(containsExp, colExpression.Parameters));
}
return myQuery;
List is better than Array in this case, because list has Contains function and Array has only extension

Linq to entities extension method inner query (EF6)

Can someone explain to me why the EF Engine is failing in the following scenario?
It works fine with the following expression:
var data = context.Programs
.Select(d => new MyDataDto
{
ProgramId = d.ProgramId,
ProgramName = d.ProgramName,
ClientId = d.ClientId,
Protocols = d.Protocols.Where(p => p.UserProtocols.Any(u => u.UserId == userId))
.Count(pr => pr.Programs.Any(pg => pg.ProgramId == d.ProgramId))
})
.ToList();
But if I encapsulate some into an extension method:
public static IQueryable<Protocol> ForUser(this IQueryable<Protocol> protocols, int userId)
{
return protocols.Where(p => p.UserProtocols.Any(u => u.UserId == userId));
}
The resulting query:
var data = context.Programs
.Select(d => new MyDataDto
{
ProgramId = d.ProgramId,
ProgramName = d.ProgramName,
ClientId = d.ClientId,
Protocols = d.Protocols.ForUser(userId)
.Count(pr => pr.Programs.Any(pg => pg.ProgramId == d.ProgramId))
})
.ToList();
Fails with the exception: LINQ to Entities does not recognize the method 'System.Linq.IQueryable1[DAL.Protocol] ForUser(System.Linq.IQueryable1[DAL.Protocol], Int32)' method, and this method cannot be translated into a store expression.
I would expect the EF Engine to build the entire expression tree, chaining the necessary expressions and then generate the SQL. Why doesn't it do that?

This is happening because the call to ForUser() is being made inside of the expression tree that the C# compiler builds when it sees the lambda you pass into Select. Entity Framework tries to figure out how to convert that function into SQL, but it can't invoke the function for a few reasons (e.g. d.Protocols does not exist at the moment).
The simplest approach that works for a case like this is to have your helper return a criteria lambda expression, and then pass that into the .Where() method yourself:
public static Expression<Func<Protocol, true>> ProtocolIsForUser(int userId)
{
return p => p.UserProtocols.Any(u => u.UserId == userId);
}
...
var protocolCriteria = Helpers.ProtocolIsForUser(userId);
var data = context.Programs
.Select(d => new MyDataDto
{
ProgramId = d.ProgramId,
ProgramName = d.ProgramName,
ClientId = d.ClientId,
Protocols = d.Protocols.Count(protocolCriteria)
})
.ToList();
More information
When you invoke a LINQ method outside of an expression tree (like you do with context.Programs.Select(...)), the Queryable.Select() extension method actually gets invoked, and its implementation returns an IQueryable<> that represents the extension method getting called on the original IQueryable<>. Here's the implementation of Select, for instance:
public static IQueryable<TResult> Select<TSource,TResult>(this IQueryable<TSource> source, Expression<Func<TSource, TResult>> selector) {
if (source == null)
throw Error.ArgumentNull("source");
if (selector == null)
throw Error.ArgumentNull("selector");
return source.Provider.CreateQuery<TResult>(
Expression.Call(
null,
GetMethodInfo(Queryable.Select, source, selector),
new Expression[] { source.Expression, Expression.Quote(selector) }
));
}
When the queryable's Provider has to generate actual data from the IQueryable<>, it analyzes the expression tree and tries to figure out how to interpret those method calls. Entity Framework has built-in knowledge of many LINQ-related functions like .Where() and .Select(), so it knows how to translate those method calls into SQL. However, it doesn't know what to do for methods that you write.
So why does this work?
var data = context.Programs.ForUser(userId);
The answer is that your ForUser method is not implemented like the Select method above: you are not adding an expression to the queryable to represent calling ForUser. Instead, you are returning the result of a .Where() call. From the IQueryable<>'s perspective, it's as if Where() was called directly, and the call to ForUser() never happened.
You can prove this by capturing the Expression property on the IQueryable<>:
Console.WriteLine(data.Expression.ToString());
... which will produce something like this:
Programs.Where(u => (u.UserId == value(Helpers<>c__DisplayClass1_0).userId))
There's no call to ForUser() anywhere in that expression.
On the other hand, if you include the ForUser() call inside of an expression tree like this:
var data = context.Programs.Select(d => d.Protocols.ForUser(id));
... then the .ForUser() method never actually gets invoked, so it never returns an IQueryable<> that knows the .Where() method got called. Instead, the expression tree for the queryable shows .ForUser() getting invoked. Outputting its expression tree would look something like this:
Programs.Select(d => d.Protocols.ForUser(value(Repository<>c__DisplayClass1_0).userId))
Entity Framework has no idea what ForUser() is supposed to do. As far as it's concerned, you could have written ForUser() to do something that's impossible to do in SQL. So it tells you that's not a supported method.

As I mentioned in my comment above, I can't tell why the EF Engine is working the way it is. Therefore, I've tried to find a way to re-write the query so I'll be able to make use of my extension methods.
The tables are:
Program -> 1..m -> ProgramProtocol -> m..1 -> Protocol
ProgramProtocol is just a join table and is not mapped in the model by Entity Framework.
The idea is simple: select "from left", select "from right" and then join the resulted sets for proper filtering:
var data = context.Programs.ForUser(userId)
.SelectMany(pm => pm.Protocols,
(pm, pt) => new {pm.ProgramId, pm.ProgramName, pm.ClientId, pt.ProtocolId})
.Join(context.Protocols.ForUser(userId), pm => pm.ProtocolId,
pt => pt.ProtocolId, (pm, pt) => pm)
.GroupBy(pm => new {pm.ProgramId, pm.ProgramName, pm.ClientId})
.Select(d => new MyDataDto
{
ProgramName = d.Key.ProgramName,
ProgramId = d.Key.ProgramId,
ClientId = d.Key.ClientId,
Protocols = d.Count()
})
.ToList();

How to call a method with no parameters which returns Expression<Func<Model, String>>?

I have a method like this (very simplified version):
public static Expression<Func<MyModel, String>> GetSomeStatus()
{
return myModel => myModel.IsTrue ? "That's true" : "That's false";
}
So, how can I call it in statement like this:
var efRequest = db.Table1.Where(...)
.Select(x => new MyAnotherModel
{
Status = ""; // call GetSomeStatus() here; x is of MyModel type
})
Note: My initial issue was to call some helper method which returns String inside Select method, but of course I've got exception like Linq to entities doesn't recognize this method..., so I've tried to rewrite it (see example above), but now I just don't understand how to call it (I'm relatively new in EF). I know about simple AsEnumerable call before Select which solves my initial issue, but I want to keep this query as IQueryable for later purposes.

It's not possible out of the box. But you can use for instance LINQKit AsExpandable and Invoke extension methods like this:
First you need to store the expression in a variable, otherwise you'll get the famous Linq to entities doesn't recognize this method... exception:
var getSomeStatus = GetSomeStatus();
and then use it inside the query (after calling AsExpandable, so the resulting query expression tree is correctly postprocessed):
var efRequest = db.Table1.AsExpandable()
.Where(...)
.Select(x => new MyAnotherModel
{
Status = getSomeStatus.Invoke(x)
});

Entity-Framework using expressions to build global and reusable filter/query-rules

Given the following linq-query:
var query1 = dbContext.MainTable.Where(m => m.MainId == _mainId).SelectMany(sub => sub.SubTable1)
.Select(sub1 => new
{
sub1.CategoryName,
VisibleDivisions = sub1.SubTable2
.Where(sub2 => sub2.Status == "Visible")
.Select(sub2 => new
{
/* select only what needed */
})
});
Starting from my main-table, I want to get all sub1's selected together with all the sub2's related to the sub1.
The query works as expected, generating a single query which will hit the database.
My question is regarding the inner Where-part, as of this filter will be used at several other parts in the application. So I would like to have this "visible-rule" defined at a single place (DRY-principle).
As of the Where is expecting an Func<SubTable2, bool> I have written the following property
public static Expression<Func<SubTable2, bool>> VisibleOnlyExpression => sub2 => sub2.Status == "Visible";
and changed my query to
var query1 = dbContext.MainTable.Where(m => m.MainId == _mainId).SelectMany(sub => sub.SubTable1)
.Select(sub1 => new
{
sub1.CategoryName,
VisibleDivisions = sub1.SubTable2
.Where(VisibleOnlyExpression.Compile())
.Select(sub2 => new
{
/* select only what needed */
})
});
This throws me an exception, stating Internal .NET Framework Data Provider error 1025..
I already tried changing to .Where(VisibleOnlyExpression.Compile()) with the same error.
I know that this is because EntityFramework is trying to transalte this into SQL which it can not.
My question is: How can I have my "filter-rules" defined at a single place (DRY) in code but have the still usable in Where-, Select-, ... -clauses which can be used on IQueryable as well as on ICollection for inner (sub-)queries?
I would love to be able to write something like:
var query = dbContext.MainTable
.Where(IsAwesome)
.SelectMany(s => s.SubTable1.Where(IsAlsoAwesome))
.Select(sub => new
{
Sub1sub2s = sub.SubTable2.Where(IsVisible),
Sub2Mains = sub.MainTable.Where(IsAwesome)
});
whereas the IsAwesome-rule is called first on IQueryable<MainTable> to get only awesome main-entries and later on ICollection<MainTable> in the sub-select to fetch only awesome main-entries related to a specific SubTable2-entry. But the rule - defining a MainTable-entry as awesome - will be the same, no matter where I call/filter for it.
I guess the solution will need the use of expression-trees and how they can be manipulated, so they will be translatable to plain SQL but I don't get the right idea or point to start with.

You can get something close to what are you asking for using the LinqKit AsExpandable and Invoke extension methods like this:
var isAvesome = IsAwesome;
var isAlsoAwesome = IsAlsoAwesome;
var isVisible = IsVisible;
var query = dbContext.MainTable
.AsExpandable()
.Where(mt => isAwesome.Invoke(mt))
.SelectMany(s => s.SubTable1.Where(st1 => isAlsoAwesome.Invoke(st1)))
.Select(sub => new
{
Sub1sub2s = sub.SubTable2.Where(st2 => isVisible.Invoke(st2)),
Sub2Mains = sub.MainTable.Where(mt => isAwesome.Invoke(mt))
});
I'm saying close because first you need to pull all the expressions needed into variables, otherwise you'll get the famous EF "Method not supported" exception. And second, the invocation is not so concise as in your wish. But at least it allows you to reuse the logic.

AFAIK what you are trying to do should be perfectly possible:
// You forgot to access ".Status" in your code.
// Also you don't have to use "=>" to initialize "IsVisible". Use the regular "=".
public static Expression<Func<SubTable2, bool>> IsVisible = sub2 =>
sub2.Status == "Visible";
...
VisibleDivisions = sub1
.SubTable2
// Don't call "Compile()" on your predicate expression. EF will do that.
.Where(IsVisibleOnly)
.Select(sub2 => new
{
/* select only what needed */
})

I would prepare extension method like below:
public static IQueryable<SubTable2> VisibleOnly(this IQueryable<SubTable2> source)
{
return source.Where(s => s.Status == "Visible");
}
An then you can use it in that way:
var query = dbContext.Table.VisibleOnly().Select(...)

Odata Error translating Linq expression at call Contains

Is it possible using ASP.NET Web APi OData make similar:
List<string> customersTitles = Odata.OrdersService.Select(o=>o.CustomerTitle).Distinct().ToList();
List<Customer> customers = Odata.CustomerService.Where(m => customersTitles .Contains(m.CustomerTitle))
Get error:
Error translating Linq expression to URI: The expression
value(System.Collections.Generic.List`1[System.String]).Contains([10007].CustomerTitle)
is not supported.}
API:
public class CustomerController : EntitySetController<Customer, int>
{
[Queryable]
public override IQueryable<Customer> Get()
{
Expression filter = this.QueryOptions.Filter.ToExpression<Customer>();
return db.Query<Customer>(filter as Expression<Func<Customer, bool>>);
}
}

The Contains construct is not supported via URIs since the list of strings existing at the client side is not a Server side resource.
Linq2Sql Provider has an innate translation for Contains, which gets translated to IN clause for SQL.
With OData, such a translation is not supported. What you need to build is an expanded query list for your where clause using all the Title values:
Because this does not work:
List<Customer> customers = Odata.CustomerService.Where(m => customersTitles .Contains(m.CustomerTitle))
the expanded query option helps us in building a query like:
List<Customer> customers = Odata.CustomerService.Where(m => m.CustomerTitle == customerTitles[0] || m.CustomerTitle == customerTitles[1]); // and so on
Here is the code for the filter building:
var titleFilterList = customerTitles.Select(title => String.Format("(CustomerTitle eq {0})", title));
var titleFilter = String.Join(" or ", titleFilterList);
var customers = Odata.CustomerService.AddQueryOption("$filter", titleFilter).Execute().ToList(); // you may have to cast this.
There is another option to do the same in a strongly typed manner using a nice extension method and building a dynamic Expression based predicate. Follow the steps from here:
http://blogs.msdn.com/b/phaniraj/archive/2008/07/17/set-based-operations-in-ado-net-data-services.aspx

The following extension method can be used LINQ to perform OData queries that test if the value of a property is contained in a set similar to how Contains works with LINQ to EF. It is based on the link provided in Raja Nadar's answer, specifically the comment at the end by Nick.
public static IQueryable<T> WherePropertyIsIn<T, TSet>(
this IQueryable<T> query,
IEnumerable<TSet> set,
Expression<Func<T, TSet>> propertyExpression
) {
var filterPredicate = set.Select(value => Expression.Equal(propertyExpression.Body, Expression.Constant(value)))
.Aggregate<Expression, Expression>(Expression.Constant(false), Expression.Or);
var filterLambdaExpression = Expression.Lambda<Func<T, bool>>(filterPredicate, propertyExpression.Parameters.Single());
return query.Where(filterLambdaExpression);
}
Usage:
var allowed_states = getAllowedStates();
var maxPopulation = getMaxPopulation();
// Instead of...
var cities = context.Cities.Where(c => allowed_states.Contains(c.State) && c.Population <= maxPopulation);
// Use...
var cities = context.Cities.Where(c => c.Population <= maxPopulation).WherePropertyIsIn(allowed_states, c => c.Cities);
Note that you must have a separate Where call (as shown above) if you wish to filter by more than WherePropertyIsIn allows. It would be nice if this could be combined into a single Where but I couldn't figure out how.

I have run into issues with Timothy's solution when using it against Configuration Manager's OData service. CM utilises WML internally and therefore is even more limited in what can be parsed.
I have changed the code to generate a bit 'cleaner' query, which avoid using Expression.Constant(false) as aggregate base -> it produces 0 that is not accepted by WML
public static IQueryable<T> WherePropertyIsIn<T, TSet>(this IQueryable<T> query,
IEnumerable<TSet> valuesList, Expression<Func<T, TSet>> propertySelector)
{
if (valuesList == null) throw new ArgumentNullException(nameof(valuesList));
//if there are no values, no entities can fullfil the condition -> return empty
if (!valuesList.Any())
return Enumerable.Empty<T>().AsQueryable();
//create a check for each value
var filters = valuesList.Select(value => Expression.Equal(propertySelector.Body, Expression.Constant(value)));
//build an expression aggregating checks with OR, use first check as starter (could be '0', but doesn't get mapped to WML)
var firstCheck = filters.First();
//we could duplicate first check, but why not just skip it
var filterPredicate = filters.Skip(1).Aggregate(firstCheck, (Func<Expression, Expression, Expression>)Expression.Or);
var filterLambdaExpression = Expression.Lambda<Func<T, bool>>(filterPredicate, propertySelector.Parameters.Single());
return query.Where(filterLambdaExpression);
}
}
A side effect is that when there are no values passed as argument, the query will immediately return an empty result, which may save time comparing to processing it by external service. At the same time it may be undesired if query is meant to actually run on server.