I would like to create a compiled query which uses reusable where predicates. An example to make this clear:
ObjectContext.Employees.Where(EmployeePredicates.CustomerPredicate)
EmployeePredicates is a static class with a property CustomerPredicate that looks like this:
public static Expression<Func<Employee, bool>> CustomerPredicate
{
get
{
return t => t.CustomerId == 1;
}
}
This works as expected.
In most cases however, you would like to pass a parameter to Expression. To achieve this I have to change the property to a static function:
public static Expression<Func<Employee, bool>> CustomerPredicate(int id)
{
return t => t.CustomerId == id;
}
And I can use this like this:
ObjectContext.Employees.Where(EmployeePredicates.CustomerPredicate(id))
This works, but now comes the tricky part. I would like to compile this query... Visual studio doesn't give me any compile errors, but when I run this example the following exception is thrown at runtime:
Internal .NET Framework Data Provider error 1025
Just so we're on the same page here is the full code that gives me the exception:
var _compiledQuery = CompiledQuery.Compile<AdventureWorksEntities, int, IQueryable<Employee>>(
(ctx, id) =>
(ctx.Employee.Where(EmployeePredicates.CustomerPredicate(id))
));
Does anyone have a clue why this exception is being thrown? I've taken this way of working from http://www.albahari.com/nutshell/linqkit.aspx. Any help would be much appreciated.
Thanks Jon,
The problem with the approach you describe, is that chaining predicates together will become very hard. What if I need to combine this predicate with another predicate that filters the employees with a certain name? Then you end up with a lot of selects to pass in the parameters.
(ctx, id, name) =>
(ctx.Employee.Select(emp => new {emp, id})
.Where(EmployeePredicates.CustomerPredicate(id))
.Select(emp => new {emp, name})
.Where(EmployeePredicates.NamePredicate(name))
It gets even worse when you're working on joined tables.
(ctx, id, name) =>
(ctx.Employee
.Join(ctx.Contact, e=> e.ContactId, c => c.Id), (emp, cont) => new Container<Employee, Customer> {Employee = emp, Contact = cont})
.Where(EmployeePredicates.CustomerPredicate(id))
.Where(EmployeePredicates.NamePredicate(name))
.Select(t => new EmployeeDTO {Name = t.cont.Name, Customer = e.emp.Customer})
Because each Where() operates on something of type T and returns something of type T, the WherePredicates in the code above must work on the type Container. This makes it very hard to reuse the Predicates. And reuse was the initial goal of this approach...
The problem is that the Entity Framework is trying to examine the expression tree represented by
(ctx, id) => (ctx.Employee.Where(EmployeePredicates.CustomerPredicate(id))
It can't do that, because it doesn't know what EmployeePredicates.CustomerPredicate does.
As for the best fix... I'm not sure. Basically it's got to know at query compile time what the full query looks like, just with the placeholders for parameters.
I suspect the best solution will involve something like this:
public static Expression<Func<Employee, int, bool>> CustomerPredicate()
{
return (t, id) => t.CustomerId == id;
}
... as that raises the abstraction by one level; it gives you an expression tree which uses id as a ParameterExpression, which is something you'll need in order to build the appropriate expression tree to call CompileQuery. It gets a little hard to think about, unfortunately :(
Related
Can someone explain to me why the EF Engine is failing in the following scenario?
It works fine with the following expression:
var data = context.Programs
.Select(d => new MyDataDto
{
ProgramId = d.ProgramId,
ProgramName = d.ProgramName,
ClientId = d.ClientId,
Protocols = d.Protocols.Where(p => p.UserProtocols.Any(u => u.UserId == userId))
.Count(pr => pr.Programs.Any(pg => pg.ProgramId == d.ProgramId))
})
.ToList();
But if I encapsulate some into an extension method:
public static IQueryable<Protocol> ForUser(this IQueryable<Protocol> protocols, int userId)
{
return protocols.Where(p => p.UserProtocols.Any(u => u.UserId == userId));
}
The resulting query:
var data = context.Programs
.Select(d => new MyDataDto
{
ProgramId = d.ProgramId,
ProgramName = d.ProgramName,
ClientId = d.ClientId,
Protocols = d.Protocols.ForUser(userId)
.Count(pr => pr.Programs.Any(pg => pg.ProgramId == d.ProgramId))
})
.ToList();
Fails with the exception: LINQ to Entities does not recognize the method 'System.Linq.IQueryable1[DAL.Protocol] ForUser(System.Linq.IQueryable1[DAL.Protocol], Int32)' method, and this method cannot be translated into a store expression.
I would expect the EF Engine to build the entire expression tree, chaining the necessary expressions and then generate the SQL. Why doesn't it do that?
This is happening because the call to ForUser() is being made inside of the expression tree that the C# compiler builds when it sees the lambda you pass into Select. Entity Framework tries to figure out how to convert that function into SQL, but it can't invoke the function for a few reasons (e.g. d.Protocols does not exist at the moment).
The simplest approach that works for a case like this is to have your helper return a criteria lambda expression, and then pass that into the .Where() method yourself:
public static Expression<Func<Protocol, true>> ProtocolIsForUser(int userId)
{
return p => p.UserProtocols.Any(u => u.UserId == userId);
}
...
var protocolCriteria = Helpers.ProtocolIsForUser(userId);
var data = context.Programs
.Select(d => new MyDataDto
{
ProgramId = d.ProgramId,
ProgramName = d.ProgramName,
ClientId = d.ClientId,
Protocols = d.Protocols.Count(protocolCriteria)
})
.ToList();
More information
When you invoke a LINQ method outside of an expression tree (like you do with context.Programs.Select(...)), the Queryable.Select() extension method actually gets invoked, and its implementation returns an IQueryable<> that represents the extension method getting called on the original IQueryable<>. Here's the implementation of Select, for instance:
public static IQueryable<TResult> Select<TSource,TResult>(this IQueryable<TSource> source, Expression<Func<TSource, TResult>> selector) {
if (source == null)
throw Error.ArgumentNull("source");
if (selector == null)
throw Error.ArgumentNull("selector");
return source.Provider.CreateQuery<TResult>(
Expression.Call(
null,
GetMethodInfo(Queryable.Select, source, selector),
new Expression[] { source.Expression, Expression.Quote(selector) }
));
}
When the queryable's Provider has to generate actual data from the IQueryable<>, it analyzes the expression tree and tries to figure out how to interpret those method calls. Entity Framework has built-in knowledge of many LINQ-related functions like .Where() and .Select(), so it knows how to translate those method calls into SQL. However, it doesn't know what to do for methods that you write.
So why does this work?
var data = context.Programs.ForUser(userId);
The answer is that your ForUser method is not implemented like the Select method above: you are not adding an expression to the queryable to represent calling ForUser. Instead, you are returning the result of a .Where() call. From the IQueryable<>'s perspective, it's as if Where() was called directly, and the call to ForUser() never happened.
You can prove this by capturing the Expression property on the IQueryable<>:
Console.WriteLine(data.Expression.ToString());
... which will produce something like this:
Programs.Where(u => (u.UserId == value(Helpers<>c__DisplayClass1_0).userId))
There's no call to ForUser() anywhere in that expression.
On the other hand, if you include the ForUser() call inside of an expression tree like this:
var data = context.Programs.Select(d => d.Protocols.ForUser(id));
... then the .ForUser() method never actually gets invoked, so it never returns an IQueryable<> that knows the .Where() method got called. Instead, the expression tree for the queryable shows .ForUser() getting invoked. Outputting its expression tree would look something like this:
Programs.Select(d => d.Protocols.ForUser(value(Repository<>c__DisplayClass1_0).userId))
Entity Framework has no idea what ForUser() is supposed to do. As far as it's concerned, you could have written ForUser() to do something that's impossible to do in SQL. So it tells you that's not a supported method.
As I mentioned in my comment above, I can't tell why the EF Engine is working the way it is. Therefore, I've tried to find a way to re-write the query so I'll be able to make use of my extension methods.
The tables are:
Program -> 1..m -> ProgramProtocol -> m..1 -> Protocol
ProgramProtocol is just a join table and is not mapped in the model by Entity Framework.
The idea is simple: select "from left", select "from right" and then join the resulted sets for proper filtering:
var data = context.Programs.ForUser(userId)
.SelectMany(pm => pm.Protocols,
(pm, pt) => new {pm.ProgramId, pm.ProgramName, pm.ClientId, pt.ProtocolId})
.Join(context.Protocols.ForUser(userId), pm => pm.ProtocolId,
pt => pt.ProtocolId, (pm, pt) => pm)
.GroupBy(pm => new {pm.ProgramId, pm.ProgramName, pm.ClientId})
.Select(d => new MyDataDto
{
ProgramName = d.Key.ProgramName,
ProgramId = d.Key.ProgramId,
ClientId = d.Key.ClientId,
Protocols = d.Count()
})
.ToList();
I have got the following expression that works with mockup data - hereby not using Entity Framework:
public static Expression<Func<Resource, bool>> FilterResourcesByUserCriteria(IEnumerable<FilterValue> filterValuesForUser)
{
Expression<Func<Resource, bool>> filter = (resource) =>
// Get filter values for the current resource in the loop
resource.ResourceFilterValues
// Group filter values for user
.GroupBy(filterValue => filterValue.FilterValue.FilterGroup.Id)
// Each group must fulfill the following logic
.All(filterGroup =>
// For each filter group, only select the user values from the same group
filterValuesForUser
.Where(filterValueForUser => filterValueForUser.FilterGroup.Id == filterGroup.Key)
.Select(filterValueForUser => filterValueForUser.FilterValue1)
// Each group must at least one value in the sublist of filter values of the current user
.Any(filterValueForUser => filterGroup
.Select(resourceFilterValue => resourceFilterValue.FilterValue.FilterValue1)
.Any(x => x == filterValueForUser))
);
}
However, I get this famous exception when I try to insert this expression in the where clause of my repository method (using Entity Framework):
Unable to create a constant value of type. Only primitive types or enumeration types are supported in this context.
I suspect this has something to do with a parameter called filterValuesForUser, which is a collection of a complex (i.e. custom) type.
Is this behavior even possible in Entity Framework where I do a subquery that is not directly related to Entity Framework? What I want to achieve here is to query on a subset of a custom list for each group in the query.
Any solutions for this or other workarounds? I'd like to minimize the amount of database calls, preferrably limit it to just one.
The exact query you are asking for is impossible with LinqToEF (due to limitation of SQL). But fear not. It is possible to salvage your problem with a slight tweaking.
public static Expression<Func<Resource, bool>> FilterResourcesByUserCriteria(FilterValue filterValueForUser)
{
//I assume you can write this part yourself.
}
public IQueryable<Resource> GetResources()
{
IQueryable<Resource> resources = _context.Resources;
IEnumerable<FilterValue> filterValuesForUser = GetFilterValues();
IEnumerable<IQueryable<Resource>> queries = from filter in filterValuesForUser
let filterExp = FilterResourcesByUserCriteria(filter)
select resources.Where(filterExp);
return Enumerable.Aggregate(queries, (l, r) => Queryable.Concat(l, r));
}
Types and Extension methods expanded for clarity.
In addition to Aron's answer, I used the PredicateBuilder utility in the LinqKit assembly to generate 1 expression rather than multiple and separate expresssions. This also avoids doing multiple database calls.
Here is how you can achieve this (pseudo-code):
public IQueryable<Resource> GetResources()
{
MyContext ctx = new MyContext ();
IEnumerable<Expression<Func<Resource, bool>>> queries =
filterValuesForUser.GroupBy(x => x.FilterGroup)
.Select(filter => SecurityFilters.FilterResourcesByUserCriteriaEF(filter.Select(y => y.FilterValue1)))
.Select(filterExpression => { return filterExpression; });
Expression<Func<Resource, bool>> query = PredicateBuilder.True<Resource>();
foreach (Expression<Func<Resource, bool>> filter in queries)
{
query = query.And(filter);
}
return ctx.Resources.AsExpandable().Where(query);
}
public static Expression<Func<Resource, bool>> FilterResourcesByUserCriteriaEF(IEnumerable<string> filterValuesForUser)
{
// From the resource's filter values, check if there are any present in the user's filter values
return (x) => x.ResourceFilterValues.Any(y => filterValuesForUser.Contains(y.FilterValue.FilterValue1));
}
I'm still having issues with getting this working in my repository but that has something do with something blocking AsExpandable() from working properly.
1.) I have an entity query like: return myEntityStore.Query<theType>(x => x.Name == "whatevs");
However, ideally I want to call a function which appends other expressions to the original query, like OrderBy's and more Where's.
Something like this:
public IQueryable<T> ProcessQuery<T>(System.Linq.Expressions.Expression<Func<T, bool>> theOriginalQuery) where T: class
{
return EntityStore.Query<T>(theOriginalQuery).Where(x => x.Active == true).OrderBy(x => x.OrderBy);
}
2.) One of the obvious problems here is that I'm working with <T>, so is there a way to specify the property as a string or something?
<T>(expression).OrderBy(x => "x.ThisColumnExistsIPromise");
3.) The Query function already transforms the expression into a Where(), so would it be sufficient to simply do a Where(expression).Where(expression)?
So, at the end of the day, is the below possible to achieve?
entityStore.Query<T>(originalExpression).Where(additionalExpression).Where(x => "x.Active == true").OrderBy(x => "x.OrderBy");
Actually the question isn't so clear. But I noticed that your problem might be solved if you used LINQ Dynamic Query Library then you could use Property name as string to do whatever you want (OrderBy, Where ....)
For more information look at this
Say I have a very simple entity like this:
public class TestGuy
{
public virtual long Id {get;set;}
public virtual string City {get;set;}
public virtual int InterestingValue {get;set;}
public virtual int OtherValue {get;set;}
}
This contrived example object is mapped with NHibernate (using Fluent) and works fine.
Time to do some reporting. In this example, "testGuys" is an IQueryable with some criteria already applied.
var byCity = testGuys
.GroupBy(c => c.City)
.Select(g => new { City = g.Key, Avg = g.Average(tg => tg.InterestingValue) });
This works just fine. In NHibernate Profiler I can see the correct SQL being generated, and the results are as expected.
Inspired by my success, I want to make it more flexible. I want to make it configurable so that the user can get the average of OtherValue as well as InterestingValue. Shouldn't be too hard, the argument to Average() seems to be a Func (since the values are ints in this case). Easy peasy. Can't I just create a method that returns a Func based on some condition and use that as an argument?
var fieldToAverageBy = GetAverageField(SomeEnum.Other);
private Func<TestGuy,int> GetAverageField(SomeEnum someCondition)
{
switch(someCondition)
{
case SomeEnum.Interesting:
return tg => tg.InterestingValue;
case SomeEnum.Other:
return tg => tg.OtherValue;
}
throw new InvalidOperationException("Not in my example!");
}
And then, elsewhere, I could just do this:
var byCity = testGuys
.GroupBy(c => c.City)
.Select(g => new { City = g.Key, Avg = g.Average(fieldToAverageBy) });
Well, I thought I could do that. However, when I do enumerate this, NHibernate throws a fit:
Object of type 'System.Linq.Expressions.ConstantExpression' cannot be converted to type 'System.Linq.Expressions.LambdaExpression'.
So I am guessing that behind the scenes, some conversion or casting or some such thing is going on that in the first case accepts my lambda, but in the second case makes into something NHibernate can't convert to SQL.
My question is hopefully simple - how can my GetAverageField function return something that will work as a parameter to Average() when NHibernate 3.0 LINQ support (the .Query() method) translates this to SQL?
Any suggestions welcome, thanks!
EDIT
Based on the comments from David B in his answer, I took a closer look at this. My assumption that Func would be the right return type was based on the intellisense I got for the Average() method. It seems to be based on the Enumerable type, not the Queryable one. That's strange.. Need to look a bit closer at stuff.
The GroupBy method has the following return signature:
IQueryable<IGrouping<string,TestGuy>>
That means it should give me an IQueryable, all right. However, I then move on to the next line:
.Select(g => new { City = g.Key, Avg = g.Average(tg => tg.InterestingValue) });
If I check the intellisense for the g variable inside the new { } object definition, it is actually listed as being of type IGrouping - NOT IQueryable>. This is why the Average() method called is the Enumerable one, and why it won't accept the Expression parameter suggested by David B.
So somehow my group value has apparently lost it's status as an IQueryable somewhere.
Slightly interesting note:
I can change the Select to the following:
.Select(g => new { City = g.Key, Avg = g.AsQueryable<TestGuy>().Average(fieldToAverageBy) });
And now it compiles! Black magic! However, that doesn't solve the issue, as NHibernate now doesn't love me anymore and gives the following exception:
Could not parse expression '[-1].AsQueryable()': This overload of the method 'System.Linq.Queryable.AsQueryable' is currently not supported, but you can register your own parser if needed.
What baffles me is that this works when I give the lambda expression to the Average() method, but that I can't find a simple way to represent the same expression as an argument. I am obviously doing something wrong, but can't see what...!?
I am at my wits end. Help me, Jon Skeet, you're my only hope! ;)
You won't be able to call a "local" method within your lambda expression. If this were a simple non-nested clause, it would be relatively simple - you'd just need to change this:
private Func<TestGuy,int> GetAverageField(SomeEnum someCondition)
to this:
private Expression<Func<TestGuy,int>> GetAverageField(SomeEnum someCondition)
and then pass the result of the call into the relevant query method, e.g.
var results = query.Select(GetAverageField(fieldToAverageBy));
In this case, however, you'll need to build the whole expression tree up for the Select clause - the anonymous type creation expression, the extraction of the Key, and the extraction of the average field part. It's not going to be fun, to be honest. In particular, by the time you've built up your expression tree, that's not going to be statically typed in the same way as a normal query expression would be, due to the inability to express the anonymous type in a declaration.
If you're using .NET 4, dynamic typing may help you, although you'd pay the price of not having static typing any more, of course.
One option (horrible though it may be) would be try to use a sort of "template" of the anonymous type projection expression tree (e.g. always using a single property), and then build a copy of that expression tree, inserting the right expression instead. Again, it's not going to be fun.
Marc Gravell may be able to help more on this - it does sound like the kind of thing which should be possible, but I'm at a loss as to how to do it elegantly at the moment.
Eh? the parameter to Queryable.Average is not Func<T, U>. It's Expression<Func<T, U>>
The way to do this is:
private Expression<Func<TestGuy,int>> GetAverageExpr(SomeEnum someCondition)
{
switch(someCondition)
{
case SomeEnum.Interesting:
return tg => tg.InterestingValue;
case SomeEnum.Other:
return tg => tg.OtherValue;
}
throw new InvalidOperationException("Not in my example!");
}
Followed by:
Expression<Func<TestGuy, int>> averageExpr = GetAverageExpr(someCondition);
var byCity = testGuys
.GroupBy(c => c.City)
.Select(g => new { City = g.Key, Avg = g.Average(averageExpr) });
I'm trying to DRY out some lambda expressions for security rights. Is it possible to take a lamda expression and apply it to a single entity for true?
Like lets say I have a Person and a DocumentFolder
Expression<Func<Person, bool>> CanSeePerson()
{
return c => !c.IsPrivate;
}
And one for the folder
Expression<Func<DocumentFolder, bool>> CanSeeFolder()
{
return c => !c.IsPrivate && c.Owner.CanSeePerson(); // <- ???
}
How the heck can I use that CanSeePerson() function on a single type to return true and maintain an expression that can be used in linq queries like such
Entities.DocumentFolder.Where(CanSeeFolder());
I know how to use the where on an iqueryable but I can't see how to apply the expression tree to a single value.
This throws the error: Unable to create a constant value of type 'Person'. Only primitive types ('such as Int32, String, and Guid') are supported in this context.
Expression<Func<DocumentFolder, bool>> CanSeeFolder()
{
return c => !c.IsPrivate &&_entities.Persons.Where(x => x.Id == c.Owner.Id).Any(CanSeePerson());
}
The difference seems to be based on putting the IQueryable Directly in the statement. This also does NOT work
Expression<Func<DocumentFolder, bool>> CanSeeFolder()
{
return c => !c.IsPrivate &&_entities.Persons.Where(CanSeePerson()).Contains(c.User);
}
but this DOES work
Expression<Func<DocumentFolder, bool>> CanSeeFolder()
{
var canSeePersons = _entities.Persons.Where(CanSeePerson());
return c => !c.IsPrivate && canSeePersons.Contains(c.User);
}
p.s. I know I suck # using this stackoverflow formatting thing lol
The reason why this works is because the CanSeePerson() function cannot be converted and used in an expression. When you put the canSeePersons variable in the function instead you are placing in an iQueryable type which can be used in an expression. Using "var" convolutes it a bit.
You defined a Expression<Func<Person, bool>> that solves your problem. The trick is to simply transform your list of objects to the list you want to filter. It is not always possible, but many times it is. You just have to be a bit creative :-)
When using extension methods you can easily come up with a solution that allows you to do this:
var visibleFolders = Entities.DocumentFolder.WhereCanSeeFolder();
Here is the (completely DRY) solution:
public static class SecurityExtensions
{
public static IQueryable<DocumentFolder> WhereCanSeeFolder(
this IQueryable<DocumentFolder> folders)
{
var visibleOwners = folders.Select(f => f.Owner)
.Where(CanSeePerson);
return
from folder in folders.Where(CanSeeFolder)
where visibleOwners.Contains(folder.Owner)
select folder;
}
private static readonly Expression<Func<DocumentFolder, bool>>
CanSeeFolder = folder => !folder.IsPrivate;
private static readonly Expression<Func<Person, bool>>
CanSeePerson = person => !person.IsPrivate;
}
I hope this helps.
The expression tree is only applied to a single value at a time, logically, within the LINQ expression.
If you're saying you want to apply it in-process later to a single value, you can just use:
// This can be cached
Func<DocumentFolder, bool> canSeeFolderDelegate = CanSeeFolder().Compile();
DocumentFolder folder = ...; // Get the value from somewhere
if (canSeeFolderDelegate(folder))
{
// Yes, you can see that folder
}
Ah, you want to evaluate it for a single entity?
var func = CanSeeFolder().Compile(); // <=== store and re-use this;
// this isn't free
bool canSee = func(obj);
Another approach might be:
bool canSee = Enumerable.Repeat(obj, 1).AsQueryable().Any(CanSeeFolder());
but this is probably still going to do the Compile somewhere in the chain, so you may as well use the more direct code (at the top).
Edit re comments:
To evaluate that at the database, you would need a restriction, for example:
bool canSee = db.Folders.Where(f => f.FolderId == id)
.Where(CanSeeFolder()).Any();
which limits us to the single row, then adds your extra filter.