I have got the following expression that works with mockup data - hereby not using Entity Framework:
public static Expression<Func<Resource, bool>> FilterResourcesByUserCriteria(IEnumerable<FilterValue> filterValuesForUser)
{
Expression<Func<Resource, bool>> filter = (resource) =>
// Get filter values for the current resource in the loop
resource.ResourceFilterValues
// Group filter values for user
.GroupBy(filterValue => filterValue.FilterValue.FilterGroup.Id)
// Each group must fulfill the following logic
.All(filterGroup =>
// For each filter group, only select the user values from the same group
filterValuesForUser
.Where(filterValueForUser => filterValueForUser.FilterGroup.Id == filterGroup.Key)
.Select(filterValueForUser => filterValueForUser.FilterValue1)
// Each group must at least one value in the sublist of filter values of the current user
.Any(filterValueForUser => filterGroup
.Select(resourceFilterValue => resourceFilterValue.FilterValue.FilterValue1)
.Any(x => x == filterValueForUser))
);
}
However, I get this famous exception when I try to insert this expression in the where clause of my repository method (using Entity Framework):
Unable to create a constant value of type. Only primitive types or enumeration types are supported in this context.
I suspect this has something to do with a parameter called filterValuesForUser, which is a collection of a complex (i.e. custom) type.
Is this behavior even possible in Entity Framework where I do a subquery that is not directly related to Entity Framework? What I want to achieve here is to query on a subset of a custom list for each group in the query.
Any solutions for this or other workarounds? I'd like to minimize the amount of database calls, preferrably limit it to just one.
The exact query you are asking for is impossible with LinqToEF (due to limitation of SQL). But fear not. It is possible to salvage your problem with a slight tweaking.
public static Expression<Func<Resource, bool>> FilterResourcesByUserCriteria(FilterValue filterValueForUser)
{
//I assume you can write this part yourself.
}
public IQueryable<Resource> GetResources()
{
IQueryable<Resource> resources = _context.Resources;
IEnumerable<FilterValue> filterValuesForUser = GetFilterValues();
IEnumerable<IQueryable<Resource>> queries = from filter in filterValuesForUser
let filterExp = FilterResourcesByUserCriteria(filter)
select resources.Where(filterExp);
return Enumerable.Aggregate(queries, (l, r) => Queryable.Concat(l, r));
}
Types and Extension methods expanded for clarity.
In addition to Aron's answer, I used the PredicateBuilder utility in the LinqKit assembly to generate 1 expression rather than multiple and separate expresssions. This also avoids doing multiple database calls.
Here is how you can achieve this (pseudo-code):
public IQueryable<Resource> GetResources()
{
MyContext ctx = new MyContext ();
IEnumerable<Expression<Func<Resource, bool>>> queries =
filterValuesForUser.GroupBy(x => x.FilterGroup)
.Select(filter => SecurityFilters.FilterResourcesByUserCriteriaEF(filter.Select(y => y.FilterValue1)))
.Select(filterExpression => { return filterExpression; });
Expression<Func<Resource, bool>> query = PredicateBuilder.True<Resource>();
foreach (Expression<Func<Resource, bool>> filter in queries)
{
query = query.And(filter);
}
return ctx.Resources.AsExpandable().Where(query);
}
public static Expression<Func<Resource, bool>> FilterResourcesByUserCriteriaEF(IEnumerable<string> filterValuesForUser)
{
// From the resource's filter values, check if there are any present in the user's filter values
return (x) => x.ResourceFilterValues.Any(y => filterValuesForUser.Contains(y.FilterValue.FilterValue1));
}
I'm still having issues with getting this working in my repository but that has something do with something blocking AsExpandable() from working properly.
Related
I would like to get a list of all organisations within x miles of a location entered by the user. This is converted to a long/lat location.
Organisations are stored in the database with long and lat.
I am using MVC with entity framework and a Unit of Work with the respository pattern to access the dataset.
Here is my EntityRepository:
public IQueryable<T> All
{
get
{
return dbSet;
}
}
public IQueryable<T> AllIncluding(params System.Linq.Expressions.Expression<Func<T, object>>[] includeProperties)
{
IQueryable<T> query = dbSet;
foreach (var includeProperty in includeProperties)
{
query = query.Include(includeProperty);
}
return query;
}
public IEnumerable<T> Where(System.Linq.Expressions.Expression<Func<T, bool>> predicate)
{
return dbSet.Where(predicate).AsEnumerable();
}
To query the data in my datacontext I use a service class for each entity, a UOW is injected into each service. The service call for organisations is:
public class OrgService :IOrgService
{
private IUnitOfWork _UoW;
public OrgService(IUnitOfWork UoW)
{
_UoW = UoW;
}
public Organisation GetOrgByID(int OrgID)
{
return _UoW.OrganisationRepo.Find(OrgID);
}
public IList<Organisation> GetAllOrgs()
{
return _UoW.OrganisationRepo.All.ToList();
}
public IList<Organisation> GetOrgsByLocation(double lat, double lng, int range)
{
/// I need to return a list of all organisations within X miles
}
}
All other queries are working as the they should however I am no trying to write the method GetOrgsByLocation(). This is the query I think I need to get my results:
var userLocation = new GeoCoordinate(lat, lng);
var result = _UoW.OrganisationRepo.Where(x => new GeoCoordinate(x.Latitude, x.Longitude))
.Where(x => x.GetDistanceTo(userLocation) < radius).ToList();
When I try to run this query I get:
"cannot implicitly convert type system.device.location.geoCoordinate to bool"
Can anyone help?
** Update - Working solution **
var userLocation = new GeoCoordinate(lat, lng);
var nearbyOrganizations = _UoW.OrganisationRepo.All.ToList()
.Select(x => new
{ //use an anonymous type or any type you want
Org = x,
Distance = new GeoCoordinate(x.Latitude, x.Longitude).GetDistanceTo(userLocation)
})
.Where(x => x.Distance < 50000)
.ToList();
foreach (var organisation in nearbyOrganizations)
{
Console.WriteLine("{0} ({1:n0} meters)", organisation.Org, organisation.Distance);
}
Thanks to the help below for this solution, though it seems all objects have to be queried in orfer for this to work, it seems that query would be better suited to run on the database, I'll have to look into this more.
The Where method has to return a boolean value.
_UoW.OrganisationRepo.Where(x => new GeoCoordinate(x.Latitude, x.Longitude))
Maybe you meant to use a .Select there? Would the following code work?
_UoW.OrganisationRepo.Select(x => new GeoCoordinate(x.Latitude, x.Longitude))
.Where(x => x.GetDistanceTo(userLocation) < radius).ToList();
Be aware that Entity Framework tries to generate some SQL from the expression provided. I'm afraid that x.GetDistanceTo(userLocation) might not work inside the .Where expression, unless you cast it to an IEnumerable, or call .AsEnumerable() or .ToList() or .ToArray() before calling .Where. Or maybe EF is smart enough to see that GeoCoordinate is not mapped to a table, and then stop generating the SQL right there.
Edit
The code you commented won't work:
_UoW.OrganisationRepo.All.Select(x => new GeoCoordinate(x.Latitude, x.Longitude).GetDistanceTo(userLocation) < radius).ToList()
Notice that you're selecting a list of bools because you're selecting the results instead of filtering by them. You won't know which organizations are within the radius. That's why we use .Select and .Where separately.
Try something like this:
_UoW.OrganisationRepo.All
.Select(x => new GeoCoordinate(x.Latitude, x.Longitude))
.ToEnumerable() //or .ToList() or .ToArray(), make sure it's outside of EF's reach (prevent SQL generation for this)
.Where(x=> x.GetDistanceTo(userLocation) < radius).ToList()
However, if you want to know which organizations are within the radius, you'll need to carry more information along the path.
var nearbyOrganizations = _UoW.OrganisationRepo.All.ToList()
.Select(x => new
{ //use an anonymous type or any type you want
Org = x,
Distance = new GeoCoordinate(x.Latitude, x.Longitude).GetDistanceTo(userLocation)
}) //it's probably outside EF's SQL generation step by now, but you could add one more .Select here that does the math if it fails (return the GeoCoordinate object)
.Where(x=> x.Distance < radius)
.ToList();
Seems like it would be useful for you to know more about .Where and .Select. They're super useful. .Where is essentialy a filter, and .Select transforms objects. And due to anonymous types, you can do whatever you want.
Notice that this will fetch all objects from the database. You probably want to use native features of the database to work with geographical data, and EF probably supports it.
I think your problem is with your first Where as you are trying to create a new GeoCoordinate class and Where is expecting a bool output from the lambda not a GeoCoordinate instance.
I would suggest making the following change:
var result = _UoW.OrganisationRepo.Where(x => new GeoCoordinate(x.Latitude, x.Longitude).GetDistanceTo(userLocation) < radius).ToList();
This will give you a list of Organisations back within the radius you are interested in.
Update
The prior won't work with IQueryable as the provider will not be able to create an SQL query because the .GetDistanceTo function is not known within SQL.
As an alternative could you not use the Spatial Data types in EF 5 onwards?
This blog post by Rick Strahl gives an example of querying geography data stored in SQL by distance from a given location
https://weblog.west-wind.com/posts/2012/jun/21/basic-spatial-data-with-sql-server-and-entity-framework-50
Can someone explain to me why the EF Engine is failing in the following scenario?
It works fine with the following expression:
var data = context.Programs
.Select(d => new MyDataDto
{
ProgramId = d.ProgramId,
ProgramName = d.ProgramName,
ClientId = d.ClientId,
Protocols = d.Protocols.Where(p => p.UserProtocols.Any(u => u.UserId == userId))
.Count(pr => pr.Programs.Any(pg => pg.ProgramId == d.ProgramId))
})
.ToList();
But if I encapsulate some into an extension method:
public static IQueryable<Protocol> ForUser(this IQueryable<Protocol> protocols, int userId)
{
return protocols.Where(p => p.UserProtocols.Any(u => u.UserId == userId));
}
The resulting query:
var data = context.Programs
.Select(d => new MyDataDto
{
ProgramId = d.ProgramId,
ProgramName = d.ProgramName,
ClientId = d.ClientId,
Protocols = d.Protocols.ForUser(userId)
.Count(pr => pr.Programs.Any(pg => pg.ProgramId == d.ProgramId))
})
.ToList();
Fails with the exception: LINQ to Entities does not recognize the method 'System.Linq.IQueryable1[DAL.Protocol] ForUser(System.Linq.IQueryable1[DAL.Protocol], Int32)' method, and this method cannot be translated into a store expression.
I would expect the EF Engine to build the entire expression tree, chaining the necessary expressions and then generate the SQL. Why doesn't it do that?
This is happening because the call to ForUser() is being made inside of the expression tree that the C# compiler builds when it sees the lambda you pass into Select. Entity Framework tries to figure out how to convert that function into SQL, but it can't invoke the function for a few reasons (e.g. d.Protocols does not exist at the moment).
The simplest approach that works for a case like this is to have your helper return a criteria lambda expression, and then pass that into the .Where() method yourself:
public static Expression<Func<Protocol, true>> ProtocolIsForUser(int userId)
{
return p => p.UserProtocols.Any(u => u.UserId == userId);
}
...
var protocolCriteria = Helpers.ProtocolIsForUser(userId);
var data = context.Programs
.Select(d => new MyDataDto
{
ProgramId = d.ProgramId,
ProgramName = d.ProgramName,
ClientId = d.ClientId,
Protocols = d.Protocols.Count(protocolCriteria)
})
.ToList();
More information
When you invoke a LINQ method outside of an expression tree (like you do with context.Programs.Select(...)), the Queryable.Select() extension method actually gets invoked, and its implementation returns an IQueryable<> that represents the extension method getting called on the original IQueryable<>. Here's the implementation of Select, for instance:
public static IQueryable<TResult> Select<TSource,TResult>(this IQueryable<TSource> source, Expression<Func<TSource, TResult>> selector) {
if (source == null)
throw Error.ArgumentNull("source");
if (selector == null)
throw Error.ArgumentNull("selector");
return source.Provider.CreateQuery<TResult>(
Expression.Call(
null,
GetMethodInfo(Queryable.Select, source, selector),
new Expression[] { source.Expression, Expression.Quote(selector) }
));
}
When the queryable's Provider has to generate actual data from the IQueryable<>, it analyzes the expression tree and tries to figure out how to interpret those method calls. Entity Framework has built-in knowledge of many LINQ-related functions like .Where() and .Select(), so it knows how to translate those method calls into SQL. However, it doesn't know what to do for methods that you write.
So why does this work?
var data = context.Programs.ForUser(userId);
The answer is that your ForUser method is not implemented like the Select method above: you are not adding an expression to the queryable to represent calling ForUser. Instead, you are returning the result of a .Where() call. From the IQueryable<>'s perspective, it's as if Where() was called directly, and the call to ForUser() never happened.
You can prove this by capturing the Expression property on the IQueryable<>:
Console.WriteLine(data.Expression.ToString());
... which will produce something like this:
Programs.Where(u => (u.UserId == value(Helpers<>c__DisplayClass1_0).userId))
There's no call to ForUser() anywhere in that expression.
On the other hand, if you include the ForUser() call inside of an expression tree like this:
var data = context.Programs.Select(d => d.Protocols.ForUser(id));
... then the .ForUser() method never actually gets invoked, so it never returns an IQueryable<> that knows the .Where() method got called. Instead, the expression tree for the queryable shows .ForUser() getting invoked. Outputting its expression tree would look something like this:
Programs.Select(d => d.Protocols.ForUser(value(Repository<>c__DisplayClass1_0).userId))
Entity Framework has no idea what ForUser() is supposed to do. As far as it's concerned, you could have written ForUser() to do something that's impossible to do in SQL. So it tells you that's not a supported method.
As I mentioned in my comment above, I can't tell why the EF Engine is working the way it is. Therefore, I've tried to find a way to re-write the query so I'll be able to make use of my extension methods.
The tables are:
Program -> 1..m -> ProgramProtocol -> m..1 -> Protocol
ProgramProtocol is just a join table and is not mapped in the model by Entity Framework.
The idea is simple: select "from left", select "from right" and then join the resulted sets for proper filtering:
var data = context.Programs.ForUser(userId)
.SelectMany(pm => pm.Protocols,
(pm, pt) => new {pm.ProgramId, pm.ProgramName, pm.ClientId, pt.ProtocolId})
.Join(context.Protocols.ForUser(userId), pm => pm.ProtocolId,
pt => pt.ProtocolId, (pm, pt) => pm)
.GroupBy(pm => new {pm.ProgramId, pm.ProgramName, pm.ClientId})
.Select(d => new MyDataDto
{
ProgramName = d.Key.ProgramName,
ProgramId = d.Key.ProgramId,
ClientId = d.Key.ClientId,
Protocols = d.Count()
})
.ToList();
A little background:
I'm using a Linq provider (MsCrm2011 Linq provider, but doesn't really matter) that doesn't support certain operations.
Specifically, it doesn't support Contains() inside an expression.
Meaning- something like this will not work: var users = DataContext.Users.Where(user => userIds.Contains(user.Id)).
The solution I've found for it was to use LinqKit's predicate, so instead of userIds.Contains(...) I'll have user.Id == userIds[0] || user.Id == userIds[1] ... || user.Id == userIds[100].
To this end, I've defined the following function, which takes an arbitrary collection and arbitrary expression, and applies the 'Or' on them:
private IQueryable<TCrmEntity> FilterByCollection<TCrmEntity, T>(IQueryable<TCrmEntity> entities, IEnumerable<T> collection, Expression<Func<TCrmEntity, T, bool>> filterFunction)
{
var predicate = PredicateBuilder.False<TCrmEntity>();
predicate = collection.Aggregate(predicate, (current, collectionElement) => current.Or(entity => filterFunction.Invoke(entity,collectionElement)));
var query = entities.AsExpandable()
.Where(predicate);
return query;
}
this way, I can use any kind of collection and any kind of expression.
For example, see this test run (using an in-memory users collection): var res = FilterByCollection(users.AsQueryable(), rolesList, (account, role) => account.Role == role) to find all users that have one of the given roles.
However, I'm getting the following exception when I'm running the above example- variable 'entity' of type 'User' referenced from scope '', but it is not defined.
Any ideas?
P.S. not sure how relevant this is, but when I'm actually using the Crm data context instead of an in-memory collection, I'm not getting this error. :/
I want to write a class which provides basically a list of entries. This entries can either be read from a database or are generated during application lifetime. This class should provide a way to filter this entries. Is it possible to implement the filters in a way that only one filter for both sources is needed?
For example:
interface IFilter
{
bool Filter(ILogEntry entry);
}
For LINQ with IEnumerable there is no problem to apply this as dynamic filters.
IEnumerable<IFilter> filters;
IEnumerable<ILogEntry> entries;
foreach(var filter in filters)
entries = entries.Where(p => filter.Filter(p));
But for Linq-To-Sql there are two problems I have no solution for:
How can convert the LogEntry provided by the context to ILogEntry?
How can I ensure that the query will be executed on the server and not locally?
It seems a little bit strange to me that the following code will actually execute the filter on the server. But if I use IFilter in this query which takes ILogEntry (which is implemented by LogEntry) as an argument the filter will be applied on client-side:
entities.Logs
.Select(p => new LogEntry() { Message = p.Message })
.Where(p => p.Message == "132");
Is there a way to write filters for ICollection and IQueryable and ensure that the filters for IQueryable will be executed on the SQL-Server?
The only way to apply that server side is by composing it as expression trees, for example:
interface IFilter {
public Expression<Func<ILogEntry, bool>> GetPredicate();
}
...
IQueryable<ILogEntry> entries = ...
foreach(var filter in filters) {
entries = entries.Where(filter.GetPredicate());
}
This then requires implementations such as:
public Expression<Func<ILogEntry, bool>> GetPredicate() {
return x => x.SomeField == 25;
}
Note: if you have an in-memory chink of data you can switch from IEnumerable<T> to IQueryable<T> by using .AsQueryable(), which allows it to use an expression-tree locally.
I'm trying to DRY out some lambda expressions for security rights. Is it possible to take a lamda expression and apply it to a single entity for true?
Like lets say I have a Person and a DocumentFolder
Expression<Func<Person, bool>> CanSeePerson()
{
return c => !c.IsPrivate;
}
And one for the folder
Expression<Func<DocumentFolder, bool>> CanSeeFolder()
{
return c => !c.IsPrivate && c.Owner.CanSeePerson(); // <- ???
}
How the heck can I use that CanSeePerson() function on a single type to return true and maintain an expression that can be used in linq queries like such
Entities.DocumentFolder.Where(CanSeeFolder());
I know how to use the where on an iqueryable but I can't see how to apply the expression tree to a single value.
This throws the error: Unable to create a constant value of type 'Person'. Only primitive types ('such as Int32, String, and Guid') are supported in this context.
Expression<Func<DocumentFolder, bool>> CanSeeFolder()
{
return c => !c.IsPrivate &&_entities.Persons.Where(x => x.Id == c.Owner.Id).Any(CanSeePerson());
}
The difference seems to be based on putting the IQueryable Directly in the statement. This also does NOT work
Expression<Func<DocumentFolder, bool>> CanSeeFolder()
{
return c => !c.IsPrivate &&_entities.Persons.Where(CanSeePerson()).Contains(c.User);
}
but this DOES work
Expression<Func<DocumentFolder, bool>> CanSeeFolder()
{
var canSeePersons = _entities.Persons.Where(CanSeePerson());
return c => !c.IsPrivate && canSeePersons.Contains(c.User);
}
p.s. I know I suck # using this stackoverflow formatting thing lol
The reason why this works is because the CanSeePerson() function cannot be converted and used in an expression. When you put the canSeePersons variable in the function instead you are placing in an iQueryable type which can be used in an expression. Using "var" convolutes it a bit.
You defined a Expression<Func<Person, bool>> that solves your problem. The trick is to simply transform your list of objects to the list you want to filter. It is not always possible, but many times it is. You just have to be a bit creative :-)
When using extension methods you can easily come up with a solution that allows you to do this:
var visibleFolders = Entities.DocumentFolder.WhereCanSeeFolder();
Here is the (completely DRY) solution:
public static class SecurityExtensions
{
public static IQueryable<DocumentFolder> WhereCanSeeFolder(
this IQueryable<DocumentFolder> folders)
{
var visibleOwners = folders.Select(f => f.Owner)
.Where(CanSeePerson);
return
from folder in folders.Where(CanSeeFolder)
where visibleOwners.Contains(folder.Owner)
select folder;
}
private static readonly Expression<Func<DocumentFolder, bool>>
CanSeeFolder = folder => !folder.IsPrivate;
private static readonly Expression<Func<Person, bool>>
CanSeePerson = person => !person.IsPrivate;
}
I hope this helps.
The expression tree is only applied to a single value at a time, logically, within the LINQ expression.
If you're saying you want to apply it in-process later to a single value, you can just use:
// This can be cached
Func<DocumentFolder, bool> canSeeFolderDelegate = CanSeeFolder().Compile();
DocumentFolder folder = ...; // Get the value from somewhere
if (canSeeFolderDelegate(folder))
{
// Yes, you can see that folder
}
Ah, you want to evaluate it for a single entity?
var func = CanSeeFolder().Compile(); // <=== store and re-use this;
// this isn't free
bool canSee = func(obj);
Another approach might be:
bool canSee = Enumerable.Repeat(obj, 1).AsQueryable().Any(CanSeeFolder());
but this is probably still going to do the Compile somewhere in the chain, so you may as well use the more direct code (at the top).
Edit re comments:
To evaluate that at the database, you would need a restriction, for example:
bool canSee = db.Folders.Where(f => f.FolderId == id)
.Where(CanSeeFolder()).Any();
which limits us to the single row, then adds your extra filter.