Develop Reference

Using Func as parameter for LinqToEntities

I have a Linq-Query to get my EF-Data. My query joins 4 tables and selects the result to a denormalized type. I will need this query very often but with different predicates. The List-ExtensionMethods (e.g. .Where() are working with a Func<T,bool> as a parameter and I wanted to do it the same - but I don´t find a way to access my predicate in my Method.
public DenormalizedType GetData(Func<Thing, bool> predicate)
{
using (var dbContext = new MyDbContext())
{
var myData = (from some in dbContext.Thing
join other in dbContext.OtherThing
on some.OtherId equals other.Id
// => HowToWhere ???
select new DenormalizedType()
{
SomeEntry = some.Entry
SomeOtherId = some.OtherId
OtherValue = other.Value
}).ToList();
}
}
I have 3 questions regarding this issue.
First (obviously): how to invoke my predicate to use a dynamic where-clause?
Second: If my initial idea doesn´t work (because teovankots answer indicates, that my approach isn´t valid for LinqToEntities) is it somehow possible, to make a method of the join only?
Third: What is the best performing approach to return my results to another software-component?

EF query provider needs to translate the LINQ query expression tree to SQL, which is not possible when you pass a Func<...> (and more generally, invocation expression like delegate, unknown method etc.).
Shortly, what you ask is not possible with Func<...> type parameters.
The first thing to consider when working with Queryable methods is to use Expression<Func<...>> whenever you would use Func<..> in Enumerable methods. So change your method argument like this:
public DenormalizedType GetData(Expression<Func<Thing, bool>> predicate)
Unfortunately using the supplied expression inside LINQ query syntax is not supported out of the box. To do that, you need some expression tree processing library.
The problem and the possible solution is explained in the LINQKit package page. The solution provided by the package is through AsExpandable and Invoke custom extension methods.
Install the nuget package, add
using LinqKit;
to the source code file, and now you can use something like this to achieve the goal:
public DenormalizedType GetData(Expression<Func<Thing, bool>> predicate)
{
using (var dbContext = new MyDbContext())
{
var myData = (from some in dbContext.Thing.AsExpandable() // <=
join other in dbContext.OtherThing
on some.OtherId equals other.Id
where predicate.Invoke(some) // <=
select new DenormalizedType()
{
SomeEntry = some.Entry
SomeOtherId = some.OtherId
OtherValue = other.Value
}).ToList();
}
}

At least I´ve found a way to solve my issue - but I´d highly appreciate hints, and or ways to make it better, because I can´t imagine that this is the holy grail or even close to...
However, the first step is my join, which I return as IQueryable. Important: No using here, because otherwise the dbContext will be disposed, which is not so nice, while working with the IQueryable:
private static MyDbContext _dbContext;
private static IQueryable<DenormalizedType> SelectType()
{
_dbContext = new MyDbContext();
var myData = (from some in dbContext.Thing
join other in dbContext.OtherThing
on some.OtherId equals other.Id
select new DenormalizedType()
{
SomeEntry = some.Entry
SomeOtherId = some.OtherId
OtherValue = other.Value
};
return myData;
}
I´ve learned a lot today. For example: IEnumerable and IQueryable have both an extension method .Where(). But only IEnumerable.Where() has a Func<T,bool> as a parameter. IQueryable takes a Expression<Func<T,bool>> for its Where(). If I want my query to be executed, with all of my conditions I need to work with the IQueryable-type, as long as all my wheres aren´t executed. So I needed to take a closer look to the Expression-Type. I didn´t understand what all this actually does, but it works ;)
The first thing I had to do, was writing my Where-Methods.. That was pretty easy after I´ve read this one: Entity Framework Filter "Expression<Func<T, bool>>". The Method looks like this:
public static IQueryable<DenormalizedType> SelectWhereCriteria(IQueryable<DenormalizedType> data, Expression<Func<DenormalizedType, bool>> predicate)
{
return data.Where(predicate);
}
The Expression itself was a little more complicated, because I have a Selection-Enum which should select specified filters. The Expression looks like:
Expression<Func<DenormalizedType, bool>> FilterBySelection(Selection selection)
{
switch(selection)
{
case Selection.Active:
return x => x.IsActive == true;
case Selection.InActive:
return x => x.IsActive == false;
case Selection.SomeOtherSelection:
return x => x.SomeOther == "Criteria"
default:
return x => true;
}
}
This Expression works fine on my IQueryable:
var selectedQuery = DataHandler.SelectWhereCriteria(query, FilterBySelection(selection));
The only thing I needed now, was ordering. I found some very cool stuff from MarcGravell (what a genius btw) Dynamic LINQ OrderBy on IEnumerable<T> where he posted some code as an answer, which you can use, to OrderBy PropertyName. His first piece of code takes an IQueryable, orders it by PropertyName (he provides Extensions for Descending OrderyBy as well) and returns an IOrderedQueryable. The ToList()-Operation is the very last operation I execute.
And one more thing: Don´t forget to Dispose the DbContext:
public static void Dispose()
{
_dbContext.Dispose();
}

You can call it easily. Install this package, add:
public DenormalizedType GetData(Expression<Func<Thing, bool>> predicate)
{
using (var dbContext = new MyDbContext())
{
var myData = (from some in dbContext.Thing
join other in dbContext.OtherThing
on some.OtherId equals other.Id
where predicate.Invoke(some) //check this
select new DenormalizedType()
{
SomeEntry = some.Entry
SomeOtherId = some.OtherId
OtherValue = other.Value
}).ToList();
}
}
You just should know that Func<T1,T2> is a method. With this signature:
T2 predicate(T1 parameter) { /*...*/ }
Your second question depends on how you connect your conponent. But as long as you get DenormalizedType not DbEntity your example looks ok.

Linq to entities extension method inner query (EF6)

Can someone explain to me why the EF Engine is failing in the following scenario?
It works fine with the following expression:
var data = context.Programs
.Select(d => new MyDataDto
{
ProgramId = d.ProgramId,
ProgramName = d.ProgramName,
ClientId = d.ClientId,
Protocols = d.Protocols.Where(p => p.UserProtocols.Any(u => u.UserId == userId))
.Count(pr => pr.Programs.Any(pg => pg.ProgramId == d.ProgramId))
})
.ToList();
But if I encapsulate some into an extension method:
public static IQueryable<Protocol> ForUser(this IQueryable<Protocol> protocols, int userId)
{
return protocols.Where(p => p.UserProtocols.Any(u => u.UserId == userId));
}
The resulting query:
var data = context.Programs
.Select(d => new MyDataDto
{
ProgramId = d.ProgramId,
ProgramName = d.ProgramName,
ClientId = d.ClientId,
Protocols = d.Protocols.ForUser(userId)
.Count(pr => pr.Programs.Any(pg => pg.ProgramId == d.ProgramId))
})
.ToList();
Fails with the exception: LINQ to Entities does not recognize the method 'System.Linq.IQueryable1[DAL.Protocol] ForUser(System.Linq.IQueryable1[DAL.Protocol], Int32)' method, and this method cannot be translated into a store expression.
I would expect the EF Engine to build the entire expression tree, chaining the necessary expressions and then generate the SQL. Why doesn't it do that?

This is happening because the call to ForUser() is being made inside of the expression tree that the C# compiler builds when it sees the lambda you pass into Select. Entity Framework tries to figure out how to convert that function into SQL, but it can't invoke the function for a few reasons (e.g. d.Protocols does not exist at the moment).
The simplest approach that works for a case like this is to have your helper return a criteria lambda expression, and then pass that into the .Where() method yourself:
public static Expression<Func<Protocol, true>> ProtocolIsForUser(int userId)
{
return p => p.UserProtocols.Any(u => u.UserId == userId);
}
...
var protocolCriteria = Helpers.ProtocolIsForUser(userId);
var data = context.Programs
.Select(d => new MyDataDto
{
ProgramId = d.ProgramId,
ProgramName = d.ProgramName,
ClientId = d.ClientId,
Protocols = d.Protocols.Count(protocolCriteria)
})
.ToList();
More information
When you invoke a LINQ method outside of an expression tree (like you do with context.Programs.Select(...)), the Queryable.Select() extension method actually gets invoked, and its implementation returns an IQueryable<> that represents the extension method getting called on the original IQueryable<>. Here's the implementation of Select, for instance:
public static IQueryable<TResult> Select<TSource,TResult>(this IQueryable<TSource> source, Expression<Func<TSource, TResult>> selector) {
if (source == null)
throw Error.ArgumentNull("source");
if (selector == null)
throw Error.ArgumentNull("selector");
return source.Provider.CreateQuery<TResult>(
Expression.Call(
null,
GetMethodInfo(Queryable.Select, source, selector),
new Expression[] { source.Expression, Expression.Quote(selector) }
));
}
When the queryable's Provider has to generate actual data from the IQueryable<>, it analyzes the expression tree and tries to figure out how to interpret those method calls. Entity Framework has built-in knowledge of many LINQ-related functions like .Where() and .Select(), so it knows how to translate those method calls into SQL. However, it doesn't know what to do for methods that you write.
So why does this work?
var data = context.Programs.ForUser(userId);
The answer is that your ForUser method is not implemented like the Select method above: you are not adding an expression to the queryable to represent calling ForUser. Instead, you are returning the result of a .Where() call. From the IQueryable<>'s perspective, it's as if Where() was called directly, and the call to ForUser() never happened.
You can prove this by capturing the Expression property on the IQueryable<>:
Console.WriteLine(data.Expression.ToString());
... which will produce something like this:
Programs.Where(u => (u.UserId == value(Helpers<>c__DisplayClass1_0).userId))
There's no call to ForUser() anywhere in that expression.
On the other hand, if you include the ForUser() call inside of an expression tree like this:
var data = context.Programs.Select(d => d.Protocols.ForUser(id));
... then the .ForUser() method never actually gets invoked, so it never returns an IQueryable<> that knows the .Where() method got called. Instead, the expression tree for the queryable shows .ForUser() getting invoked. Outputting its expression tree would look something like this:
Programs.Select(d => d.Protocols.ForUser(value(Repository<>c__DisplayClass1_0).userId))
Entity Framework has no idea what ForUser() is supposed to do. As far as it's concerned, you could have written ForUser() to do something that's impossible to do in SQL. So it tells you that's not a supported method.

As I mentioned in my comment above, I can't tell why the EF Engine is working the way it is. Therefore, I've tried to find a way to re-write the query so I'll be able to make use of my extension methods.
The tables are:
Program -> 1..m -> ProgramProtocol -> m..1 -> Protocol
ProgramProtocol is just a join table and is not mapped in the model by Entity Framework.
The idea is simple: select "from left", select "from right" and then join the resulted sets for proper filtering:
var data = context.Programs.ForUser(userId)
.SelectMany(pm => pm.Protocols,
(pm, pt) => new {pm.ProgramId, pm.ProgramName, pm.ClientId, pt.ProtocolId})
.Join(context.Protocols.ForUser(userId), pm => pm.ProtocolId,
pt => pt.ProtocolId, (pm, pt) => pm)
.GroupBy(pm => new {pm.ProgramId, pm.ProgramName, pm.ClientId})
.Select(d => new MyDataDto
{
ProgramName = d.Key.ProgramName,
ProgramId = d.Key.ProgramId,
ClientId = d.Key.ClientId,
Protocols = d.Count()
})
.ToList();

Why doesn't IOrderedEnumerable retain order after where filtering

I've created a simplification of the issue. I have an ordered IEnumerable, I'm wondering why applying a where filter could unorder the objects
This does not compile while it should have the potential to
IOrderedEnumerable<int> tmp = new List<int>().OrderBy(x => x);
//Error Cannot Implicitly conver IEnumerable<int> To IOrderedEnumerable<int>
tmp = tmp.Where(x => x > 1);
I understand that there would be no gaurenteed execution order if coming from an IQueryable such as using linq to some DB Provider.
However, when dealing with Linq To Object what senario could occur that would unorder your objects, or why wasn't this implemented?
EDIT
I understand how to properly order this that is not the question. My Question is more of a design question. A Where filter on linq to objects should enumerate the give enumerable and apply filtering. So why is that we can only return an IEnumerable instead of an IOrderedEnumerable?
EDIT
To Clarify the senario in when this would be userful. I'm building Queries based on conditions in my code, I want to reuse as much code as possible. I have a function that is returning an OrderedEnumerable, however after applying the additional where I would have to reorder this even though it would be in its original ordered state

Rene's answer is correct, but could use some additional explanation.
IOrderedEnumerable<T> does not mean "this is a sequence that is ordered". It means "this is a sequence that has had an ordering operation applied to it and you may now follow that up with a ThenBy to impose additional ordering requirements."
The result of Where does not allow you to follow it up with ThenBy, and therefore you may not use it in a context where an IOrderedEnumerable<T> is required.
Make sense?
But of course, as others have said, you almost always want to do the filtering first and then the ordering. That way you are not spending time putting items into order that you are just going to throw away.
There are of course times when you do have to order and then filter; for example, the query "songs in the top ten that were sung by a woman" and the query "the top ten songs that were sung by a woman" are potentially very different! The first one is sort the songs -> take the top ten -> apply the filter. The second is apply the filter -> sort the songs -> take the top ten.

The signature of Where() is this:
public static IEnumerable<TSource> Where<TSource>(this IEnumerable<TSource> source, Func<TSource, bool> predicate)
So this method takes an IEnumerable<int> as first argument. The IOrderedEnumerable<int> returned from OrderBy implements IEnumerable<int> so this is no problem.
But as you can see, Where returns an IEnumerable<int> and not an IOrderedEnumerable<int>. And this cannot be casted into one another.
Anyway, the object in that sequence will still have the same order. So you could just do it like this
IEnumerable<int> tmp = new List<int>().OrderBy(x => x).Where(x => x > 1);
and get the sequence you expected.
But of course you should (for performance reasons) filter your objects first and sort them afterwards when there are fewer objects to sort:
IOrderedEnumerable<int> tmp = new List<int>().Where(x => x > 1).OrderBy(x => x);

The tmp variable's type is IOrderedEnumerable.
Where() is a function just like any other with a return type, and that return type is IEnumerable. IEnumerable and IOrderedEnumerable are not the same.
So when you do this:
tmp = tmp.Where(x => x > 1);
You are trying to assign the result of a Where() function call, which is an IEnuemrable, to the tmp variable, which is an IOrderedEnumerable. They are not directly compatible, there is no implicit cast, and so the compiler sends you an error.
The problem is you are being too specific with the tmp variable's type. You can make one simple change that will make this all work by being just be a little less specific with your tmp variable:
IEnumerable<int> tmp = new List<int>().OrderBy(x => x);
tmp = tmp.Where(x => x > 1);
Because IOrderedEnumerable inherits from IEnumerable, this code will all work. As long as you don't want to call ThenBy() later on, this should give you exactly the same results as you expect without any other loss of ability to use the tmp variable later.
If you really need an IOrderedEnumerable, you can always just call .OrderBy(x => x) again:
IOrderedEnumerable<int> tmp = new List<int>().OrderBy(x => x);
tmp = tmp.Where(x => x > 1).OrderBy(x => x);
And again, in most cases (not all, but most) you want to get your filtering out of the way before you start sorting. In other words, this is even better:
var tmp = new List<int>().Where(x => x > 1).OrderBy(x => x);

why wasn't this implemented?
Most likely because the LINQ designers decided that the effort to implement, test, document etc. isn't worth enough compared to the potential use cases. In fact your are the first one I hear complaining about that.
But if it's so important to you, you can add that missing functionality yourself (similar to #Jon Skeet MoreLINQ extension library). For instance, something like this:
namespace MyLinq
{
public static class Extensions
{
public static IOrderedEnumerable<T> Where<T>(this IOrderedEnumerable<T> source, Func<T, bool> predicate)
{
return new WhereOrderedEnumerable<T>(source, predicate);
}
class WhereOrderedEnumerable<T> : IOrderedEnumerable<T>
{
readonly IOrderedEnumerable<T> source;
readonly Func<T, bool> predicate;
public WhereOrderedEnumerable(IOrderedEnumerable<T> source, Func<T, bool> predicate)
{
if (source == null) throw new ArgumentNullException(nameof(source));
if (predicate == null) throw new ArgumentNullException(nameof(predicate));
this.source = source;
this.predicate = predicate;
}
public IOrderedEnumerable<T> CreateOrderedEnumerable<TKey>(Func<T, TKey> keySelector, IComparer<TKey> comparer, bool descending) =>
new WhereOrderedEnumerable<T>(source.CreateOrderedEnumerable(keySelector, comparer, descending), predicate);
public IEnumerator<T> GetEnumerator() => Enumerable.Where(source, predicate).GetEnumerator();
IEnumerator IEnumerable.GetEnumerator() => GetEnumerator();
}
}
}
And putting it into action:
using System;
using System.Collections.Generic;
using System.Linq;
using MyLinq;
var test = Enumerable.Range(0, 100)
.Select(n => new { Foo = 1 + (n / 20), Bar = 1 + n })
.OrderByDescending(e => e.Foo)
.Where(e => (e.Bar % 2) == 0)
.ThenByDescending(e => e.Bar) // Note this compiles:)
.ToList();

Make a Search Method generic using LINQ

I have a method in my project that repeats over and over:
public PAC PAC_GetByCodiPac(string codiPac)
{
var sel = _gam.PAC.Where(pac => pac.CODI_PAC == codiPac);
if (sel.Count() > 0)
return sel.First();
return null;
}
The table PAC means (patient), so I have these methods for all the tables I have.
How can I make a generic method for this?
Thanks in advance.

Here is your generic method. Note, that as others pointed out FirstOrDefault is better than count and then first, so I'm using it here. But it's also possible to write the expression so that it mimics what your original code does. Please let me know if you need additional help with this.
public static T GetByCodi<T>(IQueryable<T> table, string codi, string fieldName) where T : class
{
// x
ParameterExpression parameter = Expression.Parameter(typeof(T), "x");
Expression currentExpression = parameter;
Type currentType = typeof(T);
PropertyInfo property = currentType.GetProperty(fieldName);
// x.CODI_xxx
currentExpression = Expression.Property(currentExpression, property);
// x.CODI_xxx == codi
currentExpression = Expression.Equal(currentExpression, Expression.Constant(codi));
// x => x.CODI_xxx == codi
LambdaExpression lambdaExpression = Expression.Lambda(currentExpression, parameter);
return table.FirstOrDefault((Func<T, bool>)lambdaExpression.Compile());
}
You use it like this:
PAC xxx = GetByCodi<PAC>(_gam.PAC, codiPac, "CODI_PAC");
Edit 1:
I changed the code according to the comment so that you can pass arbitrary ID field name in.

I see that what you asked is a very straight forward where query even doesn't require to have have it on a separate method.
Also you can simply enhance your query link the following:
public PAC PAC_GetByCodiPac(string codiPac)
{
return _gam.PAC.FirstOrDefault(pac => pac.CODI_PAC == codiPac);
}
FirstOrDefault will return the first item on the array, if not it will return null.

If you want a generic method that lets you specify any table and any predicate for records from that table then you can't really get any better than the built-in Where<T>(...) and (as others have already pointed out) the FirstOrDefault<T>(...) extension methods.
Your code would then look like so:
var result = _gam.PAC.Where(pac => pac.CODI_PAC == codiPac).FirstOrDefault();
// OR
var result = _gam.PAC.FirstOrDefault(pac => pac.CODI_PAC == codiPac);
The best you could get then, writing your own generic method, would be this:
public T FirstOrDefault<T>(IQueryable<T> source,
Expression<Func<T, bool>> predicate)
{
return source.Where(predicate).FirstOrDefault();
// OR
// return source.FirstOrDefault(predicate);
}
And that is really just redundant. Especially when your calling code would be actually longer using the helper method:
var result = FirstOrDefault(_gam.PAC, pac => pac.CODI_PAC == codiPac);
// versus
var result = _gam.PAC.FirstOrDefault(pac => pac.CODI_PAC == codiPac);
And even worse, your code is no longer using a fluent, composable syntax. This just makes readability and maintenance more difficult.
If you stick with using the IQueryable<T> extension methods then you can do composition like this:
var result = _gam.PAC
.Where(pac => pac.CODI_PAC == codiPac)
.Where(pac => pac.SomeOtherProperty == someOtherValue)
.FirstOrDefault();
// OR
var result = (from pac in _gam.PAC
where pac.CODI_PAC == codiPac
where pac.SomeOtherProperty == someOtherValue
select pac).FirstOrDefault();
One very important thing to note here is that the predicate parameter in the IQueryable<T>.Where<T>(...) extension method is of type Expression<Func<T, bool>>. This allows the IQueryable<T> provider to construct the native SQL (or other native provider query) at the very last moment before returning a result.
Not using Expression<Func<T, bool>> means that your query would be the equivalent of this:
var result =
_gam.PAC
.ToArray()
.Where(pac => pac.CODI_PAC == codiPac)
.FirstOrDefault();
And that would mean the query will load every record from the "PAC" table into memory before selecting the first filtered result and throwing out the rest of the results.
The bottom-line is that by making a generic helper method you are rewriting existing framework code and you open yourself to performance and maintenance issues while also reducing code readability.
I hope this helps.

I'm not sure if you are asking for this, but this method could be in a static class and method and so you'd be able to call it from everywhere.

An easy solution will be:
//a generic method
private PAC PAC_GetPAC(Func<PAC, bool> predicate)
{
return _gam.PAC.Where(predicate).FirstOrDefault();
}
public PAC PAC_GetPACById(long id)
{
return PAC_GetPAC(p => p.ID == id);
}
public PAC PAC_GetByCodiPac(string codiPac)
{
return PAC_GetPAC(p => pac.CODI_PAC == codiPac);
}

Intersect LINQ query

If I have an IEnumerable where ClassA exposes an ID property of type long.
Is it possible to use a Linq query to get all instances of ClassA with ID belonging to a second IEnumerable?
In other words, can this be done?
IEnumerable<ClassA> = original.Intersect(idsToFind....)?
where original is an IEnumerable<ClassA> and idsToFind is IEnumerable<long>.

Yes.
As other people have answered, you can use Where, but it will be extremely inefficient for large sets.
If performance is a concern, you can call Join:
var results = original.Join(idsToFind, o => o.Id, id => id, (o, id) => o);
If idsToFind can contain duplicates, you'll need to either call Distinct() on the IDs or on the results or replace Join with GroupJoin (The parameters to GroupJoin would be the same).

I will post an answer using Intersect.
This is useful if you want to intersect 2 IEnumerables of the same type.
First we will need an EqualityComparer:
public class KeyEqualityComparer<T> : IEqualityComparer<T>
{
private readonly Func<T, object> keyExtractor;
public KeyEqualityComparer(Func<T, object> keyExtractor)
{
this.keyExtractor = keyExtractor;
}
public bool Equals(T x, T y)
{
return this.keyExtractor(x).Equals(this.keyExtractor(y));
}
public int GetHashCode(T obj)
{
return this.keyExtractor(obj).GetHashCode();
}
}
Secondly we apply the KeyEqualityComparer to the Intersect function:
var list3= list1.Intersect(list2, new KeyEqualityComparer<ClassToCompare>(s => s.Id));

You can do it, but in the current form, you'd want to use the Where extension method.
var results = original.Where(x => yourEnumerable.Contains(x.ID));
Intersect on the other hand will find elements that are in both IEnumerable's. If you are looking for just a list of ID's, you can do the following which takes advantage of Intersect
var ids = original.Select(x => x.ID).Intersect(yourEnumerable);

A simple way would be:
IEnumerable<ClassA> result = original.Where(a => idsToFind.contains(a.ID));

Use the Where method to filter the results:
var result = original.Where(o => idsToFind.Contains(o.ID));

Naming things is important. Here is an extension method base on the Join operator:
private static IEnumerable<TSource> IntersectBy<TSource, TKey>(
this IEnumerable<TSource> source,
IEnumerable<TKey> keys,
Func<TSource, TKey> keySelector)
=> source.Join(keys, keySelector, id => id, (o, id) => o);
You can use it like this var result = items.IntersectBy(ids, item => item.id).

I've been tripping up all morning on Intersect, and how it doesn't work anymore in core 3, due to it being client side not server side.
From a list of items pulled from a database, the user can then choose to display them in a way that requires children to attached to that original list to get more information.
What use to work was:
itemList = _context.Item
.Intersect(itemList)
.Include(i => i.Notes)
.ToList();
What seems to now work is:
itemList = _context.Item
.Where(item => itemList.Contains(item))
.Include(i => i.Notes)
.ToList();
This seems to be working as expected, without any significant performance difference, and is really no more complicated than the first.