Develop Reference

Factoring Linq Where clause for different EF objets [duplicate]

I'm getting the following error when trying to do a linq query:
LINQ to Entities does not recognize the method 'Boolean
IsCharityMatching(System.String, System.String)' method, and this
method cannot be translated into a store expression.
I've read lots of previous questions where people get the same error, and if I understand this correctly it's because LINQ to Entities requires the whole linq query expression to be translated to a server query, and therefore you can't call an outside method in it. I haven't been able to convert my scenario into something that works yet, and my brain is starting to melt down, so I was hoping someone could point me in the right direction. We're using Entity Framework and the specification pattern (and I'm new to both).
Here's the code that uses the specification:
ISpecification<Charity> specification = new CharitySearchSpecification(charityTitle, charityReference);
charities = charitiesRepository.Find(specification).OrderBy(p => p.RegisteredName).ToList();
Here's the linq expression:
public System.Linq.Expressions.Expression<Func<Charity, bool>> IsSatisfied()
{
return p => p.IsCharityMatching(this.charityName, this.charityReference);
}
Here's the IsCharityMatching method:
public bool IsCharityMatching(string name, string referenceNumber)
{
bool exists = true;
if (!String.IsNullOrEmpty(name))
{
if (!this.registeredName.ToLower().Contains(name.ToLower()) &&
!this.alias.ToLower().Contains(name.ToLower()) &&
!this.charityId.ToLower().Contains(name.ToLower()))
{
exists = false;
}
}
if (!String.IsNullOrEmpty(referenceNumber))
{
if (!this.charityReference.ToLower().Contains(referenceNumber.ToLower()))
{
exists = false;
}
}
return exists;
}
Let me know if you need any more information.
Many thanks,
Annelie

As you've figured out, Entity Framework can't actually run your C# code as part of its query. It has to be able to convert the query to an actual SQL statement. In order for that to work, you will have to restructure your query expression into an expression that Entity Framework can handle.
public System.Linq.Expressions.Expression<Func<Charity, bool>> IsSatisfied()
{
string name = this.charityName;
string referenceNumber = this.referenceNumber;
return p =>
(string.IsNullOrEmpty(name) ||
p.registeredName.ToLower().Contains(name.ToLower()) ||
p.alias.ToLower().Contains(name.ToLower()) ||
p.charityId.ToLower().Contains(name.ToLower())) &&
(string.IsNullOrEmpty(referenceNumber) ||
p.charityReference.ToLower().Contains(referenceNumber.ToLower()));
}

I got the same error in this code:
var articulos_en_almacen = xx.IV00102.Where(iv => alm_x_suc.Exists(axs => axs.almacen == iv.LOCNCODE.Trim())).Select(iv => iv.ITEMNMBR.Trim()).ToList();
this was the exactly error:
System.NotSupportedException: 'LINQ to Entities does not recognize the method 'Boolean Exists(System.Predicate`1[conector_gp.Models.almacenes_por_sucursal])' method, and this method cannot be translated into a store expression.'
I solved this way:
var articulos_en_almacen = xx.IV00102.ToList().Where(iv => alm_x_suc.Exists(axs => axs.almacen == iv.LOCNCODE.Trim())).Select(iv => iv.ITEMNMBR.Trim()).ToList();
I added a .ToList() before my table, this decouple the Entity and linq code, and avoid my next linq expression be translated
NOTE: this solution isn't optimal, because avoid entity filtering, and simply loads all table into memory

I ran into the same problem today, this was the first link I hit. However I was not looking for verifying my query. So if somebody else has the same issue and are looking for this solution it is added here. My issue was in another link.
It is the most common exception occurs when working with entity framework and converting data inside IQueryable result for filtering.
using (var context = new CustomerContext())
{
var item = context.InvoiceItems
.Where(i => i.Code == code.ToString())
.FirstOrDefault();
}
Several solutions exist. Move ToString() call to a separate line.
using (var context = new CustomerContext())
{
string codeStr = code.ToString();
var item = context.InvoiceItems
.Where(i => i.Code == codeStr)
.FirstOrDefault();
}
Use EF Extension Method,
using (var context = new CustomerContext())
{
var item = context.InvoiceItems
.Where(i => i.Code == SqlFunctions.StringConvert(code))
.FirstOrDefault();
}
Convert IQueryable result to IEnumerable before Filtering
using (var context = new CustomerContext())
{
var item = context.InvoiceItems.AsEnumerable()
.Where(i => i.Code == code.ToString())
.FirstOrDefault();
}

If anyone is looking for a VB.Net answer (as I was initially), here it is:
Public Function IsSatisfied() As Expression(Of Func(Of Charity, String, String, Boolean))
Return Function(charity, name, referenceNumber) (String.IsNullOrWhiteSpace(name) Or
charity.registeredName.ToLower().Contains(name.ToLower()) Or
charity.alias.ToLower().Contains(name.ToLower()) Or
charity.charityId.ToLower().Contains(name.ToLower())) And
(String.IsNullOrEmpty(referenceNumber) Or
charity.charityReference.ToLower().Contains(referenceNumber.ToLower()))
End Function

I got the same error in this code:
Solution
IQueryable to .toList() is the best option

Call a method in Select() of an Entity Framework Core method expression

I have 2 API endpoints which have similarities so I am trying to reuse some similar code. I have an EF Core method syntax to retrieve data and this query is used in both.
Rather than repeat it in both, as the Select()s are identical, I am trying to put it into a method.
This is one of the queries
var exhibitors = await Db.CompanyDescriptions
.Join(Db.Companies, cd => cd.CompanyId, c => c.CompanyID, (cd, c) => new { cd, c })
.Where(q => uniques.Any(id => id == q.cd.CompanyId)).Where(q => q.cd.EventId == eventId)
.Select(s => new ExhibitorModel()
{
Id = s.cd.Id,
EventId = s.cd.EventId,
}).ToListAsync();
Snipped for brevity, but I have this same code in another method (mostly) and then the methods go separate ways in terms of the unit of work, so I am trying to do something like this:
private ExhibitorModel MarshallExhibitorModel(CompanyDescriptions cd, Company c)
{
return new ExhibitorModel()
{
Id = cd.Id,
EventId = cd.EventId,
};
}
And then be able to use that in both Select()s, but I am unsure of how to form that syntax.
In this line
.Select(s => new ExhibitorModel()
How can I say call MarshallExhibitorModel and pass in s.companydescription and s.company ?

Here you have to do expression expanding. It means that you have to preprocess Expression Tree before feeding LINQ Provider with the query. Easiest way to do that is using LINQKit
Create function which returns Lambda Expression with the same parameters types:
public static Expression<Func<CompanyDescriptions, Company, ExhibitorModel>> MarshallExhibitorModel()
{
return (cd, c) => ExhibitorModel()
{
Id = cd.Id,
EventId = cd.EventId,
};
}
And then use this function in your query. Don't forget to call AsExpandable() at least once in the query.
var exhibitors = await Db.CompanyDescriptions
.Join(Db.Companies, cd => cd.CompanyId, c => c.CompanyID, (cd, c) => new { cd, c })
.Where(q => uniques.Any(id => id == q.cd.CompanyId)).Where(q => q.cd.EventId == eventId)
.AsExpandable() // important
.Select(s => MarshallExhibitorModel().Invoke(s.cd, s.c))
.ToListAsync();
Even better you can use such function in different situations:
...
.AsExpandable() // important
.Select(s => new SomeDTO
{
Exhibitor = MarshallExhibitorModel().Invoke(s.cd, s.c))
}
.ToListAsync();
This approach also can be used for generating complex reusable predicates which depend on parameters.

First, including the anonymous type (cd, c) => new { cd, c } would make code-reuse almost impossible. It looks like you only reference cd.Id, cd.CompanyId and cd.EventId. You could define an explicit type with these fields. Since you're only referring to CompanyDescriptions, it doesn't look like you need the join at all in this case. But even if you did, I am wondering why you aren't using navigation properties.
Ignoring that, you are then calling IQueryable<>.Select(), which takes an argument of type Expression<Func<,>>. This forces the C# compiler to construct an expression tree instead of compiling your lambda directly to a Func<,>.
You can define both an Expression<Func<,>> and a Func<,> from the same source;
public static Expression<Func<CompanyDescriptions,ExhibitorModel>> expression =
cd => new ExhibitorModel()
{
Id = cd.Id,
EventId = cd.EventId,
};
public static Func<CompanyDescriptions,ExhibitorModel> func = expression.Compile();
Just make sure you use the right one for IQueryable<>.Select(expression) vs IEnumerable<>.Select(func).

Just call function inside select. Add static in MarshallExhibitorModel() function too.
.Select(s => MarshallExhibitorModel(s.cd, s.c)

EF + LINQ: How do I define and utilize a variable (maybe a Func variable) to define part of a Select statement?

Before try to use a variable (this has sql server do the calculation):
IQueryable<MyEntity> query = _dbSet; // IQueryable<TEntity>
var results = query.Select(m => new MyViewModel
{
MyCalculation = m.Column1 * m.Column2
}).ToList();
What I want to do (dynamically create part of my select statement from a Func variable or some other kind of variable to allow this):
IQueryable<MyEntity> query = _dbSet; // IQueryable<TEntity>
Func<MyEntity, decimal> funcVariableAttempt = m => m.Column1 * m.Column2;
var results = query.Select(m => new MyViewModel
{
MyCalculation = funcVariableAttempt.Invoke(m) // My foolish attempt does not work.
}).ToList();
The error I get when I try what I want (aka my foolish attempt):
LINQ to Entities does not recognize the method 'System.Decimal Invoke(MyProject.Repository.Models.MyEntity)' method, and this method cannot be translated into a store expression.
How do I define and utilize a variable (maybe a Func variable) to define part of a Select statement?

It's a completely valid question that I stumbeled across earlier and found a solution that works well for me.
The problem with your Func<MyEntity, decimal> is that it is a delegate and that the O/R mapper has to have access to the internal expression (the multiplication of two properties in your case). But this information is compiled into the delegate and hidden forever.
If you however start off with a Expression<Func<MyEntity, decimal>> customCalculation, things look more promising as you have the internal logic as an expression tree.
Problem with this is: you cannot invoke an expression. customCalculation(m) doesn't compile.
The compiler would let you write
m => new MyViewModel { MyCalculation = customCalculation.Compile()(m) }
, but this wouldn't be understood by most O/R mappers.
But you see that it at least somehow contains the customCalculation lambda expression and also how it relates to its surrounding expressions.
Getting from here to the expression tree as in your original working version involves some expression manipulation:
We have to replace customCalculation.Compile()(m) with the body of the lambda that is to be Compile()d, but with the lambda's parameter(s) replaced with the respective expression(s) of the delegate invocation. So if customCalculation were x => x.Column1 * x.Column2, customCalculation.Compile()(m) would have to be replaced with m.Column1 * m.Column2
Doing so is not trivial, since the lambda itself has to be dug out of a field inside an instance of a compiler generated closure class.
I've posted my implementation of this expression manipulator in another similar question . Hope that helps.
With that, you should be able to:
var customCalculation = (Expression<Func<MyEntity, decimal>>)(x => x.Column1 * x.Column2);
var selector = Express.Prepare((Expression<Func<MyEntity, MyViewModel>>)(m => new MyViewModel { MyCalculation = customCalculation.Compile()(m) }));
var result = query.Select(selector).ToList();

As you already know, your funcVariableAttempt makes no sense to your database, so you have to call your method in the linq-to-object context. i.e. for instance first fetch the data as an Enumerable, then call your method:
var results = query.Select(m => new {
Column1= col1,
Column2= col2
}).AsEnumerable()
.Select(m => new MyViewModel
{
MyCalculation = Foo(m.Column1, m.Column2)
});
*Note: Code is not tested.

You should call ToList() first and perform the Select() on the result in memory.
var results = query.ToList()
.Select(m => new MyViewModel {
MyCalculation = Foo(m.Column1, m.Column2)
});
You're trying to perform the Select as part of the query. You should just use a regular function for the mapping calculation. Lambda functions are useful with LINQ but in this case they're not needed.

Linq to entities extension method inner query (EF6)

Can someone explain to me why the EF Engine is failing in the following scenario?
It works fine with the following expression:
var data = context.Programs
.Select(d => new MyDataDto
{
ProgramId = d.ProgramId,
ProgramName = d.ProgramName,
ClientId = d.ClientId,
Protocols = d.Protocols.Where(p => p.UserProtocols.Any(u => u.UserId == userId))
.Count(pr => pr.Programs.Any(pg => pg.ProgramId == d.ProgramId))
})
.ToList();
But if I encapsulate some into an extension method:
public static IQueryable<Protocol> ForUser(this IQueryable<Protocol> protocols, int userId)
{
return protocols.Where(p => p.UserProtocols.Any(u => u.UserId == userId));
}
The resulting query:
var data = context.Programs
.Select(d => new MyDataDto
{
ProgramId = d.ProgramId,
ProgramName = d.ProgramName,
ClientId = d.ClientId,
Protocols = d.Protocols.ForUser(userId)
.Count(pr => pr.Programs.Any(pg => pg.ProgramId == d.ProgramId))
})
.ToList();
Fails with the exception: LINQ to Entities does not recognize the method 'System.Linq.IQueryable1[DAL.Protocol] ForUser(System.Linq.IQueryable1[DAL.Protocol], Int32)' method, and this method cannot be translated into a store expression.
I would expect the EF Engine to build the entire expression tree, chaining the necessary expressions and then generate the SQL. Why doesn't it do that?

This is happening because the call to ForUser() is being made inside of the expression tree that the C# compiler builds when it sees the lambda you pass into Select. Entity Framework tries to figure out how to convert that function into SQL, but it can't invoke the function for a few reasons (e.g. d.Protocols does not exist at the moment).
The simplest approach that works for a case like this is to have your helper return a criteria lambda expression, and then pass that into the .Where() method yourself:
public static Expression<Func<Protocol, true>> ProtocolIsForUser(int userId)
{
return p => p.UserProtocols.Any(u => u.UserId == userId);
}
...
var protocolCriteria = Helpers.ProtocolIsForUser(userId);
var data = context.Programs
.Select(d => new MyDataDto
{
ProgramId = d.ProgramId,
ProgramName = d.ProgramName,
ClientId = d.ClientId,
Protocols = d.Protocols.Count(protocolCriteria)
})
.ToList();
More information
When you invoke a LINQ method outside of an expression tree (like you do with context.Programs.Select(...)), the Queryable.Select() extension method actually gets invoked, and its implementation returns an IQueryable<> that represents the extension method getting called on the original IQueryable<>. Here's the implementation of Select, for instance:
public static IQueryable<TResult> Select<TSource,TResult>(this IQueryable<TSource> source, Expression<Func<TSource, TResult>> selector) {
if (source == null)
throw Error.ArgumentNull("source");
if (selector == null)
throw Error.ArgumentNull("selector");
return source.Provider.CreateQuery<TResult>(
Expression.Call(
null,
GetMethodInfo(Queryable.Select, source, selector),
new Expression[] { source.Expression, Expression.Quote(selector) }
));
}
When the queryable's Provider has to generate actual data from the IQueryable<>, it analyzes the expression tree and tries to figure out how to interpret those method calls. Entity Framework has built-in knowledge of many LINQ-related functions like .Where() and .Select(), so it knows how to translate those method calls into SQL. However, it doesn't know what to do for methods that you write.
So why does this work?
var data = context.Programs.ForUser(userId);
The answer is that your ForUser method is not implemented like the Select method above: you are not adding an expression to the queryable to represent calling ForUser. Instead, you are returning the result of a .Where() call. From the IQueryable<>'s perspective, it's as if Where() was called directly, and the call to ForUser() never happened.
You can prove this by capturing the Expression property on the IQueryable<>:
Console.WriteLine(data.Expression.ToString());
... which will produce something like this:
Programs.Where(u => (u.UserId == value(Helpers<>c__DisplayClass1_0).userId))
There's no call to ForUser() anywhere in that expression.
On the other hand, if you include the ForUser() call inside of an expression tree like this:
var data = context.Programs.Select(d => d.Protocols.ForUser(id));
... then the .ForUser() method never actually gets invoked, so it never returns an IQueryable<> that knows the .Where() method got called. Instead, the expression tree for the queryable shows .ForUser() getting invoked. Outputting its expression tree would look something like this:
Programs.Select(d => d.Protocols.ForUser(value(Repository<>c__DisplayClass1_0).userId))
Entity Framework has no idea what ForUser() is supposed to do. As far as it's concerned, you could have written ForUser() to do something that's impossible to do in SQL. So it tells you that's not a supported method.

As I mentioned in my comment above, I can't tell why the EF Engine is working the way it is. Therefore, I've tried to find a way to re-write the query so I'll be able to make use of my extension methods.
The tables are:
Program -> 1..m -> ProgramProtocol -> m..1 -> Protocol
ProgramProtocol is just a join table and is not mapped in the model by Entity Framework.
The idea is simple: select "from left", select "from right" and then join the resulted sets for proper filtering:
var data = context.Programs.ForUser(userId)
.SelectMany(pm => pm.Protocols,
(pm, pt) => new {pm.ProgramId, pm.ProgramName, pm.ClientId, pt.ProtocolId})
.Join(context.Protocols.ForUser(userId), pm => pm.ProtocolId,
pt => pt.ProtocolId, (pm, pt) => pm)
.GroupBy(pm => new {pm.ProgramId, pm.ProgramName, pm.ClientId})
.Select(d => new MyDataDto
{
ProgramName = d.Key.ProgramName,
ProgramId = d.Key.ProgramId,
ClientId = d.Key.ClientId,
Protocols = d.Count()
})
.ToList();

Entity-Framework using expressions to build global and reusable filter/query-rules

Given the following linq-query:
var query1 = dbContext.MainTable.Where(m => m.MainId == _mainId).SelectMany(sub => sub.SubTable1)
.Select(sub1 => new
{
sub1.CategoryName,
VisibleDivisions = sub1.SubTable2
.Where(sub2 => sub2.Status == "Visible")
.Select(sub2 => new
{
/* select only what needed */
})
});
Starting from my main-table, I want to get all sub1's selected together with all the sub2's related to the sub1.
The query works as expected, generating a single query which will hit the database.
My question is regarding the inner Where-part, as of this filter will be used at several other parts in the application. So I would like to have this "visible-rule" defined at a single place (DRY-principle).
As of the Where is expecting an Func<SubTable2, bool> I have written the following property
public static Expression<Func<SubTable2, bool>> VisibleOnlyExpression => sub2 => sub2.Status == "Visible";
and changed my query to
var query1 = dbContext.MainTable.Where(m => m.MainId == _mainId).SelectMany(sub => sub.SubTable1)
.Select(sub1 => new
{
sub1.CategoryName,
VisibleDivisions = sub1.SubTable2
.Where(VisibleOnlyExpression.Compile())
.Select(sub2 => new
{
/* select only what needed */
})
});
This throws me an exception, stating Internal .NET Framework Data Provider error 1025..
I already tried changing to .Where(VisibleOnlyExpression.Compile()) with the same error.
I know that this is because EntityFramework is trying to transalte this into SQL which it can not.
My question is: How can I have my "filter-rules" defined at a single place (DRY) in code but have the still usable in Where-, Select-, ... -clauses which can be used on IQueryable as well as on ICollection for inner (sub-)queries?
I would love to be able to write something like:
var query = dbContext.MainTable
.Where(IsAwesome)
.SelectMany(s => s.SubTable1.Where(IsAlsoAwesome))
.Select(sub => new
{
Sub1sub2s = sub.SubTable2.Where(IsVisible),
Sub2Mains = sub.MainTable.Where(IsAwesome)
});
whereas the IsAwesome-rule is called first on IQueryable<MainTable> to get only awesome main-entries and later on ICollection<MainTable> in the sub-select to fetch only awesome main-entries related to a specific SubTable2-entry. But the rule - defining a MainTable-entry as awesome - will be the same, no matter where I call/filter for it.
I guess the solution will need the use of expression-trees and how they can be manipulated, so they will be translatable to plain SQL but I don't get the right idea or point to start with.

You can get something close to what are you asking for using the LinqKit AsExpandable and Invoke extension methods like this:
var isAvesome = IsAwesome;
var isAlsoAwesome = IsAlsoAwesome;
var isVisible = IsVisible;
var query = dbContext.MainTable
.AsExpandable()
.Where(mt => isAwesome.Invoke(mt))
.SelectMany(s => s.SubTable1.Where(st1 => isAlsoAwesome.Invoke(st1)))
.Select(sub => new
{
Sub1sub2s = sub.SubTable2.Where(st2 => isVisible.Invoke(st2)),
Sub2Mains = sub.MainTable.Where(mt => isAwesome.Invoke(mt))
});
I'm saying close because first you need to pull all the expressions needed into variables, otherwise you'll get the famous EF "Method not supported" exception. And second, the invocation is not so concise as in your wish. But at least it allows you to reuse the logic.

AFAIK what you are trying to do should be perfectly possible:
// You forgot to access ".Status" in your code.
// Also you don't have to use "=>" to initialize "IsVisible". Use the regular "=".
public static Expression<Func<SubTable2, bool>> IsVisible = sub2 =>
sub2.Status == "Visible";
...
VisibleDivisions = sub1
.SubTable2
// Don't call "Compile()" on your predicate expression. EF will do that.
.Where(IsVisibleOnly)
.Select(sub2 => new
{
/* select only what needed */
})

I would prepare extension method like below:
public static IQueryable<SubTable2> VisibleOnly(this IQueryable<SubTable2> source)
{
return source.Where(s => s.Status == "Visible");
}
An then you can use it in that way:
var query = dbContext.Table.VisibleOnly().Select(...)