Develop Reference

Expression<> & Func<> clarification in entity framework dot net core 2

so i am currently in the process of migrating our code base from EF 6 to EF core 2.0 and i am just running into a few problems with Expressions.
So we have some common filters that we have abstracted out to an Expression.
public static Expression<Func<AutotaskContractServiceUnit, bool>> ThisMonth()
{
var startDate = Clock.Now.Start();
var endDate = Clock.Now.End();
return csu => csu.StartDate.Date >= startDate.Date && csu.EndDate.Date <= endDate.Date;
}
And this is consumed as follows
return from vc in ContractRepository.GetAll()
let isAssigned = vc.AssetId != null
join v in _veeamRepository.GetAll() on vc.AssetId equals v.Id into veeams
from v in veeams.DefaultIfEmpty()
join ac in AutotaskContractRepository.GetAll() on vc.ContractId equals ac.Id
join ats in AutotaskServiceRepository.GetAll() on vc.ServiceId equals ats.Id
let billed = ac.ContractServiceUnits // navigation property
.AsQueryable()
.Where(AutotaskContractServiceUnitFilters.ThisMonth())
.Where(csu => csu.ServiceID == vc.ServiceId)
.OrderByDescending(csu => csu.Id)
.FirstOrDefault()
select new VeeamBillingModel
{
// property selection
};
The actual issue is with this part of the query
let billed = ac.ContractServiceUnits
.AsQueryable()
.Where(AutotaskContractServiceUnitFilters.ThisMonth())
.Where(csu => csu.ServiceID == vc.ServiceId)
.OrderByDescending(csu => csu.Id)
.FirstOrDefault()
As i get this exception
Could not parse expression '<>h__TransparentIdentifier4.<>h__TransparentIdentifier3.ac.ContractServiceUnits.AsQueryable()': This overload of the method 'System.Linq.Queryable.AsQueryable' is currently not supported.
So my question is, how can you achieve the same result in EF Core?
Things i've tried
Remove the AsQueryable(). The problem with this is that the compiler then complains that it cannot convert an Expression<> to Func<>
Argument 2: cannot convert from 'System.Linq.Expressions.Expression<System.Func<Portal.Autotask.ContractServiceUnit.Entities.AutotaskContractServiceUnit, bool>>' to 'System.Func<Portal.Autotask.ContractServiceUnit.Entities.AutotaskContractServiceUnit, bool>'
Change the Expression<> to a Func<> which compiles fine, but as soon as it tries to execute the query it throws an exception because its not an Expression<>.
I've also tried LinqKit (briefly) but couldn't get it working. I think because its from a joined table the .AsExpandable() doesn't work correctly.
The only way i have managed to get this working is to manually put the expression in the .Where clause.
let billed = ac.ContractServiceUnits
.Where(x => x.StartDate.Date >= start.Date && x.EndDate <= end.Date)
.Where(csu => csu.ServiceID == vc.ServiceId)
.OrderByDescending(csu => csu.Id)
.FirstOrDefault()
But this is not a good solution really as we have other expressions where the query logic could change and manually updating all the queries is too time consuming.
If anyone can shed a bit of light on this, that would be ace,
Cheers!

Linq to entities extension method inner query (EF6)

Can someone explain to me why the EF Engine is failing in the following scenario?
It works fine with the following expression:
var data = context.Programs
.Select(d => new MyDataDto
{
ProgramId = d.ProgramId,
ProgramName = d.ProgramName,
ClientId = d.ClientId,
Protocols = d.Protocols.Where(p => p.UserProtocols.Any(u => u.UserId == userId))
.Count(pr => pr.Programs.Any(pg => pg.ProgramId == d.ProgramId))
})
.ToList();
But if I encapsulate some into an extension method:
public static IQueryable<Protocol> ForUser(this IQueryable<Protocol> protocols, int userId)
{
return protocols.Where(p => p.UserProtocols.Any(u => u.UserId == userId));
}
The resulting query:
var data = context.Programs
.Select(d => new MyDataDto
{
ProgramId = d.ProgramId,
ProgramName = d.ProgramName,
ClientId = d.ClientId,
Protocols = d.Protocols.ForUser(userId)
.Count(pr => pr.Programs.Any(pg => pg.ProgramId == d.ProgramId))
})
.ToList();
Fails with the exception: LINQ to Entities does not recognize the method 'System.Linq.IQueryable1[DAL.Protocol] ForUser(System.Linq.IQueryable1[DAL.Protocol], Int32)' method, and this method cannot be translated into a store expression.
I would expect the EF Engine to build the entire expression tree, chaining the necessary expressions and then generate the SQL. Why doesn't it do that?

This is happening because the call to ForUser() is being made inside of the expression tree that the C# compiler builds when it sees the lambda you pass into Select. Entity Framework tries to figure out how to convert that function into SQL, but it can't invoke the function for a few reasons (e.g. d.Protocols does not exist at the moment).
The simplest approach that works for a case like this is to have your helper return a criteria lambda expression, and then pass that into the .Where() method yourself:
public static Expression<Func<Protocol, true>> ProtocolIsForUser(int userId)
{
return p => p.UserProtocols.Any(u => u.UserId == userId);
}
...
var protocolCriteria = Helpers.ProtocolIsForUser(userId);
var data = context.Programs
.Select(d => new MyDataDto
{
ProgramId = d.ProgramId,
ProgramName = d.ProgramName,
ClientId = d.ClientId,
Protocols = d.Protocols.Count(protocolCriteria)
})
.ToList();
More information
When you invoke a LINQ method outside of an expression tree (like you do with context.Programs.Select(...)), the Queryable.Select() extension method actually gets invoked, and its implementation returns an IQueryable<> that represents the extension method getting called on the original IQueryable<>. Here's the implementation of Select, for instance:
public static IQueryable<TResult> Select<TSource,TResult>(this IQueryable<TSource> source, Expression<Func<TSource, TResult>> selector) {
if (source == null)
throw Error.ArgumentNull("source");
if (selector == null)
throw Error.ArgumentNull("selector");
return source.Provider.CreateQuery<TResult>(
Expression.Call(
null,
GetMethodInfo(Queryable.Select, source, selector),
new Expression[] { source.Expression, Expression.Quote(selector) }
));
}
When the queryable's Provider has to generate actual data from the IQueryable<>, it analyzes the expression tree and tries to figure out how to interpret those method calls. Entity Framework has built-in knowledge of many LINQ-related functions like .Where() and .Select(), so it knows how to translate those method calls into SQL. However, it doesn't know what to do for methods that you write.
So why does this work?
var data = context.Programs.ForUser(userId);
The answer is that your ForUser method is not implemented like the Select method above: you are not adding an expression to the queryable to represent calling ForUser. Instead, you are returning the result of a .Where() call. From the IQueryable<>'s perspective, it's as if Where() was called directly, and the call to ForUser() never happened.
You can prove this by capturing the Expression property on the IQueryable<>:
Console.WriteLine(data.Expression.ToString());
... which will produce something like this:
Programs.Where(u => (u.UserId == value(Helpers<>c__DisplayClass1_0).userId))
There's no call to ForUser() anywhere in that expression.
On the other hand, if you include the ForUser() call inside of an expression tree like this:
var data = context.Programs.Select(d => d.Protocols.ForUser(id));
... then the .ForUser() method never actually gets invoked, so it never returns an IQueryable<> that knows the .Where() method got called. Instead, the expression tree for the queryable shows .ForUser() getting invoked. Outputting its expression tree would look something like this:
Programs.Select(d => d.Protocols.ForUser(value(Repository<>c__DisplayClass1_0).userId))
Entity Framework has no idea what ForUser() is supposed to do. As far as it's concerned, you could have written ForUser() to do something that's impossible to do in SQL. So it tells you that's not a supported method.

As I mentioned in my comment above, I can't tell why the EF Engine is working the way it is. Therefore, I've tried to find a way to re-write the query so I'll be able to make use of my extension methods.
The tables are:
Program -> 1..m -> ProgramProtocol -> m..1 -> Protocol
ProgramProtocol is just a join table and is not mapped in the model by Entity Framework.
The idea is simple: select "from left", select "from right" and then join the resulted sets for proper filtering:
var data = context.Programs.ForUser(userId)
.SelectMany(pm => pm.Protocols,
(pm, pt) => new {pm.ProgramId, pm.ProgramName, pm.ClientId, pt.ProtocolId})
.Join(context.Protocols.ForUser(userId), pm => pm.ProtocolId,
pt => pt.ProtocolId, (pm, pt) => pm)
.GroupBy(pm => new {pm.ProgramId, pm.ProgramName, pm.ClientId})
.Select(d => new MyDataDto
{
ProgramName = d.Key.ProgramName,
ProgramId = d.Key.ProgramId,
ClientId = d.Key.ClientId,
Protocols = d.Count()
})
.ToList();

Out of Memory Lambda Compile versus inline delegates

Using 4.5.1 with an application that on the server side shuffles chart data with many REST requests simultaneously.
Use IQueryable to build queries. For example, I originally had the following:
var query = ctx.Respondents
.Join(
ctx.Respondents,
other => other.RespondentId,
res => res.RespondentId,
(other, res) => new ChartJoin { Respondent = res, Occasion = null, BrandVisited = null, BrandInfo = null, Party = null, Item = null }
)
. // bunch of other joins filling out the ChartJoin
.Where(x => x.Respondent.status == 1)
. // more Where clauses dynamically applied
.GroupBy(x => new CommonGroupBy { Year = (int)x.Respondent.currentVisitYear, Month = (int)x.Respondent.currentVisitMonth })
.OrderBy(x => x.Key.Year)
.ThenBy(x => x.Key.Month)
.Select(x => new AverageEaterCheque
{
Year = x.Key.Year,
Month = x.Key.Month,
AverageCheque = (double)(x.Sum(m => m.BrandVisited.DOLLAR_TOTAL) / x.Sum(m => m.BrandVisited.NUM_PAID)),
Base = x.Count(),
Days = x.Select(m => m.Respondent.visitDate).Distinct().Count()
});
To allow for dynamic grouping (via the client), the GroupBy was generated with C# expressions returning a Dictionary. The Select also had to be generated with expressions. The above Select became something like:
public static Expression<Func<IGrouping<IDictionary<string, object>, ChartJoin>, AverageEaterCheque>> GetAverageEaterChequeSelector()
{
// x =>
var ParameterType = typeof(IGrouping<IDictionary<string, object>, ChartJoin>);
var parameter = Expression.Parameter(ParameterType);
// x => x.Sum(m => m.BrandVisited.DOLLAR_TOTAL) / x.Sum(m => m.BrandVisited.NUM_PAID)
var m = Expression.Parameter(typeof(ChartJoin), "m");
var mBrandVisited = Expression.PropertyOrField(m, "BrandVisited");
PropertyInfo DollarTotalPropertyInfo = typeof(BrandVisited).GetProperty("DOLLAR_TOTAL");
PropertyInfo NumPaidPropertyInfo = typeof(BrandVisited).GetProperty("NUM_PAID");
....
return a lambda...
}
When I did a test run locally I got an Out of Memory error. Then I started reading blogs from Totin and others that Lambda compiles, expression trees in general are expensive. Had no idea it would blow my application. And I need the ability to dynamically add grouping which lead me to using Expression trees for the GroupBy and Select clauses.
Would love some pointers on how to chase down the memory offenders in my application? Have seen some people use dotMemory but would be great with some practical tips as well. Very little experience in monitoring C#, DotNet.

Since you're compiling the expression into a delegate, the operation is performed using LINQ to Objects, rather than using the IQueryable overload. This means that the entirety of the data set is being pulled into memory, and all of the processing done by the application, instead of that processing being done in the database and only the final results being sent to the application.
Apparently pulling down the entire table into memory is enough to run your application out of memory.
You need to not compile the lambda, and leave it as an expression, thus allowing the query provider to translate it into SQL, as is done with your original code.

EF: LINQ - orderby using child collection with condition - ArgumentException

I'm running into troubles trying to sort IQueryable of my EF Entity.
My object structure is something like this:
Item
Item.CustomFieldValue [List<CustomFieldValue>]
Item.CustomFieldValue.DefinitionID
Item.CustomFieldValue.Value
and I'm working with
IQueryable<Item>
I'd need to sort it conditionally with values having desired definition id being sorted first something like this:
queryable = queryable
.OrderBy(p => p.CustomFieldValue
.Where(p2 => p2.DefinitionID == defId)
.Select(p3 => p3.Value)
.OrderBy(p4 => p4)
);
This however throws ArgumentException "DbSortClause expressions must have a type that is order comparable.".
I indeed understand what's the exception trying to say to me, I just can't figure out on how to change this so that valid query is generated.
Any help greatly appreciated
EDIT:
To bring some more light into the issue, I want to achieve something similar that this query does
SELECT * FROM ticketnumber t, customfieldvalue c
WHERE t.id like '%00000047%' and c.ticketnumberid = t.id
ORDER BY CASE
WHEN DefinitionId = 2125 THEN 1
ELSE 2
END, c.Value ASC
Alternatively, as time is starting to become a factor for me, is there a way I could append OrderBy in string form?

You probably want to use FirstOrDefault() at the end of the end of the first OrderBy so you won't be dealing with enumerables but with values.
queryable = queryable
.OrderBy(p => p.CustomFieldValue
.Where(p2 => p2.DefinitionID == defId)
.Select(p3 => p3.Value)
.OrderBy(p4 => p4)
.FirstOrDefault()
);

Modification of Joanvo's answer did the trick, this is the working code [I've removed the inner OrderBy]
queryable = queryable.OrderBy(p => p.CustomFieldValue.Where(p2 => p2.DefinitionID == defId).Select(p3 => p3.Value).FirstOrDefault());

Join between in memory collection and EntityFramework

Is there any mechanism for doing a JOIN between an in-memory collection and entity framework while preserving the order.
What I am trying is
var itemsToAdd =
myInMemoryList.Join(efRepo.All(), listitem => listitem.RECORD_NUMBER,
efRepoItem => efRepoItem.RECORD_NUMBER, (left, right) => right);
which gives me the rather curiously titled "This method supports the LINQ to Entities infrastructure and is not intended to be used directly from your code." error.
Now of course I can do this iteratively with something like
foreach (var item in myInMemoryList)
{
var ho = efRepo.Where(h => h.RECORD_NUMBER == item.RECORD_NUMBER).FirstOrDefault();
tmp.Add(ho);
}
but this is an N+1 query. Which is nasty as myInMemoryList might be quite large!
Resharper can refactor that for me to
tmp = (from TypeOfItemInTheList item in myInMemoryList
select efRepo.Where(h => h.RECORD_NUMBER == item.RECORD_NUMBER)
.FirstOrDefault());
which I suspect is still doing N+1 queries. So any ideas for a better approach to getting ef entities that match (on key field) with an in-memory collection. The resulting set must be in the same order as the in-memory collection was.

No you cannot join in-memory collection with database result set without loading whole result set to the memory and performing the join with linq-to-objects. Try using contains instead of join:
var myNumbers = myInMemoryList.Select(i => i.RECORD_NUMBER);
var itemsToAdd = efRepo.Where(e => myNumbers.Contains(e.RECORD_NUMBER));
This will generate query with IN operator

You can read how you can do this with the PredicateBuilder from the LINQKit or Stored Procedures in my blog post.
http://kalcik.net/2014/01/05/joining-data-in-memory-with-data-in-database-table/

try this:
var list = (from n in efRepo
where myInMemoryList.Select(m=>m.RECORD_NUMBER).Contains(n.RECORD_NUMBER)
select n).ToList();
Contains will be translated to IN operator in SQL (only if your RECORD_NUMBER member is a primitive type like int, string, Guid, etc)

What about loading the whole efRepo? I mean something like this (ToArray()):
var itemsToAdd = myInMemoryList.Join(
efRepo.ToArray(),
listitem => listitem.RECORD_NUMBER, efRepoItem => efRepoItem.RECORD_NUMBER, (left, right) => right);