Develop Reference

Writing an equivalent LINQ query , on a DbContext, using Reflection, is missing a where condition

I can't seem to create an equivalent query in reflection, of a LINQ query.
The goal is to run a query on an Entity Framework DBContext, but specifying the Tablename, and the where clause. There are several tables, so I do not want to copy/paste the same LINQ, for different tables.
The code below works with LINQ, but the reflections equivalent (although I cast it to System.Linq.IQueryable), does not allow me to add a .where clause.
I included using System.Linq; in the class.
Examples I researched from stackoverflow, exposes the Where() clause.
It's probably something minor I am overlooking. (missing a reference?)
// Link to TREESTRUCTURES Database
var DaContext = new WebApplication7.Models.Db_Entities.TREE_STRUCTURESEntities();
DaContext.Database.CommandTimeout=300;
// This works 100% for "HEADCOUNT_NEW"
var daTreeLeaves = DaContext.HEADCOUNT_NEW
.Where(x => x.PARENT_COMPONENT == "" && x.COMPONENT_DESCRIPTION == "Headcounts")
.Select(x => new { x.COMPONENT, x.COMPONENT_DESCRIPTION })
.OrderBy(x => x.COMPONENT)
.ToList();
// Reflections! = This works - Selecting from "HEADCOUNT_NEW" or any other specified available table;
PropertyInfo dbsetinfo = DaContext.GetType().GetProperties().FirstOrDefault(p => p.Name.ToUpper().Equals("HEADCOUNT_NEW"));
// I can't seem to get the `.Where()` clause right here (although this code does return values :) )
// I omitted the Where here as it would not compile.
System.Linq.IQueryable anyDbSet = ((IQueryable)dbsetinfo.GetValue(DaContext));
// Below is my attempt (pseudocode), but the error
// "IQueryable does not contain a definition for Where" is returned by VStudio
System.Linq.IQueryable anyDbSet2 = ((IQueryable)dbsetinfo.GetValue(DaContext)).Where("Id=" + pseudocode);

You need to cast to IQueryable<HEADCOUNT_NEW>
Where is an extension method which works on IQueryable<T>, not IQueryable.

Linq to entities extension method inner query (EF6)

Can someone explain to me why the EF Engine is failing in the following scenario?
It works fine with the following expression:
var data = context.Programs
.Select(d => new MyDataDto
{
ProgramId = d.ProgramId,
ProgramName = d.ProgramName,
ClientId = d.ClientId,
Protocols = d.Protocols.Where(p => p.UserProtocols.Any(u => u.UserId == userId))
.Count(pr => pr.Programs.Any(pg => pg.ProgramId == d.ProgramId))
})
.ToList();
But if I encapsulate some into an extension method:
public static IQueryable<Protocol> ForUser(this IQueryable<Protocol> protocols, int userId)
{
return protocols.Where(p => p.UserProtocols.Any(u => u.UserId == userId));
}
The resulting query:
var data = context.Programs
.Select(d => new MyDataDto
{
ProgramId = d.ProgramId,
ProgramName = d.ProgramName,
ClientId = d.ClientId,
Protocols = d.Protocols.ForUser(userId)
.Count(pr => pr.Programs.Any(pg => pg.ProgramId == d.ProgramId))
})
.ToList();
Fails with the exception: LINQ to Entities does not recognize the method 'System.Linq.IQueryable1[DAL.Protocol] ForUser(System.Linq.IQueryable1[DAL.Protocol], Int32)' method, and this method cannot be translated into a store expression.
I would expect the EF Engine to build the entire expression tree, chaining the necessary expressions and then generate the SQL. Why doesn't it do that?

This is happening because the call to ForUser() is being made inside of the expression tree that the C# compiler builds when it sees the lambda you pass into Select. Entity Framework tries to figure out how to convert that function into SQL, but it can't invoke the function for a few reasons (e.g. d.Protocols does not exist at the moment).
The simplest approach that works for a case like this is to have your helper return a criteria lambda expression, and then pass that into the .Where() method yourself:
public static Expression<Func<Protocol, true>> ProtocolIsForUser(int userId)
{
return p => p.UserProtocols.Any(u => u.UserId == userId);
}
...
var protocolCriteria = Helpers.ProtocolIsForUser(userId);
var data = context.Programs
.Select(d => new MyDataDto
{
ProgramId = d.ProgramId,
ProgramName = d.ProgramName,
ClientId = d.ClientId,
Protocols = d.Protocols.Count(protocolCriteria)
})
.ToList();
More information
When you invoke a LINQ method outside of an expression tree (like you do with context.Programs.Select(...)), the Queryable.Select() extension method actually gets invoked, and its implementation returns an IQueryable<> that represents the extension method getting called on the original IQueryable<>. Here's the implementation of Select, for instance:
public static IQueryable<TResult> Select<TSource,TResult>(this IQueryable<TSource> source, Expression<Func<TSource, TResult>> selector) {
if (source == null)
throw Error.ArgumentNull("source");
if (selector == null)
throw Error.ArgumentNull("selector");
return source.Provider.CreateQuery<TResult>(
Expression.Call(
null,
GetMethodInfo(Queryable.Select, source, selector),
new Expression[] { source.Expression, Expression.Quote(selector) }
));
}
When the queryable's Provider has to generate actual data from the IQueryable<>, it analyzes the expression tree and tries to figure out how to interpret those method calls. Entity Framework has built-in knowledge of many LINQ-related functions like .Where() and .Select(), so it knows how to translate those method calls into SQL. However, it doesn't know what to do for methods that you write.
So why does this work?
var data = context.Programs.ForUser(userId);
The answer is that your ForUser method is not implemented like the Select method above: you are not adding an expression to the queryable to represent calling ForUser. Instead, you are returning the result of a .Where() call. From the IQueryable<>'s perspective, it's as if Where() was called directly, and the call to ForUser() never happened.
You can prove this by capturing the Expression property on the IQueryable<>:
Console.WriteLine(data.Expression.ToString());
... which will produce something like this:
Programs.Where(u => (u.UserId == value(Helpers<>c__DisplayClass1_0).userId))
There's no call to ForUser() anywhere in that expression.
On the other hand, if you include the ForUser() call inside of an expression tree like this:
var data = context.Programs.Select(d => d.Protocols.ForUser(id));
... then the .ForUser() method never actually gets invoked, so it never returns an IQueryable<> that knows the .Where() method got called. Instead, the expression tree for the queryable shows .ForUser() getting invoked. Outputting its expression tree would look something like this:
Programs.Select(d => d.Protocols.ForUser(value(Repository<>c__DisplayClass1_0).userId))
Entity Framework has no idea what ForUser() is supposed to do. As far as it's concerned, you could have written ForUser() to do something that's impossible to do in SQL. So it tells you that's not a supported method.

As I mentioned in my comment above, I can't tell why the EF Engine is working the way it is. Therefore, I've tried to find a way to re-write the query so I'll be able to make use of my extension methods.
The tables are:
Program -> 1..m -> ProgramProtocol -> m..1 -> Protocol
ProgramProtocol is just a join table and is not mapped in the model by Entity Framework.
The idea is simple: select "from left", select "from right" and then join the resulted sets for proper filtering:
var data = context.Programs.ForUser(userId)
.SelectMany(pm => pm.Protocols,
(pm, pt) => new {pm.ProgramId, pm.ProgramName, pm.ClientId, pt.ProtocolId})
.Join(context.Protocols.ForUser(userId), pm => pm.ProtocolId,
pt => pt.ProtocolId, (pm, pt) => pm)
.GroupBy(pm => new {pm.ProgramId, pm.ProgramName, pm.ClientId})
.Select(d => new MyDataDto
{
ProgramName = d.Key.ProgramName,
ProgramId = d.Key.ProgramId,
ClientId = d.Key.ClientId,
Protocols = d.Count()
})
.ToList();

LINQ TO ENTITIES USING PREDICATE(Func(x,bool) Not Working

I have a Where clause predicate which is fetching records .Now if I use the same in a where clause as a Predicate(ie Func(x,bool) ) it returns no records.My impression was that both were equivalent. - see below
var nonPredicate=this.ObjectSet.Where(x=>!String.IsNullOrEmpty(x.Email) && x.Email.Contains("AND")).ToList();
Func<Model,bool) clause=x=> x=>!String.IsNullOrEmpty(x.Email) && x.Email.Contains("AND");
var predicateRes=this.ObjectSet.Where(clause).ToList();
I expected same results in both cases -but the first produces 29 record result and the second 0 records.
Any Help would be great
Thanks

Linq to Entitites uses Expression<Func<?,?>> for it's query methods. Lambda expressions are automatically coersed to an expression (Expression<Func<?,?>>) or delegate (Func<?,?>) depending on the type of the variable or argument it is stored in.
// Inline expression
var nonPredicate = this.ObjectSet.Where(x =>
!string.IsNullOrEmpty(x.Email) && x.Email.Contains("AND")).ToList();
// Delegate type
Func<Model,bool> clauseDelegate = x =>
!string.IsNullOrEmpty(x.Email) && x.Email.Contains("AND");
// Expression type
Expression<Func<Model,bool>> clauseExpr = x =>
!string.IsNullOrEmpty(x.Email) && x.Email.Contains("AND");
var predicateRes = this.ObjectSet.Where(clauseExpr).ToList();
Expressions are an object representation of the lambda, which allows the library to translate it into SQL for communication with the database. The same is not possible with delegates, since the structure is already compiled into IL or machine code.

error when using linqkit's predicate with an expression method

A little background:
I'm using a Linq provider (MsCrm2011 Linq provider, but doesn't really matter) that doesn't support certain operations.
Specifically, it doesn't support Contains() inside an expression.
Meaning- something like this will not work: var users = DataContext.Users.Where(user => userIds.Contains(user.Id)).
The solution I've found for it was to use LinqKit's predicate, so instead of userIds.Contains(...) I'll have user.Id == userIds[0] || user.Id == userIds[1] ... || user.Id == userIds[100].
To this end, I've defined the following function, which takes an arbitrary collection and arbitrary expression, and applies the 'Or' on them:
private IQueryable<TCrmEntity> FilterByCollection<TCrmEntity, T>(IQueryable<TCrmEntity> entities, IEnumerable<T> collection, Expression<Func<TCrmEntity, T, bool>> filterFunction)
{
var predicate = PredicateBuilder.False<TCrmEntity>();
predicate = collection.Aggregate(predicate, (current, collectionElement) => current.Or(entity => filterFunction.Invoke(entity,collectionElement)));
var query = entities.AsExpandable()
.Where(predicate);
return query;
}
this way, I can use any kind of collection and any kind of expression.
For example, see this test run (using an in-memory users collection): var res = FilterByCollection(users.AsQueryable(), rolesList, (account, role) => account.Role == role) to find all users that have one of the given roles.
However, I'm getting the following exception when I'm running the above example- variable 'entity' of type 'User' referenced from scope '', but it is not defined.
Any ideas?
P.S. not sure how relevant this is, but when I'm actually using the Crm data context instead of an in-memory collection, I'm not getting this error. :/

Linq To Sql - Changing Sort Order At Run-Time with well known static typing

This is not another question about 'How Can I Sort Dynamically (based on an arbitrary user provided field)?'
The question is -- how can I change sort order when I know the potential sorts in advance? (And thus avoid reflection / custom Expression building typically associated with truly dynamic sorting.)
Take for instance this subquery (shortened for this example) of a larger query:
(from solutionIds in context.csExtendedQAIncident_Docs
where solutionIds.tiRecordStatus == 1
&& (from solutionProductAssocation in context.csProductDocs
where solutionProductAssocation.iSiteId == Settings.Current.WebUtility().Onyx.SiteId
&& (from allowedProduct in context.KB_User_Allowed_Products
where allowedProduct.UserId == userId
select allowedProduct.ModelCode
).Contains(solutionProductAssocation.chModelCd)
select solutionProductAssocation.chIdNo).Distinct().Contains(solutionIds.chIdNo)
).OrderByDescending(s => s.dtUpdateDate)
.Select(s => s.chIdNo)
.Take(count ?? Settings.Current.WCFServices().Output.HomePage.MaxRows)
The OrderByDescending portion works as I would expect.
Now -- I want to factor that out like the following:
Expression<Func<csExtendedQAIncident_Doc, IComparable>> ordering = (s) => s.dtUpdateDate;
if (viewType == HomepageViewType.MostViewed)
ordering = (s) => s.vchUserField8;
else if (viewType == HomepageViewType.MostEffective)
ordering = (s) => s.vchUserField4;
and then use:
OrderByDescending(ordering)
This does compile, but blows up at run-time.
Unsupported overload used for query operator 'OrderByDescending'.
This of course comes from deep in the bowels of System.Data.Linq.SqlClient.QueryConverter -- in particular VisitSequenceOperatorCall. Reflectoring that code reveals that the following conditions must be met for OrderByDescending to properly evaluate. 'mc' is the MethodCallExpression passed into the method.
if (((mc.Arguments.Count != 2) || !this.IsLambda(mc.Arguments[1]))
|| (this.GetLambda(mc.Arguments[1]).Parameters.Count != 1))
{
break;
}
So essentially that MethodCallExpression has to have 2 arguments, the second of which has to be a Expressions.LambdaExpression with a single parameter (presumably the sort field). If that code breaks out, the exception that I got is thrown.
So clearly I have not constructed the expression correctly. Without digging in any further here, does anyone know how to correctly construct the sorting Expression?

I think the unsupported part of your code is the use of IComparable as a general return type for your ordering expression. If you consider the plain use of OrderByDescending, the compiler-generated lambda expression has a return type of the type of the property that you're ordering by: for example, an Expression<Func<csExtendedQAIncident_doc, string>> for a string property.
One possible answer, although I'm not sure whether it works in your case, is to first create an unordered query:
IQueryable<Foo> unorderedQuery = from f in db.Foo select f;
And then, depending on the sort:
IOrderedQueryable<Foo> orderedQuery = unorderedQuery
.OrderBy(f => f.DefaultSortKey);
if (sortBy == SortByName)
orderedQuery = unorderedQuery.OrderBy(f => f.Name);
else if (sortBy == SortByDate)
orderedQuery = unorderedQuery.OrderBy(f => f.Date);
// etc.

I believe that this will not work unless the two possible fields have the identical type.
Then the linq to sql will (if possible) correctly create the relevant sql.
so for example if both of those fields were DateTimes:
Expression<Func<csExtendedQAIncident_Doc, DateTime>> ordering =
s => s.dtUpdateDate;
if (viewType == HomepageViewType.MostViewed)
ordering = (s) => s.vchUserField8; // a DateTime
else if (viewType == HomepageViewType.MostEffective)
ordering = (s) => s.vchUserField4; // another DateTime
Then this would work just fine (I tested it and it worked)
You could instead do a per type order by either a series of nested switch/if statements of by constructing a dictionary or similar structure to get them.
For the linq to sql to work without explicit dynamic creation of the query I believe it must know the precise type of the query as opposed to just it being an IComparable...