Develop Reference

Generic Query With PredicateBuilder in Linqkit

I've been using LinqKit to create generic queries for quite some time.
One thing that has always bothered me is the fact that you always have to test whether the value sent in the filter is valid.
For example: Suppose I have a string filter. Conditions can be Equal, StartsWith, EndsWith and Contains.
My method would look something like this:
public List<MyModel> Get(MyModelFilter filter)
{
if (string.IsNullOrEmpty(filter.prop))
{
predicate = predicate.And(_myModel => myModel.Prop.Contains(filter.prop));
}
// Plus a giant amount of if's with multiple filters
return DbSet.AsExpandable()
.Where(predicate)
.ToList();
}
To end this bunch of If's, I decided to create a generic method to apply the filter to the properties.
My idea is to pass the property where the filter will be applied, and the filter definition, and encapsulate the Expression creation logic
It would be something of the type:
public List<MyModel> Get(MyModelFilter filter)
{
predicate = predicate.And(_myModel => myModel.Prop, filter.PropFilterDefinition);
// Goodnye If's, Only others filter impl
return DbSet.AsExpandable()
.Where(predicate)
.ToList();
}
For this, I've created some extension methods to handle this
public static Expression<Func<TPredicate, bool>> And<TPredicate>(
this ExpressionStarter<TPredicate> predicate,
Func<TPredicate, string> property, StringFilterDefinition filter,
bool ignoreNull = true)
{
if (InvalidStringFilter(filter, ignoreNull))
{
return predicate;
}
// This is LinqKit's And Extension Method
return predicate.And(BuildPredicate(property, filter));
}
private static Expression<Func<TPredicate, bool>> BuildPredicate<TPredicate>(
Func<TPredicate, string> property,
StringFilterDefinition filter)
{
if (filter.Filter == StringFilterComparators.Equal)
{
return x => property.Invoke(x) == filter.Value;
}
if (filter.Filter == StringFilterComparators.BeginsWith)
{
return x => property.Invoke(x).StartsWith(filter.Value);
}
if (filter.Filter == StringFilterComparators.EndsWith)
{
return x => property.Invoke(x).EndsWith(filter.Value);
}
return x => property.Invoke(x).Contains(filter.Value);
}
private static bool InvalidStringFilter(
StringFilterDefinition filter,
bool ignoreNullValue = true)
{
if (filter?.Filter == null)
{
return true;
}
return ignoreNullValue && string.IsNullOrEmpty(filter.Value);
}
The problem is that the filter is not applied, and the answer is in Invoke right up there. EF can not translate the above expression to SQL.
The EF error is
Microsoft.EntityFrameworkCore.Query.Internal.SqlServerQueryCompilationContextFactory[8]
The LINQ expression '(__property_0.Invoke([x]) == __filter_Value_1)'
could not be translated and will be evaluated locally. To configure
this warning use the DbContextOptionsBuilder.ConfigureWarnings API
(event id 'RelationalEventId.QueryClientEvaluationWarning').
ConfigureWarnings can be used when overriding the
DbContext.OnConfiguring method or using AddDbContext on the
application service provider.
The question is:
How can I make this construction work?
Also, any suggestions on how best this?

You seem to forgot that besides the PredicateBuilder, the really useful feature provided by LINQKit AsExpandable, Expand and Invoke custom extension methods is to be able to correctly embed expressions inside the expression tree.
In order to utilize that feature, you should use Expression<Func<...>> instead of Func<...>. In the posted code, replace all occurrences of Func<TPredicate, string> with Expression<Func<TPredicate, string>> and the issue should be solved.

Polymorphism by generic types where types do not share a base class

Background
This is a refactoring question. I have a bunch of methods that more or less have exactly the same code but they act on different types. There is essentially one method per type and I want to combine them all into one that can use a generic type.
Current Code
Perhaps the below code will help explain what I'm trying -
The below methods differ mostly in the DbSet<> entity argument. Inside the method code, they use mostly exactly the same properties but in one or two lines they may use properties that are not shared by the entity types. For example, AccountId (from Account entity) and CustomerId (from Customer entity).
int? MethodToRefactor(DbSet<Account> entity, List someCollection, string[] moreParams)
{
int? keyValue = null;
foreach (var itemDetail in someCollection)
{
string refText = GetRefTextBySource(itemDetail, moreParams);
//Only the below two lines differ in all MethodToRefactor because they use entity's properties that are not shared by all entities
if (entity.Count(a => a.Name == refText) > 0)
keyValue = entity.Where(a => a.Name == refText).First().AccountId;
if (...some conditional code...)
break;
}
return keyValue;
}
int? MethodToRefactor(DbSet<Customer> entity, List someCollection, string[] moreParams)
{
int? keyValue = null;
foreach (var itemDetail in someCollection)
{
string refText = GetRefTextBySource(itemDetail, moreParams);
//Only the below two lines differ in all MethodToRefactor because they use entity's properties that are not shared by all entities
if (entity.Count(c => c.CustomerName == refText) > 0)
keyValue = entity.Where(c => c.CustomerName == refText).First().CustomerId;
if (...some conditional code...)
break;
}
return keyValue;
}
Below is the code that calls the above methods -
void Caller()
{
foreach (var entity in EntityCollection)
{
if (entity.Name == "Account")
{
id = MethodToRefactor(db.Accounts,...);
}
else if (entity.Name == "Customer")
{
id = MethodToRefactor(db.Customers,...);
}
}
}
Problem
This is not scalable for one thing because it requires copying/pasting a new MethodToRefactor for each newly added entity. It is difficult to maintain as well. I can perhaps refactor the code common to all MethodToRefactors in a separate method and do an ifelse inside it per entity but then I would basically be merging the Caller with MethodToRefactor. I'm looking for a neater solution with minimal changes in Caller method, as described below.
Ideal/desired refactored code
This is a great candidate for generic/template types. As seen below, I can change the actual entity to be a generic T and pass the two lines that do not use the common properties among the entities as expressions/methods.
Below is the C# type of pseudocode that demonstrates the ideal solution but I don't know how to actually do it in C#.
int? MethodToRefactor<T>(DbSet<T> entity, Expression<Func<T, T> filterMethod,
Expression<Func<T, T> getIdMethod, List someCollection, string[] moreParams) where T : Account, Customer //This will fail
{
int? keyValue = null;
foreach (var itemDetail in someCollection)
{
string refText = GetRefTextBySource(itemDetail, moreParams);
if (filterMethod(entity) == true)
keyValue = getIdMethod(entity);
if (...some conditional code...)
break;
}
return keyValue;
}
void Caller()
{
foreach (var entity in EntityCollection)
{
if (entity.Name == "Account")
{
id = MethodToRefactor<Account>(db.Accounts, () => {entity.Count(a => a.Name == refText) > 0}, () => {entity.Where(a => a.Name == refText).First().AccountId},...);
}
else if (entity.Name == "Customer")
{
id = MethodToRefactor<Customer>(db.Customer, () => {entity.Count(c => c.CustomerName == refText) > 0}, () => {entity.Where(c => c.CustomerName == refText).First().CustomerId},...);
}
}
}
Benefits / Goals Achieved
1. We combined all of MethodToRefactors into one and eliminated all duplicate code.
2. We abstracted away entity specific operations to the Caller. This is important because that logic is moved to the one logical place that knows how different entities differ from each other (Caller had a per entity ifelse to begin with) and how those differences are to be used.
2. By delegating the entity specific code to the Caller we also made it more flexible so that we don't have to create one MethodToRefactor per entity specific logic.
Note: I'm not a big fan of Adapter, Strategy etc, I prefer solutions that can achieve those goals using C# language features. That doesn't mean I'm anti-classical-design-patterns, it's just that I don't like the idea of creating a bunch of new classes when I can do it by refactoring into a couple of methods.

If the entities do not have the same base class, the best you can do is to have a class constraint.
Since both expressions are essentially the same, you should just pass one expression and a function to get the key value from the entity.
The Count and First methods can also be merged into a single statement and then checking for null.
int? MethodToRefactor<T>(DbSet<T> entities, Func<string, Expression<Func<T, bool>>> expressionFilter, Func<T, int> getIdFunc, IList<string> someCollection, string[] moreParams)
where T : class
{
int? keyValue = null;
foreach (var itemDetail in someCollection)
{
string refText = GetRefTextBySource(itemDetail, moreParams);
var entity = entities.FirstOrDefault(expressionFilter(refText));
if (entity != null)
{
keyValue = getIdFunc(entity);
}
if (...some conditional code...)
break;
}
return keyValue;
}
You would call the method like this
id = MethodToRefactor<Account>(db.Accounts, txt => a => a.Name == txt, a => a.AccountId, ...);
id = MethodToRefactor<Customer>(db.Customers, txt => c => c.CustomerName == txt, c => c.CustomerId, ...);

Here is how you can do it.
Given a type T, all we need is an accessor to a string property to be compared with refText and also an accessor to a int property for keyValue. The first is expressed by Expression<Func<T, string>> nameSelector and the second by Expression<Func<T, int>> keySelector, so these should be the additional parameters to the MethodToRefactor.
What about the implementation, the code
if (entity.Count(a => a.Name == refText) > 0)
keyValue = entity.Where(a => a.Name == refText).First().AccountId;
can be made more optimal (using a single database query returning just one field) like this (pseudo code):
keyValue = entity.Where(e => nameSelector(e) == refText)
.Select(e => (int?)keySelector(e))
.FirstOrDefault();
The int? cast is needed to allow returning null when refText does not exist.
In order to implement that, we need two expressions derived from the arguments:
Expression<Func<T, bool>> predicate = e => nameSelector(e) == refText;
and
Expression<Func<T, int?>> nullableKeySelector = e => (int?)keySelector(e);
Of course the above is not a valid syntax, but can easily be build with System.Linq.Expressions.
With all that being said, the refactored method could be like this:
int? MethodToRefactor<T>(
DbSet<T> entitySet,
Expression<Func<T, string>> nameSelector,
Expression<Func<T, int>> keySelector,
List someCollection,
string[] moreParams)
where T : class
{
int? keyValue = null;
foreach (var itemDetail in someCollection)
{
string refText = GetRefTextBySource(itemDetail, moreParams);
// Build the two expressions needed
var predicate = Expression.Lambda<Func<T, bool>>(
Expression.Equal(nameSelector.Body, Expression.Constant(refText)),
nameSelector.Parameters);
var nullableKeySelector = Expression.Lambda<Func<T, int?>>(
Expression.Convert(keySelector.Body, typeof(int?)),
keySelector.Parameters);
// Execute the query and process the result
var key = entitySet.Where(predicate).Select(nullableKeySelector).FirstOrDefault();
if (key != null)
keyValue = key;
if (...some conditional code...)
break;
}
return keyValue;
}
and the usage:
Account:
id = MethodToRefactor(db.Accounts, e => e.Name, e => e.AccountId, ...);
Customer:
id = MethodToRefactor(db.Customer, e => e.CustomerName, e => e.CustomerId, ...);

I understand you do not have a base class, but your method is definately only applicable to classes of your dal. As such, i would definatelty mark the available classes with an interface. This will help others on your team to get a hint of where they can use your method. I always added a base interface to my dal classes.
I do not think defining the key property is the responsability of your caller. The key is something the entity should provide.
Having an interface, you can already abstract the key property to it, having
internal interface IEntity
{
int Key { get; }
}
Of course you can have it generic by the keytype, if you have more than one.
As for your search term property, this is something you need to decide. Either it is also a property of the entity (if this property/ies (why only one???) is used in more than one place), or is used only in this method. I would guess for the sake of simplicity, this is used only here.
In this case your method would look like:
int? MethodToRefactor<T>(EfContext context, IEnumerable<Expression<Func<T, string>>> searchFields, IEnumerable<string> someCollection, string[] moreParams)
where T : class, IEntity
{
int? keyValue = null;
foreach (var itemDetail in someCollection)
{
string refText = GetRefTextBySource(itemDetail, moreParams);
if (searchFields.Any())
{
var filter = searchFields.Skip(1).Aggregate(EqualsValue(searchFields.First(), refText), (e1, e2) => CombineWithOr(e1, EqualsValue(e2, refText)));
var entity = context.Set<T>().FirstOrDefault(filter);
if (entity != null)
{
keyValue = entity.Key;
}
if (... some condition ...)
break;
}
}
return keyValue;
}
private Expression<Func<T, bool>> EqualsValue<T>(Expression<Func<T, string>> propertyExpression, string strValue)
{
var valueAsParam = new {Value = strValue}; // this is just to ensure that your strValue will be an sql parameter, and not a constant in the sql
// this will speed up further calls by allowing the server to reuse a previously calculated query plan
// this is a trick for ef, if you use something else, you can maybe skip this
return Expression.Lambda<Func<T, bool>>(
Expression.Equal(propertyExpression.Body, Expression.MakeMemberAccess(Expression.Constant(valueAsParam), valueAsParam.GetType().GetProperty("Value"))),
propertyExpression.Parameters); // here you can cache the property info
}
private class ParamReplacer : ExpressionVisitor // this i guess you might have already
{
private ParameterExpression NewParam {get;set;}
public ParamReplacer(ParameterExpression newParam)
{
NewParam = newParam;
}
protected override Expression VisitParameter(ParameterExpression expression)
{
return NewParam;
}
}
private Expression<Func<T, bool>> CombineWithOr<T>(Expression<Func<T, bool>> e1, Expression<Func<T, bool>> e2) // this is also found in many helper libraries
{
return Expression.Lambda<Func<T, bool>>(Expression.Or(e1.Body, new ParamReplacer(e1.Parameters.Single()).VisitAndConvert(e2.Body, MethodBase.GetCurrentMethod().Name)), e1.Parameters);
}
Now this will obviously require you to implement the key property on all your entities, which in my opinion is not such a bad thing. Apparently you use your key properties anyways for other stuff too (otherwise why would this method return a key only).
On another note, you are retrieving the whole entity when a match is found, but then you care only about the key. This could be made better by retrieving only the key, e.g. adding a select to the end of the expression. Unfortunately in this case you would need a bit more "magic" in order for ef (or your linq provider) to understand the .Select(e => e.Key) expression (at least ef won't out of the box). Since i hope you need the whole entity in your "... some condition...", im not including this version in this answer (also to keep it short :P).
So finally your caller would look like:
void Caller()
{
foreach (var entity in EntityCollection)
{
if (entity.Name == "Account")
{
id = MethodToRefactor<Account>(db, new [] {a => a.Name}, ...);
}
else if (entity.Name == "Customer")
{
id = MethodToRefactor<Customer>(db, new [] {c => c.FirstName, c => c.LastName}, ...);
}
}
}

Use a function or Expression to replace the "new {x.ID,x.Name}" part

I may have function calls like this :
foo(**new {x.ID,x.Name}**);
and LINQ:
(IQueryable<SomeTableName>).where(x=>x.ID>1).select(x=>**new {x.ID,x.Name}**);
Is it possible to replace the "new {x.ID,x.Name}" part with a function, expression or variable, so I can change from time to time, in only 1 place?
Thank you.
I know I can make an Expression, which be used inside LINQ only
public static Func<SomeTableName, Object> Select_RS_Food = x => new { x.ID,x.Name };
but I also want to use for normal anonymous object creating. Like:
foo(CreateMyObject(x));
Update
Current:
return new { myTable.ID, myTable.Name};
//And
db.SomeTable.Select (x=> new { myTable.ID, myTable.Name));
Expect:
return SomeMagicCode;
//And
db.SomeTable.Select (x=> SomeMagicCode);

You could define a helper class like this to allow you to specify only some of the generic type parameters when creating a Func<T, R>:
public static class Funk<T>
{
public static Func<T, R> Make<R>(Func<T, R> func)
{
return func;
}
}
And use it like this:
var selector = Funk<SomeTableName>.Make(x => new {x.ID, x.Name});
var result = db.SomeTable.Where(x => x.ID>1).Select(selector);
This takes advantage of type inference to determine the return type, so you only have to define the input type (SomeTableName). The var keyword makes the selector variable implicitly typed, you won't have to specify Func<SomeTableName, _AnonymousType1_>. You can now reuse selector in multiple queries.
You will not be able to save this as property or field of a class, however, because implicit typing is not supported in type members. If you want to be able to reuse this selection function, you must use a named type or perhaps dynamic—but I wouldn't expect this to work with any current ORM.
For the other lambda expressions (x => x.ID == 1 and x => x.ID > 1), there are no anonymous types involved so you can simply do this:
public static Func<SomeTableName, bool> IDEquals1 = x => x.ID == 1;
public static Func<SomeTableName, bool> IDGreaterThan1 = x => x.ID == 1;
But by the way, if your using an ORM like entity framework, you probably need to use an Expression instead. Maybe something kind of like this:
public static Expression<Func<T, bool>> IDEquals1 = x => x.ID == 1;
Or even this:
public interface IHasID
{
int ID { get; }
}
public SomeTableName : IHasID { ... }
public static Expression<Func<T, bool>> IDEquals1<T>() where T : IHasID
{
return x => x.ID == 1;
}

Expression for OrderBy linq with inner expression

I have the following expresssion for sorting:
this.Students = this.Students.OrderBy(x => x.ExamData.OrderByDescending(p => p.ExamDate).ThenByDescending(p => p.ExamId).FirstOrDefault().TotalMarks);
While my idea is to abstract the Expression for
x => x.ExamData.OrderByDescending(p => p.ExamDate).ThenByDescending(
p => p.ExamId).FirstOrDefault().TotalMarks
to be made as an lambda Expression so that I can use like
this.Students = this.Students.OrderBy(sortExpression);
It's because I have many sort fields like the TotalMarks defined above, and I want to just create the Expression from the sort field and then call the OrderBy.
I know from this link, we can create an expression where child property is used, but not getting with the inner expressions.
Currently I have given a switch case and written the same stuff in each case like
this.Students = this.Students.OrderBy(x => x.ExamData.OrderByDescending(p => p.ExamDate).ThenByDescending(p => p.ExamId).FirstOrDefault().SubjectName);
So my idea is to create kindof ExpressionBuilder with a static method which builds the expression passing on the fieldName, like
public static Expression BuildSortExpression(string fieldName) {}

You can easily pull most of the logic into a method:
private int sortExpression(Student x) {
return x.ExamData.OrderByDescending(p => p.ExamDate).ThenByDescending(p => p.ExamId).FirstOrDefault().TotalMarks;
}
Assuming TotalMarks is an int.
Then you would just need to use:
this.Students.OrderBy(x => sortExpression(x));
or add it as a property of Student.
Warning:
If you are using this with an ORM (linq to SQL, Entity framework, etc), this will not execute as efficiently as the previous code!

Trying to create a reusable expression variable will end up being a lot more work than just creating your own extension method to do the whole ordering:
public static IQueryable<Student> OrderByMarks(this IQueryable<Student> students)
{
return students.OrderBy(student => student.ExamData
.OrderByDescending(exam => exam.ExamDate)
.ThenBy(exam => exam.ExamId)
.FirstOrDefault().TotalMarks);
}
Then you can use it like so:
this.Students = this.Students.OrderByMarks();

Got a solution for this using the idea by Ben.
Created a
Dictionary<string, Func<Student, Object>>
with the sort fields as key and the func as
new Func<Student, object>((Student student) => { return GetLatestExam(student).TotalMarks; })
And the GetLatestExam static method as
private static Study GetLatestExam(Student student)
{
return student.ExamData.OrderByDescending(p => p.ExamDate).ThenByDescending(p => p.ExamId).FirstOrDefault();
}
Then in the actual sort, I just have to call like this:
public void Sort(string sortField, bool sortAscending)
{
// based on the sort field, get the sorting expression and execute.
if(sortAscending)
{
this.Students= this.Students.OrderBy(student=>this._customSortColumns[sortField](student));
}
else
{
this.Patients = this.Patients.OrderByDescending(student=>this._customSortColumns[sortField](student));
}
}

LinqToSql Select to a class then do more queries

I have a LINQ query running with multiple joins and I want to pass it around as an IQueryable<T> and apply additional filters in other methods.
The problem is that I can't work out how to pass around a var data type and keep it strongly typed, and if I try to put it in my own class (EG: .Select((a,b) => new MyClass(a,b))) I get errors when I try to add later Where clauses because my class has no translations into SQL. Is there any way I can do one of the following:
Make my class map to SQL?
Make the var data-type implement an interface (So I can pass it round as though it's that)?
Something I haven't though of that'll solve my issue?
Example:
public void Main()
{
using (DBDataContext context = new DBDataContext())
{
var result = context.TableAs.Join(
context.TableBs,
a => a.BID,
b => b.ID,
(a,b) => new {A = a, B = b}
);
result = addNeedValue(result, 4);
}
}
private ???? addNeedValue(???? result, int value)
{
return result.Where(r => r.A.Value == value);
}
PS: I know in my example I can flatten out the function easily, but in the real thing it'd be an absolute mess if I tried.

All you have to do is de-anonymize your type that you're constructing as a result of the query - new { A = a, B = b }
Just create a class with those properties and name it appropriately. Then your query will have a type of IQueryable<your type>
Like so:
public class MyClass
{
public int A { get; set; }
public int B { get; set; }
}
public void Main()
{
using (DBDataContext context = new DBDataContext())
{
var result = context.TableAs.Join(
context.TableBs,
a => a.BID,
b => b.ID,
(a,b) => new MyClass {A = a, B = b}
);
result = addNeedValue(result, 4);
}
}
private IQueryable<MyClass> addNeedValue(IQueryable<MyClass> result, int value)
{
return result.Where(r => r.A.Value == value);
}

Here's two different approaches; the first applies the filter before doing the Join, since the joined queries don't have to just be basic tables. The second approach applies the filter after the join, using an intermediary projection (and changing it to return the predicate, rather than applying it internally).
This has been tested successfully on .NET 3.5 and .NET 4; note that in 3.5 (SP1) the Expression.Invoke (for the second example) won't work on EF, but is fine for LINQ-to-SQL.
If you want to run the example, I've used Northwind (just because that is what I had locally):
using System;
using System.Linq;
using System.Linq.Expressions;
using ConsoleApplication1; // my data-context's namespace
static class Program
{
public static void Main()
{
using (var context = new TestDataContext())
{
context.Log = Console.Out; // to check it has worked
IQueryable<Order> lhs = context.Orders;
IQueryable<Order_Detail> rhs = context.Order_Details;
// how ever many predicates etc here
rhs = addBeforeJoin(rhs, 4);
var result = lhs.Join(rhs,
a => a.OrderID,
b => b.OrderID,
(a, b) => new { A = a, B = b }
);
// or add after
result = result.Where(row => row.B, addAfterJoin(100));
Console.WriteLine(result.Count());
}
}
private static IQueryable<Order_Detail> addBeforeJoin(IQueryable<Order_Detail> query, int value)
{
return query.Where(r => r.Quantity >= value);
}
private static Expression<Func<Order_Detail, bool>> addAfterJoin(int value)
{
return r => r.Quantity <= value;
}
private static IQueryable<TSource> Where<TSource, TProjection>(
this IQueryable<TSource> source,
Expression<Func<TSource, TProjection>> selector,
Expression<Func<TProjection, bool>> predicate)
{
return source.Where(
Expression.Lambda<Func<TSource, bool>>(
Expression.Invoke(predicate, selector.Body),
selector.Parameters));
}
}

You are returning what's known as an 'anonymous type' from your join - it's not a specific database type, but is instead just an object containing A and B. Because the type is anonymous, I don't think you're going to be able to write a function that returns that type - or a collection of that type.
If it were a type that SQL / LINQ-to-SQL knew about then you'd probably be able to do this.
One way to get LINQ-to-SQL to recognize the type might be to create a stored procedure which performs the join and selects the relevant columns. If you add that stored procedure to your DBML file, LINQ-to-SQL will understand the type of each 'row' in the result. You should then be able to have your function return an IQueryable<ResultSetRow>.

I definitely don't recommend it, but you can probably use dynamic here, if you're on .NET 4.0.