Develop Reference

In linq to EF can I create a dynamic query to search on a field specified by the user

We have some user defined data stored in a table with columns called "Date01", "Date02", "Text01", "Text02", "Number01", "Number02" etc. The user configures which columns are used and what the data stored in those columns means.
We now need to allow the user to search this data.
I could write something like this:
if (property.Name.StartsWith("Date") && search.Value is DateTime)
{
switch (property.Name)
{
case "Date01":
results = table.Where(clf => clf.Date01.HasValue && clf.Date01.Value.Date == ((DateTime)search.Value).Date);
break;
case "Date02":
results = table.Where(clf => clf.Date02.HasValue && clf.Date02.Value.Date == ((DateTime)search.Value).Date);
break;
case "Date03":
....
}
}
else if (property.Name.StartsWith("Text") && search.Value is string)
{
switch (property.Name)
{
case "Text01":
results = table.Where(clf => clf.Text01 == (string)search.Value);
break;
case "Text02":
results = table.Where(clf => clf.Text02 == (string)search.Value);
break;
case "Text03":
....
}
}
but this strikes me as potentially inefficient, difficult to maintain and difficult to expand.
"search" is a dictionary that's keyed on what the user calls the column and holds the value being searched for.
What I'd like to write is something like this:
results = table.Where(t => GetProperty(t, search.Term) == search.Value);
but I know this won't work.
I'd be prepared to keep the if/switch statement so I can do the appropriate equality test but I'd really like to avoid a big 20 item switch/if statement for each type.

You could use something like this to build an expression:
public Expression<Func<T, bool>> GetMatchExpression<T>(
string propertyName,
object propertyValue)
{
var parameterExp = Expression.Parameter(typeof(T), "t");
var propertyExp = Expression.PropertyOrField(parameterExp, propertyName);
var method = propertyExp.Type.GetMethod("Equals", new[] { propertyExp.Type });
var someValue = Expression.Constant(propertyValue);
var methodExpression = Expression.Call(propertyExp, method, someValue);
return Expression.Lambda<Func<T, bool>>(methodExpression, parameterExp);
}
And you would use it like this:
results = table.Where(GetMatchExpression<Table>(search.Term, search.Value));

Okay, so talks of dynamic LINQ etc is skirting around the underlying issue of the database not being maintainable.
Please note, this is psuedo code taken off other implementations I've used that work.
In EF you can set up data inheritence by using Discriminator fields, so your table would become (please name them something different than Date and Text)...
ID
Date
Text
Discriminator
This Discriminator refers to the type of the object in code, and then gets defined in your DbContext, such as:
protected override void OnModelCreating(ModelBuilder builder)
{
base.OnModelCreating(builder);
builder.Entity<YourTableName>()
.HasDiscriminator<string>("Discriminator")
.HasValue<YourObjectType>("YourObjectTypeName")
.HasValue<YourObjectType2>("YourObjectType2Name")
}
Then, you use object inheritence in your models, such as:
public class YourTableName
{
public int ID { get; set; }
public DateTime Date{ get; set; }
public string Text { get; set; }
}
public class YourObjectType: YourTableName
{
}
public class YourObjectType2: YourTableName
{
}
And then, finally, your data retrieval solution can change pretty nicely, to (the same result is achieved for Text too, this is just an example of how easy the call is now, without dynamics):
public T GetByType<T>(DateTime date) where T:YourTableName
{
return table.OfType<T>().Where(a => a.Date.HasValue && a.Date.Value.Date == ((DateTime)search.Value).Date);
}
Nice and maintainable :)

How to make LINQ execute a (SQL) LIKE range search

I am in big need of help, i have been trying to do this for some time now.
So I have this Query:
Select name from BlaBlaBla
order by
case when name like '9%' then 1 end,
case when name like '8%' then 1 end,
case when name like '7%' then 1 end,
case when name like '6%' then 1 end,
case when name like '5%' then 1 end,
case when name like '4%' then 1 end,
case when name like '3%' then 1 end,
case when name like '2%' then 1 end,
case when name like '1%' then 1 end,
case when name like '0%' then 1 end,
name
And I want to implement it in a new C#, Asp.Net, class, in my Solution, to the Domain Project, so it will be an OrderType Filter, for some function...
for now I have this:
var param = Expression.Parameter(typeof(T), "item");
var paramName = Expression.Property(param, "Name");
var regexMatch = Expression.Constant("^[0-9]");
var startsWithDigit = Expression.Call(typeof(Regex), "IsMatch",
null, paramName);
var lambda = Expression.Lambda<Func<T, bool>>(startsWithDigit,
param);
return namesList.OrderBy(lambda)
.ThenBy(BlaBla1())
.ThenByDescending(BlaBla2())
.ThenByDescending(BlaBla3())
.ThenBy(BlaBla4());
But it tells me, that Expression does not contain "IsMatch" method.
Can you please help me?
Thank you!!!

The problem here is that expressions containing Regex can't be translated to SQL, so even when you'd succeed in building a correct expression, you can't use it in LINQ to a SQL backend. However, SQL's LIKE method also supports range wildcards like [0-9], so the trick is to make your LINQ translate to SQL containing a LIKE statement.
LINQ-to-SQL offers the possibility to use the SQL LIKE statement explicitly:
return namesList.OrderBy(r => SqlMethods.Like(r.Name, "[0-9]%")) ...
This SqlMethods class can only be used in LINQ-to-SQL though. In Entity Framework there are string functions that translate to LIKE implicitly, but none of them enable the range wildcard ([x-y]). In EF a statement like ...
return namesList.OrderBy(r => r.Name.StartsWith("[0-9]")) ...
... would translate to nonsense:
[Name] LIKE '~[0-9]%' ESCAPE '~'
I.e. it vainly looks for names starting with the literal string "[0-9]". So as long as you keep using LINQ-to-SQL SqlMethods.Like is the way to go.
In Entity Framework 6.1.3 (and lower) we have to use a slightly different way to obtain the same result ...
return namesList.OrderBy(r => SqlFunctions.PatIndex("[0-9]%", c.Name) == 1) ...
... because PatIndex in SqlFunctions also supports range pattern matching.
But in Entity Framwork 6.2 we're back on track with LINQ-to-SQL because of the new DbFunctions.Like function:
return namesList.OrderBy(r => DbFunctions.Like(r.Name, "[0-9]%")) ...
Finally, also Entity Framework core has a Like function:
return namesList.OrderBy(r => EF.Functions.Like(r.Name, "[0-9]%")) ...

Below you see a sample for this kind of way to handle cases for your orderings.
static void Main(string[] args)
{
List<Obvious> list = new List<Obvious>();
for (int i = 0; i < 100; i++)
{
list.Add(new Obvious(i.ToString(), i));
}
string name = list[30].name;
switch (name)
{
case "9":
list.OrderBy(o => o.perc)
.ThenByDescending(o => o.name);
break;
default:
list.OrderByDescending(o => o.name)
.ThenBy(o => o.perc);
break;
}
}
public class Obvious
{
public string name { get; set; }
public int perc { get; set; }
public Obvious(string _name, int _perc)
{
this.name = _name;
this.perc = _perc;
}
}

If I was you I wouldn't try using Expressions to solve this issue since it brings in a lot of complexity.
I see that you would like to have a generic method, so it can work with different domain entities, yet you are expecting that each entity has a Name property.
You can solve this in a more simple way by defining interface that contains Name property. Like this:
public static void Main()
{
var test = new List<YourDomainEntity>()
{
new YourDomainEntity() { Name = "1test", OtherProperty = "1"},
new YourDomainEntity() { Name = "2test", OtherProperty = "2" },
new YourDomainEntity() { Name = "2test", OtherProperty = "1" }
};
var k = Foo(test).ToList();
}
public interface INameOrderable
{
string Name { get; set; }
}
public interface IOtherPropertyOrderable
{
string OtherProperty { get; set; }
}
public static IEnumerable<T> Foo<T>(IEnumerable<T> list) where T : INameOrderable, IOtherPropertyOrderable
{
return list.OrderBy(a => a.Name, new NamesDescComparer()).ThenBy(b => b.OtherProperty);
}
public class NamesDescComparer : IComparer<string>
{
public int Compare(string x, string y) => -String.CompareOrdinal(x, y);
}
class YourDomainEntity : INameOrderable, IOtherPropertyOrderable
{
public string OtherProperty { get; set; }
public string Name { get; set; }
}
I believe the method Foo is what you are looking for.
Note the where T : INameOrderable part. It restricts usage of this method to entities that implement INameOrderable interface

Query and map complex objects

My goal is to query and map complex objects with as little overhead as possible. I am working with a large database with lots of related tables. I am trying to use LINQ select and projection to select only the necessary information i need to make the object.
This is the original query I had which was fast and worked great.
List<ClientDTO> clientList = dbClients.Select(client =>
new ClientDTO
{
ID = client.ClientID,
FirstName = client.FirstName,
LastName = client.LastName,
//etc....
Products = client.Products
.Select(prod => new ProductDTO
{
ID = prod.ID,
DateOfTransaction = prod.Date,
//etc...
}).ToList(),
Items = client.Items
.Select(item => new ItemDTO
{
ID = item.ID,
Date = item.Date,
//etc...
}
});
Keep in mind the Client table has over 50 related tables, so this query worked great in that it only selected the fields I needed to make the object.
Now what I needed to do is make mappers for these Objects and try to build the same query statement but using the mappers this time. Here is what I ended up with.
List<ClientDTO> clients = dbClients.ProjectToClientDTO();
Using these Mappers
public static List<ClientDTO> ProjectToClientDTO(this IQueryable<Clients> query)
{
var clientList = query.Select(client => new
{
ID = client.ClientID,
FirstName = client.FirstName,
LastName = client.LastName,
//etc...
Products = client.Products.AsQueryable().ProjectToProductDTO().ToList(),
Items = client.Items.AsQueryable().ProjectToItemDTO().ToList()
}
List<ClientDTO> dtoClientList = new List<ClientDTO>();
foreach (var client in clientList)
{
ClientDTO clientDTO = new ClientDTO();
clientDTO.EncryptedID = EncryptID(client.ID, client.FirstName, client.LastName);
//etc...
clientDTO.Products = client.Products;
clientDTO.Items = client.Items;
}
return dtoClientList;
}
public static IQueryable<ProductDTO> ProjectToProductDTO(this IQueryable<Products> query)
{
return query.Select(prod => new ProductDTO
{
ID = prod.ID,
DateOfTransaction = prod.Date,
//etc...
});
}
public static IQueryable<ItemDTO> ProjectToItemDTO(this IQueryable<Items> query)
{
return query.Select(item => new ItemDTO
{
ID = item.ID,
Date = item.Date,
//etc...
});
}
After trying to run this I get the following error.
LINQ to Entities does not recognize the method 'ProjectToProductDTO(IQueryable[Products])', and this method cannot be translated into a store expression."}
Can I make LINQ invoke these methods to build the query?
Or is there a better way to query and map these objects without grabbing 50+ tables of unnecessary data for hundreds of clients?
UPDATE
User Tuco mentioned that I could try looking into expression trees. After reading up on them for a bit I came up with this.
public static Expression<Func<Product, ProductDTO>> test = prod =>
new ProductDTO()
{
ID= prod.ID,
Date= prod.Date,
//etc...
};
And use it as such.
Products = client.Products.Select(prod => test.Compile()(prod)),
But running this I receive this error.
The LINQ expression node type 'Invoke' is not supported in LINQ to Entities

You are very close with your 2nd approach!
Let's say you define the projection of the product entity to the DTO (the mapper as you call it) like you did:
Expression<Func<Product, ProductDTO>> productProjection = prod => new ProductDTO
{
ID = prod.ID,
DateOfTransaction = prod.Date
// ...
};
and the projection of the client entity to it's DTO like this (slightly simpler, but logically equivalent to what you did):
Expression<Func<Client, ClientDTO>> clientProjection = client => new ClientDTO
{
ID = client.ClientID,
FirstName = client.FirstName,
// ...
Products = client.Products.Select(productProjection.Compile()).ToList(),
// ...
};
The compiler let's you do so, but the queryable will not understand that. However what you have achieved is that the productProjection is somehow contained in the expression tree. All you have to do is some expression manipulation.
If you look at the subtree the compiler builds for the argument to .Select you'll find a MethodCallExpression - the call to .Compile(). It's .Object expression - the thing that is to be compiled - is a MemberExpression accessing a field named productProjection(!) on an ConstantExpression containing an instance of an oddly named compiler generated closure class.
So: Find .Compile() calls and replace them with what would be compiled, ending up with the very expression tree you had in your original version.
I'm maintaining a helper class for expression stuff called Express. (See another answer that deals with .Compile().Invoke(...) for a similar situation).
clientProjection = Express.Uncompile(clientProjection);
var clientList = dbClients.Select(clientProjection).ToList();
Here's the relevant snipped of the Express class.
public static class Express
{
/// <summary>
/// Replace .Compile() calls to lambdas with the lambdas themselves.
/// </summary>
public static Expression<TDelegate> Uncompile<TDelegate>(Expression<TDelegate> lambda)
=> (Expression<TDelegate>)UncompileVisitor.Singleton.Visit(lambda);
/// <summary>
/// Evaluate an expression to a value.
/// </summary>
private static object GetValue(Expression x)
{
switch (x.NodeType)
{
case ExpressionType.Constant:
return ((ConstantExpression)x).Value;
case ExpressionType.MemberAccess:
var xMember = (MemberExpression)x;
var instance = xMember.Expression == null ? null : GetValue(xMember.Expression);
switch (xMember.Member.MemberType)
{
case MemberTypes.Field:
return ((FieldInfo)xMember.Member).GetValue(instance);
case MemberTypes.Property:
return ((PropertyInfo)xMember.Member).GetValue(instance);
default:
throw new Exception(xMember.Member.MemberType + "???");
}
default:
// NOTE: it would be easy to compile and invoke the expression, but it's intentionally not done. Callers can always pre-evaluate and pass a member of a closure.
throw new NotSupportedException("Only constant, field or property supported.");
}
}
private sealed class UncompileVisitor : ExpressionVisitor
{
public static UncompileVisitor Singleton { get; } = new UncompileVisitor();
private UncompileVisitor() { }
protected override Expression VisitMethodCall(MethodCallExpression node)
{
if (node.Method.Name != "Compile" || node.Arguments.Count != 0 || node.Object == null || !typeof(LambdaExpression).IsAssignableFrom(node.Object.Type))
return base.VisitMethodCall(node);
var lambda = (LambdaExpression)GetValue(node.Object);
return lambda;
// alternatively recurse on the lambda if it possibly could contain .Compile()s
// return Visit(lambda); // recurse on the lambda
}
}
}

Use LINQKit to expand user-defined lambda functions into the lambdas needed in the query:
https://github.com/scottksmith95/LINQKit

IQueryable and custom filtering - use Expression<Func<>>?

I face a problem, which is new to me
I have the following entity (I use the fluent nhibernate but it doesn't matter here)
public class SomeEntity
{
public virtual string Name { get; set; }
}
filter classes:
public class FilterOptions
{
public string logic { get; set; } // "and", "or"
public FilterItems[] filters { get; set; }
}
public class FilterItems
{
public string #operator { get; set; }
public string value { get; set; } //string value provided by user
}
the #operator property can have the following values
EndsWith
DoesNotContain
Contains
StartsWith
NotEqual
IsEqualTo
all I wanna do is to do some filter operations basing on the 2 filters:
private IQueryable<SomeEntity> BuildQuery(FilterOptions opts)
{
IQueryable<SomeEntity> query = Session.Query<SomeEntity>();
var firstFilter = opts.filters[0];
var secondFilter = opts.filters[1];
}
because the fact the #operator property can have so many options, I wonder if there's a posibility to have the external method with the swich operator, and use that method inside of the .Where method.
Something like
var query = query.Where(firstSwitchFilterMethod && secondFilterMethod)
pseudo code:
firstSwitchFilterMethod:
if (firstFilter.#operator == "Contains")
return SomeEntity.Name.Contains(firstFilter.value);
and so on...
Any ideas ? I'm thinking about the Expression<Func<>> - is it good direction ? If so, how to use it in my case ?
Or, maybe build my own the extension method for SomeEntity which will be using that filter class ?

You can not make arbitrary function call expression and expect that it can be translated into SQL. But there are some functions, like StartsWith, that can. Here is one example, how You can build Your own expression:
protected IQueryable<T> GetFiltered<T>(IQueryable<T> query, string filterOnProperty, string startsWithString, string endsWithString)
{
LambdaExpression startsWithLambda = (Expression<Func<string, string, bool>>)((x, s) => x.StartsWith(s));
MethodInfo startsWithMI = (startsWithLambda.Body as MethodCallExpression).Method;
LambdaExpression endsWithLambda = (Expression<Func<string, string, bool>>)((x, s) => x.EndsWith(s));
MethodInfo endsWithMI = (endsWithLambda.Body as MethodCallExpression).Method;
ParameterExpression param = Expression.Parameter(typeof(T));
Expression nameProp = Expression.Property(param, filterOnProperty);
Expression filteredOk = Expression.Constant(true);
Expression startsWithStringExpr = Expression.Constant(startsWithString);
Expression startsWithCondition = Expression.Call(nameProp, startsWithMI, startsWithStringExpr);
filteredOk = Expression.AndAlso(filteredOk, startsWithCondition);
Expression endsWithStringExpr = Expression.Constant(endsWithString);
Expression endsWithCondition = Expression.Call(nameProp, endsWithMI, endsWithStringExpr);
filteredOk = Expression.AndAlso(filteredOk, endsWithCondition);
Expression<Func<T, bool>> where = Expression.Lambda<Func<T, bool>>(filteredOk, new ParameterExpression[] { param });
return query.Where(where);
}
usage is simple
DCDataContext dc = new DCDataContext();
var query = dc.testtables.AsQueryable();
query = GetFiltered(query, "name", "aaa", "2");

NHibernate or any other LINQ provider doesn't really care how you build the expression - the only thing that matters is that the final expression only contain constructs that the LINQ provider understands and knows what to do with.
Yes, you can have a method return an Expression<Func<>> and then add that expression to the LINQ query using the Where() method.
However, you cannot ask NHibernate to analyze your compiled code and convert the if statement to SQL. You method will need to analyze the options and return a suitable expression that gets inserted into the full LINQ query.
You can write it in a single method like this:
IQueryable<SomeEntity> query = Session.Query<SomeEntity>();
if (isEqOp)
query = query.Where(e => e.Name == options.Value)
if (isContainsOp)
query = query.Where(e => e.Name.Contains(options.Value))
query.ToList();
Or if you want the filter logic in a separate method as you said you can also split it like this:
public Expression<Func<Entity, bool>> GetExpression(options)
{
if (isEqOp)
return (Entity e) => e.Name == options.Value;
if (isContainsOp)
return (Entity e) => e.Name.Contains(options.Value);
}
{
IQueryable<SomeEntity> query = Session.Query<SomeEntity>();
query = query.Where(GetExpression(options))
...
}

Function that can take different tables and operate on common columns

I have a number of different tables, each with some common columns and some columns unique to each table. I'm using the Entity Framework to operate on these tables, and LINQ queries for filtering, sorting, etc. The code below is a highly simplified example of a situation I'm running into.
var filter = "A";
using (var dc = new MyDataEntities())
{
var rpt1 = dc.Table1.Where(x => x.Name.StartsWith(filter));
var rpt2 = dc.Table2.Where(x => x.Name.StartsWith(filter));
var rpt3 = dc.Table3.Where(x => x.Name.StartsWith(filter));
}
If I decide I want to change the filter from StartsWith to Contains for example, I have to change it in 3 places. If I decide I want to add sorting, I have to add it in 3 places, etc. (As you can guess my actual code has this logic occurring across multiple classes, and a lot more than 3 instances.)
Is there some way to write a function that can take any table and operate on the common columns in that table? So my end result would be something like:
using (var dc = new MyDataEntities())
{
var rpt1 = Filter(dc.Table1);
var rpt2 = Filter(dc.Table2);
var rpt3 = Filter(dc.Table3);
}
That way I could put the logic to filter on the Name column on any of my tables into one Filter() function. Ideas?

Let me define all of the classes involved:
public class Table1
{
public string Name { get; set; }
}
public class Table2
{
public string Name { get; set; }
}
public class Table3
{
public string Name { get; set; }
}
as you can tell there POCO's will be more complex but for this example it should be fine:
public class Example
{
public void Test()
{
var t1 = new List<Table1>();
var t2 = new List<Table2>();
var t3 = new List<Table3>();
var filter = "hello";
Func<string, bool> filterFunc = (x) => x.StartsWith(filter);
var rpt1 = t1.Where(x => filterFunc(x.Name));
var rpt2 = t2.Where(x => filterFunc(x.Name));
var rpt3 = t3.Where(x => filterFunc(x.Name));
}
}
As you can see I've abstracted the filter out into a function delegate
Now a possible better solution, depends on if this really makes sense or not, is to put all of the shared columns into a base class that all of these derive from:
public class TableCommon
{
public string Name { get; set; }
}
public class Table1 : TableCommon
{}
public class Table2 : TableCommon
{}
public class Table3 : TableCommon
{}
public class Example
{
public void Test2()
{
var t1 = new List<Table1>();
var t2 = new List<Table2>();
var t3 = new List<Table3>();
var rpt1 = FilterData(t1);
var rpt2 = FilterData(t2);
var rpt3 = FilterData(t3);
}
public IEnumerable<T> FilterData<T>(IEnumerable<T> data) where T : TableCommon
{
var filter = "hello";
Func<T, bool> pred = (x) => x.Name.StartsWith(filter);
return data.Where(pred);
}
}
What's nice about this is now you can hide away your filter logic, or even have the ability to pass in different filter by making the pred variable a parameter and allowing this function to be a lot more generic.
Now if you are not comfortable with this way using a base class and a Type constraint on FilterData, then you will have to use reflection, I've had to do this for other reasons, and it gets quite messy and unreadable very fast. That or maybe something like dynamic linq which again can be very messy.