Develop Reference

How can I build a linq query at runtime with multiple group by clauses with different types

I'm in a situation where I need to specify queries on a EF-context at runtime. Our consultants configure these queries at a customer site for customer specific situations.
In order to facilitate that I was thinking of using linq to build the queries, based on a criteria-list which the consultants specify in a front-end of some kind (right now, winforms). The consultants basically specify a property from an object, specify the operator and then the value. For example: give me all clients where the [status] [equals] [1].
At the moment I have an expression builder which creates the where clause at runtime, and so far I can manage one group by clause. Where I'm running against the wall here, is when a consultant configures multiple group by clauses of different types (f.e. a string and a datetime property).
For example, I must be able to process this query: select bsn as a, dateofbirth as b from clients where status = 1 group by bsn, dateofbirth (where bsn = string and dateofbirth = datetime).
At the moment, this is the code which "glues" the query together:
public List<ClientV2> ExportClients(List<CriteriaV2> criteriaList)
{
var whereExpression = BuildWhereExpressionChain(criteriaList.Where(c => c.Operator != CriteriaOperatorV2.GROUPBY).ToList());
var groupByExpression = BuildGroupByExpression(criteriaList.Where(c => c.Operator == CriteriaOperatorV2.GROUPBY).ToList());
var sourceClients = _context.Clients.Where(whereExpression).GroupBy(groupByExpression).ToList();
IEnumerable<Client> resultClients = sourceClients.SelectMany(group => group);
return ClientToClientV2.MapList(resultClients.ToList());
}
This is the where-clause builder:
private Expression<Func<Client, bool>> BuildWhereExpressionChain(List<CriteriaV2> criteriaList)
{
var expressionList = new List<Expression<Func<Client, bool>>>();
var paramExp = Expression.Parameter(typeof(Client));
foreach (var crit in criteriaList)
{
var propertyItem = PropertyTranslator.GetPropertyItem(crit.Property);
if (propertyItem == null) throw new InvalidFilterCriteriaException("Property " + crit.Property + " niet toegestaan als filter criterium");
var propInfo = typeof(Client).GetProperty(propertyItem.InternalName);
var left = Expression.Property(paramExp, propInfo);
Expression right;
if (propInfo.PropertyType.IsEnum)
right = Expression.Constant(Enum.ToObject(propInfo.PropertyType, PropertyTranslator.TranslateEnum(propertyItem.Type, crit.Value)));
else if (propInfo.PropertyType == typeof(DateTime) || propInfo.PropertyType == typeof(DateTime?))
right = Expression.Constant(DateTime.Parse(crit.Value), propInfo.PropertyType);
else
right = Expression.Constant(crit.Value, typeof(string));
var exp = BuildExpression(left, right, crit.Operator);
expressionList.Add(Expression.Lambda<Func<Client, bool>>(exp, new ParameterExpression[] { paramExp }));
}
var firstExpression = expressionList.First();
expressionList.Skip(1).ToList().ForEach(ex => { firstExpression = firstExpression.And(ex); });
return firstExpression;
}
And this is the part where I am stuck (it does work for one clause of type string):
private Expression<Func<Client, string>> BuildGroupByExpression(List<CriteriaV2> criteriaList)
{
var expressionList = new List<Expression<Func<Client, string>>>();
var paramExp = Expression.Parameter(typeof(Client));
foreach (var crit in criteriaList)
{
var propertyItem = PropertyTranslator.GetPropertyItem(crit.Property);
if (propertyItem == null) throw new InvalidFilterCriteriaException("Property " + crit.Property + " niet toegestaan als group by criterium");
var propInfo = typeof(Client).GetProperty(propertyItem.InternalName);
var body = Expression.Property(paramExp, propInfo);
var lambda = Expression.Lambda<Func<Client, string>>(body, paramExp);
expressionList.Add(lambda);
}
var firstExpression = expressionList.First();
expressionList.Skip(1).ToList().ForEach(ex => { firstExpression = firstExpression.And(ex); });
return firstExpression;
}
Would it be possible to make the BuildGroupByExpression() in such way that it results in an expression which contains multiple clauses of different types that I can use directly in .GroupBy(expression);?
I'm not that much of an expert on linq, but I have a feeling that what I want could be possible. If I do stupid things here, please point them out and I'll work on it.

I dont think so. At least using the approach where the Expression is built they way you have. At least I ended up building expressions per type.
The main reason is
var lambda = Expression.Lambda<Func<Client, string>>(body, paramExp);
I dont know how you can make this Lambda defintion dynamic or generic.
There is a different approach, and a library that uses a string interpretation at runtime to build expressions. See
Install-Package System.Linq.Dynamic.Library
see also https://msdn.microsoft.com/en-US/vstudio/bb894665.aspx

How to parametrize a query in Entity Framework?

I am new to EF. I have a table with a list of projects. I have found a query in my software that finds all projects .
public Project[] FindAll()
{
var projects = new List<Project>();
using (var db = new ProjetDbConext())
{
var qProjects = from project in db.ProjectSet
where project.CreateDateTime != null
select project;
projects = qProjects.ToList();
}
return projects.ToArray();
}
This seems to be fine , but I am not sure how to parametrize it. I need this because I am implementing a search feature trying to re use some query logic from EF.
This takes a List of tuples . Each tuple basically has an attribute and a list of search terms.
eg. Tuple(FirstName , { Prasaanth ,Bill } ; Tuple( LastName , { Neelakandan , Gates } ;
This means I need to write a select query where I search projects where FirstName is Prasaanth or Bill . If the list has only one term.
eg. Tuple( Company , { Microsoft} ; then i need to search only one where condition in my query.
public Project[] LoadSearchProjects(List<System.Tuple<string, List<string>>> searchTerms)
{
var projects = new List<Project>();
using (var db = new ProjetDbConext())
{
foreach (System.Tuple<string, List<string>> pair in searchTerms)
{
string attribute = pair.Item1;
List<string> terms = pair.Item2;
/// logic here
}
}
return projects.ToArray();
}
I can always write an if condition where I do :
if(attribute.equals("FirstName"){
// query project.FirstName in the where conditon
}
But I have too many attributes to search on.
I know the ADO.NET way of doing this :
mycommands = new SqlCommand(" select projects from from Persons where '"+attibute+"' = some search terms ...
I don't know how to do something like this in my EF query.
1 ) Is there a way EF allows me to do the search on dynamic attributes ? or parametrize using '"+attribute+"' ??
2) Is there a better data structure I could use to simplify my structure instead of using List<Tuple<string, List<string>> ?
3) I was recommend to use 3rd party LINQKit or dynamic linq but am not sure how to integrate that to EF querying.
My apologies if much of this sounds like collegeboy code. Please let me know if any additional details needed.
Regards,
Prasaanth
UPDATE :
Working method as per Andriy's answer. My question here is this doesnt work if any particular entry in my database say Name is Null.
private static Expression<Func<TEntity, bool>> BuildStringFilter<TEntity, TProp>(
Tuple<string, List<string>> filter)
{
// entity is the Project table
var entity = Expression.Parameter(typeof (TEntity));
var prop = Expression.Property(entity, filter.Item1);
//check if contains returns true
var body = filter.Item2
.Select(v => Expression.Equal(Expression.Call(prop,
typeof (String).GetMethod("Contains"),
new Expression[] { Expression.Constant(v) }), Expression.Constant(true)))
.Aggregate(Expression.Or);
var result = (Expression<Func<TEntity, bool>>) Expression.Lambda(body, entity);
return result;
}
Any way I can modify the expression so that the Contains method :
prop,
typeof (String).GetMethod("Contains"),
new Expression[] { Expression.Constant(v)
works if the value of the attribute (prop) is null ?

You can build filter expression using snippet:
public static Expression<Func<TEntity, bool>> BuildFilter<TEntity, TProp>(
KeyValuePair<string, IEnumerable<TProp>> filter)
{
var entity = Expression.Parameter(typeof(TEntity));
var prop = Expression.Property(entity, filter.Key);
var body = filter.Value
.Select(v => Expression.Equal(prop, Expression.Constant(v)))
.Aggregate((curr, next) => Expression.Or(curr, next));
var result = (Expression<Func<TEntity, bool>>)Expression.Lambda(body, entity);
return result;
}
And call it like:
var filter = new KeyValuePair<string, IEnumerable<string>> (
"FirstName",
new [] {"Alice", "Bob"}
);
var predicate = BuildFilter<Item, string>(filter);
var result = ctx.Items.Where(predicate);
Also, see How to: Use Expression Trees to Build Dynamic Queries.

I'm spitballing here, but I think something like this would be simpler:
public Project[] Find(Expression<Func<Project, bool> filter = null)
{
using (var db = new ProjetDbConext())
{
var query = db.ProjectSet.Where(p => p.CreateDateTime != null);
if(filter != null)
query = query.Where(filter);
return query.ToArray();
}
}
Use it like:
var projects = repo.Find(p => p.id > 100);

Construct LINQ query using variables in asp.net WebAPI

I am trying to build a method in my asp.net WebAPI to grab data based on the arguments passed on the method. The method is used to perform a search on restaurant data. I have a variable called 'type' that determines the type of data search performed. The second variable 'keyword' is the keyword searched by the user. The WHERE condition in my LINQ query depends on the type and needs to be dynamic, so I have used a separate variable outside the LINQ query to define the condition. I have tried assigning this variable to my WHERE statement on the LINQ query but it doesn't seem to work. Can someone help with it please? I have been stuck on this for a few days now
public IQueryable<RestaurantView> GetRestaurantsForSearch(string keyword, int type, string location)
{
//
var condition = "";
if(type == 1)
{
condition = "x.RestaurantName.Contains(keyword)";
} else if(type == 2){
condition = "x.Cuisine.Equals(keyword)";
}
else {
condition = "x.Rating.Equals(keyword)";
}
var query = from x in db.Restaurants
join y in db.Cuisine on x.RestaurantCuisine equals y.CuisineID
where condition
select new RestaurantView
{
RestaurantID = x.RestaurantID,
RestaurantName = x.RestaurantName,
RestaurantCuisine = y.CuisineName,
RestaurantDecription = x.RestaurantDecription
};
return query;
}

Try this:
Predicate<Restaurant> pred;
if (type == 1) pred = x => x.RestaurantName.Contains(keyword);
else if (type == 2) pred = x => x.Cuisine.Equals(keyword);
else pred = x => x.Rating.Equals(keyword);
var query = from x in db.Restaurants
join y in db.Cuisine on x.RestaurantCuisine equals y.CuisineID
where pred(x)
select new RestaurantView
{
RestaurantID = x.RestaurantID,
RestaurantName = x.RestaurantName,
RestaurantCuisine = y.CuisineName,
RestaurantDecription = x.RestaurantDecription
};
return query;

You need to look a dynamic linq library i think then you can execute string statements inside your linq
http://weblogs.asp.net/scottgu/archive/2008/01/07/dynamic-linq-part-1-using-the-linq-dynamic-query-library.aspx
or you can execute direct query
http://msdn.microsoft.com/en-us/library/system.data.linq.datacontext.executequery.aspx

If you are ok with dropping your comprehensive LINQ query in favour of the extension method syntax, it's pretty simple (I'm on a netbook without VS, so I apologize that this is untested but should give you the idea):
var query = db.Restaurants
.Include("Cuisine")
if(type == 1)
{
query= query.Where(x => x.RestaurantName.Contains(keyword));
}
else if(type == 2)
{
query = query.Where(x => x.Cuisine == keyword);
}
else {
query = query.Where(x => x.Rating == keyword);
}
This builds out your expression tree differently based on your logic checks, which will result in a different SQL query being generated based on the value of type.
I notice that in your join, Cuisine appears to be an Entity, but in your logic checks, you attempt to filter by comparing Cuisine to a string so I think there is some disconnect.

var query = from x in db.Restaurants
join y in db.Cuisine on x.RestaurantCuisine equals y.CuisineID
where condition
select new RestaurantView
{
RestaurantID = x.RestaurantID,
RestaurantName = x.RestaurantName,
RestaurantCuisine = y.CuisineName,
RestaurantDecription = x.RestaurantDecription
};
return query;
}
how to get the return query value in client side to assign for grid view binding

LINQ "'s' is not in scope" when creating where clause dynamically

So, I'm getting the error that's in the title. I'll jump straight into the relevant code. First, the manual creation of the where statement (which does work) and then the dynamic creation which does not work. Both create the same queryExpression.
First, the queryExpression that is generated with both methods:
{Table(Products).Select
(s => new SalesData() {
ProductID = s.ProductID,
StoreName = s.StoreName,
Count = s.Count
}).Where(s => (s.Count > 500))}
Now, the manual method that does work:
IQueryable<SalesData> data = ( from s in sales.Products
select new SalesData
{
ProductID = s.ProductID,
StoreName = s.StoreName,
Count = s.Count
} ).AsQueryable<SalesData>();
data = data.Where(s => s.Count > 500);
this.dataGridView1.DataSource = null;
this.dataGridView1.DataSource = (from d in data
select new
{
d.ProductID,
d.StoreName,
d.Count
} );
This works as expected; my DataGridView is populated with the data.
Now, to create the where clause dynamically:
IQueryable<SalesData> data = ( from s in sales.Products
select new SalesData
{
ProductID = s.ProductID,
StoreName = s.StoreName,
Count = s.Count
} ).AsQueryable<SalesData>();
if (this.filter.Predicate != null)
{
Expression<Func<SalesData, bool>> lambda =
Expression.Lambda<Func<SalesData, bool>>(this.filter.Predicate,
new ParameterExpression[] { this.filter.PE });
data = data.Where(lambda);
}
this.dataGridView1.DataSource = null;
this.dataGridView1.DataSource = (from d in data <---- fails here
select new
{
d.ProductID,
d.StoreName,
d.Count
} );
The only thing that is different is that in the second snippet I'm creating the lambda expression dynamically. I've noted where it fails in the second snippet.
Using the debugger I can see the queryExpression for data and it is the same with both methods, so I don't think my problem is with my actual expression creation. If that code is needed I can post it here.
My question is, what am I doing wrong and how do I fix it?
Edit:
Here is the function that gives filter.Predicate it's value:
public static Expression GetPredicate(List<FilterItem> itemList, ParameterExpression pe)
{
List<Expression> expressions = new List<Expression>();
List<string> combiners = new List<string>();
foreach (FilterItem item in itemList)
{
Expression left = Expression.PropertyOrField(pe, item.Field);
Expression right = Expression.Constant(Convert.ToInt32(item.Value), typeof(int));
expressions.Add(Expression.GreaterThan(left, right));
combiners.Add(item.Combiner);
}
int expressionCount = expressions.Count();
Expression predicateBody = expressions[0];
if (expressionCount > 1)
{
for (int x = 1; x <= expressionCount; x++)
{
switch (combiners[x - 1])
{
case "AND":
predicateBody = Expression.And(predicateBody, expressions[x]);
break;
case "OR":
predicateBody = Expression.Or(predicateBody, expressions[x]);
break;
default:
break;
}
}
}
return predicateBody;
}
Inside filter I have:
this.Predicate = BuildPredicate.GetPredicate(this.filterList, this.PE);
this.PE is:
public ParameterExpression PE
{
get
{
return Expression.Parameter(typeof(SalesData), "s");
}
}

From the father of C#, Anders Hejlsberg:
Parameters are referenced in expressions through their object identity, not by comparing their names. In fact, from an expression tree's point of view, the name of a parameter is purely informational. The reason for this design is the same reason that types are referenced through their System.Type objects and not their names--expression trees are fully bound and are not in the business of implementing name lookup rules (which may differ from language to language).
From MSDN Forums
The short and long of it is that filter.Predicate has a value of {(s.Count > 500)} but this doesn't mean that s has meaning here by sharing a name with the initial LINQ expression.

I was able to solve my problem thanks to the dialog on this page and using Predicate Builder.
A better method for what I'm trying to do can be found at Scott Gu's site. This is better for my purposes because I need to determine the field and the value dynamically, plus allow for grouping (easier with this method).

Dynamic query in LINQ

How do I write a dynamic query for Linq, if I have say Customer class which holds the fields:
string name
string address
int phoneno
I have to query based on information given similar to
query = string.Empty;
if(!string.IsNullorEmpty(name))
{
query += "#name = name";
}
if(!string.IsNullorEmpty(address))
{
query += "#address = address";
}
if(!string.IsNullorEmpty(phoneno))
{
query += "#phoneno = phoneno";
}
var result = from condition in customer
where(query)
select condition;
Edit #1:
the items are changeable at run time like
private Customer[] GetCustomers(Dictionary<string,string> attributes)
{
here the attribute may be, name alone, or name and address, or name address and phoneno
foreach(string field in attributes.key)
{
query += field == attributes[key];
}
Customers[] =ExecuteQuery(query);
}
Is this kind of query supported by LINQ?
Edit #2:
Hi Mouk,
As I am new to C#, I am still struggling, this is not working for me.
var query = _ConfigFile.ConnectionMasterSection;
for(int i = 0; i < filter.count; i++)
{
query = result.Where(p => typeof(ConnectionMaster).GetProperty(filter[i].Attribute).Name == filter[i].Value);
}
This yeilds Empty, where as i used this
var query = _ConfigFile.ConnectionMasterSection;
//Hard coded
res.Where(q => q.category == filter[0].Value);
And it worked as I expected.
Hi Bryan Watts,
I tried your code also and I getting this error: "Lambda Parameter not in scope".
for(int i = 0; i < filter.count; i++)
{
Field item = filter[i];
MemberExpression param = Expression.MakeMemberAccess(Expression.Parameter(typeof(Connection), "p"), typeof(Connection).GetProperty(item.Attribute));
MemberExpression constant = Expression.MakeMemberAccess(Expression.Constant(item), typeof(Field).GetProperty("Value"));
}
try
{
var myquery = Queryable.Where(coll, Expression.Lambda<Func<Connection, bool>>(
Expression.Equal(param, constant), Expression.Parameter(typeof(Connection),"p")));
}
What is the mistake here?

Check out this http://www.albahari.com/nutshell/predicatebuilder.aspx, it allows for strongly typed predicate building, it can be really nice. If you want actually dynamic string built predicates than you can use the LINQ Dynamic Query Library provided by ScottGu.
Both will accomplish what you want although I would recommend the first option before the second.
Allowing you to do:
var predicate = PredicateBuilder.True<MyLinqType>();
if(!string.IsNullOrEmpty(name))
predicate = predicate.And(p => p.name == name);
...
var myResults = Context.MyLinTypeQueryTable.Where(predicate);
And more.

Here you go:
var result = from customer in Customers
where string.IsNullOrEmpty(phoneNo) || customer.PhoneNo == phoneNo
where string.IsNullOrEmpty(address) || customer.Address == address
select customer;
If you're concerned that this generate the optimal SQL query underneath, as always you should attach a SQL Query Analyzer and check. But I believe the expression parser in Linq To Sql will collapse down the where clauses as appropriate based on the value of the arguments.

You can use the fluent interface and add a new Where clause fpr each condition. Something like:
var result = from cus in customers select cus;
if(!string.IsNullOrEmpty(name))
result= result.Where(p => p.Name == name);
EDIT upon hte comment:
if you are querying over a collection in memory, you could retrieve the properties using reflection.
private Customer[] GetCustomers(Dictionary<string,string> attributes)
{
var result = from cus in customers select cus;
foreach(string key in attributes.Keys)
result= result.Where(p => GetProperty(p, key )== attributes[key]);
return result.ToList();
}
Supposing GetProperty retrieve the property by reflection.
Using Linq2Sql this method will result in retrieving all record an then iterating over them using reflection.

I've had good experience with Dynamic LINQ.
I used it for a rich HTML table that could be filtered and sorted server side. The server receives a request containing a request parameter where the key is the name of the property (for example 'Lastname') and the value is the value that the property needs to be sorted on (for example 'Smith'). Using that information I built a query string that I passed to the Dynamic LINQ's Where method.
Roughly, you could think of something like the following:
public static IQueryable<T> Filter<T>(this IQueryable<T> query, Dictionary<string, string> dictionary)
{
Type t = typeof(T);
StringBuilder sb = new StringBuilder();
PropertyInfo[] properties = t.GetProperties();
foreach(string key in dictionary.Keys)
{
PropertyInfo property = properties.Where(p => p.Name == key).SingleOrDefault();
if(property != null)
{
if (sb.Length > 0) sb.Append(" && ");
string value = dictionary[key];
sb.Append(string.Format(#"{0}.ToString().Contains(""{1}"")", key, value));
}
}
if (sb.Length > 0)
return query.Where(sb.ToString());
else
return query;
}
The code is out of the top of my head and thus untested.
Of course, this is the most basic version: it does a simple string comparison. If you want to have numerical comparison (meaning you want for example the User where UserID is exactly 100, not where the UserID.ToString().Contains("100")), or query nested Properties (Customer.Company.CompanyAddress for example), or query Collections this gets more complicated. You should also think about security: while Dynamic LINQ is not vulnerable to SQL injection, you shouldn't let it blindly parse all user input.

It sounds like you need to dynamically compose queries.
See my answer to this question.
It explains how queries against an IQueryable<T> are composed by the compiler, and what you can do to add dynamic elements.
Edit
Here is an example of how you would dynamically build a set of Where conditions on top of an IQueryable<Customer>:
// This method ANDs equality expressions for each property, like so:
//
// customers.Where(c => c.Property1 == value1 && c.Property2 == value2 && ...);
private IQueryable<Customer> FilterQuery(IQueryable<Customer> customers, IDictionary<string, string> filter)
{
var parameter = Expression.Parameter(typeof(Customer), "c");
Expression filterExpression = null;
foreach(var filterItem in filter)
{
var property = typeof(Customer).GetProperty(filterItem.Key);
var propertyAccess = Expression.MakeMemberAccess(parameter, property);
var equality = Expression.Equal(propertyAccess, Expression.Constant(filterItem.Value));
if(filterExpression == null)
{
filterExpression = equality;
}
else
{
filterExpression = Expression.And(filterExpression, equality);
}
}
if(filterExpression != null)
{
var whereBody = Expression.Lambda<Func<Customer, bool>>(filterExpression, parameter);
customers = customers.Where(whereBody);
}
return customers;
}