Develop Reference

Parsing ODataQueryOptions<type> without EF/Nhibernate

I have a project with a large codebase that uses an in-house data access layer to work with the database. However, we want to support OData access to the system. I'm quite comfortable with expression trees in C#. How do I get at something I can parse here in order to get the structure of their actual query?
Is there a way to get an AST out of this thing that I can turn into sql code?

Essentially, you need to implement you own Query Provider which known how to translate the expression tree to an underlying query.
A simplified version of a controller method would be:
[ODataRoute("foo")]
public List<Foo> GetFoo(ODataQueryOptions<Foo> queryOptions)
{
var queryAllFoo = _myQueryProvider.QueryAll<Foo>();
var modifiedQuery = queryOptions.ApplyTo(queryAllFoo);
return modifiedQuery.ToList();
}
However!
This is not trivial, it took me about 1 month to implement custom OData query processing
You need to build the EDM model, so the WebApi OData can process and build right expression trees
It might involve reflection, creation of types at runtime in a dynamic assembly (for the projection), compiling lambda expressions for the best performance
WebAPI OData component has some limitations, so if you want to get relations working, you need to spend much more extra time, so in our case we did some custom query string transformation (before processing) and injecting joins into expression trees when needed
There are too many details to explain in one answer, it's a long way..
Good luck!

You can use ODataQueryOptions<T> to get abstract syntax trees for the $filter and $orderby query options. ($skip and $top are also available as parsed integers.) Since you don't need/want LINQ support, you could then simply pass the ASTs to a repository method, which would then visit the ASTs to build up the appropriate SQL stored proc invocation. You will not call ODataQueryOptions.ApplyTo. Here's a sketch:
public IEnumerable<Thing> Get(ODataQueryOptions<Thing> opts)
{
var filter = opts.Filter.FilterClause.Expression;
var ordering = opts.OrderBy.OrderByClause.Expression;
var skip = opts.Skip.Value;
var top = opts.Top.Value;
return this.Repository.GetThings(key, filter, ordering, skip, top);
}
Note that filter and ordering in the above are instances of Microsoft.OData.Core.UriParser.Semantic.SingleValueNode. That class has a convenient Accept<T> method, but you probably do not want your repository to depend on that class directly. That is, you should probably use a helper to produce an intermediate form that is independent of Microsoft's OData implementation.
If this is a common pattern, consider using parameter binding so you can get the various query options directly from the controller method's parameter list.

Dynamic Linq on Dapper dynamic collections - possible?

We are investigating using LinQ to query an internal dynamic collection created by Dapper. The question is:
How to execute dynamic LinQ against the collection using Scott Guthrie dynamic linq (or some other technology if possible)? (http://weblogs.asp.net/scottgu/dynamic-linq-part-1-using-the-linq-dynamic-query-library)
This is what we want to do (much simplified):
Use Dapper to return a dynamic collection (here called rows):
rows = conn.Query(“select ACCOUNT, UNIT, AMOUNT from myTable”);
If we use a “static” LinQ query there is no problem. So this works fine:
var result = rows.Where(w => w.AMOUNT > 0);
But we would like to write something similar to this using dynamic Linq:
var result = rows.Where("AMOUNT > 0");
But we can’t get this to work.
The error we get is:
No property or field ‘AMOUNT’ exists in type ‘Object’
(We have tried a lot of other syntax also – but cant get it to work)
Please note: We do not want to use dynamic SQL when Dapper requests data from the database (that is easy). We want to execute many small dynamic Linq statements on the collection that the Dapper query returns.
Can it be that ScottGu dynamic Linq only works with ‘LinQ to SQL’?
Is there some other alternative approach to achieve the same thing?
(Performance is a key issue)
/Erik

conn.Query("...")
returns an IEnumerable<dynamic>. However, "dynamic LINQ" pre-dates dynamic, and presumably nobody has updated it to work with dynamic; it could certainly be done, but it is work (and isn't trivial). Options:
use Query<T> for some T
make the required changes to "dynamic LINQ", and preferably make those changes available to the wider community

I suspect that conn.Query(“select ACCOUNT, UNIT, AMOUNT from myTable”); returns IEnumerable<object>. In order for DLinq to work, you need to have IEnumerable<TheActualType>.
You can try this:
conn.Query<dynamic>("yourQueryString")
.ToList()
//.ToAnonymousList()
.Where("AMOUNT > 0");
If that doesn't work, you could try and use ToAnonymousList, that tries to return the IList<TheActualType.

Where clause with multiple unknown conditions

I am currently developing a Staff management system for my company. The fields may vary and change time to time, so I have an interface for each field like this:
public interface IStaffInfoField
{
// ...
IQueryable<Staff> Filter(IQueryable<Staff> pList, string pAdditionalData);
// ...
}
For each field, I implement the Filter method, for example with Name:
class NameStaffInfoField : BaseStaffInfoField
{
// ...
public override IQueryable<Staff> Filter(IQueryable<Staff> pList, string pAdditionalData)
{
return pList.Where(q => q.Name.Contains(pAdditionalData));
}
// ...
}
Now the users want to search with multiple conditions, it's easy, I just iterate through the list and call Filter. However, they also want a OR condition (say, staff which have name A, OR name B, AND Department Name C, OR Age 30). Note: Users are end-users and they input the search queries through comboboxes and textboxes.
Can I modify my pattern or the lambda expression somehow to achieve that? Because throughout the progress, I don't save the original list to Union it for OR condition. I think it will be slow if I save the expression and Union it for OR condition.
The only solution I can think of now is to add a method to interface that require raw SQL WHERE statement. But my entire program hasn't used pure SQL query yet, is it bad to use it now?

Since your method returns IQueryable, clients already can use it for arbitrarily complicated queries.
IQueryable<Staff> result = xxx.Filter( .... );
result = result.Where( ... );
if ( ... )
result = result.Where( s => ( s.Age > 30 || s.Salary < 1 ) && s.Whatever == "something" );
The IQueryable is very flexible. The query tree is evaluated and translated to sql when you start to enumerate results.
I only wonder why would you need the interface at all?! Since your Filter method expects the IQueryable, this means that client already has the IQueryable! Why would she call your Filter method then if she can already apply arbitrarily complicated query operators on her own?
Edit:
After your additional explanation, if I were you I would create a simple interface to let users create their own query trees containing OR and AND clauses and create a simple function that would translate the user query tree to linq expression tree.
In other words, do not let end users work at linq query tree level, this is too abstract and also too dangerous to let users touch such low level layer of your code. But abstract trees manually translated to linq trees sound safe and easy.

You can download Albahari's LINQKit. It contains a PredicateBuilder that allows you, among other useful things, to concatenate LINQ expressions with OR in a dynamic way.
var predicate = PredicateBuilder.False<Staff>();
predicate = predicate.Or(s => s.Name.Contains(data));
predicate = predicate.Or(s => s.Age > 30);
return dataContext.Staff.Where(predicate);
You can also download the source code and see how it is implemented.

If your users are end users, and they enter criteria through a UI, you may want to look at a UI control that supports IQueryable. Telerik has a large number of pre-baked controls. In most cases end users interact with a grid and they apply filters to the columns. There are several other vendors that do the same thing.
A second option, if you want to make your life difficult, you could take the input text that the user supplies, parse it into a expression tree and then map that expression tree to a IQueryable. If you are not familiar with parsers this task will be fairly difficult to implement.

Transforming a filter string into a C# delegate

I have a set of classes which I am using for a Data Access Layer for some clients. As part of the data access I am allowing a set of filters to be sent in this format:
"{Member[.Member....]}{Operator}{Value}"
I would like to turn these strings into delegates for use in a LINQ query like this:
.Where([delegate returned by a factory])
Here is a more concrete example:
IEnumerable<Parent> parents = GetSomeParents();
string filter = "Child.Id=5";
var expression = FilterFactory<Parent>.GetExpression(filter);
parents = parents.Where(expression);
expression would contain the delegate: parent => parent.Child.Id == 5
Is there a way using reflection to construct the FilterFactory in a Generic way to handle any member path I send in? Paths with indexing aren't required, but would be nice.

Yes absolutely! This is a really fun thing to do too.
One way you can do this is to use the LINQ Dynamic Query Library and get the expression compiler they have in there. I also have something very similar in my project MetaSharp.
But you could also do it yourself if the syntax or features don't quite match what you're wanting. The general idea is that you need to parse the string and build up an Expression tree that represents what you are parsing. In .NET the expression tree objects can be found in System.Linq.Expressions. Once you have your tree you can call Compile() on it and it will be dynamically compiled into a delegate right then. Try reading about the State Machine and Visitor patterns to figure out the best way to parse an arbitrary expresssion like you have above.
PS I would not recommend using regular expressions!

Dynamic "WHERE" like queries on memory objects

What would be the best approach to allow users to define a WHERE-like constraints on objects which are defined like this:
Collection<object[]> data
Collection<string> columnNames
where object[] is a single row.
I was thinking about dynamically creating a strong-typed wrapper and just using Dynamic LINQ but maybe there is a simpler solution?
DataSet's are not really an option since the collections are rather huge (40,000+ records) and I don't want to create DataTable and populate it every time I run a query.

What kind of queries do you need to run? If it's just equality, that's relatively easy:
public static IEnumerable<object[]> WhereEqual(
this IEnumerable<object[]> source,
Collection<string> columnNames,
string column,
object value)
{
int columnIndex = columnNames.IndexOf(column);
if (columnIndex == -1)
{
throw new ArgumentException();
}
return source.Where(row => Object.Equals(row[columnIndex], value);
}
If you need something more complicated, please give us an example of what you'd like to be able to write.

If I get your point : you'd like to support users writting the where clause externally - I mean users are real users and not developers so you seek solution for the uicontrol, code where condition bridge. I just though this because you mentioned dlinq.
So if I'm correct what you want to do is really :
give the user the ability to use column names
give the ability to describe a bool function (which will serve as where criteria)
compose the query dynamically and run
For this task let me propose : Rules from the System.Workflow.Activities.Rules namespace. For rules there're several designers available not to mention the ones shipped with Visual Studio (for the web that's another question, but there're several ones for that too).I'd start with Rules without workflow then examine examples from msdn. It's a very flexible and customizable engine.
One other thing: LINQ has connection to this problem as a function returning IQueryable can defer query execution, you can previously define a query and in another part of the code one can extend the returned queryable based on the user's condition (which then can be sticked with extension methods).

When just using object, LINQ isn't really going to help you very much... is it worth the pain? And Dynamic LINQ is certainly overkill. What is the expected way of using this? I can think of a few ways of adding basic Where operations.... but I'm not sure how helpful it would be.

How about embedding something like IronPython in your project? We use that to allow users to define their own expressions (filters and otherwise) inside a sandbox.

I'm thinking about something like this:
((col1 = "abc") or (col2 = "xyz")) and (col3 = "123")
Ultimately it would be nice to have support for LIKE operator with % wildcard.

Thank you all guys - I've finally found it. It's called NQuery and it's available from CodePlex. In its documentation there is even an example which contains a binding to my very structure - list of column names + list of object[]. Plus fully functional SQL query engine.
Just perfect.