I have been trying to pass in a dynamic list of Expressions to a MongoDB C# Driver query using Linq ... This method works for me with regular Linq queries against an ORM, for example, but results in an error when applied to a MongoDB query ... (FYI: I am also using LinqKit's PredicateBuilder)
//
// I create a List of Expressions which I can then add individual predicates to on an
// "as-needed" basis.
var filters = new List<Expression<Func<Session, Boolean>>>();
//
// If the Region DropDownList returns a value then add an expression to match it.
// (the WebFormsService is a home built service for extracting data from the various
// WebForms Server Controls... in case you're wondering how it fits in)
if (!String.IsNullOrEmpty(WebFormsService.GetControlValueAsString(this.ddlRegion)))
{
String region = WebFormsService.GetControlValueAsString(this.ddlRegion).ToLower();
filters.Add(e => e.Region.ToLower() == region);
}
//
// If the StartDate has been specified then add an expression to match it.
if (this.StartDate.HasValue)
{
Int64 startTicks = this.StartDate.Value.Ticks;
filters.Add(e => e.StartTimestampTicks >= startTicks);
}
//
// If the EndDate has been specified then add an expression to match it.
if (this.EndDate.HasValue)
{
Int64 endTicks = this.EndDate.Value.Ticks;
filters.Add(e => e.StartTimestampTicks <= endTicks);
}
//
// Pass the Expression list to the method that executes the query
var data = SessionMsgsDbSvc.GetSessionMsgs(filters);
The GetSessionMsgs() method is defined in a Data services class ...
public class SessionMsgsDbSvc
{
public static List<LocationOwnerSessions> GetSessionMsgs(List<Expression<Func<Session, Boolean>>> values)
{
//
// Using the LinqKit PredicateBuilder I simply add the provided expressions
// into a single "AND" expression ...
var predicate = PredicateBuilder.True<Session>();
foreach (var value in values)
{
predicate = predicate.And(value);
}
//
// ... and apply it as I would to any Linq query, in the Where clause.
// Additionally, using the Select clause I project the results into a
// pre-defined data transfer object (DTO) and only the DISTINCT DTOs are returned
var query = ApplCoreMsgDbCtx.Sessions.AsQueryable()
.Where(predicate)
.Select(e => new LocationOwnerSessions
{
AssetNumber = e.AssetNumber,
Owner = e.LocationOwner,
Region = e.Region
})
.Distinct();
var data = query.ToList();
return data;
}
}
Using the LinqKit PredicateBuilder I simply add the provided expressions into a single "AND" expression ... and apply it as I would to any Linq query, in the Where() clause. Additionally, using the Select() clause I project the results into a pre-defined data transfer object (DTO) and only the DISTINCT DTOs are returned.
This technique typically works when I an going against my Telerik ORM Context Entity collections ... but when I run this against the Mongo Document Collection I get the following error ...
Unsupported filter: Invoke(e => (e.Region.ToLower() == "central"),
{document})
There is certainly something going on beneath the covers that I am unclear on. In the C# Driver for MongoDB documentation I found the following NOTE ...
"When projecting scalars, the driver will wrap the scalar into a
document with a generated field name because MongoDB requires that
output from an aggregation pipeline be documents"
But honestly I am not sure what that neccessarily means or if it's related to this problem or not. The appearence of "{document}" in the error suggests that it might be relevant though.
Any additional thoughts or insight would be greatly appreciated though. Been stuck on this for the better part of 2 days now ...
I did find this post but so far am not sure how the accepted solution is much different than what I have done.
I'm coming back to revisit this after 4 years because while my original supposition did work it worked the wrong way which was it was pulling back all the records from Mongo and then filtering them in memory and to compound matters it was making a synchronous call into the database which is always a bad idea.
The magic happens in LinqKit's expand extension method
That flattens the invocation expression tree into something the Mongo driver can understand and thus act upon.
.Where(predicate.Expand())
Related
First, let me say that I've researched this problem and read the following stack overflow articles, but none of them really address this situation.
How can I use Linq to join between objects and entities?
inner join in linq to entities
Situation
I have two classes
public class Section{
public string SchoolId{get;set;}
public string CourseId{get;set;}
public string SectionId{get;set;}
}
public class RelatedItem{
public string SchoolId{get;set;}
public string CourseId{get;set;}
public string SectionId{get;set;}
//..
}
I have an array of Section coming from one source and is an actual collection of Objects.
RelatedItem I'm getting via a LINQ to Entities call against a DbContext.
My goal is to get all of the RelatedItems based on the Sections I have from the other source.
I'm writing a query like this
Section[] mySections = GetSections(); //Third Party Source
IQueryable<RelatedItem> relatedItems = DbContext.RelatedItems
.Where(r=>
mySections.Any(s=> s.SchoolId == r.SchoolId &&
s.CourseId == r.CourseId &&
s.SectionId == r.SectionId)
);
Problem
At runtime, I receive the following error
Unable to create a constant value of type
'ProjectNamespace.Section'. Only primitive types or
enumeration types are supported in this context.
I found a work around, but it involves doing the following, but it doesn't take advantage of any of my table indexes.
var sectionIds = sections.Select(s=>string.Concat(s.SchoolId, "|",s.CourseId, "|",s.SectionId));
IQueryable<RelatedItem> relatedItems = DbContext.RelatedItems
.Where(r=>
sectionIds.Contains(string.Concat(r.SchoolId, "|",r.CourseId, "|",r.SectionId))
);
This block of code works, and currently is pretty fast (but this is dev, and my record count is small). Aside from converting my related items to a collection in memory, does anyone have any other suggestions?
Try using Contains instead:
Section[] mySections = GetSections(); //Third Party Source
IQueryable<RelatedItem> relatedItems = DbContext.RelatedItems.Where(r=>
mySections.Select(s => s.SchoolId).Contains(r.SchoolId) &&
mySections.Select(s => s.CourseId).Contains(r.CourseId) &&
mySections.Select(s => s.SectionId).Contains(r.SectionId)
);
Contains should translate to WHERE IN clauses in SQL.
This won't work if using .NET 3.5 and LINQ to Entities, as it wasn't implemented in that version.
Proper way to solve this is to implement IEquitable. Here is an example on how to do it Does LINQ to Entities support IEquatable in a 'where' clause predicate?
One tip when implementing Equals() and GetHashCode() do not call any .NET methods (like getType()) only compare primitives SchoolId, CourseId, SectionId, it should get converted to expression tree and work just fine.
I found this code below in a file called Filter.cs in a project created with Microsoft App Studio. Although I am a veteran C# programmer, I'm short on experience with LINQ predicate expression builders. I can tell that the code below it is "meta-logic" for flexibly building a query given a list of filter predicates containing type field info and a set of data values to inject into the sub-expressions. What I can't figure out is how the "expression" variable in the following statement:
query = query.Where(expression).AsQueryable()"
.. is concatenating the per-field expressions into a more complex query expression that is finally executed at the end of the code to create the ObservableCollection result. If it was "query +=" I could infer a chaining action like an Event handler field, but as a straight assignment statement it baffles me since I would expect it to replace the last value the expression variable got from the last loop iteration, thereby resetting it in the process and losing its previous value(s). What is going on here?
public class Filter<T>
{
public static ObservableCollection<T> FilterCollection(
FilterSpecification filter, IEnumerable<T> data)
{
IQueryable<T> query = data.AsQueryable();
foreach (var predicate in filter.Predicates)
{
Func<T, bool> expression;
var predicateAux = predicate;
switch (predicate.Operator)
{
case ColumnOperatorEnum.Contains:
expression = x => predicateAux.GetFieldValue(x).ToLower().Contains(predicateAux.Value.ToString().ToLower());
break;
case ColumnOperatorEnum.StartsWith:
expression = x => predicateAux.GetFieldValue(x).ToLower().StartsWith(predicateAux.Value.ToString().ToLower());
break;
case ColumnOperatorEnum.GreaterThan:
expression = x => String.Compare(predicateAux.GetFieldValue(x).ToLower(), predicateAux.Value.ToString().ToLower(), StringComparison.Ordinal) > 0;
break;
case ColumnOperatorEnum.LessThan:
expression = x => String.Compare(predicateAux.GetFieldValue(x).ToLower(), predicateAux.Value.ToString().ToLower(), StringComparison.Ordinal) < 0;
break;
case ColumnOperatorEnum.NotEquals:
expression = x => !predicateAux.GetFieldValue(x).Equals(predicateAux.Value.ToString(), StringComparison.InvariantCultureIgnoreCase);
break;
default:
expression = x => predicateAux.GetFieldValue(x).Equals(predicateAux.Value.ToString(), StringComparison.InvariantCultureIgnoreCase);
break;
}
// Why doesn't this assignment wipe out the expression function value from the last loop iteration?
query = query.Where(expression).AsQueryable();
}
return new ObservableCollection<T>(query);
}
My understanding is that you are having trouble understanding why this line executed in a loop
query = query.Where(expression).AsQueryable();
produces an effect similar to "concatenation" of expressions. A short answer is that it is similar to why
str = str + suffix;
produces a longer string, even though it is an assignment.
A longer answer is that the loop is building an expression one predicate at a time, and appends a Where to the sequence of conditions. Even though it is an assignment, it is built from the previous state of the object, so the previous expression is not "lost", because it is used as a base of a bigger, more complex, filter expression.
To understand it better, imagine that the individual expressions produced by the switch statement are placed into an array of IQueryable objects, instead of being appended to query. Once the array of parts is built, you would be able to do this:
var query = data.AsQueryable()
.Where(parts[0]).AsQueryable()
.Where(parts[1]).AsQueryable()
...
.Where(parts[N]).AsQueryable();
Now observe that each parts[i] is used only once; after that, it is no longer needed. That is why you can build the chain of expressions incrementally in a loop: after the first iteration, query contains a chain that includes the first term; after the second iteration, it contains two first terms, and so on.
It doesn't "wipe it out" since it is chaining. It's handling it by assigning back to query. It's effectively like writing:
var queryTmp = query;
query = queryTmp.Where(expression).AsQueryable();
Each time you call .Where(expression).AsQueryable(), a new IQueryable<T> is being returned, and set to query. This IQueryable<T> is the result of the last .Where call. This means you effectively get a query that looks like:
query.Where(expression1).AsQueryable().Where(expression2).AsQueryable()...
Code essentially generates sequence of Where/AsQueryable calls. Not sure why you expect each loop to append expressions.
Essentially result is
query = query
.Where(expression0).AsQueryable()
.Where(expression1).AsQueryable()
.Where(expression2).AsQueryable()
where I think you expect more like
query = query
.Where(v => expression0(v) && expression1(v) && expression2(v) ...).AsQueryable()
The query variable name is a bit misleading. This code doesn't build up a long filter in the expression variable and then run it against the data set - it runs each filter against the data set, one at a time, until all of the filters have been run. The query variable just contains everything from the data that is left over from the previously run filters.
So this line:
query = query.Where(expression).AsQueryable();
is applying the filter to the existing data stored in query, and then saving the new (filtered) result back into the variable. The value of expression is overwritten each time through the loop, but we don't need it anymore because the filter has already been applied.
I'm busy with a LINQ to SQL project that basically creates multiple threads for each entity type in my database, which constantly queries information from the DB in a thread.
Here's a pseudo example:
streamer.DefineDataExpression<Contacts>(x => x.FirstName == "Bob");
while(true)
{
List<Contacts> MyContactsResult = streamer.ResultList;
// do whatever with MyContactsResult
}
The above code doesn't exist, but this is what I have so far for the 'streamer' class (it obviously doesn't work, but you can see what I'm trying to achieve above):
public void DefineExpression(System.Linq.Expressions.Expression<System.Func<T, bool>> expression)
{
using (var db = new LINQDataContext())
{
ResultList = db.GetTable<T>().Where(expression);
}
}
How do I go about creating a method like 'DefineExpression' that will allow me to query a LINQ type dynamically?
Why not use the Dynamic LINQ provider, as mentioned by Scott Guthrie. I think that would give you everything you are looking for, because you can define the query as a string. Therefore, you can more easily build a string representation of your query, and execute on the fly.
Say I have a LINQ-to-XML query that generates an anonymous type like this:
var aQuery =
(from a in document.Root.Elements("items")
select new {
id = a.Attribute("id").Value,
type = a.Attribute("type").Value,
modified = a.Attribute("modified").Value
});
if there a way to store that query expression in a variable or constant and then execute at runtime? The basic idea is that I have a bunch of these expressions and it would be handy if they could all be defined in one place and then invoked dynamically thru a single method where I just need to pass in the XML document and which expression to use. Thanks.
You could define them as methods quite easily, though you'd forfit the right to use anonymous types.
public static IQueryable<Item> GetItemsFromXml(XDocument document)
{
return (from a in document.Root.Elements("items")
select new Item
{
Id = a.Attribute("id").Value,
Type = a.Attribute("type").Value,
Modified = a.Attribute("modified").Value
});
}
Having said that, patterns like the repository pattern are used to wrap the whole process of accessing data.
I'm trying to implement encrypted columns in EF4 and using the CTP5 features to allow simple use of POCO's to query the database. Sorry that this is a lot of words, but I hope the below gives enough to explain the need and the problem!
So, bit of background, and my progress so far:
The intention is that if you query the tables without using our DAL then the data is rubbish, but I don't want the developers to worry about if/when/how the data is encrypted.
For simplicity, at this stage I'm working on the assumption any string column will be encrypted.
Now, I have successfully implemented this for returning the data using the Objectmaterialized event, and for data commits using the SavingChanges event.
So given the following class:
public class Thing
{
public int ID { get; set; }
[Required]
public string Name { get; set; }
public DateTime Date { get; set; }
public string OtherString { get; set; }
}
The below query returns all the required values and the POCO materialized has clear data in it.
var things = from t in myDbContext.Things
select t;
where myDbContext.Things is a DbSet<Thing>
Likewise, passing an instance of Thing to Things.Add()
(with clear string data in the Name and/or OtherString values)
and then calling myDbContext.SaveChanges() encrypts the strings before it gets to the data store.
Now, the problem I have is in this query:
var things = from t in myDbContext.Things
where t.Name == "Hairbrush"
select t;
This results in the unencrypted value being compared to the encrypted value in the DB. Obviously I don't want to get all the records from the database, materialize them, and then filter the results based on any supplied Where clause... so what I need to do is: intercept that query and rewrite it by encrypting any strings in the Where clause.
So I've looked at:
writing a query provider, but that doesn't seem like the right solution... (is it?)
writing my own IQueryable wrapper for the DbSet which will capture the expression, run over it using an expression tree visitor and then forward the new expression to the DbSet...
Attempts at both have left me somewhat lost! I prefer the second solution i think since it feels a bit neater, and is probably clearer to other developers in future. But I'm happy to go with either or another better option!!
The main thing I am struggling with is when/how the LINQ expression is applied to the object... I think i've got myself a bit confused as to where the expression executes in the IQueryable object thus I'm not sure which method I need to implement in my wrapper to then grab and manipulate the expression being passed in...
I'm sure I'm missing something fairly obvious here and I'm waiting for that light bulb moment... but its not coming!!
Any help will be very gratefully received!
Thought I'd let you know what my final solution was.
In the end I have gone a wrapper class which implements a Where method, but without going to the extent of implementing IQueryable entirely. LINQ will still execute against the class (at least to the extent that I want/need it to) and will call the Where method with the expression from the LINQ.
I then traverse this ExpressionTree and replace my strings with encrypted values before forwarding the new expressiontree to the internal DbSet. and then returning the result.
Its pretty crude, and has its limitation, but works for our particular circumstance without problem.
Thanks,
Ben
you should use the QueryInterceptor attribute, search here in SO or in google and you find examples on how to use it.
a snippet:
[QueryInterceptor("Orders")]
public Expression<Func<Order, bool>> FilterOrders()
{
return o => o.Customer.Name == /* Current principal name. */;
}
// Insures that the user accessing the customer(s) has the appropriate
// rights as defined in the QueryRules object to access the customer
// resource(s).
[QueryInterceptor ("Customers")]
public Expression<Func<Customer, bool>> FilterCustomers()
{
return c => c.Name == /* Current principal name. */ &&
this.CurrentDataSource.QueryRules.Contains(
rule => rule.Name == c.Name &&
rule.CustomerAllowedToQuery == true
);
}
You can use David Fowler's Query Interceptor:
https://github.com/davidfowl/QueryInterceptor
One example of its use:
IQueryable q = ...;
IQueryable modifed = q.InterceptWith(new MyInterceptor());
And on class MyInterceptor:
protected override Expression VisitBinary(BinaryExpression node) {
if (node.NodeType == ExpressionType.Equal) {
// Change == to !=
return Expression.NotEqual(node.Left, node.Right);
}
return base.VisitBinary(node);
}