I found this code below in a file called Filter.cs in a project created with Microsoft App Studio. Although I am a veteran C# programmer, I'm short on experience with LINQ predicate expression builders. I can tell that the code below it is "meta-logic" for flexibly building a query given a list of filter predicates containing type field info and a set of data values to inject into the sub-expressions. What I can't figure out is how the "expression" variable in the following statement:
query = query.Where(expression).AsQueryable()"
.. is concatenating the per-field expressions into a more complex query expression that is finally executed at the end of the code to create the ObservableCollection result. If it was "query +=" I could infer a chaining action like an Event handler field, but as a straight assignment statement it baffles me since I would expect it to replace the last value the expression variable got from the last loop iteration, thereby resetting it in the process and losing its previous value(s). What is going on here?
public class Filter<T>
{
public static ObservableCollection<T> FilterCollection(
FilterSpecification filter, IEnumerable<T> data)
{
IQueryable<T> query = data.AsQueryable();
foreach (var predicate in filter.Predicates)
{
Func<T, bool> expression;
var predicateAux = predicate;
switch (predicate.Operator)
{
case ColumnOperatorEnum.Contains:
expression = x => predicateAux.GetFieldValue(x).ToLower().Contains(predicateAux.Value.ToString().ToLower());
break;
case ColumnOperatorEnum.StartsWith:
expression = x => predicateAux.GetFieldValue(x).ToLower().StartsWith(predicateAux.Value.ToString().ToLower());
break;
case ColumnOperatorEnum.GreaterThan:
expression = x => String.Compare(predicateAux.GetFieldValue(x).ToLower(), predicateAux.Value.ToString().ToLower(), StringComparison.Ordinal) > 0;
break;
case ColumnOperatorEnum.LessThan:
expression = x => String.Compare(predicateAux.GetFieldValue(x).ToLower(), predicateAux.Value.ToString().ToLower(), StringComparison.Ordinal) < 0;
break;
case ColumnOperatorEnum.NotEquals:
expression = x => !predicateAux.GetFieldValue(x).Equals(predicateAux.Value.ToString(), StringComparison.InvariantCultureIgnoreCase);
break;
default:
expression = x => predicateAux.GetFieldValue(x).Equals(predicateAux.Value.ToString(), StringComparison.InvariantCultureIgnoreCase);
break;
}
// Why doesn't this assignment wipe out the expression function value from the last loop iteration?
query = query.Where(expression).AsQueryable();
}
return new ObservableCollection<T>(query);
}
My understanding is that you are having trouble understanding why this line executed in a loop
query = query.Where(expression).AsQueryable();
produces an effect similar to "concatenation" of expressions. A short answer is that it is similar to why
str = str + suffix;
produces a longer string, even though it is an assignment.
A longer answer is that the loop is building an expression one predicate at a time, and appends a Where to the sequence of conditions. Even though it is an assignment, it is built from the previous state of the object, so the previous expression is not "lost", because it is used as a base of a bigger, more complex, filter expression.
To understand it better, imagine that the individual expressions produced by the switch statement are placed into an array of IQueryable objects, instead of being appended to query. Once the array of parts is built, you would be able to do this:
var query = data.AsQueryable()
.Where(parts[0]).AsQueryable()
.Where(parts[1]).AsQueryable()
...
.Where(parts[N]).AsQueryable();
Now observe that each parts[i] is used only once; after that, it is no longer needed. That is why you can build the chain of expressions incrementally in a loop: after the first iteration, query contains a chain that includes the first term; after the second iteration, it contains two first terms, and so on.
It doesn't "wipe it out" since it is chaining. It's handling it by assigning back to query. It's effectively like writing:
var queryTmp = query;
query = queryTmp.Where(expression).AsQueryable();
Each time you call .Where(expression).AsQueryable(), a new IQueryable<T> is being returned, and set to query. This IQueryable<T> is the result of the last .Where call. This means you effectively get a query that looks like:
query.Where(expression1).AsQueryable().Where(expression2).AsQueryable()...
Code essentially generates sequence of Where/AsQueryable calls. Not sure why you expect each loop to append expressions.
Essentially result is
query = query
.Where(expression0).AsQueryable()
.Where(expression1).AsQueryable()
.Where(expression2).AsQueryable()
where I think you expect more like
query = query
.Where(v => expression0(v) && expression1(v) && expression2(v) ...).AsQueryable()
The query variable name is a bit misleading. This code doesn't build up a long filter in the expression variable and then run it against the data set - it runs each filter against the data set, one at a time, until all of the filters have been run. The query variable just contains everything from the data that is left over from the previously run filters.
So this line:
query = query.Where(expression).AsQueryable();
is applying the filter to the existing data stored in query, and then saving the new (filtered) result back into the variable. The value of expression is overwritten each time through the loop, but we don't need it anymore because the filter has already been applied.
Related
I was writing unit tests to compare an original response to a filtered response using a request object as a parameter. In doing so I noticed that if I change the request object after getting a response the IEnumerable list will change - As I type this, my thinking is that because it is an IEnumerable with LINQ, the request.Filter property is a reference in the LINQ query, which is what causes this behavior. If I converted this to a list instead of an IEnumerable, I suspect the behavior would go away because the .ToList() will evaluate the LINQ expressions instead of deferring. Is that the case?
public class VendorResponse {
public IEnumerable<string> Vendors { get; set; }
}
var request = new VendorRequest() {
Filter = ""
};
var response = await _service.GetVendors(request);
int vendorCount = response.Vendors.Count(); // 20
request.Filter = "at&t";
int newCount = response.Vendors.Count(); // 17
public async Task<VendorResponse> GetVendors(VendorRequest request)
{
var vendors = await _dataService.GetVendors();
return new VendorResponse {
Vendors = vendors.Where(v => v.IndexOf(request.Filter) >= 0)
}
}
If deferred execution is preferable, you can capture the current state of request.Filter with a local variable and use that in the Where predicate
public async Task<VendorResponse> GetVendors(VendorRequest request)
{
var filter = request.Filter;
var vendors = await _dataService.GetVendors();
return new VendorResponse {
Vendors = vendors.Where(v => v.IndexOf(filter) >= 0)
}
}
Yes!
This is an example of deferred execution of an IEnumerable, which just encapsulates a query on some data without encapsulating the result of that query.
An IEnumerable can be enumerated (via its IEnumerator), and "knows" how to enumerate the query it encapsulates, but this will not actually happen until something executes the enumeration.
In your case the enumeration is executed by the call to .Count() which needs to know how many items are in the result of the query. The enumeration occurs every time you call .Count(), so changing the filter between the two invocations leads to you getting two different results.
As you have correctly deduced, calling .ToList() and capturing the result in a variable before performing any further operations would lead to you capturing the resulting data rather than the query, and so lead to both counts having the same value.
Try this out yourself. In future, be sure to force the evaluation of the enumerable before passing to other queries, or returning out to unknown code, otherwise you or your users will encounter unexpected behaviour and possible performance issues.
Hope this helps :)
Edit 1:
As Moho has pointed out, and you have also alluded to in your original post, this is also a result of the request.Filter being captured by the IEnumerable as a reference type. If you can capture the value and pass this in instead, the result of the IEnumerable will no longer be modified by changing the filter.
I have this
List<Expression> levl1expressions;
Collection contains binary expressions like Expression.NotEqual, Expression.Equal
etc
I have another collection which is And and Or Conditions
List<Expression> levl2expressions;
I would like to execute these two expression collections
levl1expressions[0]+levl2expressions[0]+levl1expressions[1]+levl2expressions[1]....
Is this possible?
eg:
object.Name = "something" && object.Category != "myCategory"(//I transformed the string to expressions)
levl1expressions[0] = Expression.Equal(
Expression.Property(Expression.Parameter(typeof(MyObject), "m")),
Expression.Constant("something")
levl1expressions[1] = Expression.NotEqual(....)
levl2expressions[0]= Expression.And(/*Would like to join levl1expressions*/)
If I understand your problem correctly, you should not use lists at all. Instead you can create a single expression which will look like that:
var finalExpression = Expression.And(Expression.Equal(...), Expression.NotEqual(...));
If you would like to combine more logical operators then you can use a result of Expression.And as the right operand:
var finalExpression = Expression.And(Expression.Equal(...), Expression.And(Expression.NotEqual(...), Expression.Equal(...)));
To invoke the expression you need to compile first:
var action = Expression.Lambda<Action<bool>>(finalExpression).Compile();
Here Action<bool> is specifying what kind of function you create. Action<bool> basically means a function which is returning a boolean and has no parameters. Once you have that you can simply call it:
var result = action();
Bear in mind that expression compilation process is very expensive. Cache the result if you can.
I have been trying to pass in a dynamic list of Expressions to a MongoDB C# Driver query using Linq ... This method works for me with regular Linq queries against an ORM, for example, but results in an error when applied to a MongoDB query ... (FYI: I am also using LinqKit's PredicateBuilder)
//
// I create a List of Expressions which I can then add individual predicates to on an
// "as-needed" basis.
var filters = new List<Expression<Func<Session, Boolean>>>();
//
// If the Region DropDownList returns a value then add an expression to match it.
// (the WebFormsService is a home built service for extracting data from the various
// WebForms Server Controls... in case you're wondering how it fits in)
if (!String.IsNullOrEmpty(WebFormsService.GetControlValueAsString(this.ddlRegion)))
{
String region = WebFormsService.GetControlValueAsString(this.ddlRegion).ToLower();
filters.Add(e => e.Region.ToLower() == region);
}
//
// If the StartDate has been specified then add an expression to match it.
if (this.StartDate.HasValue)
{
Int64 startTicks = this.StartDate.Value.Ticks;
filters.Add(e => e.StartTimestampTicks >= startTicks);
}
//
// If the EndDate has been specified then add an expression to match it.
if (this.EndDate.HasValue)
{
Int64 endTicks = this.EndDate.Value.Ticks;
filters.Add(e => e.StartTimestampTicks <= endTicks);
}
//
// Pass the Expression list to the method that executes the query
var data = SessionMsgsDbSvc.GetSessionMsgs(filters);
The GetSessionMsgs() method is defined in a Data services class ...
public class SessionMsgsDbSvc
{
public static List<LocationOwnerSessions> GetSessionMsgs(List<Expression<Func<Session, Boolean>>> values)
{
//
// Using the LinqKit PredicateBuilder I simply add the provided expressions
// into a single "AND" expression ...
var predicate = PredicateBuilder.True<Session>();
foreach (var value in values)
{
predicate = predicate.And(value);
}
//
// ... and apply it as I would to any Linq query, in the Where clause.
// Additionally, using the Select clause I project the results into a
// pre-defined data transfer object (DTO) and only the DISTINCT DTOs are returned
var query = ApplCoreMsgDbCtx.Sessions.AsQueryable()
.Where(predicate)
.Select(e => new LocationOwnerSessions
{
AssetNumber = e.AssetNumber,
Owner = e.LocationOwner,
Region = e.Region
})
.Distinct();
var data = query.ToList();
return data;
}
}
Using the LinqKit PredicateBuilder I simply add the provided expressions into a single "AND" expression ... and apply it as I would to any Linq query, in the Where() clause. Additionally, using the Select() clause I project the results into a pre-defined data transfer object (DTO) and only the DISTINCT DTOs are returned.
This technique typically works when I an going against my Telerik ORM Context Entity collections ... but when I run this against the Mongo Document Collection I get the following error ...
Unsupported filter: Invoke(e => (e.Region.ToLower() == "central"),
{document})
There is certainly something going on beneath the covers that I am unclear on. In the C# Driver for MongoDB documentation I found the following NOTE ...
"When projecting scalars, the driver will wrap the scalar into a
document with a generated field name because MongoDB requires that
output from an aggregation pipeline be documents"
But honestly I am not sure what that neccessarily means or if it's related to this problem or not. The appearence of "{document}" in the error suggests that it might be relevant though.
Any additional thoughts or insight would be greatly appreciated though. Been stuck on this for the better part of 2 days now ...
I did find this post but so far am not sure how the accepted solution is much different than what I have done.
I'm coming back to revisit this after 4 years because while my original supposition did work it worked the wrong way which was it was pulling back all the records from Mongo and then filtering them in memory and to compound matters it was making a synchronous call into the database which is always a bad idea.
The magic happens in LinqKit's expand extension method
That flattens the invocation expression tree into something the Mongo driver can understand and thus act upon.
.Where(predicate.Expand())
I've been experimenting with Linq to see what it can do - and I'm really loving it so far :)
I wrote some queries for an algorithm, but I didn't get the results I expected... the Enumeration always returned empty:
case #1
List<some_object> next = new List<some_object>();
some_object current = null;
var valid_next_links =
from candidate in next
where (current.toTime + TimeSpan.FromMinutes(5) <= candidate.fromTime)
orderby candidate.fromTime
select candidate;
current = something;
next = some_list_of_things;
foreach (some_object l in valid_next_links)
{
//do stuff with l
}
I changed the query declaration to be inline like this, and it worked fine:
case #2
foreach (some_object l in
(from candidate in next
where (current.toTime + TimeSpan.FromMinutes(5) <= candidate.fromTime)
orderby candidate.fromTime
select candidate))
{
//do stuff with l
}
Does anybody know why it doesn't work in case #1 ?
The way I understood it, the query wasn't evaluated when you declared it, so I don't see how there is a difference.
Changes to current will be captured, but the query already knows the value of next. Adding extra items to the existing list will make them show up in the query, but changing the value of the variable to refer to a different list entirely won't have any effect. Basically, if you mentally expand the query from a query expression into a "normal" form, any variable present in a lambda expression will be captured as a variable, but any variable present directly as an argument will be evaluated immediately. That will only capture the reference value of the variable, not the items present in the list, but it still means changing the variable value itself won't be seen. Your first query expands to:
var valid_next_links = next
.Where(candidate => (current.toTime + TimeSpan.FromMinutes(5) <= candidate.fromTime))
.OrderBy(candidate => candidate.fromTime);
I have a query expression that takes an array of strings and should be generating a query that returns some items based on the query. However, it does not return anything. I think there is a closure problem, but I am not sure what it is.
public static Expression<Func<Item, bool>> IsKnownByIn(string[] query )
{
var i = PredicateBuilder.True<Item>();
foreach (string keyword in query)
{
string temp = keyword;
i = i.And(p=> p.Name.Contains(temp) || p.ID.ToString().Contains(temp));
}
return i;
}
I tried replacing the .Contains(temp) with .Contains(keyword) which causes the only the last string in query to be returned. In addition, replacing the i.And with i.Or causes every item (even those that do not contain any of the strings in query) to be returned. Any ideas on where the problem may be hiding?
When you tried the Or, you got them all because you start with true, i.e.
true or something or something
will always be true. Change the construction of i to be False instead.
In the code you present, your assigning keyword to temp is the correct approach when using PredicateBuilder and building up within a loop.
EDIT:
Based on your answer to my question, I would do this:
public static Expression<Func<Item, bool>> IsKnownByIn(string[] query )
{
var i = PredicateBuilder.False<Item>();
foreach (string keyword in query)
{
string temp = keyword;
i = i.Or(p=> p.Name.Contains(temp) || p.ID.ToString().Contains(temp));
}
return i;
}
So do you have items with all keywords specified in query? These should be the only ones which can pass your filter. Probably, you should use Or instead. The issue with Or you've seen might be because you start with PredicateBuilder.True, so 'true or expr1 or expr2 ... exprn' will yield true always. If you've used PredicateBuilder.False you should probably show more code so we can tell what is the issue.