I've had a look at other questions similar to this one but I couldn't find any workable answers.
I've been using the following code to generate unique keys for storing the results of my linq queries to the cache.
string key = ((LambdaExpression)expression).Body.ToString();
foreach (ParameterExpression param in expression.Parameters)
{
string name = param.Name;
string typeName = param.Type.Name;
key = key.Replace(name + ".", typeName + ".");
}
return key;
It seems to work fine for simple queries containing integers or booleans but when my query contains nested constant expressions e.g.
// Get all the crops on a farm where the slug matches the given slug.
(x => x.Crops.Any(y => slug == y.Slug) && x.Deleted == false)
The key returned is thus:
(True AndAlso (Farm.Crops.Any(y =>
(value(OzFarmGuide.Controllers.FarmController+<>c__DisplayClassd).slug
== y.Slug)) AndAlso (Farm.Deleted == False)))
As you can see any crop name I pass will give the same key result. Is there a way I can extract the value of the given parameter so that I can differentiate between my queries?
Also converting the y to say the correct type name would be nice.....
As Polity and Marc said in their comments, what you need is a partial evaluator of the LINQ expression. You can read how to do that using ExpressionVisitor in Matt Warren's LINQ: Building an IQueryable Provider - Part III. The article Caching the results of LINQ queries by Pete Montgomery (linked to by Polity) describes some more specifics regarding this kind of caching, e.g. how to represent collections in the query.
Also, I'm not sure I would rely on ToString() like this. I think it's meant mostly for debugging purposes and it might change in the future. The alternative would be creating your own IEqualityComparer<Expression> that can create a hash code for any expression and can compare two expressions for equality. I would probably do that using ExpressionVisitor too, but doing so would be quite tedious.
I've been trying to figure out a scenario where this kind of approach could be useful without leading to bloated cache that is insanely hard to maintain.
I know this isn't directly answering your question, but I want to raise a few questions about this approach that, at first, may sound tempting:
How did you plan to manage parameter ordering? Ie. (x => x.blah == "slug" && !x.Deleted) cache key should equal (x => !x.Deleted && x.blah == "slug") cache key.
How did you plan to avoid duplicate objects in cache? Ie. Same farm from multiple queries would by design be cached separately with each query. Say, for each slug that appears in the farm, we have a separate copy of the farm.
Extending the above with more parameters, such as parcel, farmer etc. would lead to more matching queries with each having a separate copy of the farm cached. The same applies to each type you might query plus the parameters might not be in the same order
Now, what happens if you update the farm? Without knowing which cached queries would contain your farm, you'd be forced to kill your whole cache. Which kind of is counterproductive to what you're trying to achieve.
I can see the reasoning behind this approach. A 0-maintenance performance layer. However, if the above points are not taken into consideration, the approach will first kill the performance, then lead to a lot of attempts to maintain it, then prove to be completely unmaintainable.
I've been down that road. Eventually wasted a lot of time and gave up.
I found a much better approach by caching each resulting entity separately when the results come from the backend with an extension method for each type separately or through a common interface.
Then you can build extension method for your lambda expressions to first try the cache before hitting the db.
var query = (x => x.Crops.Any(y => slug == y.Slug) && x.Deleted == false);
var results = query.FromCache();
if (!results.Any()) {
results = query.FromDatabase();
results.ForEach(x = x.ToCache());
}
Of course, you will still need to track which queries have actually hit the database to avoid query A returning 3 farms from DB satisfying query B with one matching farm from cache while the database would actually have 20 matching farms available. So, each query stll need to hit DB at least once.
And you need to track queries returning 0 results to avoid them consequently hitting the DB for nothing.
But all in all, you get away with a lot less code and as a bonus, when you update a farm, you can
var farm = (f => f.farmId == farmId).FromCache().First();
farm.Name = "My Test Farm";
var updatedFarm = farm.ToDatabase();
updatedFarm.ToCache();
What about this?
public class KeyGeneratorVisitor : ExpressionVisitor
{
protected override Expression VisitParameter(ParameterExpression node)
{
return Expression.Parameter(node.Type, node.Type.Name);
}
protected override Expression VisitMember(MemberExpression node)
{
if (CanBeEvaluated(node))
{
return Expression.Constant(Evaluate(node));
}
else
{
return base.VisitMember(node);
}
}
private static bool CanBeEvaluated(MemberExpression exp)
{
while (exp.Expression.NodeType == ExpressionType.MemberAccess)
{
exp = (MemberExpression) exp.Expression;
}
return (exp.Expression.NodeType == ExpressionType.Constant);
}
private static object Evaluate(Expression exp)
{
if (exp.NodeType == ExpressionType.Constant)
{
return ((ConstantExpression) exp).Value;
}
else
{
MemberExpression mexp = (MemberExpression) exp;
object value = Evaluate(mexp.Expression);
FieldInfo field = mexp.Member as FieldInfo;
if (field != null)
{
return field.GetValue(value);
}
else
{
PropertyInfo property = (PropertyInfo) mexp.Member;
return property.GetValue(value, null);
}
}
}
}
This will replace the complex constant expressions to their original values as well as the parameter names to their type names. So just have to create a new KeyGeneratorVisitor instance and call its Visit or VisitAndConvert method with your expression.
Please note that the Expression.ToString method will be also invoked on your complex types, so either override their ToString methods or write a custom logic for them in the Evaluate method.
How about:
var call = expression.Body as MethodCallExpression;
if (call != null)
{
List<object> list = new List<object>();
foreach (Expression argument in call.Arguments)
{
object o = Expression.Lambda(argument, expression.Parameters).Compile().DynamicInvoke();
list.Add(o);
}
StringBuilder keyValue = new StringBuilder();
keyValue.Append(expression.Body.ToString());
list.ForEach(e => keyValue.Append(String.Format("_{0}", e.ToString())));
string key = keyValue.ToString();
}
Related
I have a case where I need to send tens of thousands of ids to the graphql server in the filtering query.
The query now generated by the HT is something like this:
_dbContext.
Forms
.Where(c=>staticLiistOfIds.Contains(c.Id))
.Select(c=>new {C.Name,C.Age});
I have two problems with this:
slow performance
SQL Server Limit I guess is around 32K
I have found a Nuget library to convert this static list to a temp table,so now I want to override the HT middle to rewrite the above query generated to the following:
_dbContext.
Forms
.Where(c=>_dbContext..AsQueryableValues(staticLiistOfIds).Contains(c.Id))
.Select(c=>new {C.Name,C.Age});
This will create a temp table for this static list of ids so I will be able to solve the above two problems that I have.
So since i didn't get answers, I had to ask from the Slack of HotChocolate's Team and hopefully, they provided me with the documentation extending-filtering/extending-iqueryable:
in case the link was broken, here is
Extending IQueryable The default filtering implementation uses
IQueryable under the hood. You can customize the translation of
queries by registering handlers on the QueryableFilterProvider.
The following example creates a StringOperationHandler that supports
case-insensitive filtering:
// The QueryableStringOperationHandler already has an implemenation of CanHandle
// It checks if the field is declared in a string operation type and also checks if
// the operation of this field uses the `Operation` specified in the override property further
// below
public class QueryableStringInvariantEqualsHandler : QueryableStringOperationHandler
{
// For creating a expression tree we need the `MethodInfo` of the `ToLower` method of string
private static readonly MethodInfo _toLower = typeof(string)
.GetMethods()
.Single(
x => x.Name == nameof(string.ToLower) &&
x.GetParameters().Length == 0);
// This is used to match the handler to all `eq` fields
protected override int Operation => DefaultFilterOperations.Equals;
public override Expression HandleOperation(
QueryableFilterContext context,
IFilterOperationField field,
IValueNode value,
object parsedValue)
{
// We get the instance of the context. This is the expression path to the propert
// e.g. ~> y.Street
Expression property = context.GetInstance();
// the parsed value is what was specified in the query
// e.g. ~> eq: "221B Baker Street"
if (parsedValue is string str)
{
// Creates and returnes the operation
// e.g. ~> y.Street.ToLower() == "221b baker street"
return Expression.Equal(
Expression.Call(property, _toLower),
Expression.Constant(str.ToLower()));
}
// Something went wrong 😱
throw new InvalidOperationException();
}
}
This operation handler can be registered on the convention:
public class CustomFilteringConvention : FilterConvention
{
protected override void Configure(IFilterConventionDescriptor descriptor)
{
descriptor.AddDefaults();
descriptor.Provider(
new QueryableFilterProvider(
x => x
.AddDefaultFieldHandlers()
.AddFieldHandler<QueryableStringInvariantEqualsHandler>()));
}
}
// and then
services.AddGraphQLServer()
.AddFiltering<CustomFilteringConvention>();
To make this registration easier, Hot Chocolate also supports
convention and provider extensions. Instead of creating a custom
FilterConvention, you can also do the following:
services
.AddGraphQLServer()
.AddFiltering()
.AddConvention<IFilterConvention>(
new FilterConventionExtension(
x => x.AddProviderExtension(
new QueryableFilterProviderExtension(
y => y.AddFieldHandler<QueryableStringInvariantEqualsHandler>()))));
but I was suggested that doing this way(sending up to 100k list of string ids to the graphQL server) is not a good approach. so I decided to take another approach by writing a custom simple dynamic LINQ generates.
Thanks.
I am really struggling to write a generic class, that needs to check if all members of a passed expression are not null, by invoking the actual object.
I am calling the method like:
new TestClass<Class1, Class2>.IsAnyMemberNull(x => x.Property1.List1, object1);
and the Method looks like:
public bool IsAnyMemberNull(Expression<Func<T, IList<TM>>> listExpression, T entityDb)
{
var expressionsToCheck = new List<MemberExpression>();
var expression = listExpression.Body as MemberExpression;
while (expression != null)
{
expressionsToCheck.Add(expression);
expression = expression.Expression as MemberExpression;
}
for (var i = expressionsToCheck.Count - 1; i >= 0; i--)
{
var objectMember = Expression.Convert(expressionsToCheck[i], typeof (object));
var lambda = Expression.Lambda<Func<T, object>>(objectMember);
var value = lambda.Compile().Invoke(entityDb);
if (value == null)
return true;
}
return false;
}
When executing, I get the exception:
incorrect number of parameters supplied for lambda declaration
Any ideas, what I have done wrong?
While it is possible to get your code working, so that correct lambdas are built and compiled, using repeatedly compiled lambdas to achieve null-checking is a costly overkill.
Generally, using lambdas so casually (compiling a lambda for each property in chain and a single object) comes with a notable performance hit. I've run some tests, and on my computer executing this method 1000 times for a given object yielded ~300-700 ms time (depending on the number of properties in the chain). Dunno how many entities you deal with, but it's not a good sign, and better replacements are available. Please read further...
The question is, what are you using this for? That method of yours reminds me of null-conditional operators quite a lot. Generally, if you:
just want to check if any property in the property chain is a null
use C# 6
have all parameter chain lambdas (like x => x.Property1.List1) known at runtime
Then you might scrap all that IsAnyMemberNull method altogether in favour of something like:
object1.Property1.List1 == null
Much more concise and no additional methods are required. I ran it 1 million times and it was still within ~23ms time. It means it's dozens thousands faster than creating all those lambdas.
If you can't use null-coalescing operators for whatever reason (especially when the expression is built dynamically), you might instead decide to use Field/Property reflection.
I took the liberty of removing all that generic class wrapping in favour of a generic method. From your usage example, it seemed the only purpose of the generic class was to access a specific method with class' generic type parameters. It means one new class would have to be made and stored for each variation of the method, for no apparent reason, if I'm not mistaken, for the rest of application's lifetime. Generic methods in specific classes are generally preferred to specific methods in generic classes, in cases like these.
Also, I removed IList, because I see no reason how requiring the last parameter to be of IList type serves the purpose of the function; it only limits its applicability with no apparent gain.
Overall, the result is the following:
public bool IsAnyMemberNull<TEntity, TMember>(Expression<Func<TEntity, TMember>> paramChain, TEntity entityDb)
{
var expressionsToCheck = new List<MemberExpression>();
var expression = paramChain.Body as MemberExpression;
while (expression != null)
{
expressionsToCheck.Add(expression);
expression = expression.Expression as MemberExpression;
}
object value = entityDb;
for (var i = expressionsToCheck.Count - 1; i >= 0; i--)
{
var member = expressionsToCheck[i].Member;
if (member is PropertyInfo) value = (member as PropertyInfo).GetValue(value);
else if (member is FieldInfo) value = (member as FieldInfo).GetValue(value);
else throw new Exception(); // member generally should be a property or field, shouldn't it?
if (value == null)
return true;
}
return false;
}
After running this ~1000 times, it took about 4-6ms; 50-100 times better than lambdas, though null-propagation still reigns supreme.
Called as following (assuming it still resides in TestClass, which it doesn't need to):
new TestClass().IsAnyMemberNull<Class1,Class2>(x => x.Property1.List1, object1);
(Class1 and Class2 might not be necessary, thanks to type inference)
Hope this helps. It's not exactly what you asked for, but I'm afraid with all this lambda-spawning you could run into serious performance issues; especially if this code was to be used many times per request.
You have a problem in lambda expression creation - it is simpler than you thought. You should build lambda for each expressionToCheck with original expression parameter:
for (var i = expressionsToCheck.Count - 1; i >= 0; i--)
{
var lambda = Expression.Lambda<Func<T, object>>(expressionsToCheck[i], listExpression.Parameters);
var value = lambda.Compile().Invoke(entityDb);
if (value == null)
return true;
}
I have 3 variables in my filter:
int? Owner, int? Watcher, int? CreatedBy.
Now, depending of what is entered in filter (if CreatedBy.HasValue etc), i want to combine my query in NHibernate with OR statements. So far I have
if (filter.Owner.HasValue)
{
criteria.Add(Expression.Disjunction()
.Add(Restrictions.Eq("ou.User.Id", filter.Owner.Value))
.Add(Restrictions.Eq("ou.Status", 0)));
}
if (filter.Watcher.HasValue)
{
criteria.Add(Expression.Disjunction()
.Add(Restrictions.Eq("ou.User.Id", filter.Watcher.Value))
.Add(Restrictions.Eq("ou.Status", 1)));
}
if (filter.CreatedBy.HasValue)
{
criteria.Add(Restrictions.Eq("u.Id", filter.CreatedBy));
}
before, I added createAlias etc... But, how to combine those 3 queryies in one query with OR according to variables entered in filter?
I'd say, that the first part (you already have) is ok and also the second would be very straightforward. The way how to process that could be:
// this would be the collector of criteria
var restrictions = new List<ICriterion>();
if (filter.Owner.HasValue)
{
restrcitons.Add(Expression.Disjunction()
.Add(Restrictions.Eq("ou.User.Id", filter.Owner.Value))
.Add(Restrictions.Eq("ou.Status", 0))
);
}
if (filter.Watcher.HasValue)
{
restricitons.Add(Expression.Disjunction()
.Add(Restrictions.Eq("ou.User.Id", filter.Watcher.Value))
.Add(Restrictions.Eq("ou.Status", 1))
);
}
if (filter.CreatedBy.HasValue)
{
restrictions.Add(Restrictions.Eq("u.Id", filter.CreatedBy));
}
// now we can inject the result of the above code into
// Disjunction or Conjunction...
if(restrictions.Count > 0)
{
var disjunction = Restrictions.Disjunction();
restrictions .ForEach(r => disjunction.Add(r));
criteria.Add(disjunction)
}
Also, I would CreateAlias somewhere inside of the if - close to decision if the JOIN will be required or not (based on search parameters). Unless you are checking that elsewhere.
And maybe try to move the if into methods. They can take restrictions and criteria and decide how to handle them. it would be like this
RestrictOwner (filter, restrictions, criteria);
RestrictWatcher(filter, restrictions, criteria);
RestrictCreator(filter, restrictions, criteria);
if(restrictions.Count ...
And later, they could be maybe even more generic...
I am not very familiar with lambda expressions. So I have the following expression:
EnabledPropertySelector = l => l.FranchiseInfo.ExternalSystemType == ExternalSystemTypes.Master
And two properties:
public string VisibilityPropertyName { get; set; }
public object VisibilityPropertyValue { get; set; }
I want to extract some data from the expression so in the end I can get the values of the two properties:
VisibilityPropertyName == 'FranchiseInfo.ExternalSystemType';
VisibilityPropertyValue == ExternalSystemTypes.Master;
VisibilityPropertyName is always a string. This is the name of the property.
VisibilityPropertyValue can be of any type.
EDIT:
I have a lot of properties. Some of them are dependent on other properties. For every single property I have to manually write the name and the value of the parent property:
{ VisibilityPropertyName = 'FranchiseInfo.ExternalSystemType', VisibilityPropertyValue = ExternalSystemTypes.Master, EnabledPropertySelector = l => l.FranchiseInfo.ExternalSystemType == ExternalSystemTypes.Master}
Instead of writing all this I want to write only the expression and populate the properties from it.
This is the declaration of the expresion:
Expression<Func<TEntity, bool?>> EnabledPropertySelector
First off all, you need an Expression. What's the type of EnabledPropertySelector? It'll need to be something like Expression<Func<T, bool>> where T is whatever the type of "l" in your example is.
If you already have an Expression then you can use the Expression API to extract whatever you need:-
var body = EnabledPropertySelector.Body as BinaryExpression;
var left = body.Left as PropertyExpression;
var outerMemberName = left.Member.Name;
var innerMemberName = (left.Expression as PropertyExpression).Member.Name
VisibilityPropertyName = innerMemberName + "." + outerMemberName;
var right = body.Right as PropertyExpression;
var rightValueDelegate = Expression.Lambda<Func<object>>(right).Compile();
VisibilityPropertyValue = rightValueDelegate();
etc.
I really recommend doing some reading to properly grok the expression API before diving in though; there are a lot of corner cases depending on how flexible you need to be. E.g. is the expression always of the form parameter.Property.Property == constant? It gets really complicated really quickly, so you'll want a solid understanding of the fundamentals before trying to handle any real-world cases.
There's a reasonable introduction to expression trees on MSDN, but some focused googling might get you a better understanding quicker.
You can use Funciton and Action class, I'm not very sure of what you want be able to do, but I can give an tip.
Functions returns a value:
Function<InputType1,InputType2,ResultType> functionVariableName;
Usage:
functionVariableName = (param1, param2) => {
//...process both params
return result;
};
Actions, do not return values:
Action<InputType1,InputType2> actionVariableName;
Usage:
actionVariableName= (param1, param2) => {
//...process both params
};
If the lambda expression is simple (one line, with out if expression) you can make the lambda with out {}:
functionVariableName = (param1, param2) => //process values and result;
Hope this helps...
if you want to create an IEnumerable where the two properties are equal:
var results = EnabledPropertySelector.Where(l => l.FranchiseInfo.ExternalSystemType ==
ExternalSystemTypes.Master.ToString());
I'm trying to implement encrypted columns in EF4 and using the CTP5 features to allow simple use of POCO's to query the database. Sorry that this is a lot of words, but I hope the below gives enough to explain the need and the problem!
So, bit of background, and my progress so far:
The intention is that if you query the tables without using our DAL then the data is rubbish, but I don't want the developers to worry about if/when/how the data is encrypted.
For simplicity, at this stage I'm working on the assumption any string column will be encrypted.
Now, I have successfully implemented this for returning the data using the Objectmaterialized event, and for data commits using the SavingChanges event.
So given the following class:
public class Thing
{
public int ID { get; set; }
[Required]
public string Name { get; set; }
public DateTime Date { get; set; }
public string OtherString { get; set; }
}
The below query returns all the required values and the POCO materialized has clear data in it.
var things = from t in myDbContext.Things
select t;
where myDbContext.Things is a DbSet<Thing>
Likewise, passing an instance of Thing to Things.Add()
(with clear string data in the Name and/or OtherString values)
and then calling myDbContext.SaveChanges() encrypts the strings before it gets to the data store.
Now, the problem I have is in this query:
var things = from t in myDbContext.Things
where t.Name == "Hairbrush"
select t;
This results in the unencrypted value being compared to the encrypted value in the DB. Obviously I don't want to get all the records from the database, materialize them, and then filter the results based on any supplied Where clause... so what I need to do is: intercept that query and rewrite it by encrypting any strings in the Where clause.
So I've looked at:
writing a query provider, but that doesn't seem like the right solution... (is it?)
writing my own IQueryable wrapper for the DbSet which will capture the expression, run over it using an expression tree visitor and then forward the new expression to the DbSet...
Attempts at both have left me somewhat lost! I prefer the second solution i think since it feels a bit neater, and is probably clearer to other developers in future. But I'm happy to go with either or another better option!!
The main thing I am struggling with is when/how the LINQ expression is applied to the object... I think i've got myself a bit confused as to where the expression executes in the IQueryable object thus I'm not sure which method I need to implement in my wrapper to then grab and manipulate the expression being passed in...
I'm sure I'm missing something fairly obvious here and I'm waiting for that light bulb moment... but its not coming!!
Any help will be very gratefully received!
Thought I'd let you know what my final solution was.
In the end I have gone a wrapper class which implements a Where method, but without going to the extent of implementing IQueryable entirely. LINQ will still execute against the class (at least to the extent that I want/need it to) and will call the Where method with the expression from the LINQ.
I then traverse this ExpressionTree and replace my strings with encrypted values before forwarding the new expressiontree to the internal DbSet. and then returning the result.
Its pretty crude, and has its limitation, but works for our particular circumstance without problem.
Thanks,
Ben
you should use the QueryInterceptor attribute, search here in SO or in google and you find examples on how to use it.
a snippet:
[QueryInterceptor("Orders")]
public Expression<Func<Order, bool>> FilterOrders()
{
return o => o.Customer.Name == /* Current principal name. */;
}
// Insures that the user accessing the customer(s) has the appropriate
// rights as defined in the QueryRules object to access the customer
// resource(s).
[QueryInterceptor ("Customers")]
public Expression<Func<Customer, bool>> FilterCustomers()
{
return c => c.Name == /* Current principal name. */ &&
this.CurrentDataSource.QueryRules.Contains(
rule => rule.Name == c.Name &&
rule.CustomerAllowedToQuery == true
);
}
You can use David Fowler's Query Interceptor:
https://github.com/davidfowl/QueryInterceptor
One example of its use:
IQueryable q = ...;
IQueryable modifed = q.InterceptWith(new MyInterceptor());
And on class MyInterceptor:
protected override Expression VisitBinary(BinaryExpression node) {
if (node.NodeType == ExpressionType.Equal) {
// Change == to !=
return Expression.NotEqual(node.Left, node.Right);
}
return base.VisitBinary(node);
}