Develop Reference

How to properly convert a list to IQueryable on a asp net core web API

im trying to deploy a simple search function that uses a simple tag system, probably there are better ways to bt this is the solution i landed on:
public async Task<ActionResult<IEnumerable<t_usuarios_pub>>> Gett_usuarios_pubByTag(string tag)
{
string[] TagList;
TagList = tag.Split(',');
List<t_usuarios_pub> results = new List<t_usuarios_pub>();
var pubs = from m in _context.t_usuarios_pub select m;
if (!String.IsNullOrEmpty(tag))
{
foreach (var Itag in TagList)
{
pubs = pubs.Where(s => (s.tag.Contains(Itag) && s.estatus<2)).OrderBy(x => x.estatus);
foreach(var x in pubs)
{
if (!results.Contains(x))
{
results.Add(x);
}
}
}
}
return await results.AsQueryable().ToListAsync();
}
problem is that the List results cant be converted to IQueryable, this is the error stack.
Any idea of how i can properly implement it ?
System.InvalidOperationException:
The source 'IQueryable' doesn't implement 'IAsyncEnumerable<UserPostAPI.Models.t_usuarios_pub>'.
Only sources that implement 'IAsyncEnumerable' can be used for Entity Framework asynchronous operations.
´´´

Since results is a local variable of List<> type, I don't believe you need to await the last line. It might just work with:
return results.AsQueryable();
In which case, your method may not even need to be an async method. Or, depending on what _context is, perhaps the await needs to be on the pubs filter call:
pubs = await pubs.Where(s => (s.tag.Contains(Itag) && s.estatus<2))
.OrderBy(x => x.estatus)
.ToListAsync();
Furthermore, since your method says it returns an IEnumerable<>, you probably don't need the .AsQueryable(), either:
return result;
There's also a lot of refactoring you could do -- here are just a few things you may want to consider:
Move the String.IsNullOrEmpty(tag) check to the beginning of the method as a guard.
If this is user input, trim the split tags in TagList to avoid any issues with extra whitespace, and then be sure to remove any resulting empty members.
You could put the estatus < 2 check in your query to reduce having to re-check it again for every item in pubs for every ITag.

MongoDB: Combine Aggregation and Filter

Please see the following post for some background: MongoDB C# Driver - Return last modified rows only
After almost two years of running this code, we've been experiencing performance problems lately and as much as I keep on saying that the code is not the issue, Infrastructure are insisting it's because I'm doing full table scans.
The thing is that the problem is environment specific. Our QA environment runs like a dream all the time but Dev and Prod are very slow at times and fine at other - it's very erratic. They have the same data and code on but Dev and Prod have another app that is also running on the database.
My data has an Id as well as an _id (or AuditId) - I group the data by Id and then return the last _id for that record where it was not deleted. We have multiple historic records for the same ID and I would like to return the last one (see original post).
So I have the following method:
private static FilterDefinition<T> ForLastAuditIds<T>(IMongoCollection<T> collection) where T : Auditable, IMongoAuditable
{
var pipeline = new[] { new BsonDocument { { "$group", new BsonDocument { { "_id", "$Id" }, { "LastAuditId", new BsonDocument { { "$max", "$_id" } } } } } } };
var lastAuditIds = collection.Aggregate<Audit>(pipeline).ToListAsync().Result.ToList().Select(_ => _.LastAuditId);
var forLastAuditIds = Builders<T>.Filter.Where(_ => lastAuditIds.Contains(_.AuditId) && _.Status != "DELETE");
return forLastAuditIds;
}
This method is called by the one below, which accepts an Expression that it appends to the FilterDefinition created by ForLastAuditIds.
protected List<T> GetLatest<T>(IMongoCollection<T> collection,
Expression<Func<T, bool>> filter, ProjectionDefinition<T, T> projection = null,
bool disableRoleCheck = false) where T : Auditable, IMongoAuditable
{
var forLastAuditIds = ForLastAuditIds(collection);
var limitedList = (
projection != null
? collection.Find(forLastAuditIds & filter, new FindOptions()).Project(projection)
: collection.Find(forLastAuditIds & filter, new FindOptions())
).ToListAsync().Result.ToList();
return limitedList;
}
Now, all of this works really well and is re-used by all of my code that calls Collections, but this specific collection is a lot bigger than the others and we are getting slowdowns just on that one.
My question is: Is there a way for me to take the aggregate and Filter Builder and combine them to return a single FilterDefinition that I could use without running the full table scan first?
I really hope I am making sense.

Assuming I fully understand what you want, this should be as easy as this:
First, put a descending index on the LastAuditId field:
db.collection.createIndex{ "LastAuditId": -1 /* for sorting */ }
Or even extend the index to cover for other fields that you have in your filter:
db.collection.createIndex{ "Status": 1, "LastAuditId": -1 /* for sorting */ }
Make sure, however, that you understand how indexes can/cannot support certain queries. And always use explain() to see what's really going on.
The next step is to realize that you must always filter as much as possible as the very first step to reduce the amount of sorting required.
So, if you need to e.g. filter by Name then by all means do it as the very first step if your business requirements permit it. Be careful, however, that filtering at the start changes your semantics in the sense that you will get the last modified documents per each Id that passed the preceeding $match stage as opposed to the last documents per each Id that happen to also pass the following $match stage.
Anyway, most importantly, once you've got a sorted set, you can easily and quickly get the latest full document by using $group with $first which - with the right index in place - will not do a collection scan anymore (it'll be an index scan for now and hence way faster).
Finally, you want to run the equivalent of the following MongoDB query through C# leveraging the $$ROOT variable in order to avoid a second query (I can put the required code together for you once you post your Audit, Auditable and IMongoAuditable types as well as any potential serializers/conventions):
db.getCollection('collection').aggregate({
$match: {
/* some criteria that you currently get in the "Expression<Func<BsonDocument, bool>> filter" */
}
}, {
$sort: {
"ModifiedDate": -1 // this will use the index!
}
}, {
$group: {
"_id": "$Id",
"document": { $first: "$$ROOT" } // no need to do a separate subsequent query or a $max/$min across the entire group because we're sorted!
}
}, {
$match: { // some additional filtering depending on your needs
"document.Status": { $ne: "Delete" }
}
})
Lastly, kindly note that it might be a good idea to move to the latest version of MongoDB because they are currently putting a lot of effort into optimizing aggregation cases like yours, e.g. this one: https://jira.mongodb.org/browse/SERVER-9507

Optional Design Pattern, Advantages [closed]

Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 8 years ago.
Improve this question
So, it's pretty well known that the infamous NullReferenceException is the most common exception in software products. I've been reading some articles, and found myself with the Optional approach.
Its aim is to create some kind of encapsulation around a nullable value
public sealed class Optional<T> where T : class {
private T value;
private Optional(T value) {
this.value = value;
}
//Used to create an empty container
public static Optional<T> Empty() {
return new Optional(null);
}
//Used to create a container with a non-null value
public static Optional<T> For(T value) {
return new Optional(value);
}
//Used to check if the container holds a non-null value
public bool IsPresent {
get { return value != null; }
}
//Retrieves the non-null value
public T Value {
get { return value; }
}
}
Afterwards, the now optional value can be returned like this:
public Optional<ICustomer> FindCustomerByName(string name)
{
ICustomer customer = null;
// Code to find the customer in database
if(customer != null) {
return Optional.Of(customer);
} else {
return Optional.Empty();
}
}
And handled like this:
Optional<ICustomer> optionalCustomer = repository.FindCustomerByName("Matt");
if(optionalCustomer.IsPresent) {
ICustomer foundCustomer = optionalCustomer.Value;
Console.WriteLine("Customer found: " + customer.ToString());
} else {
Console.WriteLine("Customer not found");
}
I don't see any improvement, just shifted complexity.
The programmer must remember to check if a value IsPresent, in the same way he must remember to check if a value != null.
And if he forgets, he would get a NullReferenceException on both approaches.
What am I missing? What advantages (if any) does the Optional pattern provide over something like Nullable<T> and the null coalescing operator?

Free your mind
If you think of Option as Nullable by a different name then you are absolutely correct - Option is simply Nullable for reference types.
The Option pattern makes more sense if you view it as a monad or as a specialized collection that contain either one or zero values.
Option as a collection
Consider a simple foreach loop with a list that cannot be null:
public void DoWork<T>(List<T> someList) {
foreach (var el in someList) {
Console.WriteLine(el);
}
}
If you pass an empty list to DoWork, nothing happens:
DoWork(new List<int>());
If you pass a list with one or more elements in it, work happens:
DoWork(new List<int>(1));
// 1
Let's alias the empty list to None and the list with one entry in it to Some:
var None = new List<int>();
var Some = new List(1);
We can pass these variables to DoWork and we get the same behavior as before:
DoWork(None);
DoWork(Some);
// 1
Of course, we can also use LINQ extension methods:
Some.Where(x => x > 0).Select(x => x * 2);
// List(2)
// Some -> Transform Function(s) -> another Some
None.Where(x => x > 0).Select(x => x * 2);
// List()
// None -> None
Some.Where(x => x > 100).Select(x => x * 2);
// List() aka None
// Some -> A Transform that eliminates the element -> None
Interesting side note: LINQ is monadic.
Wait, what just happened?
By wrapping the value that we want inside a list we were suddenly able to only apply an operation to the value if we actually had a value in the first place!
Extending Optional
With that consideration in mind, let's add a few methods to Optional to let us work with it as if it were a collection (alternately, we could make it a specialized version of IEnumerable that only allows one entry):
// map makes it easy to work with pure functions
public Optional<TOut> Map<TIn, TOut>(Func<TIn, TOut> f) where TIn : T {
return IsPresent ? Optional.For(f(value)) : Empty();
}
// foreach is for side-effects
public Optional<T> Foreach(Action<T> f) {
if (IsPresent) f(value);
return this;
}
// getOrElse for defaults
public T GetOrElse(Func<T> f) {
return IsPresent ? value : f();
}
public T GetOrElse(T defaultValue) { return IsPresent ? value: defaultValue; }
// orElse for taking actions when dealing with `None`
public void OrElse(Action<T> f) { if (!IsPresent) f(); }
Then your code becomes:
Optional<ICustomer> optionalCustomer = repository.FindCustomerByName("Matt");
optionalCustomer
.Foreach(customer =>
Console.WriteLine("Customer found: " + customer.ToString()))
.OrElse(() => Console.WriteLine("Customer not found"));
Not much savings there, right? And two more anonymous functions - so why would we do this? Because, just like LINQ, it enables us to set up a chain of behavior that only executes as long as we have the input that we need. For example:
optionalCustomer
.Map(predictCustomerBehavior)
.Map(chooseIncentiveBasedOnPredictedBehavior)
.Foreach(scheduleIncentiveMessage);
Each of these actions (predictCustomerBehavior, chooseIncentiveBasedOnPredictedBehavior, scheduleIncentiveMessage) is expensive - but they will only happen if we have a customer to begin with!
It gets better though - after some study we realize that we cannot always predict customer behavior. So we change the signature of predictCustomerBehavior to return an Optional<CustomerBehaviorPrediction> and change our second Map call in the chain to FlatMap:
optionalCustomer
.FlatMap(predictCustomerBehavior)
.Map(chooseIncentiveBasedOnPredictedBehavior)
.Foreach(scheduleIncentiveMessage);
which is defined as:
public Optional<TOut> FlatMap<TIn, TOut>(Func<TIn, Optional<TOut>> f) where TIn : T {
var Optional<Optional<TOut>> result = Map(f)
return result.IsPresent ? result.value : Empty();
}
This starts to look a lot like LINQ (FlatMap -> Flatten, for example).
Further possible refinements
In order to get more utility out of Optional we should really make it implement IEnumerable. Additionally, we can take advantage of polymorphism and create two sub-types of Optional, Some and None to represent the full list and the empty list case. Then our methods can drop the IsPresent checks, making them easier to read.
TL;DR
The advantages of LINQ for expensive operations are obvious:
someList
.Where(cheapOp1)
.SkipWhile(cheapOp2)
.GroupBy(expensiveOp)
.Select(expensiveProjection);
Optional, when viewed as a collection of one or zero values provides a similar benefit (and there's no reason it couldn't implement IEnumerable so that LINQ methods would work on it as well):
someOptional
.FlatMap(expensiveOp1)
.Filter(expensiveOp2)
.GetOrElse(generateDefaultValue);
Further suggested reading
Option (F#)
When null is not enough (C#)
The neophytes guide to Scala Part 5: The Option type
The Marvel of Monads (C#)
Eric Lippert's series on LINQ and monads

it would probally make more sense if you used something like this
interface ICustomer {
String name { get; }
}
public class OptionalCustomer : ICustomer {
public OptionalCustomer (ICustomer value) {
this.value = value;
}
public static OptionalCustomer Empty() {
return new OptionalCustomer(null);
}
ICustomer value;
public String name { get {
if (value == null ) {
return "No customer found";
}
return value.Name;
}
}
}
now if your pass an "empty" optional customer object you can still call the .Name property (without getting nullpointers)

The advantage of Optional is you know if something may not exist.
The problem with many types of queries that return a null is that that could mean 2 things:
The query didn't return a result
The query returned a result whose value was null.
I know you're asking specifically about C# but Java just introduced Optionals in Java 8 so there are a lot of articles about it so I'll use Java as an example. but it's completely the same idea as in C#:
Consider the Java Map.get(key) method
Object value = map.get(key);
if(value ==null){
//is there an entry in the map key =>null or does key not exist?
}
to get around that you have to have an additional method containsKey( k)
With optional, you only need one method
Optional<Object> result = map.get(key);
if(result.isPresent()){
Object value = result.get();
//if value is null, then we know that key =>null
}
More info see this Java article : http://www.oracle.com/technetwork/articles/java/java8-optional-2175753.html

Did you mean: Null Object pattern
The article linked to me in the comments contains a conclusion section explained this programming tool.
... The purpose of Optional is not to replace every single null reference in your codebase but rather to help design better APIs in which—just by reading the signature of a method—users can tell whether to expect an optional value. .... deal with the absence of a value; as a result, you protect your code against unintended null pointer exceptions.
Anyway, let it crash and find the reason. If you do not want endlessly embedded if statements than use an implementation pattern Guard Clause pattern, which says the following:
While programs have a main flow, some situations require deviations from the
main flow. The guard clause is a way to express simple and local exceptional
situations with purely local consequences.

Converting a lambda expression into a unique key for caching

I've had a look at other questions similar to this one but I couldn't find any workable answers.
I've been using the following code to generate unique keys for storing the results of my linq queries to the cache.
string key = ((LambdaExpression)expression).Body.ToString();
foreach (ParameterExpression param in expression.Parameters)
{
string name = param.Name;
string typeName = param.Type.Name;
key = key.Replace(name + ".", typeName + ".");
}
return key;
It seems to work fine for simple queries containing integers or booleans but when my query contains nested constant expressions e.g.
// Get all the crops on a farm where the slug matches the given slug.
(x => x.Crops.Any(y => slug == y.Slug) && x.Deleted == false)
The key returned is thus:
(True AndAlso (Farm.Crops.Any(y =>
(value(OzFarmGuide.Controllers.FarmController+<>c__DisplayClassd).slug
== y.Slug)) AndAlso (Farm.Deleted == False)))
As you can see any crop name I pass will give the same key result. Is there a way I can extract the value of the given parameter so that I can differentiate between my queries?
Also converting the y to say the correct type name would be nice.....

As Polity and Marc said in their comments, what you need is a partial evaluator of the LINQ expression. You can read how to do that using ExpressionVisitor in Matt Warren's LINQ: Building an IQueryable Provider - Part III. The article Caching the results of LINQ queries by Pete Montgomery (linked to by Polity) describes some more specifics regarding this kind of caching, e.g. how to represent collections in the query.
Also, I'm not sure I would rely on ToString() like this. I think it's meant mostly for debugging purposes and it might change in the future. The alternative would be creating your own IEqualityComparer<Expression> that can create a hash code for any expression and can compare two expressions for equality. I would probably do that using ExpressionVisitor too, but doing so would be quite tedious.

I've been trying to figure out a scenario where this kind of approach could be useful without leading to bloated cache that is insanely hard to maintain.
I know this isn't directly answering your question, but I want to raise a few questions about this approach that, at first, may sound tempting:
How did you plan to manage parameter ordering? Ie. (x => x.blah == "slug" && !x.Deleted) cache key should equal (x => !x.Deleted && x.blah == "slug") cache key.
How did you plan to avoid duplicate objects in cache? Ie. Same farm from multiple queries would by design be cached separately with each query. Say, for each slug that appears in the farm, we have a separate copy of the farm.
Extending the above with more parameters, such as parcel, farmer etc. would lead to more matching queries with each having a separate copy of the farm cached. The same applies to each type you might query plus the parameters might not be in the same order
Now, what happens if you update the farm? Without knowing which cached queries would contain your farm, you'd be forced to kill your whole cache. Which kind of is counterproductive to what you're trying to achieve.
I can see the reasoning behind this approach. A 0-maintenance performance layer. However, if the above points are not taken into consideration, the approach will first kill the performance, then lead to a lot of attempts to maintain it, then prove to be completely unmaintainable.
I've been down that road. Eventually wasted a lot of time and gave up.
I found a much better approach by caching each resulting entity separately when the results come from the backend with an extension method for each type separately or through a common interface.
Then you can build extension method for your lambda expressions to first try the cache before hitting the db.
var query = (x => x.Crops.Any(y => slug == y.Slug) && x.Deleted == false);
var results = query.FromCache();
if (!results.Any()) {
results = query.FromDatabase();
results.ForEach(x = x.ToCache());
}
Of course, you will still need to track which queries have actually hit the database to avoid query A returning 3 farms from DB satisfying query B with one matching farm from cache while the database would actually have 20 matching farms available. So, each query stll need to hit DB at least once.
And you need to track queries returning 0 results to avoid them consequently hitting the DB for nothing.
But all in all, you get away with a lot less code and as a bonus, when you update a farm, you can
var farm = (f => f.farmId == farmId).FromCache().First();
farm.Name = "My Test Farm";
var updatedFarm = farm.ToDatabase();
updatedFarm.ToCache();

What about this?
public class KeyGeneratorVisitor : ExpressionVisitor
{
protected override Expression VisitParameter(ParameterExpression node)
{
return Expression.Parameter(node.Type, node.Type.Name);
}
protected override Expression VisitMember(MemberExpression node)
{
if (CanBeEvaluated(node))
{
return Expression.Constant(Evaluate(node));
}
else
{
return base.VisitMember(node);
}
}
private static bool CanBeEvaluated(MemberExpression exp)
{
while (exp.Expression.NodeType == ExpressionType.MemberAccess)
{
exp = (MemberExpression) exp.Expression;
}
return (exp.Expression.NodeType == ExpressionType.Constant);
}
private static object Evaluate(Expression exp)
{
if (exp.NodeType == ExpressionType.Constant)
{
return ((ConstantExpression) exp).Value;
}
else
{
MemberExpression mexp = (MemberExpression) exp;
object value = Evaluate(mexp.Expression);
FieldInfo field = mexp.Member as FieldInfo;
if (field != null)
{
return field.GetValue(value);
}
else
{
PropertyInfo property = (PropertyInfo) mexp.Member;
return property.GetValue(value, null);
}
}
}
}
This will replace the complex constant expressions to their original values as well as the parameter names to their type names. So just have to create a new KeyGeneratorVisitor instance and call its Visit or VisitAndConvert method with your expression.
Please note that the Expression.ToString method will be also invoked on your complex types, so either override their ToString methods or write a custom logic for them in the Evaluate method.

How about:
var call = expression.Body as MethodCallExpression;
if (call != null)
{
List<object> list = new List<object>();
foreach (Expression argument in call.Arguments)
{
object o = Expression.Lambda(argument, expression.Parameters).Compile().DynamicInvoke();
list.Add(o);
}
StringBuilder keyValue = new StringBuilder();
keyValue.Append(expression.Body.ToString());
list.ForEach(e => keyValue.Append(String.Format("_{0}", e.ToString())));
string key = keyValue.ToString();
}

Filtering an Objects Properties by Name

I am doing some simple sanity validation on various types. The current test I'm working on is checking to make sure their properties are populated. In this case, populated is defined as not null, having a length greater than zero (if a string), or not equal to 0 (if an integer).
The "tricky" part of this test is that some properties are immune to this check. Right now I use a giant if statement that weeds out properties that don't need to be checked.
//Gets all the properties of the currect feature.
System.Reflection.PropertyInfo[] pi = t.GetProperties();
for(int i = 0; i < pi.Length; i++)
{
if(!pi[i].Name.Equals("PropertyOne")
&& !pi[i].Name.Equals("PropertyTwo")
&& !pi[i].Name.Equals("PropertyThree")
//... repeat a bunch more times
&& !pi[i].Name.IndexOf("ValueOne") != -1
&& !pi[i].Name.IndexOf("ValueTwo") != -1
//... repeat a bunch more times
{
//Perform the validation check.
}
}
When profiling, I noticed the if statement is actually performing worse than the reflection (not that the reflection is blazing fast). Is there a more efficient way to filter the properties of several different types?
I've thought about a massive regular expression but I'm unsure on how to format it, plus it would probably be unreadable given its size. I've also considered storing the values in a List and then using Linq but I'm not sure how to handle the cases that use String.IndexOf() to find if the property contains a certain value.
Thanks in advance.

Make a HashSet "exactNames" with PropertyOne, PropertyTwo etc, and then a List "partialNames" with ValueOne, ValueTwo etc. Then:
var matchingProperties = pi.Where(exactNames.Contains(pi.Name) ||
partialNames.Any(name => pi.Name.Contains(name));
foreach (PropertyInfo property in matchingProperties)
{
// Stuff
}
(Odd indenting just to avoid wrapping.)
Note that you could cache the set of properties to validate on a per-type basis, so you only need to go through this check once per type.

Your idea help speed up my program, thank you. However, you had some syntax issues, plus you were matching items found in the lists and I needed items not in the list. Here is the code I ended up using.
List<System.Reflection.PropertyInfo> pi = type.GetProperties().ToList();
var matchingProperties = pi.Where( prop => !PropertyExclusionSet.Contains( prop.Name )
&& !PropertiesPartialSet.Any( name => prop.Name.Contains( name ) ) );

You might consider decorating your properties with attributes that tell what action needs to be done to them.
public class MyClass {
[CheckMe]
public int PropertyOne { get; set; }
[DontCheckMe]
public int PropertyTwo { get; set; }
}