Develop Reference

Simple NUnit test fail because exception is not thrown ( out of the test is thrown)

I'm creating an extension generic method Apply, what it does is not important, the major problem is that I cant understand how to test it, a simple test like ApplyOnNullFunctionThrow fail to say that no exception is thrown, but if I run a "manual test" from main it throws
The main:
class Program
{
static void Main(string[] args)
{
foreach (var i in first.Apply(second,f)) Console.WriteLine("{0}", i);
Console.ReadLine();
}
}
The extension method:
public static class Applier
{
public static IEnumerable<T> Apply<T>(this IEnumerable<T> first, IEnumerable<T> second,Func<T,T,T> f)
{
if(f == null ) throw new ArgumentNullException();
if (null==first || null==second) throw new ArgumentNullException();
var x=second.GetEnumerator();
foreach (var item in first)
{
yield return x.MoveNext() ? f(item, x.Current) : f(item, default);
}
while (x.MoveNext())
{
yield return f(default, x.Current);
}
}
}
The test:
class ApplierTests
{
[TestCase(new[]{1,2,3},new[]{10,20,30},null)]
public void ApplyOnNullFunctionThrow<T>(IEnumerable<T> first, IEnumerable<T> second,Func<T,T,T> f)
{
Assert.Throws<ArgumentNullException>(() => first.Apply(second, f));
}
}

In your Apply method you have yield keyword which means that the elements of the sequence will be returned one at a time. So when you just call the method with yield keyword, under the hood it does nothing, but just returns an iterator (IEnumerable<T>) which you can use in, for example, foreach. The bottom line here is that the code of your original Apply method isn't executed until MoveNext is called on the iterator returned, therefore it doesn't throw the exception. To fix this you may want to have a wrapper which is without yield keyword:
public static IEnumerable<T> Apply<T>(this IEnumerable<T> first, IEnumerable<T> second,Func<T,T,T> f)
{
if(f == null ) throw new ArgumentNullException();
if (null==first || null==second) throw new ArgumentNullException();
return ApplyInternal(first, second, f);
}
private static IEnumerable<T> ApplyInternal<T>(...) {
var x=second.GetEnumerator();
foreach (var item in first)
{
yield return x.MoveNext() ? f(item, x.Current) : f(item, default);
}
while (x.MoveNext())
{
yield return f(default, x.Current);
}
}
Instead of creating a private method you also can use C# 7 feature called local functions as alternative
See also, the exception is thrown when you call the method in Main, because you use the result of the method in the foreach statement which is compiled into repeated calls to MoveNext.

Calling method with IEnumerable<T> sequence as argument, if that sequence is not empty

I have method Foo, which do some CPU intensive computations and returns IEnumerable<T> sequence. I need to check, if that sequence is empty. And if not, call method Bar with that sequence as argument.
I thought about three approaches...
Check, if sequence is empty with Any(). This is ok, if sequence is really empty, which will be case most of the times. But it will have horrible performance, if sequence will contains some elements and Foo will need them compute again...
Convert sequence to list, check if that list it empty... and pass it to Bar. This have also limitation. Bar will need only first x items, so Foo will be doing unnecessary work...
Check, if sequence is empty without actually reset the sequence. This sounds like win-win, but I can't find any easy build-in way, how to do it. So I create this obscure workaround and wondering, whether this is really a best approach.
Condition
var source = Foo();
if (!IsEmpty(ref source))
Bar(source);
with IsEmpty implemented as
bool IsEmpty<T>(ref IEnumerable<T> source)
{
var enumerator = source.GetEnumerator();
if (enumerator.MoveNext())
{
source = CreateIEnumerable(enumerator);
return false;
}
return true;
IEnumerable<T> CreateIEnumerable(IEnumerator<T> usedEnumerator)
{
yield return usedEnumerator.Current;
while (usedEnumerator.MoveNext())
{
yield return usedEnumerator.Current;
}
}
}
Also note, that calling Bar with empty sequence is not option...
EDIT:
After some consideration, best answer for my case is from Olivier Jacot-Descombes - avoid that scenario completely. Accepted solution answers this question - if it is really no other way.

I don't know whether your algorithm in Foo allows to determine if the enumeration will be empty without doing the calculations. But if this is the case, return null if the sequence would be empty:
public IEnumerable<T> Foo()
{
if (<check if sequence will be empty>) {
return null;
}
return GetSequence();
}
private IEnumerable<T> GetSequence()
{
...
yield return item;
...
}
Note that if a method uses yield return, it cannot use a simple return to return null. Therefore a second method is needed.
var sequence = Foo();
if (sequence != null) {
Bar(sequence);
}
After reading one of your comments
Foo need to initialize some resources, parse XML file and fill some HashSets, which will be used to filter (yield) returned data.
I suggest another approach. The time consuming part seems to be the initialization. To be able to separate it from the iteration, create a foo calculator class. Something like:
public class FooCalculator<T>
{
private bool _isInitialized;
private string _file;
public FooCalculator(string file)
{
_file = file;
}
private EnsureInitialized()
{
if (_isInitialized) return;
// Parse XML.
// Fill some HashSets.
_isInitialized = true;
}
public IEnumerable<T> Result
{
get {
EnsureInitialized();
...
yield return ...;
...
}
}
}
This ensures that the costly initialization stuff is executed only once. Now you can safely use Any().
Other optimizations are conceivable. The Result property could remember the position of the first returned element, so that if it is called again, it could skip to it immediately.

You would like to call some function Bar<T>(IEnumerable<T> source) if and only if the enumerable source contains at least one element, but you're running into two problems:
There is no method T Peek() in IEnumerable<T> so you would need to actually begin to evaluate the enumerable to see if it's nonempty, but...
You don't want to even partially double-evaluate the enumerable since setting up the enumerable might be expensive.
In that case your approach looks reasonable. You do, however, have some issues with your imlementation:
You need to dispose enumerator after using it.
As pointed out by Ivan Stoev in comments, if the Bar() method attempts to evaluate the IEnumerable<T> more than once (e.g. by calling Any() then foreach (...)) then the results will be undefined because usedEnumerator will have been exhausted by the first enumeration.
To resolve these issues, I'd suggest modifying your API a little and create an extension method IfNonEmpty<T>(this IEnumerable<T> source, Action<IEnumerable<T>> func) that calls a specified method only if the sequence is nonempty, as shown below:
public static partial class EnumerableExtensions
{
public static bool IfNonEmpty<T>(this IEnumerable<T> source, Action<IEnumerable<T>> func)
{
if (source == null|| func == null)
throw new ArgumentNullException();
using (var enumerator = source.GetEnumerator())
{
if (!enumerator.MoveNext())
return false;
func(new UsedEnumerator<T>(enumerator));
return true;
}
}
class UsedEnumerator<T> : IEnumerable<T>
{
IEnumerator<T> usedEnumerator;
public UsedEnumerator(IEnumerator<T> usedEnumerator)
{
if (usedEnumerator == null)
throw new ArgumentNullException();
this.usedEnumerator = usedEnumerator;
}
public IEnumerator<T> GetEnumerator()
{
var localEnumerator = System.Threading.Interlocked.Exchange(ref usedEnumerator, null);
if (localEnumerator == null)
// An attempt has been made to enumerate usedEnumerator more than once;
// throw an exception since this is not allowed.
throw new InvalidOperationException();
yield return localEnumerator.Current;
while (localEnumerator.MoveNext())
{
yield return localEnumerator.Current;
}
}
IEnumerator IEnumerable.GetEnumerator()
{
return GetEnumerator();
}
}
}
Demo fiddle with unit tests here.

If you can change Bar then how about change it to TryBar that returns false when IEnumerable<T> was empty?
bool TryBar(IEnumerable<Foo> source)
{
var count = 0;
foreach (var x in source)
{
count++;
}
return count > 0;
}
If that doesn't work for you could always create your own IEnumerable<T> wrapper that caches values after they have been iterated once.

One improvement for your IsEmpty would be to check if source is ICollection<T>, and if it is, check .Count (also, dispose the enumerator):
bool IsEmpty<T>(ref IEnumerable<T> source)
{
if (source is ICollection<T> collection)
{
return collection.Count == 0;
}
var enumerator = source.GetEnumerator();
if (enumerator.MoveNext())
{
source = CreateIEnumerable(enumerator);
return false;
}
enumerator.Dispose();
return true;
IEnumerable<T> CreateIEnumerable(IEnumerator<T> usedEnumerator)
{
yield return usedEnumerator.Current;
while (usedEnumerator.MoveNext())
{
yield return usedEnumerator.Current;
}
usedEnumerator.Dispose();
}
}
This will work for arrays and lists.
I would, however, rework IsEmpty to return:
IEnumerable<T> NotEmpty<T>(IEnumerable<T> source)
{
if (source is ICollection<T> collection)
{
if (collection.Count == 0)
{
return null;
}
return source;
}
var enumerator = source.GetEnumerator();
if (enumerator.MoveNext())
{
return CreateIEnumerable(enumerator);
}
enumerator.Dispose();
return null;
IEnumerable<T> CreateIEnumerable(IEnumerator<T> usedEnumerator)
{
yield return usedEnumerator.Current;
while (usedEnumerator.MoveNext())
{
yield return usedEnumerator.Current;
}
usedEnumerator.Dispose();
}
}
Now, you would check if it returned null.

The accepted answer is probably the best approach but, based on, and I quote:
Convert sequence to list, check if that list it empty... and pass it to Bar. This have also limitation. Bar will need only first x items, so Foo will be doing unnecessary work...
Another take would be creating an IEnumerable<T> that partially caches the underlying enumeration. Something along the following lines:
interface IDisposableEnumerable<T>
:IEnumerable<T>, IDisposable
{
}
static class PartiallyCachedEnumerable
{
public static IDisposableEnumerable<T> Create<T>(
IEnumerable<T> source,
int cachedCount)
{
if (source == null)
throw new NullReferenceException(
nameof(source));
if (cachedCount < 1)
throw new ArgumentOutOfRangeException(
nameof(cachedCount));
return new partiallyCachedEnumerable<T>(
source, cachedCount);
}
private class partiallyCachedEnumerable<T>
: IDisposableEnumerable<T>
{
private readonly IEnumerator<T> enumerator;
private bool disposed;
private readonly List<T> cache;
private readonly bool hasMoreItems;
public partiallyCachedEnumerable(
IEnumerable<T> source,
int cachedCount)
{
Debug.Assert(source != null);
Debug.Assert(cachedCount > 0);
enumerator = source.GetEnumerator();
cache = new List<T>(cachedCount);
var count = 0;
while (enumerator.MoveNext() &&
count < cachedCount)
{
cache.Add(enumerator.Current);
count += 1;
}
hasMoreItems = !(count < cachedCount);
}
public void Dispose()
{
if (disposed)
return;
enumerator.Dispose();
disposed = true;
}
public IEnumerator<T> GetEnumerator()
{
foreach (var t in cache)
yield return t;
if (disposed)
yield break;
while (enumerator.MoveNext())
{
yield return enumerator.Current;
cache.Add(enumerator.Current)
}
Dispose();
}
IEnumerator IEnumerable.GetEnumerator()
=> GetEnumerator();
}
}

Tentative locks in C#?

Suppose I'd like to allow parallel execution of some code, but need other code wait for all these operations to finish.
Let's imagine a softlock in addition to lock:
public static class MySimpleCache
{
private static readonly SynchronizedCollection<KeyValuePair<string, string>> Collection = new SynchronizedCollection<KeyValuePair<string, string>>();
public static string Get(string key, Func<string> getter)
{
// Allow parallel enumerations here,
// but force modifications to the collections to wait.
softlock(Collection.SyncRoot)
{
if (Collection.Any(kvp => kvp.Key == key))
{
return Collection.First(kvp => kvp.Key == key).Value;
}
}
var data = getter();
// Wait for previous soft-locks before modifying the collection and let subsequent softlocks wait
lock (Collection.SyncRoot)
{
Collection.Add(new KeyValuePair<string, string>(key, data));
}
return data;
}
}
Is there any design-pattern or language/framework features in C#/.NET to achieve this in a straightforward and reliable fashion, or would one have to implement this from the ground up?
I'm currently limited to .NET 3.5 and I'm mostly interested in the conceptual issue, not so much in other possible collections that might solve the example in itself.

In situations like this you can use a ReaderWriterLockSlim, it will allow multiple readers until someone wants to write, it then blocks all readers and only allows a single writer through.
public static class MySimpleCache
{
private static readonly SynchronizedCollection<KeyValuePair<string, string>> Collection = new SynchronizedCollection<KeyValuePair<string, string>>();
private static readonly ReaderWriterLockSlim Lock = new ReaderWriterLockSlim();
public static string Get(string key, Func<string> getter)
{
//This allows multiple readers to run concurrently.
Lock.EnterReadLock();
try
{
var result = Collection.FirstOrDefault(kvp => kvp.Key == key);
if (!Object.Equals(result, default(KeyValuePair<string, string>)))
{
return result.Value;
}
}
finally
{
Lock.ExitReadLock();
}
var data = getter();
//This blocks all future EnterReadLock(), once all finish it allows the function to continue
Lock.EnterWriteLock();
try
{
Collection.Add(new KeyValuePair<string, string>(key, data));
return data;
}
finally
{
Lock.ExitWriteLock();
}
}
}
However, you may want to check to see while you where waiting to take the write lock someone else may have entered the record in to the cache, in that case you can use a EnterUpgradeableReadLock(), this allows unlimited people to be inside EnterReadLock() but only a single person can be in the upgrade lock (and there will still be no write locks). The upgrade-able lock is useful when you know you will likely be writing but there is a opportunity to not write.
public static class MySimpleCache
{
private static readonly SynchronizedCollection<KeyValuePair<string, string>> Collection = new SynchronizedCollection<KeyValuePair<string, string>>();
private static readonly ReaderWriterLockSlim Lock = new ReaderWriterLockSlim();
public static string Get(string key, Func<string> getter)
{
//This allows multiple readers to run concurrently.
Lock.EnterReadLock();
try
{
var result = Collection.FirstOrDefault(kvp => kvp.Key == key);
if (!Object.Equals(result, default(KeyValuePair<string, string>)))
{
return result.Value;
}
}
finally
{
Lock.ExitReadLock();
}
//This allows unlimited EnterReadLock to run concurrently, but only one thread can be in upgrade mode, other threads will block.
Lock.EnterUpgradeableReadLock();
try
{
//We need to check to see if someone else filled the cache while we where waiting.
var result = Collection.FirstOrDefault(kvp => kvp.Key == key);
if (!Object.Equals(result, default(KeyValuePair<string, string>)))
{
return result.Value;
}
var data = getter();
//This blocks all future EnterReadLock(), once all finish it allows the function to continue
Lock.EnterWriteLock();
try
{
Collection.Add(new KeyValuePair<string, string>(key, data));
return data;
}
finally
{
Lock.ExitWriteLock();
}
}
finally
{
Lock.ExitUpgradeableReadLock();
}
}
}
P.S. You mentioned in a comment that the value could be null so FirstOrDefault() would not work. In that case use a extension method to make a TryFirst() function.
public static class ExtensionMethods
{
public static bool TryFirst<T>(this IEnumerable<T> #this, Func<T, bool> predicate, out T result)
{
foreach (var item in #this)
{
if (predicate(item))
{
result = item;
return true;
}
}
result = default(T);
return false;
}
}
//Used like
Lock.EnterReadLock();
try
{
KeyValuePair<string, string> result;
bool found = Collection.TryFirst(kvp => kvp.Key == key, out result);
if (found)
{
return result.Value;
}
}
finally
{
Lock.ExitReadLock();
}

How to pass method on LINQ-generated object as delegate?

I have the following method (where SetTypes is an enum):
public IEnumerable<Chart> RetrieveCharts(IEnumerable<string> filters, SetTypes set)
{
switch (set)
{
case SetTypes.Set: return GetChartsSet(filters);
case SetTypes.Subset: return GetChartsSubset(filters);
// Other cases here...
default:
throw new NotImplementedException();
}
}
I am duplicating code across my method calls ... the only thing that's changing is the method call on the HashSet instance:
private IEnumerable<Chart> GetChartsSet(IEnumerable<string> filters)
{
using (var db = new ChartContext())
{
return db.Charts.Where(c => new HashSet<string>(c.ProductFilters.Select(f => f.Filter)).SetEquals(filters)).Select(c => c);
}
}
private IEnumerable<Chart> GetChartsSubset(IEnumerable<string> filters)
{
using (var db = new ChartContext())
{
return db.Charts.Where(c => new HashSet<string>(c.ProductFilters.Select(f => f.Filter)).IsProperSubsetOf(filters)).Select(c => c);
}
}
// More duplicated methods follow...
Instead of having multiple methods simply to call a different method on HashSet, I'd prefer a consolidated method which takes the HashSet method as an argument - but I'm not sure how to proceed. How do I do this?
Bonus points if you also tell me how to get rid of that switch statement. :)

You could make a common implementation that takes an additional Predicate<HashSet<string>> on the set:
private IEnumerable<Chart> GetWithPredicate(Predicate<HashSet<string>> pred) {
using (var db = new ChartContext()) {
return db.Charts
.Where(c => pred(new HashSet<string>(c.ProductFilters.Select(f => f.Filter))))
.Select(c => c);
}
}
and then reuse it in different methods:
private IEnumerable<Chart> GetChartsSet(IEnumerable<string> filters) {
return GetWithPredicate(s => s.SetEquals(filters));
}
private IEnumerable<Chart> GetChartsSubset(IEnumerable<string> filters) {
return GetWithPredicate(s => s.IsProperSubsetOf(filters));
}

IQueryable failback to IEnumerable

Having IQueryable<T> returned from Linq-2-SQL data context is it possible to wrap it with custom IQueryable<T> implementation that on execution stage (for instance, while enumeration) will handle underlying IQueryProvider's NotSupportedException and failback to enumerating whole data context and applying Expression using in-memory IQueryProvider (System.Linq.EnumerableQuery<T>)?
Code example:
IQueryable<User> users = usersRepository.GetAll();
// linq-to-sql can handle it
users.Where(u => u.Id > 10).ToList();
// linq-to-sql IQueryProvider will throw NotSupportedException in run-time,
// because ComplexFilter body was not captured as Expression
users.Where(u => ComplexFilter(u)).ToList();
// will succeed, because EnumerableQuery do not need Expression Tree
users.AsEnumerable().Where(u => ComplexFilter(u)).ToList();
public static bool ComplexFilter(User user)
{
return user.Id + user.Name.Length > 12;
}
I looking for way to wrap IQueryable that will automatically failback to IEnumerable if underlying IQueryProvider throws NotSupportedException:
IQueryable<User> safeUsersProxy = new SafeUsersProxy<User>(users);
// I want it to try underlying linq-to-sql provider first
// and then automatically failback to IEnumerable and return
// the result anyway (do not care about whole data enumerating in such cases)
safeUsersProxy.Where(u => ComplexFilter(u)).ToList();
The point
This will allow convenient way to execute all queries on DB level whenever possible, and only if Linq-to-SQL was unable to convert Expression into the SQL use slow EnumerableQuery fetching the whole dataset.

Now, I always thought that if someone asks you some rope to hang himself you should only ask two questions, "are you sure?" and "how much rope do you need?". So you ask me for some rope, you told us you are sure you want this rope, so who am I to not give you this rope?
This is what you asked... Let's call it a v0.5 . It even has a codename: Armed and Dangerous. I've even given to v0.1 a codename: Running with Scissors. It is disponible at http://pastebin.com/6qLs8TPt . (the v0.2 is Lost in Space, because I've forgot it on another computer :-) ). I've added to the v0.3 Playing with Fire http://pastebin.com/pRbKt1Z2 a simple logger and a Stack Trace augmenter, and to the v0.4 Crossing without Looking http://pastebin.com/yEhc9vjg a little EF compatibility.
I checked it with simple queries and Join(s). It seems to work. Much could be added. For example an "intelligent" projector that analyzes the Select that probably is removed from the query and tries to rebuild it... But this can be a good start. There is a small problem: as I've written in a comment, depending on the presence/not presence of a Select, the LINQ-to-SQL loads the returned objects in its Object Tracker. If in the "original" query there is a Select (so no object tracking), and I remove it to execute it locally, then the full object will be loaded, and the object tracking will be turned "on". Clearly you can context.ObjectTrackingEnabled = false.
Now it should be compatible with simple EF queries. Not compatible with AsNoTracking()/Include().
The NotSupportedException is tracked in an Exception property. Note that this property is modified only in the most "external" IQueryable. So
// Optional :-) You can ignore the logger and live happy...
// Not optional: you can live happy very far from me!
SafeQueryable.Logger = (iqueriable, expression, e) => Console.WriteLine(e);
var q1 = context.Items.AsSafe();
var q2 = q1.Where(x => x.ID.GetHashCode() > 0);
var q3 = q2.Select(x => x.ID);
var ex = ((ISafeQueryable)q3).Exception;
How to use it? There is a single extension method, AsSafe(). You use it on a IQueryable<T> and then you execute the query... And live happy... Playing with Fire! :-)
Examples of use:
// Queries that are NotSupportedException with LINQ-to-SQL
var q1 = context.Items.AsSafe().Where(x => x.ID.GetHashCode() > 0).Take(2).Select(x => x.ID);
var q2 = context.Items.Where(x => x.ID.GetHashCode() > 0).AsSafe().Take(2).Select(x => x.ID);
var q3 = context.Items.Where(x => x.ID.GetHashCode() > 0).Take(2).AsSafe().Select(x => x.ID);
var q4 = context.Items.Where(x => x.ID.GetHashCode() > 0).Take(2).Select(x => x.ID).AsSafe();
//// Queries that are OK with LINQ-to-SQL
//var q1 = context.Items.AsSafe().Where(x => x.ID > 0).Take(2).Select(x => x.ID);
//var q2 = context.Items.Where(x => x.ID > 0).AsSafe().Take(2).Select(x => x.ID);
//var q3 = context.Items.Where(x => x.ID > 0).Take(2).AsSafe().Select(x => x.ID);
//var q4 = context.Items.Where(x => x.ID > 0).Take(2).Select(x => x.ID).AsSafe();
var r1 = q1.ToList();
var r2 = q2.First();
var r3 = q3.Max();
// The Aggregate isn't normally supported by LINQ-to-SQL
var r4 = q4.Aggregate((x, y) => x + y);
var ex1 = ((ISafeQueryable)q1).Exception;
var ex2 = ((ISafeQueryable)q2).Exception;
var ex3 = ((ISafeQueryable)q3).Exception;
var ex4 = ((ISafeQueryable)q4).Exception;
As you can see, you can use the AsSafe() at any step, and it will work. This happens because the query is wrapped by AsSafe() in a SafeQueryable<T> that is both a IQueryProvider and a IQueryable<T> (like the classes of LINQ-to-SQL). In this way every other IQueryable method you call on it will produce other SafeQueryable<T> objects (so it self-reproduce :-) )
using System;
using System.Collections;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Linq.Expressions;
using System.Reflection;
/// <summary>
/// v0.5 Codename: Armed and Dangerous
///
/// (previous version v0.1 Codename: Running with Scissors: http://pastebin.com/6qLs8TPt)
/// (previous version v0.2 Codename: Lost in Space: lost in space :-) )
/// (previous version v0.3 Codename: Playing with Fire: http://pastebin.com/pRbKt1Z2)
/// (previous version v0.4 Codename: Crossing without Looking: http://pastebin.com/yEhc9vjg)
///
/// Support class with an extension method to make "Safe" an IQueryable
/// or IQueryable<T>. Safe as "I work for another company, thousand
/// of miles from you. I do hope I won't ever buy/need something from
/// your company".
/// The Extension methods wraps a IQueryable in a wrapper that then can
/// be used to execute a query. If the original Provider doesn't suppport
/// some methods, the query will be partially executed by the Provider
/// and partially executed locally.
///
/// Minimal support for EF.
///
/// Note that this **won't** play nice with the Object Tracking!
///
/// Not suitable for programmers under 5 years (of experience)!
/// Dangerous if inhaled or executed.
/// </summary>
public static class SafeQueryable
{
/// <summary>
/// Optional logger to log the queries that are "corrected. Note that
/// there is no "strong guarantee" that the IQueriable (that is also
/// an IQueryProvider) is executing its (as in IQueriable.Expression)
/// Expression, so an explicit Expression parameter is passed. This
/// because the IQueryProvider.Execute method receives an explicit
/// expression parameter. Clearly there is a "weak guarantee" that
/// unless you do "strange things" this won't happen :-)
/// </summary>
public static Action<IQueryable, Expression, NotSupportedException> Logger { get; set; }
/// <summary>
/// Return a "Safe" IQueryable<T>.
/// </summary>
/// <typeparam name="T"></typeparam>
/// <param name="source"></param>
/// <returns></returns>
public static IQueryable<T> AsSafe<T>(this IQueryable<T> source)
{
if (source is SafeQueryable<T>)
{
return source;
}
return new SafeQueryable<T>(source);
}
}
/// <summary>
/// Simple interface useful to collect the Exception, or to recognize
/// a SafeQueryable<T>.
/// </summary>
public interface ISafeQueryable
{
NotSupportedException Exception { get; }
}
/// <summary>
/// "Safe" wrapper around a IQueryable<T;>
/// </summary>
/// <typeparam name="T"></typeparam>
public class SafeQueryable<T> : IOrderedQueryable<T>, IQueryProvider, ISafeQueryable
{
protected static readonly FieldInfo StackTraceStringField = typeof(Exception).GetField("_stackTraceString", BindingFlags.Instance | BindingFlags.NonPublic);
// The query. Note that it can be "transformed" to a "safe" version
// of itself. When it happens, IsSafe becomes true
public IQueryable<T> Query { get; protected set; }
// IsSafe means that the query has been "corrected" if necessary and
// won't throw a NotSupportedException
protected bool IsSafe { get; set; }
// Logging of the "main" NotSupportedException.
public NotSupportedException Exception { get; protected set; }
public SafeQueryable(IQueryable<T> query)
{
Query = query;
}
/* IQueryable<T> */
public IEnumerator<T> GetEnumerator()
{
if (IsSafe)
{
return Query.GetEnumerator();
}
return new SafeEnumerator(this);
}
IEnumerator IEnumerable.GetEnumerator()
{
return GetEnumerator();
}
public Type ElementType
{
get { return Query.ElementType; }
}
public Expression Expression
{
get { return Query.Expression; }
}
public IQueryProvider Provider
{
get { return this; }
}
/* IQueryProvider */
public IQueryable<TElement> CreateQuery<TElement>(Expression expression)
{
return CreateQueryImpl<TElement>(expression);
}
public IQueryable CreateQuery(Expression expression)
{
Type iqueryableArgument = GetIQueryableTypeArgument(expression.Type);
MethodInfo createQueryImplMethod = typeof(SafeQueryable<T>)
.GetMethod("CreateQueryImpl", BindingFlags.Instance | BindingFlags.NonPublic)
.MakeGenericMethod(iqueryableArgument);
return (IQueryable)createQueryImplMethod.Invoke(this, new[] { expression });
}
public TResult Execute<TResult>(Expression expression)
{
return ExecuteImpl<TResult>(expression);
}
public object Execute(Expression expression)
{
Type iqueryableArgument = GetIQueryableTypeArgument(expression.Type);
MethodInfo executeImplMethod = typeof(SafeQueryable<T>)
.GetMethod("ExecuteImpl", BindingFlags.Instance | BindingFlags.NonPublic)
.MakeGenericMethod(iqueryableArgument);
return executeImplMethod.Invoke(this, new[] { expression });
}
/* Implementation methods */
// Gets the T of IQueryablelt;T>
protected static Type GetIQueryableTypeArgument(Type type)
{
IEnumerable<Type> interfaces = type.IsInterface ?
new[] { type }.Concat(type.GetInterfaces()) :
type.GetInterfaces();
Type argument = (from x in interfaces
where x.IsGenericType
let gt = x.GetGenericTypeDefinition()
where gt == typeof(IQueryable<>)
select x.GetGenericArguments()[0]).FirstOrDefault();
return argument;
}
protected IQueryable<TElement> CreateQueryImpl<TElement>(Expression expression)
{
return new SafeQueryable<TElement>(Query.Provider.CreateQuery<TElement>(expression));
}
protected TResult ExecuteImpl<TResult>(Expression expression)
{
if (IsSafe && Query.Expression == expression)
{
TResult result = Query.Provider.Execute<TResult>(expression);
return result;
}
try
{
// Note that thanks to how everything knits together, if you
// call query1.First(); query1.First(); the second call will
// get to use the query cached by the first one (technically
// the cached query will be only the "query1" part)
// We try executing it directly
TResult result = Query.Provider.Execute<TResult>(expression);
// Success!
if (!IsSafe && CanCache(expression, true))
{
IsSafe = true;
}
return result;
}
catch (NotSupportedException e1)
{
// Clearly there was a NotSupportedException :-)
Tuple<IEnumerator<T>, bool, TResult> result = HandleEnumerationFailure<TResult>(e1, expression, true);
if (result == null)
{
throw;
}
// Success!
return result.Item3;
}
}
// Is used both indirectly by GetEnumerator() and by Execute<>.
// The returned Tuple<,,> has the first two elements that are valid
// when used by the GetEnumerator() and the last that is valid
// when used by Execute<>.
protected Tuple<IEnumerator<T>, bool, TResult> HandleEnumerationFailure<TResult>(NotSupportedException e1, Expression expression, bool singleResult)
{
// We "augment" the exception with the full stack trace
AugmentStackTrace(e1, 3);
if (SafeQueryable.Logger != null)
{
SafeQueryable.Logger(this, expression, e1);
}
// We save this first exception
Exception = e1;
{
var query = Query;
MethodInfo executeSplittedMethod = typeof(SafeQueryable<T>).GetMethod("ExecuteSplitted", BindingFlags.Instance | BindingFlags.NonPublic);
MethodCallExpression call;
Expression innerExpression = expression;
Type iqueryableArgument;
// We want to check that there is a MethodCallExpression with
// at least one argument, and that argument is an Expression
// of type IQueryable<iqueryableArgument>, and we save the
// iqueryableArgument
while ((call = innerExpression as MethodCallExpression) != null &&
call.Arguments.Count > 0 &&
(innerExpression = call.Arguments[0] as Expression) != null &&
(iqueryableArgument = GetIQueryableTypeArgument(innerExpression.Type)) != null)
{
try
{
Tuple<IEnumerator<T>, bool, TResult> result2 = (Tuple<IEnumerator<T>, bool, TResult>)executeSplittedMethod.MakeGenericMethod(iqueryableArgument, typeof(TResult)).Invoke(this, new object[] { expression, call, innerExpression, singleResult });
return result2;
}
catch (TargetInvocationException e2)
{
if (!(e2.InnerException is NotSupportedException))
{
throw;
}
}
}
return null;
}
}
// Is used both indirectly by GetEnumerator() and by Execute<>.
// The returned Tuple<,,> has the first two elements that are valid
// when used by the GetEnumerator() and the last that is valid
// when used by Execute<>.
protected Tuple<IEnumerator<T>, bool, TResult> ExecuteSplitted<TInner, TResult>(Expression expression, MethodCallExpression call, Expression innerExpression, bool singleResult)
{
// The NotSupportedException should happen here
IQueryable<TInner> innerQueryable = Query.Provider.CreateQuery<TInner>(innerExpression);
// We try executing it directly
IEnumerator<TInner> innerEnumerator = innerQueryable.GetEnumerator();
bool moveNextSuccess = innerEnumerator.MoveNext();
IEnumerator<T> enumerator;
TResult singleResultValue;
// Success!
{
// Now we wrap the partially used enumerator in an
// EnumerableFromStartedEnumerator
IEnumerable<TInner> innerEnumerable = new EnumerableFromStartedEnumerator<TInner>(innerEnumerator, moveNextSuccess, innerQueryable);
// Then we apply an AsQueryable, that does some magic
// to make the query appear to be a Queryable
IQueryable<TInner> innerEnumerableAsQueryable = Queryable.AsQueryable(innerEnumerable);
// We rebuild a new expression by changing the "old"
// inner parameter of the MethodCallExpression with the
// queryable we just built
var arguments = call.Arguments.ToArray();
arguments[0] = Expression.Constant(innerEnumerableAsQueryable);
MethodCallExpression call2 = Expression.Call(call.Object, call.Method, arguments);
Expression expressionWithFake = new SimpleExpressionReplacer(call, call2).Visit(expression);
// We "execute" locally the whole query through a second
// "outer" instance of the EnumerableQuery (this class is
// the class that "implements" the "fake-magic" of
// AsQueryable)
IQueryable<T> queryable = new EnumerableQuery<T>(expressionWithFake);
if (singleResult)
{
enumerator = null;
moveNextSuccess = false;
singleResultValue = queryable.Provider.Execute<TResult>(queryable.Expression);
}
else
{
enumerator = queryable.GetEnumerator();
moveNextSuccess = enumerator.MoveNext();
singleResultValue = default(TResult);
}
}
// We could enter here with a new query from Execute<>(),
// with IsSafe == true . It would be useless to try to cache
// that query.
if (!IsSafe && CanCache(expression, singleResult))
{
Stopwatch sw = Stopwatch.StartNew();
// We redo the same things to create a second copy of
// the query that is "complete", not partially
// enumerated. This second copy will be cached in the
// SafeQueryable<T>.
// Note that forcing the Queryable.AsQueryable to not
// "recast" the query to the original IQueryable<T> is
// quite complex :-) We have to
// .AsEnumerable().Select(x => x) .
IEnumerable<TInner> innerEnumerable = innerQueryable.AsEnumerable().Select(x => x);
IQueryable<TInner> innerEnumerableAsQueryable = Queryable.AsQueryable(innerEnumerable);
// Note that we cache the SafeQueryable<>.Expression!
var arguments = call.Arguments.ToArray();
arguments[0] = Expression.Constant(innerEnumerableAsQueryable);
MethodCallExpression call2 = Expression.Call(call.Object, call.Method, arguments);
Expression expressionWithFake = new SimpleExpressionReplacer(call, call2).Visit(Expression);
IQueryable<T> queryable = new EnumerableQuery<T>(expressionWithFake);
// Now the SafeQueryable<T> has a query that *just works*
Query = queryable;
IsSafe = true;
sw.Stop();
Console.WriteLine(sw.ElapsedTicks);
}
return Tuple.Create(enumerator, moveNextSuccess, singleResultValue);
}
protected bool CanCache(Expression expression, bool singleResult)
{
// GetEnumerator() doesn't permit changing the query
if (!singleResult)
{
return true;
}
// The expression is equal to the one in Query.Expression
// (should be very rare!)
if (Query.Expression == expression)
{
return true;
}
MethodCallExpression call;
Expression innerExpression = expression;
Type iqueryableArgument;
// We walk back the expression to see if a smaller part of it is
// the "original" Query.Expression . This happens for example
// when one of the operators that returns a single value
// (.First(), .FirstOrDefault(), .Single(), .SingleOrDefault(),
// .Any(), .All()., .Min(), .Max(), ...) are used.
while ((call = innerExpression as MethodCallExpression) != null &&
call.Arguments.Count > 0 &&
(innerExpression = call.Arguments[0] as Expression) != null &&
(iqueryableArgument = GetIQueryableTypeArgument(innerExpression.Type)) != null)
{
if (Query.Expression == innerExpression)
{
return true;
}
}
return false;
}
// The StackTrace of an Exception "stops" at the catch. This method
// "augments" it to include the full stack trace.
protected static void AugmentStackTrace(Exception e, int skipFrames = 2)
{
// Playing with a private field here. Don't do it at home :-)
// If not present, do nothing.
if (StackTraceStringField == null)
{
return;
}
string stack1 = e.StackTrace;
string stack2 = new StackTrace(skipFrames, true).ToString();
string stack3 = stack1 + stack2;
StackTraceStringField.SetValue(e, stack3);
}
/* Utility classes */
// An IEnumerator<T> that applies the AsSafe() paradigm, knowing that
// normally the exception happens only on the first MoveFirst().
protected class SafeEnumerator : IEnumerator<T>
{
protected readonly SafeQueryable<T> SafeQueryable_;
protected IEnumerator<T> Enumerator { get; set; }
public SafeEnumerator(SafeQueryable<T> safeQueryable)
{
SafeQueryable_ = safeQueryable;
}
public T Current
{
get
{
return Enumerator != null ? Enumerator.Current : default(T);
}
}
public void Dispose()
{
if (Enumerator != null)
{
Enumerator.Dispose();
}
}
object IEnumerator.Current
{
get { return Current; }
}
public bool MoveNext()
{
// We handle exceptions only on first MoveNext()
if (Enumerator != null)
{
return Enumerator.MoveNext();
}
try
{
// We try executing it directly
Enumerator = SafeQueryable_.Query.GetEnumerator();
bool result = Enumerator.MoveNext();
// Success!
SafeQueryable_.IsSafe = true;
return result;
}
catch (NotSupportedException e1)
{
// Clearly there was a NotSupportedException :-)
Tuple<IEnumerator<T>, bool, T> result = SafeQueryable_.HandleEnumerationFailure<T>(e1, SafeQueryable_.Query.Expression, false);
if (result == null)
{
throw;
}
Enumerator = result.Item1;
return result.Item2;
}
}
public void Reset()
{
if (Enumerator != null)
{
Enumerator.Reset();
}
}
}
}
// A simple expression visitor to replace some nodes of an expression
// with some other nodes
public class SimpleExpressionReplacer : ExpressionVisitor
{
public readonly Dictionary<Expression, Expression> Replaces;
public SimpleExpressionReplacer(Dictionary<Expression, Expression> replaces)
{
Replaces = replaces;
}
public SimpleExpressionReplacer(IEnumerable<Expression> from, IEnumerable<Expression> to)
{
Replaces = new Dictionary<Expression, Expression>();
using (var enu1 = from.GetEnumerator())
using (var enu2 = to.GetEnumerator())
{
while (true)
{
bool res1 = enu1.MoveNext();
bool res2 = enu2.MoveNext();
if (!res1 || !res2)
{
if (!res1 && !res2)
{
break;
}
if (!res1)
{
throw new ArgumentException("from shorter");
}
throw new ArgumentException("to shorter");
}
Replaces.Add(enu1.Current, enu2.Current);
}
}
}
public SimpleExpressionReplacer(Expression from, Expression to)
{
Replaces = new Dictionary<Expression, Expression> { { from, to } };
}
public override Expression Visit(Expression node)
{
Expression to;
if (node != null && Replaces.TryGetValue(node, out to))
{
return base.Visit(to);
}
return base.Visit(node);
}
}
// Simple IEnumerable<T> that "uses" an IEnumerator<T> that has
// already received a MoveNext(). "eats" the first MoveNext()
// received, then continues normally. For shortness, both IEnumerable<T>
// and IEnumerator<T> are implemented by the same class. Note that if a
// second call to GetEnumerator() is done, the "real" IEnumerator<T> will
// be returned, not this proxy implementation.
public class EnumerableFromStartedEnumerator<T> : IEnumerable<T>, IEnumerator<T>
{
public readonly IEnumerator<T> Enumerator;
public readonly IEnumerable<T> Enumerable;
// Received by creator. Return value of MoveNext() done by caller
protected bool FirstMoveNextSuccessful { get; set; }
// The Enumerator can be "used" only once, then a new enumerator
// can be requested by Enumerable.GetEnumerator()
// (default = false)
protected bool Used { get; set; }
// The first MoveNext() has been already done (default = false)
protected bool DoneMoveNext { get; set; }
public EnumerableFromStartedEnumerator(IEnumerator<T> enumerator, bool firstMoveNextSuccessful, IEnumerable<T> enumerable)
{
Enumerator = enumerator;
FirstMoveNextSuccessful = firstMoveNextSuccessful;
Enumerable = enumerable;
}
public IEnumerator<T> GetEnumerator()
{
if (Used)
{
return Enumerable.GetEnumerator();
}
Used = true;
return this;
}
IEnumerator IEnumerable.GetEnumerator()
{
return GetEnumerator();
}
public T Current
{
get
{
// There are various school of though on what should
// happens if called before the first MoveNext() or
// after a MoveNext() returns false. We follow the
// "return default(TInner)" school of thought for the
// before first MoveNext() and the "whatever the
// Enumerator wants" for the after a MoveNext() returns
// false
if (!DoneMoveNext)
{
return default(T);
}
return Enumerator.Current;
}
}
public void Dispose()
{
Enumerator.Dispose();
}
object IEnumerator.Current
{
get
{
return Current;
}
}
public bool MoveNext()
{
if (!DoneMoveNext)
{
DoneMoveNext = true;
return FirstMoveNextSuccessful;
}
return Enumerator.MoveNext();
}
public void Reset()
{
// This will 99% throw :-) Not our problem.
Enumerator.Reset();
// So it is improbable we will arrive here
DoneMoveNext = true;
}
}
Note that there is a bug in the LINQ-to-SQL:
var q5 = context.Items.Where(x => x.ID > context.Nodis.Min(y => y.ID.GetHashCode()));
var r5 = q5.ToList();
This won't throw NotSupportedException but won't execute correctly. I think it could be a problem with many queries that use the context.SomeTable "internally" in the query.

If I'm not mistaken, you need to materialize the collection through .ToList () before using your method. Try to convert the collection into a list and store it in a variable. After that, try to use his method in the collection that was generated in this variable.

I don't think there's an automatic way to do this, other than writing your own ExtressionTree
But analyzing your code I think that what you need is to use an Extension Method to encapsulate your query
Something like this (I already tested the concept using LinqToSql and it supports the Length property):
public static class Extensions
{
public static IQueryable<User> WhereUserMatches(this IQueryable<User> source)
{
return source.Where(x => x.Id + x.Name.Length > 12);
}
}
Usage
var myFileterdUsers = users.WhereUserMatches();
And since you are working with IQueryable then your condition will be sent to the server not in memory
With this approach you can encapsulate your complex queries and actually execute them in the server instead of memory.
A word about hiding the Exception
If the query cannot be handled by the provider I wouldn't recommend you to hide the exception by falling back to use IEnumerable (in memory query) because this could cause undesired behavior in your application
I'd recommend you to be explicit and throw an exception if the underlying query provider cannot convert it to a valid ExpressionTree
Exceptions are good, they are our friends, they let us know if we are doing something wrong
Hiding the exception is the opposite and it's considered a bad practice, hiding the exception will give you the ilusion that your application works even when most likely it won't do exactly what you think it does