Develop Reference

using AsQueryable after ToList [duplicate]

I know some differences of LINQ to Entities and LINQ to Objects which the first implements IQueryable and the second implements IEnumerable and my question scope is within EF 5.
My question is what's the technical difference(s) of those 3 methods? I see that in many situations all of them work. I also see using combinations of them like .ToList().AsQueryable().
What do those methods mean, exactly?
Is there any performance issue or something that would lead to the use of one over the other?
Why would one use, for example, .ToList().AsQueryable() instead of .AsQueryable()?

There is a lot to say about this. Let me focus on AsEnumerable and AsQueryable and mention ToList() along the way.
What do these methods do?
AsEnumerable and AsQueryable cast or convert to IEnumerable or IQueryable, respectively. I say cast or convert with a reason:
When the source object already implements the target interface, the source object itself is returned but cast to the target interface. In other words: the type is not changed, but the compile-time type is.
When the source object does not implement the target interface, the source object is converted into an object that implements the target interface. So both the type and the compile-time type are changed.
Let me show this with some examples. I've got this little method that reports the compile-time type and the actual type of an object (courtesy Jon Skeet):
void ReportTypeProperties<T>(T obj)
{
Console.WriteLine("Compile-time type: {0}", typeof(T).Name);
Console.WriteLine("Actual type: {0}", obj.GetType().Name);
}
Let's try an arbitrary linq-to-sql Table<T>, which implements IQueryable:
ReportTypeProperties(context.Observations);
ReportTypeProperties(context.Observations.AsEnumerable());
ReportTypeProperties(context.Observations.AsQueryable());
The result:
Compile-time type: Table`1
Actual type: Table`1
Compile-time type: IEnumerable`1
Actual type: Table`1
Compile-time type: IQueryable`1
Actual type: Table`1
You see that the table class itself is always returned, but its representation changes.
Now an object that implements IEnumerable, not IQueryable:
var ints = new[] { 1, 2 };
ReportTypeProperties(ints);
ReportTypeProperties(ints.AsEnumerable());
ReportTypeProperties(ints.AsQueryable());
The results:
Compile-time type: Int32[]
Actual type: Int32[]
Compile-time type: IEnumerable`1
Actual type: Int32[]
Compile-time type: IQueryable`1
Actual type: EnumerableQuery`1
There it is. AsQueryable() has converted the array into an EnumerableQuery, which "represents an IEnumerable<T> collection as an IQueryable<T> data source." (MSDN).
What's the use?
AsEnumerable is frequently used to switch from any IQueryable implementation to LINQ to objects (L2O), mostly because the former does not support functions that L2O has. For more details see What is the effect of AsEnumerable() on a LINQ Entity?.
For example, in an Entity Framework query we can only use a restricted number of methods. So if, for example, we need to use one of our own methods in a query we would typically write something like
var query = context.Observations.Select(o => o.Id)
.AsEnumerable().Select(x => MySuperSmartMethod(x))
ToList – which converts an IEnumerable<T> to a List<T> – is often used for this purpose as well. The advantage of using AsEnumerable vs. ToList is that AsEnumerable does not execute the query. AsEnumerable preserves deferred execution and does not build an often useless intermediate list.
On the other hand, when forced execution of a LINQ query is desired, ToList can be a way to do that.
AsQueryable can be used to make an enumerable collection accept expressions in LINQ statements. See here for more details: Do i really need use AsQueryable() on collection?.
Note on substance abuse!
AsEnumerable works like a drug. It's a quick fix, but at a cost and it doesn't address the underlying problem.
In many Stack Overflow answers, I see people applying AsEnumerable to fix just about any problem with unsupported methods in LINQ expressions. But the price isn't always clear. For instance, if you do this:
context.MyLongWideTable // A table with many records and columns
.Where(x => x.Type == "type")
.Select(x => new { x.Name, x.CreateDate })
...everything is neatly translated into a SQL statement that filters (Where) and projects (Select). That is, both the length and the width, respectively, of the SQL result set are reduced.
Now suppose users only want to see the date part of CreateDate. In Entity Framework you'll quickly discover that...
.Select(x => new { x.Name, x.CreateDate.Date })
...is not supported (at the time of writing). Ah, fortunately there's the AsEnumerable fix:
context.MyLongWideTable.AsEnumerable()
.Where(x => x.Type == "type")
.Select(x => new { x.Name, x.CreateDate.Date })
Sure, it runs, probably. But it pulls the entire table into memory and then applies the filter and the projections. Well, most people are smart enough to do the Where first:
context.MyLongWideTable
.Where(x => x.Type == "type").AsEnumerable()
.Select(x => new { x.Name, x.CreateDate.Date })
But still all columns are fetched first and the projection is done in memory.
The real fix is:
context.MyLongWideTable
.Where(x => x.Type == "type")
.Select(x => new { x.Name, DbFunctions.TruncateTime(x.CreateDate) })
(But that requires just a little bit more knowledge...)
What do these methods NOT do?
Restore IQueryable capabilities
Now an important caveat. When you do
context.Observations.AsEnumerable()
.AsQueryable()
you will end up with the source object represented as IQueryable. (Because both methods only cast and don't convert).
But when you do
context.Observations.AsEnumerable().Select(x => x)
.AsQueryable()
what will the result be?
The Select produces a WhereSelectEnumerableIterator. This is an internal .Net class that implements IEnumerable, not IQueryable. So a conversion to another type has taken place and the subsequent AsQueryable can never return the original source anymore.
The implication of this is that using AsQueryable is not a way to magically inject a query provider with its specific features into an enumerable. Suppose you do
var query = context.Observations.Select(o => o.Id)
.AsEnumerable().Select(x => x.ToString())
.AsQueryable()
.Where(...)
The where condition will never be translated into SQL. AsEnumerable() followed by LINQ statements definitively cuts the connection with entity framework query provider.
I deliberately show this example because I've seen questions here where people for instance try to 'inject' Include capabilities into a collection by calling AsQueryable. It compiles and runs, but it does nothing because the underlying object does not have an Include implementation anymore.
Execute
Both AsQueryable and AsEnumerable don't execute (or enumerate) the source object. They only change their type or representation. Both involved interfaces, IQueryable and IEnumerable, are nothing but "an enumeration waiting to happen". They are not executed before they're forced to do so, for example, as mentioned above, by calling ToList().
That means that executing an IEnumerable obtained by calling AsEnumerable on an IQueryable object, will execute the underlying IQueryable. A subsequent execution of the IEnumerable will again execute the IQueryable. Which may be very expensive.
Specific Implementations
So far, this was only about the Queryable.AsQueryable and Enumerable.AsEnumerable extension methods. But of course anybody can write instance methods or extension methods with the same names (and functions).
In fact, a common example of a specific AsEnumerable extension method is DataTableExtensions.AsEnumerable. DataTable does not implement IQueryable or IEnumerable, so the regular extension methods don't apply.

ToList()
Execute the query immediately
AsEnumerable()
lazy (execute the query later)
Parameter: Func<TSource, bool>
Load EVERY record into application memory, and then handle/filter them. (e.g. Where/Take/Skip, it will select * from table1, into the memory, then select the first X elements) (In this case, what it did: Linq-to-SQL + Linq-to-Object)
AsQueryable()
lazy (execute the query later)
Parameter: Expression<Func<TSource, bool>>
Convert Expression into T-SQL (with the specific provider), query remotely and load result to your application memory.
That’s why DbSet (in Entity Framework) also inherits IQueryable to get the efficient query.
Do not load every record, e.g. if Take(5), it will generate select top 5 * SQL in the background. This means this type is more friendly to SQL Database, and that is why this type usually has higher performance and is recommended when dealing with a database.
So AsQueryable() usually works much faster than AsEnumerable() as it generate T-SQL at first, which includes all your where conditions in your Linq.

ToList() will being everything in memory and then you will be working on it.
so, ToList().where ( apply some filter ) is executed locally.
AsQueryable() will execute everything remotely i.e. a filter on it is sent to the database for applying.
Queryable doesn't do anything til you execute it. ToList, however executes immediately.
Also, look at this answer Why use AsQueryable() instead of List()?.
EDIT :
Also, in your case once you do ToList() then every subsequent operation is local including AsQueryable(). You can't switch to remote once you start executing locally.
Hope this makes it a little bit more clearer.

Encountered a bad performance on below code.
void DoSomething<T>(IEnumerable<T> objects){
var single = objects.First(); //load everything into memory before .First()
...
}
Fixed with
void DoSomething<T>(IEnumerable<T> objects){
T single;
if (objects is IQueryable<T>)
single = objects.AsQueryable().First(); // SELECT TOP (1) ... is used
else
single = objects.First();
}
For an IQueryable, stay in IQueryable when possible, try not be used like IEnumerable.
Update. It can be further simplified in one expression, thanks Gert Arnold.
T single = objects is IQueryable<T> q?
q.First():
objects.First();

Populate navigation properties when used in method passed to Where clause

I have an endpoint which should list all Candidatures filtered by search term, so what I do is to create a method which accepts candidature entity and searchTerm as params. Then I pass that method to Where clause, but the problem is that I got NullReferenceException because navigation properties are nulls. If I put statement inside Where clause instead of the method then it doesn't throw Exception. The question is how to fix this, but I want to keep the external method because there will be a lot more logic, but I need to have access to all navigation properties i.e. they should be populated.
if (!string.IsNullOrEmpty(searchTerm))
{
query = query.Where(c => FilterBySearchTerm(c, searchTerm));
}
var result = await query.Select(c => new CandidaturesResponseModel()
{
Id = c.Id,
Name = c.PrimaryMember.FullName, // that's filled
}).ToListAsync();
private bool FilterBySearchTerm(Candidature c, string searchTerm)
{
return c.PrimaryMember.FirstName.Contains(searchTerm); // here is the exception because PrimaryMember navigation property is null. So I want this to be filled.
}

The issue is that you're materializing the query by using your FilterBySearchTerm method. EF cannot translate random methods to SQL, so it has to go ahead and run the query, get the results back and then apply your Where. EF would actually throw an exception in the past, but EF Core handles this silently.
Anyways, once the query is run, you're done. Your filtering is happening in-memory, and at that point, without an Include, your related entities are not there to work with. Long and short, you'll need to build your filter in place (rather than using a separate method) in order for EF to be able to translate that to SQL.
An alternative approach which may serve you better is pass a queryable to your FilterBySearchTerm method. For example, instead of doing:
query = query.Where(c => FilterBySearchTerm(c, searchTerm));
Do
query = FilterBySearchTerm(query, searchTerm);
Then, inside FilterBySearchTerm, you can directly apply Where clauses to the passed in query. That allows you to build an actual query that EF can understand, while also encapsulating the logic.

Just use Include method to add PrimaryMember
query = query.Include(x=> x.PrimaryMember).Where(c => FilterBySearchTerm(c, searchTerm));

Linq/lambda toDictionary() throwing exception

I have a query that looks something like this
return (from item in context.MERCHANDISE_ITEMS
where item.ORGID == orgId
select new MerchandiseModel()
{
Description = item.DESCRIPTION,
...,
Images = context.MERCHANDISE_IMAGES.Where(i => i.ITEMID == item.ID).ToDictionary(i => i.ID, i => i.IMAGENAME)
}).FirstOrDefault();
Images is a Dictionary and MERCHANDISE_IMAGES.ID is int and MERCHANDISE_IMAGES.IMAGENAME is string.
This is throwing System.NotSupportedException, I've tried doing linq (no lambda), same exception.
Here is the exception:
LINQ to Entities does not recognize the method
'System.Collections.Generic.Dictionary`2[System.Int32,System.String] ToDictionary[MERCHANDISE_IMAGES,Int32,String](System.Collections.Generic.IEnumerable`1[Hylton.Infrastructure.EntityFrameworkORM.MERCHANDISE_IMAGES],
System.Func`2[Hylton.Infrastructure.EntityFrameworkORM.MERCHANDISE_IMAGES,
System.Int32], System.Func`2[Hylton.Infrastructure.EntityFrameworkORM.MERCHANDISE_IMAGES,System.String])' method,
and this method cannot be translated into a store expression.
I have no idea what I'm doing wrong, I've looked through every post on this issue and everything I've seen is doing it the same way. Can anyone point out what I'm missing?

Short version: EF can't convert ToDictionary to SQL.
In my experience, the best way to go is to simplify the Select to an anonymous type with all the stuff from the DBOs you'll need, call ToList (or something else to force the evaluation) and then do the ToDictionary and other massaging of the data.
The added benefit of this is by forcing it to evaluate, you'll avoid leaving the using your context is in and closing it, which will cause an error when you try and use the return.

Why does this LINQ expression break my loop & conversion logic?

Background
ArticleService is a class that provides methods for front-end layers to facilitate business with the back-end.
Two of its base responsibilities are to convert ViewModels (ArticleViewModel) to the appropriate Models (Article) when persisting data, and the reverse, convert Models to ViewModels when fetching data... so often that I created a private method building ViewModel objects:
private ArticleViewModel BuildViewModel(Article a)
{
return new ArticleViewModel { Title = a.Title /* all properties */ }
}
Moving along, the ArticleService provides a method to fetch all articles from the data store, returning them as ViewModels: public IEnumerable<ArticleViewModel> All()
A calling class uses it like this: var articleViewModels = _articleService.All();
Simple, right?
Problem:
I originally wrote All() lazily with a classic foreach loop:
private IEnumerable<ArticleViewModel> All()
{
var viewModels = new List<ArticleViewModel>();
foreach (var article in _db.Articles)
viewModels.Add(BuildViewModel(article));
return viewModels;
}
It worked fine - articleViewModels was an instantiate list of all view models.
Next, I used ReSharper to convert this loop to a LINQ statement for performance and prettiness, and then combined the assignment statement with the return statement. Result:
private IEnumerable<ArticleViewModel> All()
{
return _db.Articles.Select(article => BuildViewModel(article)).ToList();
}
I debugged the LINQ statement and the beast awakened:
LINQ to Entities does not recognize the method 'ArticleViewModel
BuildViewModel(Article)' and this method cannot be translated into a store expression.
Question - Why does this LINQ statement break my code?
Note: Stepping back to an explicit declaration, assignment, return worked with the LINQ statement, so I'm almost certain it's something to do with the lambda logic.

Question - Why does this LINQ statement break my code?
Because LINQ to Entities is trying to translate BuildViewModel into SQL. It doesn't know how, so it dies miserably.
In your original version, you were streaming the entity from the database to your local box, and then doing the projection using BuildViewModel client-side. That's fine.
so I'm almost certain it's something to do with the lambda logic.
Nope. It's because LINQ to Entities can't translate BuildViewModel into SQL. It doesn't matter if you use a lambda expression to express the projection or not.
You can rewrite the code like this:
return _db.Articles.
.AsEnumerable()
.Select(article => BuildViewModel(article)).ToList();
This causes _db.Articles to be treated as a plain old enumerable, and then does the projection client side. Now LINQ to Entities doesn't have to figure out what to do with BuildViewModel.

The answer by Jason explains the error, but doesn't [didn't] list the simple fix (if you still want to use LINQ). Just add a call to .AsEnumerable() right after "_db.Articles". This will force any future LINQ statements to be performed using LINQ to Objects (in memory) rather than trying to use the IQueryable to perform the statements against the database.

Simply your function BuildViewModel is not traslable to raw SQL. That is.

Understanding .AsEnumerable() in LINQ to SQL

Given the following LINQ to SQL query:
var test = from i in Imports
where i.IsActive
select i;
The interpreted SQL statement is:
SELECT [t0].[id] AS [Id] .... FROM [Imports] AS [t0] WHERE [t0].[isActive] = 1
Say I wanted to perform some action in the select that cannot be converted to SQL. Its my understanding that the conventional way to accomplish this is to do AsEnumerable() thus converting it to a workable object.
Given this updated code:
var test = from i in Imports.AsEnumerable()
where i.IsActive
select new
{
// Make some method call
};
And updated SQL:
SELECT [t0].[id] AS [Id] ... FROM [Imports] AS [t0]
Notice the lack of a where clause in the executed SQL statement.
Does this mean the entire "Imports" table is cached into memory?
Would this slow performance at all if the table contained a large amount of records?
Help me to understand what is actually happening behind the scenes here.

The reason for AsEnumerable is to
AsEnumerable(TSource)(IEnumerable(TSource))
can be used to choose between query
implementations when a sequence
implements IEnumerable(T) but also has
a different set of public query
methods available
So when you were calling the Where method before, you were calling a different Where method from the IEnumerable.Where. That Where statement was for LINQ to convert to SQL, the new Where is the IEnumerable one that takes an IEnumerable, enumerates it and yields the matching items. Which explains why you see the different SQL being generated. The table will be taken in full from the database before the Where extension will be applied in your second version of the code. This could create a serious bottle neck, because the entire table has to be in memory, or worse the entire table would have to travel between servers. Allow SQL server to execute the Where and do what it does best.

At the point where the enumeration is enumerated through, the database will then be queried, and the entire resultset retrieved.
A part-and-part solution can be the way. Consider
var res = (
from result in SomeSource
where DatabaseConvertableCriterion(result)
&& NonDatabaseConvertableCriterion(result)
select new {result.A, result.B}
);
Let's say also that NonDatabaseConvertableCriterion requires field C from result. Because NonDatabaseConvertableCriterion does what its name suggests, this has to be performed as an enumeration. However, consider:
var partWay =
(
from result in SomeSource
where DatabaseConvertableCriterion(result)
select new {result.A, result.B, result.C}
);
var res =
(
from result in partWay.AsEnumerable()
where NonDatabaseConvertableCriterion select new {result.A, result.B}
);
In this case, when res is enumerated, queried or otherwise used, as much work as possible will be passed to the database, which will return enough to continue the job. Assuming that it is indeed really impossible to rewrite so that all the work can be sent to the database, this may be a suitable compromise.

There are three implementations of AsEnumerable.
DataTableExtensions.AsEnumerable
Extends a DataTable to give it an IEnumerable interface so you can use Linq against the DataTable.
Enumerable.AsEnumerable<TSource> and ParallelEnumerable.AsEnumerable<TSource>
The AsEnumerable<TSource>(IEnumerable<TSource>) method has no effect
other than to change the compile-time type of source from a type that
implements IEnumerable<T> to IEnumerable<T> itself.
AsEnumerable<TSource>(IEnumerable<TSource>) can be used to choose
between query implementations when a sequence implements
IEnumerable<T> but also has a different set of public query methods
available. For example, given a generic class Table that implements
IEnumerable<T> and has its own methods such as Where, Select, and
SelectMany, a call to Where would invoke the public Where method of
Table. A Table type that represents a database table could have a
Where method that takes the predicate argument as an expression tree
and converts the tree to SQL for remote execution. If remote execution
is not desired, for example because the predicate invokes a local
method, the AsEnumerable<TSource> method can be used to hide the
custom methods and instead make the standard query operators
available.
In other words.
If I have an
IQueryable<X> sequence = ...;
from a LinqProvider, like Entity Framework, and I do,
sequence.Where(x => SomeUnusualPredicate(x));
that query will be composed and run on the server. This will fail at runtime because the EntityFramework doesn't know how to convert SomeUnusualPredicate into SQL.
If I want that to run the statement with Linq to Objects instead, I do,
sequence.AsEnumerable().Where(x => SomeUnusualPredicate(x));
now the server will return all the data and the Enumerable.Where from Linq to Objects will be used instead of the Query Provider's implementation.
It won't matter that Entity Framework doesn't know how to interpret SomeUnusualPredicate, my function will be used directly. (However, this may be an inefficient approach since all rows will be returned from the server.)

I believe the AsEnumerable just tells the compiler which extension methods to use (in this case the ones defined for IEnumerable instead of those for IQueryable).
The execution of the query is still deferred until you call ToArray or enumerate on it.