Develop Reference

using AsQueryable after ToList [duplicate]

I know some differences of LINQ to Entities and LINQ to Objects which the first implements IQueryable and the second implements IEnumerable and my question scope is within EF 5.
My question is what's the technical difference(s) of those 3 methods? I see that in many situations all of them work. I also see using combinations of them like .ToList().AsQueryable().
What do those methods mean, exactly?
Is there any performance issue or something that would lead to the use of one over the other?
Why would one use, for example, .ToList().AsQueryable() instead of .AsQueryable()?

There is a lot to say about this. Let me focus on AsEnumerable and AsQueryable and mention ToList() along the way.
What do these methods do?
AsEnumerable and AsQueryable cast or convert to IEnumerable or IQueryable, respectively. I say cast or convert with a reason:
When the source object already implements the target interface, the source object itself is returned but cast to the target interface. In other words: the type is not changed, but the compile-time type is.
When the source object does not implement the target interface, the source object is converted into an object that implements the target interface. So both the type and the compile-time type are changed.
Let me show this with some examples. I've got this little method that reports the compile-time type and the actual type of an object (courtesy Jon Skeet):
void ReportTypeProperties<T>(T obj)
{
Console.WriteLine("Compile-time type: {0}", typeof(T).Name);
Console.WriteLine("Actual type: {0}", obj.GetType().Name);
}
Let's try an arbitrary linq-to-sql Table<T>, which implements IQueryable:
ReportTypeProperties(context.Observations);
ReportTypeProperties(context.Observations.AsEnumerable());
ReportTypeProperties(context.Observations.AsQueryable());
The result:
Compile-time type: Table`1
Actual type: Table`1
Compile-time type: IEnumerable`1
Actual type: Table`1
Compile-time type: IQueryable`1
Actual type: Table`1
You see that the table class itself is always returned, but its representation changes.
Now an object that implements IEnumerable, not IQueryable:
var ints = new[] { 1, 2 };
ReportTypeProperties(ints);
ReportTypeProperties(ints.AsEnumerable());
ReportTypeProperties(ints.AsQueryable());
The results:
Compile-time type: Int32[]
Actual type: Int32[]
Compile-time type: IEnumerable`1
Actual type: Int32[]
Compile-time type: IQueryable`1
Actual type: EnumerableQuery`1
There it is. AsQueryable() has converted the array into an EnumerableQuery, which "represents an IEnumerable<T> collection as an IQueryable<T> data source." (MSDN).
What's the use?
AsEnumerable is frequently used to switch from any IQueryable implementation to LINQ to objects (L2O), mostly because the former does not support functions that L2O has. For more details see What is the effect of AsEnumerable() on a LINQ Entity?.
For example, in an Entity Framework query we can only use a restricted number of methods. So if, for example, we need to use one of our own methods in a query we would typically write something like
var query = context.Observations.Select(o => o.Id)
.AsEnumerable().Select(x => MySuperSmartMethod(x))
ToList – which converts an IEnumerable<T> to a List<T> – is often used for this purpose as well. The advantage of using AsEnumerable vs. ToList is that AsEnumerable does not execute the query. AsEnumerable preserves deferred execution and does not build an often useless intermediate list.
On the other hand, when forced execution of a LINQ query is desired, ToList can be a way to do that.
AsQueryable can be used to make an enumerable collection accept expressions in LINQ statements. See here for more details: Do i really need use AsQueryable() on collection?.
Note on substance abuse!
AsEnumerable works like a drug. It's a quick fix, but at a cost and it doesn't address the underlying problem.
In many Stack Overflow answers, I see people applying AsEnumerable to fix just about any problem with unsupported methods in LINQ expressions. But the price isn't always clear. For instance, if you do this:
context.MyLongWideTable // A table with many records and columns
.Where(x => x.Type == "type")
.Select(x => new { x.Name, x.CreateDate })
...everything is neatly translated into a SQL statement that filters (Where) and projects (Select). That is, both the length and the width, respectively, of the SQL result set are reduced.
Now suppose users only want to see the date part of CreateDate. In Entity Framework you'll quickly discover that...
.Select(x => new { x.Name, x.CreateDate.Date })
...is not supported (at the time of writing). Ah, fortunately there's the AsEnumerable fix:
context.MyLongWideTable.AsEnumerable()
.Where(x => x.Type == "type")
.Select(x => new { x.Name, x.CreateDate.Date })
Sure, it runs, probably. But it pulls the entire table into memory and then applies the filter and the projections. Well, most people are smart enough to do the Where first:
context.MyLongWideTable
.Where(x => x.Type == "type").AsEnumerable()
.Select(x => new { x.Name, x.CreateDate.Date })
But still all columns are fetched first and the projection is done in memory.
The real fix is:
context.MyLongWideTable
.Where(x => x.Type == "type")
.Select(x => new { x.Name, DbFunctions.TruncateTime(x.CreateDate) })
(But that requires just a little bit more knowledge...)
What do these methods NOT do?
Restore IQueryable capabilities
Now an important caveat. When you do
context.Observations.AsEnumerable()
.AsQueryable()
you will end up with the source object represented as IQueryable. (Because both methods only cast and don't convert).
But when you do
context.Observations.AsEnumerable().Select(x => x)
.AsQueryable()
what will the result be?
The Select produces a WhereSelectEnumerableIterator. This is an internal .Net class that implements IEnumerable, not IQueryable. So a conversion to another type has taken place and the subsequent AsQueryable can never return the original source anymore.
The implication of this is that using AsQueryable is not a way to magically inject a query provider with its specific features into an enumerable. Suppose you do
var query = context.Observations.Select(o => o.Id)
.AsEnumerable().Select(x => x.ToString())
.AsQueryable()
.Where(...)
The where condition will never be translated into SQL. AsEnumerable() followed by LINQ statements definitively cuts the connection with entity framework query provider.
I deliberately show this example because I've seen questions here where people for instance try to 'inject' Include capabilities into a collection by calling AsQueryable. It compiles and runs, but it does nothing because the underlying object does not have an Include implementation anymore.
Execute
Both AsQueryable and AsEnumerable don't execute (or enumerate) the source object. They only change their type or representation. Both involved interfaces, IQueryable and IEnumerable, are nothing but "an enumeration waiting to happen". They are not executed before they're forced to do so, for example, as mentioned above, by calling ToList().
That means that executing an IEnumerable obtained by calling AsEnumerable on an IQueryable object, will execute the underlying IQueryable. A subsequent execution of the IEnumerable will again execute the IQueryable. Which may be very expensive.
Specific Implementations
So far, this was only about the Queryable.AsQueryable and Enumerable.AsEnumerable extension methods. But of course anybody can write instance methods or extension methods with the same names (and functions).
In fact, a common example of a specific AsEnumerable extension method is DataTableExtensions.AsEnumerable. DataTable does not implement IQueryable or IEnumerable, so the regular extension methods don't apply.

ToList()
Execute the query immediately
AsEnumerable()
lazy (execute the query later)
Parameter: Func<TSource, bool>
Load EVERY record into application memory, and then handle/filter them. (e.g. Where/Take/Skip, it will select * from table1, into the memory, then select the first X elements) (In this case, what it did: Linq-to-SQL + Linq-to-Object)
AsQueryable()
lazy (execute the query later)
Parameter: Expression<Func<TSource, bool>>
Convert Expression into T-SQL (with the specific provider), query remotely and load result to your application memory.
That’s why DbSet (in Entity Framework) also inherits IQueryable to get the efficient query.
Do not load every record, e.g. if Take(5), it will generate select top 5 * SQL in the background. This means this type is more friendly to SQL Database, and that is why this type usually has higher performance and is recommended when dealing with a database.
So AsQueryable() usually works much faster than AsEnumerable() as it generate T-SQL at first, which includes all your where conditions in your Linq.

ToList() will being everything in memory and then you will be working on it.
so, ToList().where ( apply some filter ) is executed locally.
AsQueryable() will execute everything remotely i.e. a filter on it is sent to the database for applying.
Queryable doesn't do anything til you execute it. ToList, however executes immediately.
Also, look at this answer Why use AsQueryable() instead of List()?.
EDIT :
Also, in your case once you do ToList() then every subsequent operation is local including AsQueryable(). You can't switch to remote once you start executing locally.
Hope this makes it a little bit more clearer.

Encountered a bad performance on below code.
void DoSomething<T>(IEnumerable<T> objects){
var single = objects.First(); //load everything into memory before .First()
...
}
Fixed with
void DoSomething<T>(IEnumerable<T> objects){
T single;
if (objects is IQueryable<T>)
single = objects.AsQueryable().First(); // SELECT TOP (1) ... is used
else
single = objects.First();
}
For an IQueryable, stay in IQueryable when possible, try not be used like IEnumerable.
Update. It can be further simplified in one expression, thanks Gert Arnold.
T single = objects is IQueryable<T> q?
q.First():
objects.First();

How can I determine if a LINQ query is going to be LINQ to SQL vs. LINQ to Objects?

Usually the distinction between LINQ to SQL and LINQ to Objects isn't much of an issue, but how can I determine which is happening?
It would be useful to know when writing the code, but I fear one can only be sure at run time sometimes.

It's not micro optimization to make the distinction between Linq-To-Sql and Linq-To-Objects. The latter requires all data to be loaded into memory before you start filtering it. Of course, that can be a major issue.
Most LINQ methods are using deferred execution, which means that it's just building the query but it's not yet executed (like Select or Where). Few others are executing the query and materialize the result into an in-memory collection (like ToLIst or ToArray). If you use AsEnumerable you are also using Linq-To-Objects and no SQL is generated for the parts after it, which means that the data must be loaded into memory (yet still using deferred execution).
So consider the following two queries. The first selects and filters in the database:
var queryLondonCustomers = from cust in db.customers
where cust.City == "London"
select cust;
whereas the second selects all and filters via Linq-To-Objects:
var queryLondonCustomers = from cust in db.customers.AsEnumerable()
where cust.City == "London"
select cust;
The latter has one advantage: you can use any .NET method since it doesn't need to be translated to SQL (e.g. !String.IsNullOrWhiteSpace(cust.City)).
If you just get something that is an IEnumerable<T>, you can't be sure if it's actually a query or already an in-memory object. Even the try-cast to IQueryable<T> will not tell you for sure what it actually is because of the AsQueryable-method. Maybe you could try-cast it to a collection type. If the cast succeeds you can be sure that it's already materialized but otherwise it doesn't tell you if it's using Linq-To-Sql or Linq-To-Objects:
bool isMaterialized = queryLondonCustomers as ICollection<Customer> != null;
Related: EF ICollection Vs List Vs IEnumerable Vs IQueryable

The first solution comes into my mind is checking the query provider.
If the query is materialized, which means the data is loaded into memory, EnumerableQuery(T) is used. Otherwise, a special query provider is used, for example, System.Data.Entity.Internal.Linq.DbQueryProvider for entityframework.
var materialized = query
.AsQueryable()
.Provider
.GetType()
.GetGenericTypeDefinition() == typeof(EnumerableQuery<>);
However the above are ideal cases because someone can implement a custom query provider behaves like EnumerableQuery.

I had the same question, for different reasons.
Judging purely on your title & initial description (which is why google search brought me here).
Pre compilation, given an instance that implements IQueryable, there's no way to know the implementation behind the interface.
At runtime, you need to check the instance's Provider property like #Danny Chen mentioned.
public enum LinqProvider
{
Linq2SQL, Linq2Objects
}
public static class LinqProviderExtensions
{
public static LinqProvider LinqProvider(this IQueryable query)
{
if (query.Provider.GetType().IsGenericType && query.Provider.GetType().GetGenericTypeDefinition() == typeof(EnumerableQuery<>))
return LinqProvider.Linq2Objects;
if (typeof(ICollection<>).MakeGenericType(query.ElementType).IsAssignableFrom(query.GetType()))
return LinqProvider.Linq2Objects;
return LinqProvider.Linq2SQL;
}
}
In our case, we are adding additional filters dynamically, but ran into issues with different handling of case-sensitivity/nullreference handling on different providers.
Hence, at runtime we had to tweak the filters that we add based on the type of provider, and ended up adding this extension method:

Using EF core in net core 6
To see if the provider is an EF provider, use the following code:
if (queryable.Provider is Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryProvider)
{
// Queryable is backed by EF and is not an in-memory/client-side queryable.
}
One could get the opposite by testing the provider against System.Linq.EnumerableQuery (base type of EnumerableQuery<T> - so you don't have to test generics).
This is useful if you have methods like EF.Functions.Like(...) which can only be executed in the database - and you want to branch to something else in case of client-side execution.

What am I using? (Linq/sql) [duplicate]

This question already has answers here:
Linq query or Lambda expression?
(5 answers)
Closed 7 years ago.
My project currently uses for following syntax for "linq"
var le = domainModel.EntityDataContext.Table_name_here.Query()
.WithFullReadCheck(domainModel.Security)
.Where(x => x.Metadata.AdvisorID == advisorId).FirstOrDefault();
It is said the above code is linq, being unaware about linq I decided to learn it. However, on https://msdn.microsoft.com/en-us/library/gg509017.aspx , things are pretty different.
what is being used in my code? Is it a version of linq? Is it something else?

What you are using is LINQ. There are 2 different notations you can use for writing LINQ - Lambda Syntax and Query Syntax
The difference is explained here: LINQ: Dot Notation vs Query Expression
There is more information on this on MSDN here: https://msdn.microsoft.com/en-us/library/bb308959.aspx
The MSDN articles starts explaining LINQ with SQL-like syntax ("query syntax") and then goes on to explain Lambda Expressions and Expression Trees ("lamba syntax").

That is extension method syntax for a query. The .Query().WithFullReadCheck() are much less common, but the rest .Where and .FirstOrDefault are pretty common query extensions. LINQ is a special syntax for expressing the same thing that can be done with chaining the query extension methods. Behind the scenes, they are identical.
See this question which gives an example of both LINQ and extension method syntax and has a good discussion of the differences between them: Extension methods syntax vs query syntax

Linq (language integrated query) is the language you use to query the data. You don't care where do they come from, that's the whole point of using it. You can use the same code to query arrays, collections, databases, xml files, directories, whatever... Linq translates your code in the background, so if you use it for instance to get the data from a database, it produces SQL to be sent to the database.
There are two syntax versions available:
1.) lambda syntax
2.) chainable methods
It doesn't matter which one you chose, just try to be consistent with it and use what makes you more comfortable / makes more sense in your situation.

LINQ is 'Language INtegrated Query'. In practical terms it means two things to most people:
The way that inline delegates are converted by the compiler into expression trees, not anonymous methods. That means that x => x.Metadata.AdvisorID == advisorId is not compiled into IL at compile time: instead it's compiled into code which builds an equivalent Expression object, which can be passed to a provider such as Entity Framework or LINQ to SQL, in order to generate a database query.
The other part it means to most people is the so-called "syntactic sugar", which effectively calls .Where(), .OrderBy() etc on your behalf:
from a in something
where a.Name == "Bob"
select a;
In both cases, the Queryable class in .NET provides extension methods for building a chain of Expression objects, for example .Where(), .OrderBy(), .Select() etc - these take an IQueryable (either untyped or with a generic parameter) and return another. They lightly wrap an Expression object at each point which represents the query as a whole.
Meaning:
someQueryable.Where(x => x.Id > 3).OrderBy(x => x.Date).Select(x => x.Ref)
Returns an object implementing IQueryable which holds an Expression which looks like:
Select(OrderBy(Where(someQueryable, x => x.Id > 3), x => x.Date), x => x.Ref)
... which is readable by LINQ providers to produce something like:
SELECT Ref FROM someTable WHERE Id > 3 ORDER BY Date
Finally, note that .Where(), .OrderBy() and the like are NOT limited to queryable/LINQ objects. They exist for IEnumerable (and IEnumerable<>), too - but this is not LINQ, and simply performs the operations at the instant that the method is called.

LINQ passthrough provider?

I've implemented simple IQueryable and IQueryProvider classes that collect statistical data on LINQ expression trees. This part works fine. Next, I would like to pass the expression tree off to the default LINQ-to-Objects provider for evaluation, since I don't need to execute it any differently. In order words, I'd like my provider to collect stats as a side-effect, passing the query on to the default LINQ implementation.
However, I am having difficulty getting a handle to the default provider. I thought that I could simply save a reference to the original IEnumerable collection and then return the default provider (from my custom IQueryable) like:
IQueryProvider IQueryable.Provider
{
get { return _my_provider.OriginalIEnum().AsQueryable().Provider; }
}
but this does not work correctly. The code eventually throws a StackOverflowException. What I think is happening (gleaned from single-stepping in debug mode) is that the LINQ runtime fetches the provider from the above method, then it fetches the expression tree from my custom IQueryable, and then it notices that the top-level expression is my custom IQueryable. So it starts the process over again, trying to find the appropriate provider. It does this endlessly until a stack overflow occurs.
Right now, the only thing I can think of is to come up with another visitor which produces another expression tree with the custom IQueryable nodes removed so that the LINQ runtime will call the default provider. That's a fair amount of work, since I need to visit every leaf to ensure that there are no nested Call expressions which call my custom IQueryable again. Is there a simpler approach?
Thanks for the help.

It turns out that the real issue here is exactly the one described in Pass LINQ expression to another QueryProvider.

Error, method not supported by LINQ to Entities

Why I am getting this error:
The method 'Single' is not supported by LINQ to Entities. Consider using the method 'First' instead.
public ActionResult Details(int id)
Line 27: {
var result = (from d in _db.MovieSet
Line 29: where d.Id == id
Line 30: select d).Single();
//
//
}
Code compiles safe, but only breaks if call is made to the respective section. I am new to LINQ, therefore do not know which methods are for LINQtoSQL or LINQtoEntities. This means more errors! We cannot remember all methods this way.
My question is, if there are limitations to the methods applicable to certain types / scenarios, why do they appear in Intellisense?
EDIT: Any work-around / technique helpful to have an idea if one is supported ?

Microsoft has a complete list of supported and unsupported methods in Linq to Entities. That's where to go to find out this information.
You'll notice that the Single and SingleOrDefault methods are in fact listed as "not supported" in the section on Paging Methods.
As Jared pointed out, the compiler does not know at compile time which provider you are using, so it has no way to enforce compile-time safety of extension methods that the provider may or may not implement. You'll have to rely on the documentation instead.

In the case of LINQtoSQL / Entities, the queries are all broken down into expression trees which are then passed to the provider APIs. The providers cannot provide compile time information about the trees they do or do not support because there is no syntactic difference. The only choice is for them to provide runtime data.
For example once in expression tree form, both the Single and First appear as a MethodCallExpression instance.

Unfortunately, it's yet another indication of both the relative immaturity of EF and the Object Relational Impedance Mismatch.
Documentation is your friend if you choose to go this route.