To keep my code cleaner I often try to break down parts of my data access code in LINQ to SQL into private sub-methods, just like with plain-old business logic code. Let me give a very simplistic example:
public IEnumerable<Item> GetItemsFromRepository()
{
var setA = from a in this.dataContext.TableA
where /* criteria */
select a.Prop;
return DoSubquery(setA);
}
private IEnumerable<Item> DoSubQuery(IEnumerable<DateTimeOffset> set)
{
return from item in set
where /* criteria */
select item;
}
I'm sure no one's imagination would be stretched by imagining more complex examples with deeper nesting or using results of sets to filter other queries.
My basic question is this: I've seen some significant performance differences and even exceptions being thrown by just simply reorganizing LINQ to SQL code in private methods. Can anyone explain the rules for these behaviors so that I can make informed decisions about how to write efficient, clean data access code?
Some questions I've had:
1) When does passage of System.Linq.Table instace to a method cause query execution?
2) When does using a System.Linq.Table in another query cause execution?
3) Are there limits to what types of operations (Take, First, Last, order by, etc.) can be applied to System.Linq.Table passed a parameters into a method?
The most important rule in terms of LINQ-to-SQL would be: don't return IEnumerable<T> unless you must - as the semantic is unclear. There are two schools of thought beyond that:
if you return IQueryable<T>, it is composable, meaning the where from later queries is combined to make a single TSQL, but as a down-side, it is hard to fully test
otherwise, return List<T> or similar, so it is clear that everything beyond that point is LINQ-to-Objects
Currently, you are doing something in the middle: collapsing it to LINQ-to-Objects (via IEnumerable<T>), but without it being obvious - and keeping the connection open in the middle (again, only a problem because it isn't obvious)
Remove the implicit cast:
public IQueryable<Item> GetItemsFromRepository()
{
var setA = from a in this.dataContext.TableA
where /* criteria */
select a.Prop;
return DoSubquery(setA);
}
private IQueryable<Item> DoSubQuery(IQueryable<DateTimeOffset> set)
{
return from item in set
where /* criteria */
select item;
}
The implicit cast from IQueryable<Item> to IEnumerable<Item> is essentially the same as calling AsEnumerable() on your IQueryable<Item>. There are of course times when you want that, but you should leave things as IQueryable by default, so that the entire query can be performed on the database, rather than merely the GetItemsFromRepository() bit with the rest being done in memory.
The secondary questions:
1) When does passage of System.Linq.Table instace to a method cause query execution?
When something needs a final result, such as Max(), ToList(), etc. that is neither a queryable object, nor a loaded-as-it-goes enumerable.
Note though, that while AsEnumerable() does not cause query execution, it does mean that when execution does happen only that before the AsEnumerable() will be performed against the source datasource, this will then produce an on-demand in-memory datasource against which the rest will be performed.
2) When does using a System.Linq.Table in another query cause
execution?
The same as above. Table<T> implements IQueryable<T>. If you e.g. join two of them together, that won't yet cause anything to be executed.
3) Are there limits to what types of operations (Take,
First, Last, order by, etc.) can be applied to System.Linq.Table
passed a parameters into a method?
Those that are definted by IQueryable<T>.
Edit: Some clarification on the differences and similarities between IEnumerable and IQueryable.
Just about anything you can do on an IQueryable you can do on an IEnumerable and vice-versa, but how it's performed will be different.
Any given IQueryable implementation can be used in linq queries and will have all the linqy extension methods like Take(), Select(), GroupBy and so on.
Just how this is done, depends on the implementation. For example, System.Linq.Data.Table implements those methods by the query being turned into an SQL query, the results of which are turned into a objects on a as-loaded basis. So if mySource is a table then:
var filtered = from item in mySource
where item.ID < 23
select new{item.ID, item.Name};
foreach(var i in filtered)
Console.WriteLine(i.Name);
Gets turned into SQL like:
select id, name from mySourceTable where id < 23
And then an enumerator is created from that such that on each call to MoveNext() another row is read from the results, and a new anonymous object created from it.
On the other hand, if mySource where a List or a HashSet, or anything else that implements IEnumerable<T> but doesn't have its own query engine, then the linq-to-objects code will turn it into something like:
foreach(var item in mySource)
if(item.ID < 23)
yield return new {item.ID, item.Name};
Which is about as efficiently as that code could be done in memory. The results will be the same, but the way to get them, would be different:
Now, since all IQueryable<T> can be converted into the equivalent IEnumerable<T> we could, if we wanted to, take the first mySource (where execution happens in a database) and do the following instead:
var filtered = from item in mySource.AsEnumerable()
where item.ID < 23
select new{item.ID, item.Name};
Here, while there is still nothing executed against the database until we iterate through the results or call something that examines all of those results, once we do so, it's as if we split the execution into two separate steps:
var asEnum = mySource.AsEnumerable();
var filtered = from item in asEnum
where item.ID < 23
select new{item.ID, item.Name};
The implemenatation of the first line would be to execute the SQL SELECT * FROM mySourceTable, and the execution of the rest would be like the linq-to-objects example above.
It's not hard to see how, if the database contained 10 items with an id < 23, and 50,000 items with an id higher, this is now much, much less performant.
As well as offering the explicity AsEnumerable() method, all IQueryable<T> can be implicitly cast to IEnumerable<T>. This lets us do foreach on them and use them with any other existing code that handles IEnumerable<T>, but if we accidentally do it at in inappropriate time, we can make queries much slower, and this is what was happening when your DoSubQuery was defined to take an IEnumerable<DateTimeOffset> and return an IEnumerable<Item>; it implicitly called AsEnumerable() on your IQueryable<DateTimeOffset> and your IQueryable<Item> and caused what could have been performed on the database to be performed in-memory.
For this reason, 99% of the time, we want to keep dealing in IQueryable until the very last moment.
As an example of the opposite though, just to point out that AsEnumerable() and the casts to IEnumerable<T> aren't there out of madness, we should consider two things. The first is that IEnumerable<T> lets us do things that can't be done otherwise, such as joining two completely different sources that don't know about each other (e.g. two different databases, a database and an XML file, etc.)
Another is that sometimes IEnumerable<T> is actually more efficient too. Consider:
IQueryable<IGrouping<string, int>> groupingQuery = from item in mySource select item.ID group by item.Name;
var list1 = groupingQuery.Select(grp => new {Name=grp.Key, Count=grp.Count()}).ToList();//fine
foreach(var grp in groupingQuery)//disaster!
Console.WriteLine(grp.Count());
Here groupingQuery is set up as a queryable that does some grouping, but which hasn't been executed in anyway. When we create list1, then first we create a new IQueryable based on that, and the query engine does it's best to work out what the best SQL for it is, and comes up with something like:
select name, count(id) from mySourceTable group by name
Which is pretty efficiently performed. Then the rows are turned into objects, which are then put into a list.
On the other hand, with the second query, there isn't as natural a SQL conversion for a group by that doesn't perform aggregate methods on all of the non-grouped items, so the best the query engine can come up with is to first do:
select distinct name from mySourceTable,
And then for every name it receives, to do:
select id from mySourceTable where name = '{name found in last query goes here}'
And so on, should this mean 2 SQL queries, or 200,000.
In this case, we're much better working on mySource.AsEnumerable() because here it is more efficient to grab the whole table into memory first. (Even better still would be to work on mySource.Select(item => new {item.ID, item.Name}).AsEnumerable() because then we still only retrieve the columns we care about from the database, and switch to in-memory at that point).
The last bit is worth remembering because it breaks our rule that we should stay with IQueryable<T> as long as possible. It isn't something to worry about much, but it is worth keeping an eye on if you do grouping and find yourself with a very slow query.
Related
Below is an example from my textbook:
var allGenres = from genre in myEntities.Genres.Include("Reviews")
orderby genre.Name
select new { genre.Name, genre.Reviews };
Repeater1.DataSource = allGenres.ToList();
Repeater1.DataBind();
and the book says:
as soon as you call ToList(), the query is executed and the relevant genres and reviews are retrieved from the database and assigned to the DataSource property
so my question is, if I get rid of Repeater1.DataSource = allGenres.ToList();, what does var allGenres contain? since the query hasn't been executed?
There are three stages to understand.
First, when the query is created.
Second, when the query variable is iterated over (deferred execution).
Third, forcing a query for immediate results.
var allGenres = from genre in myEntities.Genres.Include("Reviews")
orderby genre.Name
select new { genre.Name, genre.Reviews };
In this code, the query is only created. Its dead as a person in a cemetery.
If you need deferred execution, then you can iterate over the results using for loop etc.
To force immediate execution, you can use conversion operators like
ToList, ToArray, ToLookup, and ToDictionary.
Hope it helps.
You can put a breakpoint on this line:
var allGenres = from genre in myEntities.Genres.Include("Reviews")
orderby genre.Name
select new { genre.Name, genre.Reviews };
and notice nothing happens. After the following line, voila SQL Profiler will display the sql query happening:
Repeater1.DataSource = allGenres.ToList();
Your object allGenres is an object that implements IQueryable<...>. This means that it represents a sequence, and it has functions to get the first element of the sequence, and once you've got the element you can get the next one, until there are no more elements.
The <...> part defines what kind of elements are in the sequence. So IQueryable<Book> says that you can query for a sequence of Books, which you can enumerate one after another. Every element of the sequence will be an object of class Book
This enumeration is provided using a base interface of IQueryable, namely IEnumerable<Book>.
At the lowest level, this Enumeration is done as follows:
IEnumerable<Book> books = ... // for example new List<Book>(), or new Book[10]
IEnumerator<Book> bookEnumerator = books.Getenumerator();
// as long as there are books, print the title:
// the first MoveNext() moves to the first element
// every other MoveNext() move to the next element
// it returns false if there is no such element (no first, or no next)
while (bookEnumerator.MoveNext())
{
// the enumerator points to the next element. Property Current contains this element
Book book = bookEnumerator.Current;
Console.WriteLine(book.Title);
}
Normally we won't use this low level functionality. You'll see the foreach var more often:
foreach (Book book in books)
{
Console.WriteLine(book.Title);
}
foreach will do the GetEnumerator() / MoveNext() / Current for you
Note that the IEnumerable does not represent the enumeration itself, it represents the ability to enumerate. Quite often people are not that precise, and call the IEnumerable the sequence itself. But remember: to access the elements of the sequence you need to Enumerate over them (either by using MoveNext, or by calling foreach).
An IQueryable<...> seems very similar as an IEnumerable<...>. The difference is that it is usually meant to be processed by a different process, like a database management system, or a server on a different computer, it can also represent the lines in a CSV file, or whatever. The purpose of the IQueryable is to separate how the data is fetched from the manipulation of the fetched data.
Just like an IEnumerable holds the ability to enumerate, the IQueryable hold the ability to query data.
For this, the IQueryable has an Expression and a Provider. The Expression defines in a generic way what data must be fetched. The Provider knows who must provide the data (the database), and what language this data provider needs (SQL).
Perhaps you have noticed there are two kinds of LINQ statements. The ones that return an IQueryable and the ones that don't. The first group are functions like Where, Select, Join, GroupBy. They all return an IQueryable of some kind.
As long as you concatenate functions of this group, the Expression is changed. The query is not executed yet. The return value is still an object that represents the ability to query. These functions use deferred execution (or lazy execution), meaning that the query is not executed yet. You'll recognize these functions because they return IQueryable<...>. The remarks section of the description of these function also mentions that execution is deferred.
Only after you call GetEnumerator() / MoveNext(), either directly, or indirectly using foreach the query is executed.
If you start enumerating, the Expression is sent to the Provider, who will translate the Expression into the language that the executor of the query understands (SQL) and will order the executor to execute the query. The fetched data is converted to an IEnumerable<...> which is then enumerated as if the data was local.
It depends a bit on who made the Provider, but sometimes the Provider is really smart. it doesn't fetch all millions of Products from your products database, but it fetches a Page of Products. While you enumerate over this page the Provider fetches the next page. This will improve processing speed because there won't be fetched much more products than you actually will use, besides you can start enumerating before all Products are fetched.
I mentioned the LINQ functions that use deferred execution (= return IQueryable). The other group of functions will execute the query. This group of functions contain functions like ToList, ToDictionary, Max, FirstOrDefault, Count They don't return IQueryable<...>, but some TResult. If you look at the source code (google for "reference source queryable tolist"), you will see that they will do this by calling foreach or GetEnumerator.
Because the Expression must be translated to SQL, the query has less possibilities than an IEnumerable<...>. For instance, you can't use any locally defined methods in your query. If you do that, you will get a run-time exception as soon as the query is executed, telling you that the expression can't be translated into SQL.
What kind of expressions can be executed depends on who must execute your query. There is a list of Supported and Unsupported LINQ methods for ling-to-entities.
If you are using Entity Framework the allGeneres will be an instance of IQueryable<Genre> interface. The behavior is called deferred execution. For More reference https://learn.microsoft.com/en-us/dotnet/framework/data/adonet/ef/language-reference/query-execution#deferred-query-execution
What is the difference between returning IQueryable<T> vs. IEnumerable<T>, when should one be preferred over the other?
IQueryable<Customer> custs = from c in db.Customers
where c.City == "<City>"
select c;
IEnumerable<Customer> custs = from c in db.Customers
where c.City == "<City>"
select c;
Will both be deferred execution and when should one be preferred over the other?
Yes, both will give you deferred execution.
The difference is that IQueryable<T> is the interface that allows LINQ-to-SQL (LINQ.-to-anything really) to work. So if you further refine your query on an IQueryable<T>, that query will be executed in the database, if possible.
For the IEnumerable<T> case, it will be LINQ-to-object, meaning that all objects matching the original query will have to be loaded into memory from the database.
In code:
IQueryable<Customer> custs = ...;
// Later on...
var goldCustomers = custs.Where(c => c.IsGold);
That code will execute SQL to only select gold customers. The following code, on the other hand, will execute the original query in the database, then filtering out the non-gold customers in the memory:
IEnumerable<Customer> custs = ...;
// Later on...
var goldCustomers = custs.Where(c => c.IsGold);
This is quite an important difference, and working on IQueryable<T> can in many cases save you from returning too many rows from the database. Another prime example is doing paging: If you use Take and Skip on IQueryable, you will only get the number of rows requested; doing that on an IEnumerable<T> will cause all of your rows to be loaded in memory.
The top answer is good but it doesn't mention expression trees which explain "how" the two interfaces differ. Basically, there are two identical sets of LINQ extensions. Where(), Sum(), Count(), FirstOrDefault(), etc all have two versions: one that accepts functions and one that accepts expressions.
The IEnumerable version signature is: Where(Func<Customer, bool> predicate)
The IQueryable version signature is: Where(Expression<Func<Customer, bool>> predicate)
You've probably been using both of those without realizing it because both are called using identical syntax:
e.g. Where(x => x.City == "<City>") works on both IEnumerable and IQueryable
When using Where() on an IEnumerable collection, the compiler passes a compiled function to Where()
When using Where() on an IQueryable collection, the compiler passes an expression tree to Where(). An expression tree is like the reflection system but for code. The compiler converts your code into a data structure that describes what your code does in a format that's easily digestible.
Why bother with this expression tree thing? I just want Where() to filter my data.
The main reason is that both the EF and Linq2SQL ORMs can convert expression trees directly into SQL where your code will execute much faster.
Oh, that sounds like a free performance boost, should I use AsQueryable() all over the place in that case?
No, IQueryable is only useful if the underlying data provider can do something with it. Converting something like a regular List to IQueryable will not give you any benefit.
Yes, both use deferred execution. Let's illustrate the difference using the SQL Server profiler....
When we run the following code:
MarketDevEntities db = new MarketDevEntities();
IEnumerable<WebLog> first = db.WebLogs;
var second = first.Where(c => c.DurationSeconds > 10);
var third = second.Where(c => c.WebLogID > 100);
var result = third.Where(c => c.EmailAddress.Length > 11);
Console.Write(result.First().UserName);
In SQL Server profiler we find a command equal to:
"SELECT * FROM [dbo].[WebLog]"
It approximately takes 90 seconds to run that block of code against a WebLog table which has 1 million records.
So, all table records are loaded into memory as objects, and then with each .Where() it will be another filter in memory against these objects.
When we use IQueryable instead of IEnumerable in the above example (second line):
In SQL Server profiler we find a command equal to:
"SELECT TOP 1 * FROM [dbo].[WebLog] WHERE [DurationSeconds] > 10 AND [WebLogID] > 100 AND LEN([EmailAddress]) > 11"
It approximately takes four seconds to run this block of code using IQueryable.
IQueryable has a property called Expression which stores a tree expression which starts being created when we used the result in our example (which is called deferred execution), and at the end this expression will be converted to an SQL query to run on the database engine.
Both will give you deferred execution, yes.
As for which is preferred over the other, it depends on what your underlying datasource is.
Returning an IEnumerable will automatically force the runtime to use LINQ to Objects to query your collection.
Returning an IQueryable (which implements IEnumerable, by the way) provides the extra functionality to translate your query into something that might perform better on the underlying source (LINQ to SQL, LINQ to XML, etc.).
A lot has been said previously, but back to the roots, in a more technical way:
IEnumerable is a collection of objects in memory that you can enumerate - an in-memory sequence that makes it possible to iterate through (makes it way easy for within foreach loop, though you can go with IEnumerator only). They reside in the memory as is.
IQueryable is an expression tree that will get translated into something else at some point with ability to enumerate over the final outcome. I guess this is what confuses most people.
They obviously have different connotations.
IQueryable represents an expression tree (a query, simply) that will be translated to something else by the underlying query provider as soon as release APIs are called, like LINQ aggregate functions (Sum, Count, etc.) or ToList[Array, Dictionary,...]. And IQueryable objects also implement IEnumerable, IEnumerable<T> so that if they represent a query the result of that query could be iterated. It means IQueryable don't have to be queries only. The right term is they are expression trees.
Now how those expressions are executed and what they turn to is all up to so called query providers (expression executors we can think them of).
In the Entity Framework world (which is that mystical underlying data source provider, or the query provider) IQueryable expressions are translated into native T-SQL queries. Nhibernate does similar things with them. You can write your own one following the concepts pretty well described in LINQ: Building an IQueryable Provider link, for example, and you might want to have a custom querying API for your product store provider service.
So basically, IQueryable objects are getting constructed all the way long until we explicitly release them and tell the system to rewrite them into SQL or whatever and send down the execution chain for onward processing.
As if to deferred execution it's a LINQ feature to hold up the expression tree scheme in the memory and send it into the execution only on demand, whenever certain APIs are called against the sequence (the same Count, ToList, etc.).
The proper usage of both heavily depends on the tasks you're facing for the specific case. For the well-known repository pattern I personally opt for returning IList, that is IEnumerable over Lists (indexers and the like). So it is my advice to use IQueryable only within repositories and IEnumerable anywhere else in the code. Not saying about the testability concerns that IQueryable breaks down and ruins the separation of concerns principle. If you return an expression from within repositories consumers may play with the persistence layer as they would wish.
A little addition to the mess :) (from a discussion in the comments))
None of them are objects in memory since they're not real types per se, they're markers of a type - if you want to go that deep. But it makes sense (and that's why even MSDN put it this way) to think of IEnumerables as in-memory collections whereas IQueryables as expression trees. The point is that the IQueryable interface inherits the IEnumerable interface so that if it represents a query, the results of that query can be enumerated. Enumeration causes the expression tree associated with an IQueryable object to be executed.
So, in fact, you can't really call any IEnumerable member without having the object in the memory. It will get in there if you do, anyways, if it's not empty. IQueryables are just queries, not the data.
In general terms I would recommend the following:
Return IQueryable<T> if you want to enable the developer using your method to refine the query you return before executing.
Return IEnumerable if you want to transport a set of Objects to enumerate over.
Imagine an IQueryable as that what it is - a "query" for data (which you can refine if you want to). An IEnumerable is a set of objects (which has already been received or was created) over which you can enumerate.
In general you want to preserve the original static type of the query until it matters.
For this reason, you can define your variable as 'var' instead of either IQueryable<> or IEnumerable<> and you will know that you are not changing the type.
If you start out with an IQueryable<>, you typically want to keep it as an IQueryable<> until there is some compelling reason to change it. The reason for this is that you want to give the query processor as much information as possible. For example, if you're only going to use 10 results (you've called Take(10)) then you want SQL Server to know about that so that it can optimize its query plans and send you only the data you'll use.
A compelling reason to change the type from IQueryable<> to IEnumerable<> might be that you are calling some extension function that the implementation of IQueryable<> in your particular object either cannot handle or handles inefficiently. In that case, you might wish to convert the type to IEnumerable<> (by assigning to a variable of type IEnumerable<> or by using the AsEnumerable extension method for example) so that the extension functions you call end up being the ones in the Enumerable class instead of the Queryable class.
There is a blog post with brief source code sample about how misuse of IEnumerable<T> can dramatically impact LINQ query performance: Entity Framework: IQueryable vs. IEnumerable.
If we dig deeper and look into the sources, we can see that there are obviously different extension methods are perfomed for IEnumerable<T>:
// Type: System.Linq.Enumerable
// Assembly: System.Core, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089
// Assembly location: C:\Windows\Microsoft.NET\Framework\v4.0.30319\System.Core.dll
public static class Enumerable
{
public static IEnumerable<TSource> Where<TSource>(
this IEnumerable<TSource> source,
Func<TSource, bool> predicate)
{
return (IEnumerable<TSource>)
new Enumerable.WhereEnumerableIterator<TSource>(source, predicate);
}
}
and IQueryable<T>:
// Type: System.Linq.Queryable
// Assembly: System.Core, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089
// Assembly location: C:\Windows\Microsoft.NET\Framework\v4.0.30319\System.Core.dll
public static class Queryable
{
public static IQueryable<TSource> Where<TSource>(
this IQueryable<TSource> source,
Expression<Func<TSource, bool>> predicate)
{
return source.Provider.CreateQuery<TSource>(
Expression.Call(
null,
((MethodInfo) MethodBase.GetCurrentMethod()).MakeGenericMethod(
new Type[] { typeof(TSource) }),
new Expression[]
{ source.Expression, Expression.Quote(predicate) }));
}
}
The first one returns enumerable iterator, and the second one creates query through the query provider, specified in IQueryable source.
The main difference between “IEnumerable” and “IQueryable” is about where the filter logic is executed. One executes on the client side (in memory) and the other executes on the database.
For example, we can consider an example where we have 10,000 records for a user in our database and let's say only 900 out which are active users, so in this case if we use “IEnumerable” then first it loads all 10,000 records in memory and then applies the IsActive filter on it which eventually returns the 900 active users.
While on the other hand on the same case if we use “IQueryable” it will directly apply the IsActive filter on the database which directly from there will return the 900 active users.
I would like to clarify a few things due to seemingly conflicting responses (mostly surrounding IEnumerable).
(1) IQueryable extends the IEnumerable interface. (You can send an IQueryable to something which expects IEnumerable without error.)
(2) Both IQueryable and IEnumerable LINQ attempt lazy loading when iterating over the result set. (Note that implementation can be seen in interface extension methods for each type.)
In other words, IEnumerables are not exclusively "in-memory". IQueryables are not always executed on the database. IEnumerable must load things into memory (once retrieved, possibly lazily) because it has no abstract data provider. IQueryables rely on an abstract provider (like LINQ-to-SQL), although this could also be the .NET in-memory provider.
Sample use case
(a) Retrieve list of records as IQueryable from EF context. (No records are in-memory.)
(b) Pass the IQueryable to a view whose model is IEnumerable. (Valid. IQueryable extends IEnumerable.)
(c) Iterate over and access the data set's records, child entities and properties from the view. (May cause exceptions!)
Possible Issues
(1) The IEnumerable attempts lazy loading and your data context is expired. Exception thrown because provider is no longer available.
(2) Entity Framework entity proxies are enabled (the default), and you attempt to access a related (virtual) object with an expired data context. Same as (1).
(3) Multiple Active Result Sets (MARS). If you are iterating over the IEnumerable in a foreach( var record in resultSet ) block and simultaneously attempt to access record.childEntity.childProperty, you may end up with MARS due to lazy loading of both the data set and the relational entity. This will cause an exception if it is not enabled in your connection string.
Solution
I have found that enabling MARS in the connection string works unreliably. I suggest you avoid MARS unless it is well-understood and explicitly desired.
Execute the query and store results by invoking resultList = resultSet.ToList() This seems to be the most straightforward way of ensuring your entities are in-memory.
In cases where the you are accessing related entities, you may still require a data context. Either that, or you can disable entity proxies and explicitly Include related entities from your DbSet.
I recently ran into an issue with IEnumerable v. IQueryable. The algorithm being used first performed an IQueryable query to obtain a set of results. These were then passed to a foreach loop, with the items instantiated as an Entity Framework (EF) class. This EF class was then used in the from clause of a Linq to Entity query, causing the result to be IEnumerable.
I'm fairly new to EF and Linq for Entities, so it took a while to figure out what the bottleneck was. Using MiniProfiling, I found the query and then converted all of the individual operations to a single IQueryable Linq for Entities query. The IEnumerable took 15 seconds and the IQueryable took 0.5 seconds to execute. There were three tables involved and, after reading this, I believe that the IEnumerable query was actually forming a three table cross-product and filtering the results.
Try to use IQueryables as a rule-of-thumb and profile your work to make your changes measurable.
We can use both for the same way, and they are only different in the performance.
IQueryable only executes against the database in an efficient way. It means that it creates an entire select query and only gets the related records.
For example, we want to take the top 10 customers whose name start with ‘Nimal’. In this case the select query will be generated as select top 10 * from Customer where name like ‘Nimal%’.
But if we used IEnumerable, the query would be like select * from Customer where name like ‘Nimal%’ and the top ten will be filtered at the C# coding level (it gets all the customer records from the database and passes them into C#).
In addition to first 2 really good answers (by driis & by Jacob) :
IEnumerable
interface is in the System.Collections namespace.
The IEnumerable object represents a set of data in memory and can move on this data only forward. The query represented by the IEnumerable object is executed immediately and completely, so the application receives data quickly.
When the query is executed, IEnumerable loads all the data, and if we need to filter it, the filtering itself is done on the client side.
IQueryable interface is located in the System.Linq namespace.
The IQueryable object provides remote access to the database and allows you to navigate through the data either in a direct order from beginning to end, or in the reverse order. In the process of creating a query, the returned object is IQueryable, the query is optimized. As a result, less memory is consumed during its execution, less network bandwidth, but at the same time it can be processed slightly more slowly than a query that returns an IEnumerable object.
What to choose?
If you need the entire set of returned data, then it's better to use IEnumerable, which provides the maximum speed.
If you DO NOT need the entire set of returned data, but only some filtered data, then it's better to use IQueryable.
In addition to the above, it's interesting to note that you can get exceptions if you use IQueryable instead of IEnumerable:
The following works fine if products is an IEnumerable:
products.Skip(-4);
However if products is an IQueryable and it's trying to access records from a DB table, then you'll get this error:
The offset specified in a OFFSET clause may not be negative.
This is because the following query was constructed:
SELECT [p].[ProductId]
FROM [Products] AS [p]
ORDER BY (SELECT 1)
OFFSET #__p_0 ROWS
and OFFSET can't have a negative value.
I'm looking to get a better understanding on when we should look to use IEnumerable over IQueryablewith LINQ to Entities.
With really basic calls to the database, IQueryable is way quicker, but when do i need to think about using an IEnumerable in its place?
Where is an IEnumerable optimal over an IQueryable??
Basically, IQueryables are executed by a query provider (for example a database) and some operations cannot be or should not be done by the database. For example, if you want to call a C# function (here as an example, capitalize a name correctly) using a value you got from the database you may try something like;
db.Users.Select(x => Capitalize(x.Name)) // Tries to make the db call Capitalize.
.ToList();
Since the Select is executed on an IQueryable, and the underlying database has no idea about your Capitalize function, the query will fail. What you can do instead is to get the correct data from the database and convert the IQueryable to an IEnumerable (which is basically just a way to iterate through collections in-memory) to do the rest of the operation in local memory, as in;
db.Users.Select(x => x.Name) // Gets only the name from the database
.AsEnumerable() // Do the rest of the operations in memory
.Select(x => Capitalize(x)) // Capitalize in memory
.ToList();
The most important thing when it comes to performance of IQueryable vs. IEnumerable from the side of EF, is that you should always try to filter the data using an IQueryable to get as little data as possible to convert to an IEnumerable. What the AsEnumerable call basically does is to tell the database "give me the data as it is filtered now", and if you didn't filter it, you'll get everything fetched to memory, even data you may not need.
IEnumerable represents a sequence of elements which you enumerate one by one until you find the answer you need, so for example if I wanted all entities that had some property greater than 10, I'd need to go through each one in turn and return only those that matched. Pulling every row of a database table into memory in order to do this would not maybe be a great idea.
IQueryable on the other hand represents a set of elements on which operations like filtering can be deferred to the underlying data source, so in the filtering case, if I were to implement IQueryable on top of a custom data source (or use LINQ to Entities!) then I could give the hard work of filtering / grouping etc to the data source (e.g. a database).
The major downside of IQueryable is that implementing it is pretty hard - queries are constructed as Expression trees which as the implementer you then have to parse in order to resolve the query. If you're not planning to write a provider though then this isn't going to hurt you.
Another aspect of IQueryable that it's worth being aware of (although this is really just a generic caveat about passing processing off to another system that may make different assumptions about the world) is that you may find things like string comparison work in the manner they are supported in the source system, not in the manner they are implemented by the consumer, e.g. if your source database is case-insensitive but your default comparison in .NET is case-sensitive.
I query a database and push the result out to the console and a file using two methods like this:
var result = from p in _db.Pages
join r in rank on p.PkId equals r.Key
orderby r.NumPages descending
select new KeyNameCount
{
PageID = p.PkId,
PageName = p.Name,
Count = r.NumPages
};
WriteFindingsToFile(result, parentId);
WriteFindingsToConsole(result, parentId);
IEnumerable<T>, not IQuerable<T> is used as parametertype for result when used as a parameter in the two method calls.
In both calls the result is iterated in a foreach. This leads to two identical calls against the database, one for each method.
I could refactor the two methods into one and only use one foreach, but that would fast become very hard to maintain (adding write to html, write to xml, etc.)
I am pretty sure this is a farly common question, but using google has not made me any wiser, so I turn to you guys :-)
Every time you access a LINQ query it will requery the database to refresh the data. To stop this happening use .ToArray() or .ToList().
e.g.
var result = (from p in _db.Pages
select p).ToList(); //will query now
Write(result); //will not requery
Write(result2); //will not requery
It's important to understand that a raw LINQ query is run when it is used, not when it is written in the code (e.g. don't dispose of your _db before then).
It can be surprising when you realise how it really works. This allows method chaining and later modification of the query to be reflected in the final query run on the DB. It is important to always keep in mind as it can cause run-time bugs that will not be caught at compile time, usually because the DB connection is closed before the list is used, as you are passing around what appear to be a simple IEnumerable.
EDIT: Changed to remove my opinion and reflect the discussion in the comments. Personally I think the compiler should assume that the end result of chained queries is immediately run unless you explicitly say that it'll be further modified later. Just to avoid the run-time bugs it inevitably causes.
If you look at the function definition for IQuerayble, you will see that it also implements IEnumerable. So you can pass IQueryable as parameter to an IEnumerable function without actually enumerating it.
But because of Linqs deffered execution pattern, the IQueryable will only be executed against the database when you iterate over it (with a for loop for example as in your case, or with ToList() or functions like First/Single).
Here is a blog post that explains how this works.
If you change your code to the following you will hit the database only once and then pass the result in memory to your functions:
var result = (from p in _db.Pages
join r in rank on p.PkId equals r.Key
orderby r.NumPages descending
select new KeyNameCount
{
PageID = p.PkId,
PageName = p.Name,
Count = r.NumPages
}).ToList();
WriteFindingsToFile(result, parentId);
WriteFindingsToConsole(result, parentId);
Any time you iterate over your LINQ result the databased will be queried too.
This is called Deferred query execution. You may have a deeper look into at one (out of many) corresponding MSDN articles!
Execution of the query is deferred until the query variable is iterated over in a foreach or For Each loop
Given the following LINQ to SQL query:
var test = from i in Imports
where i.IsActive
select i;
The interpreted SQL statement is:
SELECT [t0].[id] AS [Id] .... FROM [Imports] AS [t0] WHERE [t0].[isActive] = 1
Say I wanted to perform some action in the select that cannot be converted to SQL. Its my understanding that the conventional way to accomplish this is to do AsEnumerable() thus converting it to a workable object.
Given this updated code:
var test = from i in Imports.AsEnumerable()
where i.IsActive
select new
{
// Make some method call
};
And updated SQL:
SELECT [t0].[id] AS [Id] ... FROM [Imports] AS [t0]
Notice the lack of a where clause in the executed SQL statement.
Does this mean the entire "Imports" table is cached into memory?
Would this slow performance at all if the table contained a large amount of records?
Help me to understand what is actually happening behind the scenes here.
The reason for AsEnumerable is to
AsEnumerable(TSource)(IEnumerable(TSource))
can be used to choose between query
implementations when a sequence
implements IEnumerable(T) but also has
a different set of public query
methods available
So when you were calling the Where method before, you were calling a different Where method from the IEnumerable.Where. That Where statement was for LINQ to convert to SQL, the new Where is the IEnumerable one that takes an IEnumerable, enumerates it and yields the matching items. Which explains why you see the different SQL being generated. The table will be taken in full from the database before the Where extension will be applied in your second version of the code. This could create a serious bottle neck, because the entire table has to be in memory, or worse the entire table would have to travel between servers. Allow SQL server to execute the Where and do what it does best.
At the point where the enumeration is enumerated through, the database will then be queried, and the entire resultset retrieved.
A part-and-part solution can be the way. Consider
var res = (
from result in SomeSource
where DatabaseConvertableCriterion(result)
&& NonDatabaseConvertableCriterion(result)
select new {result.A, result.B}
);
Let's say also that NonDatabaseConvertableCriterion requires field C from result. Because NonDatabaseConvertableCriterion does what its name suggests, this has to be performed as an enumeration. However, consider:
var partWay =
(
from result in SomeSource
where DatabaseConvertableCriterion(result)
select new {result.A, result.B, result.C}
);
var res =
(
from result in partWay.AsEnumerable()
where NonDatabaseConvertableCriterion select new {result.A, result.B}
);
In this case, when res is enumerated, queried or otherwise used, as much work as possible will be passed to the database, which will return enough to continue the job. Assuming that it is indeed really impossible to rewrite so that all the work can be sent to the database, this may be a suitable compromise.
There are three implementations of AsEnumerable.
DataTableExtensions.AsEnumerable
Extends a DataTable to give it an IEnumerable interface so you can use Linq against the DataTable.
Enumerable.AsEnumerable<TSource> and ParallelEnumerable.AsEnumerable<TSource>
The AsEnumerable<TSource>(IEnumerable<TSource>) method has no effect
other than to change the compile-time type of source from a type that
implements IEnumerable<T> to IEnumerable<T> itself.
AsEnumerable<TSource>(IEnumerable<TSource>) can be used to choose
between query implementations when a sequence implements
IEnumerable<T> but also has a different set of public query methods
available. For example, given a generic class Table that implements
IEnumerable<T> and has its own methods such as Where, Select, and
SelectMany, a call to Where would invoke the public Where method of
Table. A Table type that represents a database table could have a
Where method that takes the predicate argument as an expression tree
and converts the tree to SQL for remote execution. If remote execution
is not desired, for example because the predicate invokes a local
method, the AsEnumerable<TSource> method can be used to hide the
custom methods and instead make the standard query operators
available.
In other words.
If I have an
IQueryable<X> sequence = ...;
from a LinqProvider, like Entity Framework, and I do,
sequence.Where(x => SomeUnusualPredicate(x));
that query will be composed and run on the server. This will fail at runtime because the EntityFramework doesn't know how to convert SomeUnusualPredicate into SQL.
If I want that to run the statement with Linq to Objects instead, I do,
sequence.AsEnumerable().Where(x => SomeUnusualPredicate(x));
now the server will return all the data and the Enumerable.Where from Linq to Objects will be used instead of the Query Provider's implementation.
It won't matter that Entity Framework doesn't know how to interpret SomeUnusualPredicate, my function will be used directly. (However, this may be an inefficient approach since all rows will be returned from the server.)
I believe the AsEnumerable just tells the compiler which extension methods to use (in this case the ones defined for IEnumerable instead of those for IQueryable).
The execution of the query is still deferred until you call ToArray or enumerate on it.