What is the use of IQueryable in the context of LINQ?
Is it used for developing extension methods or any other purpose?
Marc Gravell's answer is very complete, but I thought I'd add something about this from the user's point of view, as well...
The main difference, from a user's perspective, is that, when you use IQueryable<T> (with a provider that supports things correctly), you can save a lot of resources.
For example, if you're working against a remote database, with many ORM systems, you have the option of fetching data from a table in two ways, one which returns IEnumerable<T>, and one which returns an IQueryable<T>. Say, for example, you have a Products table, and you want to get all of the products whose cost is >$25.
If you do:
IEnumerable<Product> products = myORM.GetProducts();
var productsOver25 = products.Where(p => p.Cost >= 25.00);
What happens here, is the database loads all of the products, and passes them across the wire to your program. Your program then filters the data. In essence, the database does a SELECT * FROM Products, and returns EVERY product to you.
With the right IQueryable<T> provider, on the other hand, you can do:
IQueryable<Product> products = myORM.GetQueryableProducts();
var productsOver25 = products.Where(p => p.Cost >= 25.00);
The code looks the same, but the difference here is that the SQL executed will be SELECT * FROM Products WHERE Cost >= 25.
From your POV as a developer, this looks the same. However, from a performance standpoint, you may only return 2 records across the network instead of 20,000....
In essence its job is very similar to IEnumerable<T> - to represent a queryable data source - the difference being that the various LINQ methods (on Queryable) can be more specific, to build the query using Expression trees rather than delegates (which is what Enumerable uses).
The expression trees can be inspected by your chosen LINQ provider and turned into an actual query - although that is a black art in itself.
This is really down to the ElementType, Expression and Provider - but in reality you rarely need to care about this as a user. Only a LINQ implementer needs to know the gory details.
Re comments; I'm not quite sure what you want by way of example, but consider LINQ-to-SQL; the central object here is a DataContext, which represents our database-wrapper. This typically has a property per table (for example, Customers), and a table implements IQueryable<Customer>. But we don't use that much directly; consider:
using(var ctx = new MyDataContext()) {
var qry = from cust in ctx.Customers
where cust.Region == "North"
select new { cust.Id, cust.Name };
foreach(var row in qry) {
Console.WriteLine("{0}: {1}", row.Id, row.Name);
}
}
this becomes (by the C# compiler):
var qry = ctx.Customers.Where(cust => cust.Region == "North")
.Select(cust => new { cust.Id, cust.Name });
which is again interpreted (by the C# compiler) as:
var qry = Queryable.Select(
Queryable.Where(
ctx.Customers,
cust => cust.Region == "North"),
cust => new { cust.Id, cust.Name });
Importantly, the static methods on Queryable take expression trees, which - rather than regular IL, get compiled to an object model. For example - just looking at the "Where", this gives us something comparable to:
var cust = Expression.Parameter(typeof(Customer), "cust");
var lambda = Expression.Lambda<Func<Customer,bool>>(
Expression.Equal(
Expression.Property(cust, "Region"),
Expression.Constant("North")
), cust);
... Queryable.Where(ctx.Customers, lambda) ...
Didn't the compiler do a lot for us? This object model can be torn apart, inspected for what it means, and put back together again by the TSQL generator - giving something like:
SELECT c.Id, c.Name
FROM [dbo].[Customer] c
WHERE c.Region = 'North'
(the string might end up as a parameter; I can't remember)
None of this would be possible if we had just used a delegate. And this is the point of Queryable / IQueryable<T>: it provides the entry-point for using expression trees.
All this is very complex, so it is a good job that the compiler makes it nice and easy for us.
For more information, look at "C# in Depth" or "LINQ in Action", both of which provide coverage of these topics.
Although Reed Copsey and Marc Gravell already described about IQueryable (and also IEnumerable) enough,mI want to add little more here by providing a small example on IQueryable and IEnumerable as many users asked for it
Example: I have created two table in database
CREATE TABLE [dbo].[Employee]([PersonId] [int] NOT NULL PRIMARY KEY,[Gender] [nchar](1) NOT NULL)
CREATE TABLE [dbo].[Person]([PersonId] [int] NOT NULL PRIMARY KEY,[FirstName] [nvarchar](50) NOT NULL,[LastName] [nvarchar](50) NOT NULL)
The Primary key(PersonId) of table Employee is also a forgein key(personid) of table Person
Next i added ado.net entity model in my application and create below service class on that
public class SomeServiceClass
{
public IQueryable<Employee> GetEmployeeAndPersonDetailIQueryable(IEnumerable<int> employeesToCollect)
{
DemoIQueryableEntities db = new DemoIQueryableEntities();
var allDetails = from Employee e in db.Employees
join Person p in db.People on e.PersonId equals p.PersonId
where employeesToCollect.Contains(e.PersonId)
select e;
return allDetails;
}
public IEnumerable<Employee> GetEmployeeAndPersonDetailIEnumerable(IEnumerable<int> employeesToCollect)
{
DemoIQueryableEntities db = new DemoIQueryableEntities();
var allDetails = from Employee e in db.Employees
join Person p in db.People on e.PersonId equals p.PersonId
where employeesToCollect.Contains(e.PersonId)
select e;
return allDetails;
}
}
they contains same linq. It called in program.cs as defined below
class Program
{
static void Main(string[] args)
{
SomeServiceClass s= new SomeServiceClass();
var employeesToCollect= new []{0,1,2,3};
//IQueryable execution part
var IQueryableList = s.GetEmployeeAndPersonDetailIQueryable(employeesToCollect).Where(i => i.Gender=="M");
foreach (var emp in IQueryableList)
{
System.Console.WriteLine("ID:{0}, EName:{1},Gender:{2}", emp.PersonId, emp.Person.FirstName, emp.Gender);
}
System.Console.WriteLine("IQueryable contain {0} row in result set", IQueryableList.Count());
//IEnumerable execution part
var IEnumerableList = s.GetEmployeeAndPersonDetailIEnumerable(employeesToCollect).Where(i => i.Gender == "M");
foreach (var emp in IEnumerableList)
{
System.Console.WriteLine("ID:{0}, EName:{1},Gender:{2}", emp.PersonId, emp.Person.FirstName, emp.Gender);
}
System.Console.WriteLine("IEnumerable contain {0} row in result set", IEnumerableList.Count());
Console.ReadKey();
}
}
The output is same for both obviously
ID:1, EName:Ken,Gender:M
ID:3, EName:Roberto,Gender:M
IQueryable contain 2 row in result set
ID:1, EName:Ken,Gender:M
ID:3, EName:Roberto,Gender:M
IEnumerable contain 2 row in result set
So the question is what/where is the difference? It does not seem to
have any difference right? Really!!
Let's have a look on sql queries generated and executed by entity
framwork 5 during these period
IQueryable execution part
--IQueryableQuery1
SELECT
[Extent1].[PersonId] AS [PersonId],
[Extent1].[Gender] AS [Gender]
FROM [dbo].[Employee] AS [Extent1]
WHERE ([Extent1].[PersonId] IN (0,1,2,3)) AND (N'M' = [Extent1].[Gender])
--IQueryableQuery2
SELECT
[GroupBy1].[A1] AS [C1]
FROM ( SELECT
COUNT(1) AS [A1]
FROM [dbo].[Employee] AS [Extent1]
WHERE ([Extent1].[PersonId] IN (0,1,2,3)) AND (N'M' = [Extent1].[Gender])
) AS [GroupBy1]
IEnumerable execution part
--IEnumerableQuery1
SELECT
[Extent1].[PersonId] AS [PersonId],
[Extent1].[Gender] AS [Gender]
FROM [dbo].[Employee] AS [Extent1]
WHERE [Extent1].[PersonId] IN (0,1,2,3)
--IEnumerableQuery2
SELECT
[Extent1].[PersonId] AS [PersonId],
[Extent1].[Gender] AS [Gender]
FROM [dbo].[Employee] AS [Extent1]
WHERE [Extent1].[PersonId] IN (0,1,2,3)
Common script for both execution part
/* these two query will execute for both IQueryable or IEnumerable to get details from Person table
Ignore these two queries here because it has nothing to do with IQueryable vs IEnumerable
--ICommonQuery1
exec sp_executesql N'SELECT
[Extent1].[PersonId] AS [PersonId],
[Extent1].[FirstName] AS [FirstName],
[Extent1].[LastName] AS [LastName]
FROM [dbo].[Person] AS [Extent1]
WHERE [Extent1].[PersonId] = #EntityKeyValue1',N'#EntityKeyValue1 int',#EntityKeyValue1=1
--ICommonQuery2
exec sp_executesql N'SELECT
[Extent1].[PersonId] AS [PersonId],
[Extent1].[FirstName] AS [FirstName],
[Extent1].[LastName] AS [LastName]
FROM [dbo].[Person] AS [Extent1]
WHERE [Extent1].[PersonId] = #EntityKeyValue1',N'#EntityKeyValue1 int',#EntityKeyValue1=3
*/
So you have few questions now, let me guess those and try to answer them
Why are different scripts generated for same result?
Lets find out some points here,
all queries has one common part
WHERE [Extent1].[PersonId] IN (0,1,2,3)
why? Because both function IQueryable<Employee> GetEmployeeAndPersonDetailIQueryable and
IEnumerable<Employee> GetEmployeeAndPersonDetailIEnumerable of SomeServiceClass contains one common line in linq queries
where employeesToCollect.Contains(e.PersonId)
Than why is the
AND (N'M' = [Extent1].[Gender]) part is missing in IEnumerable execution part, while in both function calling we used Where(i => i.Gender == "M") inprogram.cs`
Now we are in the point where difference came between IQueryable and
IEnumerable
What entity framwork does when an IQueryable method called, it tooks linq statement written inside the method and try to find out if more linq expressions are defined on the resultset, it then gathers all linq queries defined until the result need to fetch and constructs more appropriate sql query to execute.
It provide a lots of benefits like,
only those rows populated by sql server which could be valid by the
whole linq query execution
helps sql server performance by not selecting unnecessary rows
network cost get reduce
like here in example sql server returned to application only two rows after IQueryable execution` but returned THREE rows for IEnumerable query why?
In case of IEnumerable method, entity framework took linq statement written inside the method and constructs sql query when result need to fetch. it does not include rest linq part to constructs the sql query. Like here no filtering is done in sql server on column gender.
But the outputs are same? Because 'IEnumerable filters the result further in application level after retrieving result from sql server
SO, what should someone choose?
I personally prefer to define function result as IQueryable<T> because there are lots of benefit it has over IEnumerable like, you could join two or more IQueryable functions, which generate more specific script to sql server.
Here in example you can see an IQueryable Query(IQueryableQuery2) generates a more specific script than IEnumerable query(IEnumerableQuery2) which is much more acceptable in my point of view.
It allows for further querying further down the line. If this was beyond a service boundary say, then the user of this IQueryable object would be allowed to do more with it.
For instance if you were using lazy loading with nhibernate this might result in graph being loaded when/if needed.
Related
I have a situation where the translated sql direct form Visual Studio is returning a different number of records from the Linq. I am expecting 4 items but in the count() of the resulting list I only have 1. This is the case despite creating more records - it always returns 1.
db.DCLVUnknowns.Where(x => x.DCLVUnknownId == Report.DCLid).ToList();
SELECT
[Extent1].[DCLVUnknownId] AS [DCLVUnknownId],
[Extent1].[Gender] AS [Gender],
[Extent1].[Height] AS [Height],
[Extent1].[Weight] AS [Weight],
[Extent1].[Age] AS [Age],
[Extent1].[Race] AS [Race],
[Extent1].[DCLid] AS [DCLid]
FROM [dbo].[DCLVUnknown] AS [Extent1]
Strange thing is I have the same linq expression running fine for other entities and there is no problem. It is consistently happening at the same spot every time.
db.DCLVUnknowns
Is a query of the entire table, not the query for what you want.
If you want to inspect the IQueryable of the full query, try:
var results = db.DCLVUnknowns.Where(x => x.DCLVUnknownId == Report.DCLid);
var theResultSet = results.ToList();
Here results should translate as roughly:
SELECT
[Extent1].[DCLVUnknownId] AS [DCLVUnknownId],
[Extent1].[Gender] AS [Gender],
[Extent1].[Height] AS [Height],
[Extent1].[Weight] AS [Weight],
[Extent1].[Age] AS [Age],
[Extent1].[Race] AS [Race],
[Extent1].[DCLid] AS [DCLid]
FROM [dbo].[DCLVUnknown] AS [Extent1]
WHERE [Extent1].[DCLVUnknownId] = DCLid
Assuming DCLVUnknownId is a PK/ Identity, you should see one result in theResultSet.
In LINQ, I am trying to inner join custom function written for full-text search and an Iqueryable result.
However, I get the following error when I try to to_ret.select(--something--).ToList()
Nested query does not have appropriate key
LINQ Code:
var sql_query = db.search(st);
var to_ret = from ts in sql_query
from t in table
where t.Id == ts.Value select t;
to_ret = to_ret.Include(x => x.table1)
.Include(x=> x.table2.Select(y=> y.table2Col));
to_ret.select(-something-).toList();
SQL Code:
create function [dbo].[search]
(#keywords nvarchar(4000))
returns table
as
return (
select [key] from containstable(tb,(Name,Description),#keywords)
)
Code that works in place of above LINQ Code :
var ids = (from t in table join ts in db.search(st) on t.Id equals ts.Value select t.Id).ToList();
to_ret = to_ret.Where(x => ids.Contains(x.Id));
However, the code that works isn't efficient enough as it eagerly loads all the ids for comparison
Do not join the table using LINQ, it is not effective.
You need to include all the joined functions into [dbo].[search] table valued function (like a view). Then do just call the [dbo].[search] from EF and filter it.
I have mentioned joined Fulltext table valued function here.
Note that fulltext and filtering together in one query could take time, because it is not easy job for query optimizer. Query optimizer selects to perform first fulltext on entire table(s) and then filtering or the opposite way.
I am facing serious performance issues... My query is supposed to filter Products with SQL directly in database. When I execute this code, it doesn't, and it returns all products and filters them in C#.
MyContext context = new MyContext();
Func<Product, bool> query = (p => p.UPC.StartsWith("817"));
var products = context.Products.Where(query).Take(10);
I've noticed that the products variable is of type TakeIterator. When I change the code slightly, I get the filtering OK, but it forces me to put the query logic directly in the same method, which is what I want to avoid.
MyContext context = new MyContext();
var products = context.Products.Where(p => p.UPC.StartsWith("817")).Take(10);
This second version is of an undisclosed type by the Visual Studio debugger, but it shows as the query I am trying to end with, which is good!
{SELECT TOP (10)
[Extent1].[Id] AS [Id],
[Extent1].[Brand] AS [Brand],
[Extent1].[Description] AS [Description],
[Extent1].[UPC] AS [UPC]
FROM [dbo].[Products] AS [Extent1]
WHERE [Extent1].[UPC] LIKE N'817%'}
I need to figure out how to get a Func passed as an argument and execute the query in the same fashion as the first C# code excerpt, but with optimisations of the second.
Try this instead:
MyContext context = new MyContext();
Expression<Func<Product, bool>> query = (p => p.UPC.StartsWith("817"));
var products = context.Products.Where(query).Take(10);
And see this question for reference on why:
Why would you use Expression<Func<T>> rather than Func<T>?
The accepted answer there is so complete that I don't dare try to explain it better!
The following code:
using (var db = new Entities())
{
db.Blogs.First().Posts.Skip(10).Take(5).ToList();
}
Will generate the following SQL:
-- statement #1
SELECT TOP ( 1 ) [c].[Id] AS [Id],
[c].[Title] AS [Title],
[c].[Subtitle] AS [Subtitle],
[c].[AllowsComments] AS [AllowsComments],
[c].[CreatedAt] AS [CreatedAt]
FROM [dbo].[Blogs] AS [c]
-- statement #2
SELECT [Extent1].[Id] AS [Id],
[Extent1].[Title] AS [Title],
[Extent1].[Text] AS [Text],
[Extent1].[PostedAt] AS [PostedAt],
[Extent1].[BlogId] AS [BlogId],
[Extent1].[UserId] AS [UserId]
FROM [dbo].[Posts] AS [Extent1]
WHERE [Extent1].[BlogId] = 1 /* #EntityKeyValue1 */
(from http://ayende.com/blog/4351/nhibernate-vs-entity-framework-4-0)
NB The Skip and Take have not been translated to SQL resulting in ALL posts from the blog being loaded from the database, instead of just the 5 we require.
This seems dangerously, horribly inefficient. Unbelievably so, what gives?
The reason it's happening is the call to First, which is causing the Blog object to be materialized. Any further traversal requires more queries.
Try db.Blogs.Take(1).SelectMany(b => b.Posts).Skip(10).Take(5).ToList(); instead to do it in one query. You probably want to add some sort of ordering of blogs before the .Take(1), to ensure a deterministic result.
Edit
You actually have to use OrderBy before Skip (otherwise LINQ to Entities will throw an exception), which makes it something like:
db.Blogs.OrderBy(b => b.Id).Take(1) // Filter to a single blog (while remaining IQueryable)
.SelectMany(b => b.Posts) // Select the blog's posts
.OrderBy(p => p.PublishedDate).Skip(10).Take(5).ToList(); // Filter to the correct page of posts
As he suggests in his post, you could use EQL to perform this query instead. Something like:
// Create a query that takes two parameters.
string queryString =
#"SELECT VALUE product FROM
AdventureWorksEntities.Products AS product
order by product.ListPrice SKIP #skip LIMIT #limit";
ObjectQuery<Product> productQuery =
new ObjectQuery<Product>(queryString, context);
// Add parameters to the collection.
productQuery.Parameters.Add(new ObjectParameter("skip", 3));
productQuery.Parameters.Add(new ObjectParameter("limit", 5));
// Iterate through the collection of Contact items.
foreach (Product result in productQuery)
Console.WriteLine("ID: {0}; Name: {1}",
result.ProductID, result.Name);
Code taken from here: http://msdn.microsoft.com/en-us/library/bb738702.aspx
You can try to get your first blog and use the blog id to filter posts like this:
Blog blog = db.Blogs.First();
blog.posts = Posts.Where(r=>r.blogID=blog.id).Skip(10).Take(5).ToList();
I use asp.net 4 c# and ef4.
I have this code, it should compile a query and return a single scalar value (I use anonymous type).
My code does not have apparently errors, but because is the first time I write a compiled query I would like to know if is well written or could be improved for a performance boost.
var query = CompiledQuery.Compile((CmsConnectionStringEntityDataModel ctx)
=> from o in ctx.CmsOptions
where o.OptionId == 7
select new
{
Value = o.Value
});
uxHtmlHead.Text = query(context).FirstOrDefault().Value;// I print the result in a Label
SQL Profile Output:
SELECT TOP (1)
[Extent1].[OptionId] AS [OptionId],
[Extent1].[Value] AS [Value]
FROM [dbo].[CmsOptions] AS [Extent1]
WHERE 7 = [Extent1].[OptionId]
Many Thanks
Result after Wouter advice (please guys have a double check again):
static readonly Func<CmsConnectionStringEntityDataModel, int, string> compiledQueryHtmlHead =
CompiledQuery.Compile<CmsConnectionStringEntityDataModel, int, string>(
(ctx, id) => ctx.CmsOptions.FirstOrDefault(o => o.OptionId == id).Value);
using (var context = new CmsConnectionStringEntityDataModel())
{
int id = 7;
uxHtmlHead.Text = compiledQueryHtmlHead.Invoke(context, id);
}
Resulting SQL (I do not understand why with a LEFT JOIN)
exec sp_executesql N'SELECT
[Project1].[Value] AS [Value]
FROM ( SELECT 1 AS X ) AS [SingleRowTable1]
LEFT OUTER JOIN (SELECT
[Extent1].[Value] AS [Value]
FROM [dbo].[CmsOptions] AS [Extent1]
WHERE [Extent1].[OptionId] = #p__linq__0 ) AS [Project1] ON 1 = 1',N'#p__linq__0 int',#p__linq__0=7
There are 2 things you can improve on.
First, precompiling a query is definitely a good idea but if you have a look at your code you will see that it precompiles the query each and every time instead of only once.
You need to move the precompiled query to a static variable that is initialized only once.
Another thing you need to be careful of is that when precompiling a query you shouldn't modify the query anymore before executing it.
You are building a precompiled query that will select all rows and then you say 'firstordefault' which changes the precompiled query to a SELECT TOP (1) and you lose the benefit of precompiling. You need to move the FirstOrDefault part inside your precompiled query and return only one result.
Have a look at this documentation. If you look at the examples you can see how they use a static field to hold the compiled query and how they specify the return value.