I have a table of 200,000 record where I am getting only the top 10 using .Take() but it is taking about 10 seconds to get the data.
My question is: does the .Take() method get all the data from the database and filter the top 10 on the client side?
Here is my code:
mylist = (from mytable in db.spdata().OrderByDescending(f => f.Weight)
group feed by mytable.id into g
select g.FirstOrDefault()).Take(10).ToList();
spdata() is a function Import from stored procedure.
Thanks
The stored procedure probably returns a lot of data to the client which is very slow. You cannot remote a query to an sproc. That would be possible using a view or a table-valued function.
There's no way to use an sproc in a query. You can only execute it by itself.
Your intention probably was to execute the Take(10) on the server. For that to work you need to switch to an inline query, a view or a TVF.
The extension method Take does not fetch all the results from the database. That is not how Takeworks.
However your db.spdata() call probably does fetch all rows.
I'm not 100% sure but as I remember you get an IEnumerable result when you call an SP using EF DataContext...
There are a couple of was to optimize the performance:
Pass the search criteria s as SP params and do the filtering in the stored procedure.
Or if you have a quite simple query in the SP where you are not declaring any variables and where you are just joining some tables then:
Create an indexed view where specify the query that you need and call the Take method on it.
What this will give you? You can map to the created view and EF will now be returning an IQueryable result and not an IEnumerable. This will optimize the sql command and rather the receiving all of the data and then taking the 10 elements that you need, a sql command that just retrieves the 10 elements will be formed.
I also advice you to see what is the deference between IEnumerable vs IQueryable.
It does, because you are sorting the data before grouping, which is not possible to do in SQL.
You should use an aggregate to get the highest weight from each group, then sort the weights to get the ten largest:
mylist = (
from mytable in db.spdata()
group feed by mytable.id into g
select g.Max(f => f.Weight)
).OrderByDescending(w => w).Take(10).ToList();
Related
I have a question about LINQ pagination and stored procedure pagination:
If I am using LINQ pagination, then what logic use Microsoft? Means it load all data into list and then performed pagination logic?
For example, if I have 1 million records in a SQL Server database and I want to use LINQ pagination then how will it work?
Code snippet:
_list = _list.Skip(StartIndex).Take(FetchRecords).ToList();
If LINQ loads all records first, then do I have to go for stored procedure pagination?
You need to remember that
The ToList(IEnumerable) method forces immediate query evaluation and returns a List that contains the query results.
https://learn.microsoft.com/en-us/dotnet/api/system.linq.enumerable.tolist
ie without ToList method call, query is not executed hence no data.
Secondly you should also use OrderBy method eg. OrderBy(o => o.ID); as you should explicitly know in which order items are skipped and taken.
Further consider using 'where' and 'select' method to limit the data payload.
I have a Linq query that reads from a SQL table and 1 of the fields it returns are from a custom function (in C#).
Something like:
var q = from my in MyTable
select new
{
ID = my.ID,
Amount = GetAmount(ID)
};
If I do a q.Dump() in LinqPad, it shows the results, which tells me that it runs the custom function without trying to send it to SQL.
Now I want to union this to another query, with:
var q1 = (from p in AnotherQuery.Union(q)...
and the I get the error that Method has no supported translation to SQL.
So, my logic tells me that I need to dump q in memory and then try to union to that. I've tried doing that with ToList() and creating a secondary query that populates itself from the List, but that leads to a long list of different errors. Am I on the right track, by trying to get q in memory and union on that, or are there better ways of doing this?
You can't use any custom functions in a LINQ query that gets translated - only the functions supported by the given LINQ provider. If you want your query to happen on the server, you need to stick with the supported functions (even if it sometimes means having to inline code that would otherwise be reused).
The difference between your two queries boils down to when (and where) the projection happens. In your first case, the data from MyTable is returned from the DB - in your sample, just the ID. Then, the projection happens on top of this - the GetAmount method is called in your application for each of ID.
On the other hand, there's no such way for this to happen in your second query, since you're not using GetAmount in the final projection.
You either need to replace the custom function with inlined query the provider understands, or refactor all your queries to use the supported functions in addition with whatever you need to do in-memory. There's no point in giving you any sample code, since it depends entirely on your actual query, and what you're really trying to query for.
im calling a table with 200.000 rows and 6 columns, but i only want 2 of these columns to be used in one controller, so i want to know if there is a better way to call them from the server without compromising performance, because as i know Linq queries get the whole table and them makes the filtering, i think maybe Views is a good way, but i want to know if there are others and betters, Thanks.
for example:
var items = from i in db.Items select new {i.id,i.name};
in case i have 1.000.000 items, will it be a trouble for the server?
Your initial assumption is incorrect.
In general LINQ queries do not get the whole table. the query is converted into a "server side expression" (i.e. a SQL statement) and the statement is resolved on the server and only the requested data is returned.
Given the statement you provided you will return only two columns but you will get 1,000,000 objects in the result if you do not do any filtering. But that isn't a problem with LINQ, that's a problem with you not filtering. If you included a where clause you would only get the rows you requested.
var items = from i in db.Items
where i.Whatever == SomeValue
select new { i.id, i.name };
Your original query would be translated (roughly) into the following SQL:
SELECT id, name FROM Items
You didn't include a where clause so you're going to get everything.
With the version that included a where clause you'd get the following SQL generated:
SELECT id, name FROM Items WHERE Whatever = SomeValue
Only the rows that match the condition would be returned to your application and converted into objects.
I have an app using Entity Framework. I want to add a tree view listing products, grouped by their categories. I have an old SQL query that will grab all of the products and categories and arrange them into parent nodes and children. I am trying to translate it into LINQ that uses the EF. But the SQL has a WITH sub-query that I am not familiar with using. I have tried using Linqer and LinqPad to sort it out, but they choke on the WITH clause and I am not sure how to fix it. Is this sort of thing possible in LINQ?
Here is the query:
declare #id int
set #id=0
WITH ChildIDs(id,parentid,type,ChildLevel) AS
(
SELECT id,parentid,type,0 AS ChildLevel
FROM dbo.brooks_product
WHERE id = #id
UNION ALL
SELECT e.id,e.parentid,e.type,ChildLevel + 1
FROM dbo.brooks_product AS e
INNER JOIN ChildIDs AS d
ON e.parentid = d.id
WHERE showitem='yes' AND tribflag=1
)
SELECT ID,parentid,type,ChildLevel
FROM ChildIDs
WHERE type in('product','productchild','productgroup','menu')
ORDER BY ChildLevel, type
OPTION (MAXRECURSION 10);
When I run the query, I get data that looks like this (a few thousand rows, truncated here):
ID.....parentid.....type.....ChildLevel
35429..0............menu.....1
49205..0............menu.....1
49206..49205........menu.....2
169999.49206........product..3
160531.169999.......productchild..4
and so on.
The WITH block is a Common Table Expression, and in this case is used to create a recursive query.
This will be VERY difficult in Linq as Linq doesn't play well with recursion. If you need all of the data on one result set that a Stored Procedure would be easier. Another option is to do the recursion in C# (not in Linq but a recursive function) and do multiple round-trips. The performance will not be as good but if you result set is small it may not make much difference (and you will get a better object model).
You may be able to solve this using LINQ to Entities, but it is non-trivial and I suspect it will be very time consuming.
In situations like this, you may prefer to build a SQL View or Table-Valued Function that returns the results for which you're looking. Then import that View or Table-Valued Function into your EF model and you can pull data directly from it using LINQ.
Querying the View in LINQ is no different than querying a table.
To get data from a Table-Valued Function in LINQ, you pass the function's parameters in after the name of the function, like so:
var query = from tvf in _db.MyTableValuedFunction(parameters)
select tvf;
EDIT
As suggested by #thepirat000, Table-Valued Function support is not available in Entity Framework versions prior to version 5. In order to use this functionality, EF must be running with .NET 4.5 or higher.
At the end of the day, I could not get this to work. I ended up writing out a SQL query dynamically and sending that straight to the database. It works fine, and I am not relying on any direct user input so there is no chance of SQL injection. But it seems so old school! For the rest of my program I am using EF and LINQ.
Thanks for the replies!
I have some database ang now it contains a table with about 100 rows. But in future it will have not 100 but 1 000 000+ rows and I have to be careful with my web application I'm developing now.
Problem is next: at web page I need to create paged list what will show records to user. And here is a sample of code that I plan to use
public IQueryable<MyTable> GetRows(int from, int to)
{
var queryRes = (from row in SomeDataContext.MyTable
order by row.id
select row).AsQueriable();
return queryRes.Take(to).Skip(from);
}
It is only sample of code. I did not run it.
But question is what will go on in this case? I see tow scenarios
It will load all rows from database and at server side and records in range from 'from' to 'to' will be returned. Other will be ignored. In this case my application will have big troubles. Imagine load 1 000 000 rows from database every time. It will be disaster.
It will construct SQL request what will return only rows I need without loading others. That's exactly what I need.
I think that it will be 2 scenario but I'm not sure and can't check it. Am I correct?
As a side-note, you don't have to call AsQueryable. It is enough to do
var queryRes = SomeDataContext.MyTable.OrderBy(r => r.Id);
return queryRes.Take(to).Skip(from);
And to answer your question - scenario 2 will be executed. You can always check the generated SQL by using the SQL Server Profiler, but in case you are using Entity Framework, you can even do queryRes.ToString(). And as #Aron correctly pointed out - the query will be actually executed against the database only when enumerating the results (e.g. calling queryRes.ToList()).
These questions address the issue of looking up the SQL code in more detail:
How to view generated SQL from Entity Framework?
exact sql query executed by Entity Framework
Strictly speaking, neither 1 nor 2 is correct. Running the code DOES NOT hit the database. It constructs an expression tree. The calling code can still modify the expression tree further without hitting the database.
With the IQueryable interface no SQL is run. It is at the point when you call IEnumerable.GetEnumerator() that the underlying Linq Provider converts the WHOLE expression into a query. In this case a SQL query, and then run it.
So for example, with this code. You could have
void Main()
{
var foo = from x in GetRows(10, 10)
where x.Id > 1000
select x;
foreach(var f in foo)
{
//Stuff
}
}
The sql that is actually run will actually be closer to
SELECT a,b,c FROM
(SELECT a,b,c, ROW_NUMBER() OVER (ORDER BY ...) as row_number
FROM Table
WHERE id > 1000) t0
WHERE to.row_number BETWEEN 10 and 20;
To be honest you are going about this wrong. You don't need a GetRows method. I would directly call the Linq query when constructing the table itself. You should take a look at the IRepository pattern that MVC scaffolding uses.
Finally if this is meant to be called as a WebQuery for AJAX I would look at the two OData implementations in .net (WCF Data Services and WebAPI OData).
You are right.
The 2. scenario is what will happen. When the query is eventuallty exectuted.
I Would sugges to reverse the Take - Skip, so you start by Skip
queryRes.Skip(from).Take(to)
Debuggen this method will not make any calls to the database. It just returns the query - not the resualt.
If you want to test exactly what will happen, try download LinqPad - it is a great to for demystifying linq queries.