We have a DB table having a binary column. Each row stores the file content of a file. We want to avoid loading the file content (each can be 10-20Mb in size) into app server's memory unless when necessary.
In the following queries, will the file content be loaded into app server's memory? If yes, during which function call?
// Query 1
dataContext.DataFiles.OrderBy(w => w.atimestamp).ToList();
// Query 2
var filesizes = from DataFiles in dataContext.DataFiles
select DataFiles.FileContent.Length;
Linq methods which return IEnumerable<TSource> or IQueriable<TSource> are implemented to be deferred executed.
Being differed executed, only when you call ToList()/ToArray() (or one of the linq extension methods that do not return a collection like Sum()/Max()/FirstOrDefault() etc.) will actually be executed and will retrieve the data
In your first query, the OrderBy() is not yet executed until you try to retrieve specific items or call the ToList(). However, even though this is deferred executed, this one will have to consume the entire collection the moment you requested the first item (whereas Select() for example will continue to be deferred for the entire collection - see this for some more details)
In your second query, after that line, nothing will yet to be executed.
You can read this MSDN blog about linq and deferred execution and this question from yesterday that also askes about the matter of when linq and deferred
Related
I've always worked with Linq and that's why I always brought only the necessary records for operation - obviously everything was hand-coded.
Now, I'm studying Data Binding because I understand that I can speed up the whole process a lot.
However, I have a question about the initial load of the BindingSource. I noticed that the sample codes always contain the .Load () command without specifying an initial filter.
Example:
dbContext ctx = new dbContex();
ctx.Table.Load(); <-- Here is my doubt
myBindingSource.DataSource = ctx.Table.Local.ToBindingList()
Let's assume that this table has 10,000 records. Will it load 10,000 records at once? Doesn't this type of operation make the load very slow and consume a lot of network bandwidth?
According to documentation
One common way to do this is to write a LINQ query and then call ToList on it, only to immediately discard the created list. The Load extension method works just like ToList except that it avoids the creation of the list altogether.
So, if you just call
ctx.Table.Load()
it will load all the data on that table.
You can also query it before calling Load()
context.Posts.Where(p => p.Tags.Contains("stackoverflow")).Load();
To retrieve data, first I wrote LINQ query, that I expect not to executed on database until I call the FirstAsync() method.
var query =
from tkn in Db.Set<TableA>()
where tkn.IsActive == true
where tkn.Token == token
select tkn.RelatedObjectMappedToTableB;
var retrievedObject = await clientQuery.FirstAsync();
The problem is while debugging I can see the values of the related object in the Watch of visual studio, before reaching the call to FirstAsync()
This means to me that database is already queried and EF has not waited until I ask it to do so.
Why is it doing so? Is my opinion about when a linq query is executed wrong?
This means to me that database is already queried and ef has not waited until I ask it to do so.
No, the database is being queried exactly because you asked it to do so.
Is my opinion about when a linq query is executed wrong?
No, it is not, however,
Why is it doing so?
LINQ retrieves the values in a lazy way, which means that it waits until you perform some enumeration on it. Using Visual Studio's Watch Window is asking the query to be evaluated so you can see the actual details.
LINQ support two execution behaviors deferred execution and immediate execution. Deferred execution means that the evaluation of an expression is delayed until its realized value is actually required. It greatly improves performance by avoiding unnecessary execution.
Deferred execution is applicable to any in-memory collection as well as remote LINQ providers like LINQ-to-SQL, LINQ-to-Entities, LINQ-to-XML, etc.
In the provided example in question, it seems that query executed at where but that's not happening actually. The query executed when you call
.FirstAsync(), It means you really need data and C# execute the query, extract data and store data into memory.
If you want immediate execution behavior you can use .ToList() it will extract the complete projected data and save into memory.
var result =(from tkn in Db.Set<TableA>()
where tkn.IsActive == true
& tkn.Token == token
select tkn.RelatedObjectMappedToTableB).ToList();
For more details see article
Assuming the two following possible blocks of code inside of a view, with a model passed to it using something like return View(db.Products.Find(id));
List<UserReview> reviews = Model.UserReviews.OrderByDescending(ur => ur.Date).ToList();
if (myUserReview != null)
reviews = reviews.Where(ur => ur.Id != myUserReview.Id).ToList();
IEnumerable<UserReview> reviews = Model.UserReviews.OrderByDescending(ur => ur.Date);
if (myUserReview != null)
reviews = reviews.Where(ur => ur.Id != myUserReview.Id);
What are the performance implications between the two? By this point, is all of the product related data in memory, including its navigation properties? Does using ToList() in this instance matter at all? If not, is there a better approach to using Linq queries on a List without having to call ToList() every time, or is this the typical way one would do it?
Read http://blogs.msdn.com/b/charlie/archive/2007/12/09/deferred-execution.aspx
Deferred execution is one of the many marvels intrinsic to linq. The short version is that your data is never touched (it remains idle in the source be that in-memory, or in-database, or wherever). When you construct a linq query all you are doing is creating an IEnumerable class that is 'capable' of enumerating your data. The work doesn't begin until you actually start enumerating and then each piece of data comes all the way from the source, through the pipeline, and is serviced by your code one item at a time. If you break your loop early, you have saved some work - later items are never processed. That's the simple version.
Some linq operations cannot work this way. Orderby is the best example. Orderby has to know every piece of data because it possible that the last piece retrieved from the source very well could be the first piece that you are supposed to get. So when an operation such as orderby is in the pipe, it will actually cache your dataset internally. So all data has been pulled from the source, and has gone through the pipeline, up to the orderby, and then the orderby becomes your new temporary data source for any operations that come afterwards in the expression. Even so, orderby tries as much as possible to follow the deferred execution paradigm by waiting until the last possible moment to build its cache. Including orderby in your query still doesn't do any work, immediately, but the work begins once you start enumerating.
To answer your question directly, your call to ToList is doing exactly that. OrderByDescending is caching the data from your datasource => ToList additionally persists it into a variable that you can actually touch (reviews) => where starts pulling records one at a time from reviews, and if it matches then your final ToList call is storing the results into yet another list in memory.
Beyond the memory implications, ToList is additionally thwarting deferred execution because it also STOPS the processing of your view at the time of the call, to entirely process that entire linq expression, in order to build its in-memory representation of the results.
Now none of this is a real big deal if the number of records we're talking about are in the dozens. You'll never notice the difference at runtime because it happens so quick. But when dealing with large scale datasets, deferring as much as possible for as long as possible in hopes that something will happen allowing you to cancel a full enumeration... in addition to the memory savings... gold.
In your version without ToList: OrderByDescending will still cache a copy of your dataset as processed through the pipeline up to that point, internally, sorted of course. That's ok, you gotta do what you gotta do. But that doesn't happen until you actually try to retrieve your first record later in your view. Once that cache is complete, you get your first record, and for every next record you are then pulling from that cache, checking against the where clause, you get it or you don't based upon that where and have saved a couple of in memory copies and a lot of work.
Magically, I bet even your lead-in of db.Products.Find(id) doesn't even start spinning until your view starts enumerating (if not using ToList). If db.Products is a Linq2SQL datasource, then every other element you've specified will reduce into SQL verbiage, and your entire expression will be deferred.
Hope this helps! Read further on Deferred Execution. And if you want to know 'how' that works, look into c# iterators (yield return). There's a page somewhere on MSDN that I'm having trouble finding that contains the common linq operations, and whether they defer execution or not. I'll update if I can track that down.
/*edit*/ to clarify - all of the above is in the context of raw linq, or Linq2Objects. Until we find that page, common sense will tell you how it works. If you close your eyes and imagine implementing orderby, or any other linq operation, if you can't think of a way to implement it with 'yield return', or without caching, then execution is not likely deferred and a cache copy is likely and/or a full enumeration... orderby, distinct, count, sum, etc... Linq2SQL is a whole other can of worms. Even in that context, ToList will still stop and process the whole expression and store the results because a list, is a list, and is in memory. But Linq2SQL is uniquely capable of deferring many of those aforementioned clauses, and then some, by incorporating them into the generated SQL that is sent to the SQL server. So even orderby can be deferred in this way because the clause will be pushed down into your original datasource and then ignored in the pipe.
Good luck ;)
Not enough context to know for sure.
But ToList() guarantees that the data has been copied into memory, and your first example does that twice.
The second example could involve queryable data or some other on-demand scenario. Even if the original data was all already in memory and even if you only added a call to ToList() at the end, that would be one less copy in-memory than the first example.
And it's entirely possible that in the second example, by the time you get to the end of that little snippet, no actual processing of the original data has been done at all. In that case, the database might not even be queried until some code actually enumerates the final reviews value.
As for whether there's a "better" way to do it, not possible to say. You haven't defined "better". Personally, I tend to prefer the second example...why materialize data before you need it? But in some cases, you do want to force materialization. It just depends on the scenario.
I was reading up on LINQ. I know that there are deferred and immediate queries. I know with deferred types running the query when it's enumerated allows any changes to the data set to be reflected in each enumeration. But I can't seem to find an answer if there's a mechanism in place to prevent the query from running if no changes occurred to the data set since the last enumeration.
I read on MSDN referring to LINQ queries:
Therefore, it follows that if a query is enumerated twice it will be executed twice.
Have I overlooked an obvious - but...?
Indeed, there is none. Actually, that's not quite true - some LINQ providers will spot trivial but common examples like:
int id = ...
var customer = ctx.Customers.SingleOrDefault(x => x.Id == id);
and will intercept that via the identity-manager, i.e. it will check whether it has a matching record already in the context; if it does: it doesn't execute anything.
You should note that the re-execution also has nothing to do with whether or not data has changed; it will re-execute (or not) regardless.
There are two take-away messages here:
don't iterate any enumerable more than once: not least, it isn't guaranteed to work at all
if you want to buffer data, put it into a list/array
So:
var list = someQuery.ToList();
will never re-execute the query, no matter how many times you iterate over list. Because list is not a query: it is a list.
A third take-away would be:
if you have a context that lives long enough that it is interesting to ask about data migration, then you are probably holding your data-context for far, far too long - they are intended to be short-lived
I have a fairly complex linq to entities query that I display on a website. It uses paging so I never pull down more than 50 records at a time for display.
But I also want to give the user the option to export the full results to Excel or some other file format.
My concern is that there could potentially be a large # of records all being loaded into memory at one time to do this.
Is there a way to process a linq result set 1 record at a time like you could w/ a datareader so only 1 record is really being kept in memory at a time?
I've seen suggestions that if you enumerate over the linq query w/ a foreach loop that the records will not all be read into memory at once and would not overwelm the server.
Does anyone have a link to something I could read to verify this?
I'd appreciate any help.
Thanks
set the ObjectContext to MergeOption.NoTracking (since it is a read only operation). If you are using the same ObjectContext for saving other data, Detach the object from the context.
how to detach
foreach( IQueryable)
{
//do something
objectContext.Detach(object);
}
Edit: If you are using NoTracking option, there is no need to detach
Edit2: I wrote to Matt Warren about this scenario. And am posting relevant private correspondences here, with his approval
The results from SQL server may not
even be all produced by the server
yet. The query has started on the
server and the first batch of results
are transferred to the client, but no
more are produced (or they are cached
on the server) until the client
requests to continue reading them.
This is what is called ‘firehose
cursor’ mode, or sometimes referred to
as streaming. The server is sending
them as fast as it can, and the client
is reading them as fast as it can
(your code), but there is a data
transfer protocol underneath that
requires acknowledgement from the
client to continue sending more data.
Since IQueryable inherits from IEnumerable, I believe the underlying query sent to the server would be the same. However, when we do a IEnumerable.ToList(), the data reader, which is used by the underlying connection, would start populating the object, the objects get loaded into the app domain and might run out of memory these objects cannot yet be disposed.
When you are using foreach and IEunmerable the data reader reads the SQL result set one at a time, the objects are created and then disposed. The underlying connection might receive data in chunks and might not send a response to SQL Server back until all the chunks are read. Hence you will not run into 'out of memory` exception
Edit3:
When your query is running, you actually can open your SQL Server "Activity Monitor" and see the query, the Task State as SUSPENDED and Wait Type as Async_network_IO - which actually states that the result is in the SQL Server network buffer. You can read more about it here and here
Look at the return value of the LINQ query. It should be IEnumerable<>, which only loads one object at a time. If you then use something like .ToList(), they will all be loaded into memory. Just make sure your code doesn't maintain a list or use more than one instance at a time and you will be fine.
Edit: To add on to what people have said about foreach... If you do something like:
var query = from o in Objects
where o.Name = "abc"
select o;
foreach (Object o in query)
{
// Do something with o
}
The query portion uses deferred execution (see examples), so the objects are not in memory yet. The foreach iterates through the results, but only getting one object at a time. query uses IEnumerator, which has Reset() and MoveNext(). The foreach calls MoveNext() each round until there are no more results.