We have several different projects using ASP.NET and DevExpress ASPxGridView components. Throughout the development of these projects, several techniques on databinding have been used and we're now finding that some of these projects are eating up all the memory on the server.
Originally, we were using a call to a stored procedure and binding a DataSet to the the gridview, but on DX recommendation, modified this to an ObjectDataSource and created and object that ultimately uses a Linq statement against the DB and returns a generic list of objects which is then bound.
Unfortunately, this does not cure the problem at hand. We're still noticing large amounts of memory being eaten up and I'm trying to get to the bottom of this. When running through RedGate memory profiler, I notice that there are lots of strings, RuntimeTypeHandles and instances of my object created everytime we rebind to the grid.
The DataBind is done on page load, and the grid uses postbacks on sorting, but this is causing MBs of memory to leak on every bind, so I'm wondering what techniques I can use / best practices for managing the objects we have control over? I've implemented IDisposable in the data object, disposing of the linq context and setting any other objects to null, but it doesn't seem to make a difference. I seem to be creating an instance of the data object on every call, and even calling dispose makes no difference.
Wow, lots of plumbing and moving parts in there.
Is it possible to narrow things down a bit? That is, can you strip stuff off the page and see how it performs?
Forgive this, but when you say 'leaking memory' what do you mean and how do you know? The GC is 'lazy' and won't do anything until there is pressure to do so. This is a good thing but it also means memory may appear to accumulate until a collection is needed, and then you may find it frees a lot up. Memory profilers often look like a saw-tooth for this reason.
How are you storing the grid data to make the paging work? I've seen datasets persisted in viewstate, which means the data goes to the client along with the grid. If you're querying again on post-back page-load you're wasting a lot of space there.
Another common problem is event subscriptions keeping large objects alive longer than they should. I've actually seen code where a datagrid was placed in session state which kept the page alive for as long as the session was. On each post-back this happened again and again until poof. In this case, GC couldn't help us becuase the objects were indeed still 'in-use'.
So try to simplify - turn off sorting, get rid of the 3rd party control, use a smaller data set, etc. Using a memory profiler and something that puts the server under pressure, measure this scenario. If you find no 'leaks' then start adding stuff back to see when it goes haywire.
You may be returning too much data to your iis server each time. Remember that using standard linq datasource with the devexpress grid, each time you do a callback for sorting, or paging or any other callback, the whole data is loaded in memory and then sorted and paged.
This means that if you are loading a very large amount of data you will easily waste server memory. Think that you may have much users opening the same page and this will load the whole data in memory for each user, and the GC may not have time enough to free all that stuff.
DevExpress provides for this the LinqServerModeDataSource, that does all the paging and sorting in the data server.
If you cannot use that, try to retrieve a smaller set of data by filtering it.
Related
I apologize if this question is a bit nebulous.
I am writing a C# application which does data manipulation against a SQL Server database. For a group of items, I read data for each item, do calculations on the data, then write the results to the database.
The problem I am having is that the application starts to slow down relative to the time it takes to process each item when the number of items to be processed increases.
I am trying to be very careful as far as freeing memory for allocated objects as I am through with them. I want to have nothing hanging around from the processing of one item when I start the processing for the next item. I make use of "using" structures for data tables and the BulkCopy class to try to force memory cleanup.
Yet, I start to get geometrically increasing run times per item the more items I try to process in one invocation of the program.
My program is a WinForms app. I don't seem to be eating up the server's memory with what I am doing. I am trying to make the processing of each item isolated from the processing of all other items, to make sure it would not matter how many items I process in each invocation of the application.
Has anyone seen this behavior in their applications and know what to look for to correct this?
A couple of things you can be watchful for - if you're using "using" statements - are you making sure that you're not keeping your connection open while manipulating your objects? Best to make sure you get your data from the database, close the connection, do your manipulation and then send the data back.
Are you using stored procedures for fetching/sending complex objects? You can also experiment with doing some of you data manipulation inside of the stored procedure or in functions called from them - you do NOT want to offload your entire business classes to the database, but you can do some of it there, depending on what you're doing.
Make sure your data structure is optimized as well (primary key indices, foreign keys, triggers etc. you can get some scripts from http://www.brentozar.com/first-aid/ to check the optimization of your database.
As mentioned above, try using some parallel/asynchronous patterns to divy up your work - await/async is very helpful for this, especially if you want to have calculations while also sending previous data back to the server.
Thanks for all the input. I checked the issues of opening/closing connections, etc. to see that I was being tidy. The thing that really helped was removing the primary keys on the destination data table. These were setup relative to what an end user would require, but they really gummed up the speed of data inserts. A heads up to folks to think about database constraints for updating data vs. using the data.
Also, found performance issues in selecting with a filter from an in memory DataTable. Somehow what I was doing get bogged down with a larger number of rows (30,000). I realized that I was mishandling the data and did not really need to do this. But it did show me the need to micro-test each step of my logic when trying to drag so much data around.
Work on C#.In my application several time need to select\collect datafrom DB.Fro this task I do the bellow step
1)Write SP
2)Execute the Sp
3)Fill result to Generic collection(ORM)
4)By the collection Bind the control
I want to know is there any mechanism or technique \Advanced technique available help to collect data from database.Thanks in advance
When i again and again rapidly hit the db.then it's performance become bottleneck .What to do?
It sounds like you should be caching some results. In a high load application, caching even for a few seconds can have a big impact on performance. There are a myriad of cache solutions out there; if this is a web app, the inbuilt http-context .Cache should be fine (.NET 4.0 adds MemoryCache to do the same more conveniently in non-web applications).
Re loading the data; you mention ORM - in our experience here, we find most ORMs indeed are a bottleneck for "hot" code paths - a subject I'm talking on in a few hours as it happens. Because we faced this problem, we wrote an intentionally simple but really really fast micro-ORM, dapper-dot-net. It isn't as feature rich as some full ORMs, but if you are trying to load data quick for display, it is ideal.
The other thing, of course, is to look at your query and improve the performance. Look in particular at the logical IO reads, and where they are coming from. It could well be that an extra index or a little denormalization could make a really big difference to your query performance.
Ye, but the only exception is to use a DataReader or a DataTable.
For ex. datareader is usefull for limited view of rows from a large collection being retrieved.
However Datatable is important, if to apply functions on a complete collection of Data.
Plus there are different methods like connection pooling, localviews, indexes that will matter most when Data fetched is more than available Server resources.
I work on a big project in company. We collect data which we get via API methods of the CMS.
ex.
DataSet users = CMS.UserHelper.GetLoggedUser(); // returns dataset with users
Now on some pages we need many different data, not just users, also Nodes of the tree of the CMS or specific data of subtreee.
So we thought of write an own "helper class" in which we later can get different data easy.
ex:
MyHelperClass.GetUsers();
MyHelperClass.Objects.GetSingleObject( ID );
Now the problem is our "Helper Class" is really big and now we like to collect different data from the "Helper Class" and write them into a typed dataset . Later we can give a repeater that typed dataset which contains data from different tables. (which even comes from the methods I wrote before via API)
Problem is: It is so slow now, even at loading the page! Does it load or init the whole class??
By the way CMS is Kentico if anyone works with it.
I'm tired. Tried whole night..but it's soooo slow. Please give a look to that architecture.
May be you find some crimes which are not allowed :S
I hope we get it work faster. Thank you.
alt text http://img705.imageshack.us/img705/3087/classj.jpg
Bottlenecks usually come in a few forms:
Slow or flakey network.
Heavy reading/writing to disk, as disk IO is 1000s of times slower than reading or writing to memory.
CPU throttle caused by long-running or inefficiently implemented algorithm.
Lots of things could affect this, including your database queries and indexes, the number of people accessing your site, lack of memory on your web server, lots of reflection in your code, just plain slow hardware etc. No one here can tell you why your site is slow, you need to profile it.
For what its worth, you asked a question about your API architecture -- from a code point of view, it looks fine. There's nothing wrong with copying fields from one class to another, and the performance penalty incurred by wrapper class casting from object to Guid or bool is likely to be so tiny that its negligible.
Since you asked about performance, its not very clear why you're connecting class architecture to performance. There are really really tiny micro-optimizations you could apply to your classes which may or may not affect performance -- but the four or five nanoseconds you'll gain with those micro-optimizations have already been lost simply by reading this answer. Network latency and DB queries will absolutely dwarf the performance subtleties of your API.
In a comment, you stated "so there is no problem with static classes or a basic mistake of me". Performance-wise, no. From a web-app point of view, probably. In particular, static fields are global and initialized once per AppDomain, not per session -- the variables mCurrentCultureCode and mcurrentSiteName sound session-specific, not global to the AppDomain. I'd double-check those to see your site renders correctly when users with different culture settings access the site at the same time.
Are you already using Caching and Session state?
The basic idea being to defer as much of the data loading to these storage mediums as possible and not do it on individual page loads. Caching especially can be useful if you only need to get the data once and want to share it between users and over time.
If you are already doing these things, ore cant directly implement them try deferring as much of this data gathering as possible, opting to short-circuit it and not do the loading up front. If the data is only occasionally used this can also save you a lot of time in page loads.
I suggest you try to profile your application and see where the bottlenecks are:
Slow load from the DB?
Slow network traffic?
Slow rendering?
Too much traffic for the client?
The profiling world should be part of almost every senior programmer. It's part of the general toolbox. Learn it, and you'll have the answers yourself.
Cheers!
First thing first... Enable Trace for your application and try to optimize Response size, caching and work with some Application and DB Profilers... By just looking at the code I am afraid no one can be able to help you better.
I am using a Session variable to pass a datatable from 1 page to another. Sometimes the datatable can contain well over 100,000 records. After running it a few times, I get thrown a Out of Memory exception, so I guess I have a few questions?
Is Session the best way to handle this?
Does Session.Clear("session") release it from Memory? If not, does anything release the Session from memory?
If I store a datatable into a Session object and then I store another datatable into that same Session object, does it keep using up memory or does it write over the existing Session object?
I'll assume you're talking about In-Process session state.
You aren't actually storing the DataTable itself in session. You are instead storing a reference to a DataTable. Thus when you create a new DataTable, and add that to session, you're simply overwriting the reference. You still have two DataTables somewhere in memory, until garbage collection cleans up any to which there there are no live references.
Remember that garbage collection in .net is non-deterministic. That is to say, setting an object to null does not immediately release the memory. It simply marks it, and at some point in the future, the garbage collector may see the dead object and release the memory associated with it.
You probably want to rethink your design if you're running out of memory. It might be better to have the second page refetch the data again, perhaps from a caching layer (possibly implemented on an application server as suggested by another poster) or perhaps from the database itself.
use a app-server layer to hold the data and each page should take it from there...
My first question would be why you need to store the entire database in the Session or the Application? Here is a good article that goes over all of your options and it advises against storing large amounts of data in the Session or Application caches. What issue are you trying to resolve by doing this?
Edit:
Are you display all the data at once on that page? ex. scroll down through 10000 records. If so that doesn't sound very user friendly (Assumption). Have you considered paging that data? You could have a page of 50 records and n number of pages. That would make the data call a lot faster and then you could implement filters, sorting, etc.
I want to show a large amount of data in a dataset, 100,000 records approx 10 columns, this consumes a large amount of ram 700MB. I have also tried using paging which reduces this by about 15-20% but I don't really like the Previous/Next buttons involved when using paging. I'm not writing the data to disk at present, should I be? If so what is the most common method? The data isn't to be stored forever just whilst it is being viewed, then a new query may be run and a nother 70,000 records could be viewed. What is the best way to proceed?
Thanks for the advice.
The reality is that the end-user rarely needs to see the totality of their dataset, so I would use which method you like for presenting the data (listview) and build a custom pager so that the dataset is only fed with the results of the number of records desired. Otherwise, each page load would result in re-calling the dataset.
The XML method to a temp file or utilizing a temp table created through a stored proc are alternatives but you still must sift and present the data.
An important question is where this data comes from. That will help determine what options are available to you. Writing to disk would work, but it probably isn't the best choice, for three reasons:
As a user, I'd be pretty annoyed if your app suddenly chewed up 700Mb of disk space with no warning at all. But, then, I'd notice such things. I suppose a lot of users wouldn't. Still: it's a lot of space.
Depending on the source of the data, even the initial transfer could take longer than your really want to allow.
Again, as a user, there's NO WAY I'm manually digging through 700Mb worth of data. That means you almost certainly never need to show it. You want to only load the requested page, one (or a couple) pages at a time.
I would suggest memory mapped files...not sure if .NET includes support for it yet.
That is a lot of data to be working with and keeping aroudn in memory.
Is this a ASP.NET app? Or a Windows app?
I personally have found that going with a custom pager setup (to control, next previous links) and paging at the database level to be the only possible way to get the best performance, only get the data needed....
implement paging in SQL if you want to reduce the memory footprint