I'm trying to reduce the memory usage of a console application I have. It is supposed to run for hours on end, but it seems like the memory usage is gradually increasing with each second. It does use multiple threads, and does a variety of things, but I read somewhere that having a lot of calls to Console.WriteLine can also cause memory spikes.
Because the application is constantly writing to the console, I thought that maybe the memory usage is because of this. Unfortunately, I can't easily clear the console because I'm redirecting the output to a monitoring window. I've turned it off temporarily, but the memory is still increasing, which tells me there's other things I need to address.
Before I go about hunting down memory leaks, I was wondering if anyone can confirm/verify whether having thousands of Console.WriteLine's could cause a memory leak, or if that's already handled appropriately by the redirected output buffer. I've tried to do a search, but haven't found much on this.
Having thousands of calls to Console.WriteLine does not cause a memory leak. I have a long-running program (it's been up for 6 months now) that writes a few hundred lines per minute to the console, and its memory usage has remained level.
It's possible that writing thousands of lines infrequently all at once can cause memory spikes due to temporary strings, but those will be collected the next time GC is run. But a steady load of Console.WriteLine will just cause a steady memory load of uncollected strings. It won't be ever-increasing.
Related
I have a fairly simple application that takes a data file with about 500k lines of data in it, parses the data, organizes it and then inserts it into an Azure Table. There are about 2000 of these files and I need the process to work smoothly as I load all of the data.
I am using WindowsAzure.Storage v5.0.2 for inserting the data and Microsoft.tpl.dataflow v4.4.24 for parallelization. Each file is completely processed and all tasks finalized before I move onto the next file. I am also disposing of all objects I can and setting everything else to Null at the end of each file load.
Despite trying to be as careful as possible the RAM usage goes up steadily until it crashes the process. When it starts it jumps up to 1 GB of RAM used and steadily climbs until the process crashes somewhere above 9GB of RAM consumed.
Note - this is targeting x64 on a reasonably large computer. Garbage collection is happening on a regular basis, but it doesn't seem to affect the memory problem.
At this point I am completely confused about where the memory leak is coming from and don't have any idea about how to diagnose the problem.
Update
After a lot of work and following the suggestion below I found out that the parallelization I was using was allowing more simultaneous insert processes than I expected. It looked like the insert was complete and my code was starting the next insert. In reality the parallel process had just reported back a status but had not finished. This led to a huge backlog of simultaneous inserts happening, chewing up RAM and crashing the system. It also cause me to go past my IOPS limit which I believe triggered throttling, compounding the problem.
Figuring this out required a huge amount of work and many different ways of analysing everything, but the suggestion below got me going in the right direction.
Doing a search for 'troubleshoot .net memory leak' will yield lots of results, but probably the best one is http://blogs.msdn.com/b/tess/archive/2009/05/12/debug-diag-script-for-troubleshooting-net-2-0-memory-leaks.aspx.
Basically, use DebugDiag to generate a memory leak analysis report and then look for which objects are consuming all of the memory. You will likely see one type of object that you didn't realize you were continuously adding to a collection without removing it later on.
I have been reading about out of memory for some time now and I figured out that in most cases out of memory exception (at least in .NET) isn't really caused by system actually running out of memory but rather system could not allocate chunks of requested memory block due to fragmentation.
What I don't really understand is I've been in a situation where I still get out of memory exception even if I try to allocate a large chunk of contiguous memory on application startup (eg: loading 100 images). Since the application has just started up, it is assumed that not much allocations / de-allocations have been done prior to that, so there should be many free contiguous blocks available. In that case why would the application still get hit by memory fragmentation issue?
Note that I'm also fairly certain that the issue was not caused by the system actually running out of memory quota allocated for my application because loading 100 images in my specific case only takes ~200 mb or so.
In my experience, Out of Memory mostly means poor object management. It's symptomatic of creating too many objects too fast and GC is having a hard time keeping up. Setting aside the few projects that take and never give memory back (like a SQL Server) out of memory can be prevented with caching and a well defined object life cycle.
I am using ANTS memory profiler and am somewhat baffled at the moment. If I load my site up and hook ANTS up to the process I can see the Private Bytes around 90mb, I then run the same routine a number of time with the following results:
109mb
120mb
125mb
126mb
123mb
126mb
and it basically stays around 126mb for each try after. My understanding is that if I had a memory leak then it would keep going up and not settle but what I don't understand is why it grows slowly until 126mb. Does .NET have an amount of memory it is allowed to reserve and it is just hitting that limit?
Simple question: Memory is growing up to a point then stopping. Is this normal for a .NET app?
EDIT: Just realised that I probably should have posted this at programmers.stackexchange.com - Apologies.
"Memory leak" is when memory you think should not be allocated is allocated.
It is not possible to simply look at amount of consumed memory and say "you have memory leak". I.e. what if your application collects logs in memory for 3 days - in this case memory consumption will grow, but it is not an indication of a leak. On other hand if your application simply prints a line a minute, but memory usage constantly grows it likley is a leak.
In my experience, if I see memory grow and plateau, it's generally from caching. As stated before, a memory leak is simply when something remains in memory that you feel should've been released. Using a profiler is a great way to determine leaks (vs using the task manager) because it generally will ensure a 2nd generation garbage collection has occurred, allowing you to see everything that's still in memory.
When I profile, I generally will execute the commands that I want to test a couple times to ensure all caching has occurred, and then I'll create a before and after snapshot and compare the delta of memory. If you're using a managed language like C#, it's not uncommon to have a delta of +/- 10KB. Repeat that process several times and if your delta is consistently positive, you more than likely have a leak (assuming you're not intentionally allocating more memory).
Have simple C# console app which imports text data into SQL.
It takes around 300K in memory and 80% in CPU. There are 2Gb RAM available at any time and yet the Page Fault shows 500K.
The app is 32 bit and OS is either W2000 or XP 32 bit and .NET 3.5
Anyone can explain what could be the problem and how can I investigate this further?
EDIT: I am now certain that the page faults are related to the disk I/O (read). I commented out SQL part and the pure disk read generates that high number alone.
EDIT2: There are 200 hard faults/sec and 4000 soft faults/sec on average.
I wonder if the same would appear on W2008
First, how do you measure the memory the app is using? If you're looking at "working set" that's only the part that resides in physical memory. You should also take a look at the "VM Size" (or "Commit Size") where the actual virtual memory your process takes up.
If Windows kernel Balance Set Manager thinks that your app is inactive, or should be left behind to give other processes more power, it can decide to reduce the working set size. If working set size is smaller than what your application actually needs to work on, you could easily see a lot of page faults because it simply becomes a race between The Balance Set Manager and the application. Usually balance set manager monitors memory usage and can also decide to increase working set size accordingly. However, this might be prevented in certain circumstances like low physical free memory, high I/O (cache stress on physical memory), low process priorty, background/foreground status of the application etc.
It can simply be the behavior of .NET garbage collector due to vast amount of small memory blocks getting allocated and disposed in a very short time, causing a stress on both memory allocation and releasing. The "VM Size" could stay around the same size but behind the scenes it could be continously allocating/freeing memory, causing continous page faults.
Also know that the DLLs the process is using are also accounted for the process statistics. Not your app but one of the COM or .NET DLL you are using might be causing this behavior as well. You can deduce actual culprit by changing your application's behavior (e.g. removing DB access code and only leave object allocation code behind) to see which component is actually causing thrashing.
EDIT: About your question on GC impact on memory thrashing: The CLR actually grows the heap dynamically and gives the memory back to the OS as needed. That does not occur synchronously. GC runs behind the scenes and frees memory in large chunks to prevent hindering application performance. Say you are allocating many small objects and freeing them almost immediately. That causes many references to stay for a moment in memory before freeing. It is easy to imagine that it becomes like a head-to-head race between the garbage collector and the memory allocating code. While GC eventually catches up, the required new memory must be satisified from a "new memory", not the old one because old one is not freed up yet. Since actual memory we are working on stays around the same, balance set manager may not think of giving our process more memory because we're on the edge, always around the same physical memory size but constantly need "newly allocated memory" not "more memory", therefore page faults.
Page faults are normal. Memory gets swapped out and when you next access it that's a page fault and the system brings it back. This is by design.
I've got an app running on my machine right now with 500 million page faults. There's nothing to worry about!
Page faults means Memory issues
Consider increasing memory, if you have excessive page faults.
Have a large working set size.
The working set is the set of memory pages currently loaded in RAM. This is measured by Process\Working Set. A high value might indicate that you have loaded a number of assemblies.
Process\Working Set has no specific threshold value to watch, although a high or fluctuating value can indicate a memory shortage. A high or fluctuating value accompanied by a high rate of page faults clearly indicates that your server does not have enough memory.
Further reading:
Check memory under System Resources in following MSDN article:
http://msdn.microsoft.com/en-us/library/ff647791.aspx#scalenetchapt15_topic9
Please provide some code to investigate.
A possible answer to this I am currently testing it on my application. Break up your working set into smaller chunks and work with the chunks.
For instance I have a large list of objects (9000-30000). If I break up that list into a chunk of 500 or so at a time it should maintain the 500 objects in memory while I work on them.
You will want to increase or decrease the size of your chunk until you can work with it fast enough that the OS will maintain it in memory. This is theory I haven't fully tested it yet. But it should work.
My application's performance deteriorate as it continues to run through the day.
I suspect it is garbage collector, how can I verify this?
Is there a way to find out which object/function is causing garbage collection overhead?
Is there a way to manually perform garbage collection programatically to clear memory of leakages?
Thanks,
edit
On one end of the application it receives call back from a unmanaged api to accept data, processes it and then send messages out of socket on the second end. From the second end it then gets back follow up data on the messages it sent out.
The application opens 5-6 sockets to send and receive data from the second end.
It constantly logs lots of data to windows file system on a separate thread.
My measurements include timestamping (queryperformance counter) just before I send data out and the timestampinga again when I receive the followup from another process back on the socket.
I noticed out of multiple sockets I open, the performance deterioration is happening on one socket connection only.
The processing between the timestamping and sending.receiving data over socket includes iterating through 2 arraylist that has no more than 5-6 objects and couple of callbacks.
The memory usage from Task MAnager window does not go up considerably. From 96MB to 100MB after 6-7 hours run.
Following are some observations from running perfmon.
"finanlization survivors" and "promoted finalization memory from Gen 0" gradually increase with time
"Gen 0 collections" going from 1819 at the start to 6000 after 4 hours.
"Gen 1 Collections" is 10%-12% of Gen 0 collection and "Gen 2 collection" is 1% or less. COnsidering Gen 0 collection numbers are cumulative, it is probably not abig concern.
GC handles" went up from 850ish to 4000.
It's far more likely that you have a memory leak, and invoking the GC manually will not help that: it can't dispose of objects if your code hasn't released them.
Edit
Since your GC handles are increasing, this page suggests that there are non-managed resources that are not getting freed. I've had this happen with bitmaps, for example, but you might have to tell us a lot more about your application to get a more specific suggestion.
Here's a thread that may give you some useful insight.
You can call GC.Collect() to force the garbage collection, but this will not fix memory leaks. Try a memory profiler like ANTS memory profiler to find memory leaks.
ANTS memory profiler
It may not necessarily be a memory problem. Use the Windows Performance Monitor (look in administrative tools) to monitor your app's CPU, Memory, GDI object count, handle count, and see if any of these appear to be climbing throughout the life of your app.
Often, you'll find that you're using something from System.Drawing and not calling Dispose() on it, causing a handle leak. I've found that handle leaks tend to eat performance much faster than memory leaks. And handle leaks cause no GC pressure, meaning you could be leaking handles like a sieve and the GC wouldn't ever know the difference.
So, long story short: measure, measure, measure. Then you'll know what to fix.