Have simple C# console app which imports text data into SQL.
It takes around 300K in memory and 80% in CPU. There are 2Gb RAM available at any time and yet the Page Fault shows 500K.
The app is 32 bit and OS is either W2000 or XP 32 bit and .NET 3.5
Anyone can explain what could be the problem and how can I investigate this further?
EDIT: I am now certain that the page faults are related to the disk I/O (read). I commented out SQL part and the pure disk read generates that high number alone.
EDIT2: There are 200 hard faults/sec and 4000 soft faults/sec on average.
I wonder if the same would appear on W2008
First, how do you measure the memory the app is using? If you're looking at "working set" that's only the part that resides in physical memory. You should also take a look at the "VM Size" (or "Commit Size") where the actual virtual memory your process takes up.
If Windows kernel Balance Set Manager thinks that your app is inactive, or should be left behind to give other processes more power, it can decide to reduce the working set size. If working set size is smaller than what your application actually needs to work on, you could easily see a lot of page faults because it simply becomes a race between The Balance Set Manager and the application. Usually balance set manager monitors memory usage and can also decide to increase working set size accordingly. However, this might be prevented in certain circumstances like low physical free memory, high I/O (cache stress on physical memory), low process priorty, background/foreground status of the application etc.
It can simply be the behavior of .NET garbage collector due to vast amount of small memory blocks getting allocated and disposed in a very short time, causing a stress on both memory allocation and releasing. The "VM Size" could stay around the same size but behind the scenes it could be continously allocating/freeing memory, causing continous page faults.
Also know that the DLLs the process is using are also accounted for the process statistics. Not your app but one of the COM or .NET DLL you are using might be causing this behavior as well. You can deduce actual culprit by changing your application's behavior (e.g. removing DB access code and only leave object allocation code behind) to see which component is actually causing thrashing.
EDIT: About your question on GC impact on memory thrashing: The CLR actually grows the heap dynamically and gives the memory back to the OS as needed. That does not occur synchronously. GC runs behind the scenes and frees memory in large chunks to prevent hindering application performance. Say you are allocating many small objects and freeing them almost immediately. That causes many references to stay for a moment in memory before freeing. It is easy to imagine that it becomes like a head-to-head race between the garbage collector and the memory allocating code. While GC eventually catches up, the required new memory must be satisified from a "new memory", not the old one because old one is not freed up yet. Since actual memory we are working on stays around the same, balance set manager may not think of giving our process more memory because we're on the edge, always around the same physical memory size but constantly need "newly allocated memory" not "more memory", therefore page faults.
Page faults are normal. Memory gets swapped out and when you next access it that's a page fault and the system brings it back. This is by design.
I've got an app running on my machine right now with 500 million page faults. There's nothing to worry about!
Page faults means Memory issues
Consider increasing memory, if you have excessive page faults.
Have a large working set size.
The working set is the set of memory pages currently loaded in RAM. This is measured by Process\Working Set. A high value might indicate that you have loaded a number of assemblies.
Process\Working Set has no specific threshold value to watch, although a high or fluctuating value can indicate a memory shortage. A high or fluctuating value accompanied by a high rate of page faults clearly indicates that your server does not have enough memory.
Further reading:
Check memory under System Resources in following MSDN article:
http://msdn.microsoft.com/en-us/library/ff647791.aspx#scalenetchapt15_topic9
Please provide some code to investigate.
A possible answer to this I am currently testing it on my application. Break up your working set into smaller chunks and work with the chunks.
For instance I have a large list of objects (9000-30000). If I break up that list into a chunk of 500 or so at a time it should maintain the 500 objects in memory while I work on them.
You will want to increase or decrease the size of your chunk until you can work with it fast enough that the OS will maintain it in memory. This is theory I haven't fully tested it yet. But it should work.
Related
I have been reading about out of memory for some time now and I figured out that in most cases out of memory exception (at least in .NET) isn't really caused by system actually running out of memory but rather system could not allocate chunks of requested memory block due to fragmentation.
What I don't really understand is I've been in a situation where I still get out of memory exception even if I try to allocate a large chunk of contiguous memory on application startup (eg: loading 100 images). Since the application has just started up, it is assumed that not much allocations / de-allocations have been done prior to that, so there should be many free contiguous blocks available. In that case why would the application still get hit by memory fragmentation issue?
Note that I'm also fairly certain that the issue was not caused by the system actually running out of memory quota allocated for my application because loading 100 images in my specific case only takes ~200 mb or so.
In my experience, Out of Memory mostly means poor object management. It's symptomatic of creating too many objects too fast and GC is having a hard time keeping up. Setting aside the few projects that take and never give memory back (like a SQL Server) out of memory can be prevented with caching and a well defined object life cycle.
We have various tests that monitor our application's (or sub components) memory usage under various scenarios, to check for regressions. The problem is that our measurements (using Process.Refresh(); Process.PrivateMemorySize64) fluctuate wildly over different runs.
What we are currently doing is polling via a background thread every X milliseconds. We compare the max value reached to a benchmark, and pass/fail accordingly.
Interestingly, reducing the time between polls significantly reduces the maximum memory value recorded. In one example, reading the memory every 100ms gives a maximum of 360MB, every 10ms gives 147MB, and every 1ms gives 35ms.
Presumably the increased number of observations somehow makes GC behaviour more aggressive.
I suppose that the main problem is that what we are trying to measure, "maximum memory usage", is not particularly well defined.
If there's plenty of memory available, the GC might not bother to kick in, so memory usage will appear higher.
Is there a standard way to measure memory usage under dot net to guard against performance degradation?
I suppose that the main problem is that what we are trying to measure, "maximum memory usage", is not particularly well defined. If there's plenty of memory available, the GC might not bother to kick in, so memory usage will appear higher.
This is your problem. It's not arbitrary that a garbage-collected runtime would use more memory when it's available. These algorithms perform substantially better when they have a larger memory space to "play around in". For an objective analysis, you must use a tool to analyze only the reachable set, which is not affected by garbage collected behavior.
You can use PerformanceCounter class to measure your memomry usage
link : http://msdn.microsoft.com/en-us/library/system.diagnostics.performancecounter.aspx
manualy you can use Memory Profiler tool to inspect your memory
link : http://memprofiler.com/
I work with a flow control system written in. NET, the system inter-acts with external systems through TCP connections and routes transactions between different endpoints.
My problem:
At startup / initialization the private working set memory level is about 25000KB. After initialization when the system is in idle state, the private working set is stepping up with about 50-100KB per second until it reaches a limit of about 57000KB.
Information:
The system is generating page faults during the incrementation.
When the limit is reached, the private working set stays very stable and oscillates up and down with a few MB when I connect +300 clients and exchange high-frequency transactions for a couple of hours, the logic for garbage collection works very well.
I have profiled this system with a tool from Redgate called "Memory Profiler" which tells me the memory stepping up after initialization is allocated by unmanaged code, unfortunately this profiler does not support insight to memory allocated by unmanaged code so I have difficulties to find out what this allocated memory contains, why it is allocated and which code that allocates the memory.
The whole codebase is developed in C#, there are no references to COM+ assemblies and there is no communication with native windows API's (during the incrementation of this memory).
My question:
I need to be pointed in the right direction to find out why the memory is continuously incrementing in small chunks to a specific level after initialization.
If a page is in not working set this does not mean the page is stored only on disk or on disk at all. Pages on Windows can go to the standby list. If they do, they leave the WS and require a soft fault to bring them back. (I never understood why this mechanism is there, but it is). A soft fault is cheap.
Using Process Explorer's system information window you can see the number of hard and soft faults per seconds. Probably also available using perfmon. I suggest you check if you have hard faults (which I believe you don't so you don't have a problem and you can close the investigation).
Also, WS has nothing to do with memory usage, but I think you already knew that.
I am using ANTS memory profiler and am somewhat baffled at the moment. If I load my site up and hook ANTS up to the process I can see the Private Bytes around 90mb, I then run the same routine a number of time with the following results:
109mb
120mb
125mb
126mb
123mb
126mb
and it basically stays around 126mb for each try after. My understanding is that if I had a memory leak then it would keep going up and not settle but what I don't understand is why it grows slowly until 126mb. Does .NET have an amount of memory it is allowed to reserve and it is just hitting that limit?
Simple question: Memory is growing up to a point then stopping. Is this normal for a .NET app?
EDIT: Just realised that I probably should have posted this at programmers.stackexchange.com - Apologies.
"Memory leak" is when memory you think should not be allocated is allocated.
It is not possible to simply look at amount of consumed memory and say "you have memory leak". I.e. what if your application collects logs in memory for 3 days - in this case memory consumption will grow, but it is not an indication of a leak. On other hand if your application simply prints a line a minute, but memory usage constantly grows it likley is a leak.
In my experience, if I see memory grow and plateau, it's generally from caching. As stated before, a memory leak is simply when something remains in memory that you feel should've been released. Using a profiler is a great way to determine leaks (vs using the task manager) because it generally will ensure a 2nd generation garbage collection has occurred, allowing you to see everything that's still in memory.
When I profile, I generally will execute the commands that I want to test a couple times to ensure all caching has occurred, and then I'll create a before and after snapshot and compare the delta of memory. If you're using a managed language like C#, it's not uncommon to have a delta of +/- 10KB. Repeat that process several times and if your delta is consistently positive, you more than likely have a leak (assuming you're not intentionally allocating more memory).
We have a 64bit C#/.Net3.0 application that runs on a 64bit Windows server. From time to time the app can use large amount of memory which is available. In some instances the application stops allocating additional memory and slows down significantly (500+ times slower).When I check the memory from the task manager the amount of the memory used barely changes. The application keeps on running very slowly and never gives an out of memory exception.
Any ideas? Let me know if more data is needed.
You might try enabling server mode for the Garbage Collector. By default, all .NET apps run in Workstation Mode, where the GC tries to do its sweeps while keeping the application running. If you turn on server mode, it temporarily stops the application so that it can free up memory (much) faster, and it also uses different heaps for each processor/core.
Most server apps will see a performance improvement using the GC server mode, especially if they allocate a lot of memory. The downside is that your app will basically stall when it starts to run out of memory (until the GC is finished).
* To enable this mode, insert the following into your app.config or web.config:
<configuration>
<runtime>
<gcServer enabled="true"/>
</runtime>
</configuration>
The moment you are hitting the physical memory limit, the OS will start paging (that is, write memory to disk). This will indeed cause the kind of slowdown you are seeing.
Solutions?
Add more memory - this will only help until you hit the new memory limit
Rewrite your app to use less memory
Figure out if you have a memory leak and fix it
If memory is not the issue, perhaps your application is hitting CPU very hard? Do you see the CPU hitting close to 100%? If so, check for large collections that are being iterated over and over.
As with 32-bit Windows operating systems, there is a 2GB limit on the size of an object you can create while running a 64-bit managed application on a 64-bit Windows operating system.
Investigating Memory Issues (MSDN article)
There is an awful lot of good stuff mentioned in the other answers. However, I'm going to chip in my two pence (or cents - depending on where you're from!) anyway.
Assuming that this is indeed a 64-bit process as you have stated, here's a few avenues of investigation...
Which memory usage are you checking? Mem Usage or VMem Size? VMem size is the one that actually matters, since that applies to both paged and non-paged memory. If the two numbers are far out of whack, then the memory usage is indeed the cause of the slow-down.
What's the actual memory usage across the whole server when things start to slow down? Does the slow down also apply to other apps? If so, then you may have a kernel memory issue - which can be due to huge amounts of disk accessing and low-level resource usage (for example, create 20000 mutexes, or load a few thousand bitmaps via code that uses Win32 HBitmaps). You can get some indication of this on the Task Manager (although Windows 2003's version is more informative directly on this than 2008's).
When you say that the app gets significantly slower, how do you know? Are you using vast dictionaries or lists? Could it not just be that the internal data structures are getting so big so as to complicate the work any internal algorithms are performing? When you get to huge numbers some algorithms can start to become slower by orders of magnitude.
What's the CPU load of the application when it's running at full-pelt? Is actually the same as when the slow-down occurs? If the CPU usage decreases as the memory usage goes up, then that means that whatever it's doing is taking the OS longer to fulfill, meaning that it's probably putting too much load on the OS. If there's no difference in CPU load, then my guess is it's internal data structures getting so big as to slow down your algos.
I would certainly be looking at running a Perfmon on the application - starting off with some .Net and native memory counters, Cache hits and misses, and Disk Queue length. Run it over the course of the application from startup to when it starts to run like an asthmatic tortoise, and you might just get a clue from that as well.
Having skimmed through the other answers, I'd say there's a lot of good ideas. Here's one I didn't see:
Get a memory profiler, such as SciTech's MemProfiler. It will tell you what's being allocated, by what, and it will show you the whole slice n dice.
It also has video tutorials in case you don't know how to use it. In my case, I discovered I had IDisposable instances that I wasn't Using(...)