There is an application that consists of three executable files. One of them - a dispatcher, which runs other executables. The dispatcher receives a code from an executable at its completion. That is, only the dispatcher is always running, other executables unloads and loads again. The application runs on the point of service and work around the clock. At the first launch the application works fast. At the end of the day, the application works terribly slow. What could be the reason for such a behavior?
There could be lot of reasons for a slow down over time. Anywhere from a slow memory leak to anti-virus. The best you can do is try to build evidence (data) about what area of the application to look first. Try not to talk it over with many devs because everyone will have a different opinion about what might be wrong. Get the data!
How to get the data:
perfmon
perfmon is your friend. There are a lot of counters that you can look at (system wide as well as process specific). So you can start by profiling the big 4 (that's memory, disk usage, cpu and networking). There are a lot of posts out there about what counters are best, so I won't go into too much detail about the perf counters here.
windbg
If you indeed see that memory is growing and not being collected it's time to bring in the big guns. .NET is great at abstracting memory usage away from developers, but this means we have to get underneath .NET sometimes to find out what is not allowing the Garbage Collector to do its work. windbg with the sos.dll (managed extensions) is a great tool for this. The hardest part of windbg (in my experience) is just getting the sos extensions loaded properly. You have to pay close attention to what target architecture (64 or 32) you are analysing and what CLR version you are running on.
procdump
procdump by sysinternals is a great little utility to take memory snapshots from a running process. These snapshots (.dmp files) can then be analyzed by windbg.
sos
The sos.dll has shipped with the .NET Framework since v2. With v4, Visual Studio 2010 has integrated sos and allows you to analyze .dmp files!
The sos commands for memory leaks that I have found most useful are:
!eeheap -gc (overview of what is in each generation of each heap)
!dumpheap -min <size> (dumps out all objects and types, over a particular <size>)
!dumpheap -type <type> (dump out all objects of a specific <type>)
!gcroot <address> (prints out a stack so you can see what parent object is pinning in the GC)
!do <address> (prints out memory of a specific object)
Some other pointers:
Usually, you want to snapshot memory under load, so it would be good to have some way to simulate that from outside the system. So, it is good to get this running ahead of time and even work it into the QA process for the application.
For performance problems it is usually best to take regular snapshots over time with a running application. Then you can compare the snapshots when you analyse.
Well, that was a bit longer than I intended, but hopefully worth it!
You have to check the memory usage of your dispatcher application... it seems you are not disposing of unused objects.
Related
I am developing a WinForm application. When running it for a long time wihout stopping, it suddenly raise a OutOfMemoryException. I don't know how to find the reason which causes this exception. I use Task Manager to see how much memory my application to use, I see that it only take 90000 KB while my RAM is 2GB. I don't know it is a memory leak error? If yes, I remember there are some .NET tools to help finding the memory leak.
Please help me. Thanks.
If you run your application through WinDbg you will be able to get a break down of all the objects in memory. A unusually large number will pin point the exact object that is causing this behavior.
You can follow these instructions
In a nut shell you do
Attach WinDbg to process
Issue .load SOS.dll command
Issue !dumpheap –stat to view managed objects
vmmap
A useful tool to see how much and what kind memory your process is using.
CLR Profiler
Provides a graphical view of the heap and what is holding on to what. It can be a bit slow, but if the leak is in the managed heap it will point out what types are the problem pretty easily. Here's a guide.
WinDBG
Like parapura mentioned, this is will let you find your leak, but there's a steep learning curve (and almost no gui).
I'm developing a C# application which seems to have a leak.
I've used memory profiler and found that my
private bytes keep increasing but Bytes in all Heaps do not, which means that probably it's a native memory leak
Now I'm stuck, how do I find memory leaks in native code ?
First, if you have a dump of the leaking process, you can open it in WinDbg and issue the command :
!address -summary
if RegionUsageHeap is large, then it should be a native memory leak
if RegionUsageIsVAD, then it should be a .NET memory leak.
If this is a native leak, then you have 2 options :
Use DebugDiag : when prompt, choose 'Native Memory leak and Handle leak', choose the process you want to diagnose, and start working with the application until you experiment the memory leak. When complete, generate a full dump of the application (right click on the leak rule and select Full user dump). You can then analyze the generated dump (you'll need to have the symbols properly configured for this to work efficiently) : on 'advanced analysis' tab, select 'Memory pressure analyzers', open the dump file and press 'Start analysis'. This produces and html report you can analyze. You can refer to this page for a detailed walkthrough.
Use Application Verifier / WinDbg. In application verifier, select your application (.exe). In tests page, be sure Basics/Heaps is selected. In the lower pane, be sure 'Traces' is set to true. Once the configuration saved, re-run the application and generate a full dump when the leak occurs. Don't forget to clean application flags after the dump is generated. Then you can open the dump from within WinDbg, and investigate the leak with the help of '!heap' command. In particular, '!heap -l' will give you a list of leaked blocks, '!heap -p -a ' will show the details of a block, including the call stack of allocation.
If this is a .NET leak, there are third party tools to troubleshoot it. Starting from version 1.2, DebugDiag is also enable to perform .NET memory leak analysis (never tried this however).
Diagnosing native memory leaks in a managed application is (at least initially) very similar to diagnosing memory leaks in any other native application.
The way I normally approach these problems is to get the process to leak a large amount of memory, take a full process dump and then examine the dump to see what is using the most memory. For example if your process has a normal / initial private bytes of ~20MB but you can get your process to leak memory until it has ~200MB of private bytes, then there is a good chance that ~180MB of that memory is leaked - generally speaking whatever has the most memory allocated is where you should start looking.
Microsoft have a very useful tool called DebugDiag - initially developed for use in diagnosing memory leaks in IIS it is a very vesatile tool and very handy when dealing with memory issues. If you give it a crash dump it will perform some analysis and should (at the very least) tell you what module has allocated all of that memory, you can then start looking more specifically at how that module is used.
It's hard to give you a solid response without more information, but it sounds like the lib you are trying to use has a memory leak. You'll need to the lib with the appropriate tools, depending on the language it was written in. If you don't have the lib's source, contact the developers and have them fix the leak.
If you can post the name of the library and some of your source code (as well as the native method signatures), we might be able to give you some more specific advice.
private bytes in heaps managed by .net framework, you need use professional tool to analysis your source. such like use red gate memory profiler, find object created but not being disposed.
Usually I had the best results when hunting memory leaks using the ANTS Memory Profiler.
(Or other tools, personally I had best experiences with ANTS)
This question already has answers here:
Closed 12 years ago.
Possible Duplicate:
Anatomy of a “Memory Leak”
Hi All
what are the best practices for identifying memory leak in c#?
How about
ANTS Memory Profiler™: Profile the memory usage of your .NET application
ANTS Performance Profiler for optimizing your .NET code
Memory Leak Detection in .NET
First, install the latest version of the Microsoft Debugging Tools. Its a prerequisite.
To analyze memory consumption, I recommend you to download & use .Net Memory Profiler. It really simplifies all the process.
MemProfiler allows to analyze managed memory utilization from managed applications, windows services, asp.net apps and from memory dumps.
To find a leak on a process on a DEV machine, run MemProfiler and simply attach to the desired process. Once you are profiling, take a snapshots and MemProfiler will tell you which and how many instances are loaded, so you could know what is allocating your memory. Take at least 2 memory snapshots on different moments and compare them to see the “delta instances”. ( If delta is >0 then N new instances were created. Otherwise, delta<0, they were destroyed)
When to collect the memory dumps?
You may take the first one after all the “required stuff” has been loaded, in example, after your app has responded successfully to a few requests.
You probably will make a load test to the app before taking the second snapshot, because if the app has leakead A LOT, then its easier to find out where.
You can also continue to load test the app and take a 3rd, 4th snapshots, just in case.
To collect a memory dump on web application on a production environment:
You are not required to install the Debugging tools on the server. You can install it on a developer pc and then copy the folder to the production environment. E.g c:\Program Files (x86)\Debugging Tools for Windows (x86)
To identify the Asp.Net w3wp use this command (works with W7 and Server 2008, didnt tried with 2003) : %windir%\system32\inetsrv\appcmd list wp
To generate the dump, execute: AdPlus -hang -p <process id> -o <outputDir>
Other Tips:
Generally your app will appear to leak strings, and in all the cases that I analyzed that was false. In my case, I used Asp.NET+WCF+IdentityModel, and they used a LOT of strings.
If you are interested, do the “.NET Memory Profiler Tutorials” at http://memprofiler.com/tutorials/ and follow Tess Ferrandez blog http://blogs.msdn.com/tess/
If you think that probably there is a memory leak somewhere, but you have no idea where could it be, start by watching memory usage per process with Task Manager or Process Explorer, looking for a process that eats always more and more RAM:
Use AdPlus to generate the dump, avoid Task Manager, because Task Manager is a 32 bit process probably dumping a 64 bit one. MemProfiler gets confused in that case.
When analyzing the objects in the memory, you will see an ExecutionEngineException, StackOverflowException and OutOfMemoryException on the heap. Don’t Panic. They doesn't mean that your application are raising that exceptions, they are always pre-allocated just in case:
Use a memory profiler - attach it to the suspected process and run the suspected actions.
You can find some recommended profilers in the answers to this question.
Visual Studio Profiling/Performance Tools allow you to track Object Allocation and Lifetime.
Unfortunately, that's only available in the more costly editions of the VS hierarchy.
We are getting a lot of out of memory exceptions and we can't seem to diagnose what is happening. It seems to be an issue that happens that will spike memory usage from 300 MB to over 1 GB in a matter of a few minutes. Now this is an IIS application and has three application domains running in separate thread pools.
We want to record when the memory exceptions are about to happen. So we can attempt to find a pattern. What is the best way to do this?
Is there a way to query the memory usage once a minute to see how high it is and send an alert email. Or maybe write an application that will monitor the CLR's memory usage? An ideas or a direction are more than welcome.
I am using Performance Monitor, but unless I am watching the process it is not that useful. I can only see what is was and when. I have also used the Red Gate Memory Profile tool, which is awesome by the way - only I cannot seem to hit upon the page or process that is causing the exception.
Install ADPlus on your server (it's part of the Windows Debugging Tools). When you start observing unusually high memory usage capture a memory dump using:
adplus -hang -p <PID> -quiet -o <dump file folder path>
<PID> is the process ID of the worker process which you can get from tasklist.exe.
If you're not always around when this issue occurs then you could automate capturing a memory dump using DebugDiag:
Use DebugDiag in leak tracking mode to trigger a process dump when either your Private or Virtual memory use reaches a certain threshold. That said, I've not always found this reliable.
Use DebugDiag in Crash Mode to capture a dump whenever CLR exceptions are thrown. You can used the Advanced Settings to configure DebugDiag to produce a full memory dump upon encountering a CLR exception of type System.OutOfMemoryException. This is more reliable and will definitely trigger. Only use Advanced Settings -> Exceptions, don't touch the Unconfigured First Chance Exceptions, leave this setting at None.
Once you have your memory dump, fire up WinDBG then load the dump file and load up SOS and start poking about.
Tess Ferrandez's blog is a great .NET debugging resource and she has plenty of articles and labs about how to track down memory leaks (these are archive.org links due to broken links):
If it is broken, fix it you should - memory issue articles
.NET Debugging Demos Lab 6:_Memory Leak
.NET Debugging Demos Lab 6:_Memory Leak - Review
.NET Debugging Demos Lab 7: Memory Leak
.NET Debugging Demos Lab 7: Memory Leak - Review
PerfMon counters are useful and can be used to confirm that you have a memory leak, but they don't tell the whole story. WinDBG and SOS are the tools you need to use to find out where your memory is being used.
Performance Monitor, aka Perfmon, is your friend - it is free and not intrusive and lightweight and can be safely run on production servers if you use less frequent sampling (every few seconds). The minimum it can do is to sample memory and CPU usage for your processes (w3wp.exe) and store them on a file.
Since you have not shared what you are doing in the application I can't suggest performance counters to store, but there are plenty in the ASP.NET and .NET and CLR.
Since you are getting a CLR out of memory exception, my hunch is GC is not working due to a pinned object or something. I doubt it is an unmanaged resource, such as bitmap, not being released although it could be.
Here are a list of counters I suggest:
.NET CLR Memory
GC Handles
Pinned Objects
Process for the w3wp.exe running your application
A couple, but mostly Working Set
ASP.NET
Managed Memory used
Try RedGate Memory Profiler. I suppose it works with ASP.NET sites (their performance profiler definitely does).
Use a memory profiler. There are a couple of good ones out there, e.g. JetBrains dotTrace or ANTS Memory Profiler from Red Gate. There were a couple of discussions here at Stack Overflow with lots of other tips and recommendations.
If you cannot afford a good profiler, you should use what Microsoft recommends in Production Debugging for .NET Framework Applications.
I would use ADPlus in "crash" mode to capture a memory dump when the exception occurs, and then WinDbg and SOS to figure out what's taking up all of the memory.
Use dotTrace or YourToolkit .NET where you can attach the profiler to the ASP process.
They have a trial version, so you don't need to spend money right away. Using these profilers you can select the timeline where memory starts increasing (you can see the memory usage visually in a graph), so it should be quite easy to select the range and understand what is causing the memory usage to shoot up so high.
We have a 64bit C#/.Net3.0 application that runs on a 64bit Windows server. From time to time the app can use large amount of memory which is available. In some instances the application stops allocating additional memory and slows down significantly (500+ times slower).When I check the memory from the task manager the amount of the memory used barely changes. The application keeps on running very slowly and never gives an out of memory exception.
Any ideas? Let me know if more data is needed.
You might try enabling server mode for the Garbage Collector. By default, all .NET apps run in Workstation Mode, where the GC tries to do its sweeps while keeping the application running. If you turn on server mode, it temporarily stops the application so that it can free up memory (much) faster, and it also uses different heaps for each processor/core.
Most server apps will see a performance improvement using the GC server mode, especially if they allocate a lot of memory. The downside is that your app will basically stall when it starts to run out of memory (until the GC is finished).
* To enable this mode, insert the following into your app.config or web.config:
<configuration>
<runtime>
<gcServer enabled="true"/>
</runtime>
</configuration>
The moment you are hitting the physical memory limit, the OS will start paging (that is, write memory to disk). This will indeed cause the kind of slowdown you are seeing.
Solutions?
Add more memory - this will only help until you hit the new memory limit
Rewrite your app to use less memory
Figure out if you have a memory leak and fix it
If memory is not the issue, perhaps your application is hitting CPU very hard? Do you see the CPU hitting close to 100%? If so, check for large collections that are being iterated over and over.
As with 32-bit Windows operating systems, there is a 2GB limit on the size of an object you can create while running a 64-bit managed application on a 64-bit Windows operating system.
Investigating Memory Issues (MSDN article)
There is an awful lot of good stuff mentioned in the other answers. However, I'm going to chip in my two pence (or cents - depending on where you're from!) anyway.
Assuming that this is indeed a 64-bit process as you have stated, here's a few avenues of investigation...
Which memory usage are you checking? Mem Usage or VMem Size? VMem size is the one that actually matters, since that applies to both paged and non-paged memory. If the two numbers are far out of whack, then the memory usage is indeed the cause of the slow-down.
What's the actual memory usage across the whole server when things start to slow down? Does the slow down also apply to other apps? If so, then you may have a kernel memory issue - which can be due to huge amounts of disk accessing and low-level resource usage (for example, create 20000 mutexes, or load a few thousand bitmaps via code that uses Win32 HBitmaps). You can get some indication of this on the Task Manager (although Windows 2003's version is more informative directly on this than 2008's).
When you say that the app gets significantly slower, how do you know? Are you using vast dictionaries or lists? Could it not just be that the internal data structures are getting so big so as to complicate the work any internal algorithms are performing? When you get to huge numbers some algorithms can start to become slower by orders of magnitude.
What's the CPU load of the application when it's running at full-pelt? Is actually the same as when the slow-down occurs? If the CPU usage decreases as the memory usage goes up, then that means that whatever it's doing is taking the OS longer to fulfill, meaning that it's probably putting too much load on the OS. If there's no difference in CPU load, then my guess is it's internal data structures getting so big as to slow down your algos.
I would certainly be looking at running a Perfmon on the application - starting off with some .Net and native memory counters, Cache hits and misses, and Disk Queue length. Run it over the course of the application from startup to when it starts to run like an asthmatic tortoise, and you might just get a clue from that as well.
Having skimmed through the other answers, I'd say there's a lot of good ideas. Here's one I didn't see:
Get a memory profiler, such as SciTech's MemProfiler. It will tell you what's being allocated, by what, and it will show you the whole slice n dice.
It also has video tutorials in case you don't know how to use it. In my case, I discovered I had IDisposable instances that I wasn't Using(...)