Develop Reference

How to prove that a version of a program is more memory efficient?

I have an application which has to process hundred of thousands of records. Right now, I can only process 500 of them at a time. Each batch can take up to 5 minutes to process/analyze (total of ~10 hours of processing). The reason for that limit of 500 records is memory consumption. I think that one of the main reason why our program takes so much memory is that fact that we don't set the size of lists or dictionaries (e.g. new List() instead of new List(100000)). I did the changes to set the size of collection so that .NET stop creating and copying new lists with extra capacity.
Here's my question: how to prove that a version of a program is more memory efficient? Are there performance counters I should look at? Tools? Monitoring?

There are tools like .net memory profiler from where you can have in-depth analysis of memory management and memory leaks of a .net application.
.NET Memory Profiler is a powerful tool for finding memory leaks and optimizing the memory usage in programs written in C#, VB.NET or any other .NET Language. With the help of the profiling guides, the automatic memory analyzer, and specialized trackers, you can make sure that your program has no memory or resource leaks, and that the memory usage is as optimal as possible.

I don't think a profiler would give you a real overview of how much more efficient one version is than the other.
I recommend using memory performance counters to do this. You can setup a few data collection sessions using perfmon, for both of your app's versions.
You should monitor at first only the process memory (of your process, of course). You can also add some of the rest of system memory performance counters, just to see how the overall virtual memory status changes during your process' lifetime.
If there is a difference and you still can't get a good overview and conclusion, you could start digging in the .NET CLR memory performance counters. Not as easy to analyze than the previous ones I mentioned, but much more detailed.
You should probably run each session for one hour or two in order to get some good data, given the lengthy processing done by your app.

You can use the in-built Performance Profiler tool if you have Visual Studio Ultimate: Debug -> Start Performance Analysis. If you don't have this available, you can use dotTrace by JetBrains, which accomplishes the same thing.
You can also measure execution time by using the Stopwatch class. Stopwatch is specifically designed to measure elapsed time and may (if available on your hardware) provide good granularity/accuracy using an underlying high-frequency hardware timer, compared to DateTime.Now. By using the Stopwatch class and comparing the two execution times you can see which runs faster. This does not give any information with CPU or Memory Usage.

Using lots of memory does normally not really slow you down until you go out of real memory and start using swapfiles. You can check that in Task manager while your app is running. Where you can also see the peak and working amount of memory of your app.
I would profile a bit (there are several memory profilers available) to see what objects are taking up your memory.
As for the lists; I do not think setting the capacity of Lists will fix your problem. The List class grows by doubling the capacity (tested as I could not find documentation). Worst case you are using twice the memory. If you create new lists inabundance you could create an object cache to re-use the lists. But in my experience .NET this does not help much in .NET as the runtime is quite efficient.

You can use CLR GC ETL events. Download perfview from Microsoft, run your application under it, and then just check the GC page.
Just check "total memory allocation" and "time spent in GC" would give you an idea of how managed memory is used.
If you want more details, check the CLR allocation tick events.
Perfview can also analyse your managed heap live objects.

Most Accurate Way to Get Memory Usage for Silverlight Browser Application

Currently i'm tracking down a memory leak issue in a Silverlight 4 application. I can watch the memory used by iexplorer.exe increase gradually using Task Manager.
However, I am looking for a way within the Silverlight code to tell how much memory the application is currently using within the IE Process. I have tried GC.GetTotalMemory but it does not represent the actual memory usage by iexplorer.
I have developer tools for tracking down the memory leak itself, so i'm not looking for that but a method within my Silverlight code to get total IE memory usage, or at least a closer representation of what is actually being used by Silverlight instead of what the Garbage collector thinks is being used.

GC allocated memory is good estimation of dynamic portion of memory used by SL application. Otherwise it is unclear what other types of memory usage you need.
If you need some other data consider using VaDump first ( http://msdn.microsoft.com/en-us/magazine/dd882521.aspx linked from Using VADump to track memory usage - OpenProcess Failed c0000034). When you figure out what portion/type of memory you are interested in it should be relatively easy to get this information (you may need to collect it with native calls, but likely all can be PInvoke'ed).
Update: I think all information you looking for is avaialble through performance counters - usage sample - http://msdn.microsoft.com/en-us/library/system.diagnostics.performancecounter.countername.aspx (but I'm don't think you can get it from SL).

Try the visual studio profiler
https://blogs.msdn.com/b/profiler/archive/2010/04/26/vs2010-silverlight-4-profiling.aspx
"For .NET memory profiling, use “VSPerfClrEnv /samplegc” or “VSPerfClrEnv /samplegclife” in your step 1 to get either allocation profiling or allocation and lifetime profiling. "

ANTS Profiler...
http://www.red-gate.com/products/dotnet-development/ants-memory-profiler/

CLR out of memory exceptions

We are getting a lot of out of memory exceptions and we can't seem to diagnose what is happening. It seems to be an issue that happens that will spike memory usage from 300 MB to over 1 GB in a matter of a few minutes. Now this is an IIS application and has three application domains running in separate thread pools.
We want to record when the memory exceptions are about to happen. So we can attempt to find a pattern. What is the best way to do this?
Is there a way to query the memory usage once a minute to see how high it is and send an alert email. Or maybe write an application that will monitor the CLR's memory usage? An ideas or a direction are more than welcome.
I am using Performance Monitor, but unless I am watching the process it is not that useful. I can only see what is was and when. I have also used the Red Gate Memory Profile tool, which is awesome by the way - only I cannot seem to hit upon the page or process that is causing the exception.

Install ADPlus on your server (it's part of the Windows Debugging Tools). When you start observing unusually high memory usage capture a memory dump using:
adplus -hang -p <PID> -quiet -o <dump file folder path>
<PID> is the process ID of the worker process which you can get from tasklist.exe.
If you're not always around when this issue occurs then you could automate capturing a memory dump using DebugDiag:
Use DebugDiag in leak tracking mode to trigger a process dump when either your Private or Virtual memory use reaches a certain threshold. That said, I've not always found this reliable.
Use DebugDiag in Crash Mode to capture a dump whenever CLR exceptions are thrown. You can used the Advanced Settings to configure DebugDiag to produce a full memory dump upon encountering a CLR exception of type System.OutOfMemoryException. This is more reliable and will definitely trigger. Only use Advanced Settings -> Exceptions, don't touch the Unconfigured First Chance Exceptions, leave this setting at None.
Once you have your memory dump, fire up WinDBG then load the dump file and load up SOS and start poking about.
Tess Ferrandez's blog is a great .NET debugging resource and she has plenty of articles and labs about how to track down memory leaks (these are archive.org links due to broken links):
If it is broken, fix it you should - memory issue articles
.NET Debugging Demos Lab 6:_Memory Leak
.NET Debugging Demos Lab 6:_Memory Leak - Review
.NET Debugging Demos Lab 7: Memory Leak
.NET Debugging Demos Lab 7: Memory Leak - Review
PerfMon counters are useful and can be used to confirm that you have a memory leak, but they don't tell the whole story. WinDBG and SOS are the tools you need to use to find out where your memory is being used.

Performance Monitor, aka Perfmon, is your friend - it is free and not intrusive and lightweight and can be safely run on production servers if you use less frequent sampling (every few seconds). The minimum it can do is to sample memory and CPU usage for your processes (w3wp.exe) and store them on a file.
Since you have not shared what you are doing in the application I can't suggest performance counters to store, but there are plenty in the ASP.NET and .NET and CLR.
Since you are getting a CLR out of memory exception, my hunch is GC is not working due to a pinned object or something. I doubt it is an unmanaged resource, such as bitmap, not being released although it could be.
Here are a list of counters I suggest:
.NET CLR Memory
GC Handles
Pinned Objects
Process for the w3wp.exe running your application
A couple, but mostly Working Set
ASP.NET
Managed Memory used

Try RedGate Memory Profiler. I suppose it works with ASP.NET sites (their performance profiler definitely does).

Use a memory profiler. There are a couple of good ones out there, e.g. JetBrains dotTrace or ANTS Memory Profiler from Red Gate. There were a couple of discussions here at Stack Overflow with lots of other tips and recommendations.

If you cannot afford a good profiler, you should use what Microsoft recommends in Production Debugging for .NET Framework Applications.

I would use ADPlus in "crash" mode to capture a memory dump when the exception occurs, and then WinDbg and SOS to figure out what's taking up all of the memory.

Use dotTrace or YourToolkit .NET where you can attach the profiler to the ASP process.
They have a trial version, so you don't need to spend money right away. Using these profilers you can select the timeline where memory starts increasing (you can see the memory usage visually in a graph), so it should be quite easy to select the range and understand what is causing the memory usage to shoot up so high.

.NET out of memory troubleshooting

After reading a few enlightening articles about memory in the .NET technology, Out of Memory does not refer to physical memory, 597499.
I thought I understood why a C# app would throw an out of memory exception -- until I started experimenting with two servers-- both are having 2.5 gigs of ram, windows server 2003 and identical programs running.
The only significant difference between the two being one has 7% hard drive storage left and the other more than 50%.
The server with 7% storage space left is consistently throwing an out of memory while the other is performing consistently well.
My app is a C# web application that process' hundreds of MBs of String object.
Why would this difference happen seeing that the most likely reason for the out of memory issue is out of contiguous virtual address space.

All I can think of is that you're exhausting the virtual memory. Sounds like you need to run a memory profiler on the app.
I've used the Red Gate profiler in similar situations in the past. You may be surprised how much memory your strings are actually using.

Is the paging file fragmentation different on each machine? High fragmentation could slow down paging operations and thus exacerbate memory issues. If the paging file is massively fragmented, sort it out e.g. bring the server off-line, set the paging file size to zero, defrag the drive, re-create the paging file.
It's hard to give any specific advice on how to deal with perf problems with your string handling without more detail of what you are doing.

Why would this difference happen
seeing that the most likely reason for
the out of memory issue is out of
contiguous virtual address space?
With 7% free hard disk your server is probably running out of space to page out memory from either your process or other processes, hence it has to keep everything in RAM and therefore you are unable to allocate additional memory more often than on the server with 50% free space.
What solutions do you guys propose?
Since you've already run a profiler and seen at least 600MB+ of usage with all the string data you need to start tackling this problem.
The obvious answer would be to not hold all that data in memory. If you are processing a large data set then load a bit, process it and then throw that bit away and load the next bit instead of loading it all up front.
If it's data you need to serve, look at a caching strategy like LRU (least recently used) and keep only the hottest data in memory but leave the rest on disk.
You could even offload the strings into a database (in-memory or disk-based) and let that handle the cache management for you.
A slighty left-of-field solution I've had to use in the past was simply compressing the string data in memory as it arrived and decompressing it again when needed using the SharpZipLib. It wasn't that slow surprisingly.

I would agree that your best bet is to use a memory profiler. I've used .Net Memory Profiler 3.5 and was able to diagnose the issue, which in my case were undisposed Regex statements. They have demo tutorials which will walk you through the process if you're not familiar.
As you your question, any single reference to the strings, the jagged array for instance, would still prevent the string from disposing. Without knowing more about your architecture, it would be tough to make a specific recommendation. I would suggest trying to optimize your app before extending memory though. It will come back to bite you later.

An OutOfMemoryException is more likely to indicate fragmentation in your page file - not that you are out of RAM or disk space.
It is generally (wrongly) assumed that the page file is used as a swap disk - that RAM overflow is written to the page file. All allocated memory is stored in the page file and only data that is under heavy usage is copied to RAM.
There's no simple code fix to this problem other than trying to reduce the memory footprint of your application. But if you really get desperate you can always try PageDefrag, which is a free application originally developed by SysInternals.

There is a few tricks to increase memory (I dont know if it works with a web-app, but it looks like it does):
"Out of memory? Easy ways to increase the memory available to your program"
http://blogs.msdn.com/b/calvin_hsia/archive/2010/09/27/10068359.aspx

Hitting a memory limit slows down the .Net application

We have a 64bit C#/.Net3.0 application that runs on a 64bit Windows server. From time to time the app can use large amount of memory which is available. In some instances the application stops allocating additional memory and slows down significantly (500+ times slower).When I check the memory from the task manager the amount of the memory used barely changes. The application keeps on running very slowly and never gives an out of memory exception.
Any ideas? Let me know if more data is needed.

You might try enabling server mode for the Garbage Collector. By default, all .NET apps run in Workstation Mode, where the GC tries to do its sweeps while keeping the application running. If you turn on server mode, it temporarily stops the application so that it can free up memory (much) faster, and it also uses different heaps for each processor/core.
Most server apps will see a performance improvement using the GC server mode, especially if they allocate a lot of memory. The downside is that your app will basically stall when it starts to run out of memory (until the GC is finished).
* To enable this mode, insert the following into your app.config or web.config:
<configuration>
<runtime>
<gcServer enabled="true"/>
</runtime>
</configuration>

The moment you are hitting the physical memory limit, the OS will start paging (that is, write memory to disk). This will indeed cause the kind of slowdown you are seeing.
Solutions?
Add more memory - this will only help until you hit the new memory limit
Rewrite your app to use less memory
Figure out if you have a memory leak and fix it
If memory is not the issue, perhaps your application is hitting CPU very hard? Do you see the CPU hitting close to 100%? If so, check for large collections that are being iterated over and over.

As with 32-bit Windows operating systems, there is a 2GB limit on the size of an object you can create while running a 64-bit managed application on a 64-bit Windows operating system.
Investigating Memory Issues (MSDN article)

There is an awful lot of good stuff mentioned in the other answers. However, I'm going to chip in my two pence (or cents - depending on where you're from!) anyway.
Assuming that this is indeed a 64-bit process as you have stated, here's a few avenues of investigation...
Which memory usage are you checking? Mem Usage or VMem Size? VMem size is the one that actually matters, since that applies to both paged and non-paged memory. If the two numbers are far out of whack, then the memory usage is indeed the cause of the slow-down.
What's the actual memory usage across the whole server when things start to slow down? Does the slow down also apply to other apps? If so, then you may have a kernel memory issue - which can be due to huge amounts of disk accessing and low-level resource usage (for example, create 20000 mutexes, or load a few thousand bitmaps via code that uses Win32 HBitmaps). You can get some indication of this on the Task Manager (although Windows 2003's version is more informative directly on this than 2008's).
When you say that the app gets significantly slower, how do you know? Are you using vast dictionaries or lists? Could it not just be that the internal data structures are getting so big so as to complicate the work any internal algorithms are performing? When you get to huge numbers some algorithms can start to become slower by orders of magnitude.
What's the CPU load of the application when it's running at full-pelt? Is actually the same as when the slow-down occurs? If the CPU usage decreases as the memory usage goes up, then that means that whatever it's doing is taking the OS longer to fulfill, meaning that it's probably putting too much load on the OS. If there's no difference in CPU load, then my guess is it's internal data structures getting so big as to slow down your algos.
I would certainly be looking at running a Perfmon on the application - starting off with some .Net and native memory counters, Cache hits and misses, and Disk Queue length. Run it over the course of the application from startup to when it starts to run like an asthmatic tortoise, and you might just get a clue from that as well.

Having skimmed through the other answers, I'd say there's a lot of good ideas. Here's one I didn't see:
Get a memory profiler, such as SciTech's MemProfiler. It will tell you what's being allocated, by what, and it will show you the whole slice n dice.
It also has video tutorials in case you don't know how to use it. In my case, I discovered I had IDisposable instances that I wasn't Using(...)