Develop Reference

Memory leak because of pinned GC handles / no gc root visible

What is the reason for pinned GC handles when working with unmanaged .net components? This happens from time to time without any code changed or something else. When investigating the issue, I see a lot of pinned GC-Handles
These handles seem to stick in the memory for the entire application lifetime. In this case, the library is GdPicture (14). Is there any way to investigate why those instances are not cleaned up? I'm using Dispose()/using everywhere and can't find any GC roots in the managed code.
Thanks a lot!
EDIT
Another behaviour that is strange is, that the task manager shows that the application uses about 6GB ram, when the memory profiler shows the usage of 400MB (red line is live bytes)

What is the reason for pinned GC handles when working with unmanaged .net components?
Pinning is needed when working with unmanaged code. It prevents objects from being moved during garbage collection so that the unmanaged code can have a pointer to it. The garbage collector will update all .NET references, but it will not update unmanaged pointer values.
Is there any way to investigate why those instances are not cleaned up?
No. The reason always is: there's a bug in the code. Either your code (assume that first) or in a 3rd party library (libraries are used often, chances are that leaks in the library have been found by someone else before).
I'm using Dispose()/using everywhere
Seems like you missed one or it's not using the disposable pattern.
Another behaviour that is strange is, that the task manager shows that the application uses about 6GB ram, when the memory profiler shows the usage of 400MB (red line is live bytes)
A .NET memory profiler may only show the .NET part of memory (400 MB) and omit the rest (5600 MB).
Task manager is not interested in .NET. It cares about physical RAM mostly, which is why Task Manager is not a good analytics tool in general. You don't want to analyze physical RAM, you want to analyze virtual memory.
To look for memory leaks, use Process Explorer and show the "Private Bytes" and "Virtual size" column. Process Explorer can also show you a graph over time per process.
How to proceed?
Forget about the unmanaged leak for a moment. Use a .NET profiler that has the capability of taking memory snapshots and allows you to see each individual object inside as well as a statistics.
Try to figure out the steps that it takes to create more leaks in a consistent way. Then
Take a snapshot
Repeat the leak procedure 10 times
Take a snapshot
Repeat the leak procedure another 10 times
Take a snaphot
Compare snapshot of step 1 and 3. Check for managed types that differ in multiples of 10. Compare snapshot of step 3 and 5. Check the same type again. It must be a multiple of 10. You can't leak 7 objects when you run a method 10 times.
Do a code review on the places where the affected types are used based on internal knowledge on the leak procedure (which methods are called) and the managed type. Make sure it's disposed or released properly.

Strategies for Diagnosing Memory Leak in C# program updating an Azure Table

I have a fairly simple application that takes a data file with about 500k lines of data in it, parses the data, organizes it and then inserts it into an Azure Table. There are about 2000 of these files and I need the process to work smoothly as I load all of the data.
I am using WindowsAzure.Storage v5.0.2 for inserting the data and Microsoft.tpl.dataflow v4.4.24 for parallelization. Each file is completely processed and all tasks finalized before I move onto the next file. I am also disposing of all objects I can and setting everything else to Null at the end of each file load.
Despite trying to be as careful as possible the RAM usage goes up steadily until it crashes the process. When it starts it jumps up to 1 GB of RAM used and steadily climbs until the process crashes somewhere above 9GB of RAM consumed.
Note - this is targeting x64 on a reasonably large computer. Garbage collection is happening on a regular basis, but it doesn't seem to affect the memory problem.
At this point I am completely confused about where the memory leak is coming from and don't have any idea about how to diagnose the problem.
Update
After a lot of work and following the suggestion below I found out that the parallelization I was using was allowing more simultaneous insert processes than I expected. It looked like the insert was complete and my code was starting the next insert. In reality the parallel process had just reported back a status but had not finished. This led to a huge backlog of simultaneous inserts happening, chewing up RAM and crashing the system. It also cause me to go past my IOPS limit which I believe triggered throttling, compounding the problem.
Figuring this out required a huge amount of work and many different ways of analysing everything, but the suggestion below got me going in the right direction.

Doing a search for 'troubleshoot .net memory leak' will yield lots of results, but probably the best one is http://blogs.msdn.com/b/tess/archive/2009/05/12/debug-diag-script-for-troubleshooting-net-2-0-memory-leaks.aspx.
Basically, use DebugDiag to generate a memory leak analysis report and then look for which objects are consuming all of the memory. You will likely see one type of object that you didn't realize you were continuously adding to a collection without removing it later on.

does mono/.Net GC release free allocated memory back to OS after collection? if not, why?

I heard many times that once C# managed program request more memory from OS, it doesn't free it back, unless system is out of memory. Eg. when object is collected, it gets deleted, and memory that was occupied by the object is free to reuse by another managed object, but memory itself is not returned to operating system (for example, mono on unix wouldn't call brk / sbrk to decrease the amount of virtual memory available to the process back to what it was before its allocation).
I don't know if this really happens or not, but I can see that my c# applications, running on linux, use small amount of memory on beginning, then when I do something memory expensive, it allocates more of it, but later on when all objects get deleted (I can verify that by putting debug message to destructors), the memory is not free'd. On other hand no more memory is allocated when I run that memory expensive operation again. The program just keep on eating the same amount of memory until it is terminated.
Maybe it is just my misunderstanding of how GC in .net works, but if it really does work like this, why is that? What is a benefit of keeping the allocated memory for later, instead of returning it back to the system? How can it even know if system need it back or not? What about other application that would crash or couldn't start because of OOM caused by this effect?
I know that people will probably answer something like "GC manages memory better than you ever could, just don't care about it" or "GC knows what it does best" or "it doesn't matter at all, it's just virtual memory" but it does matter, on my 2gb laptop I am running OOM (and kernel OOM killer gets started because of that) very often when I am running any C# applications after some time precisely because of this irresponsible memory management.
Note: I was testing this all on mono in linux because I really have hard times understanding how windows manage memory, so debugging on linux is much easier for me, also linux memory management is open source code, memory management of windows kernel / .Net is rather mystery for me

The memory manager works this way because there is no benefit of having a lot of unused system memory when you don't need it.
If the memory manager would always try to have as little memory allocated as possible, that would mean that it would do a lot of work for no reason. It would only slow the application down, and the only benefit would be more free memory that no application is using.
Whenever the system needs more memory, it will tell the running applications to return as much as possible. The same signal is also sent to an application when you minimise it.
If this doesn't work the same with Mono in Linux, then that is a problem with that specific implementation.

Generally, if an app needs memory once, it will need it again. Releasing memory back to the OS only to request it back again is overhead, and if nothing else wants the memory: why bother?. It is trying to optimize for the very likely scenario of wanting it again. Additionally, releasing it back requires entire / contiguous blocks that can be handed back, which has very specific impact on things like compaction: it isn't quite as simple as "hey, I'm not using most of this : have it back" - it needs to figure out what blocks can be released, presumably after a full collect and compact (relocate objects etc) cycle.

Most Accurate Way to Get Memory Usage for Silverlight Browser Application

Currently i'm tracking down a memory leak issue in a Silverlight 4 application. I can watch the memory used by iexplorer.exe increase gradually using Task Manager.
However, I am looking for a way within the Silverlight code to tell how much memory the application is currently using within the IE Process. I have tried GC.GetTotalMemory but it does not represent the actual memory usage by iexplorer.
I have developer tools for tracking down the memory leak itself, so i'm not looking for that but a method within my Silverlight code to get total IE memory usage, or at least a closer representation of what is actually being used by Silverlight instead of what the Garbage collector thinks is being used.

GC allocated memory is good estimation of dynamic portion of memory used by SL application. Otherwise it is unclear what other types of memory usage you need.
If you need some other data consider using VaDump first ( http://msdn.microsoft.com/en-us/magazine/dd882521.aspx linked from Using VADump to track memory usage - OpenProcess Failed c0000034). When you figure out what portion/type of memory you are interested in it should be relatively easy to get this information (you may need to collect it with native calls, but likely all can be PInvoke'ed).
Update: I think all information you looking for is avaialble through performance counters - usage sample - http://msdn.microsoft.com/en-us/library/system.diagnostics.performancecounter.countername.aspx (but I'm don't think you can get it from SL).

Try the visual studio profiler
https://blogs.msdn.com/b/profiler/archive/2010/04/26/vs2010-silverlight-4-profiling.aspx
"For .NET memory profiling, use “VSPerfClrEnv /samplegc” or “VSPerfClrEnv /samplegclife” in your step 1 to get either allocation profiling or allocation and lifetime profiling. "

ANTS Profiler...
http://www.red-gate.com/products/dotnet-development/ants-memory-profiler/

Causes for web service memory leak

We have a web service that uses up more and more private bytes until that application stops responding. The managed heap (mostly Gen2) will show some 200-250 MB, while private bytes shows over 1GB. What are possible causes of a memory leak outside of the managed heap?
I've already checked for the following:
Prolific dynamic assemblies (Xml serialization, regex, etc.)
Session state (turned off)
System.Policy.Evidence memory leak (SP1 installed)
Threading deadlock (no use of Join, only lock)
Use of SQLOLEDB (using SqlClient)
What other sources can I check for?

Make sure your app is complied in release mode. If you compile under debug mode, and deploy that, simply instantiating a class that has an event defined (event doesn't even need to be raised), will cause a small piece of memory to leak. Instantiating enough of these objects over a long enough period of time will cause all the memory to be used. I've seen web apps that would use up all the memory within a matter of hours, simply because a debug build was used. Compiling as a release build immediately and permanently fixed the problem.

I would recommend you view snapshots of the stack at various times, and see what's using up the memory. If your application is using Java, then jmap works extremely well - you just give it the PID of the java process.
If using something else, try Lambda Probe (http://www.lambdaprobe.org/d/index.htm). It doesn't show as much detail, but will at least show you memory use.
I had a bad memory leak in my JDBC code that ended up being traced to a change in the JDBC specification a few years ago that I missed (with respect to closing statements and such). It took a combination of Lamdba Probe and then jmap to localize the problem enough to fix it.
Cheers,
-R

Also look for:
COM Assemblies being loaded
DB Connections not being closed
Cache & State (Session, Application)
Try forcing the Garbage Collector (GC) to run (write a page that does it when it loads) or try the instrumentation, but that's a bit hit and miss in my experience. Another thing would be to keep it running and see if it runs out of memory.
What could be happening is that there is plenty of memory and Windows does not signal your app to clean up. This causes the app to look like its using more and more memory because it can, when in fact the system can reclaim the memory when it needs. SQL Server and Exchange do this a lot. The idea is why cause a unnecessary cleanup when there are plenty of resources.
Rob

Garbage collection does not run until a request for memory is denied due to lack of available memory. This can often make things look like a memory leak when one is not around.
Do you have any events and event handlers within the service? Services often have static variables, and if you are creating event handlers from the static instances, connected to a non-static instance object, the static will hold a reference to the instance forever, which will stop it from releasing.

Double check that trace is not enabled. I've seen instances of trace slowly consuming memory until the app reaches it's app pool limit.

For what it is worth, my issue was not with the service, but the with HttpClient that was calling it.
The client was not properly disposed, so it kept the connection open and the memory locked.
After disposing the client the service released the memory as expected.