Develop Reference

Large Gaps within Generation #0 heap

I'm currently investigating a memory leak, which I can't really explain and now I'm searching for helpful links or ideas.
Here is a screenshot of the native memory of the application (made with the .NET Memory profiler):
The application takes around 2.2 GB (which is normal). The dump was taken when the application had around ~3.5 GB. And these Gaps in the generation #0 are what I currently can't explain. For me this seems as the garbage collector doesn't compact the gaps in generation #0.
In order to have one clear question:
How do these kind of gaps happen? For me this seems as the GC has collected dead objects but hasn't compacted the heap. I know that I can't trigger or force the GC to compact the heap.
I've searched on this site for similar questions, but most of these are about the LOH, (which seems fine in my case). The only question which has some kind of similar large gaps is this: What are GC Holes, but I can't see how 2 KB of Generation #0 pinned instances produce 1 GB holes. Another question is about the threshold to trigger the GC GC thresholds. But I can't believe that there wasn't a single compaction phase.

Holes/Gaps represent memory that is unused between two allocated instances. “Holes” appear when the heap is not fully compacted (due to pinned instances or optimizations in the garbage collector).
You can't explicitly compact the heap per se. However, the GC is sometimes able to do this performing either a full or partial collect.
So you have gaps, what does it mean? It means it makes more complicated to allocate to the heap, what can you do about it... not a lot. if its a performance issue, you could play around with the GC a bit. However if you are using lots of pinned memory there isn't much you can do. This is called fragmentation
Additional resources
Garbage Collector Basics and Performance Hints
What are GC holes?
GC behavior when pinning an object
What causes memory fragmentation in .NET
No More Memory Fragmentation on the .NET Large Object Heap

To this day I'm still not certain what exactly caused this issue but it was resolved this way. First of all I had some Windows Dumps and NMP Dumps and poked a lot in them (WinDBG was pretty helpful). There I found out, that some of the data was related to sockets.
So the only shot was to rewrite the current socket implementation from the Begin* calls to the *Async calls. (see this question)
There we followed the msdn implementation. (Create one large buffer and reuse it). Especially the comments there point in that direction:
// Allocates one large byte buffer which all I/O operations use a piece of. This gaurds
// against memory fragmentation
The gaps are not gone now, but they are much smaller and they don't grow any more...

C# application using too much ram

stackoverflow. I've written a application in C# (a raytracer). Problem is not with raytracer but with ray usage. Application keep consuming more ram over time.
Ram usage while not raytracing : start with 10mb goes up and up
Ram usege while raytracing : start 80mb then 120mb and up, if resolution is big
this number is up to 500mb.
I looked to VS Diagnostics tool and see application garbage collection is like 10 gc in 1 minute. This number is even bigger while application is raytracing. Raytracer returns a gigantic pixel array. I set pixel array to null after raytracing but application ram stucks in a range of 500-600 mb ram.
Are there any options to collect more garbage or free ram?
EDIT
This is a WinForms application, not WPF or something.

I cant be sure since there is no code, but it sounds like a memory leak. Verify that there are no static object referencing the problematic object. To verify, run GC.Collect() after you set the object to null since if its in generation 2 for example it might take a while for the garbage collector to check it and reclaim it's memory. if after GC.Collect you see that the memory is not reclaimed, you most likely have a memory leak somewhere. If you cant find the location of the rouge reference, you can use tools like this https://blogs.msdn.microsoft.com/visualstudioalm/2015/04/29/diagnosing-event-handler-leaks-with-the-memory-usage-tool-in-visual-studio-2015/
Good luck

Can i recommend you attempt to identify if any memory leaks are contributing to the progressive increase of memory usage of your application. There are plenty of free and paid tools that are available to achieve this task, along with endless advice on how to go about using them. The vs diagnostics tool has the ability to snapshot the memory usage of your application identifying where it is in use and what class types are present. Further information and example of use for the VS diagnostics tools can be found here : https://channel9.msdn.com/Shows/Visual-Studio-Toolbox/Visual-Studio-2015-Diagnostic-Tools
Further Help : What strategies and tools are useful for finding memory leaks in .NET?
Similar searches for ".net identify memory leaks" on here and other platforms will also help.
Commonly memory leaks are caused by the incorrect disposal and the retention of circular references, so identify any locations across your code where circular references are present.

C# Garbage Collector behavior

We have an application in C# that controls one of our device and reacts to signal this device gives us.
Basically, the application creates threads, treat operations (access to a database, etc) and communicates with this device.
In the life of the application, it creates objects and release them and so far, we're letting the Garbage Collector taking care of our memory. i've read that it is highly recommanded to let the GC do its stuff without interfering.
Now the problem we're facing is that the process of our application grows ad vitam eternam, growing by step. Example:
It seems to have "waves" when the application is growing and all of a sudden, the application release some memory but seems to leave memory leaks at the same time.
We're trying to investigate the application with some memory profiler but we would like to understand deeply how the garbage Collector works.
I've found an excellent article here : The Danger of Large Objects
I've also found the official documentation here : MSDN
Do you guys know another really really deep documentation of GC?
Edit :
Here is a screenshot that illustrates the behavior of the application :
You can clearly see the "wave" effect we're having on a very regular pattern.
Subsidiary question :
I've seen that my GC Collection 2 Heap is quite big and following the same pattern as the total bytes used by my application. I guess it's perfectly normal because most of our objects will survive at least 2 garbage collections (for example Singleton classes, etc)... What do you think ?

The behavior you describe is typical of problems with objects created on Large Object Heap (LOH). However, your memory consumption seems to return to some lower value later on, so check twice whether it is really a LOH issue.
You are obviously aware of that, but what is not quite obvious is that there is an exception to the size of the objects on LOH.
As described in documentation, objects above 85000 bytes in size end up on LOH. However, for some reason (an 'optimization' probably) arrays of doubles which are longer than 1000 elements also end up there:
double[999] smallArray = ...; // ends up in 'normal', Gen-0 heap
double[1001] bigArray = ...; // ends up in LOH
These arrays can result in fragmented LOH, which requires more memory, until you get an Out of memory exception.
I was bitten by this as we had an app which received some sensor readings as arrays of doubles which resulted in LOH defragmentation since every array slightly differed in length (these were readings of realtime data at various frequencies, sampled by non-realtime process). We solved the issue by implementing our own buffer pool.

I did some research on a class I was teaching a couple of years back. I don't think the references contain any information regarding the LoH but I thought it was worthwhile to share them either way (see below). Further, I suggest performing a second search for unreleased object references before blaming the garbage collector. Simply implementing a counter in the class finalizer to check that these large objects are being dropped as you believe.
A different solution to this problem, is simply to never deallocate your large objects, but instead reuse them with a pooling strategy. In my hubris I have many times before ended up blaming the GC prematurely for the memory requirements of my application growing over time, however this is more often than not a symptom of faulty implementation.
GC References:
http://blogs.msdn.com/b/clyon/archive/2007/03/12/new-in-orcas-part-3-gc-latency-modes.aspx
http://msdn.microsoft.com/en-us/library/ee851764.aspx
http://blogs.msdn.com/b/ericlippert/archive/2010/09/30/the-truth-about-value-types.aspx
http://blogs.msdn.com/b/ericlippert/archive/2009/04/27/the-stack-is-an-implementation-detail.aspx
Eric Lippert's blog is especially interesting, when it comes to understanding anything C# in detail!

Here is an update with some of my investigations :
In our application, we're using a lot of thread to make different tasks. Some of these threads have higher priority.
1) We're using a GC that is concurrent and we tried to switch it back to non-concurrent.
We've seen a dramatic improvment :
The Garbage collector is being called much often and it seems that, when called more often, it's releasing much better our memory.
I'll post a screenshot as soon as I have a good one to illustrate this.
We've found a really good article on the MSDN. We also found an interesting question on SO.
With the next Framework 4.5, 4 possibilities will be available for GC configuration.
Workstation - non-concurrent
Workstation - concurrent
Server - non-concurrent
Server - concurrent
We'll try and switch to the "server - non-concurrent" and "serveur - concurrent" to check if it's giving us better performance.
I'll keep this thread updated with our findings.

Is There a .Net Memory Profiler that will track all allocations on the Large Object Heap? [closed]

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 4 years ago.
Improve this question
Most .NET memory profilers that I've tried allow you to take snapshots of memory.
However, I'm trying to diagnose an issue where I'm ending up with huge amounts of memory allocated to .NET that is indicated as "free" by the ANTS profiler. (I've confirmed this problem with other profilers like Mem Profiler and the CLR profiler.
ANTS is showing that I have a large amount of memory fragmentation (100% of free memory with 150MB as the largest chunk.) The total size of all the objects in the heap is 180MB. I have 553 MB allocated to .NET and 152 allocated to "Unmanaged".
However, the size of the Large Object Heap (LOH) is only 175kb. This is the actual size of the objects allocated there. I don't allocate any objects that end up on the LOH permanently.
Hence, my problem, some where along the line, I suspect I am somehow allocating large objects (over the 85k limit for the LOH) and then disposing them.
I'm reading large amounts of data (estimating here at several MB) from databases (Oracle, Sql Server), copying this data to object arrays in memory, and processing the data into indexes (arrays, dictionaries, etc) for easier searching/filtering/processing.
My guess is, the data reader(s) are allocating a lot of space temporarily. However, I don't have a good way to pause the program and take a snapshot of the memory.
What I'd like is a profiler that keeps track of every object allocated on the LOH so I can figure out what is causing the LOH fragmentation and excessive memory usage (the memory is not returned to the OS, so it looks like my process is taking 1GB of memory to store 200MB of allocated objects.) I'm guess the memory is not returned because the LOH is not compacted, so I'm stuck with all this memory for the life of my process, which can be weeks (it runs as a windows service.)
Edit: My problem is that my .NET application is using a lot of memory that I can't trace.
Edit1: I've used the Visual Studio memory profiler. While it does tell me all objects that are instantiated, how many, total bytes, etc, I doesn't give me a hint as to why I end up with so much free memory. My only hint/clue is what ANTS is telling me: "Memory Fragmentation is restricting the size of the objects that can be allocated." and I have a lot of unused memory allocated to .NET that is not used.
Edit2: More profiling shows that I have some short lived large objects allocated on the LOH. However, the total amount allocated on the LOH is never more the 3 to 4 MB. However, during this time the private bytes shoot through the roof, doubling and tripling, while the size of my actually allocated objects (on all heaps) only grows slightly. For instance bytes in all heaps is 115MB but my private bytes are over 512 MB.
ANTS is telling me clearly that I am having a problem with memory fragmentation. Turns out I am creating short lived objects on the LOH. However, these objects never total more than 3 or 4 MB. So these short lived large objects (appear to?) are fragmenting the heck out of the LOH.
To respond to Eric Lippert, and the Disney Land parking lot analogy (which is great).
It's like someone parks in a spot for a few minutes, and then leaves. That spot is then reserved (no one else can park there) until I repave the parking lot!
I first starting investigating this when Visual Studio warned me of memory usage and advised switching to x64. (I forget the warning number, quick google doesn't find it). So switching to x64 alleviates the immediate problem, but doesn't address the underlying problem.
It's like I have a parking lot for 1000 cars, but after I put 100 cars in it, my parking attendants are screaming that it's full...
Luckily I have a huge VMware cluster at my disposal and an understanding admin. I've been allocated 8 cpu's and 8 GB of memory. So as far as a problem, I can deal with it, I just throw resources at it. Also, (as i said above) I switched to x64 a while back as Visual Studio kept nagging me with a warning about However, I'd like to figure out what it allocated on the LOH to see if I an mitigate this Heap fragmentation with some small code changes. Perhaps a fool's errand, given that I can just throw resources at it.
The application runs fine, it's fast with the occasional GC pause. But mostly I can live with the situation, I'd just like to know what objects are causing it. My suspicions are some short lived dictionary's that I haven't tracked down yet.
Edit3: http://msdn.microsoft.com/en-us/magazine/cc188781.aspx
ObjectAllocatedByClass does not track allocations of the large object
heap, but ObjectAllocated does. By comparing the notifications from
the two, an enterprising soul should be able to figure out what is in
the large object heap as opposed to the normal managed heap.
So it looks like this can be done. However, my C++ skills are way to rusty to dig into this (maybe sometime in the future if I get more time). I was hoping that a profiler would provide this out of box.

After experimenting, I have been able to be notified when the GC is removing things from a generation, but not when its putting it in there.
Since LOH is not generation specific, and there is no specific event I can access for notification of LOH insertions, then the only alternative I can offer is to debug the application, take a crash dump - OR - run it locally and use WinDBG. Here's how you can do it:
Download the Windows SDK 7
Copy the %Microsoft_NET_Framework%\sos.dll to the WinDBG directory
IN WinDBG Click File -> Open Executable -> point to your executable
In the command bar at the bottom type g (Go)
Monitor the memory and when you want to analyse it, go to WinDBG -> Debug menu -> Break
Type load .sos - to load the .NET extensions
Type !dumpheap -min 85000 - this will list large objects, which should be residing on the LOH
Address MT Size
0000000012a17048 000007fee7ae6ae8 400032
0000000012a78b00 000007fee7ae6ae8 400032
0000000012ada5b8 000007fee7ae6ae8 400032
0000000012b3c070 000007fee7ae6ae8 400032
0000000012b9db28 000007fee7ae6ae8 400032
Next we need to go through each of these and find out whats in them.
Copy the first column (the object address) into the clipboard
Type !do <paste from clipboard>
This will list the contents of the object, its type and size.
CLR Version: 4.0.30319.261
SOS Version: 4.0.30319.239
Name: System.String
MethodTable: 000007fee7ae6ae8
EEClass: 000007fee766ed68
Size: 400026(0x61a9a) bytes
File: C:\Windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
String: 8470737076787475867884758166807183888774746571677189..
Fields:
MT Field Offset Type VT Attr Value Name
000007fee7aec9d0 4000103 8 System.Int32 1 instance 200000 m_stringLength
000007fee7aeb510 4000104 c System.Char 1 instance 38 m_firstChar
000007fee7ae6ae8 4000105 10 System.String 0 shared static Empty
>> Domain:Value 000000000055fe50:0000000002a11420 <<
And the lines you want to look for are:
Size: 400026(0x61a9a) bytes
String: 8470737076787475867884758166807183888774746571677189..
Do this for each object
(However, I am assuming it would be a string, so check out the 'Name' property just to be sure. It could be an array.)

Oh god - #EricLippert is in the question - I know I've got this one wrong - but here goes :-)
LOH will only store a specific objects greater than 85k in size. I've found from experience this is usually strings - XML or large enumerable types that were in global variables and/or static variables.
If your LOH is small, you must be storing other objects in memory which most likely end up as Generation 2. These could be global variables like arrays, dictionaries throughout the lifecycle of the application.
So first thing to quickly check is to check if there are any variables that would be present throughout the lifetime of the application? Do these have thousands of entries?
Secondly, .NET has a performance counter statistics for memory size for each generation size. The counters can be pointed directly at an application, so this would be a good next step.
Finally, the GC will only clear out memory when it needs to. So yes, it is possible it would remain the same size if the GC doesn't feel the need (memory pressure wise) to clear out anything. You can force Garbage Collection, but it is generally recommended to let the GC do its job.
I asked a similar question about why a string was being stored in my application for so long. I was using WinDBG to analyse a crash dump I forced by going to Task Manager -> Processes -> Right Click -> Create Crash Dump. Then loaded it into WinDBG (Windows SDK 7).
Hopefully the link will provide you with more guidance to tracking down where the problem is rooted (pun intended).
And a question back to the OP: Is this impacting your server severly or are you just curious as to why the memory is sticking about?

Reasons for seeing high "% Time in GC" in Perf Mon

While monitoring our application in Perf Mon I noticed that the % of Time In GC is anywhere from 20 - 60% while our application is performing a long running process (varies between 30 seconds to 1.5 minutes). This seems a bit excessive to me. This raises two important questions.
Am I correct that this excessive?
How can I figure out why route causes GC spikes?

Yes, this does sound excessive. Reducing the amount of GC would probably be the single best step you could take to reducing the runtime of your application (if that is your goal).
A high "% time in GC" is typically caused by allocating and then throwing away thousands or millions of objects. A good way to find out what's going on is to use a memory profiler tool.
Microsoft provides the free CLR Profiler. This will show you every allocation, but will make your app run 10-60 times slower. You may need to run it on less input data so that it can finish analyzing in a reasonable amount of time.
A great commercial tool is SciTech's .NET Memory Profiler. This imposes much less runtime overhead, and there is a free trial available. By taking multiple snapshots while your process is running, you can find out what type of objects are being frequently allocated (and then destroyed).
Once you've identified the source of the allocations, you then need to examine the code and figure out how those allocations can be reduced. While there are no one-size-fits-all answers, some things I've encountered in the past include:
String.Split can create hundreds of small short-lived strings. If you're doing a lot of string manipulation, it can help to process the string by walking it character-by-character.
Creating arrays or lists of thousands of small classes (say, under 24 bytes in size) can be expensive; if those classes can be treated as value types, it can (sometimes) greatly improve things to change them to structs.
Creating thousands of small arrays can increase memory usage a lot (because each array has a small amount of overhead); sometimes these can be replaced with one large array and indexes into a sub-section of it.
Having a lot of finalizable objects (particularly if they're not being disposed) can put a lot of pressure on the garbage collector; ensure that you're correctly disposing all IDisposable objects, and note that your own types should (almost) never have finalizers.
Microsoft has an article with Garbage Collection Guidelines for improving performance.

Am I correct that this excessive?
Yes, you are correct
How can I figure out why route causes GC spikes?
1.- Do take a look at PerfView
PerfView is a performance-analysis tool that helps isolate CPU- and
memory-related performance issues.
See Also: Improving Managed Code Performance
2.- See if GC.Collect or GC.WaitForPendingFinalizers is being called anywhere in your code or third party library. The latter can cause high CPU utilization.

Another reason could be lots of gen-1 or gen-2 collections, each of which takes MUCH more time and is caused by hanging on to objects a longer time.
I've seen this happen in web apps when buggy objects hang onto actual page objects - forcing the page to live as long as the other objects referring to them.
Breaking the link between objects and pages (in this case) caused GC to drop to very low values. Our site now has 100+ hits/second and GC time is typically 1% or less.