Related
The general advice is that you should not call GC.Collect from your code, but what are the exceptions to this rule?
I can only think of a few very specific cases where it may make sense to force a garbage collection.
One example that springs to mind is a service, that wakes up at intervals, performs some task, and then sleeps for a long time. In this case, it may be a good idea to force a collect to prevent the soon-to-be-idle process from holding on to more memory than needed.
Are there any other cases where it is acceptable to call GC.Collect?
If you have good reason to believe that a significant set of objects - particularly those you suspect to be in generations 1 and 2 - are now eligible for garbage collection, and that now would be an appropriate time to collect in terms of the small performance hit.
A good example of this is if you've just closed a large form. You know that all the UI controls can now be garbage collected, and a very short pause as the form is closed probably won't be noticeable to the user.
UPDATE 2.7.2018
As of .NET 4.5 - there is GCLatencyMode.LowLatency and GCLatencyMode.SustainedLowLatency. When entering and leaving either of these modes, it is recommended that you force a full GC with GC.Collect(2, GCCollectionMode.Forced).
As of .NET 4.6 - there is the GC.TryStartNoGCRegion method (used to set the read-only value GCLatencyMode.NoGCRegion). This can itself, perform a full blocking garbage collection in an attempt to free enough memory, but given we are disallowing GC for a period, I would argue it is also a good idea to perform full GC before and after.
Source: Microsoft engineer Ben Watson's: Writing High-Performance .NET Code, 2nd Ed. 2018.
See:
https://msdn.microsoft.com/en-us/library/system.runtime.gclatencymode(v=vs.110).aspx
https://msdn.microsoft.com/en-us/library/dn906204(v=vs.110).aspx
I use GC.Collect only when writing crude performance/profiler test rigs; i.e. I have two (or more) blocks of code to test - something like:
GC.Collect(GC.MaxGeneration, GCCollectionMode.Forced);
TestA(); // may allocate lots of transient objects
GC.Collect(GC.MaxGeneration, GCCollectionMode.Forced);
TestB(); // may allocate lots of transient objects
GC.Collect(GC.MaxGeneration, GCCollectionMode.Forced);
...
So that TestA() and TestB() run with as similar state as possible - i.e. TestB() doesn't get hammered just because TestA left it very close to the tipping point.
A classic example would be a simple console exe (a Main method sort-enough to be posted here for example), that shows the difference between looped string concatenation and StringBuilder.
If I need something precise, then this would be two completely independent tests - but often this is enough if we just want to minimize (or normalize) the GC during the tests to get a rough feel for the behaviour.
During production code? I have yet to use it ;-p
The best practise is to not force a garbage collection in most cases. (Every system I have worked on that had forced garbage collections, had underlining problems that if solved would have removed the need to forced the garbage collection, and speeded the system up greatly.)
There are a few cases when you know more about memory usage then the garbage collector does. This is unlikely to be true in a multi user application, or a service that is responding to more then one request at a time.
However in some batch type processing you do know more then the GC. E.g. consider an application that.
Is given a list of file names on the command line
Processes a single file then write the result out to a results file.
While processing the file, creates a lot of interlinked objects that can not be collected until the processing of the file have complete (e.g. a parse tree)
Does not keep match state between the files it has processed.
You may be able to make a case (after careful) testing that you should force a full garbage collection after you have process each file.
Another cases is a service that wakes up every few minutes to process some items, and does not keep any state while it’s asleep. Then forcing a full collection just before going to sleep may be worthwhile.
The only time I would consider forcing
a collection is when I know that a lot
of object had been created recently
and very few objects are currently
referenced.
I would rather have a garbage collection API when I could give it hints about this type of thing without having to force a GC my self.
See also "Rico Mariani's Performance Tidbits"
These days I consider same of the above cases would be better to use a short lived worker process to do each batch of work and let the OS do the resource recovery.
One case is when you are trying to unit test code that uses WeakReference.
In large 24/7 or 24/6 systems -- systems that react to messages, RPC requests or that poll a database or process continuously -- it is useful to have a way to identify memory leaks. For this, I tend to add a mechanism to the application to temporarily suspend any processing and then perform full garbage collection. This puts the system into a quiescent state where the memory remaining is either legitimately long lived memory (caches, configuration, &c.) or else is 'leaked' (objects that are not expected or desired to be rooted but actually are).
Having this mechanism makes it a lot easier to profile memory usage as the reports will not be clouded with noise from active processing.
To be sure you get all of the garbage, you need to perform two collections:
GC.Collect();
GC.WaitForPendingFinalizers();
GC.Collect();
As the first collection will cause any objects with finalizers to be finalized (but not actually garbage collect these objects). The second GC will garbage collect these finalized objects.
You can call GC.Collect() when you know something about the nature of the app the garbage collector doesn't.
As the author, it's often tempting to think this is likely or normal. However, the truth is the GC amounts to a pretty well-written and tested expert system, and it's rare you'll know something about the low level code paths it doesn't.
The best example I can think of where you might have some extra information is an app that cycles between idle periods and very busy periods. You want the best performance possible for the busy periods and therefore want to use the idle time to do some clean up.
However, most of the time the GC is smart enough to do this anyway.
One instance where it is almost necessary to call GC.Collect() is when automating Microsoft Office through Interop. COM objects for Office don't like to automatically release and can result in the instances of the Office product taking up very large amounts of memory. I'm not sure if this is an issue or by design. There's lots of posts about this topic around the internet so I won't go into too much detail.
When programming using Interop, every single COM object should be manually released, usually though the use of Marshal.ReleseComObject(). In addition, calling Garbage Collection manually can help "clean up" a bit. Calling the following code when you're done with Interop objects seems to help quite a bit:
GC.Collect()
GC.WaitForPendingFinalizers()
GC.Collect()
In my personal experience, using a combination of ReleaseComObject and manually calling garbage collection greatly reduces the memory usage of Office products, specifically Excel.
As a memory fragmentation solution.
I was getting out of memory exceptions while writing a lot of data into a memory stream (reading from a network stream). The data was written in 8K chunks. After reaching 128M there was exception even though there was a lot of memory available (but it was fragmented). Calling GC.Collect() solved the issue. I was able to handle over 1G after the fix.
Have a look at this article by Rico Mariani. He gives two rules when to call GC.Collect (rule 1 is: "Don't"):
When to call GC.Collect()
I was doing some performance testing on array and list:
private static int count = 100000000;
private static List<int> GetSomeNumbers_List_int()
{
var lstNumbers = new List<int>();
for(var i = 1; i <= count; i++)
{
lstNumbers.Add(i);
}
return lstNumbers;
}
private static int[] GetSomeNumbers_Array()
{
var lstNumbers = new int[count];
for (var i = 1; i <= count; i++)
{
lstNumbers[i-1] = i + 1;
}
return lstNumbers;
}
private static int[] GetSomeNumbers_Enumerable_Range()
{
return Enumerable.Range(1, count).ToArray();
}
static void performance_100_Million()
{
var sw = new Stopwatch();
sw.Start();
var numbers1 = GetSomeNumbers_List_int();
sw.Stop();
//numbers1 = null;
//GC.Collect();
Console.WriteLine(String.Format("\"List<int>\" took {0} milliseconds", sw.ElapsedMilliseconds));
sw.Reset();
sw.Start();
var numbers2 = GetSomeNumbers_Array();
sw.Stop();
//numbers2 = null;
//GC.Collect();
Console.WriteLine(String.Format("\"int[]\" took {0} milliseconds", sw.ElapsedMilliseconds));
sw.Reset();
sw.Start();
//getting System.OutOfMemoryException in GetSomeNumbers_Enumerable_Range method
var numbers3 = GetSomeNumbers_Enumerable_Range();
sw.Stop();
//numbers3 = null;
//GC.Collect();
Console.WriteLine(String.Format("\"int[]\" Enumerable.Range took {0} milliseconds", sw.ElapsedMilliseconds));
}
and I got OutOfMemoryException in GetSomeNumbers_Enumerable_Range method the only workaround is to deallocate the memory by:
numbers = null;
GC.Collect();
You should try to avoid using GC.Collect() since its very expensive. Here is an example:
public void ClearFrame(ulong timeStamp)
{
if (RecordSet.Count <= 0) return;
if (Limit == false)
{
var seconds = (timeStamp - RecordSet[0].TimeStamp)/1000;
if (seconds <= _preFramesTime) return;
Limit = true;
do
{
RecordSet.Remove(RecordSet[0]);
} while (((timeStamp - RecordSet[0].TimeStamp) / 1000) > _preFramesTime);
}
else
{
RecordSet.Remove(RecordSet[0]);
}
GC.Collect(); // AVOID
}
TEST RESULT: CPU USAGE 12%
When you change to this:
public void ClearFrame(ulong timeStamp)
{
if (RecordSet.Count <= 0) return;
if (Limit == false)
{
var seconds = (timeStamp - RecordSet[0].TimeStamp)/1000;
if (seconds <= _preFramesTime) return;
Limit = true;
do
{
RecordSet[0].Dispose(); // Bitmap destroyed!
RecordSet.Remove(RecordSet[0]);
} while (((timeStamp - RecordSet[0].TimeStamp) / 1000) > _preFramesTime);
}
else
{
RecordSet[0].Dispose(); // Bitmap destroyed!
RecordSet.Remove(RecordSet[0]);
}
//GC.Collect();
}
TEST RESULT: CPU USAGE 2-3%
In your example, I think that calling GC.Collect isn't the issue, but rather there is a design issue.
If you are going to wake up at intervals, (set times) then your program should be crafted for a single execution (perform the task once) and then terminate. Then, you set the program up as a scheduled task to run at the scheduled intervals.
This way, you don't have to concern yourself with calling GC.Collect, (which you should rarely if ever, have to do).
That being said, Rico Mariani has a great blog post on this subject, which can be found here:
http://blogs.msdn.com/ricom/archive/2004/11/29/271829.aspx
One useful place to call GC.Collect() is in a unit test when you want to verify that you are not creating a memory leak (e. g. if you are doing something with WeakReferences or ConditionalWeakTable, dynamically generated code, etc).
For example, I have a few tests like:
WeakReference w = CodeThatShouldNotMemoryLeak();
Assert.IsTrue(w.IsAlive);
GC.Collect();
GC.WaitForPendingFinalizers();
Assert.IsFalse(w.IsAlive);
It could be argued that using WeakReferences is a problem in and of itself, but it seems that if you are creating a system that relies on such behavior then calling GC.Collect() is a good way to verify such code.
There are some situations where it is better safe than sorry.
Here is one situation.
It is possible to author an unmanaged DLL in C# using IL rewrites (because there are situations where this is necessary).
Now suppose, for example, the DLL creates an array of bytes at the class level - because many of the exported functions need access to such. What happens when the DLL is unloaded? Is the garbage collector automatically called at that point? I don't know, but being an unmanaged DLL it is entirely possible the GC isn't called. And it would be a big problem if it wasn't called. When the DLL is unloaded so too would be the garbage collector - so who is going to be responsible for collecting any possible garbage and how would they do it? Better to employ C#'s garbage collector. Have a cleanup function (available to the DLL client) where the class level variables are set to null and the garbage collector called.
Better safe than sorry.
The short answer is: never!
using(var stream = new MemoryStream())
{
bitmap.Save(stream, ImageFormat.Png);
techObject.Last().Image = Image.FromStream(stream);
bitmap.Dispose();
// Without this code, I had an OutOfMemory exception.
GC.Collect();
GC.WaitForPendingFinalizers();
//
}
Another reason is when you have a SerialPort opened on a USB COM port, and then the USB device is unplugged. Because the SerialPort was opened, the resource holds a reference to the previously connected port in the system's registry. The system's registry will then contain stale data, so the list of available ports will be wrong. Therefore the port must be closed.
Calling SerialPort.Close() on the port calls Dispose() on the object, but it remains in memory until garbage collection actually runs, causing the registry to remain stale until the garbage collector decides to release the resource.
From https://stackoverflow.com/a/58810699/8685342:
try
{
if (port != null)
port.Close(); //this will throw an exception if the port was unplugged
}
catch (Exception ex) //of type 'System.IO.IOException'
{
System.GC.Collect();
System.GC.WaitForPendingFinalizers();
}
port = null;
If you are creating a lot of new System.Drawing.Bitmap objects, the Garbage Collector doesn't clear them. Eventually GDI+ will think you are running out of memory and will throw a "The parameter is not valid" exception. Calling GC.Collect() every so often (not too often!) seems to resolve this issue.
i am still pretty unsure about this.
I am working since 7 years on an Application Server. Our bigger installations take use of 24 GB Ram. Its hightly Multithreaded, and ALL calls for GC.Collect() ran into really terrible performance issues.
Many third party Components used GC.Collect() when they thought it was clever to do this right now.
So a simple bunch of Excel-Reports blocked the App Server for all threads several times a minute.
We had to refactor all the 3rd Party Components in order to remove the GC.Collect() calls, and all worked fine after doing this.
But i am running Servers on Win32 as well, and here i started to take heavy use of GC.Collect() after getting a OutOfMemoryException.
But i am also pretty unsure about this, because i often noticed, when i get a OOM on 32 Bit, and i retry to run the same Operation again, without calling GC.Collect(), it just worked fine.
One thing i wonder is the OOM Exception itself...
If i would have written the .Net Framework, and i can't alloc a memory block, i would use GC.Collect(), defrag memory (??), try again, and if i still cant find a free memory block, then i would throw the OOM-Exception.
Or at least make this behavior as configurable option, due the drawbacks of the performance issue with GC.Collect.
Now i have lots of code like this in my app to "solve" the problem:
public static TResult ExecuteOOMAware<T1, T2, TResult>(Func<T1,T2 ,TResult> func, T1 a1, T2 a2)
{
int oomCounter = 0;
int maxOOMRetries = 10;
do
{
try
{
return func(a1, a2);
}
catch (OutOfMemoryException)
{
oomCounter++;
if (maxOOMRetries > 10)
{
throw;
}
else
{
Log.Info("OutOfMemory-Exception caught, Trying to fix. Counter: " + oomCounter.ToString());
System.Threading.Thread.Sleep(TimeSpan.FromSeconds(oomCounter * 10));
GC.Collect();
}
}
} while (oomCounter < maxOOMRetries);
// never gets hitted.
return default(TResult);
}
(Note that the Thread.Sleep() behavior is a really App apecific behavior, because we are running a ORM Caching Service, and the service takes some time to release all the cached objects, if RAM exceeds some predefined values. so it waits a few seconds the first time, and has increased waiting time each occurence of OOM.)
one good reason for calling GC is on small ARM computers with little memory, like the Raspberry PI (running with mono).
If unallocated memory fragments use too much of the system RAM, then the Linux OS can get unstable.
I have an application where I have to call GC every second (!) to get rid of memory overflow problems.
Another good solution is to dispose objects when they are no longer needed. Unfortunately this is not so easy in many cases.
This isn't that relevant to the question, but for XSLT transforms in .NET (XSLCompiledTranform) then you might have no choice. Another candidate is the MSHTML control.
If you are using a version of .net less than 4.5, manual collection may be inevitable (especially if you are dealing with many 'large objects').
this link describes why:
https://blogs.msdn.microsoft.com/dotnet/2011/10/03/large-object-heap-improvements-in-net-4-5/
Since there are Small object heap(SOH) and Large object heap(LOH)
We can call GC.Collect() to clear de-reference object in SOP, and move lived object to next generation.
In .net4.5, we can also compact LOH by using largeobjectheapcompactionmode
I need to reduce the memory leak in c# WPF after disposing all the used objects. But I couldn't reduce the memory consumption completely by using the following code snippet.
Here is my code:
string str;
Uri uri;
private void Button_Click(object sender, RoutedEventArgs e) // "Load" Button
{
if(img.Source!=null)
Unload();
str = "F://Photos//Parthi//IMG_20141128_172826244.jpg"; // File Size: 0.643 MB
uri = new Uri(str);
img.Source = new BitmapImage(uri);
}
private void Button_Click_1(object sender, RoutedEventArgs e) //"Unload Button"
{
Unload();
}
private void Unload()
{
Bitmap bmp = GetBitmap(img.Source as BitmapSource);
bmp.Dispose();
bmp = null;
img.Source = null;
str = string.Empty;
uri = null;
}
private Bitmap GetBitmap(BitmapSource source)
{
Bitmap bmp = new Bitmap(source.PixelWidth, source.PixelHeight, System.Drawing.Imaging.PixelFormat.Format32bppPArgb);
BitmapData data = bmp.LockBits(new System.Drawing.Rectangle(System.Drawing.Point.Empty, bmp.Size), ImageLockMode.WriteOnly, System.Drawing.Imaging.PixelFormat.Format32bppPArgb);
source.CopyPixels(Int32Rect.Empty, data.Scan0, data.Height * data.Stride, data.Stride);
bmp.UnlockBits(data);
data = null;
source = null;
return bmp;
}
After Running the sample, While checking it in Task manager, the following memory consumption reading has produced,
Before "Load" Button is clicked : 10.0 MB
After "Load" Button is clicked : 47.8 MB
After "Unload" Button is clicked : 26.0 MB
After Unloading, I need to reduce the memory closely to 10.0 MB. So please help me regarding this.
Thanks in Advance.
Using the .net platform does not have control over the memory as if it works in c / c ++
The garbage collector operates very complex policies and the fact that you see down the memory to 26 but 10 is normal.
The .net reserves a space to recharge faster data and guarantee a free space in main memory without having to require continuous OS.
What I've noticed is perfectly normal.
It 'a question that in the past I personally subjected to a Microsoft MVP during a course
View this:
http://msdn.microsoft.com/en-us/library/ee787088%28v=vs.110%29.aspx
and this: Best Practice for Forcing Garbage Collection in C#
First of all, do not use Task Manager to view your memory usage, because it only shows the memory that the windows process allocates. There are plenty of much better tools out there and even the Performance Monitor, which comes on Windows, will give you a better idea of your application performance and any memory leaks. You may start it by running perfmon.exe.
In this sample application, GC does not become aggressive with collections until heap reaches approximately 85MB. And why would I want it to? It's not a lot of memory and it works very well as a caching solution, if I decide to use the same objects again.
So, I suggest taking a look at that tool. It gives a nice overview of what's going on and it's free.
Second of all, just because you called a .Dispose() to release those resources, does not mean that memory is released right away. You're basically making it eligible for the garbage collection and when GC gets to it, it'll take care of it.
The default garbage collection behavior for a WPF application is Concurrent(<4.0)/Background(4.0=<) Workstation GC. It runs on a special thread and most of the time tries to run concurrently to the rest of the application (I say most of the time, because once in a while it'll extremely briefly suspend other threads, so that it may complete its cleanup). It doesn't wait until you're out of memory, but at the same time, it does collection only when it doesn't affect the performance greatly -- sort of a balanced trade-off.
Another factor to consider is that you have a 0.643 MB jpeg file and once you count in BitmapImage, it's a bit more... well, anything above 85,000 bytes is considered to be a large object and thus, it is placed into generation 2, which contains the large object heap. Generation 2 garbage collection is expensive and is done infrequently. However, if you're running .NET 4.5.1, and I'm not sure if you are, you may force a compaction:
GCSettings.LargeObjectHeapCompactionMode = GCLargeObjectHeapCompactionMode.CompactOnce;
GC.Collection();
As with .Dipose(), this is not going happen right away, as it's an expensive process and it'll take a little longer than the generation 1 less-than-a-millisecond sweep. And honestly, you probably wouldn't even benefit from it, since I doubt that you would have a high fragmentation in the LOH with that application. I just mentioned it so that you're aware of that option if you ever need it.
So, why am I giving you a brief lesson on the default GC behavior of a WPF application (and most of .NET applications for that matter)? Well, it's important to understand the behavior of GC and acknowledge its existence. Unlike a C++ application, where you're granted a lot of control and freedom over your memory allocation, a .NET application utilizes a GC. The trade-off is that the memory is freed when it's freed by the GC. There may even be times when it frees it prematurely, from your point of view, and that's when you would explicitly keep an object alive by calling GC.KeepAlive(..). A lot of embedded systems do not make use of a GC and if you want very precise control over your memory, I suggest that you do not either.
If you want to be aware of how memory is being handled in a .NET application, I strongly recommend educating yourself on the inner workings of the garbage collector. What I've told you about is an incredibly brief snapshot of the default behavior and there is a lot more to it. There are a few modes, which provide different behaviors.
try to manage the usage of your variables. you can do something like this :
private void Button_Click(object sender, RoutedEventArgs e)
{
img.Source = new BitmapImage(new Uri(#"F://Photos//Parthi//IMG_20141128_172826244.jpg"));
}
if 'str' and 'uri' is not necessary needed by the whole class you can just add it directly it to your object. Or if you really need to store it and reuse your data that really has a huge amount of size you can just store it on Physical file on a temp file to prevent memory leakage; instead of storing it to memory you can store it on physical storage.
hope it helps
Your Unload method calls GetBitmap, but GetBitmap returns a new object every time, so as far as I can tell, you're never actually going to dispose of img.Source properly
I am trying to write a class that wraps a buffer allocated with Marshal.AllocHGlobal. I implemented the IDisposable interface, and added a finalizer that should release the memory when I don't need it anymore (when the object goes out of scope).
When I test the class, the GC does not call the finalizer or the Dispose method of my classes, even though they are out of scope. As a result, I get an OutOfMemoryException.
Why does the GC not call the finalizer, and why does the memory not get freed?
Here is a short example that illustrates the problem. In the sample, there is nothing written to the console (except Unhandled Exception: OutOfMemoryException.)
class Buffer : IDisposable
{
public IntPtr buf { get; set; }
public Buffer()
{
buf = Marshal.AllocHGlobal(4 * 1024 * 1024);
}
~Buffer()
{
Console.WriteLine("Finalizer called");
Dispose(false);
}
public void Dispose()
{
Console.WriteLine("Dispose called");
Dispose(true);
GC.SuppressFinalize(this);
}
internal virtual void Dispose(bool disposing)
{
if (buf != IntPtr.Zero)
{
Console.WriteLine("Releasing memory");
Marshal.FreeHGlobal(buf);
buf = IntPtr.Zero;
}
}
}
class Program
{
static void Main(string[] args)
{
while(true)
{
Buffer b = new Buffer();
Thread.Sleep(20);
}
}
}
EDIT: Here is the .NET performance counters for my test program when it crashes:
You need to tell the garbage collector that your very small managed objects with a single IntPtr field have a high cost in terms of unmanaged memory. Currently, the garbage collector is blissfully unaware of the fact that each small managed object uses a large amount of unmanaged memory and has no reason to perform any collection.
You can use the GC.AddMemoryPressure when you allocate the unmanaged memory and GC.RemoveMemoryPressure when you free the unmanaged memory.
Garbage collection occurs when one of the following conditions is true:
The system has low physical memory.
The memory that is used by allocated objects on the managed heap
surpasses an acceptable threshold. This threshold is continuously
adjusted as the process runs.
The GC.Collect method is called. In almost all cases, you do not
have to call this method, because the garbage collector runs
continuously. This method is primarily used for unique situations
and testing.
Also the garbage collector tracks memory only on the managed heap, so for this program, the only condition can trigger GC is the first one.
I compiled the program, if the target CPU is x86, it will through out of memory exception when the private bytes of the process reaches about 2G. When I run the program, I noticed the private bytes increase quickly, but the working set increase very slowly, and also the system physical memory usage increase very slowly.
As private bytes and working set, this post explains:
Private Bytes refer to the amount of memory that the process executable has asked for - not necessarily the amount it is actually using. They are "private" because they (usually) exclude memory-mapped files (i.e. shared DLLs). But - here's the catch - they don't necessarily exclude memory allocated by those files. There is no way to tell whether a change in private bytes was due to the executable itself, or due to a linked library. Private bytes are also not exclusively physical memory; they can be paged to disk or in the standby page list (i.e. no longer in use, but not paged yet either).
Working Set refers to the total physical memory (RAM) used by the process. However, unlike private bytes, this also includes memory-mapped files and various other resources, so it's an even less accurate measurement than the private bytes. This is the same value that gets reported in Task Manager's "Mem Usage" and has been the source of endless amounts of confusion in recent years. Memory in the Working Set is "physical" in the sense that it can be addressed without a page fault; however, the standby page list is also still physically in memory but not reported in the Working Set, and this is why you might see the "Mem Usage" suddenly drop when you minimize an application.
Marshal.AllocHGlobal just increases the private bytes, but the working set is still small, it doesn't trigger GC either.
Please refer this: Fundamentals of Garbage Collection
IDisposable is declarative, The dispose method is only called when Garbage collection actually happens.
Yoy can force garbage collection to happen , for this You need to call
GC.Collect
http://msdn.microsoft.com/en-us/library/xe0c2357(v=vs.110).aspx
I would also recommend using preformance counters to see you app memory consumtion, and see if GC is already called. see here how to do it http://msdn.microsoft.com/en-us/library/x2tyfybc(v=vs.110).aspx
The following code is a simplified example of an issue I am seeing. This application consumes approx 4GB of memory before throwing an exception as the dictionary is too big.
class Program
{
static void Main(string[] args)
{
Program program = new Program();
while(true)
{
program.Method();
Console.ReadLine();
}
}
public void Method()
{
WasteOfMemory memory = new WasteOfMemory();
Task tast = new Task(memory.WasteMemory);
tast.Start();
}
}
public class WasteOfMemory
{
public void WasteMemory()
{
Dictionary<string, string> aMassiveList = new Dictionary<string, string>();
try
{
long i = 0;
while (true)
{
aMassiveList.Add(i.ToString(), "I am a line of text designed to waste space.... I am exceptionally useful........");
i++;
}
}
catch(Exception e)
{
Console.WriteLine("I have broken myself");
}
}
}
This is all as expected, although what we cannot currently work out is when this memory should be released from the CLR.
We have let the task complete and then simulated a memory overload situation, but the memory consumed by the dictionary is not released. As the OS is running out of memory, is it not putting pressure on the CLR to release the memory?
However and even more confusing, if we wait until the task has completed, then hit enter to run the task again the memory is released, so obviously the previous dictionary has been garbage collected (hasn't it?).
So, why is the memory not being released? And how can we get the CLR to release the memory?
Any explanations or solutions would be greatly appreciated.
EDIT: Following replies, particularly Beska's, it is obvious my description of the issue is not the the clearest, so I will try to clarify.
The code may not be the best example, sorry! It was a quick crude piece of code to try to replicate the issue.
The dictionary is used here to replicate the fact we have a large custom data object, which fills a large chunk of our memory and it is not then released after the task has completed.
In the example, the dictionary fills up to the limit of the dictionary and then throws an exception, it does NOT keep filling forever! This is well before our memory is full, and it does not cause an OutOfMemoryException. Hence the result is a large object in memory, and then the task completes.
At this point we would expect the dictionary to be out of scope, as both the task and the method 'Method' have completed. Hence, we would expect the dictionary to be garbage collected and the memory reclaimed. In reality, the memory is not freed until 'Method' is called again, creating a new WasteOfMemory instance and starting a new task.
Hopefully that will clarify the issue a bit
The garbage collector only frees locations in memory that are no longer in use that are objects which have no pointer pointing to them.
(1) your program runs infinitely without termination and
(2) you never change the pointer to your dictionary, so the GC has certainly no reason to touch the dictionary.
So for me your program is doing exactly what it is supposed to do.
Okay, I've been following this...I think there are a couple issues, some of which people have touched on, but I think not answering the real question (which, admittedly, took me a while to recognize, and I'm not sure I'm answering what you want even now.)
This is all as expected, although what we cannot currently work out is when this memory should be released from the CLR.
As others have said, while the task is running, the dictionary will not be released. It's being used. It gets bigger until you run out of memory. I'm pretty sure you understand this.
We have let the task complete and then simulated a memory overload situation, but the memory consumed by the dictionary is not released. As the OS is running out of memory, is it not putting pressure on the CLR to release the memory?
Here, I think, is the real question.
If I understand you correctly, you're saying you set this up to fill up memory. And then, after it crashes (but before you hit return to start a new task) you're trying other things outside of this program, such as running other programs in Windows to try to get the GC to collect the memory, right? Hoping that the OS would talk to the GC, and start pressuring it to do it's thing.
However and even more confusing, if we wait until the task has completed, then hit enter to run the task again the memory is released, so obviously the previous dictionary has been garbage collected (hasn't it?).
I think you answered your own question...it has not been necessarily been released until you hit return to start a new task. The new task needs memory, so it goes to the GC, and the GC happily collects the memory from the previous task, which has now ended (after throwing from full memory).
So, why is the memory not being released? And how can we get the CLR to release the memory?
I don't know that you can force the GC to release memory. Generally speaking, it does it when it wants (though some hacker types might know some slick way to force its hand.) Of course, .NET decides when to run the GC, and since nothing is happening while the program is just sitting there, it may well be deciding that it doesn't need to. As to whether the OS can pressure the GC to run, it seems from your tests the answer is "no". A bit counter-intuitive perhaps.
Is that what you were trying to get at?
The memory is not being released because the scope aMassiveList is never finished. When a function returns, it releases all non-referenced resources created inside it.
In your case, aMassiveList never leaves context. If you want your function never to return you have to find a way to 'process' your info and release it instead of storing all of them forever.
If you create a function that increasingly allocates resources and never release it you will end up consuming all the memory.
GC will only release unreferenced objects, so as the dictionary is being referenced by your program it can't be released by the GC
The way you've written the WasteMemory method, it will never exit (unless the variable "i" overflows, which won't happen this year) and BECAUSE IT WILL NEVER EXIT it will keep IN USE the reference to the internal Dictionary.
Daniel White is right, you should read about how GC works.
If the references are in use, GC will not collect the referenced memory. Otherwise, how would any program work?
I don't see what you expect the CLR/GC to do here. There's nothing to garbage-collect inside one run of your WasteMemory method.
However and even more confusing, if we wait until the task has completed, then hit enter to run the task again the memory is released, so obviously the previous dictionary has been garbage collected (hasn't it?).
When you press Enter, a new task is created and started. It's not the same task, it's a new task - a new object holding a reference to a new WasteOfMemory instance.
The old task will keep running and the memory it uses will NOT be collected because the old task keeps running in background and it keeps USING that memory.
I'm not sure why - and most importantly HOW - you observe the memory of the old task being released.
Change your method to be a using statement
Example:
Using (WateOfMemory memory = new WateOfMemory())
{
Task tast = new Task(memory.WasteMemory);
tast.Start();
}
And add disposible WateOfMemoryClass (by the way your constructor is WasteOfMemory)
#region Dispose
private IntPtr handle;
private Component component = new Component();
private bool disposed = false;
public WateOfMemory()
{
}
public WateOfMemory(IntPtr handle)
{
this.handle = handle;
}
public void Dispose()
{
Dispose(true);
GC.SuppressFinalize(this);
}
private void Dispose(bool disposing)
{
if(!this.disposed)
{
if(disposing)
{
component.Dispose();
}
CloseHandle(handle);
handle = IntPtr.Zero;
}
disposed = true;
}
[System.Runtime.InteropServices.DllImport("Kernel32")]
private extern static Boolean CloseHandle(IntPtr handle);
~WateOfMemory()
{
Dispose(false);
}
#endregion
I have a C# service which listens to a queue for XML messages, receives them, processing them using XSLTs and writing them in the database.
It does process about 60K messages a day of about 1Mb each. The memory when is idle is going down to 100MB which is really good.
However recently I have started processing messages of 12 MB in size. It does blow the memory up and even when is idle it has memory of about 500MB. Any suggestions why this might be a problem? I don't think there is a memory leak as it would have surfaced after processing so many (60K messages of 1MB).
That looks perfectly fine. Why do you think this is a problem?
The garbage collector will eventually release the unused memory, but this doesn't mean this is going to happen as soon as your service is idle.
Raymond Chen has written a good article explaining the basic idea of garbage collection:
Everybody thinks about garbage collection the wrong way
However - but this is pure speculation with the information given in your questions - there might be memory leaks related to extension methods in your XSLT. Extension methods may lead to problems in case you recompile the stylesheet every time a new XML document is transformed. The fix is simple: Once compiled, cache the stylesheet.
SIR! PUT DOWN TASK MANAGER AND WALK AWAY. Seriously. Memory management in .NET is not tuned for smallest footprint. It is tuned for efficiency. It will hold onto memory rather than release it back to the system. Resist the temptation to babysit your memory unless there is an actual problem (OOM exceptions, system issues).
What measure are you using for memory? There are lots of possible measures, and none of them really mean "used memory". With virtual memory, sharing of images, etc. things are not simple.
even when is idle it has memory of about 500MB
Most processes (and I think this includes .NET) will not release memory back to the OS once allocated to the process. But if it is not in use it will be paged from physical memory allowing other processes to make use of it.
started processing messages of 12 MB in size
If an XML document is expanded into an object model (e.g. XmlDocument) it needs a lot more memory (lots of links to other nodes). Either look at a different model (both XDocument and XPathDocument are lighter weight) or, better, process the XML with XmlReader and thus never have a fully expanded object model.
First of all, add something to your system to allow you to manually invoke garbage collection for the purpose of investigation. The CLR will not necessarily return memory to the O/S unless it is in short supply. You should use this mechanism to manually invoke garbage collection when you have the system in a quiescent state so that you can see if the memory drops back down. (You should invoke GC.Collect(2) twice to ensure objects with finalizers are actually collected and not just finalized.)
If the memory goes down to the quiescent level after a full GC then you do not have a problem: simply that .NET is not being proactive at deallocating memory.
If the memory does not go down, then you have some kind of a leak. As your messages are large, it means their text representations are likely ending up on the Large Object Heap (LOH). This particular heap is not compacted which means that it is easier to leak this memory than with the other heaps.
One thing to watch out for is string interning: if you are manually interning strings this can cause LOH fragmentation.
Very hard to say what may be the problem. I've had success using Ants Memory Profiler when investigating memory issues.
Contrary to the thought I don't think there is a memory leak; I would profile the memory.
I would check your event handlers. If you're not careful detaching those when you're done, it seems like it's easy to create object references that won't be collected by GC.
I don't know that it's the greatest practice (since the onus of beginning and cancelling event handling should fall to the subscriber), but I've gone the route of implementing IDisposable and creating my events with an explicit delegate field instance. On disposal, the field can be set to null, which effectively detaches all subscriptions.
something like this:
public class Publisher
: IDisposable
{
private EventHandler _somethingHappened;
public event EventHandler SomethingHappened
{
add { _somethingHappened += value; }
remove { _somethingHappened -= value; }
}
protected void OnSomethingHappened(object sender, EventArgs e)
{
if (_somethingHappened != null)
_somethingHappened(sender, e);
}
public void Dispose()
{
_somethingHappened = null;
}
}
Barring that (I don't know how much control you have over the publisher), you might need to be careful to detach unneeded handlers on your subscriber:
public class Subscriber
: IDisposable
{
private Publisher _publisher;
public Publisher Publisher
{
get { return _publisher; }
set {
// Detach from the old reference
DetachEvents(_publisher);
_publisher = value;
// Attach to the new
AttachEvents(_publisher);
}
}
private void DetachEvents(Publisher publisher)
{
if (publisher != null)
{
publisher.SomethingHappened -= new EventHandler(publisher_SomethingHappened);
}
}
private void AttachEvents(Publisher publisher)
{
if (publisher != null)
{
publisher.SomethingHappened += new EventHandler(publisher_SomethingHappened);
}
}
void publisher_SomethingHappened(object sender, EventArgs e)
{
// DO STUFF
}
public void Dispose()
{
// Detach from reference
DetachEvents(Publisher);
}
}