I'm noticing a lot of byte[] getting stuck in memory when profiling my program. I did some digging and found the majority of the instances created in some manner like this:
public byte[] CreateBytes(byte[] bytes)
{
using (var start = new MemoryStream())
{
using (var memStr = new MemoryStream(bytes))
{
//do stuff
return start.ToArray();
}
}
}
The returned byte[] is then passed to other methods and is used in creating another MemoryStream from within another using block:
using (var uncompressedStream = new MemoryStream(uncompressedData))
{
using (var compressedStream = new MemoryStream())
{
//Do some compression
}
}
myObject.Bytes = uncompressedData;
uncompressedData = null;
return myObject;
(uncompressedData is the value returned from CreateBytes()).
My question is, when does the byte[] get cleaned up? Do I specifically need to set it to null, and if so, where? After the second using block I no longer need it, but if I simply put uncompressedData = null; I'm not sure that's going to reclaim the memory.
I would've thought that the using statement in CreateBytes(byte[] bytes) would've disposed of the bytes, but since it's returning a reference does that postpone and/or forego the disposal?
EDIT: I added another line of code. Since I'm storing the uncompressedBtyes in another object, setting uncompressedData to null is pointless and the byte[] will live as long as myObject (or until myObject.Bytes is set to null), correct?
The byte[] gets cleaned up when two conditions are both satisfied:
There are no references to that memory block
The garbage collector decides to collect that memory block
The GC runs at non-deterministic times based on a variety of factors.
The byte array, just like any other managed object in memory, is eligible for garbage collection as soon as it is no longer accessible from any root reference. You can trust the GC to know when that's happened, and to intelligently schedule times to actually do the clean up of eligible objects. Try not to worry about when an object that's eligible for cleanup is actually cleaned up, chances are the GC knows better than you.
In addition to Erics answer:
When you are sure you do not more need that byte[], assign to it null.
This will not guarantee that memory will be reclaimed now, neither that it will be reclimed after, this is just a way to help GC to identify it like a subject for collection.
The using statement really has nothing to do with the byte array you are returning.
The MemoryStream is disposable, and disposal of that object is what the using statement is managing (note that disposal is not the same as garbage collection, but is typically a precursor thereto).
When you call ToArray(), you are creating something new that is not disposable; once it is out of scope, it will be cleaned up by the garbage collector at an indeterminate time (this is typically a well optimized process).
Related
What is an alternate/better way I can safely dispose of the variables purge and factory in the following code?
public void Run( string XmlFragment ) {
XmlNode xmlNode = null;
try
{
xmlNode = Common.ConstructXmlNodeFromString(XmlFragment, "Params");
var list = DataList();
foreach (var item in list)
{
var factory = new PurgerFactory(item);
IPurger purge = factory.Purger;
purge.Purge();
purge = null;
factory = null;
}
Common.PurgeEmail(SuccessEmail());
}
catch (Exception ex)
{
string errorMessage = $"Purge Error: {ex.Message}. {Environment.NewLine} Stack Trace: {ex.StackTrace}";
Common.PurgeEmail(FailEmail(errorMessage));
}
}
As I think you know, C# has a garbage collector. So for normal objects that don't access an unmanaged resource, just letting the garbage collector clean them is fine.
If you want to deterministically close a managed resource the primary paradigm is inheriting from IDisposable and a using statement. This will call the Dispose function upon exiting the code block of the using statement.
If you otherwise want to clean stuff up, but you don't care when it happens you can use ~(MyType). This is called when the GC does a GC cycle, this is called the Finalizer. I personally haven't encountered a use-case for this vs IDisposable. But if you just want to make sure say a file, or otherwise is deleted when this email object is Garbage Collected, then it might be a good use case for you.
Your code doesn't indicate whether purge or factory implement IDisposable. If they do, you would call
purge.Dispose();
factory.Dispose();
or use using which ensures that Dispose is called even if the method throws an exception:
using(var factory = new PurgerFactory(item))
{
// do stuff
} // factory is disposed.
What we don't need to do is set variables to null. An object gets garbage collected sometime after there are no longer any references to it. It's true that setting factory = null removes that reference to the object, but the same thing happens anyway when the variable goes out of scope. So it's redundant.
There's really no need to try to force that to happen sooner. It's not like the object will get garbage collected the instant there are no more references to it, so we don't worry so much about that.
In this case, the variables are both declared within the foreach loop, so the references go out of scope immediately after each iteration of the loop. In other words, they already go out of scope right at the point when you're setting them to null.
That's part of what's awesome about .NET. Imagine if we had to set every reference variable to null. We'd need try/catch everywhere to make sure it happened. It would be as if every single object was disposable. What a nightmare. Instead it's handled for us in most normal scenarios so that we don't have to think about it or clutter our code with it.
I have found the following construct in some open-source code:
var mstream = new MemoryStream();
// ... write some data to mstream
mstream.Close();
byte[] b = mstream.GetBuffer();
I thought this code would have "unexpected" behavior and maybe throw an exception, since the call to Close should effectively be a call to Dispose according to the MSDN documentation.
However, as far as I have been able to tell from experimenting, the call to GetBuffer() always succeeds and returns a valid result, even if I Thread.Sleep for 20 seconds or enforce garbage collection via GC.Collect().
Should the call to GetBuffer() succeed even after Close/Dispose? In that case, why is not the underlying buffer released in the MemoryStream disposal?
It doesn't need to. The buffer is managed memory, so normal garbage collection will deal with it, without needing to be included in the disposal.
It's useful to be able to get the bytes of the memory stream, even after the stream has been closed (which may have happened automatically after the steam was passed to a method that writes something to a stream and then closes said stream). And for that to work, the object needs to hold onto the buffer along with a record of how much had been written to it.
In considering the second point, it actually makes more sense in a lot of cases to call ToArray() (which as said, requires the in-memory store that GetBuffer() returns to still be alive) after you've closed the stream, because having closed the stream guarantees that any further attempt to write to the stream will fail. Hence if you have a bug where you obtain the array too early, it will throw an exception rather than just give you incorrect data. (Obviously if you explicitly want to get the current array part-way through the stream operations, that's another matter). It also guarantees that all streams are fully flushed rather than having part of their data in a temporary buffer (MemoryStream isn't buffered because MemoryStream essentially is a buffer, but you may have been using it with chained streams or writers that had their own separate buffer).
Technically there's nothing to dispose in MemoryStream. Literally nothing, it doesn't have operating system handles, unmanaged resources, nothing. it is just a wrapper around byte[]. All what you can do is set the buffer(internal array) to null, that BCL team hasn't done for some reason.
As #mike noted in comments BCL team wanted GetBuffer and ToArray to work, even after disposed, though we're not sure why?. Reference source.
Here's how Dispose implemented.
protected override void Dispose(bool disposing)
{
try {
if (disposing) {
_isOpen = false;
_writable = false;
_expandable = false;
// Don't set buffer to null - allow GetBuffer & ToArray to work.
#if FEATURE_ASYNC_IO
_lastReadTask = null;
#endif
}
}
finally {
// Call base.Close() to cleanup async IO resources
base.Dispose(disposing);
}
}
and GetBuffer is below
public virtual byte[] GetBuffer()
{
if (!this._exposable)
{
throw new UnauthorizedAccessException(Environment.GetResourceString("UnauthorizedAccess_MemStreamBuffer"));
}
return this._buffer;
}
As you can see in Dispose _buffer is untouched, and in GetBuffer no disposed checks.
Since the GC is non-deterministic you cannot force it to immediately dispose the MemoryStream, thus the instance will not be marked as disposed immediately, instead it will just be marked for disposal. This means that for some time, until it is really disposed you can use some of its functions. Since it keeps a strong reference to its buffer you can get it, here is what the GetBuffer method looks like:
public virtual byte[] GetBuffer()
{
if (!this._exposable)
{
throw new UnauthorizedAccessException(Environment.GetResourceString("UnauthorizedAccess_MemStreamBuffer"));
}
return this._buffer;
}
Unlike most interface methods, the IDisposable.Dispose does not promise to do anything. Instead, it provides a standard means by which the owner of an object can let that object know that its services are no longer required, in case the object might need to make use of that information. If an object has asked outside entities to do something on its behalf, and has promised those outside entities that it will let them know when their services are no longer required, its Dispose method can relay the notification to those entities.
If an object has a method which can only be performed while the object has outside entities acting on its behalf, an attempt to call that method after those entities have been dismissed should throw an ObjectDisposedException rather than failing in some other way. Further, if there is a method which cannot possibly be useful after the entity is dismissed, it should often throw ObjectDisposedException even if a particular didn't actually need to use the entity. On the other hand, if a particular call would have a sensible meaning after an object has dismissed all entities that were acting on its behalf, there's no particular reason why such a call shouldn't be allowed to succeed.
I would view the ObjectDisposedException much like I view the collection-modified InvalidOperationException of IEnumerator<T>.MoveNext(): if some condition (either Dispose, or modification of a collection, respectively) would prevent a method from behaving "normally", the method is allowed to throw the indicated exception, and is not allowed to behave in some other erroneous fashion. On the other hand, if the method is capable of achieving its objectives without difficulty, and if doing so would make sense, such behavior should be considered just as acceptable as would be throwing an exception. In general, objects are not required to operate under such adverse conditions, but sometimes it may be helpful for them to do so [e.g. enumeration of a ConcurrentDictionary will not be invalidated by changes to the collection, since such invalidation would make concurrent enumeration useless].
My application keeps growing in size everytime I open and close a certain Form. I used dotTrace and came up with an ever growing list of object of type System.Drawing.Internal.GPStream which are creating byte arrays continuously without every disposing them. I did some research and found out the .net does not support any means of closing such memory streams. the following
foreach (Type t in Assembly.GetExecutingAssembly().GetTypes())
{
if (t.IsClass && t.BaseType.Name.ToLower() == "form")
{
//Assembly ass = Assembly.GetExecutingAssembly();
object obj = null;
obj = Activator.CreateInstance(t);
Form f = new Form();
f = (Form)obj;
if ((string)f.Tag != "DNI") // Do Not Import
{
DataRow dr = Formsdt.NewRow();
dr["Name"] = f.Name;
dr["Text"] = f.Text;
dr["Tag"] = f.Tag;
Formsdt.Rows.Add(dr);
}
}
}
The intent of this code is to loop over all Forms and retrieve the controls in order to set access rights to users, activating and deactivating controls as needed programmatically.
Any insight is appreciated.
Note that
While( i am opening and closing the form)
==> Memory Allocation keeps on increasing
Somethings not being disposed of properly. Do you have warnings that say X class is iDisposable and you're not disposing it? Look into using 'using' blocks.
See this :
"A FileStream involves unmanaged resources which could actually be
immediately freed upon calling Dispose. A MemoryStream, on the other
hand, stores a managed byte array in its _buffer variable, which is
not freed at disposal time. In fact, the _buffer is not even nulled in
the MemoryStream's Dispose method, which is a SHAMEFUL BUG IMO because
nulling the reference could make the memory eligible for GC right at
disposal time. Instead, a lingering (but disposed) MemoryStream
reference still holds onto memory. Therefore, once you dispose it, you
should also null it if it's still in scope." – Triynko Oct 25 '10 at
20:46
Is a memory leak created if a MemoryStream in .NET is not closed?
Creating Forms in a loop will cause problems, depending the code in it's constructor it can fire events that will choke the message loop. Just to check, try adding an application.doevents in the loop and watch if memory gets released.
Maybe you will need to refactor your classes in order to determine access on a property that is outside a form. Something like:
Class MyObject
Public my_form as Form
Public Tag as string
end class
So you don't need to instantiate a Form.
Regards,
MarianoC.
I've been tracking massive memory leeking in my application and it seems the issue is the MemoryStream class. Whenever I use one, either with the 'using' key word or explicit close/dispose, the memory will never be collected by the garbage collector. What is wrong here?
byte[] bData = System.IO.File.ReadAllBytes( "F:\\application_exit_bw.png" );
using( System.IO.MemoryStream hMemoryStreamOutput = new System.IO.MemoryStream())
{
for ( int i = 0; i < 10000; i++ ) hMemoryStreamOutput.Write( bData, 0, bData.Length );
}
Thread.Sleep(Timeout.Infinite);
With explicit close/dipose the behaviour stays the same. Memory is occupied and will stay that way until I close my application, or, the application filled all of the system memory. Help?
There is nothing wrong with the MemoryStream class or the usage within your sample code. The GC in .Net doesn't clean up memory immediately after it's no longer. Instead it reclaims it when free space in the heap reaches a certain threshold or it's explicitly invoked with a GC.Collect call.
In this scenario the only way the memory would be freed is if a GC occurred immediately after the using statement and before the Thread.Sleep call. This is fairly unlikely to happen and hence if you profiled the program it would have the appearance of a memory leak when it's not actually leaking
This is a symptom of a non-deterministic GC. The GC does not make any guarantees at all about when memory will be freed. This is normal, expected and wanted behavior.
Try calling GC.Collect() to see if this fixes your problem. Also, you need to run in release mode because in debug mode the JIT extends the lifetime of local variables to the end of the method even if they are not used after some point.
Another side of the problem is what you are using to determine "memory leak". There are many different ways to measure "free' memory and depending on it you may get tottaly different results.
Memory usage show in Task manager - unlikely to go down even when all memory is considered "free" by CLR GC due to the way GC uses memory.
GC memory performance counters (and properties) - these ones actually will show GC's view on memory. You want to use them to detect managed memory leaks.
There is one more thing with MemoryStream (and any other large 86K+) allocations - they use Large Objects Heap that will be only collected on full GC, to trigger it you may need to run GC.Collect twice. In normal flow of an application it will happen rare enough, so you may not see this memory freed before application shutdown. To diagnose - check GC collection performance counter (number of GC).
And one more: if you are trying to solve memory leak because you are getting "out of memory" exception it may be caused by address space fragmentation (normally only for 32-bit processes). If it is the case - consider creating your own memory stream that does not allocate memory in single chunk and then have to copy it when growing the stream. Or at least try to preallocate space in the stream.
I have used this method for batch processing
static byte[] buffer;
public static object Read(XmlDocument xmlDocument)
{
if (buffer == null)
{
buffer = new byte[1024 * 1024 * 512];
}
if (xmlDocument != null)
{
using (MemoryStream ms = new MemoryStream(buffer))
{
xmlDocument.Save(ms);
ms.Flush();
ms.Seek(0, SeekOrigin.Begin);
object result = ReadFromStream(ms);
ms.Close();
return result;
}
}
else
{
return null;
}
}
Calling GC.Collect() is not a good practise and shouldn't be used as a solution.
You can try and see if that changes anything but do not rely on intentional GC.Collect() call...
The question is self explanatory :
Foo objfoo;
try
{
objfoo = new Foo();
..........
.........
}
catch
{
}
finally
{
objfoo = null;
}
Is it necessary to free objects like this ?
Note: Setting a local / field to null is not freeing the value. It is instead removing a reference to the value which may or may not make it elligable for collection during the next GC cyle.
To answer the question, no it is not necessary. The JIT`er will calculate the last time a local is used and will essentially remove the local as one of the object's GC roots at that time. Nulling the local out will not speed up this process.
Raymond Chen did an excellent article on this very subject
http://blogs.msdn.com/b/oldnewthing/archive/2010/08/10/10048149.aspx
You dont need to as objFoo will be eligible for garbage collection (as per your code) once you leave the try block. objfoo =null just sets the pointer to null, if any other code had a pointer to the object then it wont be garbage collected
Assigning null to a variable will not delete the object it points to.
You don't need to worry about memory cleanup. C# will do it automatically for you, if you're using .NET.
In C#, usually not, because your objects will be freed by the Garbage Collector. Setting it to null later might even extend its lifetime because (depending how the compiler optimizes your code) it is still referenced at that point.
An exception is if the object implements IDisposable, which should be disposed using its Dispose() method. But you usually wrap those in using-blocks.
No , its not necessary , let the CLR do its job.
Unless the objects are accessing unmanaged resources (files, network shares, database connections ...) there is no need to do this, since they will be marked for garbage collection once they are out of scope.
If they are using resources (the StreamReader class for example) it's necessary to dispose of the objects yourself, but that's best accomplished by using a using statement.
Automatic Garbage Collection means you do not need to free memory resources, i.e. objects created with new.
You may need to free other resources you have allocated, eg locking mutexes and maybe context specific to your own project, for which you have the finally block.
There' typically no need to do this. If your function is about to exit then your object will automatically become eligible for garbage collection.
However, if you've got a really long function or it's going to do some blocking operation after the finally block then by setting the variable to null you make it eligible for garbage collection immediately. This might be a good idea if Foo manages a large amount of data, for example.