Related
Uses of "using" in C# has a nice explanation of the utilities of the using feature.
.Net has its garbage collector. How does it handle the lack of a dipose()?
Specifically for DB connections, statements and resultsets, is it required a using() for each of them? What happens if they are left behind with no using(), dispose() and neither close()?
Update: the context is web applications, therefore there may be thousands of simultaneous users, each with his own connection/stmt/rs and the app will never be closed.
Since using is a shorthand for calling Dispose, you could imitate it with try/finally. So the real question is what's the consequence of not calling Dispose at all.
Although C# has garbage collection which would eventually release resources most of the time, you want the release of critical resources to happen as soon as you are done with them. If you use using or the equivalent try/finally, the resources would be released quickly. If you let the garbage collector to release the resources for you, your program may be starved of resources while they are "in custody" of GC (i.e. your program is no longer using them, but GC has not released them yet). Moreover, since GC offers no hard guarantee of running finalizers, some resources may not get released explicitly until your program ends, which may cause resource starvation of other processes.
You don't know when .net's garbage collector is called and run, so it's allowing you to do it yourself when you don't need it. So, when your code gets out of using() it's dispose object used in using() instead of waiting for GC to run on its own schedule.
If you don't use with DB connection, then GC will dispose it on its own way based on criteria of algorithm it's implementing. It might get too late(in terms of computer clock) to sweep it.
Garbage collector is a background thread which doesn't run every millisecond. It has specific schedule and its own algorithm which tends it to work on a specific time. E.g., some GC algorithms check for objects having no references then they sweep those objects when GC runs.
Specifically for DB connections, statements and resultsets, is it
required a using() for each of them? What happens if they are left
behind with no using(), dispose() and neither close()?
Actually the worst consequence of a memory leak is holding some memory until you restart the PC. However in this case probably the worst consequence is leaking memory until you restart the application.
If memory growth increases up to where GC cannot clean any longer, in fact if Gen 2 of Small Object Heap is overflow (Large object heap also can overflow), it will throw out of memory exception and close the application.
.Net has its garbage collector. How does it handle the lack of a
dipose()?
All the standard database connection related classes have implement Dispose and Finalize methods properly. Generally there are unmanaged resources in those classes. Unmanaged resources are the resouces (eg: file handlers, database connection handlers and etc) which could cause worse memory leaks that may hold memory until you restart the PC. However, that's where GC's finalization comes handy. If you don't call the Dispose for such Disposable object, garbage collector will execute the Finalize method(if there is a ~destructor) and clear unmanaged resources.
That's the reason why it is required to implement IDispose Pattern properly Dispose and Finalization as required. Finalization is required only if it has Unamanged resources.
The most likely consequence of failing to promptly Dispose database objects is that the program will ask a database server to open a database connections on its behalf with a promise that it will tell the server when they are no longer needed (i.e. close them), but may leave the connections open for quite awhile after they're no longer needed. Such behavior may increase the number of connections the database server will need to keep open simultaneously. Depending upon the server, there may be no consequences, or the extra connections may impair performance, or they may cause some connection requests to get needlessly denied.
Although .NET will try to ensure that database servers will get notified when database objects are abandoned, even if Dispose is not called, the code which uses database objects will generally know when it will no longer need them, long before .NET can determine that they're abandoned. Note also that while some .NET database-related libraries may keep connections open for a little while after a Dispose (so that if code needs the database again it can resume using the earlier connection) such libraries may use timers to limit how long connections are maintained in expectation of further use, rather than depend upon the garbage-collector (which might go a very long time without noticing that an object has been abandoned).
With resource leaks I mean Streams, StreamWriter (I suppose they are consuming File descriptors), Handles (GDI or user also Graphics fonts). Shortly all Closable objects can count as a resource!
If there are some resource leaks in the application. Say some InputStreams are not getting closed, are they potential memory leaks too because Garbage Collector is not going to remove them from memory?
Another question: Are resource leaks affecting the performance?
If there are some resource leaks in the application. Say some InputStreams are not getting closed, are they potential memory leaks too because Garbage Collector is not going to remove them from memory?
The GC will clean up resources whether they are closed or not. The only resources which cannot be cleaned up if not shutdown are threads.
When a Stream is discarded without closing, it is closed by the finalizer on a collection. This has two problems
exactly when or even if this happens in unpredictable meaning the file might not be flushed or a lock is retained on the file preventing it being deleted.
the finalizer is a single thread and closing a resources can take time. If you have enough of these the finalizer won't keep up and you will get an OutOfMemoryError because you have a large number of resources waiting to be cleaned.
It is always better to clean up resources when you have finished with them.
Another question: Are resource leaks affecting the performance?
They can, it depends on how much is leaked. If you don't have much idea you have to assume it a problem unless you are confident it is not. e.g. a program could leak one 24 bytes object once per day, or it could leak 100 MB per seconds. Not all leaks are the same.
It depends on what you call performance. I'll assume you're speaking of overall performance, meaning that memory consumption, speed, and the like are all important.
It depends, too, on the resource used. Some resources (e.g. file handles) are recovered when the process exits, so the leak will only be a problem when executing. Others (like server or database connections) could remain leaking even after your application execution. Others (like mutexes, etc.) should be released as soon as possible.
Now, the consequences depend on the resource. If the resource is a native object in the same process, then leaking it will probably leak the associated memory. If the resource is a mutex you locked but failed to unlock, then you are probably about to deadlock your application. If the resource is a connection, the server will keep that connection open even after you stopped using it. If the resource is a file, it could stop other applications (or even your own application) to access it again.
In the end, while some resources could be leaked, other shouldn't. As far as I am concerned, no resource should be leaked, ever, but YMMV.
So you should make an habit of always correctly releasing the resources you acquired (memory, files, connections, mutexes, etc.), no matter the perceived importance of that resource. Doing so will train you in the right coding patterns (and mindset).
RAII and Exception Safety are the keyword you're searching for if you want to explore the notions.
For C#, the Dispose pattern (IDisposable interface, and the finalizer) and the using keyword will be needed. Another solution is to use the finally class of a try/finally to free your resource, but this is difficult to maintain correctly.
In Java, you'll need Java 7 (IIRC), and use the AutoCloseable interface and the "try-with-resources" statement. As in C#, you can use the finally class of a try/finally to free your resource, with the same problems.
Memory leak, by definition is a memory that is useless, but still allocated in your proess space. Considering that CLR process on 32bit machine has approximately 1.2GB of possible memory, I would say it's extremely dangerous to have a memory leaks in your application.
Naturally, everything depends on how big, mission critical + other factors yuor application is. But, in any case, always try to avoid them, especially if you already know that they are exist and especially if you already know where they are.
EDIT
The resource leaks are the same story actually. Resource allocates the memory, so the leak of it creates a memory leak, by definition.
Hope this helps.
Yes, memory leaks means that the app needs to run the Garbage collector more often and is able yto recover less memory on each run. When the memory is exausted, the application will crash.
Files not getting closed will lead to a the app being unable to do anything related to files or sockets when the maximum number of open files is reached, which usually makes the application unusable.
Leak can occur when you keep a rooted reference to an unused object. The GC is unable to collect it as it is still reachable. Pay special attention to your static instances and event handlers attached to static events.
When you leave a disposable object undiposed, in most case it will defer the release of unmanaged resources and may cause bug (Stream not flushed, ...). When releasing memory, the garbage collector calls a Finalizer on objects containing unmanaged resources. This is more expensive than a staight call to Dispose that will lessen GC's job. It is unpredictable, the finalizer may be called really lately. So if you do not call Dispose it can lead to temporary resource starvation (no remaining file handle, ...).
So there is no memory leak for not calling Dispose on this Stream but you shouldn't rely on Finalization as it is costly and unpredictable. Moreover Dispose can do more than releasing resources, it can also properly clear a managed object (flush buffered stream, ...).
My application's performance deteriorate as it continues to run through the day.
I suspect it is garbage collector, how can I verify this?
Is there a way to find out which object/function is causing garbage collection overhead?
Is there a way to manually perform garbage collection programatically to clear memory of leakages?
Thanks,
edit
On one end of the application it receives call back from a unmanaged api to accept data, processes it and then send messages out of socket on the second end. From the second end it then gets back follow up data on the messages it sent out.
The application opens 5-6 sockets to send and receive data from the second end.
It constantly logs lots of data to windows file system on a separate thread.
My measurements include timestamping (queryperformance counter) just before I send data out and the timestampinga again when I receive the followup from another process back on the socket.
I noticed out of multiple sockets I open, the performance deterioration is happening on one socket connection only.
The processing between the timestamping and sending.receiving data over socket includes iterating through 2 arraylist that has no more than 5-6 objects and couple of callbacks.
The memory usage from Task MAnager window does not go up considerably. From 96MB to 100MB after 6-7 hours run.
Following are some observations from running perfmon.
"finanlization survivors" and "promoted finalization memory from Gen 0" gradually increase with time
"Gen 0 collections" going from 1819 at the start to 6000 after 4 hours.
"Gen 1 Collections" is 10%-12% of Gen 0 collection and "Gen 2 collection" is 1% or less. COnsidering Gen 0 collection numbers are cumulative, it is probably not abig concern.
GC handles" went up from 850ish to 4000.
It's far more likely that you have a memory leak, and invoking the GC manually will not help that: it can't dispose of objects if your code hasn't released them.
Edit
Since your GC handles are increasing, this page suggests that there are non-managed resources that are not getting freed. I've had this happen with bitmaps, for example, but you might have to tell us a lot more about your application to get a more specific suggestion.
Here's a thread that may give you some useful insight.
You can call GC.Collect() to force the garbage collection, but this will not fix memory leaks. Try a memory profiler like ANTS memory profiler to find memory leaks.
ANTS memory profiler
It may not necessarily be a memory problem. Use the Windows Performance Monitor (look in administrative tools) to monitor your app's CPU, Memory, GDI object count, handle count, and see if any of these appear to be climbing throughout the life of your app.
Often, you'll find that you're using something from System.Drawing and not calling Dispose() on it, causing a handle leak. I've found that handle leaks tend to eat performance much faster than memory leaks. And handle leaks cause no GC pressure, meaning you could be leaking handles like a sieve and the GC wouldn't ever know the difference.
So, long story short: measure, measure, measure. Then you'll know what to fix.
A memory intensive program that I wrote ran out of memory: threw an OutOfMemory exception. During attempts to reduce memory usage, I started calling GC.GetTotalMemory(true) (to write the total memory usage to debug file), which triggers a garbage collect.
For some reason, when calling this function I don't get an out of memory exception anymore. If I remove the calls again (keeping everything else the same), the exception gets thrown again. In my understanding, calls are automatically made to collect garbage when memory pressure increases, so I don't understand this behavior.
Can anyone explain why the out of memory exception is only thrown when there are no calls to GC.collect?
Update:
I'm using VS 2010, but I'm downtargeting the application to framework 3.5. I believe that defragmentation is indeed causing my problems.
I did some tests: When the exception is thrown, a call to GC.gettotalmemory tells me I am using ~800 * 10^6 bytes. However, task manager tells me the application is using 1700 mb. A rather large discrepancy. I'm now planning to allocate memory only once, and to never deallocate any large arrays but reusing them. Luckily, my program allows me to accomplish this without too much fuss.
I solved the problem by doing some smarter memory management. In particular by using a CustomList according to the suggestions on http://www.simple-talk.com/dotnet/.net-framework/the-dangers-of-the-large-object-heap/
Is your app running at full CPU? I'm pretty sure automatic garbage collection only occurs when the application is idle. Otherwise, you have to run a manual cycle.
I'm fairly sure that running out of memory does not force a garbage collection. That probably sounds incredibly unintuitive to you but I think this was done for a good reason. It prevents the program from entering a death-spiral where it constantly tries to find more space and getting all objects firmly lodged into gen #2. From which it is very hard to recover again.
The true argument you pass to GetTotalMemory() forces a full garbage collection. I would guess that this happens to free up enough space in the Large Object Heap to satisfy the memory allocation. This will of course work only once. If your program just keeps running, gobbling up memory beyond the 1.5 gigabytes or so that it has already consumed then OOM is just around the corner again. This time without any way to recover. Surviving an OOM requires drastic measures.
You'll need a good memory profiler to find out what's really going on. Unmanaged C++ in your project is always a fertile source of memory leaks. The unmanaged kind, always hard to trouble-shoot.
GC.Collect is merely a "suggestion" to free unused memory - it does not guarantee its release.
[Edit]
It appears that, while once true when I was learning the JVM years ago, this may not be the case in .NET anymore. The MSDN Library says that GC.Collect "Forces an immediate garbage collection of all generations." Good stuff (for me, anyway) about this here.
If you have unmanaged resources that occupy a lot of memory, the garbage collector won't really recognize that memory pressure. If you clean up those resources in finalizers, then forcing a collection will result in those unmanaged resources being freed, while if you don't force a collection, the garbage collector might not realize that it needs to be collecting.
If you are performing large unmanaged allocations, you can use GC.AddMemoryPressure to tell the GC that, so it can take it into account when deciding whether to run a collection or not.
I was browsing around the Microsoft Connect site and I am seeing bug reports where people are making the same claim you are. The claim being that an OutOfMemoryException is occuring which can be resolved by periodically calling GC.Collect. I saw one report where the lead engineer from the garbage collector team responded back and said a bug was fixed in .NET 4.0 that should resolve a fragmentation issue with the large object heap. That is why I asked what version you were using.
It is certainly possible that you have stumbled upon a bug in the garbage collector. As with all GC related issues this could be very version dependent.
My advice would be to:
make sure you have the latest patches and service pack
refactor the code so that it is not as memory intensive
reuse LOH objects as much as possible instead of creating new ones
continue using GC.Collect at strategic points if necessary as a workaround
We have a web service that uses up more and more private bytes until that application stops responding. The managed heap (mostly Gen2) will show some 200-250 MB, while private bytes shows over 1GB. What are possible causes of a memory leak outside of the managed heap?
I've already checked for the following:
Prolific dynamic assemblies (Xml serialization, regex, etc.)
Session state (turned off)
System.Policy.Evidence memory leak (SP1 installed)
Threading deadlock (no use of Join, only lock)
Use of SQLOLEDB (using SqlClient)
What other sources can I check for?
Make sure your app is complied in release mode. If you compile under debug mode, and deploy that, simply instantiating a class that has an event defined (event doesn't even need to be raised), will cause a small piece of memory to leak. Instantiating enough of these objects over a long enough period of time will cause all the memory to be used. I've seen web apps that would use up all the memory within a matter of hours, simply because a debug build was used. Compiling as a release build immediately and permanently fixed the problem.
I would recommend you view snapshots of the stack at various times, and see what's using up the memory. If your application is using Java, then jmap works extremely well - you just give it the PID of the java process.
If using something else, try Lambda Probe (http://www.lambdaprobe.org/d/index.htm). It doesn't show as much detail, but will at least show you memory use.
I had a bad memory leak in my JDBC code that ended up being traced to a change in the JDBC specification a few years ago that I missed (with respect to closing statements and such). It took a combination of Lamdba Probe and then jmap to localize the problem enough to fix it.
Cheers,
-R
Also look for:
COM Assemblies being loaded
DB Connections not being closed
Cache & State (Session, Application)
Try forcing the Garbage Collector (GC) to run (write a page that does it when it loads) or try the instrumentation, but that's a bit hit and miss in my experience. Another thing would be to keep it running and see if it runs out of memory.
What could be happening is that there is plenty of memory and Windows does not signal your app to clean up. This causes the app to look like its using more and more memory because it can, when in fact the system can reclaim the memory when it needs. SQL Server and Exchange do this a lot. The idea is why cause a unnecessary cleanup when there are plenty of resources.
Rob
Garbage collection does not run until a request for memory is denied due to lack of available memory. This can often make things look like a memory leak when one is not around.
Do you have any events and event handlers within the service? Services often have static variables, and if you are creating event handlers from the static instances, connected to a non-static instance object, the static will hold a reference to the instance forever, which will stop it from releasing.
Double check that trace is not enabled. I've seen instances of trace slowly consuming memory until the app reaches it's app pool limit.
For what it is worth, my issue was not with the service, but the with HttpClient that was calling it.
The client was not properly disposed, so it kept the connection open and the memory locked.
After disposing the client the service released the memory as expected.