I am launching many threads simultaneously, each one writing / reading data into/from a queue; Data is dequeued progressively while it is processed and stored to DB. For some reason the memory is not freed although the Queues are empty and I made sure all event subscription between the data reader and the data processor are unsubscribed at the end of the threads.The amount of squatted RAM is Exactly the amount of the data that is read by binary readers and put into queues.
In order to isolate the problem, I have bypassed the Processing and the DB Storing step.
Why would be the RAM still squatted long after all threads are finished, until I explicitly call GC.Collect() or terminate the program? I manually nullified the Queue after it is emptied, and also nullified the Binary Reader that read the data. I thought that would be enough for the GC to wake up and do its housekeeping at least after a few minutes.
EDIT :
The (reformulated after deletion) QUESTION :
In short, I was always told that the default GC behaviour managed the memory properly and that I shall almost never call the GC explicitly and let the framework do the job.
I would like to know why in this case memory usage drops down only when an explicit Call is made to GC.Collect
EDIT : Without GC Collect
With GC Collect (Called on a regular basis)
There are three conditions when a Garbage collection might occur (see MSDN):
1.) The system has low physical memory.
2.) The memory that is used by allocated objects on the managed heap
surpasses an acceptable threshold. This means that a threshold of
acceptable memory usage has been exceeded on the managed heap. This
threshold is continuously adjusted as the process runs.
3.) The GC.Collect method is called. In almost all cases, you do not have
to call this method, because the garbage collector runs continuously.
This method is primarily used for unique situations and testing.
From your description it sounds like the framework decided neither 1.) nor 2.) are the case hence it will only collect when you call GC.Collect()
Based on the screenshot, it appears that your memory usage never goes over 50% during the entire run of the application. It's much faster for the .NET framework to leave that memory alone than to stop your application (or at least take up a lot of CPU time in checking for liveness) in order to collect the garbage. If you cut your machine's RAM to 2 GB or so, I'm sure that the garbage collector would step up and keep memory usage within the hardware limits.
It seems you are leaking objects you should use windbg to create a dump then use sos.dll to track root for your objects.
You can follow this explanation on how to track "roots" for your objects and see what is causing these leaks.
Related
Some background:
We are running some pipelines on a buildserver and it consumes way to much memory. The pipeline does some DB imports and it builds up memory over time x times greater than the total size of an exported DB. For the import Entity Framework (core) is used (in order to be able to reuse entity definitions used in other parts of the application).
Situtation:
We are looking into where memory consumption can be reduced. Hence I was using the memory profiler.
I've noticed that sometimes the garbage collector does seem to free up memory after process X was done, and before process Y was started.
This is as expected. The 4GB memory build up is OK(ish), as long as it is released. The code that caused this consumption is running in its own Scope (speaking about dependency injection) and the DbContexts (and other things) used are registered as Scoped. Hence we have these ScopeWorkers.
await _scopeWorker.DoWork<MyProcessX>(_ => _.Import(cancellationToken));
// In some test, memory got freed up in between, but in some other test, memory never seemed to have dropped
await _scopeWorker.DoWork<MyProcessY>(_ => _.Import(cancellationToken));
But in some other test, this drop in memory was never seen.
The red arrow indicates approximately the same moment in time, after MyProcessX.Import, and a significant drop (of 4GBs) was never seen.
Of course I do not know whether the GC spread out the cleaning of this memory over a couple dozen collection moments, instead of 3, as seen in the first screenshot.
Questions
Is it possible to wait for the garbage collector to have collected basically all memory used by MyProcessX.Import, before continueing with MyProcessY.Import?
Should the garbage collector behave consistently? In other words, should I see the same memory consumption graph over time when the processes is repeated and is doing the exact same operations (so same data, as the data comes from a static source)
If the garbage collector is inconsistent in its behavior, how to make good use of the memory profiling feature in Visual Studio to spot opportunities of lowering memory?
EDIT
Yes the memory pressure on the system changes everything, as Evk pointed out. After reserving almost all physical memory on the system (31GB/32GB) and continuing the process which I was attempting to optimize memory usage I could see a definite drop in memory used. I could repeat this, as shown in the image there are actually 2 drops in memory.
Garbage collector uses the following conditions to decide whether it should start collection:
The system has low physical memory. The memory size is detected by
either the low memory notification from the operating system or low
memory as indicated by the host.
The memory that's used by allocated objects on the managed heap
surpasses an acceptable threshold. This threshold is continuously
adjusted as the process runs.
The GC.Collect method is called. In almost all cases, you don't have
to call this method because the garbage collector runs continuously.
This method is primarily used for unique situations and testing.
The first point means it depends on all processes running on current machine, not only on your process. For the same reason you don't know when GC will start, so you can't wait for that to happen.
For that same reason it cannot behave consistently in way you describe, in relation to your process. Your process may do the same thing, but OS as a whole is unlikely to ever do the same things during your process run. In one test run there were enough free memory over whole system, and in another it was not.
What you can do is force GC to run via GC.Collect (and overloads). However that's rarely a good idea.
Main thing you should ask yourself is - does high memory consumption bring any problems? Because by itself it's not a problem (assuming no memory leaks) - you have RAM to be used, not to just stay "free". If there is enough memory currently - GC might rightfully decide to not waste time on garbage collection and do that later when necessary.
I have a console program written in VB.NET (.NET 4.5.2) that acts as a service. A continuous loop runs, and then waits for a message on an MSMQ queue, and processes the message. Somehow this program has a substantial memory leak. I have gone through all the code and done everything I could to use Using statements, but yet the problem persists. The more times through the loop, the higher the memory used by the program, and this memory is never reclaimed by the garbage collector.
I ended up putting a GC.Collect() at the bottom of my loop, and was able to free up most of the memory. However, I realize that this is bad practice and could cause issues. Just wondering if there is a way to inspect what variables the GC.Collect() is getting rid of, so I can find the root of the problem?
Do While (True)
' Code to wait for message on a queue
' Code to process message (includes calls to class library)
GC.Collect()
Loop
If the garbage collector is freeing memory when you manually invoke it, you are not leaking memory. A leak in a managed memory environment is memory that the GC can't free because it is referenced somewhere on the object graph.
It could be that they way your code uses objects instances that are being promoted to gen1, gen2 or the large object heap. Instances in these generations are collected less frequently then gen0. The windows resource monitor includes a number of performance counters that can be used to profile the behavior of the managed heap. I would guess that you might have objects being promoted into gen2. Tracking the "Gen 1 Promoted Bytes/Sec" counter would give you insight as to whether this is what's happening.
In a managed memory environment the GC runs when there is memory pressure, not when object instances are no longer needed, so the mere presence of increased memory use is not necessarily the sign of a leak.
If you take the Collect out does memory usage always increase (say over a number of minutes or messages being processed form the queue), or does it go up and down somewhat like a sin wave? If it's the latter just let the GC do its thing, you don't have a leak.
Visual studio has a number of memory analysis features https://msdn.microsoft.com/en-us/library/dn342825.aspx
The SOS managed debugger extension is a very powerful tool for sifting through the managed heap, though it is not for the faint of heart. https://learn.microsoft.com/en-us/dotnet/framework/tools/sos-dll-sos-debugging-extension
I have a WCF service hosted in a console application. and I have a ChannelFactory to call WCF's operation Contracts.
Problem: whenever I call an operation that returns values, it seems that the value returned is cached somewhere by the service when it is serialized.
I am checking the service memory usage through the task manager under windows 7. When I call an operation that returns nothing, the memory is not increased, but when I call an operation that returns data, memory is increased and stays this way even after the data is returned to the client.
My guess is that this is a serialization caching issue?!?
It sounds more like garbage collector hasn't run yet and because of that the memory wasn't released. Moreover when hosting WCF service in console application the GC runs in Workstation mode which can be less effective in such case.
Memory will probably stay increased until the GC runs, which won't correspond to when data is returned to the client.
Have you tried adding a breakpoint or logging of some sort to your service method to make sure that the method is being called on each request? I don't think WCF does any caching on its own; at least, I've never had it do any in my applications that use it.
Edit:
Memory will remain in use until the garbage collector runs. If your process still has plenty of free space in its heap, then there really isn't a reason for the GC to run.
According to MSDN: http://msdn.microsoft.com/en-us/library/ee787088.aspx#conditions_for_a_garbage_collection
Garbage collection occurs when one of
the following conditions is true:
The system has low physical memory.
The memory that is used by allocated objects on the managed heap
surpasses an acceptable threshold.
This means that a threshold of
acceptable memory usage has been
exceeded on the managed heap. This
threshold is continuously adjusted as
the process runs.
The GC.Collect method is called. In almost all cases, you do not have
to call this method, because the
garbage collector runs continuously.
This method is primarily used for
unique situations and testing.
It is likely that you are allocating objects in the heap, but they are not being GC'd because the GC sees no reason to run (there is still open space in the heap generation, and no reason to spend time clearing it out).
However, if you can repeat your WCF call over and over and eventually get an Out of Memory exception, then that would indicate that you do indeed have a problem with references being held somewhere. In that case, I would use a memory profiler to determine what is being held onto, and by what.
Edit #2:
See also this thread: C# Thread not releasing memory
Use a tool like this to see what objects are being created/collected:
http://memprofiler.com/
I wouldn't worry too much if GC isn't being run after each call - this wouldn't be very efficient anyway - the garbage collector will run at a point when the .NET framework determines that objects are old enough to be collected. If memory starts to run low then this will happen more frequently.
My application's performance deteriorate as it continues to run through the day.
I suspect it is garbage collector, how can I verify this?
Is there a way to find out which object/function is causing garbage collection overhead?
Is there a way to manually perform garbage collection programatically to clear memory of leakages?
Thanks,
edit
On one end of the application it receives call back from a unmanaged api to accept data, processes it and then send messages out of socket on the second end. From the second end it then gets back follow up data on the messages it sent out.
The application opens 5-6 sockets to send and receive data from the second end.
It constantly logs lots of data to windows file system on a separate thread.
My measurements include timestamping (queryperformance counter) just before I send data out and the timestampinga again when I receive the followup from another process back on the socket.
I noticed out of multiple sockets I open, the performance deterioration is happening on one socket connection only.
The processing between the timestamping and sending.receiving data over socket includes iterating through 2 arraylist that has no more than 5-6 objects and couple of callbacks.
The memory usage from Task MAnager window does not go up considerably. From 96MB to 100MB after 6-7 hours run.
Following are some observations from running perfmon.
"finanlization survivors" and "promoted finalization memory from Gen 0" gradually increase with time
"Gen 0 collections" going from 1819 at the start to 6000 after 4 hours.
"Gen 1 Collections" is 10%-12% of Gen 0 collection and "Gen 2 collection" is 1% or less. COnsidering Gen 0 collection numbers are cumulative, it is probably not abig concern.
GC handles" went up from 850ish to 4000.
It's far more likely that you have a memory leak, and invoking the GC manually will not help that: it can't dispose of objects if your code hasn't released them.
Edit
Since your GC handles are increasing, this page suggests that there are non-managed resources that are not getting freed. I've had this happen with bitmaps, for example, but you might have to tell us a lot more about your application to get a more specific suggestion.
Here's a thread that may give you some useful insight.
You can call GC.Collect() to force the garbage collection, but this will not fix memory leaks. Try a memory profiler like ANTS memory profiler to find memory leaks.
ANTS memory profiler
It may not necessarily be a memory problem. Use the Windows Performance Monitor (look in administrative tools) to monitor your app's CPU, Memory, GDI object count, handle count, and see if any of these appear to be climbing throughout the life of your app.
Often, you'll find that you're using something from System.Drawing and not calling Dispose() on it, causing a handle leak. I've found that handle leaks tend to eat performance much faster than memory leaks. And handle leaks cause no GC pressure, meaning you could be leaking handles like a sieve and the GC wouldn't ever know the difference.
So, long story short: measure, measure, measure. Then you'll know what to fix.
I have a large 2GB file with 1.5 million listings to process. I am running a console app that performs some string manipulation then uploads each listing to the database.
I created a LINQ object and clear the object by assigning it to a new LinqObject() for each listing (loop).
When the object is complete, I add it to a list.
When the list reaches 100 objects, I submitAll on the entire list, clear the list, then repeat.
My memory usage continues to grow as the program runs. Is there anything I should be doing to keep memory usage down? I tried GC.collect. I think I want to use dispose..
Thanks in advance for looking.
It's normal for the memory usage of a program to increase when it's working. You should not try to force the garbage collector to reduce the memory usage to try to save resources, this will most likely waste resources instead.
Contrary to one's first reaction, high memory usage is not a performance problem as long as there are any free memory left at all. Having a lot of unused memory doesn't increase the performance a bit. If you try to reduce the memory usage only to keep it down, you are just wasting CPU time doing cleanup that is not needed.
If you are running out of free memory or if some other application needs it, the garbage collector will do the appropriate cleanup. In almost every situation the garbage collector will know much more about the current memory situatiuon than you can possibly anticipate when writing the code.
If you are using objects that implement the IDisposable interface, you should call the Dispose method to free unmanaged resources, but all other objects are handled by the garbage collector. Managed objects normally don't leak memory at all.
Do you need your memory usage to stay low? Absent an actual functional problem, high memory usage in and of itself is not an issue.
How large is the memory usage growing? It may be that .NET is just "settling" effectively.
It's not really clear exactly how you're doing this, but the general principle sounds okay. I suggest you take the database work out of the equation - just comment out whichever line would actually submit to the database. See how much memory that uses. Other than the StreamReader (or whatever) you shouldn't have anything else that needs disposing if you're not touching the database - just building batches of transformed objects and throwing them away.