Develop Reference

Inconsistent(?) behavior of garbage collector and (almost) out of memory issues

Some background:
We are running some pipelines on a buildserver and it consumes way to much memory. The pipeline does some DB imports and it builds up memory over time x times greater than the total size of an exported DB. For the import Entity Framework (core) is used (in order to be able to reuse entity definitions used in other parts of the application).
Situtation:
We are looking into where memory consumption can be reduced. Hence I was using the memory profiler.
I've noticed that sometimes the garbage collector does seem to free up memory after process X was done, and before process Y was started.
This is as expected. The 4GB memory build up is OK(ish), as long as it is released. The code that caused this consumption is running in its own Scope (speaking about dependency injection) and the DbContexts (and other things) used are registered as Scoped. Hence we have these ScopeWorkers.
await _scopeWorker.DoWork<MyProcessX>(_ => _.Import(cancellationToken));
// In some test, memory got freed up in between, but in some other test, memory never seemed to have dropped
await _scopeWorker.DoWork<MyProcessY>(_ => _.Import(cancellationToken));
But in some other test, this drop in memory was never seen.
The red arrow indicates approximately the same moment in time, after MyProcessX.Import, and a significant drop (of 4GBs) was never seen.
Of course I do not know whether the GC spread out the cleaning of this memory over a couple dozen collection moments, instead of 3, as seen in the first screenshot.
Questions
Is it possible to wait for the garbage collector to have collected basically all memory used by MyProcessX.Import, before continueing with MyProcessY.Import?
Should the garbage collector behave consistently? In other words, should I see the same memory consumption graph over time when the processes is repeated and is doing the exact same operations (so same data, as the data comes from a static source)
If the garbage collector is inconsistent in its behavior, how to make good use of the memory profiling feature in Visual Studio to spot opportunities of lowering memory?
EDIT
Yes the memory pressure on the system changes everything, as Evk pointed out. After reserving almost all physical memory on the system (31GB/32GB) and continuing the process which I was attempting to optimize memory usage I could see a definite drop in memory used. I could repeat this, as shown in the image there are actually 2 drops in memory.

Garbage collector uses the following conditions to decide whether it should start collection:
The system has low physical memory. The memory size is detected by
either the low memory notification from the operating system or low
memory as indicated by the host.
The memory that's used by allocated objects on the managed heap
surpasses an acceptable threshold. This threshold is continuously
adjusted as the process runs.
The GC.Collect method is called. In almost all cases, you don't have
to call this method because the garbage collector runs continuously.
This method is primarily used for unique situations and testing.
The first point means it depends on all processes running on current machine, not only on your process. For the same reason you don't know when GC will start, so you can't wait for that to happen.
For that same reason it cannot behave consistently in way you describe, in relation to your process. Your process may do the same thing, but OS as a whole is unlikely to ever do the same things during your process run. In one test run there were enough free memory over whole system, and in another it was not.
What you can do is force GC to run via GC.Collect (and overloads). However that's rarely a good idea.
Main thing you should ask yourself is - does high memory consumption bring any problems? Because by itself it's not a problem (assuming no memory leaks) - you have RAM to be used, not to just stay "free". If there is enough memory currently - GC might rightfully decide to not waste time on garbage collection and do that later when necessary.

Determine what GC.Collect() is Collecting

I have a console program written in VB.NET (.NET 4.5.2) that acts as a service. A continuous loop runs, and then waits for a message on an MSMQ queue, and processes the message. Somehow this program has a substantial memory leak. I have gone through all the code and done everything I could to use Using statements, but yet the problem persists. The more times through the loop, the higher the memory used by the program, and this memory is never reclaimed by the garbage collector.
I ended up putting a GC.Collect() at the bottom of my loop, and was able to free up most of the memory. However, I realize that this is bad practice and could cause issues. Just wondering if there is a way to inspect what variables the GC.Collect() is getting rid of, so I can find the root of the problem?
Do While (True)
' Code to wait for message on a queue
' Code to process message (includes calls to class library)
GC.Collect()
Loop

If the garbage collector is freeing memory when you manually invoke it, you are not leaking memory. A leak in a managed memory environment is memory that the GC can't free because it is referenced somewhere on the object graph.
It could be that they way your code uses objects instances that are being promoted to gen1, gen2 or the large object heap. Instances in these generations are collected less frequently then gen0. The windows resource monitor includes a number of performance counters that can be used to profile the behavior of the managed heap. I would guess that you might have objects being promoted into gen2. Tracking the "Gen 1 Promoted Bytes/Sec" counter would give you insight as to whether this is what's happening.
In a managed memory environment the GC runs when there is memory pressure, not when object instances are no longer needed, so the mere presence of increased memory use is not necessarily the sign of a leak.
If you take the Collect out does memory usage always increase (say over a number of minutes or messages being processed form the queue), or does it go up and down somewhat like a sin wave? If it's the latter just let the GC do its thing, you don't have a leak.
Visual studio has a number of memory analysis features https://msdn.microsoft.com/en-us/library/dn342825.aspx
The SOS managed debugger extension is a very powerful tool for sifting through the managed heap, though it is not for the faint of heart. https://learn.microsoft.com/en-us/dotnet/framework/tools/sos-dll-sos-debugging-extension

How can I tell if I have a memory leak?

I am using ANTS memory profiler and am somewhat baffled at the moment. If I load my site up and hook ANTS up to the process I can see the Private Bytes around 90mb, I then run the same routine a number of time with the following results:
109mb
120mb
125mb
126mb
123mb
126mb
and it basically stays around 126mb for each try after. My understanding is that if I had a memory leak then it would keep going up and not settle but what I don't understand is why it grows slowly until 126mb. Does .NET have an amount of memory it is allowed to reserve and it is just hitting that limit?
Simple question: Memory is growing up to a point then stopping. Is this normal for a .NET app?
EDIT: Just realised that I probably should have posted this at programmers.stackexchange.com - Apologies.

"Memory leak" is when memory you think should not be allocated is allocated.
It is not possible to simply look at amount of consumed memory and say "you have memory leak". I.e. what if your application collects logs in memory for 3 days - in this case memory consumption will grow, but it is not an indication of a leak. On other hand if your application simply prints a line a minute, but memory usage constantly grows it likley is a leak.

In my experience, if I see memory grow and plateau, it's generally from caching. As stated before, a memory leak is simply when something remains in memory that you feel should've been released. Using a profiler is a great way to determine leaks (vs using the task manager) because it generally will ensure a 2nd generation garbage collection has occurred, allowing you to see everything that's still in memory.
When I profile, I generally will execute the commands that I want to test a couple times to ensure all caching has occurred, and then I'll create a before and after snapshot and compare the delta of memory. If you're using a managed language like C#, it's not uncommon to have a delta of +/- 10KB. Repeat that process several times and if your delta is consistently positive, you more than likely have a leak (assuming you're not intentionally allocating more memory).

Memory not freed by Application until GC is explicitly called

I am launching many threads simultaneously, each one writing / reading data into/from a queue; Data is dequeued progressively while it is processed and stored to DB. For some reason the memory is not freed although the Queues are empty and I made sure all event subscription between the data reader and the data processor are unsubscribed at the end of the threads.The amount of squatted RAM is Exactly the amount of the data that is read by binary readers and put into queues.
In order to isolate the problem, I have bypassed the Processing and the DB Storing step.
Why would be the RAM still squatted long after all threads are finished, until I explicitly call GC.Collect() or terminate the program? I manually nullified the Queue after it is emptied, and also nullified the Binary Reader that read the data. I thought that would be enough for the GC to wake up and do its housekeeping at least after a few minutes.
EDIT :
The (reformulated after deletion) QUESTION :
In short, I was always told that the default GC behaviour managed the memory properly and that I shall almost never call the GC explicitly and let the framework do the job.
I would like to know why in this case memory usage drops down only when an explicit Call is made to GC.Collect
EDIT : Without GC Collect
With GC Collect (Called on a regular basis)

There are three conditions when a Garbage collection might occur (see MSDN):
1.) The system has low physical memory.
2.) The memory that is used by allocated objects on the managed heap
surpasses an acceptable threshold. This means that a threshold of
acceptable memory usage has been exceeded on the managed heap. This
threshold is continuously adjusted as the process runs.
3.) The GC.Collect method is called. In almost all cases, you do not have
to call this method, because the garbage collector runs continuously.
This method is primarily used for unique situations and testing.
From your description it sounds like the framework decided neither 1.) nor 2.) are the case hence it will only collect when you call GC.Collect()

Based on the screenshot, it appears that your memory usage never goes over 50% during the entire run of the application. It's much faster for the .NET framework to leave that memory alone than to stop your application (or at least take up a lot of CPU time in checking for liveness) in order to collect the garbage. If you cut your machine's RAM to 2 GB or so, I'm sure that the garbage collector would step up and keep memory usage within the hardware limits.

It seems you are leaking objects you should use windbg to create a dump then use sos.dll to track root for your objects.
You can follow this explanation on how to track "roots" for your objects and see what is causing these leaks.

How to control the memory size of continuously running windows service?

I have created a windows service which is continuously polling a database. For this purpose i have a timer in place. Ever time i am querying a database table i open a connection and close it immediately after my work is done. Right now i am doing this every 20 seconds for testing purpose, but later this time might increase to 5 - 10 minutes.
What happens is every time the database table is polled there is an increase of 10-12 KB in the size of the memory of the service running. This i can see in the task manager. Is there any way to control this.

The garbage collector should eventually kick in and free up some memory. It isn't based on time, but rather memory pressure. So you should be able to simulate longer durations by just increasing the frequency of your polling. If the GC doesn't reclaim any memory then you've got a leak somewhere. Also don't forget that database connections typically use connection pooling which means even though you're done with the connection, a pool of active connections is still waiting around.
But it's very normal not to see memory reclaimed instantly. Resist the urge to use GC.Collect unless its just for debugging. It could impact the efficiency of the garbage collector.

The memory of a .NET application is managed by the CLR and it is not recommended to interfere with it. I would instead advice you to profile your service and try to understand why it is leaking memory.
What do you do with the results of the query? Do you store them in memory (some static objects)?
Make sure you use using blocks around disposable resources such as SqlConnection and SqlCommand.
The garbage collector should take care of freeing resources.

Check if you have a memory leak. I use Windbg for this, as explained in CLR Memory Leak:
!dumpheap -stat to see the statistics of each type
!dumpheap -type <LeakedTypeName> to see the instances of the supposed leaked type
!gcroot <AddressOfObject> to track the reference that pins the allocation (ie. the cause of the leak).