Develop Reference

Thread pool - 10 phreads created?

I am creating an app that deals with huge number of data to be processed. I want to use threading in C# just to make it processes faster. Please see example code below.
private static void MyProcess(Object someData)
{
//Do some data processing
}
static void Main(string[] args)
{
for (int task = 1; task < 10; task++)
{
ThreadPool.QueueUserWorkItem(new WaitCallback(MyProcess), task);
}
}
Does this mean that a new thread will be created every loop passing the task to the "MyProcess" method (10 threads total)? Also, are the threads going to process concurrently?

The number of threads a threadpool will start depends on multiple factors, see The managed thread pool
Basically you are queing 10 worker items here which are likely to start threads immediatly.
The threads will most likly run concurrently, depending on the machine and number of processors.
If you start a large number of worker items, they will end up in a queue and start running as soon as a thread becomes available.

The calls will be scheduled on the thread pool. It does not guarantee that 10 threads will be created nor that all 10 tasks will be executed concurrently. The number of threads in the thread pool depends on the hardware and is chosen automatically to provide the best performance.
This articles contain good explanations of how it works:
https://owlcation.com/stem/C-ThreadPool-and-its-Task-Queue-Example
https://learn.microsoft.com/en-us/dotnet/api/system.threading.threadpool?redirectedfrom=MSDN&view=netframework-4.8
https://www.c-sharpcorner.com/article/thread-pool-in-net-core-and-c-sharp/
This Stackoverflow question explains the difference between ThreadPool and Thread:
Thread vs ThreadPool

Your method will be queued 9 times (you start at 1, not 0) for execution and will be executed when a thredpool thread will be available.

Effects of non-awaited Task

I have a Task which I do not await because I want it to continue its own logic in the background. Part of that logic is to delay 60 seconds and check back in to see if some minute work is to be done. The abbreviate code looks something like this:
public Dictionary<string, Task> taskQueue = new Dictionary<string, Task>();
// Entry point
public void DoMainWork(string workId, XmlDocument workInstructions){
// A work task (i.e. "workInstructions") is actually a plugin which might use its own tasks internally or any other logic it sees fit.
var workTask = Task.Factory.StartNew(() => {
// Main work code that interprets workInstructions
// .........
// .........
// etc.
}, TaskCreationOptions.LongRunning);
// Add the work task to the queue of currently running tasks
taskQueue.Add(workId, workTask);
// Delay a period of time and then see if we need to extend our timeout for doing main work code
this.QueueCheckinOnWorkTask(workId); // Note the non-awaited task
}
private async Task QueueCheckinOnWorkTask(string workId){
DateTime startTime = DateTime.Now;
// Delay 60 seconds
await Task.Delay(60 * 1000).ConfigureAwait(false);
// Find out how long Task.Delay delayed for.
TimeSpan duration = DateTime.Now - startTime; // THIS SOMETIMES DENOTES TIMES MUCH LARGER THAN EXPECTED, I.E. 80+ SECONDS VS. 60
if(!taskQueue.ContainsKey(workId)){
// Do something based on work being complete
}else{
// Work is not complete, inform outside source we're still working
QueueCheckinOnWorkTask(workId); // Note the non-awaited task
}
}
Keep in mind, this is example code just to show a extremely miniminal version of what is going on with my actual program.
My problem is that Task.Delay() is delaying for longer than the time specified. Something is blocking this from continuing in a reasonable timeframe.
Unfortunately I haven't been able to replicate the issue on my development machine and it only happens on the server every couple of days. Lastly, it seems related to the number of work tasks we have running at a time.
What would cause this to delay longer than expected? Additionally, how might one go about debugging this type of situation?
This is a follow up to my other question which did not receive an answer: await Task.Delay() delaying for longer that expected

Most often that happens because of thread pool saturation. You can clearly see its effect with this simple console application (I measure time the same way you are doing, doesn't matter in this case if we use stopwatch or not):
public class Program {
public static void Main() {
for (int j = 0; j < 10; j++)
for (int i = 1; i < 10; i++) {
TestDelay(i * 1000);
}
Console.ReadKey();
}
static async Task TestDelay(int expected) {
var startTime = DateTime.Now;
await Task.Delay(expected).ConfigureAwait(false);
var actual = (int) (DateTime.Now - startTime).TotalMilliseconds;
ThreadPool.GetAvailableThreads(out int aw, out _);
ThreadPool.GetMaxThreads(out int mw, out _);
Console.WriteLine("Thread: {3}, Total threads in pool: {4}, Expected: {0}, Actual: {1}, Diff: {2}", expected, actual, actual - expected, Thread.CurrentThread.ManagedThreadId, mw - aw);
Thread.Sleep(5000);
}
}
This program starts 100 tasks which await Task.Delay for 1-10 seconds, and then use Thread.Sleep for 5 seconds to simulate work on a thread on which continuation runs (this is thread pool thread). It will also output total number of threads in thread pool, so you will see how it increases over time.
If you run it you will see that in almost all cases (except first 8) - actual time after delay is much longer than expected, in some cases 5 times longer (you delayed for 3 seconds but 15 seconds has passed).
That's not because Task.Delay is so imprecise. The reason is continuation after await should be executed on a thread pool thread. Thread pool will not always give you a thread when you request. It can consider that instead of creating new thread - it's better to wait for one of the current busy threads to finish its work. It will wait for a certain time and if no thread became free - it will still create a new thread. If you request 10 thread pool threads at once and none is free, it will wait for Xms and create new one. Now you have 9 requests in queue. Now it will again wait for Xms and create another one. Now you have 8 in queue, and so on. This wait for a thread pool thread to become free is what causes increased delay in this console application (and most likely in your real program) - we keep thread pool threads busy with long Thread.Sleep, and thread pool is saturated.
Some parameters of heuristics used by thread pool are available for you to control. Most influential one is "minumum" number of threads in a pool. Thread pool is expected to always create new thread without delay until total number of threads in a pool reaches configurable "minimum". After that, if you request a thread, it might either still create new one or wait for existing to become free.
So the most straightforward way to remove this delay is to increase minimum number of threads in a pool. For example if you do this:
ThreadPool.GetMinThreads(out int wt, out int ct);
ThreadPool.SetMinThreads(100, ct); // increase min worker threads to 100
All tasks in the example above will complete at the expected time with no additional delay.
This is usually not recommended way to solve this problem though. It's better to avoid performing long running heavy operations on thread pool threads, because thread pool is a global resource and doing this affects your whole application. For example, if we remove Thread.Sleep(5000) in the example above - all tasks will delay for expected amount of time, because all what keeps thread pool thread busy now is Console.WriteLine statement which completes in no time, making this thread available for other work.
So to sum up: identify places where you perform heavy work on thread pool threads and avoid doing that (perform heavy work on separate, non-thread-pool threads instead). Alternatively, you might consider increasing minimum number of threads in a pool to a reasonable amount.

Where does the thread pool get new threads from when its total available worker threads has reached zero?

Just for fun, I wrote this code to simulate a deadlock. Then, I sat and watched it run patiently until the total number of available worker threads that the thread pool had went down to zero. I was curious to see what would happen. Would it throw an exception?
using System;
using System.Diagnostics;
using System.Threading;
namespace Deadlock
{
class Program
{
private static readonly object lockA = new object();
private static readonly object lockB = new object();
static void Main(string[] args)
{
int worker, io;
ThreadPool.GetAvailableThreads(out worker, out io);
Console.WriteLine($"Total number of thread pool threads: {worker}, {io}");
Console.WriteLine($"Total threads in my process: {Process.GetCurrentProcess().Threads.Count}");
Console.ReadKey();
try
{
for (int i = 0; i < 1000000; i++)
{
AutoResetEvent auto1 = new AutoResetEvent(false);
AutoResetEvent auto2 = new AutoResetEvent(false);
ThreadPool.QueueUserWorkItem(ThreadProc1, auto1);
ThreadPool.QueueUserWorkItem(ThreadProc2, auto2);
var allCompleted = WaitHandle.WaitAll(new[] { auto1, auto2 }, 20);
ThreadPool.GetAvailableThreads(out worker, out io);
var total = Process.GetCurrentProcess().Threads.Count;
if (allCompleted)
{
Console.WriteLine($"All threads done: (Iteration #{i + 1}). Total: {total}, Available: {worker}, {io}\n");
}
else
{
Console.WriteLine($"Timed out: (Iteration #{i + 1}). Total: {total}, Available: {worker}, {io}\n");
}
}
Console.WriteLine("Press any key to exit...");
}
catch(Exception ex)
{
Console.WriteLine("An exception occurred.");
Console.WriteLine($"{ex.GetType().Name}: {ex.Message}");
Console.WriteLine("The program will now exit. Press any key to terminate the program...");
}
Console.ReadKey();
}
static void ThreadProc1(object state)
{
lock(lockA)
{
Console.WriteLine("ThreadProc1 entered lockA. Going to acquire lockB");
lock(lockB)
{
Console.WriteLine("ThreadProc1 acquired both locks: lockA and lockB.");
//Do stuff
Console.WriteLine("ThreadProc1 running...");
}
}
if (state != null)
{
((AutoResetEvent)state).Set();
}
}
static void ThreadProc2(object state)
{
lock(lockB)
{
Console.WriteLine("ThreadProc2 entered lockB. Going to acquire lockA.");
lock(lockA)
{
Console.WriteLine("ThreadProc2 acquired both locks: lockA and lockB.");
// Do stuff
Console.WriteLine("ThreadProc2 running...");
}
}
if (state != null)
{
((AutoResetEvent)state).Set();
}
}
}
}
Meanwhile, I also kept the Windows Task Manager's Performance tab running and watched the total number of operating system threads go up as my program ate up more threads.
Here is what I observed:
The OS didn't create more threads as the .NET thread pool created a thread every time. In fact, for every four or five iterations that my for loop ran, the OS thread-count would go up by one or two. This was interesting, but this isn't my question. It proves what has already been established.
More interestingly, I observed that the number of threads did not decrease by 2 on every iteration of my for loop. I expected that it should have gone down by 2 because none of my deadlocked threads are expected to return since they are deadlocked, waiting on each other.
I also observed that when the total number of available worker threads in the thread pool went down to zero, the program still kept running more iterations of my for-loop. This made me curious as to where those new threads were coming from if the thread pool had already run out of threads and none of the threads had returned?
So, to clarify, my two question(s), which, perhaps are related in that a single answer may be the explanation to them, are:
When a single iteration of my for-loop ran, for some of those iterations, no thread pool threads were created. Why? And where did the thread pool get the threads to run these iterations on?
Where did the thread pool get the threads from when it ran out of its total number of available worker threads and still kept running my for loop?

ThreadPool.GetAvailableThreads(out worker, out io);
That's not a great statistic to show you how the thread pool works. Primary problem is that it is ridiculously large number on all recent .NET versions. On my dual-core laptop, it starts out at 1020 in 32-bit mode, 32767 in 64-bit mode. Far, far larger than such an anemic CPU could reasonably handle. This number has significantly inflated over the years, it started out at 50x the number of cores back in .NET 2.0. It is now dynamically calculated based on machine capabilities, the job of the CLR host. It uses a glass that's well over half-full.
The primary job of the threadpool manager is to keep threading efficient. The sweet-spot is to keep the number of executing threads limited to the number of processor cores. Running more reduces perf, the OS then has to context-switch between threads and that adds overhead.
That ideal however cannot always be met, practical tp threads that programmers write are not always well-behaved. In practice they take too long and/or spend too much time blocking on I/O or a lock instead of executing code. Where your example is of course a rather extreme case of blocking.
The thread pool manager is not otherwise aware of exactly why a tp thread takes too long to execute. All it can see is that it takes too long to complete. Getting deep insight into exactly why a thread takes too long is not practical, it takes a debugger and the kind heavily trained massively parallel neural network that programmers have between their ears.
Twice a second, the thread pool manager re-evaluates the work load and allows an extra tp thread to start when none of the active ones complete. Even though that is beyond the optimum. On the theory that this is likely to get more work done since presumably the active ones are blocking too much and not using the available cores efficiently. Also important to solve some deadlock scenarios, albeit that you never want to need that. It is just a regular thread like any other, underlying OS call is CreateThread().
So that's what you see, the number of available threads drops by one twice a second. Independent of your code, this is time-based. There is actually a feedback loop implemented in the manager that tries to dynamically calculate the optimum number of extra threads. You never got there yet with all threads blocking.
This does not go on forever, you ultimately reach the high upper limit set by the default SetMaxThreads(). No exception, assuming you did not hit an OutOfMemoryException first and you'd commonly experience in real life, it just stops adding more threads. You are still adding execution requests to the thread pool, covers bullet 3, they just never actually get started. Eventually you'll run out of memory when the number of requests get too large. You'll have to wait for a long time, takes a while to fill up a gigabyte.

The cause is QueueUserWorkItem: "Queues a method for execution. The method executes when a thread pool thread becomes available." https://msdn.microsoft.com/en-us/library/kbf0f1ct(v=vs.110).aspx
In my understanding Threadpool just slowly increases the number of threads to fit your demand, this is what you see in taskmgr. i think you could verify this by adding some things to be done to your thread.
edit: What I mean is, you just queue them, the first threads are starting, and slowly slowly (every 500ms, https://blogs.msdn.microsoft.com/pedram/2007/08/05/dedicated-thread-or-a-threadpool-thread/) more and more threads are added until limits are reached - afterwards you can still queue new ones.

The thread pool (almost) never runs out of threads. There's an injection heuristic that adds new threads in an (almost) unbounded way when it thinks this is helping throughput. This also is a guard against deadlocks based on too few threads being available.
This can be a big problem because memory usage is (almost) unbounded.
"Almost" because there is a maximum thread count but that tends to be extremely high in practice (thousands of threads).
When a single iteration of my for-loop ran, for some of those iterations, no thread pool threads were created.
The reason is not apparent to me from the data shown. You probably should measure Process.GetCurrentProcess().ThreadCount after each iteration.
Maybe the deadlock was avoided in some cases? It's not a deterministic deadlock.
On the current CLR running threads appear to be 1:1 with OS threads.
Maybe you should run a simpler benchmark?
for (int i = 0; i < 10000000; i++)
{
Task.Run(() => Thread.Sleep(Timeout.Infinite));
int workerThreads;
int cpThreads;
ThreadPool.GetAvailableThreads(out workerThreads, out cpThreads);
Console.WriteLine($"Queued: {i}, threads: {Process.GetCurrentProcess().Threads.Count}, workerThreads: {workerThreads}, workerThreads: {cpThreads}");
Thread.Sleep(100);
}

await Task.Delay takes longer than expected

I wrote a multithreaded app which uses async/await extensively. It is supposed to download some stuff at a scheduled time. To achieve that, it uses 'await Task.Delay'. Sometimes it sends thousands requests every minute.
It works as expected, but sometimes my program needs to log something big. When it does, it serializes many objects and saves them to a file. During that time, I noticed that my scheduled tasks are executed too late. I've put all the logging to a separate thread with the lowest priority and the problem doesn't occur that often anymore, but it still happens. The things is, I want to know when it happens and in order to know that I have to use something like that:
var delayTestDate = DateTime.Now;
await Task.Delay(5000);
if((DateTime.Now - delayTestDate).TotalMilliseconds > 6000/*delays up to 1 second are tolerated*/) Console.WriteLine("The task has been delayed!");
Moreover, I have found that 'Task.Run', which I also use, can also cause delays. To monitor that, I have to use even more ugly code:
var delayTestDate = DateTime.Now;
await Task.Run(() =>
{
if((DateTime.Now - delayTestDate).TotalMilliseconds > 1000/*delays up to 1 second are tolerated*/) Console.WriteLine("The task has been delayed!");
//do some stuff
delayTestDate = DateTime.Now;
});
if((DateTime.Now - delayTestDate).TotalMilliseconds > 1000/*delays up to 1 second are tolerated*/) Console.WriteLine("The task has been delayed!");
I have to use it before and after every await and Task.Run and inside every async function, which is ugly and inconvenient. I can't put it into a separate function, since it would have to be async and I would have to await it anyway. Does anybody have an idea of a more elegant solution?
EDIT:
Some information I provided in the comments:
As #YuvalItzchakov noticed, the problem may be caused by Thread Pool starvation. That's why I used System.Threading.Thread to take care of the logging outside of the Thread Pool, but as I said, the problem still sometimes occur.
I have a processor with four cores and by subtracting results of ThreadPool.GetAvailableThreads from ThreadPool.GetMaxThreads I get 0 busy worker threads and 1-2 busy completion port threads. Process.GetCurrentProcess().Threads.Count usually returns about 30. It's a Windows Forms app and although it only has a tray icon with a menu, it starts with 11 threads. When it gets to sending thousands requests per minute, it quickly gets up to 30.
As #Noseratio suggested, I tried to play with ThreadPool.SetMinThreads and ThreadPool.SetMaxThreads, but it didn't even change the numbers of busy threads mentioned above.

When you execute Task.Run it uses Thread Pool threads to execute those tasks. When you have long running tasks, you are causing starvation to the Thread Pool, since its resources are currently occupied with long running tasks.
2 Suggestions:
When running long running tasks, make sure to use Task.Factory.Startnew with TaskCreationOptions.LongRunning, which will trigger a new thread creation. You must be cautious here as well, as spinning too many new threads will cause excessive context switches which will cause your app to slow down
Use true async where you have to do IO Bound work, use apis that support the TAP such as HttpClient and Stream, which wont cause a new thread to execute blocking work.

There are overheads in async/await, as well as the tasks themselves being executed at a lower priority. If you need something to happen reliably at an accurate interval, async/await / TPL is not the interface to use.
Try creating an independent background thread that loops until it is scheduled to do work. This way you can control the priority and timing directly without going through TPL / async.
Thread backgroundThread = new Thread(BackgroundWork);
DateTime nextInterval = DateTime.Now;
public void BackgroundWork()
{
if(DateTime.Now > nextInterval){
DoWork();
nextInterval = nextInterval.Add(new TimeSpan(0,0,0,10)); // 10 seconds
}
Thread.Sleep(100);
}
Adjust the Sleep(..) and interval values as needed.

I think you're experiencing the situation described by Joe Duffy in his "CLR thread pool injection, stuttering problems" blog post:
One silly thing our thread pool currently does has to do with how it
creates new threads. Namely, it severely throttles creation of new
threads once you surpass the “minimum” number of threads, which, by
default, is the number of CPUs on the machine. We limit ourselves to
at most one new thread per 500ms once we reach or surpass this number.
One solution might be to explicitly increase the minimum number of thread pool threads before making any use of TPL, e.g.:
ThreadPool.SetMaxThreads(workerThreads: 200, completionPortThreads: 200);
ThreadPool.SetMinThreads(workerThreads: 100, completionPortThreads: 100);
Try playing with these numbers and see if the problem goes away.

Multi Thread c# application System.OutOfMemoryException after 1~5 minutes of runtime

Here is my Timer Elapsed Event, I am receiving the System.OutOfMemoryException on the line Thread thread = new Thread(threadStart);
I am receiving the error fairly fast (1~5 minutes, randomly), and it does not cause unexpected results in my program. I am just wondering what is causing this error, and I am afraid it may cause unexpected results if it is left unchecked. I have searched on the internet and am comming no where near the number of max threads.
readList contains about 46 enteries.
Any help would be appreciated.
private void glob_loopTimer_Elapsed(object sender, ElapsedEventArgs e)
{
try
{
ParameterizedThreadStart threadStart = new ParameterizedThreadStart(readHoldingRegisters);
foreach (readwriteDataGridRow.Read row in readList)
{
Thread thread = new Thread(threadStart);
thread.IsBackground = true;
thread.Start(System.Convert.ToInt32(row.Address));
}
}
catch (Exception ex)
{
UpdateConsole(new object[] { ex.Message.ToString() + " " + ex.StackTrace.ToString(), Color.Red });
Thread.CurrentThread.Abort(); // maybe?
}
}
EDIT:
Here is a bit more information.
My program is reading registers from a Serial Device using the Modbus RTU protocol.
A single register takes less than a tenth of a second to retrieve from readHoldingRegisters
I am open to suggestions on what else to use rather than threads.
note: I need to call readHoldingRegisters 40 - 100 times in a single 'pass'. The passes start when the user hits connect and end when he hits disconnect. Timers are not needed, they just offered a simple way for me to maintain the loop with a start and stop button.
EDIT: Solved
private void glob_loopTimer_Elapsed(object sender, ElapsedEventArgs e)
{
try
{
foreach (readwriteDataGridRow.Read row in readList)
{
readHoldingRegisters(row.Address);
}
}
catch (Exception ex)
{
UpdateConsole(new object[] { ex.Message.ToString() + " " + ex.StackTrace.ToString(), Color.Red });
}
}
The additional Threads were the problem and were not needed.

Ughh, do not, ever (well almost ever) abort threads. There are many preferable ways to make a System.Thread stop. Look around SO, you will find plenty of examples on why doing this is a bad idea and alternative approaches.
On with your question: The problem doesn't seem to be the number of rows in readList. It is more likely that your glob_looperTimer_Elapsed event handler is being executed many times and you are basically starting more and more threads.
What is the interval of your glob_loopTimer?

So how many times is glob_loopTimer_Elapsed called? The name implies that it is run on a periodic timer interval. If so, and if the 46 threads that get created on each invocation do not terminate about as quickly as the timer interval fires, then you could easily be spawning too many threads and running out of memory space as a result. Perhaps you could try logging when each thread starts and when each one finishes to get an idea about how many are in flight at once?
Keep in mind that every thread you allocate will have a certain amount of stack space allocated to it. Depending upon your runtime configuration, this amount of stack space may not be negligible (as in, it may be 1 MB per thread or more) and it may quickly consume your available memory even if you're not close to approaching the theoretical maximum number of threads supported by the OS.

Besides your problem I'll consider using ThreadPool or the TPL.
When using System.Thread there is no automisn to manage the threads...
Also each Thread allocates some memory which could lead to you problem.
The Threadpool and the TPL manage this resources by themselves
see also: -> Thread vs ThreadPool
Reusing threads that have already been created instead of creating new ones (an expensive process)
...
If you queue 100 thread pool tasks, it will only use as many threads as have already been created to service these requests (say 10
for example). The thread pool will make frequent checks (I believe
every 500ms in 3.5 SP1) and if there are queued tasks, it will make
one new thread. If your tasks are quick, then the number of new
threads will be small and reusing the 10 or so threads for the short
tasks will be faster than creating 100 threads up front.
If your workload consistently has large numbers of thread pool requests coming in, then the thread pool will tune itself to your
workload by creating more threads in the pool by the above process so
that there are a larger number of thread available to process requests
check Here for more in depth info on how the thread pool functions under the hood

I just know
Each thread also consumes (by default) around 1 MB of memory.