I've a c# single threaded application and currently working on to make it multi-threaded with the use of thread pools. I am stuck in deciding which model would work for my problem.
Here's my current scenario
While(1)
{
do_sometask();
wait(time);
}
And this is repeated almost forever. The new scenario has multiple threads which does the above. I could easily implement it by spawning number of threads based on the tasks I have to perform, where all the threads perform some task and wait forever.
The issue here is I may not know the number of tasks, so I can't just blindly spawn 500 threads. I thought about using threadpool, but because almost every thread loops forever and won't ever be freed up for new tasks in the queue, am not sure which other model to use.
I am looking for an idea or solution where I could break the loop in the thread and free it up instead of waiting, but come back and resume the same task after the wait(when the time gets elapsed, using something like a timer/checking timestamp of when the last task is performed).
With this I could use a limited number of threads (like in a thread pool) and serve the tasks which are coming in during the time old threads waits(virtually).
Any help is really appreciated.
If all you have is a bunch of things that happen periodically, it sounds what you want is a bunch of timers. Create a timer for each task, to fire when appropriate. So if you have two different tasks:
using System.Threading;
// Task1 happens once per minute
Timer task1Timer = new Timer(
s => DoTask1(),
null,
TimeSpan.FromMinutes(1),
TimeSpan.FromMinutes(1));
// Task2 happens once every 47 seconds
Timer task2Timer = new Timer(
s => DoTask2(),
null,
TimeSpan.FromSeconds(47),
TimeSpan.FromSeconds(47);
The timer is a pretty lightweight object, so having a whole bunch of them isn't really a problem. The timer only takes CPU resources when it fires. The callback method will be executed on a pool thread.
There is one potential problem. If you have a whole lot of timers all with the same period, then the callbacks will all be called at the same time. The threadpool should handle that gracefully by limiting the number of concurrent tasks, but I can't say for sure. But if your wait times are staggered, this is going to work well.
If you have small wait times (less than a second), then you probably need a different technique. I'll detail that if required.
With this design, you only have one thread blocked at any time.
Have one thread (the master thread) waiting on a concurrent blocking collection, such as the BlockingCollection. This thread will be blocked by a call to TryTake until something is placed in the collection, or after a certain amount of time has passed via a timeout passed into the call (more on this later).
Once it is unblocked, it may have a unit of work to be processed. It checks to see if there is one (i.e., the TryTake call didn't time out), then if there is capacity to perform this work, and if so, queues up a thread (pool, Task or whatevs) to service the work. This master thread then goes back to the blocking collection and tries to take another unit of work. The cycle continues.
As a unit of work is begun, it will be noted so that the main thread can see how many threads are working. Once this unit is completed, the notation will be removed. The thread is then freed.
You want to use a timeout so that if it is judged that too many operations are running concurrently, you will be able to re-evaluate this a set period of time down the road. Otherwise, that unit of work sits in the blocking collection until a new unit is added, which is not optimal.
Outside users of this instance can queue up new units of work by simply dropping them in the collection.
You can use a cancellation token to immediately unblock the thread when it's time to shut down operations. Have the worker operations take cancellation tokens as well so they can halt on shutdown.
I could implement it with the help of a threadpool and few conditions to check the last activity of the task before adding it to the threadpool queue.
Related
I've found a ton of related content that all beat around the bush and I've never been able to find an answer. I'm almost 100% certain that Task.Delay(int) does not use a thread, because I can run this code on my machine with only 16 logical processors:
var tasks = new List<Task>();
for(int i = 1; i < 100000; i++) tasks.Add(Task.Delay(10000));
await Task.WhenAll(tasks);
And it takes ten seconds to complete. If it were using roughly a hundred thousand threads it would take quite a bit longer, I'd think.
So my question is how does Task.Delay(int) work? Not in the manner that this poorly-entitled SO question indicates, but from a threading and hardware resources standpoint.
In the current implementation of .NET, there is a single "timer thread" that just keeps track of managed timer instances and raises their events at the appropriate times. This timer thread will block on its control signal with a timeout set to the next timer's due time. The control signal is used to add/remove/change timers, so when this blocking request times out, the timer thread knows the next timer has fired. This is a normal thread blocking operation, so internally, the thread is idled and removed from the scheduler queue until that blocking operation completes or is timed out. The timing out of those operations is handled by the OS scheduler's timer interrupt.
So technically there is a thread, but it's only one thread per process, not one thread per Task.Delay.
I again stress that this is in the current implementation of .NET. Other solutions have been proposed, such as one timer thread per CPU, or a dynamic pool of timer threads. Perhaps they were experimented with and rejected for some reason, or perhaps an alternative solution will be adopted in the future. AFAIK this is not officially documented anywhere, so this is an implementation detail.
What feedback does TPL use to dynamically adjust the number of worker threads?
My previous understanding was that it measures the rate of task completion to see if adding or removing threads is worth it. But then, why does this code keep increasing the number of threads, even though there is a bottleneck introduced by a semaphore?
Surely, there can be no more than 20 task completions per second, and more than 2 threads will not improve that.
var activeThreads = 0;
var semaphore = new SemaphoreSlim(2);
var res = Parallel.For(0, 1_000_000, i =>
{
Interlocked.Increment(ref activeThreads);
semaphore.Wait();
try
{
Thread.Sleep(100);
Console.WriteLine("Threads: " + activeThreads);
}
finally
{
Interlocked.Decrement(ref activeThreads);
semaphore.Release();
}
});
I believe the ParallelOptions is what you are looking for to specify the amount of parallelism.
Parallel.For(0, 1000, new ParallelOptions
{
MaxDegreeOfParallelism = 2
}, i => { Console.WriteLine(i); });
Personally, I think the TPL library will work in a lot of cases, but it isn't really smart about execution distribution (pardon my english). Whenever you have bottlenecks in the execution of your application, have a look at the pipeline pattern for example. Here is a link that describes different approaches to parallel execution very well imo: https://www.dotnetcurry.com/patterns-practices/1407/producer-consumer-pattern-dotnet-csharp
TL;DR: The thing that you are doing in your code that the TPL uses to justify creating a new thread is blocking. (Synchronizing or sleeping, or performing I/O would all count as blocking.)
A longer explanation...
When your task runs, it takes its thread hostage for 100 ms (because you Sleep(100)). While you are sleeping, that thread cannot be used to run other tasks because it would risk not being in a runnable state when the sleep time period expires. Typically we sleep rather than perform an asynchronous action because we need to keep our call stack intact. We are therefore relying on the stack to maintain our state. And the stack is a one-of-a-kind resource for the thread. (There's not actually a lot more to a thread than its stack.)
So the TPL (Thread pool, specifically) tries to keep occupancy high but the thread count low. One way it achieves this is by making sure that there are approximately just as many runnable threads in the system as there are virtual processors. Each time it needs to increase the thread count, it must create a relatively expensive stack for the thread, so it's best not to have so many. And a thread that is not runnable cannot be scheduled, so when the CPU becomes free, you need something to schedule to make use of the processing resources available. If the thread is sleeping, it cannot be scheduled to run. So instead, a thread will be added to the thread pool and the next task will be scheduled on it.
When you are writing parallel code like this (as in your parallel for loop) that can be partitioned and managed by the TPL you should be careful about putting your thread into a non-runnable state. Performing synchronous I/O, waiting for a synchronization object (e.g. semaphore, event or mutex etc.), or sleeping will put the thread into a state where nothing else can be done on the thread until the I/O completes, the sleep interval expires, or the synchronization object becomes signalled. The thread is no good to the TPL during this period.
In your case, you do several of these things: you wait on a semaphore, you sleep, and you perform I/O by writing to the console. The first thing is waiting on that semaphore. If it's not signalled, then you immediately have the situation where the thread is not runnable and the next task of your million-or-so tasks that need to be run must be scheduled on a different thread. If there isn't one, then the TPL can justify creating a new thread to get more tasks started. After-all, what if it's thread #987,321 that will actually wind up setting the semaphore to unblock task #1? The TPL doesn't know what your code does, so it can delay creating threads for a while in the spirit of efficiency, but for correctness, ultimately it will have to create more threads to start chipping away at the task list. There is a complex, implementation-specific heuristic that it applies to monitor, predict and otherwise get this efficiency guess right.
Now your specific question actually asked what feedback does it use to adjust the number of threads. Like I said, the actual implementation is complex and you should probably think of it as a black-box. But in a nutshell, if there are no runnable threads, it may create another thread to keep chipping away at the task list (or may wait a while before doing so, hoping that things will free up), and if there are too many idle threads, it will terminate the idle threads to reclaim their resources.
And to reiterate, as I said at the top, and to hopefully answer your question this time, the one thing you do that allows the TPL to justify creating a new thread is to block. ...even on that first semaphore.
Ran into an article analysing the thread injection algorithm in 2017. As of 2019-08-01, the hillclimbing.cpp file on GitHub hasn't really changed so the article should still be up to date.
Relevant details:
The .NET thread pool has two main mechanisms for injecting threads: a
starvation-avoidance mechanism that adds worker threads if it sees no
progress being made on queued items and a hill-climbing heuristic that
tries to maximize throughput while using as few threads as possible.
...
It calculates the desired number of threads based on the ‘current
throughput’, which is the ‘# of tasks completed’ (numCompletions)
during the current time-period (sampleDuration in seconds).
...
It also takes the current thread count (currentThreadCount) into
consideration.
...
The real .NET Thread Pool only increases the thread-count by one
thread every 500 milliseconds. It keeps doing this until the ‘# of
threads’ has reached the amount that the hill-climbing algorithm
suggests.
...
The [hill-climbing] algorithm only returns values that respect the limits
specified by ThreadPool.SetMinThreads(..) and
ThreadPool.SetMaxThreads(..)
...
In addition, [the hill-climbing algorithm] will only recommend
increasing the thread count if the CPU Utilization is below 95%
So it turns out the thread pool does have a feedback mechanism based on task completion rate. It also doesn't explicitly check whether its threads are blocked or running, but it does keep an eye on overall CPU utilization to detect blockages. All this also means it should be roughly aware of what the other threads and processes are doing.
On the other hand, it will always eagerly spawn at least as many threads as told by ThreadPool.SetMinThreads(), which defaults to the number of logical processors on the machine.
In conclusion, the test code in question was doing two things which make it keep piling up more threads:
there are lots of tasks queued up and sitting in the queue for ages, which indicates starvation
CPU utilization is negligible, which means that a new thread should be able to use it
I am writing a Windows Service that receives messages/requests and executes them asynchronously - it does not need to wait for the item to complete, nor care about the result. I am successfully able to execute the requests as Tasks using System.Threading.Tasks.Task. Most of these items execute quickly (less than a second), but some take longer (2-3 minutes).
As a windows service, I need to respond to the "Stop" command and some of the Tasks will still be running. It is preferable to not Cancel the Tasks as the longer running ones might leave data in a bad state (and a rollback is very tricky).
What is the best way to handle this? I thought of keeping a List of the tasks that I have started so that I can do a WaitAll. During the execution of the service it will process tens of thousands of requests. How would I know when to remove completed Tasks from the List so the List doesn't grow wildly? I don't think I should be holding references to that many Task objects.
Thanks in advance.
You can use CancellationToken for this purposes.
Once OnStop event occured, you just call method Cancel() on the CancellationTokenSource and it will be propagated to all tasks that you passed the token in.
There are several techniques how to correctly cancel a task.
You may explicitly check from time to time inside the task if cancellation requested.
Or I beleive there is a property ThrowWhenCancelled on a token itself, and if cancellation has been requested token will throw an CancellationException.
If you don't care about the task results, but want to have a tracking list, just keep the List<Task<T>> (or may be ConcurrentBag<Task<T>>)as a local variable. From time to time start another task that will go through the list and check the Task.Status property, if it's running or else.
I don't think keeping that many references should be an issue as long as you maintain those references correctly.
Also, it depends how do you want to stop your application. If you are fine with the task being killed along with the application you may not track them (unless they hold some resources to hold, and you need to free them) at all. But in most of the cases I would say it should be correctly finalized.
EDIT: just reread your post. RequestAdditionalTime call may help you to wait until long runnning tasks are finished.
Check it on MSDN: ServiceBase.RequestAdditionalTime Method
If you only care about tasks finishing before your service terminates I would suggest using Thread instead of Task.
new Thread(WorkerMethod).Start();
Thread created this way is a so called foreground thread and your application (service) will not end until all foreground threads end. You need to make sure all your foreground threads do not hang under any condition otherwise your application will never terminate by itself. You can achieve the same thing with Task but you would need to keep list of all tasks you have run and use Task.WaitAll to wait for all of them to finish in your Stop event.
If you need to control your threads (i.e. keep reference to them) you need to use some sort of collection.
List<Thread> threads = new List<Thread>();
Thread thrd;
threads.Add(thrd = new Thread(WorkerMethod));
thrd.Start();
But if you actually need to control and cancel you tasks/threads you should rather go with Task which makes cancellation easier.
So I loop that will loop about 20000 times. Each time it loops, I create a new thread. The thread basically calls one method. The method is rather slow it takes four seconds to complete. It goes out and scrapes some page(which we have permission to scrape). I add a one second delay in the loop which would make sure only 4 pages are being scrapped at once. My question what happens to that thread once the method is completed?
Thread.Sleep(1000);
Thread t = new Thread(() => scraping(asin.Trim(), sku.Trim()));
t.Start();
My question what happens to that thread once the method is completed?
It will get destroyed as soon as the method completes.
That being said, this is not an ideal approach. You should use the ThreadPool to avoid creating many threads.
Instead of using new Thread, consider using ThreadPool.QueueUserWorkItem to start off the task.
In addition, if you're using .NET 4, you can use Parallel.ForEach to loop through your entire collection concurrently. This will use the ThreadPool (by default) to schedule all of your tasks.
Finally, you probably should eliminate the Thread.Sleep in your loop - it will just slow down the overall operation, and probably not provide you any gains (once you've switched to using the ThreadPool).
It exits and gets destroyed.
Maybe you'd be more interested in using a ThreadPool instead and set their maximum thread count to 4?
It will reuse the same threads for doing your task, as constructing new threads involves allocating new memory for their stacks, especially if you're doing that 20000 times, it might be one of your bottlenecks.
It gets collected and discarded by the operating system. You might be better off with ThreadPool.QueueUserWorkItem. This will re-use threads so you don't have to incur the setup/teardown cost 20000 times.
Based on your edited post, the thread will be destroyed and its resources garbage collected. As #Karim has also mentioned.
If you were using a ThreadPool it would be returned to the pool. If you know exactly how many threads you plan to keep active at any given time you could create a pool with that number to save some overhead.
I am investigating the design of a work queue processor where the QueueProcessor retrieves a Command Pattern object from the Queue and executes it in a new thread.
I am trying to get my head around a potential Queue lockup scenario where nested Commands may result in a deadlock.
E.G.
A FooCommand object is placed onto the queue which the QueueProcessor then executes in its own thread.
The executing FooCommand places a BarCommand onto the queue.
Assuming that the maximum allowed threads was only 1 thread, the QueueProcessor would be in a deadlocked state since the FooCommand is infinitely waiting for the BarCommand to complete.
How can this situation be managed? Is a queue object the right object for the job? Are there any checks and balances that can be put into place to resolve this issue?
Many thanks. ( application uses C# .NET 3.0 )
You could redesign things so that FooCommand doesn't use the queue to run BarCommand but runs it directly, or you could split FooCommand into two, and have the first half stop immediately after queueing BarCommand, and have BarCommand queue the second have of FooCommand after it's done its work.
Queuing implicitly assumes an asynchronous execution model. By waiting for the command to exit, you are working synchronously.
Maybe you can split up the commands in three parts: FooCommand1 that executes until the BarCommand has to be sent, BarCommand and finally FooCommand2 that continues after BarCommand has finished. These three commands can be queued separately. Of course, BarCommand should make sure that FooCommand2 is queued.
For simple cases like this an additional monitoring thread that can spin off more threads on demand is helpful.
Basically every N seconds check to see if any jobs have been finished, if not, add another thread.
This won't necessarily handle even more complex deadlock problems, but it will solve this one.
My recommendation for the heavier problem is to restrict waits to newly spawned process, in other words, you can only wait on something you started, that way you never get deadlocks, since cycles are impossible in that situation.
If you are building the Queue object yourself there are a few things you can try:
Dynamically add new service threads. Use a timer and add a thread if the available thread count has been zero for too long.
If a command is trying to queue another command and wait for the result then you should synchronously execute the second command in the same thread. If the first thread simply waits for the second you won't get a concurrency benefit anyway.
I assume you want to queue BarCommand so it is able to run in parallel with FooCommand, but BarCommand will need the result at some later point. If this is the case then I would recommend using Future from the Parallel Extensions library.
Bart DeSmet has a good blog entry on this. Basically you want to do:
public void FooCommand()
{
Future<int> BarFuture = new Future<int>( () => BarCommand() );
// Do Foo's Processing - Bar will (may) be running in parallel
int barResult = BarFuture.Value;
// More processing that needs barResult
}
With libararies such as the Parallel Extensions I'd avoid "rolling your own" scheduling.