I am new to threaded programming. I have to run few tasks in PARALLEL and in Background (so that main UI execution thread remain responsive to user actions) and wait for each one of them to complete before proceeding further execution.
Something like:
foreach(MyTask t in myTasks)
{
t.DoSomethinginBackground(); // There could be n number of task, to save
// processing time I wish to run each of them
// in parallel
}
// Wait till all tasks complete doing something parallel in background
Console.Write("All tasks Completed. Now we can do further processing");
I understand that there could be several ways to achieve this. But I am looking for the best solution to implement in .Net 4.0 (C#).
To me it would seem like you want Parallel.ForEach
Parallel.ForEach(myTasks, t => t.DoSomethingInBackground());
Console.Write("All tasks Completed. Now we can do further processing");
You can also perform multiple tasks within a single loop
List<string> results = new List<string>(myTasks.Count);
Parallel.ForEach(myTasks, t =>
{
string result = t.DoSomethingInBackground();
lock (results)
{ // lock the list to avoid race conditions
results.Add(result);
}
});
In order for the main UI thread to remain responsive, you will want to use a BackgroundWorker and subscribe to its DoWork and RunWorkerCompleted events and then call
worker.RunWorkerAsync();
worker.RunWorkerAsync(argument); // argument is an object
You can use Task library to complete:
string[] urls = ...;
var tasks = urls.Select(url => Task.Factory.StartNew(() => DoSomething(url)));
To avoid locking UI Thread, you can use ContinueWhenAll in .NET 4.0:
Task.Factory.ContinueWhenAll(tasks.ToArray(), _ =>
Console.Write("All tasks Completed. Now we can do further processing");
);
If you are in the latest version of .NET, you can use Task.WhenAll instead
If you use Net 4.0 or up, refer to the Parallel class and Task class. Joseph Albahari wrote very clear book about that: http://www.albahari.com/threading/part5.aspx#_Creating_and_Starting_Tasks
Related
I've been developing an application which I need to run some methods as parallel and not blocking. first I used Task.Run, but IN DEBUG MODE, I see that the operation blocks and just waits for the result. I do not want this, I want all method , which call in a foreach loop, run asynchronously.
public async void f()
{
foreach (var item in childrenANDparents)
{
await Task.Run(() => SendUpdatedSiteInfo(item.Host,site_fr));
// foreach loop does not work until the task return and continues
}
}
So I changed the task.run to thread.start and it works great!
public async void f()
{
foreach (var item in childrenANDparents)
{
Thread t = new Thread(() => SendUpdatedSiteInfo(item.Host, site_fr));
t.Start();
// foreach loop works regardless of the method, in debug mode it shows me
// they are working in parallel
}
}
Would you explain what is the difference and why ? I expect the same behavior from both code and it seems they are different.
thanks
I want all method , which call in a foreach loop, run asynchronously.
It seems that you're confusing async/sync calls with parallelization.
A quote from MSDN:
Data parallelism: A form of parallel processing where the same
computation executes in parallel on different data. Data parallelism
is supported in the Microsoft .NET Framework by the Parallel.For and
Parallel.ForEach methods and by PLINQ. Compare to task parallelism.
Asynchronous operation: An operation that that does not block the current thread
of control when the operation starts.
Let's have a closer look at your code again:
foreach (var item in childrenANDparents)
{
await Task.Run(() => SendUpdatedSiteInfo(item.Host,site_fr));
}
The await keyword will cause compiler to create a StateMachine that will handle the method execution.
It's like if you say to compiler:"Start this async operation without blocking any threads and when it's completed - execute the rest of the stuff".
After Task finishes execution this thread will be released and returned to a ThreadPool and it will execute the rest of the code on a first available thread from a ThreadPool and will make attempt to execute it in a thread in which it had started the method execution (unless .ConfigureAwait(false) is used in which case it's more like 'fire and forget' mode when we don't really care which thread will do the continuation).
When you create a separate Thread you do parallelism by delegating some code to run in a separate Thread. So depending on the code itself it may or may not be executed asynchronously.
It's like if you say to compiler:"Take this piece of work start a new thread and do it there"
If you still want to use Tasks with parallelism you could create an array of tasks in a loop and then wait for all of them to finish execution:
var tasks = new[]
{
childrenANDparents.Select(item=> Task.Run(() => SendUpdatedSiteInfo(item.Host,site_fr)));
}
await Task.WhenAll(tasks);
P.S.
And yes you may as well use TPL (Task Parallel Library) and specifically Parallel loops.
You could use a simple Parallel.ForEach or PLinq
Parallel.ForEach(childrenANDparents, (item) =>
{
SendUpdatedSiteInfo(item.Host,site_fr)
});
To better understand async and await its best to start reading some docos, its a large topic, but its worth your while
https://learn.microsoft.com/en-us/dotnet/csharp/programming-guide/concepts/async/
It's often the case when I need to get a task done within X amount of seconds and if it doesn't complete, I want to keep going, processing the rest of the tasks
I've been defaulting to something like this:
Thread worker = new Thread(() => {
// do some long operation
});
Thread monitor = new Thread(() => {
Thread.Sleep(10000);
if(worker != null && worker.IsAlive) {
worker.Abort();
worker = null;
}
StartNextTask();
});
monitor.Start ();
worker.Start();
This works, but it is cumbersome because it uses two threads (yes, you can use the Task class as well to use threads from the threadpool).
AutoResetEvents and event based models don't quite work because the former blocks the monitor thread until the worker is done, and the event driven approach relies on the client to call the event and notify the monitor.
Are there alternative patterns to this that can follow the same semantics?
If you use the Task class as you say, you can also use Task.Wait() which does exactly what you want. Specify an amount of seconds to wait. The task doesn't get cancelled unless you cancel it by use of a CancellationToken
See: http://msdn.microsoft.com/en-us/library/dd235606.aspx
I’m writing a win forms that uses the report viewer for the creation of multiple PDF files. These PDF files are divided in 4 main parts, each part is responsible for the creation of a specific report. These processes are creating a minimum of 1 file up to the number of users (currently 50).
The program already exists using there 4 methods sequentially. For extra performance where the number of users is growing, I want to separate these methods from the mail process in 4 separate threads.
While I'm new to multithreading using C# I read a number of articles how to achieve this. The only thing I'm not sure of is which way I should start. As I read multiple blog posts I'm not sure if to use 4 separate threads, a thread pool or multiple background workers. (or should parallel programming be the best way?). Blog posts tell me if more than 3 threads use a thread pool, but on the other hand the tell me if using winforms, use the backgroundworker. Which option is best (and why)?
At the end my main thread has to wait for all processes to end before continuing.
Can someone tell me what's the best solution to my problem.
* Extra information after edit *
Which i forgot to tell (after i read al your comments and possible solutions). The methods share one "IEnumerable" only for reading. After firing the methods (that don't have to run sequentially), the methods trigger events for for sending status updates to the UI. I think triggering events is difficult if not impossible using separate threads so there should be some kind of callback function to report status updates while running.
some example in psuedo code.
main()
{
private List<customclass> lcc = importCustomClass()
export.CreatePDFKind1.create(lcc.First(), exportfolderpath, arg1)
export.CreatePDFKind2.create(lcc, exportfolderpath)
export.CreatePDFKind3.create(lcc.First(), exportfolderpath)
export.CreatePDFKind4.create(customclass2, exportfolderpath)
}
namespace export
{
class CreatePDFKind1
{
create(customclass cc, string folderpath)
{
do something;
reportstatus(listviewItem, status, message)
}
}
class CreatePDFKind2
{
create(IEnumerable<customclass> lcc, string folderpath)
{
foreach (var x in lcc)
{
do something;
reportstatus(listviewItem, status, message)
}
}
}
etc.......
}
From the very basic picture you have described, I would use the Task Paralell Library (TPL). Shipped with .NET Framework 4.0+.
You talk about the 'best' option of using thread pools when spawning a large-to-medium number of threads. Dispite this being correct [the most efficent way of mangaing the resources], the TPL does all of this for you - without you having to worry about a thing. The TPL also makes the use of multiple threads and waiting on their completion a doddle too...
To do what you require I would use the TPL and Continuations. A continuation not only allows you to create a flow of tasks but also handles your exceptions. This is a great introduction to the TPL. But to give you some idea...
You can start a TPL task using
Task task = Task.Factory.StartNew(() =>
{
// Do some work here...
});
Now to start a second task when an antecedent task finishes (in error or successfully) you can use the ContinueWith method
Task task1 = Task.Factory.StartNew(() => Console.WriteLine("Antecedant Task"));
Task task2 = task1.ContinueWith(antTask => Console.WriteLine("Continuation..."));
So as soon as task1 completes, fails or is cancelled task2 'fires-up' and starts running. Note that if task1 had completed before reaching the second line of code task2 would be scheduled to execute immediately. The antTask argument passed to the second lambda is a reference to the antecedent task. See this link for more detailed examples...
You can also pass continuations results from the antecedent task
Task.Factory.StartNew<int>(() => 1)
.ContinueWith(antTask => antTask.Result * 4)
.ContinueWith(antTask => antTask.Result * 4)
.ContinueWith(antTask =>Console.WriteLine(antTask.Result * 4)); // Prints 64.
Note. Be sure to read up on exception handling in the first link provided as this can lead a newcomer to TPL astray.
One last thing to look at in particular for what you want is child tasks. Child tasks are those which are created as AttachedToParent. In this case the continuation will not run until all child tasks have completed
TaskCreationOptions atp = TaskCreationOptions.AttachedToParent;
Task.Factory.StartNew(() =>
{
Task.Factory.StartNew(() => { SomeMethod() }, atp);
Task.Factory.StartNew(() => { SomeOtherMethod() }, atp);
}).ContinueWith( cont => { Console.WriteLine("Finished!") });
So in your case you would start your four tasks, then wait on their completion on the main thread.
I hope this helps.
Using a BackgroundWorker is helpful if you need to interact with the UI with respect to your background process. If you don't, then I wouldn't bother with it. You can just start 4 Task objects directly:
tasks.Add(Task.Factory.StartNew(()=>DoStuff()));
tasks.Add(Task.Factory.StartNew(()=>DoStuff2()));
tasks.Add(Task.Factory.StartNew(()=>DoStuff3()));
If you do need to interact with the UI; possibly by updating it to reflect when the tasks are finished, then I would suggest staring one BackgroundWorker and then using tasks again to process each individual unit of work. Since there is some additional overhead in using a BackgroundWorker I would avoid starting lots of them if you can avoid it.
BackgroundWorker bgw = new BackgroundWorker();
bgw.DoWork += (_, args) =>
{
List<Task> tasks = new List<Task>();
tasks.Add(Task.Factory.StartNew(() => DoStuff()));
tasks.Add(Task.Factory.StartNew(() => DoStuff2()));
tasks.Add(Task.Factory.StartNew(() => DoStuff3()));
Task.WaitAll(tasks.ToArray());
};
bgw.RunWorkerCompleted += (_, args) => updateUI();
bgw.RunWorkerAsync();
You could of course use just Task methods to do all of this, but I still find BackgroundWorkers a bit simpler to work with for the simpler cases. Using .NEt 4.5 you could use Task.WhenAll to run a continuation in the UI thread when all 4 tasks finished, but doing that in 4.0 wouldn't be quite as simple.
Without further information it's impossible to tell. The fact that they're in four separate methods doesn't make much of a difference if they're accessing the same resources. The PDF file for example. If you're having trouble understanding what I mean you should post some of the code for each method and I'll go into a little more detail.
Since the number of "parts" you have is fixed it won't make a big difference whether you use separate threads, background workers or use a thread pool. I'm not sure why people are recommending background workers. Most likely because it's a simpler approach to multithreading and more difficult to screw up.
I want to queue dependant tasks across several flows that need to be processed in order (in each flow). The flows can be processed in parallel.
To be specific, let's say I need two queues and I want the tasks in each queue to be processed in order. Here is sample pseudocode to illustrate the desired behavior:
Queue1_WorkItem wi1a=...;
enqueue wi1a;
... time passes ...
Queue1_WorkItem wi1b=...;
enqueue wi1b; // This must be processed after processing of item wi1a is complete
... time passes ...
Queue2_WorkItem wi2a=...;
enqueue wi2a; // This can be processed concurrently with the wi1a/wi1b
... time passes ...
Queue1_WorkItem wi1c=...;
enqueue wi1c; // This must be processed after processing of item wi1b is complete
Here is a diagram with arrows illustrating dependencies between work items:
The question is how do I do this using C# 4.0/.NET 4.0? Right now I have two worker threads, one per queue and I use a BlockingCollection<> for each queue. I would like to instead leverage the .NET thread pool and have worker threads process items concurrently (across flows), but serially within a flow. In other words I would like to be able to indicate that for example wi1b depends on completion of wi1a, without having to track completion and remember wi1a, when wi1b arrives. In other words, I just want to say, "I want to submit a work item for queue1, which is to be processed serially with other items I have already submitted for queue1, but possibly in parallel with work items submitted to other queues".
I hope this description made sense. If not please feel free to ask questions in the comments and I will update this question accordingly.
Thanks for reading.
Update:
To summarize "flawed" solutions so far, here are the solutions from the answers section that I cannot use and the reason(s) why I cannot use them:
TPL tasks require specifying the antecedent task for a ContinueWith(). I do not want to maintain knowledge of each queue's antecedent task when submitting a new task.
TDF ActionBlocks looked promising, but it would appear that items posted to an ActionBlock are processed in parallel. I need for the items for a particular queue to be processed serially.
Update 2:
RE: ActionBlocks
It would appear that setting the MaxDegreeOfParallelism option to one prevents parallel processing of work items submitted to a single ActionBlock. Therefore it seems that having an ActionBlock per queue solves my problem with the only disadvantage being that this requires the installation and deployment of the TDF library from Microsoft and I was hoping for a pure .NET 4.0 solution. So far, this is the candidate accepted answer, unless someone can figure out a way to do this with a pure .NET 4.0 solution that doesn't degenerate to a worker thread per queue (which I am already using).
I understand you have many queues and don't want to tie up threads. You could have an ActionBlock per queue. The ActionBlock automates most of what you need: It processes work items serially, and only starts a Task when work is pending. When no work is pending, no Task/Thread is blocked.
The best way is to use the Task Parallel Library (TPL) and Continuations. A continuation not only allows you to create a flow of tasks but also handles your exceptions. This is a great introduction to the TPL. But to give you some idea...
You can start a TPL task using
Task task = Task.Factory.StartNew(() =>
{
// Do some work here...
});
Now to start a second task when an antecedent task finishes (in error or successfully) you can use the ContinueWith method
Task task1 = Task.Factory.StartNew(() => Console.WriteLine("Antecedant Task"));
Task task2 = task1.ContinueWith(antTask => Console.WriteLine("Continuation..."));
So as soon as task1 completes, fails or is cancelled task2 'fires-up' and starts running. Note that if task1 had completed before reaching the second line of code task2 would be scheduled to execute immediately. The antTask argument passed to the second lambda is a reference to the antecedent task. See this link for more detailed examples...
You can also pass continuations results from the antecedent task
Task.Factory.StartNew<int>(() => 1)
.ContinueWith(antTask => antTask.Result * 4)
.ContinueWith(antTask => antTask.Result * 4)
.ContinueWith(antTask =>Console.WriteLine(antTask.Result * 4)); // Prints 64.
Note. Be sure to read up on exception handling in the first link provided as this can lead a newcomer to TPL astray.
One last thing to look at in particular for what you want is child tasks. Child tasks are those which are created as AttachedToParent. In this case the continuation will not run until all child tasks have completed
TaskCreationOptions atp = TaskCreationOptions.AttachedToParent;
Task.Factory.StartNew(() =>
{
Task.Factory.StartNew(() => { SomeMethod() }, atp);
Task.Factory.StartNew(() => { SomeOtherMethod() }, atp);
}).ContinueWith( cont => { Console.WriteLine("Finished!") });
I hope this helps.
Edit: Have you had a look at ConcurrentCollections in particular the BlockngCollection<T>. So in your case you might use something like
public class TaskQueue : IDisposable
{
BlockingCollection<Action> taskX = new BlockingCollection<Action>();
public TaskQueue(int taskCount)
{
// Create and start new Task for each consumer.
for (int i = 0; i < taskCount; i++)
Task.Factory.StartNew(Consumer);
}
public void Dispose() { taskX.CompleteAdding(); }
public void EnqueueTask (Action action) { taskX.Add(Action); }
void Consumer()
{
// This seq. that we are enumerating will BLOCK when no elements
// are avalible and will end when CompleteAdding is called.
foreach (Action action in taskX.GetConsumingEnumerable())
action(); // Perform your task.
}
}
A .NET 4.0 solution based on TPL is possible, while hiding away the fact that it needs to store the parent task somewhere. For example:
class QueuePool
{
private readonly Task[] _queues;
public QueuePool(int queueCount)
{ _queues = new Task[queueCount]; }
public void Enqueue(int queueIndex, Action action)
{
lock (_queues)
{
var parent = _queue[queueIndex];
if (parent == null)
_queues[queueIndex] = Task.Factory.StartNew(action);
else
_queues[queueIndex] = parent.ContinueWith(_ => action());
}
}
}
This is using a single lock for all queues, to illustrate the idea. In production code, however, I would use a lock per queue to reduce contention.
It looks like the design you already have is good and working. Your worker threads (one per queue) are long-running so if you want to use Task's instead, specify TaskCreationOptions.LongRunning so you get a dedicated worker thread.
But there isn't really a need to use the ThreadPool here. It doesn't offer many benefits for long-running work.
I have a c# console application which has some threads to do some work (download a file).
each thread may exit the application at any time any where in application, but I'll show a proper message on console. It's possible to track them but it doesn't make sense to me. I want simply check thread count or something like that to find out which one is the last thread and do something when it is exiting.
What's the best practice to do so ?
pseudo code:
if (lastThread)
{
cleanUp();
Console.ReadLine();
}
Thanks
This is one place where using the new Task Parallel Library can make life much easier. Instead of creating threads, and spinning work up on the thread, you can use multiple tasks:
var task1 = Task.Factory.StartNew( () => DoTaskOneWork() );
var task2 = Task.Factory.StartNew( () => DoTaskTwoWork() );
var task3 = Task.Factory.StartNew( () => DoTaskThreeWork() );
// Block until all tasks are done
Task.WaitAll(new[] {task1, task2, task3} );
cleanUp(); // Do your cleanup
If the "tasks" are just downloading a bunch of individual files, you could even make this simpler using PLINQ:
var fileUrls = GetListOfUrlsToDownload();
fileUrls.AsParallel().ForAll( fileUrl => DownloadAndProcessFile(fileUrl) );
cleanUp(); // Do your cleanup
A design where you lose track of your threads is not ideal.
Depending on how you spawn them it ought to be possible to track the status of each by associating some per-thread signalable object, then WaitAll on those signalable objects.
Each signalable object in turn should get signaled as its thread exits. When they are all signaled, you know the threads are all dead and you close down clean. You have to make sure that abnormal conditions in your threads do not result in that thread's associated signalable object remaining unset, or your WaitAll will never return. This means exceptions typically - could use try...finally to ensure the objects get signaled.
Your new pseudocode is
foreach (workitem in list of work)
start up thread associated with a ManualResetEvent or similar
WaitAll for all events to be signalled
cleanup
Your main thread should join with all your worker threads and block while they are running. Then when all threads are complete it performs the cleanup code and then quits.
Alternatively you can use a WaitHandle such as as a ManualResetEvent per thread and wait for all of them to be signalled.