Develop Reference

Random Tasks of Task.Factory.StartNew don't start

I need to have 5 tasks completed in parallel with max 2 executed at a time.
So, as soon as some task is finished, the next should be run - up until there are no tasks pending.
I'm using a solution by L.B. which involves using semaphores for synchronization across tasks.
void LaunchTaskPool ()
{
SemaphoreSlim maxThreadSemaphore = new SemaphoreSlim(2); //Max 2 tasks at a time.
for (int i = 0; i < 5; i++) //loop through 5 tasks to be assigned
{
maxThreadSemaphore.Wait(); //Wait for the queue
Console.WriteLine("Assigning work {0} ", i);
Task t = Task.Factory.StartNew(() =>
{
DoWork(i.ToString()); // assign tasks
}, TaskCreationOptions.LongRunning
)
.ContinueWith(
(task) => maxThreadSemaphore.Release() // step out of the queue
);
}
}
void DoWork(string workname)
{
Thread.Sleep(100);
Console.WriteLine("--work {0} starts", workname);
Thread.Sleep(1000);
Console.WriteLine("--work {0} finishes", workname);
}
The problem is that some random tasks would not even start. For example here Work 1 and 3 never started and Work 4 got run twice:
I tried adding Task.WaitAll() as suggested here, but it didn't help.
thanks in advance for your suggestions!
Constantine.

I recommend using Parallel.For() for this instead; there's no need to reinvent the wheel! You can specify MaxDegreeOfParallelism when using Parallel.For():
For example:
using System;
using System.Threading;
using System.Threading.Tasks;
namespace ConsoleApp4
{
class Program
{
static void Main()
{
Parallel.For(
0, // Inclusive start
5, // Exclusive end
new ParallelOptions{MaxDegreeOfParallelism = 2},
i => DoWork(i.ToString()));
}
static void DoWork(string workname)
{
Thread.Sleep(100);
Console.WriteLine("--work {0} starts", workname);
Thread.Sleep(1000);
Console.WriteLine("--work {0} finishes", workname);
}
}
}
(Actually I just looked, and this is already in one of the other answers in the thread you linked - is there a reason you didn't want to use that solution? If not, I guess we should close this question as a duplicate...)
Anyway to answer your actual question:
You are accessing a "modified closure" in the loop. To fix this, make a copy of the loop variable i before passing it to the task:
SemaphoreSlim maxThreadSemaphore = new SemaphoreSlim(2); //Max 2 tasks at a time.
for (int i = 0; i < 5; i++) //loop through 5 tasks to be assigned
{
maxThreadSemaphore.Wait(); //Wait for the queue
Console.WriteLine("Assigning work {0} ", i);
int copy = i; // <----- Make a copy here.
Task t = Task.Factory.StartNew(() =>
{
DoWork(copy.ToString()); // assign tasks
}, TaskCreationOptions.LongRunning
)
.ContinueWith(
(task) => maxThreadSemaphore.Release() // step out of the queue
);
}

The problem with your solution is that before the Task is started the loop has allready run through and is starting the next Task.
As #Matthew Watson recommended you should use Parallel.For.
Just out of interest this would solve your problem:
static void LaunchTaskPool()
{
SemaphoreSlim maxThreadSemaphore = new SemaphoreSlim(2); //Max 2 tasks at a time.
for (int i = 0; i < 5; i++) //loop through 5 tasks to be assigned
{
maxThreadSemaphore.Wait(); //Wait for the queue
Console.WriteLine("Assigning work {0} ", i);
StartThead(i, maxThreadSemaphore);
}
}
static void StartThead(int i, SemaphoreSlim maxThreadSemaphore)
{
Task.Factory.StartNew(
() => DoWork(i.ToString()),
TaskCreationOptions.None
).ContinueWith((task) => maxThreadSemaphore.Release());
}
static void DoWork(string workname)
{
Thread.Sleep(100);
Console.WriteLine("--work {0} starts", workname);
Thread.Sleep(1000);
Console.WriteLine("--work {0} finishes", workname);
}

I am working on implementing job queue. Using TPL for the same. Trying to restrict number of jobs at a time using ThreadPool.SetMaxThreads. no luck

****To Restrict the thread :****
int workerThreads, completionPortThreads;
ThreadPool.GetMaxThreads(out workerThreads, out completionPortThreads);
workerThreads = 2;
ThreadPool.SetMaxThreads(workerThreads, completionPortThreads);
To run the job I tried 2 options
Option 1.
ThreadPool.QueueUserWorkItem(new WaitCallback(ThreadProc),task);
Option 2:
Task runner = new Task(() => taskProcessor.ImportIntoArt(task),TaskCreationOptions.LongRunning|TaskCreationOptions.PreferFairness);
runner.Start();
I expect this code has to pick up first two jobs for processing and 3rd one should go in to the queue. As expected first two jobs will start, however 3rd one will also be picked up for processing.
Any help is highly appreciated.

Use the QueuedTaskScheduler from this package in conjunction with Task.Factory.StartNew method:
var scheduler = new QueuedTaskScheduler(TaskScheduler.Default, 2);
var jobAction = new Action<string>(
jobName =>
{
Console.WriteLine("I am job " + jobName + " and I start at " + DateTime.Now.ToLongTimeString());
Thread.Sleep(10000);
Console.WriteLine("I am job " + jobName + " and I finish at " + DateTime.Now.ToLongTimeString());
});
var jobs = Enumerable
.Range(1, 6)
.Select(num => Task.Factory.StartNew(
() => jobAction("Job" + num),
CancellationToken.None,
TaskCreationOptions.LongRunning,
scheduler))
.ToList();
Task.WhenAll(jobs).Wait();

I know you want to achieve this task using TPL, but as #stuartd has made a comment that we can't do that with threadpool, then you can achieve this task traditional way by creating required number of thread and run them infinitely and observe the collection of a task which of type query.
Please refer below code if you want to achieve the task without using other libraries.
//Declare queue of task.
static Queue<int> taskQueue = new Queue<int>();
static readonly object lockObj = new object();
//Get task to perform.
static int? GetNextTask()
{
lock (lockObj)
{
if (taskQueue.Count > 0)
return taskQueue.Dequeue();
else return null;
}
}
//Add task to queue from different thread.
static void AddTask(int task)
{
lock (lockObj)
{
taskQueue.Enqueue(task);
}
}
static void PerformThreadOperation()
{
//Run infinite for current thread.
while (true)
{
var task = GetNextTask();
//If there is task then perform some action else make thread sleep for some time you can set event to resume thread.
if (task.HasValue)
{
Console.WriteLine("Task Initiate => {0}", task.Value);
Thread.Sleep(4000);
Console.WriteLine("Task Complete => {0}", task.Value);
}
else
{
Console.WriteLine("Task not found, thread is going to be sleep for some moment.");
Console.WriteLine("Thread {0} enter in sleep mode.", Thread.CurrentThread.Name);
Thread.Sleep(5000);
}
}
}
//Create required thread to process task parallely.
static void TestThreadApplication()
{
Thread thread = new Thread(new ThreadStart(PerformThreadOperation));
Thread thread1 = new Thread(PerformThreadOperation);
thread.Start();
thread1.Start();
}
static void Main(string[] args)
{
for (int i = 0; i < 6; i++)
{
taskQueue.Enqueue(i);
}
TestThreadApplication();
Thread.Sleep(20000);
for (int i = 6; i < 10; i++)
{
taskQueue.Enqueue(i);
}
Console.ReadKey();
}

How to run multiple tasks in c# and get an event on complete of these tasks?

I am re-running a Task when its completed. Below is the function I call in the Application_Start of my application.
private void Run()
{
Task t = new Task(() => new XyzServices().ProcessXyz());
t.Start();
t.ContinueWith((x) =>
{
Thread.Sleep(ConfigReader.CronReRunTimeInSeconds);
Run();
});
}
I want to run multiple tasks, number which will be read from web.config app setttings.
I am trying something like this,
private void Run()
{
List<Task> tasks = new List<Task>();
for (int i = 0; i < ConfigReader.ThreadCount - 1; i++)
{
tasks.Add(Task.Run(() => new XyzServices().ProcessXyz()));
}
Task.WhenAll(tasks);
Run();
}
Whats the correct way to do this ?

I believe you are looking for:
Task.WaitAll(tasks.ToArray());
https://msdn.microsoft.com/en-us/library/dd270695(v=vs.110).aspx

if you want to run the tasks one after the other,
await Task.Run(() => new XyzServices().ProcessXyz());
await Task.Delay(ConfigReader.CronReRunTimeInSeconds * 1000);
if you want to run them concurrently, as the task scheduler permits,
await Task.WhenAll(new[]
{
Task.Run(() => new XyzServices().ProcessXyz()),
Task.Run(() => new XyzServices().ProcessXyz())
});
So, your method should be something like,
private async Task Run()
{
var tasks =
Enumerable.Range(0, ConfigReader.ThreadCount)
.Select(i => Task.Run(() => new XyzServices().ProcessXyz()));
await Task.WhenAll(tasks);
}

If you want to wait all tasks to finish and then restart them, Marks's answer is correct.
But if you want ThreadCount tasks to be running at any time (start a new task as soon as any one of them ends), then
void Run()
{
SemaphoreSlim sem = new SemaphoreSlim(ConfigReader.ThreadCount);
Task.Run(() =>
{
while (true)
{
sem.Wait();
Task.Run(() => { /*Your work*/ })
.ContinueWith((t) => { sem.Release(); });
}
});
}

Why do async await without being told

I am trying to make part of my system run in parallel, but some some reason do it wait for each element before it starts the next even though i have not told it to await. I would like to start executing ExecuteItem for each this.Items and then continue when they are all done.
bool singleThread = false;
public async Task Execute()
{
if (!this.singleThread)
{
var tasks = this.Items.Select(x => this.ExecuteItem(x)).ToArray();
Task.WaitAll(tasks);
}
else
{
foreach (var item in this.Items)
{
await this.ExecuteItem(item);
}
}
}
private async Task ExecuteItem(IMyItem item)
{
MappedDiagnosticsContext.Set("itemRef", item.ItemRef);
try
{
await this.HandelItem(item);
}
catch (Exception exp)
{
Logger.ErrorException(string.Format("Execution for {0} failed.", item.ItemName), exp);
Logger.Error("Error Message: ", exp.Message);
}
MappedDiagnosticsContext.Remove("itemRef");
}
To make clear my problem my code behaves as if had wrote the following
var tasks = this.Items.Select(x => await this.ExecuteItem(x)).ToArray();
To make sure it was not some kind of linq problem have i rewriten the problem code to the following, however the code still blocks tasks[i] = this.ExecuteItem(this.Items[i]);
Task[] tasks = new Task[this.Items.Count];
for (int i = 0; i < this.Items.Count; i++)
{
Console.WriteLine("Adding " + this.Items[i].ItemName);
tasks[i] = this.ExecuteItem(this.Items[i]);
}
Console.WriteLine("Waiting!!!");
Task.WaitAll(tasks);

Something in HandelItem is blocking.
async methods don't run completely asynchronously, they execute synchronously up until the point they hit an await. So all of ExecuteItem, up to HandelItem will run before the tasks list is built. This synchronous behavior would continue into HandelItem if it is an async method, so likely HandelItem is executing while building up the tasks list.
This is easily seen with this example program:
static void Main(string[] args)
{
var items = Enumerable.Range(1, 2);
Console.WriteLine("Start");
var tasks = items.Select(i => AsyncMethod(i)).ToArray();
Console.WriteLine("Got tasks");
Task.WaitAll(tasks);
Console.WriteLine("Done!");
}
static async Task AsyncMethod(int i)
{
Console.WriteLine("Enter {0}", i);
await AsyncMethod2(i);
await Task.Delay(1000);
Console.WriteLine("Exit {0}", i);
}
static async Task AsyncMethod2(int i)
{
Console.WriteLine("Enter2 {0}", i);
await Task.Delay(2000);
Console.WriteLine("Exit2 {0}", i);
}
It's output is:
Start
Enter 1
Enter2 1
Enter 2
Enter2 2
Got tasks
Exit2 2
Exit2 1
Exit 1
Exit 2
Done!
So both async methods run while building the task list, up until the point that they have to wait. So if HandelItem does something non-asynchronous, it will cause blocking.

If you want the tasks to execute in parallel; and wait until all are complete:
await Task.WhenAll(this.Items.Select(item=>this.ExecuteItem(item)));

Task Parallel Library WaitAny with specified result

I'm trying to write some code that will make a web service call to a number of different servers in parallel, so TPL seems like the obvious choice to use.
Only one of my web service calls will ever return the result I want and all the others won't. I'm trying to work out a way of effectively having a Task.WaitAny but only unblocking when the first Task that matches a condition returns.
I tried with WaitAny but couldn't work out where to put the filter. I got this far:
public void SearchServers()
{
var servers = new[] {"server1", "server2", "server3", "server4"};
var tasks = servers
.Select(s => Task<bool>.Factory.StartNew(server => CallServer((string)server), s))
.ToArray();
Task.WaitAny(tasks); //how do I say "WaitAny where the result is true"?
//Omitted: cancel any outstanding tasks since the correct server has been found
}
private bool CallServer(string server)
{
//... make the call to the server and return the result ...
}
Edit: Quick clarification just in case there's any confusion above. I'm trying to do the following:
For each server, start a Task to check it
Either, wait until a server returns true (only a max of 1 server will ever return true)
Or, wait until all servers have returned false, i.e. there was no match.

The best of what I can think of is specifying a ContinueWith for each Task, checking the result, and if true cancelling the other tasks. For cancelling tasks you may want to use CancellationToken.
var tasks = servers
.Select(s => Task.Run(...)
.ContinueWith(t =>
if (t.Result) {
// cancel other threads
}
)
).ToArray();
UPDATE: An alternative solution would be to WaitAny until the right task completed (but it has some drawbacks, e.g. removing the finished tasks from the list and creating a new array out of the remaining ones is quite a heavy operation):
List<Task<bool>> tasks = servers.Select(s => Task<bool>.Factory.StartNew(server => CallServer((string)server), s)).ToList();
bool result;
do {
int idx = Task.WaitAny(tasks.ToArray());
result = tasks[idx].Result;
tasks.RemoveAt(idx);
} while (!result && tasks.Count > 0);
// cancel other tasks
UPDATE 2: Nowadays I would do it with Rx:
[Fact]
public async Task AwaitFirst()
{
var servers = new[] { "server1", "server2", "server3", "server4" };
var server = await servers
.Select(s => Observable
.FromAsync(ct => CallServer(s, ct))
.Where(p => p)
.Select(_ => s)
)
.Merge()
.FirstAsync();
output.WriteLine($"Got result from {server}");
}
private async Task<bool> CallServer(string server, CancellationToken ct)
{
try
{
if (server == "server1")
{
await Task.Delay(TimeSpan.FromSeconds(1), ct);
output.WriteLine($"{server} finished");
return false;
}
if (server == "server2")
{
await Task.Delay(TimeSpan.FromSeconds(2), ct);
output.WriteLine($"{server} finished");
return false;
}
if (server == "server3")
{
await Task.Delay(TimeSpan.FromSeconds(3), ct);
output.WriteLine($"{server} finished");
return true;
}
if (server == "server4")
{
await Task.Delay(TimeSpan.FromSeconds(4), ct);
output.WriteLine($"{server} finished");
return true;
}
}
catch(OperationCanceledException)
{
output.WriteLine($"{server} Cancelled");
throw;
}
throw new ArgumentOutOfRangeException(nameof(server));
}
The test takes 3.32 seconds on my machine (that means it didn't wait for the 4th server) and I got the following output:
server1 finished
server2 finished
server3 finished
server4 Cancelled
Got result from server3

You can use OrderByCompletion() from the AsyncEx library, which returns the tasks as they complete. Your code could look something like:
var tasks = servers
.Select(s => Task.Factory.StartNew(server => CallServer((string)server), s))
.OrderByCompletion();
foreach (var task in tasks)
{
if (task.Result)
{
Console.WriteLine("found");
break;
}
Console.WriteLine("not found yet");
}
// cancel any outstanding tasks since the correct server has been found

Using Interlocked.CompareExchange will do just that, only one Task will be able to write on serverReturedData
public void SearchServers()
{
ResultClass serverReturnedData = null;
var servers = new[] {"server1", "server2", "server3", "server4"};
var tasks = servers.Select(s => Task<bool>.Factory.StartNew(server =>
{
var result = CallServer((string)server), s);
Interlocked.CompareExchange(ref serverReturnedData, result, null);
}).ToArray();
Task.WaitAny(tasks); //how do I say "WaitAny where the result is true"?
//
// use serverReturnedData as you want.
//
}
EDIT: As Jasd said, the above code can return before the variable serverReturnedData have a valid value (if the server returns a null value, this can happen), to assure that you could wrap the result in a custom object.

Here's a generic solution based on svick's answer:
public static async Task<T> GetFirstResult<T>(
this IEnumerable<Func<CancellationToken, Task<T>>> taskFactories,
Action<Exception> exceptionHandler,
Predicate<T> predicate)
{
T ret = default(T);
var cts = new CancellationTokenSource();
var proxified = taskFactories.Select(tf => tf(cts.Token)).ProxifyByCompletion();
int i;
for (i = 0; i < proxified.Length; i++)
{
try
{
ret = await proxified[i].ConfigureAwait(false);
}
catch (Exception e)
{
exceptionHandler(e);
continue;
}
if (predicate(ret))
{
break;
}
}
if (i == proxified.Length)
{
throw new InvalidOperationException("No task returned the expected value");
}
cts.Cancel(); //we have our value, so we can cancel the rest of the tasks
for (int j = i+1; j < proxified.Length; j++)
{
//observe remaining tasks to prevent process crash
proxified[j].ContinueWith(
t => exceptionHandler(t.Exception), TaskContinuationOptions.OnlyOnFaulted)
.Forget();
}
return ret;
}
Where ProxifyByCompletion is implemented as:
public static Task<T>[] ProxifyByCompletion<T>(this IEnumerable<Task<T>> tasks)
{
var inputTasks = tasks.ToArray();
var buckets = new TaskCompletionSource<T>[inputTasks.Length];
var results = new Task<T>[inputTasks.Length];
for (int i = 0; i < buckets.Length; i++)
{
buckets[i] = new TaskCompletionSource<T>();
results[i] = buckets[i].Task;
}
int nextTaskIndex = -1;
foreach (var inputTask in inputTasks)
{
inputTask.ContinueWith(completed =>
{
var bucket = buckets[Interlocked.Increment(ref nextTaskIndex)];
if (completed.IsFaulted)
{
Trace.Assert(completed.Exception != null);
bucket.TrySetException(completed.Exception.InnerExceptions);
}
else if (completed.IsCanceled)
{
bucket.TrySetCanceled();
}
else
{
bucket.TrySetResult(completed.Result);
}
}, CancellationToken.None,
TaskContinuationOptions.ExecuteSynchronously, TaskScheduler.Default);
}
return results;
}
And Forget is an empty method to suppress CS4014:
public static void Forget(this Task task) //suppress CS4014
{
}