Develop Reference

Recursively call a method with the same thread

I have the following method:
public async Task ScrapeObjects(int page = 1)
{
try
{
while (!isObjectSearchCompleted)
{
..do calls..
}
}
catch (HttpRequestException ex)
{
Thread.Sleep(TimeSpan.FromSeconds(60));
ScrapeObjects(page);
Log.Fatal(ex, ex.Message);
}
}
I call this long running method async and I don't wait for it to finish. Thing is that an exception my occur and in that case I want to handle it. But then I want to start from where I left and with the same thread. At the current state a new thread gets used when I recursively call the method after handling the exception. I would like to keep using the same thread. Is there a way to do so? Thank you!

You probably need to move the try/catch block inside the while loop, and add a counter with the errors occurred, to bail out in case of continuous faulted attempts.
public async Task ScrapeObjects()
{
int failedCount = 0;
int page = 1;
while (!isObjectSearchCompleted)
{
try
{
//..do calls..
}
catch (HttpRequestException ex)
{
failedCount++;
if (failedCount < 3)
{
Log.Info(ex, ex.Message);
await Task.Delay(TimeSpan.FromSeconds(60));
}
else
{
Log.Fatal(ex, ex.Message);
throw; // or return;
}
}
}
}
As a side note it is generally better to await Task.Delay instead of Thread.Sleep inside asynchronous methods, to avoid blocking a thread without a reason.

One simple question before you read the long answer below:
Why you need the same thread? Are you accessing thread static / contextual data?
If yes, there will be ways to solve that easily than limiting your tasks to run on the same thread.
How to limit tasks to run on a single thread
As long as you use async calls on the default synchronization context, and as soon as the code is resumed from an await, it is possible that the thread can change after an await. This is because the default context schedules tasks to the next available thread in the thread pool. Like in the below case, before can be different from after:
public async Task ScrapeObjects(int page = 1)
{
var before = Thread.CurrentThread.ManagedThreadId;
await Task.Delay(1000);
var after = Thread.CurrentThread.ManagedThreadId;
}
The only reliable way to guarantee that your code could come back on the same thread is to schedule your async code onto a single threaded synchronization context:
class SingleThreadSynchronizationContext : SynchronizationContext
{
private readonly BlockingCollection<Action> _actions = new BlockingCollection<Action>();
private readonly Thread _theThread;
public SingleThreadSynchronizationContext()
{
_theThread = new Thread(DoWork);
_theThread.IsBackground = true;
_theThread.Start();
}
public override void Send(SendOrPostCallback d, object state)
{
// Send requires run the delegate immediately.
d(state);
}
public override void Post(SendOrPostCallback d, object state)
{
// Schedule the action by adding to blocking collection.
_actions.Add(() => d(state));
}
private void DoWork()
{
// Keep picking up actions to run from the collection.
while (!_actions.IsAddingCompleted)
{
try
{
var action = _actions.Take();
action();
}
catch (InvalidOperationException)
{
break;
}
}
}
}
And you need to schedule ScrapeObjects to the custom context:
SynchronizationContext.SetSynchronizationContext(new SingleThreadSynchronizationContext());
await Task.Factory.StartNew(
() => ScrapeObjects(),
CancellationToken.None,
TaskCreationOptions.DenyChildAttach | TaskCreationOptions.LongRunning,
TaskScheduler.FromCurrentSynchronizationContext()
).Unwrap();
By doing that, all your async code shall be scheduled to the same context, and run by the thread on that context.
However
This is typically dangerous, as you suddenly lose the ability to use the thread pool. If you block the thread, the entire async operation is blocked, meaning you will have deadlocks.

Managing task cancellation and completion on a concurrent thread

[ This question needs to be reimagined. One of my thread queues MUST run on an STA thread, and the code below does not accommodate that. In particular it seems Task<> chooses its own thread and that just is not going to work for me. ]
I have a task queue (BlockingCollection) that I'm running through on a dedicated thread. That queue receives a series of Task<> objects that it runs sequentially within that thread via a while loop.
I need a means of Cancelling that series of tasks, and a means of knowing that the tasks are all complete. I have not been able to figure out how to do this.
Here's a fragment of my queuing class. ProcessQueue is run on a separate thread from main. QueueJob calls occur on the main thread.
using Job = Tuple<Task<bool>, string>;
public class JobProcessor
{
private readonly BlockingCollection<Job> m_queue = new BlockingCollection<Job>();
volatile bool cancel_queue = false;
private bool ProcessQueue()
{
while (true)
{
if (m_queue.IsAddingCompleted)
break;
Job tuple;
if (!m_queue.TryTake(out tuple, Timeout.Infinite))
break;
var task = tuple.Item1;
var taskName = tuple.Item2;
try
{
Console.WriteLine("Task {0}::{1} starting", this.name, taskName);
task.RunSynchronously();
Console.WriteLine("Task {0}::{1} completed", this.name, taskName);
}
catch (Exception e)
{
string message = e.Message;
}
if (cancel_queue) // CANCEL BY ERASING TASKS AND NOT RUNNING.
{
while (m_queue.TryTake(out tuple))
{
}
}
} // while(true)
return true;
}
public Task<bool> QueueJob(Func<bool> input)
{
var task = new Task<bool>(input);
try
{
m_queue.Add(Tuple.Create(task, input.Method.Name));
}
catch (InvalidOperationException)
{
Task<bool> dummy = new Task<bool>(() => false);
dummy.Start();
return dummy;
}
return task;
}
Here are the functions that trouble me:
public void ClearQueue()
{
cancel_queue = true;
// wait for queue to become empty. HOW?
cancel_queue = false;
}
public void WaitForCompletion()
{
// wait for all tasks to be completed.
// not sufficient to wait for empty queue because the last task
// must also execute and finish. HOW?
}
}
Here is some usage:
class SomeClass
{
void Test()
{
JobProcessor jp = new JobProcessor();
// launch Processor loop on separate thread... code not shown.
// send a bunch of jobs via QueueJob... code not show.
// launch dialog... code not shown.
if (dialog_result == Result.Cancel)
jp.ClearQueue();
if (dialog_result == Result.Proceed)
jp.WaitForCompletion();
}
}
The idea is after the work is completed or cancelled, new work may be posted. In general though, new work may come in asynchronously. WaitForCompletion might in fact be "when all work is done, inform the user and then do other stuff", so it doesn't strictly have to be a synchronous function call like above, but I can't figure how to make these happen.
(One further complication, I expect to have several queues that interact. While I am careful to keep things parallelized in a way to prevent deadlocks, I am not confident what happens when cancellation is introduced into the mix, but this is probably beyond scope for this question.)

WaitForCompletion() sounds easy enough. Create a semaphore or event, create a task whose only action is to signal the semaphore, queue up the task, wait on the semaphore.
When the thread finishes the last 'real' task, the semaphore task will be run and so the thread that called WaitForCompletion will become ready/running:)
Would not a similar approach work for cancellation? Have a very high priority thread that you create/signal that drains the queue of all pending jobs, disposing them, queueing up the semaphore task and waiting for the 'last task done' signal?

Calling TaskCompletionSource.SetResult in a non blocking manner

I've discovered that TaskCompletionSource.SetResult(); invokes the code awaiting the task before returning. In my case that result in a deadlock.
This is a simplified version that is started in an ordinary Thread
void ReceiverRun()
while (true)
{
var msg = ReadNextMessage();
TaskCompletionSource<Response> task = requests[msg.RequestID];
if(msg.Error == null)
task.SetResult(msg);
else
task.SetException(new Exception(msg.Error));
}
}
The "async" part of the code looks something like this.
await SendAwaitResponse("first message");
SendAwaitResponse("second message").Wait();
The Wait is actually nested inside non-async calls.
The SendAwaitResponse(simplified)
public static Task<Response> SendAwaitResponse(string msg)
{
var t = new TaskCompletionSource<Response>();
requests.Add(GetID(msg), t);
stream.Write(msg);
return t.Task;
}
My assumption was that the second SendAwaitResponse would execute in a ThreadPool thread but it continues in the thread created for ReceiverRun.
Is there anyway to set the result of a task without continuing its awaited code?
The application is a console application.

I've discovered that TaskCompletionSource.SetResult(); invokes the code awaiting the task before returning. In my case that result in a deadlock.
Yes, I have a blog post documenting this (AFAIK it's not documented on MSDN). The deadlock happens because of two things:
There's a mixture of async and blocking code (i.e., an async method is calling Wait).
Task continuations are scheduled using TaskContinuationOptions.ExecuteSynchronously.
I recommend starting with the simplest possible solution: removing the first thing (1). I.e., don't mix async and Wait calls:
await SendAwaitResponse("first message");
SendAwaitResponse("second message").Wait();
Instead, use await consistently:
await SendAwaitResponse("first message");
await SendAwaitResponse("second message");
If you need to, you can Wait at an alternative point further up the call stack (not in an async method).
That's my most-recommended solution. However, if you want to try removing the second thing (2), you can do a couple of tricks: either wrap the SetResult in a Task.Run to force it onto a separate thread (my AsyncEx library has *WithBackgroundContinuations extension methods that do exactly this), or give your thread an actual context (such as my AsyncContext type) and specify ConfigureAwait(false), which will cause the continuation to ignore the ExecuteSynchronously flag.
But those solutions are much more complex than just separating the async and blocking code.
As a side note, take a look at TPL Dataflow; it sounds like you may find it useful.

As your app is a console app, it runs on the default synchronization context, where the await continuation callback will be called on the same thread the awaiting task has become completed on. If you want to switch threads after await SendAwaitResponse, you can do so with await Task.Yield():
await SendAwaitResponse("first message");
await Task.Yield();
// will be continued on a pool thread
// ...
SendAwaitResponse("second message").Wait(); // so no deadlock
You could further improve this by storing Thread.CurrentThread.ManagedThreadId inside Task.Result and comparing it to the current thread's id after the await. If you're still on the same thread, do await Task.Yield().
While I understand that SendAwaitResponse is a simplified version of your actual code, it's still completely synchronous inside (the way you showed it in your question). Why would you expect any thread switch in there?
Anyway, you probably should redesign your logic the way it doesn't make assumptions about what thread you are currently on. Avoid mixing await and Task.Wait() and make all of your code asynchronous. Usually, it's possible to stick with just one Wait() somewhere on the top level (e.g. inside Main).
[EDITED] Calling task.SetResult(msg) from ReceiverRun actually transfers the control flow to the point where you await on the task - without a thread switch, because of the default synchronization context's behavior. So, your code which does the actual message processing is taking over the ReceiverRun thread. Eventually, SendAwaitResponse("second message").Wait() is called on the same thread, causing the deadlock.
Below is a console app code, modeled after your sample. It uses await Task.Yield() inside ProcessAsync to schedule the continuation on a separate thread, so the control flow returns to ReceiverRun and there's no deadlock.
using System;
using System.Collections.Concurrent;
using System.Threading;
using System.Threading.Tasks;
namespace ConsoleApplication
{
class Program
{
class Worker
{
public struct Response
{
public string message;
public int threadId;
}
CancellationToken _token;
readonly ConcurrentQueue<string> _messages = new ConcurrentQueue<string>();
readonly ConcurrentDictionary<string, TaskCompletionSource<Response>> _requests = new ConcurrentDictionary<string, TaskCompletionSource<Response>>();
public Worker(CancellationToken token)
{
_token = token;
}
string ReadNextMessage()
{
// using Thread.Sleep(100) for test purposes here,
// should be using ManualResetEvent (or similar synchronization primitive),
// depending on how messages arrive
string message;
while (!_messages.TryDequeue(out message))
{
Thread.Sleep(100);
_token.ThrowIfCancellationRequested();
}
return message;
}
public void ReceiverRun()
{
LogThread("Enter ReceiverRun");
while (true)
{
var msg = ReadNextMessage();
LogThread("ReadNextMessage: " + msg);
var tcs = _requests[msg];
tcs.SetResult(new Response { message = msg, threadId = Thread.CurrentThread.ManagedThreadId });
_token.ThrowIfCancellationRequested(); // this is how we terminate the loop
}
}
Task<Response> SendAwaitResponse(string msg)
{
LogThread("SendAwaitResponse: " + msg);
var tcs = new TaskCompletionSource<Response>();
_requests.TryAdd(msg, tcs);
_messages.Enqueue(msg);
return tcs.Task;
}
public async Task ProcessAsync()
{
LogThread("Enter Worker.ProcessAsync");
var task1 = SendAwaitResponse("first message");
await task1;
LogThread("result1: " + task1.Result.message);
// avoid deadlock for task2.Wait() with Task.Yield()
// comment this out and task2.Wait() will dead-lock
if (task1.Result.threadId == Thread.CurrentThread.ManagedThreadId)
await Task.Yield();
var task2 = SendAwaitResponse("second message");
task2.Wait();
LogThread("result2: " + task2.Result.message);
var task3 = SendAwaitResponse("third message");
// still on the same thread as with result 2, no deadlock for task3.Wait()
task3.Wait();
LogThread("result3: " + task3.Result.message);
var task4 = SendAwaitResponse("fourth message");
await task4;
LogThread("result4: " + task4.Result.message);
// avoid deadlock for task5.Wait() with Task.Yield()
// comment this out and task5.Wait() will dead-lock
if (task4.Result.threadId == Thread.CurrentThread.ManagedThreadId)
await Task.Yield();
var task5 = SendAwaitResponse("fifth message");
task5.Wait();
LogThread("result5: " + task5.Result.message);
LogThread("Leave Worker.ProcessAsync");
}
public static void LogThread(string message)
{
Console.WriteLine("{0}, thread: {1}", message, Thread.CurrentThread.ManagedThreadId);
}
}
static void Main(string[] args)
{
Worker.LogThread("Enter Main");
var cts = new CancellationTokenSource(5000); // cancel after 5s
var worker = new Worker(cts.Token);
Task receiver = Task.Run(() => worker.ReceiverRun());
Task main = worker.ProcessAsync();
try
{
Task.WaitAll(main, receiver);
}
catch (Exception e)
{
Console.WriteLine("Exception: " + e.Message);
}
Worker.LogThread("Leave Main");
Console.ReadLine();
}
}
}
This is not much different from doing Task.Run(() => task.SetResult(msg)) inside ReceiverRun. The only advantage I can think of is that you have an explicit control over when to switch threads. This way, you can stay on the same thread for as long as possible (e.g., for task2, task3, task4, but you still need another thread switch after task4 to avoid a deadlock on task5.Wait()).
Both solutions would eventually make the thread pool grow, which is bad in terms of performance and scalability.
Now, if we replace task.Wait() with await task everywhere inside ProcessAsync in the above code, we will not have to use await Task.Yield and there still will be no deadlocks. However, the whole chain of await calls after the 1st await task1 inside ProcessAsync will actually be executed on the ReceiverRun thread. As long as we don't block this thread with other Wait()-style calls and don't do a lot of CPU-bound work as we're processing messages, this approach might work OK (asynchronous IO-bound await-style calls still should be OK, and they may actually trigger an implicit thread switch).
That said, I think you'd need a separate thread with a serializing synchronization context installed on it for processing messages (similar to WindowsFormsSynchronizationContext). That's where your asynchronous code containing awaits should run. You'd still need to avoid using Task.Wait on that thread. And if an individual message processing takes a lot of CPU-bound work, you should use Task.Run for such work. For async IO-bound calls, you could stay on the same thread.
You may want to look at ActionDispatcher/ActionDispatcherSynchronizationContext from #StephenCleary's
Nito Asynchronous Library for your asynchronous message processing logic. Hopefully, Stephen jumps in and provides a better answer.

"My assumption was that the second SendAwaitResponse would execute in a ThreadPool thread but it continues in the thread created for ReceiverRun."
It depends entirely on what you do within SendAwaitResponse. Asynchrony and concurrency are not the same thing.
Check out: C# 5 Async/Await - is it *concurrent*?

A little late to the party, but here's my solution which i think is added value.
I've been struggling with this also, i've solved it by capturing the SynchronizationContext on the method that is awaited.
It would look something like:
// just a default sync context
private readonly SynchronizationContext _defaultContext = new SynchronizationContext();
void ReceiverRun()
{
while (true) // <-- i would replace this with a cancellation token
{
var msg = ReadNextMessage();
TaskWithContext<TResult> task = requests[msg.RequestID];
// if it wasn't a winforms/wpf thread, it would be null
// we choose our default context (threadpool)
var context = task.Context ?? _defaultContext;
// execute it on the context which was captured where it was added. So it won't get completed on this thread.
context.Post(state =>
{
if (msg.Error == null)
task.TaskCompletionSource.SetResult(msg);
else
task.TaskCompletionSource.SetException(new Exception(msg.Error));
});
}
}
public static Task<Response> SendAwaitResponse(string msg)
{
// The key is here! Save the current synchronization context.
var t = new TaskWithContext<Response>(SynchronizationContext.Current);
requests.Add(GetID(msg), t);
stream.Write(msg);
return t.TaskCompletionSource.Task;
}
// class to hold a task and context
public class TaskWithContext<TResult>
{
public SynchronizationContext Context { get; }
public TaskCompletionSource<TResult> TaskCompletionSource { get; } = new TaskCompletionSource<Response>();
public TaskWithContext(SynchronizationContext context)
{
Context = context;
}
}

AsyncLock for multiple methods configuration

What I need to achieve is:
When MasterAsync is executing, HumbleSlave1Async and HumbleSlave2Async can NOT.
Vice Versa- When one or both of Slaves are executing, MasterAsync can NOT.
Difficult part- Slaves can NOT block each other.
(Link to used AsyncLock).
private async Task MasterAsync()
{
using (await _asyncLock.LockAsync())
{
await Task.Delay(2000);
}
}
private async Task HumbleSlave1Async()
{
using (await _asyncLock.LockAsync())
{
await Task.Delay(5000);
}
}
private async Task HumbleSlave2Async()
{
using (await _asyncLock.LockAsync())
{
await Task.Delay(5000);
}
}
I am not sure how to solve it, was thinking about use of two different locks for each slave in MasterAsync but then one lock would be in another:
private async Task MasterAsync()
{
using (await _asyncLock1.LockAsync())
{
using (await _asyncLock2.LockAsync())
{
await Task.Delay(2000);
}
}
}
private async Task HumbleSlave1Async()
{
using (await _asyncLock1.LockAsync())
{
await Task.Delay(5000);
}
}
private async Task HumbleSlave2Async()
{
using (await _asyncLock2.LockAsync())
{
await Task.Delay(5000);
}
}
Does it make sense and is it safe(deadlocks, etc...) especially when I used AsyncLock ?

When MasterAsync is executing, HumbleSlave1Async and HumbleSlave2Async can NOT.
Vice Versa- When one or both of Slaves are executing, MasterAsync can NOT.
Difficult part- Slaves can NOT block each other.
First, double-check whether you really want this. The majority of the time, a restructuring of the code responsibilities will simplify what synchronization you need (and is usually more efficient, too).
That said, your scenario fits a reader/writer lock. A RWL is a lock that can be taken two different ways, as a "writer" (which does not allow any other locks at the same time) or as a "reader" (which allows other readers but not writers). Stephen Toub has an async-compatible one here and I have one as part of my AsyncEx library.
Update: Example code:
private readonly AsyncReaderWriterLock _lock = new AsyncReaderWriterLock();
private async Task MasterAsync()
{
using (await _lock.WriterLockAsync())
{
await Task.Delay(2000);
}
}
private async Task HumbleSlave1Async()
{
using (await _lock.ReaderLockAsync())
{
await Task.Delay(5000);
}
}
private async Task HumbleSlave2Async()
{
using (await _lock.ReaderLockAsync())
{
await Task.Delay(5000);
}
}

A possible problem with your approach is that when HumbleSlave2Async is executing, and MasterAsync has been called, acquired asyncLock1 and is awaiting asyncLock2, you won't be able to execute HumbleSlave1Async (because asyncLock1 is taken by MasterAsync). So, your condition #3 will not be satisfied.
Maybe you should use something like AsyncManualResetEvent to make this work.

This is certainly not going to deadlock because you acquire locks always in the same order:
_asyncLock1 (or you don't acquire it for HumbleSlave2Async)
_asyncLock2 (or you don't acquire it for HumbleSlave1Async)
A total lock order guarantees freedom of deadlocks.

c# lock and listen to CancellationToken

I want to use lock or a similar synchronization to protect a critical section. At the same time I want to listen to a CancellationToken.
Right now I'm using a mutex like this, but mutex doesn't have as good performance. Can I use any of other synchronization classes (including the new .Net 4.0) instead of the mutex?
WaitHandle.WaitAny(new[] { CancelToken.WaitHandle, _mutex});
CancelToken.ThrowIfCancellationRequested();

Take a look at the new .NET 4.0 Framework feature SemaphoreSlim Class. It provides SemaphoreSlim.Wait(CancellationToken) method.
Blocks the current thread until it can enter the SemaphoreSlim, while
observing a CancellationToken
From some point of view using Semaphore in such simple case could be an overhead because initially it was designed to provide an access for multiple threads, but perhaps you might find it useful.
EDIT: The code snippet
CancellationToken token = new CancellationToken();
SemaphoreSlim semaphore = new SemaphoreSlim(1,1);
bool tokenCanceled = false;
try {
try {
// block section entrance for other threads
semaphore.Wait(token);
}
catch (OperationCanceledException) {
// The token was canceled and the semaphore was NOT entered...
tokenCanceled = true;
}
// critical section code
// ...
if (token.IsCancellationRequested)
{
// ...
}
}
finally {
if (!tokenCanceled)
semaphore.Release();
}

private object _lockObject = new object();
lock (_lockObject)
{
// critical section
using (token.Register(() => token.ThrowIfCancellationRequested())
{
// Do something that might need cancelling.
}
}
Calling Cancel() on a token will result in the ThrowIfCancellationRequested() being invoked as that was what is hooked up to the Register callback. You can put whatever cancellation logic you want in here. This approach is great because you can cancel blocking calls by forcing the conditions that will cause the call to complete.
ThrowIfCancellationRequested throws a OperationCanceledException. You need to handle this on the calling thread or your whole process could be brought down. A simple way of doing this is by starting your task using the Task class which will aggregate all the exceptions up for you to handle on the calling thread.
try
{
var t = new Task(() => LongRunningMethod());
t.Start();
t.Wait();
}
catch (AggregateException ex)
{
ex.Handle(x => true); // this effectively swallows any exceptions
}
Some good stuff here covering co-operative cancellation

You can use Monitor.TryEnter with timeout to wait for the lock and check periodically for cancellation.
private bool TryEnterSyncLock(object syncObject)
{
while(!Monitor.TryEnter(syncObject, TimeSpan.FromMilliseconds(100)))
{
if (cts_.IsCancellationRequested)
return false;
}
return true;
}
Note that I would not recommend this in high contention situations as it can impact performance. I would use it as a safety mechanism against deadlocks in case you cannot use SemaphoreSlim as it has different same thread re-entrancy semantics than Monitor.Enter.
After returning true, lock on syncObject has to be released using Monitor.Exit.