Develop Reference

Task.WhenAny and SemaphoreSlim class

When using WaitHandle.WaitAny and Semaphore class like the following:
var s1 = new Semaphore(1, 1);
var s2 = new Semaphore(1, 1);
var handles = new [] { s1, s2 };
var index = WaitHandle.WaitAny(handles);
handles[index].Release();
It seems guaranteed that only one semaphore is acquired by WaitHandle.WaitAny.
Is it possible to obtain similar behavior for asynchronous (async/await) code?

I cannot think of a built-in solution. I'd do it like this:
var s1 = new SemaphoreSlim(1, 1);
var s2 = new SemaphoreSlim(1, 1);
var waits = new [] { s1.WaitAsync(), s2.WaitAsync() };
var firstWait = await Task.WhenAny(waits);
//The wait is still running - perform compensation.
if (firstWait == waits[0])
waits[1].ContinueWith(_ => s2.Release());
if (firstWait == waits[1])
waits[0].ContinueWith(_ => s1.Release());
This acquires both semaphores but it immediately releases the one that came second. This should be equivalent. I cannot think of a negative consequence of acquiring a semaphore needlessly (except performance of course).

Here is a generalized implementation of a WaitAnyAsync method, that acquires asynchronously any of the supplied semaphores:
/// <summary>
/// Asynchronously waits to enter any of the semaphores in the specified array.
/// </summary>
public static async Task<SemaphoreSlim> WaitAnyAsync(SemaphoreSlim[] semaphores,
CancellationToken cancellationToken = default)
{
// Fast path
cancellationToken.ThrowIfCancellationRequested();
var acquired = semaphores.FirstOrDefault(x => x.Wait(0));
if (acquired != null) return acquired;
// Slow path
using var cts = CancellationTokenSource.CreateLinkedTokenSource(
cancellationToken);
Task<SemaphoreSlim>[] acquireTasks = semaphores
.Select(async s => { await s.WaitAsync(cts.Token); return s; })
.ToArray();
Task<SemaphoreSlim> acquiredTask = await Task.WhenAny(acquireTasks);
cts.Cancel(); // Cancel all other tasks
var releaseOtherTasks = acquireTasks
.Where(task => task != acquiredTask)
.Select(async task => (await task).Release());
try { await Task.WhenAll(releaseOtherTasks); }
catch (OperationCanceledException) { } // Ignore
catch
{
// Consider any other error (possibly SemaphoreFullException or
// ObjectDisposedException) as a failure, and propagate the exception.
try { (await acquiredTask).Release(); } catch { }
throw;
}
try { return await acquiredTask; }
catch (OperationCanceledException)
{
// Propagate an exception holding the correct CancellationToken
cancellationToken.ThrowIfCancellationRequested();
throw; // Should never happen
}
}
This method becomes increasingly inefficient as the contention gets higher and higher, so I wouldn't recommend using it in hot paths.

Variation of #usr's answer which solved my slightly more general problem (after quite some time going down the rathole of trying to marry AvailableWaitHandle with Task...)
class SemaphoreSlimExtensions
public static Task AwaitButReleaseAsync(this SemaphoreSlim s) =>
s.WaitAsync().ContinueWith(_t -> s.Release(), TaskContinuationOptions.ExecuteSynchronously);
public static bool TryTake(this SemaphoreSlim s) =>
s.Wait(0);
In my use case, the await is just a trigger for synchronous logic that then walks the full set - the TryTake helper is in my case a natural way to handle the conditional acquisition of the semaphore and the processing that's contingent on that.
var sems = new[] { new SemaphoreSlim(1, 1), new SemaphoreSlim(1, 1) };
await Task.WhenAny(from s in sems select s.AwaitButReleaseAsync());
Putting it here as I believe it to be clean, clear and relatively efficient but would be happy to see improvements on it

Parallel.Foreach in dynamic List

I am working on the file uploading to server.I am using Parallel.Foreach to split the task.In my Sample Source
Task.Factory.StartNew (delegate
{
Parallel.ForEach (metaDatas, new ParallelOptions{ MaxDegreeOfParallelism = 3 }, itemModel =>
{
apimanager.ContentConnector.uploadItem (0, itemModel.PhysicalFileName, itemModel.ParentId, itemModel.Size, itemModel.path, fetchDataDelegate, finishedUploadingDataDelegate, failedToUploadDataDelegate);
});
Console.WriteLine ("Upload Completed");
});
in my case metaDatas list is dynamically update.how can i use the latest metaDatas List in Parallel.ForEach.

I met the same problem.
I found a non standard solution that can help.
var buffer = new BlockingCollection<Task>(maxThreads);
// process while download all threads
while (!Queue.IsEmpty || buffer.Count > 0)
{
SomeClass item;
while (Queue.TryDequeue(out item))
{
var localItem = item;
var localTask = new Task(() =>
{
// work with your item
// in this place you can append Queue
buffer.Take(); // free buffer for adding new threads
});
buffer.Add(localTask);
localTask.Start();
}
// in this point Queue is empty,
// but buffer have remaining thread
Task.Delay(50).Wait();
}
So, Queue is ConcurrentQueue<SomeClass>, your dymanic source
It's not perfect solution, but it's work.

How can I make this Observable more reusable?

This is an observable sequence that retrieves paginated data from a web service. Each web service response contains a nextRecordsUrl that indicates where to get the next set of records. What is the best way to convert this Observable to something more reusable?
Web services setup:
var auth = new AuthenticationClient ();
await auth.UsernamePasswordAsync (consumerKey, consumerSecret, userName, password + passwordSecurityToken);
var forceClient = new ForceClient (auth.InstanceUrl, auth.AccessToken, auth.ApiVersion);
The Observable:
var observable = Observable.Create<QueryResult<Account>> (async (IObserver<QueryResult<Account>> o) =>
{
try
{
var queryResult = await forceClient.QueryAsync<Account> ("SELECT Id, Name from Account");
if (queryResult != null)
{
o.OnNext (queryResult);
while (!string.IsNullOrEmpty (queryResult.nextRecordsUrl))
{
queryResult = await forceClient.QueryContinuationAsync<Account> (queryResult.nextRecordsUrl);
if (queryResult != null)
{
o.OnNext (queryResult);
}
}
}
o.OnCompleted ();
}
catch (Exception ex)
{
o.OnError (ex);
}
return () => {};
});
Subscribing to the observable and collecting the results:
var accounts = new List<Account> ();
observable.Subscribe (
observer => accounts.AddRange (observer.records),
ex => Console.WriteLine (ex.Message),
() => {});
EDIT: Using Brandon's solution I can now generate the list of results with Aggregate
List<Account> accounts = await forceClient.QueryPages<Account> ("SELECT Id, Name from Account")
.Aggregate (new List<Account> (), (list, value) =>
{
list.AddRange (value.records);
return list;
});

Believe it or not, the Rx-Expiremental library (also maintained by MS) has an operator for this called Expand. Expand is used to take each element from an observable and run it through a function which produces another observable of the same type. That observable is then flattened in to the original, and each item from that goes through the same process.
Imagine being given a tree node with an observable of child nodes. You could use expand to easily traverse this tree. Since a linked-list is just a constrained version of a tree, and since what you have is effectively a linked list where each node is an observable, you can use expand.
public static IObservable<QueryResult<TResult>> QueryPages<TResult>(this ForceClient forceClient, string query)
{
return Observable.FromAsync(() => forceClient.QueryAsync<TResult>(query))
.Where(QueryResultIsValid)
.Expand(result =>
Observable.FromAsync(() => forceClient.QueryContinuationAsync<TResult>(queryResult.nextRecordsUrl))
.Where(QueryResultIsValid)
);
}
public static bool QueryResultIsValid(QueryResult<TResult> result)
{
return result != null;
}

Is something like this what you are looking for?
public static IObservable<QueryResult<TResult>> QueryPages<TResult>(this ForceClient forceClient, string query)
{
return Observable.Create<QueryResult<T>> (async (observer, token) =>
{
// No need for try/catch. Create() will call OnError if your task fails.
// Also no need for OnCompleted(). Create() calls it when your task completes
var queryResult = await forceClient.QueryAsync<TResult> (query);
while (queryResult != null)
{
observer.OnNext (queryResult);
// check the token *after* we call OnNext
// because if an observer unsubscribes
// it typically occurs during the notification
// e.g. they are using .Take(..) or
// something.
if (string.IsNullOrEmpty(queryResult.nextRecordsUrl) ||
token.IsCancellationRequested)
{
break;
}
queryResult = await forceClient.QueryContinuationAsync<TResult> (queryResult.nextRecordsUrl);
}
// No need to return anything. Just the Task itself is all that Create() wants.
}
});
// Usage:
var forceClient = // ...
var foos = forceClient.QueryPages<Foo>("SELECT A, B, C FROM Foo");
Notice I switched it to the overload that provides a cancellation token so that you can stop fetching pages if the observer unsubscribes (Your original version would have continued fetching pages even though the observer had stopped listening). Also note that the async Create awaits your Task and calls OnError or OnCompleted for you so you do not need to worry about that most of the time.

ReactiveExtensions Observable FromAsync calling twice Function

Ok, Trying to understand Rx, kinda of lost here.
FromAsyncPattern is now deprecated so I took the example from here (section Light up Task with Rx), and it works, I just made a few changes, not using await just wait the observable and subscribing.....
What I don't understand is Why is called Twice the function SumSquareRoots?
var res = Observable.FromAsync(ct => SumSquareRoots(x, ct))
.Timeout(TimeSpan.FromSeconds(5));
res.Subscribe(y => Console.WriteLine(y));
res.Wait();
class Program
{
static void Main(string[] args)
{
Samples();
}
static void Samples()
{
var x = 100000000;
try
{
var res = Observable.FromAsync(ct => SumSquareRoots(x, ct))
.Timeout(TimeSpan.FromSeconds(5));
res.Subscribe(y => Console.WriteLine(y));
res.Wait();
}
catch (TimeoutException)
{
Console.WriteLine("Timed out :-(");
}
}
static Task<double> SumSquareRoots(long count, CancellationToken ct)
{
return Task.Run(() =>
{
var res = 0.0;
Console.WriteLine("Why I'm called twice");
for (long i = 0; i < count; i++)
{
res += Math.Sqrt(i);
if (i % 10000 == 0 && ct.IsCancellationRequested)
{
Console.WriteLine("Noticed cancellation!");
ct.ThrowIfCancellationRequested();
}
}
return res;
});
}
}

The reason that this is calling SumSquareRoots twice is because you're Subscribing twice:
// Subscribes to res
res.Subscribe(y => Console.WriteLine(y));
// Also Subscribes to res, since it *must* produce a result, even
// if that result is then discarded (i.e. Wait doesn't return IObservable)
res.Wait();
Subscribe is the foreach of Rx - just like if you foreach an IEnumerable twice, you could end up doing 2x the work, multiple Subscribes means multiple the work. To undo this, you could use a blocking call that doesn't discard the result:
Console.WriteLine(res.First());
Or, you could use Publish to "freeze" the result and play it back to > 1 subscriber (kind of like how you'd use ToArray in LINQ):
res = res.Publish();
res.Connect();
// Both subscriptions get the same result, SumSquareRoots is only called once
res.Subscribe(Console.WriteLine);
res.Wait();
The general rule you can follow is, that any Rx method that doesn't return IObservable<T> or Task<T> will result in a Subscription(*)
* - Not technically correct. But your brain will feel better if you think of it this way.

Processing a batch of request with Reactive Extension

I'm learning Reactive Extensions, and I've been trying to find out if it's a match for a task like this.
I have a Process() method that processes a batch of requests as a unit of work, and invoking a callback when all requests have completed.
The important thing here is that each request will call the callback either synchronous or asynchronous depending on it's implementation, and the batch processor must be able to handle both.
But no threads are started from the batch processor, any new threads (or other async execution) will be initiated from inside the request handlers if necessary. I don't know if this match the use cases of rx.
My current working code looks (almost) like this:
public void Process(ICollection<IRequest> requests, Action<List<IResponse>> onCompleted)
{
IUnitOfWork uow = null;
try
{
uow = unitOfWorkFactory.Create();
var responses = new List<IResponse>();
var outstandingRequests = requests.Count;
foreach (var request in requests)
{
var correlationId = request.CorrelationId;
Action<IResponse> requestCallback = response =>
{
response.CorrelationId = correlationId;
responses.Add(response);
outstandingRequests--;
if (outstandingRequests != 0)
return;
uow.Commit();
onCompleted(responses);
};
requestProcessor.Process(request, requestCallback);
}
}
catch(Exception)
{
if (uow != null)
uow.Rollback();
}
if (uow != null)
uow.Commit();
}
How would you implement this using rx? Is it reasonable?
Note, that the unit of work is to be committed synchronously even if there are async requests that have not yet returned.

My approach to this is two-step.
First create a general-purpose operator that turns Action<T, Action<R>> into Func<T, IObservable<R>>:
public static class ObservableEx
{
public static Func<T, IObservable<R>> FromAsyncCallbackPattern<T, R>(
this Action<T, Action<R>> call)
{
if (call == null) throw new ArgumentNullException("call");
return t =>
{
var subject = new AsyncSubject<R>();
try
{
Action<R> callback = r =>
{
subject.OnNext(r);
subject.OnCompleted();
};
call(t, callback);
}
catch (Exception ex)
{
return Observable.Throw<R>(ex, Scheduler.ThreadPool);
}
return subject.AsObservable<R>();
};
}
}
Next, turn the call void Process(ICollection<IRequest> requests, Action<List<IResponse>> onCompleted) into IObservable<IResponse> Process(IObservable<IRequest> requests):
public IObservable<IResponse> Process(IObservable<IRequest> requests)
{
Func<IRequest, IObservable<IResponse>> rq2rp =
ObservableEx.FromAsyncCallbackPattern
<IRequest, IResponse>(requestProcessor.Process);
var query = (
from rq in requests
select rq2rp(rq)).Concat();
var uow = unitOfWorkFactory.Create();
var subject = new ReplaySubject<IResponse>();
query.Subscribe(
r => subject.OnNext(r),
ex =>
{
uow.Rollback();
subject.OnError(ex);
},
() =>
{
uow.Commit();
subject.OnCompleted();
});
return subject.AsObservable();
}
Now, not only does this run the processing async, but it also ensures the correct order of the results.
In fact, since you are starting with a collection, you could even do this:
var rqs = requests.ToObservable();
var rqrps = rqs.Zip(Process(rqs),
(rq, rp) => new
{
Request = rq,
Response = rp,
});
Then you would have an observable that pairs up each request/response without the need for a CorrelationId property.
I hope this helps.

This is part of the genius of Rx, as you're free to return results either synchronously or asynchronously:
public IObservable<int> AddNumbers(int a, int b) {
return Observable.Return(a + b);
}
public IObservable<int> AddNumbersAsync(int a, int b) {
return Observable.Start(() => a + b, Scheduler.NewThread);
}
They both have the IObservable type, so they work identically. If you want to find out when all IObservables complete, Aggregate will do this, as it will turn 'n' items in an Observable into 1 item that is returned at the end:
IObservable<int> listOfObservables[];
listObservables.ToObservable()
.Merge()
.Aggregate(0, (acc, x) => acc+1)
.Subscribe(x => Console.WriteLine("{0} items were run", x));