Develop Reference

Add delay to parallel API call

I'm using Polly to make parallel API calls. The server however can't process more than 25 calls per second and so I'm wondering if there is a way to add a 1s delay after each batch of 25 calls?
var policy = Policy
.Handle<HttpRequestException>()
.RetryAsync(3);
foreach (var mediaItem in uploadedMedia)
{
var mediaRequest = new HttpRequestMessage { *** }
async Task<string> func()
{
var response = await client.SendAsync(mediaRequest);
return await response.Content.ReadAsStringAsync();
}
tasks.Add(policy.ExecuteAsync(() => func()));
}
await Task.WhenAll(tasks);
I added a count as per the suggestion below but doesn't seem to work
foreach (var mediaItem in uploadedMedia.Items)
{
var mediaRequest = new HttpRequestMessage
{
RequestUri = new Uri($"https://u48ydao1w4.execute-api.ap-southeast-2.amazonaws.com/test/downloads/thumbnails/{mediaItem.filename.S}"),
Method = HttpMethod.Get,
Headers = {
{ "id-token", id_Token },
{ "access-token", access_Token }
}
};
async Task<string> func()
{
if (count == 24)
{
Thread.Sleep(1000);
count = 0;
}
var response = await client.SendAsync(mediaRequest);
count++;
return await response.Content.ReadAsStringAsync();
}
tasks.Add(policy.ExecuteAsync(() => func()));
}
await Task.WhenAll(tasks);
foreach (var t in tasks)
{
var postResponse = await t;
urls.Add(postResponse);
}

Just scanned quickly over the code and perhaps another similar solution would be to add a Thread.Sleep(calculatedDelay):
foreach (var mediaItem in uploadedMedia.Items)
{
Thread.Sleep(calculatedDelay);
var mediaRequest = new HttpRequestMessage
Where calculatedDelay is some value based on 1000/25.
However I feel you would need a better solution than putting in a delay of some specified value as you cannot be sure of overhead delays issues in transferring data. Also you dont indicate what happens when you reach the 25+ limit, how does the server respond.. do you get an error or is it handled more elegantly? Here is perhaps the area where you can find a more reliable solution?

There are many ways to do this, however it's fairly easy to write a simple thread safe reusable async rate limiter.
The advantages with the async approach, it doesn't block thread pool threads, it's fairly efficient, and would work well in existing async workflows and pipelines like TPL Dataflow, and Reactive Extensions.
Example
// 3 calls every 3 seconds as an example
var rateLimiter = new RateLimiter(3, TimeSpan.FromSeconds(3));
// create some work
var task1 = Task.Run(async () =>
{
for (var i = 0; i < 5; i++)
{
await rateLimiter.WaitAsync();
Console.WriteLine($"{Thread.CurrentThread.ManagedThreadId} : {DateTime.Now}");
}
}
);
var task2 = Task.Run(async () =>
{
for (var i = 0; i < 5; i++)
{
await rateLimiter.WaitAsync();
Console.WriteLine($"{Thread.CurrentThread.ManagedThreadId} : {DateTime.Now}");
}
}
);
await Task.WhenAll(task1, task2);
Output
4 : 10/25/2020 05:16:15
5 : 10/25/2020 05:16:15
4 : 10/25/2020 05:16:15
5 : 10/25/2020 05:16:18
5 : 10/25/2020 05:16:18
5 : 10/25/2020 05:16:18
5 : 10/25/2020 05:16:21
5 : 10/25/2020 05:16:21
5 : 10/25/2020 05:16:21
4 : 10/25/2020 05:16:24
Full Demo Here
Usage
private RateLimiter _rateLimiter = new RateLimiter(25 , TimeSpan.FromSeconds(1));
...
async Task<string> func()
{
await _rateLimiter.WaitAsync();
var response = await client.SendAsync(mediaRequest);
return await response.Content.ReadAsStringAsync();
}
Class
public class RateLimiter
{
private readonly CancellationToken _cancellationToken;
private readonly TimeSpan _timeSpan;
private bool _isProcessing;
private readonly int _count;
private readonly Queue<DateTime> _completed = new Queue<DateTime>();
private readonly Queue<TaskCompletionSource<bool>> _waiting = new Queue<TaskCompletionSource<bool>>();
private readonly object _sync = new object();
public RateLimiter(int count, TimeSpan timeSpan, CancellationToken cancellationToken = default)
{
_count = count;
_timeSpan = timeSpan;
_cancellationToken = cancellationToken;
}
private void Cleanup()
{
// if the cancellation was request, we need to throw on all waiting items
while (_cancellationToken.IsCancellationRequested && _waiting.Any())
if (_waiting.TryDequeue(out var item))
item.TrySetCanceled();
_waiting.Clear();
_completed.Clear();
_isProcessing = false;
}
private async Task ProcessAsync()
{
try
{
while (true)
{
_cancellationToken.ThrowIfCancellationRequested();
var time = DateTime.Now - _timeSpan;
lock (_sync)
{
// remove anything out of date from the queue
while (_completed.Any() && _completed.Peek() < time)
_completed.TryDequeue(out _);
// signal waiting tasks to process
while (_completed.Count < _count && _waiting.Any())
{
if (_waiting.TryDequeue(out var item))
item.TrySetResult(true);
_completed.Enqueue(DateTime.Now);
}
if (!_waiting.Any() && !_completed.Any())
{
Cleanup();
break;
}
}
var delay = (_completed.Peek() - time) + TimeSpan.FromMilliseconds(20);
if (delay.Ticks > 0)
await Task.Delay(delay, _cancellationToken);
Console.WriteLine(delay);
}
}
catch (OperationCanceledException)
{
lock (_sync)
Cleanup();
}
}
public ValueTask WaitAsync()
{
// ReSharper disable once InconsistentlySynchronizedField
_cancellationToken.ThrowIfCancellationRequested();
lock (_sync)
{
try
{
if (_completed.Count < _count && !_waiting.Any())
{
_completed.Enqueue(DateTime.Now);
return new ValueTask();
}
var tcs = new TaskCompletionSource<bool>();
_waiting.Enqueue(tcs);
return new ValueTask(tcs.Task);
}
finally
{
if (!_isProcessing)
{
_isProcessing = true;
_ = ProcessAsync();
}
}
}
}
}
Note 1 : It would be optimal to use this with a max degree of parallelism.
Note 2 : Although I tested this, I only wrote it for this answer as a novel solution.

The Polly library currently lacks a rate-limiting policy (requests/time). Fortunately this functionality is relatively easy to implement using a SemaphoreSlim. The trick to make the rate-limiting happen is to configure the capacity of the semaphore equal to the dividend (requests), and delay the Release of the semaphore for a time span equal to the divisor (time), after acquiring the semaphore. This way the rate limit will be applied consistently to any possible time window.
Update: I realized that the Polly library is extensible, and allows to implement custom policies with custom functionality. So I'm scraping my original suggestion in favor of the custom RateLimitAsyncPolicy class below:
public class RateLimitAsyncPolicy : AsyncPolicy
{
private readonly SemaphoreSlim _semaphore;
private readonly TimeSpan _timeUnit;
public RateLimitAsyncPolicy(int maxOperationsPerTimeUnit, TimeSpan timeUnit)
{
// Arguments validation omitted
_semaphore = new SemaphoreSlim(maxOperationsPerTimeUnit);
_timeUnit = timeUnit;
}
protected async override Task<TResult> ImplementationAsync<TResult>(
Func<Context, CancellationToken, Task<TResult>> action,
Context context,
CancellationToken cancellationToken,
bool continueOnCapturedContext)
{
await _semaphore.WaitAsync(cancellationToken)
.ConfigureAwait(continueOnCapturedContext);
ScheduleSemaphoreRelease();
return await action(context, cancellationToken).ConfigureAwait(false);
}
private async void ScheduleSemaphoreRelease()
{
await Task.Delay(_timeUnit);
_semaphore.Release();
}
}
This policy ensures that no more than maxOperationsPerTimeUnit operations will be started during any time window of timeUnit size. The duration of the operations is not taken into account. In other words no restriction is imposed on how many operations can be running concurrently at any given moment. This restriction can be optionally imposed by the BulkheadAsync policy. Combining these two policies (the RateLimitAsyncPolicy and the BulkheadAsync) is possible, as shown in the example below:
var policy = Policy.WrapAsync
(
Policy
.Handle<HttpRequestException>()
.RetryAsync(retryCount: 3),
new RateLimitAsyncPolicy(
maxOperationsPerTimeUnit: 25, timeUnit: TimeSpan.FromSeconds(1)),
Policy.BulkheadAsync( // Optional
maxParallelization: 25, maxQueuingActions: Int32.MaxValue)
);
The order is important only for the RetryAsync policy, that must be placed first for a reason explained in the documentation:
BulkheadPolicy: Usually innermost unless wraps a final TimeoutPolicy. Certainly inside any WaitAndRetry. The Bulkhead intentionally limits the parallelization. You want that parallelization devoted to running the delegate, not occupied by waits for a retry.
Similarly the RateLimitPolicy must follow the Retry, so that each retry to be considered an independent operation, and to count towards the rate limit.

You should use Microsoft's Reactive Framework (aka Rx) - NuGet System.Reactive and add using System.Reactive.Linq; - then you can do this:
HttpRequestMessage MakeMessage(MediaItem mi) => new HttpRequestMessage
{
RequestUri = new Uri($"https://u48ydao1w4.execute-api.ap-southeast-2.amazonaws.com/test/downloads/thumbnails/{mi.filename}"),
Method = HttpMethod.Get,
Headers = {
{ "id-token", id_Token },
{ "access-token", access_Token }
}
};
var urls = await
uploadedMedia
.Items
.ToObservable()
.Buffer(24)
.Zip(Observable.Timer(TimeSpan.Zero, TimeSpan.FromSeconds(1.0)), (mrs, _) => mrs)
.SelectMany(mrs => mrs.ToObservable().SelectMany(mr => Observable.FromAsync(() => client.SendAsync(MakeMessage(mr)))))
.SelectMany(x => Observable.FromAsync(() => x.Content.ReadAsStringAsync()))
.ToList();
I haven't been able to test it, but it should be fairly close.

C# .NET CORE: cancell other tasks if one returned some kind of result

How to cancel all tasks, if one of them return i.e. false (bool) result?
Is it possible to identify which task returned a result?
class Program
{
private static Random _rnd = new Random();
static void Main(string[] args)
{
var tasksCounter = _rnd.Next(4, 7);
var cts = new CancellationTokenSource();
var tasks = new Task<bool>[tasksCounter];
for (int i = 0; i < tasks.Length; i++)
{
tasks[i] = CreateTask(cts);
}
Console.WriteLine("Waiting..");
Task.WaitAny(tasks);
Console.WriteLine("Done!");
Console.ReadKey();
}
private static Task<bool> CreateTask(CancellationTokenSource cts)
{
return Task.Factory.StartNew(TaskAction, cts.Token).Unwrap();
}
private static async Task<bool> TaskAction()
{
var delay = _rnd.Next(2, 5);
await Task.Delay(delay * 1000);
var taskResult = _rnd.Next(10) < 4;
return await Task.FromResult(taskResult);
}
}
I tried to use Task.WaitAll, Task.WaitAny etc, but none of these methods provide useful (in my case) functionality.

Edit:
As #ckuri stated in the comments, it would be easier to leverage the already existing properties of Task instead of writing a custom result class. Adjusted my answer accordingly.
One solution would be to check for the result in the CreateTask() method and pass a CancellationToken into your TaskAction() method:
private static Task<bool> CreateTask(CancellationTokenSource cts)
{
return Task.Run(async () =>
{
var result = await TaskAction(cts.Token);
// If result is false, cancel all tasks
if (!result)
cts.Cancel();
return result;
});
}
private static async Task<bool> TaskAction(CancellationToken token)
{
// Check for cancellation
token.ThrowIfCancellationRequested();
var delay = Rnd.Next(2, 5);
// Pass the cancellation token to Task.Delay()
await Task.Delay(delay * 1000, token);
var taskResult = Rnd.Next(10) < 4;
// Check for cancellation
token.ThrowIfCancellationRequested();
return taskResult;
}
Now you can do something like this in your Main method to receive all tasks which did not get cancelled:
try
{
// Wait for all tasks inside a try catch block because `WhenAll` throws a `AggregationException`
// containing a `TaskCanceledException`, if the token gets canceled
await Task.WhenAll(tasks);
}
catch { }
var tasksWithResult = tasks.Where(t => !t.IsCanceled).ToList();

I don't have an interpreter on hand, so excuse any mistakes. Your Task needs access to the token if you want to cancel all tasks from the task itself. See https://learn.microsoft.com/en-us/dotnet/api/system.threading.cancellationtokensource?view=netcore-3.0 for an example. I.e. pass it as an argument to the task so one task can cancel the other tasks.
private static async Task<bool> TaskAction(CancellationTokenSource cts)
{
var delay = _rnd.Next(2, 5);
await Task.Delay(delay * 1000);
var taskResult = _rnd.Next(10) < 4;
if (!taskResult)
cts.Cancel()
return await Task.FromResult(taskResult);
}
Just be sure to capture the AggregateException and check if one of the inner excepts is an TaskCanceledException.

C# async within an action

I would like to write a method which accept several parameters, including an action and a retry amount and invoke it.
So I have this code:
public static IEnumerable<Task> RunWithRetries<T>(List<T> source, int threads, Func<T, Task<bool>> action, int retries, string method)
{
object lockObj = new object();
int index = 0;
return new Action(async () =>
{
while (true)
{
T item;
lock (lockObj)
{
if (index < source.Count)
{
item = source[index];
index++;
}
else
break;
}
int retry = retries;
while (retry > 0)
{
try
{
bool res = await action(item);
if (res)
retry = -1;
else
//sleep if not success..
Thread.Sleep(200);
}
catch (Exception e)
{
LoggerAgent.LogException(e, method);
}
finally
{
retry--;
}
}
}
}).RunParallel(threads);
}
RunParallel is an extention method for Action, its look like this:
public static IEnumerable<Task> RunParallel(this Action action, int amount)
{
List<Task> tasks = new List<Task>();
for (int i = 0; i < amount; i++)
{
Task task = Task.Factory.StartNew(action);
tasks.Add(task);
}
return tasks;
}
Now, the issue: The thread is just disappearing or collapsing without waiting for the action to finish.
I wrote this example code:
private static async Task ex()
{
List<int> ints = new List<int>();
for (int i = 0; i < 1000; i++)
{
ints.Add(i);
}
var tasks = RetryComponent.RunWithRetries(ints, 100, async (num) =>
{
try
{
List<string> test = await fetchSmthFromDb();
Console.WriteLine("#" + num + " " + test[0]);
return test[0] == "test";
}
catch (Exception e)
{
Console.WriteLine(e.StackTrace);
return false;
}
}, 5, "test");
await Task.WhenAll(tasks);
}
The fetchSmthFromDb is a simple Task> which fetches something from the db and works perfectly fine when invoked outside of this example.
Whenever the List<string> test = await fetchSmthFromDb(); row is invoked, the thread seems to be closing and the Console.WriteLine("#" + num + " " + test[0]); not even being triggered, also when debugging the breakpoint never hit.
The Final Working Code
private static async Task DoWithRetries(Func<Task> action, int retryCount, string method)
{
while (true)
{
try
{
await action();
break;
}
catch (Exception e)
{
LoggerAgent.LogException(e, method);
}
if (retryCount <= 0)
break;
retryCount--;
await Task.Delay(200);
};
}
public static async Task RunWithRetries<T>(List<T> source, int threads, Func<T, Task<bool>> action, int retries, string method)
{
Func<T, Task> newAction = async (item) =>
{
await DoWithRetries(async ()=>
{
await action(item);
}, retries, method);
};
await source.ParallelForEachAsync(newAction, threads);
}

The problem is in this line:
return new Action(async () => ...
You start an async operation with the async lambda, but don't return a task to await on. I.e. it runs on worker threads, but you'll never find out when it's done. And your program terminates before the async operation is complete -that's why you don't see any output.
It needs to be:
return new Func<Task>(async () => ...
UPDATE
First, you need to split responsibilities of methods, so you don't mix retry policy (which should not be hardcoded to a check of a boolean result) with running tasks in parallel.
Then, as previously mentioned, you run your while (true) loop 100 times instead of doing things in parallel.
As #MachineLearning pointed out, use Task.Delay instead of Thread.Sleep.
Overall, your solution looks like this:
using System.Collections.Async;
static async Task DoWithRetries(Func<Task> action, int retryCount, string method)
{
while (true)
{
try
{
await action();
break;
}
catch (Exception e)
{
LoggerAgent.LogException(e, method);
}
if (retryCount <= 0)
break;
retryCount--;
await Task.Delay(millisecondsDelay: 200);
};
}
static async Task Example()
{
List<int> ints = new List<int>();
for (int i = 0; i < 1000; i++)
ints.Add(i);
Func<int, Task> actionOnItem =
async item =>
{
await DoWithRetries(async () =>
{
List<string> test = await fetchSmthFromDb();
Console.WriteLine("#" + item + " " + test[0]);
if (test[0] != "test")
throw new InvalidOperationException("unexpected result"); // will be re-tried
},
retryCount: 5,
method: "test");
};
await ints.ParallelForEachAsync(actionOnItem, maxDegreeOfParalellism: 100);
}
You need to use the AsyncEnumerator NuGet Package in order to use the ParallelForEachAsync extension method from the System.Collections.Async namespace.

Besides the final complete reengineering, I think it's very important to underline what was really wrong with the original code.
0) First of all, as #Serge Semenov immediately pointed out, Action has to be replaced with
Func<Task>
But there are still other two essential changes.
1) With an async delegate as argument it is necessary to use the more recent Task.Run instead of the older pattern new TaskFactory.StartNew (or otherwise you have to add Unwrap() explicitly)
2) Moreover the ex() method can't be async since Task.WhenAll must be waited with Wait() and without await.
At that point, even though there are logical errors that need reengineering, from a pure technical standpoint it does work and the output is produced.
A test is available online: http://rextester.com/HMMI93124

Waiting for a single task to fail out of a List<Task<..>> more cleanly, possibly with LINQ?

In my application I have a List<Task<Boolean>> that I Task.Wait[..] on to determine if they completed successfully (Result = true). Though if during my waiting a Task completes and returns a falsey value I want to cancel all other Task I am waiting on and do something based on this.
I have created two "ugly" methods to do this
// Create a CancellationToken and List<Task<..>> to work with
CancellationToken myCToken = new CancellationToken();
List<Task<Boolean>> myTaskList = new List<Task<Boolean>>();
//-- Method 1 --
// Wait for one of the Tasks to complete and get its result
Boolean finishedTaskResult = myTaskList[Task.WaitAny(myTaskList.ToArray(), myCToken)].Result;
// Continue waiting for Tasks to complete until there are none left or one returns false
while (myTaskList.Count > 0 && finishedTaskResult)
{
// Wait for the next Task to complete
finishedTaskResult = myTaskList[Task.WaitAny(myTaskList.ToArray(), myCToken)].Result;
if (!finishedTaskResult) break;
}
// Act on finishTaskResult here
// -- Method 2 --
// Create a label to
WaitForOneCompletion:
int completedTaskIndex = Task.WaitAny(myTaskList.ToArray(), myCToken);
if (myTaskList[completedTaskIndex].Result)
{
myTaskList.RemoveAt(completedTaskIndex);
goto WaitForOneCompletion;
}
else
;// One task has failed to completed, handle appropriately
I was wondering if there was a cleaner way to do this, possibly with LINQ?

Jon Skeet, Stephen Toub, and myself all have variations on the "order by completion" approach.
However, I find that usually people don't need this kind of complexity, if they focus their attention a bit differently.
In this case, you have a collection of tasks, and want them canceled as soon as one of them returns false. Instead of thinking about it from a controller perspective ("how can the calling code do this"), think about it from the task perspective ("how can each task do this").
If you introduce a higher-level asynchronous operation of "do the work and then cancel if necessary", you'll find your calling code cleans up nicely:
public async Task DoWorkAndCancel(Func<CancellationToken, Task<bool>> work,
CancellationTokenSource cts)
{
if (!await work(cts.Token))
cts.Cancel();
}
List<Func<CancellationToken, Task<bool>>> allWork = ...;
var cts = new CancellationTokenSource();
var tasks = allWork.Select(x => DoWorkAndCancel(x, cts));
await Task.WhenAll(tasks);

You can use the following method to take a sequence of tasks and create a new sequence of tasks that represents the initial tasks but returned in the order that they all complete:
public static IEnumerable<Task<T>> Order<T>(this IEnumerable<Task<T>> tasks)
{
var taskList = tasks.ToList();
var taskSources = new BlockingCollection<TaskCompletionSource<T>>();
var taskSourceList = new List<TaskCompletionSource<T>>(taskList.Count);
foreach (var task in taskList)
{
var newSource = new TaskCompletionSource<T>();
taskSources.Add(newSource);
taskSourceList.Add(newSource);
task.ContinueWith(t =>
{
var source = taskSources.Take();
if (t.IsCanceled)
source.TrySetCanceled();
else if (t.IsFaulted)
source.TrySetException(t.Exception.InnerExceptions);
else if (t.IsCompleted)
source.TrySetResult(t.Result);
}, CancellationToken.None, TaskContinuationOptions.PreferFairness, TaskScheduler.Default);
}
return taskSourceList.Select(tcs => tcs.Task);
}
Now that you have the ability to order the tasks based on their completion you can write the code basically exactly as your requirements dictate:
foreach(var task in myTaskList.Order())
if(!await task)
cancellationTokenSource.Cancel();

Using Task.WhenAny implementation, you can create like an extension overload that receives a filter too.
This method returns a Task that will complete when any of the supplied tasks have completed and the result pass the filter.
Something like this:
static class TasksExtensions
{
public static Task<Task<T>> WhenAny<T>(this IList<Task<T>> tasks, Func<T, bool> filter)
{
CompleteOnInvokePromiseFilter<T> action = new CompleteOnInvokePromiseFilter<T>(filter);
bool flag = false;
for (int i = 0; i < tasks.Count; i++)
{
Task<T> completingTask = tasks[i];
if (!flag)
{
if (action.IsCompleted) flag = true;
else if (completingTask.IsCompleted)
{
action.Invoke(completingTask);
flag = true;
}
else completingTask.ContinueWith(t =>
{
action.Invoke(t);
});
}
}
return action.Task;
}
}
class CompleteOnInvokePromiseFilter<T>
{
private int firstTaskAlreadyCompleted;
private TaskCompletionSource<Task<T>> source;
private Func<T, bool> filter;
public CompleteOnInvokePromiseFilter(Func<T, bool> filter)
{
this.filter = filter;
source = new TaskCompletionSource<Task<T>>();
}
public void Invoke(Task<T> completingTask)
{
if (completingTask.Status == TaskStatus.RanToCompletion &&
filter(completingTask.Result) &&
Interlocked.CompareExchange(ref firstTaskAlreadyCompleted, 1, 0) == 0)
{
source.TrySetResult(completingTask);
}
}
public Task<Task<T>> Task { get { return source.Task; } }
public bool IsCompleted { get { return source.Task.IsCompleted; } }
}
You can use this extension method like this:
List<Task<int>> tasks = new List<Task<int>>();
...Initialize Tasks...
var task = await tasks.WhenAny(x => x % 2 == 0);
//In your case would be something like tasks.WhenAny(b => b);

Is there an elegant Python reusable wait similar to my C# version?

I was looking for something just like this:
public static class Retry
{
public static void Do(
Action action,
TimeSpan retryInterval,
int retryCount = 3)
{
Do<object>(() =>
{
action();
return null;
}, retryInterval, retryCount);
}
public static T Do<T>(
Func<T> action,
TimeSpan retryInterval,
int retryCount = 3)
{
var exceptions = new List<Exception>();
for (int retry = 0; retry < retryCount; retry++)
{
try
{
if (retry > 0)
Thread.Sleep(retryInterval);
return action();
}
catch (Exception ex)
{
exceptions.Add(ex);
}
}
throw new AggregateException(exceptions);
}
}
from this post: Cleanest way to write retry logic?
I am a decent enough in Python to know that this could be really nice if someone has some tips. This is for test code where this comes up very often but is rarely handled elegantly.

You can do something like this, adding exception handling and other bells and whistles as you please:
def retry_fn(retry_count, delay, fn, *args, *kwargs):
retry = True
while retry and retry_count:
retry_count -= 1
success, results = fn(*args, **kwargs):
if success or not retry_count:
return results
time.sleep(delay)