I have a long running Task that uses callbacks to feed data incrementally (rather than a single ContinueWith() callback at the end).
I want to be able to pass the Task object back in this callback for task identification purposes (using Task.CurrentId)
However, I cannot work out how to pass the Task object into the task delegate. There appears to be no overload to do this, and I can't use the closure to do it since the task object is not defined at that point.
eg.
public Task StartDoingSomeStuff(CallbackDelegate callback)
{
Task task = Task.Factory.StartNew(() =>
{
while(whatever)
{
var results = DoSomeStuff();
callback(results, task); //CS0165. How do I get hold of the task?
}
});
return task;
}
gives:
error CS0165: Use of unassigned local variable 'task'
Split declaring the variable and assigning the task to it into two statements. Make sure that you don't use the variable before the task has been assigned:
public Task StartDoingSomeStuff(CallbackDelegate callback)
{
var gate = new object();
lock (gate)
{
Task task = null;
task = Task.Factory.StartNew(() =>
{
lock (gate)
{
while (whatever)
{
var results = DoSomeStuff();
callback(results, task);
}
}
});
return task;
}
}
Another solution is to create a task key and a dictionary mapping keys to tasks, then pass the key as state to the task action:
var taskMap = new Dictionary<object, Task>();
var taskKey = new object();
taskMap.Add(taskKey, Task.Factory.StartNew(key => { callback(results, key); }, taskKey));
of course you then have to look up the task from the key, which may or may not suit your scenario.
This should work although it is bad practice:
public Task StartDoingSomeStuff(CallbackDelegate callback)
{
Task task = null;
task = Task.Factory.StartNew(() =>
{
while(whatever)
{
var results = DoSomeStuff();
callback(results, task); //CS0165. How do I get hold of the task?
}
});
return task;
}
Related
If I have a task running on a worker thread and when it finds something wrong, is it possible to pause and wait for the user to intervene before continuing?
For example, suppose I have something like this:
async void btnStartTask_Click(object sender, EventArgs e)
{
await Task.Run(() => LongRunningTask());
}
// CPU-bound
bool LongRunningTask()
{
// Establish some connection here.
// Do some work here.
List<Foo> incorrectValues = GetIncorrectValuesFromAbove();
if (incorrectValues.Count > 0)
{
// Here, I want to present the "incorrect values" to the user (on the UI thread)
// and let them select whether to modify a value, ignore it, or abort.
var confirmedValues = WaitForUserInput(incorrectValues);
}
// Continue processing.
}
Is it possible to substitute WaitForUserInput() with something that runs on the UI thread, waits for the user's intervention, and then acts accordingly? If so, how? I'm not looking for complete code or anything; if someone could point me in the right direction, I would be grateful.
What you're looking for is almost exactly Progress<T>, except you want to have the thing that reports progress get a task back with some information that they can await and inspect the results of. Creating Progress<T> yourself isn't terribly hard., and you can reasonably easily adapt it so that it computes a result.
public interface IPrompt<TResult, TInput>
{
Task<TResult> Prompt(TInput input);
}
public class Prompt<TResult, TInput> : IPrompt<TResult, TInput>
{
private SynchronizationContext context;
private Func<TInput, Task<TResult>> prompt;
public Prompt(Func<TInput, Task<TResult>> prompt)
{
context = SynchronizationContext.Current ?? new SynchronizationContext();
this.prompt += prompt;
}
Task<TResult> IPrompt<TResult, TInput>.Prompt(TInput input)
{
var tcs = new TaskCompletionSource<TResult>();
context.Post(data => prompt((TInput)data)
.ContinueWith(task =>
{
if (task.IsCanceled)
tcs.TrySetCanceled();
if (task.IsFaulted)
tcs.TrySetException(task.Exception.InnerExceptions);
else
tcs.TrySetResult(task.Result);
}), input);
return tcs.Task;
}
}
Now you simply need to have an asynchronous method that accepts the data from the long running process and returns a task with whatever the user interface's response is.
You can use TaskCompletionSource to generate a task that can be awaited within the LongRunningTask.
using System;
using System.Collections.Generic;
using System.Threading.Tasks;
namespace ConsoleApp5
{
class Program
{
private static event Action<string> Input;
public static async Task Main(string[] args)
{
var inputTask = InputTask();
var longRunningTask = Task.Run(() => LongRunningTask());
await Task.WhenAll(inputTask, longRunningTask);
}
private static async Task InputTask()
{
await Task.Yield();
while(true)
{
var input = await Console.In.ReadLineAsync();
Input?.Invoke(input);
}
}
static async Task<bool> LongRunningTask()
{
SomeExpensiveCall();
var incorrectValues = GetIncorrectValuesFromAbove();
if (incorrectValues.Count > 0)
{
var confirmedValues = await WaitForUserInput(incorrectValues).ConfigureAwait(false);
}
// Continue processing.
return true;
}
private static void SomeExpensiveCall()
{
}
private static Task<string> WaitForUserInput(IList<string> incorrectValues)
{
var taskCompletionSource = new TaskCompletionSource<string>();
Console.Write("Input Data: ");
try
{
void EventHandler(string input)
{
Input -= EventHandler;
taskCompletionSource.TrySetResult(input);
}
Input += EventHandler;
}
catch(Exception e)
{
taskCompletionSource.TrySetException(e);
}
return taskCompletionSource.Task;
}
private static IList<string> GetIncorrectValuesFromAbove()
{
return new List<string> { "Test" };
}
}
}
Of course in this example you could have just called await Console.In.ReadLineAsync() directly, but this code is to simulate an environment where you only have an event based API.
There are several ways to solve this problem, with the Control.Invoke being probably the most familiar. Here is a more TPL-ish approach. You start by declaring a UI related scheduler as a class field:
private TaskScheduler _uiScheduler;
Then initialize it:
public MyForm()
{
InitializeComponent();
_uiScheduler = TaskScheduler.FromCurrentSynchronizationContext();
}
Then you convert your synchronous LongRunning method to an asynchronous method. This means that it must return Task<bool> instead of bool. It must also have the async modifier, and by convention be named with the Async suffix:
async Task<bool> LongRunningAsync()
Finally you use the await operator in order to wait for the user's input, which will be a Task configured to run on the captured UI scheduler:
async Task<bool> LongRunningAsync()
{
// Establish some connection here.
// Do some work here.
List<Foo> incorrectValues = GetIncorrectValuesFromAbove();
if (incorrectValues.Count > 0)
{
// Here, I want to present the "incorrect values" to the user (on the UI thread)
// and let them select whether to modify a value, ignore it, or abort.
var confirmedValues = await Task.Factory.StartNew(() =>
{
return WaitForUserInput(incorrectValues);
}, default, TaskCreationOptions.None, _uiScheduler);
}
// Continue processing.
}
Starting the long running task is the same as before. The Task.Run understands async delegates, so you don't have to do something special after making the method async.
var longRunningTask = Task.Run(() => LongRunningAsync());
This should be enough, provided that you just intend to show a dialog box to the user. The Form.ShowDialog is a blocking method, so the WaitForUserInput method needs not to be asynchronous. If you had to allow the user to interact freely with the main form, the problem would be much more difficult to solve.
Another example using Invoke() and a ManualResetEvent. Let me know if you need help with the form code; setting up a constructor, using DialogResult, or creating a property to hold the "confirmedValues":
bool LongRunningTask()
{
// Establish some connection here.
// Do some work here.
List<Foo> incorrectValues = GetIncorrectValuesFromAbove();
var confirmedValues;
if (incorrectValues.Count > 0)
{
DialogResult result;
ManualResetEvent mre = new ManualResetEvent(false);
this.Invoke((MethodInvoker)delegate
{
// pass in incorrectValues to the form
// you'll have to build a constructor in it to accept them
frmSomeForm frm = new frmSomeForm(incorrectValues);
result = frm.ShowDialog();
if (result == DialogResult.OK)
{
confirmedValues = frm.confirmedValues; // get the confirmed values somehow
}
mre.Set(); // release the block below
});
mre.WaitOne(); // blocks until "mre" is set
}
// Continue processing.
}
Is it possible to use Async when using ForEach? Below is the code I am trying:
using (DataContext db = new DataLayer.DataContext())
{
db.Groups.ToList().ForEach(i => async {
await GetAdminsFromGroup(i.Gid);
});
}
I am getting the error:
The name 'Async' does not exist in the current context
The method the using statement is enclosed in is set to async.
List<T>.ForEach doesn't play particularly well with async (neither does LINQ-to-objects, for the same reasons).
In this case, I recommend projecting each element into an asynchronous operation, and you can then (asynchronously) wait for them all to complete.
using (DataContext db = new DataLayer.DataContext())
{
var tasks = db.Groups.ToList().Select(i => GetAdminsFromGroupAsync(i.Gid));
var results = await Task.WhenAll(tasks);
}
The benefits of this approach over giving an async delegate to ForEach are:
Error handling is more proper. Exceptions from async void cannot be caught with catch; this approach will propagate exceptions at the await Task.WhenAll line, allowing natural exception handling.
You know that the tasks are complete at the end of this method, since it does an await Task.WhenAll. If you use async void, you cannot easily tell when the operations have completed.
This approach has a natural syntax for retrieving the results. GetAdminsFromGroupAsync sounds like it's an operation that produces a result (the admins), and such code is more natural if such operations can return their results rather than setting a value as a side effect.
This little extension method should give you exception-safe async iteration:
public static async Task ForEachAsync<T>(this List<T> list, Func<T, Task> func)
{
foreach (var value in list)
{
await func(value);
}
}
Since we're changing the return type of the lambda from void to Task, exceptions will propagate up correctly. This will allow you to write something like this in practice:
await db.Groups.ToList().ForEachAsync(async i => {
await GetAdminsFromGroup(i.Gid);
});
Starting with C# 8.0, you can create and consume streams asynchronously.
private async void button1_Click(object sender, EventArgs e)
{
IAsyncEnumerable<int> enumerable = GenerateSequence();
await foreach (var i in enumerable)
{
Debug.WriteLine(i);
}
}
public static async IAsyncEnumerable<int> GenerateSequence()
{
for (int i = 0; i < 20; i++)
{
await Task.Delay(100);
yield return i;
}
}
More
The simple answer is to use the foreach keyword instead of the ForEach() method of List().
using (DataContext db = new DataLayer.DataContext())
{
foreach(var i in db.Groups)
{
await GetAdminsFromGroup(i.Gid);
}
}
Here is an actual working version of the above async foreach variants with sequential processing:
public static async Task ForEachAsync<T>(this List<T> enumerable, Action<T> action)
{
foreach (var item in enumerable)
await Task.Run(() => { action(item); }).ConfigureAwait(false);
}
Here is the implementation:
public async void SequentialAsync()
{
var list = new List<Action>();
Action action1 = () => {
//do stuff 1
};
Action action2 = () => {
//do stuff 2
};
list.Add(action1);
list.Add(action2);
await list.ForEachAsync();
}
What's the key difference? .ConfigureAwait(false); which keeps the context of main thread while async sequential processing of each task.
This is not an old question, but .Net 6 introduced Parallel.ForeachAsync:
var collectionToIterate = db.Groups.ToList();
await Parallel.ForEachAsync(collectionToIterate, async (i, token) =>
{
await GetAdminsFromGroup(i);
});
ForeachAsync also accepts a ParallelOptions object, but usually you don't want to mess with the MaxDegreeOfParallelism property:
ParallelOptions parallelOptions = new ParallelOptions { MaxDegreeOfParallelism = 4 };
var collectionToIterate = db.Groups.ToList();
await Parallel.ForEachAsync(collectionToIterate, parallelOptions , async (i, token) =>
{
await GetAdminsFromGroup(i);
});
From Microsoft Docs: https://learn.microsoft.com/en-us/dotnet/api/system.threading.tasks.paralleloptions.maxdegreeofparallelism?view=net-6.0
By default, For and ForEach will utilize however many threads the underlying scheduler provides, so changing MaxDegreeOfParallelism from the default only limits how many concurrent tasks will be used.
Generally, you do not need to modify this setting....
Add this extension method
public static class ForEachAsyncExtension
{
public static Task ForEachAsync<T>(this IEnumerable<T> source, int dop, Func<T, Task> body)
{
return Task.WhenAll(from partition in Partitioner.Create(source).GetPartitions(dop)
select Task.Run(async delegate
{
using (partition)
while (partition.MoveNext())
await body(partition.Current).ConfigureAwait(false);
}));
}
}
And then use like so:
Task.Run(async () =>
{
var s3 = new AmazonS3Client(Config.Instance.Aws.Credentials, Config.Instance.Aws.RegionEndpoint);
var buckets = await s3.ListBucketsAsync();
foreach (var s3Bucket in buckets.Buckets)
{
if (s3Bucket.BucketName.StartsWith("mybucket-"))
{
log.Information("Bucket => {BucketName}", s3Bucket.BucketName);
ListObjectsResponse objects;
try
{
objects = await s3.ListObjectsAsync(s3Bucket.BucketName);
}
catch
{
log.Error("Error getting objects. Bucket => {BucketName}", s3Bucket.BucketName);
continue;
}
// ForEachAsync (4 is how many tasks you want to run in parallel)
await objects.S3Objects.ForEachAsync(4, async s3Object =>
{
try
{
log.Information("Bucket => {BucketName} => {Key}", s3Bucket.BucketName, s3Object.Key);
await s3.DeleteObjectAsync(s3Bucket.BucketName, s3Object.Key);
}
catch
{
log.Error("Error deleting bucket {BucketName} object {Key}", s3Bucket.BucketName, s3Object.Key);
}
});
try
{
await s3.DeleteBucketAsync(s3Bucket.BucketName);
}
catch
{
log.Error("Error deleting bucket {BucketName}", s3Bucket.BucketName);
}
}
}
}).Wait();
If you are using EntityFramework.Core there is an extension method ForEachAsync.
The example usage looks like this:
using Microsoft.EntityFrameworkCore;
using System.Threading.Tasks;
public class Example
{
private readonly DbContext _dbContext;
public Example(DbContext dbContext)
{
_dbContext = dbContext;
}
public async void LogicMethod()
{
await _dbContext.Set<dbTable>().ForEachAsync(async x =>
{
//logic
await AsyncTask(x);
});
}
public async Task<bool> AsyncTask(object x)
{
//other logic
return await Task.FromResult<bool>(true);
}
}
I would like to add that there is a Parallel class with ForEach function built in that can be used for this purpose.
The problem was that the async keyword needs to appear before the lambda, not before the body:
db.Groups.ToList().ForEach(async (i) => {
await GetAdminsFromGroup(i.Gid);
});
This is method I created to handle async scenarios with ForEach.
If one of tasks fails then other tasks will continue their execution.
You have ability to add function that will be executed on every exception.
Exceptions are being collected as aggregateException at the end and are available for you.
Can handle CancellationToken
public static class ParallelExecutor
{
/// <summary>
/// Executes asynchronously given function on all elements of given enumerable with task count restriction.
/// Executor will continue starting new tasks even if one of the tasks throws. If at least one of the tasks throwed exception then <see cref="AggregateException"/> is throwed at the end of the method run.
/// </summary>
/// <typeparam name="T">Type of elements in enumerable</typeparam>
/// <param name="maxTaskCount">The maximum task count.</param>
/// <param name="enumerable">The enumerable.</param>
/// <param name="asyncFunc">asynchronous function that will be executed on every element of the enumerable. MUST be thread safe.</param>
/// <param name="onException">Acton that will be executed on every exception that would be thrown by asyncFunc. CAN be thread unsafe.</param>
/// <param name="cancellationToken">The cancellation token.</param>
public static async Task ForEachAsync<T>(int maxTaskCount, IEnumerable<T> enumerable, Func<T, Task> asyncFunc, Action<Exception> onException = null, CancellationToken cancellationToken = default)
{
using var semaphore = new SemaphoreSlim(initialCount: maxTaskCount, maxCount: maxTaskCount);
// This `lockObject` is used only in `catch { }` block.
object lockObject = new object();
var exceptions = new List<Exception>();
var tasks = new Task[enumerable.Count()];
int i = 0;
try
{
foreach (var t in enumerable)
{
await semaphore.WaitAsync(cancellationToken);
tasks[i++] = Task.Run(
async () =>
{
try
{
await asyncFunc(t);
}
catch (Exception e)
{
if (onException != null)
{
lock (lockObject)
{
onException.Invoke(e);
}
}
// This exception will be swallowed here but it will be collected at the end of ForEachAsync method in order to generate AggregateException.
throw;
}
finally
{
semaphore.Release();
}
}, cancellationToken);
if (cancellationToken.IsCancellationRequested)
{
break;
}
}
}
catch (OperationCanceledException e)
{
exceptions.Add(e);
}
foreach (var t in tasks)
{
if (cancellationToken.IsCancellationRequested)
{
break;
}
// Exception handling in this case is actually pretty fast.
// https://gist.github.com/shoter/d943500eda37c7d99461ce3dace42141
try
{
await t;
}
#pragma warning disable CA1031 // Do not catch general exception types - we want to throw that exception later as aggregate exception. Nothing wrong here.
catch (Exception e)
#pragma warning restore CA1031 // Do not catch general exception types
{
exceptions.Add(e);
}
}
if (exceptions.Any())
{
throw new AggregateException(exceptions);
}
}
}
Recently I want to implement a health check for a list of service calls. They are all async task (e.g. Task<IHttpOperationResponse<XXX_Model>> method_name(...)
I would like to put all of them into a list. I followed the answer of this post: Storing a list of methods in C# However, they are async methods.
I put it like this:
a collection of async method
List<Action> _functions = new List<Action> {
() => accountDetailsServiceProvider.GetEmployer(EmployerId),
() => accountServiceProvider.GetAccountStatus(EmployerId)
}
Can someone direct me to the right way to implement putting async methods in to a list and invoke them iteratively?
Thanks in advance!
First, you need to make your methods async. That means they must return a Task. For example:
public static async Task Foo()
{
await Task.Delay(1);
Console.WriteLine("Foo!");
}
public static async Task Bar()
{
await Task.Delay(1);
Console.WriteLine("Bar!");
}
Then to put them in a list, you must define the list as containing the right type. Since an async method actually returns something, it's a Func, not an action. It returns a Task.
var actions = new List<Func<Task>>
{
Foo, Bar
};
To invoke them, Select over the list (using Linq) to invoke them. This creates a list of Tasks in place of the list of Funcs.
var tasks = actions.Select( x => x() );
Then just await them:
await Task.WhenAll(tasks);
Full example:
public static async Task MainAsync()
{
var actions = new List<Func<Task>>
{
Foo, Bar
};
var tasks = actions.Select( x => x() );
await Task.WhenAll(tasks);
}
Output:
Foo!
Bar!
Example on DotNetFiddle
If your methods return a Boolean value, then the return type becomes Task<bool> and the rest follows suit:
public static async Task<bool> Foo()
{
await Task.Delay(1);
Console.WriteLine("Foo!");
return true;
}
public static async Task<bool> Bar()
{
await Task.Delay(1);
Console.WriteLine("Bar!");
return true;
}
var actions = new List<Func<Task<bool>>>
{
Foo, Bar
};
var tasks = actions.Select( x => x() );
await Task.WhenAll(tasks);
After you have awaited them, you can convert the tasks to their results with one more LINQ statement:
List<bool> results = tasks.Select( task => task.Result ).ToList();
I think you are just looking for something simple like this?
var myList = new List<Action>()
{
async() => { await Foo.GetBarAsync(); },
...
};
I would recommend you to change the type from Action to Func<Task> like so instead.
var myList = new List<Func<Task>>()
{
async() => { await Foo.GetBarAsync(); },
};
You can read more about why here: https://blogs.msdn.microsoft.com/pfxteam/2012/02/08/potential-pitfalls-to-avoid-when-passing-around-async-lambdas/
To invoke (simplified)
foreach (var action in myList)
{
await action.Invoke();
}
Based on the comments:
However, my task requires a boolean value for each method call,
because I have to report the status to the frontend whether the
service is down or not
Create a wrapper method for the method which will return required boolean value
public async Task<Result> Check(string name, Func<Task> execute)
{
try
{
await execute();
return new Result(name, true, string.Empty);
}
catch (Exception ex)
{
return new Result(name, false, ex.Message);
}
}
public class Result
{
public string Name { get; }
public bool Success { get; }
public string Message { get; }
public Result(string name, bool success, string message)
=> (Name, Success, Message) = (name, success, message);
}
Then you don't need to have collection of delegates, instead you will have collection of Task.
var tasks = new[]
{
Check(nameof(details.GetEmployer), () => details.GetEmployer(Id)),
Check(nameof(accounts.GetAccountStatus), () => accounts.GetAccountStatus(Id)),
};
var completed = await Task.WhenAll(tasks);
foreach (var task in completed)
{
Console.WriteLine($"Task: {task.Name}, Success: {task.Success};");
}
Based on some questions on SO, mainly this one:
Throttling asynchronous tasks
I have implemented the SemaphoreSlim object to concurrently processes requests over a range of methods in my application. Most of these methods are taking in lists of IDs and getting single byte arrays back per ID in a concurrent fashion from the web. The implementation looks like this:
using (var semaphore = new SemaphoreSlim(MaxConcurrency))
{
var tasks = fileMetadata.GroupBy(x => x.StorageType).Select(async storageTypeFileMetadata=>
{
await semaphore.WaitAsync();
try
{
var fileManager = FileManagerFactory.CreateFileManager((StorageType)storageTypeFileMetadata.Key);
await fileManager.UpdateFilesAsync(storageTypeFileMetadata);
}
finally
{
semaphore.Release();
}
});
await Task.WhenAll(tasks);
}
Is there a way to abstract out a method or some reusable code snippet for the semaphore code, and pass in the work I need done, so it can be reused without re-writing the semaphore code each time? The only difference amongst multiple methods using this same semaphore pattern is the list I am iterating and the work it is doing in the try{}.
I am thinking something like pass list.select(x=> my task method with my work in it) to a semaphore method which is all the wrapper semaphore code.
So I'm guessing something like:
public static class Extension
{
public static async Task ExecuteAsync<T>(this IEnumerable<T> items, Func<T, Task> task, int concurrency)
{
var tasks = new List<Task>();
using (var semaphore = new SemaphoreSlim(concurrency))
{
foreach (var item in items)
{
tasks.Add(ExecuteInSemaphore(semaphore, task, item));
}
await Task.WhenAll(tasks);
}
}
private static async Task ExecuteInSemaphore<T>(SemaphoreSlim semaphore, Func<T, Task> task, T item)
{
await semaphore.WaitAsync();
try
{
await task(item);
}
finally
{
semaphore.Release();
}
}
}
Then you would use it like:
await fileMetadata.GroupBy(x => x.StorageType).ExecuteAsync(storageTypeFileMetadata =>
{
var fileManager = FileManagerFactory.CreateFileManager((StorageType)storageTypeFileMetadata.Key);
return fileManager.UpdateFilesAsync(storageTypeFileMetadata);
}, 4);
Is it possible to use Async when using ForEach? Below is the code I am trying:
using (DataContext db = new DataLayer.DataContext())
{
db.Groups.ToList().ForEach(i => async {
await GetAdminsFromGroup(i.Gid);
});
}
I am getting the error:
The name 'Async' does not exist in the current context
The method the using statement is enclosed in is set to async.
List<T>.ForEach doesn't play particularly well with async (neither does LINQ-to-objects, for the same reasons).
In this case, I recommend projecting each element into an asynchronous operation, and you can then (asynchronously) wait for them all to complete.
using (DataContext db = new DataLayer.DataContext())
{
var tasks = db.Groups.ToList().Select(i => GetAdminsFromGroupAsync(i.Gid));
var results = await Task.WhenAll(tasks);
}
The benefits of this approach over giving an async delegate to ForEach are:
Error handling is more proper. Exceptions from async void cannot be caught with catch; this approach will propagate exceptions at the await Task.WhenAll line, allowing natural exception handling.
You know that the tasks are complete at the end of this method, since it does an await Task.WhenAll. If you use async void, you cannot easily tell when the operations have completed.
This approach has a natural syntax for retrieving the results. GetAdminsFromGroupAsync sounds like it's an operation that produces a result (the admins), and such code is more natural if such operations can return their results rather than setting a value as a side effect.
This little extension method should give you exception-safe async iteration:
public static async Task ForEachAsync<T>(this List<T> list, Func<T, Task> func)
{
foreach (var value in list)
{
await func(value);
}
}
Since we're changing the return type of the lambda from void to Task, exceptions will propagate up correctly. This will allow you to write something like this in practice:
await db.Groups.ToList().ForEachAsync(async i => {
await GetAdminsFromGroup(i.Gid);
});
Starting with C# 8.0, you can create and consume streams asynchronously.
private async void button1_Click(object sender, EventArgs e)
{
IAsyncEnumerable<int> enumerable = GenerateSequence();
await foreach (var i in enumerable)
{
Debug.WriteLine(i);
}
}
public static async IAsyncEnumerable<int> GenerateSequence()
{
for (int i = 0; i < 20; i++)
{
await Task.Delay(100);
yield return i;
}
}
More
The simple answer is to use the foreach keyword instead of the ForEach() method of List().
using (DataContext db = new DataLayer.DataContext())
{
foreach(var i in db.Groups)
{
await GetAdminsFromGroup(i.Gid);
}
}
Here is an actual working version of the above async foreach variants with sequential processing:
public static async Task ForEachAsync<T>(this List<T> enumerable, Action<T> action)
{
foreach (var item in enumerable)
await Task.Run(() => { action(item); }).ConfigureAwait(false);
}
Here is the implementation:
public async void SequentialAsync()
{
var list = new List<Action>();
Action action1 = () => {
//do stuff 1
};
Action action2 = () => {
//do stuff 2
};
list.Add(action1);
list.Add(action2);
await list.ForEachAsync();
}
What's the key difference? .ConfigureAwait(false); which keeps the context of main thread while async sequential processing of each task.
This is not an old question, but .Net 6 introduced Parallel.ForeachAsync:
var collectionToIterate = db.Groups.ToList();
await Parallel.ForEachAsync(collectionToIterate, async (i, token) =>
{
await GetAdminsFromGroup(i);
});
ForeachAsync also accepts a ParallelOptions object, but usually you don't want to mess with the MaxDegreeOfParallelism property:
ParallelOptions parallelOptions = new ParallelOptions { MaxDegreeOfParallelism = 4 };
var collectionToIterate = db.Groups.ToList();
await Parallel.ForEachAsync(collectionToIterate, parallelOptions , async (i, token) =>
{
await GetAdminsFromGroup(i);
});
From Microsoft Docs: https://learn.microsoft.com/en-us/dotnet/api/system.threading.tasks.paralleloptions.maxdegreeofparallelism?view=net-6.0
By default, For and ForEach will utilize however many threads the underlying scheduler provides, so changing MaxDegreeOfParallelism from the default only limits how many concurrent tasks will be used.
Generally, you do not need to modify this setting....
Add this extension method
public static class ForEachAsyncExtension
{
public static Task ForEachAsync<T>(this IEnumerable<T> source, int dop, Func<T, Task> body)
{
return Task.WhenAll(from partition in Partitioner.Create(source).GetPartitions(dop)
select Task.Run(async delegate
{
using (partition)
while (partition.MoveNext())
await body(partition.Current).ConfigureAwait(false);
}));
}
}
And then use like so:
Task.Run(async () =>
{
var s3 = new AmazonS3Client(Config.Instance.Aws.Credentials, Config.Instance.Aws.RegionEndpoint);
var buckets = await s3.ListBucketsAsync();
foreach (var s3Bucket in buckets.Buckets)
{
if (s3Bucket.BucketName.StartsWith("mybucket-"))
{
log.Information("Bucket => {BucketName}", s3Bucket.BucketName);
ListObjectsResponse objects;
try
{
objects = await s3.ListObjectsAsync(s3Bucket.BucketName);
}
catch
{
log.Error("Error getting objects. Bucket => {BucketName}", s3Bucket.BucketName);
continue;
}
// ForEachAsync (4 is how many tasks you want to run in parallel)
await objects.S3Objects.ForEachAsync(4, async s3Object =>
{
try
{
log.Information("Bucket => {BucketName} => {Key}", s3Bucket.BucketName, s3Object.Key);
await s3.DeleteObjectAsync(s3Bucket.BucketName, s3Object.Key);
}
catch
{
log.Error("Error deleting bucket {BucketName} object {Key}", s3Bucket.BucketName, s3Object.Key);
}
});
try
{
await s3.DeleteBucketAsync(s3Bucket.BucketName);
}
catch
{
log.Error("Error deleting bucket {BucketName}", s3Bucket.BucketName);
}
}
}
}).Wait();
If you are using EntityFramework.Core there is an extension method ForEachAsync.
The example usage looks like this:
using Microsoft.EntityFrameworkCore;
using System.Threading.Tasks;
public class Example
{
private readonly DbContext _dbContext;
public Example(DbContext dbContext)
{
_dbContext = dbContext;
}
public async void LogicMethod()
{
await _dbContext.Set<dbTable>().ForEachAsync(async x =>
{
//logic
await AsyncTask(x);
});
}
public async Task<bool> AsyncTask(object x)
{
//other logic
return await Task.FromResult<bool>(true);
}
}
I would like to add that there is a Parallel class with ForEach function built in that can be used for this purpose.
The problem was that the async keyword needs to appear before the lambda, not before the body:
db.Groups.ToList().ForEach(async (i) => {
await GetAdminsFromGroup(i.Gid);
});
This is method I created to handle async scenarios with ForEach.
If one of tasks fails then other tasks will continue their execution.
You have ability to add function that will be executed on every exception.
Exceptions are being collected as aggregateException at the end and are available for you.
Can handle CancellationToken
public static class ParallelExecutor
{
/// <summary>
/// Executes asynchronously given function on all elements of given enumerable with task count restriction.
/// Executor will continue starting new tasks even if one of the tasks throws. If at least one of the tasks throwed exception then <see cref="AggregateException"/> is throwed at the end of the method run.
/// </summary>
/// <typeparam name="T">Type of elements in enumerable</typeparam>
/// <param name="maxTaskCount">The maximum task count.</param>
/// <param name="enumerable">The enumerable.</param>
/// <param name="asyncFunc">asynchronous function that will be executed on every element of the enumerable. MUST be thread safe.</param>
/// <param name="onException">Acton that will be executed on every exception that would be thrown by asyncFunc. CAN be thread unsafe.</param>
/// <param name="cancellationToken">The cancellation token.</param>
public static async Task ForEachAsync<T>(int maxTaskCount, IEnumerable<T> enumerable, Func<T, Task> asyncFunc, Action<Exception> onException = null, CancellationToken cancellationToken = default)
{
using var semaphore = new SemaphoreSlim(initialCount: maxTaskCount, maxCount: maxTaskCount);
// This `lockObject` is used only in `catch { }` block.
object lockObject = new object();
var exceptions = new List<Exception>();
var tasks = new Task[enumerable.Count()];
int i = 0;
try
{
foreach (var t in enumerable)
{
await semaphore.WaitAsync(cancellationToken);
tasks[i++] = Task.Run(
async () =>
{
try
{
await asyncFunc(t);
}
catch (Exception e)
{
if (onException != null)
{
lock (lockObject)
{
onException.Invoke(e);
}
}
// This exception will be swallowed here but it will be collected at the end of ForEachAsync method in order to generate AggregateException.
throw;
}
finally
{
semaphore.Release();
}
}, cancellationToken);
if (cancellationToken.IsCancellationRequested)
{
break;
}
}
}
catch (OperationCanceledException e)
{
exceptions.Add(e);
}
foreach (var t in tasks)
{
if (cancellationToken.IsCancellationRequested)
{
break;
}
// Exception handling in this case is actually pretty fast.
// https://gist.github.com/shoter/d943500eda37c7d99461ce3dace42141
try
{
await t;
}
#pragma warning disable CA1031 // Do not catch general exception types - we want to throw that exception later as aggregate exception. Nothing wrong here.
catch (Exception e)
#pragma warning restore CA1031 // Do not catch general exception types
{
exceptions.Add(e);
}
}
if (exceptions.Any())
{
throw new AggregateException(exceptions);
}
}
}