Develop Reference

How to convert a list of generic tasks of different types that are stored in a List<Task>, to a Task<List<object>>?

I am looking to create a Task<List> that when invoked with a set of methods executes in parallel, runs and returns the result in the same order of the tasks in an array. The Tasks can return different types. I tried below. Not sure if I am heading in right direction.
private async Task<IList<object>> RunTasks<T>(IList<Task> taskList)
{
var allTasks = Task.WhenAll(taskList);
var ret = new object[taskList.Count];
await allTasks;
for (int i=0;i<taskList.Count;i++)
{
ret[i] = taskList[i].IsFaulted ?
default : ((Task<T>)taskList[i]).Result;
}
//otherPolicies.AppsPermissionsPolicy = teamsAppPermissionDocFromAAE
// .ToMTTeamsAppPermissionPolicy().ToMTPolicyDocument();
//Wrap AAE TeamsApp doc response into other Policies
return ret;
}
If Task1 & Task2 returns different types in taskList do we need T for RunTasks ? if so, What type do we pass in to Invoke RunTasks?. If we dont need it, then how do we convert the Tasks return type to its corresponding object in the for loop immediately after tasks completed before we return the object array with results?

I think that converting a List<Task> to a List<Task<object>> cannot be done without reflection. Or without the dynamic keyword, like in the implementation below:
public static Task<object[]> WhenAllToObject(IEnumerable<Task> tasks)
{
ArgumentNullException.ThrowIfNull(tasks);
return Task.WhenAll(tasks.Select(async task =>
{
// First await the task, to ensure that it ran successfully.
await task.ConfigureAwait(false);
// Then try to get its result, if it's a generic Task<T>.
try
{
return await (dynamic)task;
}
catch (Microsoft.CSharp.RuntimeBinder.RuntimeBinderException)
{
throw new InvalidOperationException("Non-generic task found.");
}
}));
}
Usage example:
List<Task> heterogeneousListOfTasks = new()
{
Task.FromResult(13),
Task.FromResult("Hello"),
Task.FromResult(true),
};
object[] results = await WhenAllToObject(heterogeneousListOfTasks);
Alternative: This one is inspired by IS4's ingenious type-matching trick. The advantage over the above implementation is that the validation of the tasks argument happens synchronously. That's because the pattern matching for the Task case is a non-async method (it elides async and await).
public static Task<object[]> WhenAllToObject(IEnumerable<Task> tasks)
{
ArgumentNullException.ThrowIfNull(tasks);
return Task.WhenAll(tasks.Select(task =>
{
if (task == null) throw new ArgumentException(
$"The {nameof(tasks)} argument included a null value.", nameof(tasks));
Task<object> taskOfObject = ToTaskOfObject((dynamic)task);
if (taskOfObject == null) throw new ArgumentException(
$"The {nameof(tasks)} argument included a non-generic Task.", nameof(tasks));
return taskOfObject;
}));
}
private static Task<object> ToTaskOfObject(Task task) // Not async
=> null;
private static async Task<object> ToTaskOfObject<T>(Task<T> task)
=> await task.ConfigureAwait(false);
Both implementations have similar behavior with the Task.WhenAll method, but not identical. The Task.WhenAll propagates all exceptions of all tasks. On the contrary the WhenAllToObject propagates only the first exception of each task.

If the number of tasks is 2 (you said task1 and task2) you may consider doing this without a loop:
var t1 = DoOneThingAsync();
var t2 = DoAnotherThingAsync();
await Task.WhenAll(t1, t2); // Without this you can miss exceptions
var result1 = await t1;
var result2 = await t2;
For a dynamic number of tasks I would still suggest separating them by type:
List<Task<int>> xs = ...;
List<Task<string>> ys = ...;
var groupX = await Task.WhenAll(xs);
var groupY = await Task.WhenAll(ys);

I'd argue that in most cases, you shouldn't need this, as once you get the result as object, you'll most likely want to cast it back to some concrete type anyway, so why not work with the tasks directly until you are sure what the types are?
Anyway, the non-generic Task class doesn't expose the Result property like Task<T> does, and there aren't any simple workarounds. We'll have to use some tricks!
By far my favourite way is to use dynamic combined with overloads, resulting in a sort of runtime type-based pattern matching:
public static async Task<IEnumerable<object>> TasksResults(IEnumerable<Task> tasks)
{
await Task.WhenAll(tasks);
return tasks.Select(t => TaskResult((dynamic)t));
}
static object TaskResult<T>(Task<T> task)
{
return task.Result;
}
static object TaskResult(Task task)
{
return null;
}
Using (dynamic) makes the expression behave as if t had actually the most concrete type of the object inside. If it is actually Task<T> (with a result), the first overload is chosen, otherwise the second overload is picked. No exception handling necessary.
I've also taken the liberty to use IEnumerable for the argument and return type. Feel free to use .ToList() if you need to.

Async method that returns result and Task

I'm trying to design a public method that returns a quick result and, if needed, kicks off a long-running background task.
The result is retrieved within the method pretty quickly, and once it is retrieved it needs to be immediately available to the caller, without waiting for the potential background task to complete.
The background task is either not needed at all, or it must run for a significant amount of time - longer than the caller and all other logic in the program would take to complete without it.
Looking through the MS docs, the best design option I can come up with is to return two things from this method: the result, as well as a Task for the background task.
I don't see a better way to do this, but there are some downsides: first, the responsibility for making sure the background task completes falls on the caller of the method, even though the caller really just wants to consume the immediate result and not be concerned with the behind-the-scene stuff. Second, it is awkward to return null if the background task isn't needed to begin with: now the caller must ensure the task isn't null, in addition to making sure it completes if it is null.
What other options are available for this sort of situation?
Here's an example of what my current design looks like:
public async Task<Tuple<string, Task>> GetAndSave() {
var networkResult = await GetFromNetworkAsync();
if (NeedsToSave(networkResult)) {
var saveTask = SaveToDiskAsync(networkResult);
return new Tuple<string, Task>(networkResult, saveTask);
}
else {
return new Tuple<string, Task>(networkResult, null);
}
}

first, the responsibility for making sure the background task completes falls on the caller of the method, even though the caller really just wants to consume the immediate result and not be concerned with the behind-the-scene stuff.
If it's important to make sure the background task completes then instead of returning the Task you could hand it off to another object (that has been injected into the class that has your GetAndSave method). For example:
public class Foo
{
readonly Action<Task> _ensureCompletion;
public Foo(Action<Task> ensureCompletion)
{
_ensureCompletion = ensureCompletion;
}
public async Task<string> GetAndSaveAsync() // Your method
{
string networkResult = await GetFromNetworkAsync();
if (NeedsToSave(networkResult))
{
Task saveTask = SaveToDiskAsync(networkResult);
_ensureCompletion(saveTask); // This is a synchronous call... no await keyword here. And it returns `void`
}
return networkResult;
}
Task<string> GetFromNetworkAsync() {...}
bool NeedsToSave(string x) {...}
Task SaveToDiskAsync(string x) {...}
}
Now you can inject whatever follow-up behavior you desire for the saveTask. For example, you could write stuff out to the console depending on how it goes:
async Task DoStuff()
{
var foo = new Foo(async task =>
// ^^^^^ More on this in a moment
{
try
{
await task;
Console.Writeline("It worked!");
}
catch (Exception e)
{
Console.Writeline(e.ToString());
}
});
var immediateResult = await foo.GetAndSaveAsync();
// do stuff with `immediateResult`
}
Now, the thing that might be confusing about this (and the power behind this type of solution) is how you can have a synchronous call on the one hand:
_ensureCompletion(saveTask); // This is a synchronous call... no await keyword here
...that does asynchronous things:
var foo = new Foo(async task => ...);
// ^^^^^ This statement lambda is asynchronous
(The injected delegate might even write out to the console on a different thread than whatever thread called GetAndSaveAsync()!)
There's no magic here. It all comes down to SynchronizationContext and the inner workings of async/await.
When the compiler encounters the await keyword (in an async context) then it will do a few things:
Turn everything after the await into a continuation (basically a delegate with some state)
Replace the await keyword with something like SynchronizationContext.Current.Post(continuation)
In other words, this:
async void EnsureCompletion(Task task)
{
try
{
await task;
Console.Writeline("It worked!");
}
catch (Exception e)
{
Console.Writeline(e.ToString());
}
}
...gets turned into something like this:
void EnsureCompletion(Task task)
{
try
{
task.ContinueWith(t => SynchronizationContext.Current.Post(_ =>
{
if (t.IsCompletedSuccessfully)
{
Console.Writeline("It worked!");
}
else
{
Console.Writeline(task.Exception.ToString());
}
});
}
catch (Exception e)
{
Console.Writeline(e.ToString());
}
}
As you can see, EnsureCompletion is an entirely synchronous function that does (almost) the exact same thing as its asynchronous form. Notice how it returns void and everything. This is how you can jam an asynchronous statement lambda into a delegate parameter that has a synchronous signature.
I hinted that the console writing might happen on a totally different thread. That's because async/await is orthogonal to threading. It depends on whatever the currently assigned implementation of SynchronizationContext is programmed to do. Unless you're using WPF or WinForms then by default there will be no SynchronizationContext and continuations (the things that get passed to SynchronizationContext.Post) will just be tossed over to whatever thread happens to be free in the default thread pool.
Second, it is awkward to return null if the background task isn't needed to begin with: now the caller must ensure the task isn't null, in addition to making sure it completes if it is null.
I'm of the opinion that null was a design mistake in C#. (Ask me what some other ones are :) ). If you return null from this method:
public Task DoSomethingAsync()
{
return null; // Compiles just fine
}
...then anyone who uses it will encounter a NullReferenceException when they await it.
async Task Blah()
{
await DoSomethingAsync(); // Wham! NullReferenceException :(
}
(the reason why comes back to how the compiler desugurs the await keyword)
...and it's awkward to have to check for nulls everywhere.
Much better would be to just return Task.CompletedTask as #juharr said.
But if you just hand off the background task like I showed above then you don't have to worry about this.

How to return a Task<T> without await

Is it possible to return a task from a method which first calls multiple Task<T> returning methods and then returns some type that includes the results from previous calls without using await?
For example, the below is straight forward:
public Task<SomeType> GetAsync() => FirstOrDefaultAsync();
However, I would like to do something like this:
public Task<SomeType> GetAsync()
{
var list = GetListAsync(); // <-- Task<List<T>>
var count = GetCountAsync(); // <-- Task<int>
return new SomeType // <-- Obviously compiler error
{
List /* <-- List<T> */ = list, // <-- Also compiler error
Count /* <-- int */ = count, // <-- Also compiler error
};
}
Is it possible to do this without having to write:
public async Task<SomeType> GetAsync()
{
return new Type2
{
List = await GetListAsync(),
Count = await GetCountAsync(),
};
}

Frankly, the version already in the question is correct:
public async Task<SomeType> GetAsync()
{
return new Type2
{
List = await GetListAsync(),
Count = await GetCountAsync(),
};
}
I realize you asked "without using await", but: the hacks to avoid the await are suboptimal; in particular, you should almost never use ContinueWith - that is the legacy API, and the Task implementation is now optimized for await, not ContinueWith.
As for:
Because I read that having multiple await is bad for performance. I try to await on last call
No; once you have one incomplete await, it pretty much doesn't matter how many more you have - they're effectively free. The issue of having one vs zero incomplete await is comparable to the ContinueWith, so : you're not gaining anything by avoiding the await.
Conclusion: just use the await. It is simpler and more direct, and the internals are optimized for it.
As a minor optimization, you might want to add ConfigureAwait(false), i.e.
public async Task<SomeType> GetAsync()
{
return new Type2
{
List = await GetListAsync().ConfigureAwait(false),
Count = await GetCountAsync().ConfigureAwait(false),
};
}
Or if they should run concurrently, and the implementation supports it:
public Task<SomeType> GetAsync()
{
var list = GetListAsync();
var count = GetCountAsync();
return new SomeType
{
List = await list.ConfigureAwait(false),
Count = await count.ConfigureAwait(false),
};
}

You can use Task.WhenAll together with Task.ContinueWith.
public Task<SomeType> GetAsync()
{
var list = GetListAsync();
var count = GetCountAsync();
return Task.WhenAll(list, count).ContinueWith(_ => new Type2
{
List = list.Result,
Count = count.Result,
});
}
Edit
As suggested in comments, you're better off just using await. I also advice to read the post linked by GSerg - Performance of Task.ContinueWith in non-async method vs. using async/await

The problem is that the Task.WhenAll method does not accept tasks with different result types. All tasks must be of the same type. Fortunately this is easy to fix. The WhenAll variant bellow waits for two tasks with different types, and returns a task with the combined results.
public static Task<TResult> WhenAll<T1, T2, TResult>(
Task<T1> task1, Task<T2> task2, Func<T1, T2, TResult> factory)
{
return Task.WhenAll(task1, task2).ContinueWith(t =>
{
var tcs = new TaskCompletionSource<TResult>();
if (t.IsFaulted)
{
tcs.SetException(t.Exception.InnerExceptions);
}
else if (t.IsCanceled)
{
tcs.SetCanceled();
}
else
{
tcs.SetResult(factory(task1.Result, task2.Result));
}
return tcs.Task;
}, default, TaskContinuationOptions.ExecuteSynchronously,
TaskScheduler.Default).Unwrap();
}
It can be used like this:
public Task<SomeType> GetAsync()
{
return WhenAll(GetListAsync(), GetCountAsync(),
(list, count) => new SomeType { List = list, Count = count });
}
The advantage over the other solutions is at the handling of exceptions. If both GetListAsync and GetCountAsync fail, the task returned from GetAsync will preserve both exceptions in a shallow AggregateException (not nested in another AggregateException).
Btw this answer is inspired by a Stephen Cleary's answer here.

How to wait the result of async operations without await?

void A()
{
foreach (var document in documents)
{
var res = records.BulkWriteAsync(operationList, writeOptions); // res is Task<BulkWriteResult<JobInfoRecord>>
}
}
After foreach I would like to wait the result of all BulkWriteAsync, how to do this? I don't want to mark A() as async and do the following
await records.BulkWriteAsync(operationList, writeOptions);
Is it good solution?
void A()
{
var tasks = new List<Task<BulkWriteResult<JobInfoRecord>>>();
foreach (var document in documents)
{
var task = records.BulkWriteAsync(operationList, writeOptions);
tasks.Add(task);
}
Task.WaitAll(tasks.ToArray());
}
I call A() in try catch if I will mark public async void A() as async I never be in catch

Well, first you want a Task that represents all the operations. The simplest way to do this is with a bit of LINQ:
Task.WhenAll(documents.Select(i => records.BulkWriteAsync(...)));
Then, you ideally want to await that task. If that isn't possible, you can try
task.GetAwaiter().GetResult();
However, make sure that none of the tasks have thread affinity - that's a great way to get a deadlock. Waiting for a task on the UI thread while the task itself needs the UI thread is a typical example.
The whole point of await is that it allows you to handle asynchronous code as if it were synchronous. So from the outside, it appears as if you never left the method until you actually get to a return (or the end of the method). For this to work, however, your method must return a Task (or Task<T>), and the callee must await your method in turn.
So a code like this:
try
{
tasks = Task.WhenAll(documents.Select(i => ...));
await tasks;
}
catch (Exception ex)
{
// Handle the exception
}
will appear to run completely synchronously, and all exceptions will be handled as usual (though since we're using Task.WhenAll, some will be wrapped in AggregateException).
However, this isn't actually possible to handle with the way .NET and C# is built, so the C# compiler cheats - await is basically a return that gives you a promise of the result you'll get in the future. And when that happens, the control returns back to where the await left the last time. Task is that promise - if you use async void, there's no way for the callee to know what's happening, and it has no option but to continue as if the asynchronous method was a run-and-forget method. If you use async Task, you can await the async method and everything "feels" synchronous again. Don't break the chain, and the illusion is perfect :)

I want await to throw AggregateException, not just the first Exception

When awaiting a faulted task (one that has an exception set), await will rethrow the stored exception. If the stored exception is an AggregateException it will rethrow the first and discard the rest.
How can we use await and at the same time throw the original AggregateException so that we do not accidentally lose error information?
Note, that it is of course possible to think of hacky solutions for this (e.g. try-catch around the await, then call Task.Wait). I really wish to find a clean solution. What is the best-practice here?
I thought of using a custom awaiter but the built-in TaskAwaiter contains lots of magic that I'm not sure how to fully reproduce. It calls internal APIs on TPL types. I also do not want to reproduce all of that.
Here is a short repro if you want to play with it:
static void Main()
{
Run().Wait();
}
static async Task Run()
{
Task[] tasks = new[] { CreateTask("ex1"), CreateTask("ex2") };
await Task.WhenAll(tasks);
}
static Task CreateTask(string message)
{
return Task.Factory.StartNew(() => { throw new Exception(message); });
}
Only one of the two exceptions is thrown in Run.
Note, that other questions on Stack Overflow do not address this specific problem. Please be careful when suggesting duplicates.

I disagree with the implication in your question title that await's behavior is undesired. It makes sense in the vast majority of scenarios. In a WhenAll situation, how often do you really need to know all of the error details, as opposed to just one?
The main difficulty with AggregateException is the exception handling, i.e., you lose the ability to catch a particular type.
That said, you can get the behavior you want with an extension method:
public static async Task WithAggregateException(this Task source)
{
try
{
await source.ConfigureAwait(false);
}
catch
{
// source.Exception may be null if the task was canceled.
if (source.Exception == null)
throw;
// EDI preserves the original exception's stack trace, if any.
ExceptionDispatchInfo.Capture(source.Exception).Throw();
}
}

I know I'm late but i found this neat little trick which does what you want. Since the full set of exceptions are available with on awaited Task, calling this Task's Wait or a .Result will throw an aggregate exception.
static void Main(string[] args)
{
var task = Run();
task.Wait();
}
public static async Task Run()
{
Task[] tasks = new[] { CreateTask("ex1"), CreateTask("ex2") };
var compositeTask = Task.WhenAll(tasks);
try
{
await compositeTask.ContinueWith((antecedant) => { }, TaskContinuationOptions.ExecuteSynchronously);
compositeTask.Wait();
}
catch (AggregateException aex)
{
foreach (var ex in aex.InnerExceptions)
{
Console.WriteLine(ex.Message);
}
}
}
static Task CreateTask(string message)
{
return Task.Factory.StartNew(() => { throw new Exception(message); });
}

Here is a shorter implementation of Stephen Cleary's WithAggregateException extension method:
public static async Task WithAggregateException(this Task source)
{
try { await source.ConfigureAwait(false); }
catch when (source.IsCanceled) { throw; }
catch { source.Wait(); }
}
public static async Task<T> WithAggregateException<T>(this Task<T> source)
{
try { return await source.ConfigureAwait(false); }
catch when (source.IsCanceled) { throw; }
catch { return source.Result; }
}
This approach is based on a suggestion by Stephen Toub in this API proposal in GitHub.
Update: I added a special handling of the cancellation case, to prevent the awkwardness of propagating an AggregateException that contains an OperationCanceledException. Now the OperationCanceledException is propagated directly, and the Task.IsCanceled status is preserved. Kudos to #noseratio for pointing out this flaw in the comments of this answer. Of course now this implementation is not much shorter than Stephen Cleary's approach!

Exception Handling (Task Parallel Library)
I could say more but it would just be padding. Play with it, it does work as they say. You just have to be careful.
maybe you want this
God (Jon Skeet) explains await exception handling
(personally i shy away from await, but thats just my preference)
in response to comments (too long for a comment reply)
Then use threads as your starting point for an analogous argument as the best practises there will be the source of ones for here.
Exceptions happily get swallowed unless you implement code to pass them out (for instance the async pattern that the await is preumably wrapping ... you add them to an event args object when you raise an event). When you have a scenario where you fire up an arbitrary number of threads and execute on them you have no control over order or the point at which you terminate each thread. Moreover you would never use this pattern if an error on one was relevant to another. Therefor you are strongly implying that execution of the rest is completley independent - IE you are strongly implying that exceptions on these threads have already been handled as exceptions. If you want to do something beyond handling exceptions in these threads in the threads they occur in (which is bizzarre) you should add them to a locking collection that is passed in by reference - you are no longer considering exceptions as exceptions but as a piece of information - use a concurrent bag, wrap the exception in the info you need to identify the context it came from - which would of been passed into it.
Don't conflate your use cases.

I don't want to give up the practice to only catch the exceptions I expect. This leads me to the following extension method:
public static async Task NoSwallow<TException>(this Task task) where TException : Exception {
try {
await task;
} catch (TException) {
var unexpectedEx = task.Exception
.Flatten()
.InnerExceptions
.FirstOrDefault(ex => !(ex is TException));
if (unexpectedEx != null) {
throw new NotImplementedException(null, unexpectedEx);
} else {
throw task.Exception;
}
}
}
The consuming code could go like this:
try {
await Task.WhenAll(tasks).NoSwallow<MyException>();
catch (AggregateException ex) {
HandleExceptions(ex);
}
A bone-headed exception will have the same effect as in synchronous world, even in case it is thrown concurrently with a MyException by chance. The wrapping with NotImplementedException helps to not loose the original stack trace.

Extension that wraps original aggregation exception and doesn't change return type, so it can still be used with Task<T>
public static Task<T> UnswallowExceptions<T>(this Task<T> t)
=> t.ContinueWith(t => t.IsFaulted ? throw new AggregateException("whatever", t.Exception) : t.Result);
Example:
Task<T[]> RunTasks(Task<T>[] tasks) =>
Task.WhenAll(CreateSometasks()).UnswallowExceptions();
try
{ var result = await CreateTasks(); }
catch(AggregateException ex) { } //ex is original aggregation exception here
NOTE This method will throw if task was canceled, use another approach if cancelling is important for you