Develop Reference

Do I always need to use async/await?

I wanted to ask you about async/await. Namely, why does it always need to be used? (all my friends say so)
Example 1.
public async Task Boo()
{
await WriteInfoIntoFile("file.txt");
some other logic...
}
I have a Boo method, inside which I write something to files and then execute some logic. Asynchrony is used here so that the stream does not stop while the information is being written to the file. Everything is logical.
Example 2.
public async Task Bar()
{
var n = await GetNAsync(nId);
_uow.NRepository.Remove(n);
await _uow.CompleteAsync();
}
But for the second example, I have a question. Why here asynchronously get the entity, if without its presence it will still be impossible to work further?

why does it always need to be used?
It shouldn't always be used. Ideally (and especially for new code), it should be used for most I/O-based operations.
Why here asynchronously get the entity, if without its presence it will still be impossible to work further?
Asynchronous code is all about freeing up the calling thread. This brings two kinds of benefits, depending on where the code is running.
If the calling thread is a UI thread inside a GUI application, then asynchrony frees up the UI thread to handle user input. In other words, the application is more responsive.
If the calling thread is a server-side thread, e.g., an ASP.NET request thread, then asynchrony frees up that thread to handle other user requests. In other words, the server is able to scale further.

Depending on the context, you might or might not get some benefit. In case you call the second function from a desktop application, it allows the UI to stay responsive while the async code is being executed.

Why here asynchronously get the entity, if without its presence it will still be impossible to work further?
You are correct in the sense that this stream of work cannot proceed, but using async versions allows freeing up the thread to do other work:
I like this paragraph from Using Asynchronous Methods in ASP.NET MVC 4 to explain the benefits:
Processing Asynchronous Requests
In a web app that sees a large number of concurrent requests at start-up or has a bursty load (where concurrency increases suddenly), making web service calls asynchronous increases the responsiveness of the app. An asynchronous request takes the same amount of time to process as a synchronous request. If a request makes a web service call that requires two seconds to complete, the request takes two seconds whether it's performed synchronously or asynchronously. However during an asynchronous call, a thread isn't blocked from responding to other requests while it waits for the first request to complete. Therefore, asynchronous requests prevent request queuing and thread pool growth when there are many concurrent requests that invoke long-running operations.

Not sure what you mean by
without its presence it will still be impossible to work further
regarding example 2. As far as I can tell this code gets an entity by id from its repository asynchronously, removes it, then completes the transaction on its Unit of Work. Do you mean why it does not simply remove the entry by id? That would certainly be an improvement, but would still leave you with an asynchronous method as CompleteAsync is obviously asynchronous?
As to your general question, I don't think there is a general concensus to always use async/await.

In your second example there with the async/await keywords you are getting the value of the n variable asynchronously. This might be necessary because the GetNAsync method is likely performing some time-consuming operation, such as querying a database or perhaps you might be calling a webservice downstream, that could block the main thread of execution. By calling the method asynchronously, the rest of the code in the Bar method can continue to run while the query is being performed in the background.
But if in the GetNAsync you are just calling another method locally that is doing some basic CPU bound task then the async is pointless in my view. Aync works well when you are sure you need to wait such as network calls or I/O bound calls that will definitely add latency to your stack.

Why should I return Task<IActionResult> in a Controller?

So I have been trying to get the grasp for quite some time now but couldn't see the sense in declaring every controller-endpoint as an async method.
Let's look at a GET-Request to visualize the question.
This is my way to go with simple requests, just do the work and send the response.
[HttpGet]
public IActionResult GetUsers()
{
// Do some work and get a list of all user-objects.
List<User> userList = _dbService.ReadAllUsers();
return Ok(userList);
}
Below is the async Task<IActionResult> option I see very often, it does the same as the method above but the method itself is returning a Task. One could think, that this one is better because you can have multiple requests coming in and they get worked on asynchronously BUT I tested this approach and the approach above with the same results. Both can take multiple requests at once. So why should I choose this signature instead of the one above? I can only see the negative effects of this like transforming the code into a state-machine due to being async.
[HttpGet]
public async Task<IActionResult> GetUsers()
{
// Do some work and get a list of all user-objects.
List<User> userList = _dbService.ReadAllUsers();
return Ok(userList);
}
This approach below is also something I don't get the grasp off. I see a lot of code having exactly this setup. One async method they await and then returning the result. Awaiting like this makes the code sequential again instead of having the benefits of Multitasking/Multithreading. Am I wrong on this one?
[HttpGet]
public async Task<IActionResult> GetUsers()
{
// Do some work and get a list of all user-objects.
List<User> userList = await _dbService.ReadAllUsersAsync();
return Ok(userList);
}
It would be nice if you could enlighten me with facts so I can either continue developing like I do right now or know that I have been doing it wrong due to misunderstanding the concept.

Please read the "Synchronous vs. Asynchronous Request Handling" section of Intro to Async/Await on ASP.NET.
Both can take multiple requests at once.
Yes. This is because ASP.NET is multithreaded. So, in the synchronous case, you just have multiple threads invoking the same action method (on different controller instances).
For non-multithreaded platforms (e.g., Node.js), you have to make the code asynchronous to handle multiple requests in the same process. But on ASP.NET it's optional.
Awaiting like this makes the code sequential again instead of having the benefits of Multitasking/Multithreading.
Yes, it is sequential, but it's not synchronous. It's sequential in the sense that the async method executes one statement at a time, and that request isn't complete until the async method completes. But it's not synchronous - the synchronous code is also sequential, but it blocks a thread until the method completes.
So why should I choose this signature instead of the one above?
If your backend can scale, then the benefit of asynchronous action methods is scalability. Specifically, asynchronous action methods yield their thread while the asynchronous operation is in progress - in this case, GetUsers is not taking up a thread while the database is performing its query.
The benefit can be hard to see in a testing environment, because your server has threads to spare, so there's no observable difference between calling an asynchronous method 10 times (taking up 0 threads) and calling a synchronous method 10 times (taking up 10 threads, with another 54 to spare). You can artificially restrict the number of threads in your ASP.NET server and then do some tests to see the difference.
In a real-world server, you usually want to make it as asynchronous as possible so that your threads are available for handling other requests. Or, as described here:
Bear in mind that asynchronous code does not replace the thread pool. This isn’t thread pool or asynchronous code; it’s thread pool and asynchronous code. Asynchronous code allows your application to make optimum use of the thread pool. It takes the existing thread pool and turns it up to 11.
Bear in mind the "if" above; this particularly applies to existing code. If you have just a single SQL server backend, and if pretty much all your actions query the db, then changing them to be asynchronous may not be useful, since the scalability bottleneck is usually the db server and not the web server. But if your web app could use threads to handle non-db requests, or if your db backend is scalable (NoSQL, SQL Azure, etc), then changing action methods to be asynchronous would likely help.
For new code, I recommend asynchronous methods by default. Asynchronous makes better use of server resources and is more cloud-friendly (i.e., less expensive for pay-as-you-go hosting).

If your DB Service class has an async method for getting the user then you should see benefits. As soon as the request goes out to the DB then it is waiting on a network or disk response back from the service at the other end. While it is doing this the thread can be freed up to do other work, such as servicing other requests. As it stands in the non-async version the thread will just block and wait for a response.
With async controller actions you can also get a CancellationToken which will allow you to quit early if the token is signalled because the client at the other end has terminated the connection (but that may not work with all web servers).
[HttpGet]
public async Task<IActionResult> GetUsers(CancellationToken ct)
{
ct.ThrowIfCancellationRequested();
// Do some work and get a list of all user-objects.
List<User> userList = await _dbService.ReadAllUsersAsync(ct);
return Ok(userList);
}
So, if you have an expensive set of operations and the client disconnects, you can stop processing the request because the client won't ever receive it. This frees up the application's time and resources to deal with requests where the client is interested in a result coming back.
However, if you have a very simple action that requires no async calls then I probably wouldn't worry about it, and leave it as it is.

You are missing something fundamental here.
When you use async Task<> you are effectively saying, "run all the I/O code asynchronously and free up processing threads to... process".
At scale, your app will be able to serve more requests/second because your I/O is not tying up your CPU intensive work and vice versa.
You're not seeing much benefit now, because you're probably testing locally with plenty of resources at hand.
See
Understanding CPU and I/O bound for asynchronous operations
Async in depth

As you know, ASP.NET is based on a multi-threaded model of request execution. To put it simply, each request is run in its own thread, contrary to other single thread execution approaches like the event-loop in nodejs.
Now, threads are a finite resource. If you exhaust your thread pool (available threads) you can't continue handling tasks efficiently (or in most cases at all). If you use synchronous execution for your action handlers, you occupy those threads from the pool (even if they don't need to be). It is important to understand that those threads handle requests, they don't do input/output. I/O is handled by separate processes which are independent of the ASP.NET request threads. So, if in your action you have a DB fetch operation that takes let's say 15 seconds to execute, you're forcing your current thread to wait idle 15 seconds for the DB result to be returned so it can continue executing code. Therefore, if you have 50 such requests, you'll have 50 threads occupied in basically sleep mode. Obviously, this is not very scalable and becomes a problem soon enough. In order to use your resources efficiently, it is a good idea to preserve the state of the executing request when an I/O operation is reached and free the thread to handle another request in the meantime. When the I/O operation completes, the state is reassembled and given to a free thread in the pool to resume the handling. That's why you use async handling. It lets you use your finite resources more efficiently and prevents such thread starvation. Of course, it is not an ultimate solution. It helps you scale your application for higher load. If you don't have the need for it, don't use it as it just adds overhead.

The result returns a task from an asynchronous (non-blocking) operation which represents some work that should be done. The task can tell you if the work is completed and if the operation returns a result, the task gives you the result which won't be available until the task is completed. You can learn more about C# asynchronous programming from the official Microsoft Docs:
https://learn.microsoft.com/en-us/dotnet/api/system.threading.tasks.task?view=net-5.0

Async operations return a Task(it is not the result, but a promise made that will have the results once the task is completed.
The async methods should be awaited so that it waits till the task is completed. if you await an async method, the return values won't be a task anymore and will be the results you expected.
Imagine this async method:
async Task<SomeResult> SomeMethod()
{
...
}
the following code:
var result = SomeMethod(); // the result is a Task<SomeResult> which may have not completed yet
var result = await SomeMethod(); // the result is type of SomeResult
Now, why do we use async methods instead of sync methods:
Imagine that a Method is doing some job that may take a long time to complete, when it is done sync all other requests will be waiting till the execution of this long taking job to complete, as they are all happening in the same thread. However when it is async, the task will be done in a (possibly another) thread and will not block the other requests.

How can I run code that only has async APIs without deadlocking in an environment where I know the Thread Pool will be saturated?

I am trying to write helper methods for calling from BizTalk. BizTalk doesn't understand Tasks or async/await, so the helper methods must return a normal .NET return type not lifted into a Task<T>.
The helper methods use the IdentityModel.Clients.ActiveDirectory library and subsequently HttpClient to make a call out to an HTTP-based API and then cache this result. These classes only have an asynchronous API i.e. all the methods that do the work return Task<T> and end with -Async.
The way that BizTalk manages its thread pool essentially guarantees that the thread pool will be saturated (at a default of 25 worker threads) if it there is a high message load; for example a large number of files have been dropped at once - this is a feasible scenario in normal usage and not actually a problem. I have observed this through debugging.
When the helper code makes an API call, this is quite expensive as it returns a lot of data, and I only want one call to be in progress at a time. If all the implementation was synchronous I would just use a lock statement around the cache refresh, as the delay in processing messages is acceptable for the sake of ensuring synchronisation. Locking has been shown to deadlock, which makes sense to me, as the application architecture essentially guarantees that no threads will be available to complete the asynchronous methods. This is a more extreme case of the commonly-given advice of not locking in the context of asynchronous code, in that it is not just likely but certain to deadlock.
I've tried using SemaphoreSlim.WaitAsync to do the equivalent to locking but in a non-blocking fashion i.e. still prevent more than one thread from entering the block, but by making them yield instead of blocking. This doesn't solve the problem, as the top-level helper methods must still block to wait for the cache update to complete. My hypothesis is that the moment this wait yields the thread, it then gets swallowed up processing a new message - which then blocks it, preventing the thread which entered the semaphore from continuing.
The following pseudocode (i.e. please don't try and correct me on coding style issues irrelevant to the problem. It also can't be an MCVE unless you have an installation of BizTalk handy) illustrates the problem I am trying to solve:
public class Helper
{
public static string GetCachedValue(string key)
{
// need to wait until the cache is updated, but blocking here makes all worker threads unavailable
CacheData().Wait();
return _cache.GetValue(key);
}
private static DateTime _lastRead;
private static readonly Dictionary<string, string> Cache = new Dictionary<string, string>();
private static readonly SemaphoreSlim throttle = new SemaphoreSlim(1);
private static async Task CacheData()
{
try{
// stop more than one thread from entering this block at a time
await throttle.WaitAsync();
if(_lastRead < DateTime.Now.AddMinutes(-10))
{
var context = new AuthenticationContext(/* uninteresting parameters*/);
var token = await context.GetTokenAsync();
// can't use HttpClientFactory here because the .NET 4.5.2 implementation doesn't supply any way of setting the web proxy
var client = new HttpClient();
var data = await client.GetAsync("api/Data");
// unimportant cache update code
_lastRead = DateTime.Now;
}
}
finally
{
throttle.Release();
}
}
}
Is my understanding of what the fundamental problem is here correct?
How do I solve it?

Your question is actually "how do I do sync-over-async when the thread pool is saturated", and the only real answer is "you can't". The problem with sync-over-async is that it blocks a thread and then might require another thread to unblock that one.
One thing you can try is to install your own context (temporarily) on your thread. I have an AsyncContext type in my AsyncEx library that can do this. So at your BizTalk entry point, you could use that instead of blocking directly:
// Old code:
// var result = MyLogicAsync().GetAwaiter().GetResult();
var result = AsyncContext.Run(() => MyLogicAsync());
This will allow await continuations to run on your own thread by default. It kind of behaves similar to a UI message loop (just without UI).
Unfortunately, you can't guarantee that this will always work, because the continuations only capture that context by default. And for general-purpose libraries like ActiveDirectory and HttpClient, capturing context is considered bad practice; most library code goes out of its way to use the thread pool by always using ConfigureAwait(false).
So, the only way to avoid deadlocks in sync-over-async code is to ensure the thread pool is not saturated. If there was some way to limit BizTalk to some value and have the thread pool larger than that, then that would work.
A much more ideal solution would be to go "sync all the way". You'd want to replace HttpClient with WebClient, but from your description it sounds like ActiveDirectory doesn't support sync APIs.
You might get away with a hybrid solution: use WebClient to make the API call synchronous, and wrap all ActiveDirectory calls in AsyncContext.Run. I'm thinking this might work because the ASP.NET Core team has removed most/all ConfigureAwait(false) calls in their code. So the HTTP API would be synchronous, and the ActiveDirectory would be asynchronous but with its continuations running in the thread that is waiting for it (not requiring a thread pool thread).
But even if you get this working, it's not guaranteed in the future. Missing ConfigureAwait(false) can be considered a "bug", and if they "fix" the bug by adding ConfigureAwait(false) back in, then your code would be subject to deadlocks again.
The only truly guaranteed solution would be to force the asynchronous APIs to be synchronous, by writing your own proxy WebAPI that wraps the ActiveDirectory calls. Then your BizTalk code would talk to that API (and the other API) using WebClient, and all the BizTalk code would be synchronous at that point.

I have to say first, it looks like you just creating a lot of unnecessary problems by trying to do all this fancy stuff in a helper class. Because A) there's really no practical case for async operations in a helper class as it's not going to change message processing time, B) you should not be calling http endpoints in a helper class.
So, the correct solution is to call the http endpoint using an Ordered Delivery Send Port.
I say Ordered Delivery because of this: "and I only want one call to be in progress at a time", but...it really doesn't matter what you want, you should do serial calls only if the service requires it or has capacity issues.
If you want to internally cache the results, there are many examples of this such as this here (all .Net rules apply): Implement Caching for your BizTalk applications using "static" classes and methods

If the number of threads is the primary issue, you can reduce or increase the number of threads by adding the address and maxconnection setting to the connectionMangement section of the BTSNTSvc64.exe.config or BTSNTSvc.exe.config file if it is 32 bit. Or as Johns said in his answer, if you want it single threaded, tick Ordered delivery on the Send Port.
<system.net>
<connectionManagement>
<add address="*" maxconnection="25" />
<add address="https://LIMITEDSERVER.com*" maxconnection="5" />
<add address="https://GRUNTYSERVER.com*" maxconnection="50" />
</connectionManagement>
</system.net>
If you want it Async then you need to have a one way send port in BizTalk talking to your target system, and a Receive Location in BizTalk that the other system can send the response to.

When working with asynchronous web services, what should "await" and what shouldn't?

I'm developing an MVC web application that allows me to manage my data asynchronously through a web service.
It is my understanding this allows the CPU threads that access the app pool for the server upon which this website is running to return to the app pool after making a request so that they can be used to service other requests without stalling the entire thread.
Assuming my understanding is correct (although it may be badly worded), I got to thinking about when I should await things. Consider the function below:
public async Task<ActionResult> Index()
{
List<User> users = new List<User>();
using (var client = new HttpClient())
{
client.BaseAddress = new Uri("http://localhost:41979");
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(
new MediaTypeWithQualityHeaderValue("application/json"));
HttpResponseMessage response = await client.GetAsync("api/user/");
if (response.IsSuccessStatusCode)
{
users = await response.Content.ReadAsAsync<List<User>>();
}
}
return View(users);
}
All of my functions look similar except that they do different things with the data returned by the web service and I got to wondering, should I not await the return as well?
Something like:
return await View(users);
or
await return View(users);
I mean the website runs just fine so far except that I've had a bit of confusion to do with exactly what the web service should send back to the client website, but as I'm new to development involving a web service, I'm still wondering if I'm doing things properly and this has been eating at me for some time.

You can only await named or anonymous methods and lambda expressions which expose an asynchronous operation via Task, Task<T> or a custom awaiter. Since View by itself does nothing asynchronous, you can't await it.
What is actually the point of using await? Usually, you have some IO bound operations which is asynchronous by nature. By using async API's, you allow the thread to be non-blocked by returning to the thread-pool, making use of it to serve different requests. async does not change the nature of HTTP. It is still "request-response". When an async method yields control, it does not return a response to the client. It will only return once the action has completed.
View(object obj) returns a ViewResult, which in turn will transform your object into the desired output. ViewResult is not awaitable, as it doesn't expose any "promise" via an awaitable object. Thus, you can't asynchronously wait on it.

I got to thinking about when I should await things
It's better to always await the result from asynch calls.
If you don't await, you fire & forget, you don't receive response on your end in both success and error cases.

You "await" when you need to unwrap the async task as a value. Other wise you can return a task and have it run at a later time if needed. Also note that .Wait() is not the same as await. Wait() will block until the task has finished (side note I know). Also check your code, you have a syntax error in your method signature:
public async <Task>ActionResult> Index()
Should be
public async Task<ActionResult> Index()

I think it is a very good question and also very hard to answer it especially in the case of a web site.
I've had a bit of confusion to do with exactly what the web service should send back to the client website'
The most important thing to understand if you use async/await, then your action method code still serialized, I mean the next line will be executed only after the async operation finished. However there will be (at least) three threads:
To original web server worker thread in which the action method is
called. Originally the MVC infrastructure got this thread from the
pool, and dedicated this thread to serve the current request.
Optional thread(s) what are started by the async operation what can
be called with await.
A continuation thread (also a worker from the
pool) in which your action method is continuing after the await. Note
this thread will have the same context (HttpContext, user culture
etc) what was the original worker so it's transparent for you, but it will be an other thread freshly allocated from the pool.
For the first sight it has no sense: Why all those thread hocus-pocus in case if the operations in the action method are still serialized? I takes the same time... To understand this we must take a look to a desktop application say a WPF app.
To be short: There is a message loop and a dedicated thread which reads the UI (and other virtual) events from a queue and executes the event handlers in the context of this thread, like a button click event. In case you block this thread for 2 second the UI will be unresponsive for that time. So that's why we would like to do many things in async way in an other thread, and let the dedicated (UI) thread to fast return and process other messages from the queue. However this could be very inconvenient, because we sometime wanted to continue something after the async operation ended and we had the result. The C# feature async/await made this very convenient. This case the continuation thread is always the dedicated UI thread but in its (very) later round in its endless loop. To wrap it up:
In an event handler you can use async/await to still execute your operations in serialized, the continuation will be done always in the original dedicated UI thread but during the await this thread can loop, and process other messages.
Now back to the MVC action method: In this case your action method is the analogy of the event handler. Although it is still hard to understand the use of the thread complication: This case blocking the action method would not block anything (as it blocked the dedicated UI thread in WPF). Here are the facts:
Other clients (browsers) requests will be served by other threads. (so we are
not blocking anything by blocking this thread (*) keep reading
This request will not be served faster even if we use async/await because
the operations are serialized and we must wait (await) the result of
the async operation. (Do not confuse async/await with parallel processing)
So the use of async/await and the transparent thread tricks is not so obvious as it was in the case of the desktop application.
Still: Pool is not an endless resource. Releasing ASAP back to the pool (by await) the worker thread what was originally dedicated to serve the request, and let continue in an other thread after completion the async operation may lead better utilization of the thread pool and may lead better performance of the web server under extreme stress.

Aysnc webapi post return before finishing

I am making a asp.net webapi call that posts some data to the server and will need to be processed. The client does not need to wait for the processing to finish. I would like to return something like this
HttpResponseMessage objReturn = Request.CreateResponse(HttpStatusCode.Ok);
//start a thread to do some work processing the data
//return while the data is being processed
return objReturn;
Most of the example I find are about how to use async methods and wait for the processing to complete. I need the opposite.
thanks for you suggestions.
more code for those asking, the following code gives me a warning that the method lacks await and will run synchronously.
public async Task<HttpResponseMessage> Post()
{
HttpResponseMessage objReturn = Request.CreateResponse(HttpStatusCode.Ok);
//data processing logic
//something longer running that the client doesnt need to wait for
//like converting a pdf to jpg or other I/O operations
return objReturn;
}

If I read your question correctly, you want a user to call an API, quickly receive a response, and have that trigger a longer running task.
In general, you do not use Web Api to run this longer task, instead yo use a service (i.e. Windows Service).
That service will sit there ... waiting for work ...
your Api will give it work! (Using a database, queues, files, etc.)
However, depending on how important this is, how much effort, and how much time ... you may not want to create a whole separate service. There are some "tools" that can help you.
QueueBackgroundWorkItem
http://hangfire.io/
^^ They will help you run long tasks in your Api directly! ^^

The warning explains most of your problems. Just decorating a method with async does not mean that it runs asynchronous automatically. If you don't have any asynchronous work in your data processing logic it will run synchronously. Event if you have some asynchronous calls in there, the compiler can decide to run it synchronously if it think that's the better option. Remember that asynchronous work does NOT involve another thread.
Some hints what you can do. First, you should make your I/O calls asynchronous. The .NET framework offers a lot you can use here. Second, you should not do that work in a controller. A controller should be small and don't do heavy processing, because it is your communicator to the rest of the world. Pass everything that needs more processing to a queue where a worker role (such as a Windows Service) picks up the work that needs to be done. With that the controller has nothing to do as passing data to the queue, give a result to the client that it was put into the queue ... and done. After that your controller can pick up additional work.