Develop Reference

Do I always need to use async/await?

I wanted to ask you about async/await. Namely, why does it always need to be used? (all my friends say so)
Example 1.
public async Task Boo()
{
await WriteInfoIntoFile("file.txt");
some other logic...
}
I have a Boo method, inside which I write something to files and then execute some logic. Asynchrony is used here so that the stream does not stop while the information is being written to the file. Everything is logical.
Example 2.
public async Task Bar()
{
var n = await GetNAsync(nId);
_uow.NRepository.Remove(n);
await _uow.CompleteAsync();
}
But for the second example, I have a question. Why here asynchronously get the entity, if without its presence it will still be impossible to work further?

why does it always need to be used?
It shouldn't always be used. Ideally (and especially for new code), it should be used for most I/O-based operations.
Why here asynchronously get the entity, if without its presence it will still be impossible to work further?
Asynchronous code is all about freeing up the calling thread. This brings two kinds of benefits, depending on where the code is running.
If the calling thread is a UI thread inside a GUI application, then asynchrony frees up the UI thread to handle user input. In other words, the application is more responsive.
If the calling thread is a server-side thread, e.g., an ASP.NET request thread, then asynchrony frees up that thread to handle other user requests. In other words, the server is able to scale further.

Depending on the context, you might or might not get some benefit. In case you call the second function from a desktop application, it allows the UI to stay responsive while the async code is being executed.

Why here asynchronously get the entity, if without its presence it will still be impossible to work further?
You are correct in the sense that this stream of work cannot proceed, but using async versions allows freeing up the thread to do other work:
I like this paragraph from Using Asynchronous Methods in ASP.NET MVC 4 to explain the benefits:
Processing Asynchronous Requests
In a web app that sees a large number of concurrent requests at start-up or has a bursty load (where concurrency increases suddenly), making web service calls asynchronous increases the responsiveness of the app. An asynchronous request takes the same amount of time to process as a synchronous request. If a request makes a web service call that requires two seconds to complete, the request takes two seconds whether it's performed synchronously or asynchronously. However during an asynchronous call, a thread isn't blocked from responding to other requests while it waits for the first request to complete. Therefore, asynchronous requests prevent request queuing and thread pool growth when there are many concurrent requests that invoke long-running operations.

Not sure what you mean by
without its presence it will still be impossible to work further
regarding example 2. As far as I can tell this code gets an entity by id from its repository asynchronously, removes it, then completes the transaction on its Unit of Work. Do you mean why it does not simply remove the entry by id? That would certainly be an improvement, but would still leave you with an asynchronous method as CompleteAsync is obviously asynchronous?
As to your general question, I don't think there is a general concensus to always use async/await.

In your second example there with the async/await keywords you are getting the value of the n variable asynchronously. This might be necessary because the GetNAsync method is likely performing some time-consuming operation, such as querying a database or perhaps you might be calling a webservice downstream, that could block the main thread of execution. By calling the method asynchronously, the rest of the code in the Bar method can continue to run while the query is being performed in the background.
But if in the GetNAsync you are just calling another method locally that is doing some basic CPU bound task then the async is pointless in my view. Aync works well when you are sure you need to wait such as network calls or I/O bound calls that will definitely add latency to your stack.

Make all conroller methods async easy way

I have a web api with controller methods being like this:
public HttpResponseMessage DoSmth() {
doingSmth();
}
I've read about how server threads here: Why should I return Task<IActionResult> in a Controller?
and here: https://learn.microsoft.com/en-us/archive/msdn-magazine/2014/october/async-programming-introduction-to-async-await-on-asp-net#synchronous-vs-asynchronous-request-handling
The implementation of business\data access layers is syncronous (and huge) and there is no time left to refactor everything into async task<>.
Is there some way to refactor only the controller, i.e. "topmost" methods into being asyncronous so server won't run out of threads?

Is there some way to refactor only the controller, i.e. "topmost" methods into being asyncronous so server won't run out of threads?
No.
You can think of it this way: synchronous calls block a thread. As long as your business logic / DAL methods are synchronous, they will block a thread.
As some have noted in the comments, Task.Run will allow those methods to block a thread pool thread instead of the calling thread, and that's a fine approach for many GUI apps (block thread pool threads instead of the GUI thread), but it's counterproductive on server apps (block one thread pool thread instead of another thread pool thread).
I recommend first analyzing your app and determining which requests are the slowest or most frequently called, and changing just those to be async all the way. You can use the flag argument hack (attribution: me) to avoid code duplication in your logic/DAL types when creating asynchronous equivalents of synchronous methods.

How can I run code that only has async APIs without deadlocking in an environment where I know the Thread Pool will be saturated?

I am trying to write helper methods for calling from BizTalk. BizTalk doesn't understand Tasks or async/await, so the helper methods must return a normal .NET return type not lifted into a Task<T>.
The helper methods use the IdentityModel.Clients.ActiveDirectory library and subsequently HttpClient to make a call out to an HTTP-based API and then cache this result. These classes only have an asynchronous API i.e. all the methods that do the work return Task<T> and end with -Async.
The way that BizTalk manages its thread pool essentially guarantees that the thread pool will be saturated (at a default of 25 worker threads) if it there is a high message load; for example a large number of files have been dropped at once - this is a feasible scenario in normal usage and not actually a problem. I have observed this through debugging.
When the helper code makes an API call, this is quite expensive as it returns a lot of data, and I only want one call to be in progress at a time. If all the implementation was synchronous I would just use a lock statement around the cache refresh, as the delay in processing messages is acceptable for the sake of ensuring synchronisation. Locking has been shown to deadlock, which makes sense to me, as the application architecture essentially guarantees that no threads will be available to complete the asynchronous methods. This is a more extreme case of the commonly-given advice of not locking in the context of asynchronous code, in that it is not just likely but certain to deadlock.
I've tried using SemaphoreSlim.WaitAsync to do the equivalent to locking but in a non-blocking fashion i.e. still prevent more than one thread from entering the block, but by making them yield instead of blocking. This doesn't solve the problem, as the top-level helper methods must still block to wait for the cache update to complete. My hypothesis is that the moment this wait yields the thread, it then gets swallowed up processing a new message - which then blocks it, preventing the thread which entered the semaphore from continuing.
The following pseudocode (i.e. please don't try and correct me on coding style issues irrelevant to the problem. It also can't be an MCVE unless you have an installation of BizTalk handy) illustrates the problem I am trying to solve:
public class Helper
{
public static string GetCachedValue(string key)
{
// need to wait until the cache is updated, but blocking here makes all worker threads unavailable
CacheData().Wait();
return _cache.GetValue(key);
}
private static DateTime _lastRead;
private static readonly Dictionary<string, string> Cache = new Dictionary<string, string>();
private static readonly SemaphoreSlim throttle = new SemaphoreSlim(1);
private static async Task CacheData()
{
try{
// stop more than one thread from entering this block at a time
await throttle.WaitAsync();
if(_lastRead < DateTime.Now.AddMinutes(-10))
{
var context = new AuthenticationContext(/* uninteresting parameters*/);
var token = await context.GetTokenAsync();
// can't use HttpClientFactory here because the .NET 4.5.2 implementation doesn't supply any way of setting the web proxy
var client = new HttpClient();
var data = await client.GetAsync("api/Data");
// unimportant cache update code
_lastRead = DateTime.Now;
}
}
finally
{
throttle.Release();
}
}
}
Is my understanding of what the fundamental problem is here correct?
How do I solve it?

Your question is actually "how do I do sync-over-async when the thread pool is saturated", and the only real answer is "you can't". The problem with sync-over-async is that it blocks a thread and then might require another thread to unblock that one.
One thing you can try is to install your own context (temporarily) on your thread. I have an AsyncContext type in my AsyncEx library that can do this. So at your BizTalk entry point, you could use that instead of blocking directly:
// Old code:
// var result = MyLogicAsync().GetAwaiter().GetResult();
var result = AsyncContext.Run(() => MyLogicAsync());
This will allow await continuations to run on your own thread by default. It kind of behaves similar to a UI message loop (just without UI).
Unfortunately, you can't guarantee that this will always work, because the continuations only capture that context by default. And for general-purpose libraries like ActiveDirectory and HttpClient, capturing context is considered bad practice; most library code goes out of its way to use the thread pool by always using ConfigureAwait(false).
So, the only way to avoid deadlocks in sync-over-async code is to ensure the thread pool is not saturated. If there was some way to limit BizTalk to some value and have the thread pool larger than that, then that would work.
A much more ideal solution would be to go "sync all the way". You'd want to replace HttpClient with WebClient, but from your description it sounds like ActiveDirectory doesn't support sync APIs.
You might get away with a hybrid solution: use WebClient to make the API call synchronous, and wrap all ActiveDirectory calls in AsyncContext.Run. I'm thinking this might work because the ASP.NET Core team has removed most/all ConfigureAwait(false) calls in their code. So the HTTP API would be synchronous, and the ActiveDirectory would be asynchronous but with its continuations running in the thread that is waiting for it (not requiring a thread pool thread).
But even if you get this working, it's not guaranteed in the future. Missing ConfigureAwait(false) can be considered a "bug", and if they "fix" the bug by adding ConfigureAwait(false) back in, then your code would be subject to deadlocks again.
The only truly guaranteed solution would be to force the asynchronous APIs to be synchronous, by writing your own proxy WebAPI that wraps the ActiveDirectory calls. Then your BizTalk code would talk to that API (and the other API) using WebClient, and all the BizTalk code would be synchronous at that point.

I have to say first, it looks like you just creating a lot of unnecessary problems by trying to do all this fancy stuff in a helper class. Because A) there's really no practical case for async operations in a helper class as it's not going to change message processing time, B) you should not be calling http endpoints in a helper class.
So, the correct solution is to call the http endpoint using an Ordered Delivery Send Port.
I say Ordered Delivery because of this: "and I only want one call to be in progress at a time", but...it really doesn't matter what you want, you should do serial calls only if the service requires it or has capacity issues.
If you want to internally cache the results, there are many examples of this such as this here (all .Net rules apply): Implement Caching for your BizTalk applications using "static" classes and methods

If the number of threads is the primary issue, you can reduce or increase the number of threads by adding the address and maxconnection setting to the connectionMangement section of the BTSNTSvc64.exe.config or BTSNTSvc.exe.config file if it is 32 bit. Or as Johns said in his answer, if you want it single threaded, tick Ordered delivery on the Send Port.
<system.net>
<connectionManagement>
<add address="*" maxconnection="25" />
<add address="https://LIMITEDSERVER.com*" maxconnection="5" />
<add address="https://GRUNTYSERVER.com*" maxconnection="50" />
</connectionManagement>
</system.net>
If you want it Async then you need to have a one way send port in BizTalk talking to your target system, and a Receive Location in BizTalk that the other system can send the response to.

C# async/await - UI thread vs. other threads [duplicate]

I've been reading about the new async and await operators in C# and tried to figure out in which circumstances they would possibly be useful to me. I studied several MSDN articles and here's what I read between the lines:
You can use async for Windows Forms and WPF event handlers, so they can perform lengthy tasks without blocking the UI thread while the bulk of the operation is being executed.
async void button1_Click(object sender, EventArgs e)
{
// even though this call takes a while, the UI thread will not block
// while it is executing, therefore allowing further event handlers to
// be invoked.
await SomeLengthyOperationAsync();
}
A method using await must be async, which means that the usage of any async function somewhere in your code ultimately forces all methods in the calling sequence from the UI event handlers up until the lowest-level async method to be async as well.
In other words, if you create a thread with an ordinary good old ThreadStart entry point (or a Console application with good old static int Main(string[] args)), then you cannot use async and await because at one point you would have to use await, and make the method that uses it async, and hence in the calling method you also have to use await and make that one async and so on. But once you reach the thread entry point (or Main()), there's no caller to which an await would yield control to.
So basically you cannot use async and await without having a GUI that uses the standard WinForms and WPF message loop. I guess all that makes indeed sense, since MSDN states that async programming does not mean multithreading, but using the UI thread's spare time instead; when using a console application or a thread with a user defined entry point, multithreading would be necessary to perform asynchronous operations (if not using a compatible message loop).
My question is, are these assumptions accurate?

So basically you cannot use async and await without having a GUI that uses the standard WinForms and WPF message loop.
That's absolutely not the case.
In Windows Forms and WPF, async/await has the handy property of coming back to the UI thread when the asynchronous operation you were awaiting has completed, but that doesn't mean that's the only purpose to it.
If an asynchronous method executes on a thread-pool thread - e.g. in a web service - then the continuation (the rest of the asynchronous method) will simply execute in any thread-pool thread, with the context (security etc) preserved appropriately. This is still really useful for keeping the number of threads down.
For example, suppose you have a high traffic web service which mostly proxies requests to other web services. It spends most of its time waiting for other things, whether that's due to network traffic or genuine time at another service (e.g. a datbase). You shouldn't need lots of threads for that - but with blocking calls, you naturally end up with a thread per request. With async/await, you'd end up with very few threads, because very few requests would actually need any work performed for them at any one point in time, even if there were a lot of requests "in flight".
The trouble is that async/await is most easily demonstrated with UI code, because everyone knows the pain of either using background threads properly or doing too much work in the UI thread. That doesn't mean it's the only place the feature is useful though - far from it.
Various server-side technologies (MVC and WCF for example) already have support for asynchronous methods, and I'd expect others to follow suit.

A method using await must be async, which means that the usage of any async function somewhere in your code ultimately forces all methods in the calling sequence from the UI event handlers up until the lowest-level async method to be async as well.
Not true - methods marked async just mean they can use await, but callers of those methods have no restrictions. If the method returns Task or Task<T> then they can use ContinueWith or anything else you could do with tasks in 4.0
A good non-UI example is MVC4 AsyncController.
Ultimately, async/await is mostly about getting the compiler rewriting so you can write what looks like synchronous code and avoid all the callbacks like you had to do before async/await was added. It also helps with the SynchronizationContext handling, useful for scenarios with thread affinity (UI frameworks, ASP.NET), but even without those, it's still useful. Main can always do DoStuffAsync().Wait(); for instance. :)

My question is, are these assumptions accurate?
No.
You can use async for Windows Forms and WPF event handlers, so they can perform lengthy tasks without blocking the UI thread while the bulk of the operation is being executed.
True. Also true for other UI applications including Silverlight and Windows Store.
And also true for ASP.NET. In this case, it's the HTTP request thread that is not blocked.
A method using await must be async, which means that the usage of any async function somewhere in your code ultimately forces all methods in the calling sequence from the UI event handlers up until the lowest-level async method to be async as well.
This is a best practice ("async all the way down"), but it's not strictly required. You can block on the result of an asynchronous operation; many people choose to do this in Console applications.
an ordinary good old ThreadStart entry point
Well... I do have to take issue with "ordinary good old". As I explain on my blog, Thread is pretty much the worst option you have for doing background operations.
I recommend you review my introduction to async and await, and follow up with the async / await FAQ.

async-await is only wrapper for Task class manipulations, which is part of so named Tasks Parallel Library - TPL(published before async-await auto code generation tech.)
So fact is you may not use any references to UI controls within async - await.
Typically async-await is powerfull tool for any web and server relations, loading resources, sql. It works with smart waiting data with alive UI.
Typically TPL application: from simple big size loop till multi stages parallel calculations in complex calculations based on shared data (ContinueWith and so on)

Is async and await exclusively for GUI-based asynchronous programming?

I've been reading about the new async and await operators in C# and tried to figure out in which circumstances they would possibly be useful to me. I studied several MSDN articles and here's what I read between the lines:
You can use async for Windows Forms and WPF event handlers, so they can perform lengthy tasks without blocking the UI thread while the bulk of the operation is being executed.
async void button1_Click(object sender, EventArgs e)
{
// even though this call takes a while, the UI thread will not block
// while it is executing, therefore allowing further event handlers to
// be invoked.
await SomeLengthyOperationAsync();
}
A method using await must be async, which means that the usage of any async function somewhere in your code ultimately forces all methods in the calling sequence from the UI event handlers up until the lowest-level async method to be async as well.
In other words, if you create a thread with an ordinary good old ThreadStart entry point (or a Console application with good old static int Main(string[] args)), then you cannot use async and await because at one point you would have to use await, and make the method that uses it async, and hence in the calling method you also have to use await and make that one async and so on. But once you reach the thread entry point (or Main()), there's no caller to which an await would yield control to.
So basically you cannot use async and await without having a GUI that uses the standard WinForms and WPF message loop. I guess all that makes indeed sense, since MSDN states that async programming does not mean multithreading, but using the UI thread's spare time instead; when using a console application or a thread with a user defined entry point, multithreading would be necessary to perform asynchronous operations (if not using a compatible message loop).
My question is, are these assumptions accurate?

So basically you cannot use async and await without having a GUI that uses the standard WinForms and WPF message loop.
That's absolutely not the case.
In Windows Forms and WPF, async/await has the handy property of coming back to the UI thread when the asynchronous operation you were awaiting has completed, but that doesn't mean that's the only purpose to it.
If an asynchronous method executes on a thread-pool thread - e.g. in a web service - then the continuation (the rest of the asynchronous method) will simply execute in any thread-pool thread, with the context (security etc) preserved appropriately. This is still really useful for keeping the number of threads down.
For example, suppose you have a high traffic web service which mostly proxies requests to other web services. It spends most of its time waiting for other things, whether that's due to network traffic or genuine time at another service (e.g. a datbase). You shouldn't need lots of threads for that - but with blocking calls, you naturally end up with a thread per request. With async/await, you'd end up with very few threads, because very few requests would actually need any work performed for them at any one point in time, even if there were a lot of requests "in flight".
The trouble is that async/await is most easily demonstrated with UI code, because everyone knows the pain of either using background threads properly or doing too much work in the UI thread. That doesn't mean it's the only place the feature is useful though - far from it.
Various server-side technologies (MVC and WCF for example) already have support for asynchronous methods, and I'd expect others to follow suit.

A method using await must be async, which means that the usage of any async function somewhere in your code ultimately forces all methods in the calling sequence from the UI event handlers up until the lowest-level async method to be async as well.
Not true - methods marked async just mean they can use await, but callers of those methods have no restrictions. If the method returns Task or Task<T> then they can use ContinueWith or anything else you could do with tasks in 4.0
A good non-UI example is MVC4 AsyncController.
Ultimately, async/await is mostly about getting the compiler rewriting so you can write what looks like synchronous code and avoid all the callbacks like you had to do before async/await was added. It also helps with the SynchronizationContext handling, useful for scenarios with thread affinity (UI frameworks, ASP.NET), but even without those, it's still useful. Main can always do DoStuffAsync().Wait(); for instance. :)

My question is, are these assumptions accurate?
No.
You can use async for Windows Forms and WPF event handlers, so they can perform lengthy tasks without blocking the UI thread while the bulk of the operation is being executed.
True. Also true for other UI applications including Silverlight and Windows Store.
And also true for ASP.NET. In this case, it's the HTTP request thread that is not blocked.
A method using await must be async, which means that the usage of any async function somewhere in your code ultimately forces all methods in the calling sequence from the UI event handlers up until the lowest-level async method to be async as well.
This is a best practice ("async all the way down"), but it's not strictly required. You can block on the result of an asynchronous operation; many people choose to do this in Console applications.
an ordinary good old ThreadStart entry point
Well... I do have to take issue with "ordinary good old". As I explain on my blog, Thread is pretty much the worst option you have for doing background operations.
I recommend you review my introduction to async and await, and follow up with the async / await FAQ.

async-await is only wrapper for Task class manipulations, which is part of so named Tasks Parallel Library - TPL(published before async-await auto code generation tech.)
So fact is you may not use any references to UI controls within async - await.
Typically async-await is powerfull tool for any web and server relations, loading resources, sql. It works with smart waiting data with alive UI.
Typically TPL application: from simple big size loop till multi stages parallel calculations in complex calculations based on shared data (ContinueWith and so on)