Develop Reference

Get ALL stacktraces in async/await application

I want to get info about all call stacks (or get all stacktraces) in my asynchronous C# application. I know, how to get stacktraces of all existing threads.
But how to get info about all call stacks released by await, which do not have a running thread on it?
CONTEXT EXAMPLE
Suppose the following code:
private static async Task Main()
{
async Task DeadlockMethod(SemaphoreSlim lock1, SemaphoreSlim lock2)
{
await lock1.WaitAsync();
await Task.Delay(500);
await lock2.WaitAsync(); // this line causes the deadlock
}
SemaphoreSlim lockA = new SemaphoreSlim(1);
SemaphoreSlim lockB = new SemaphoreSlim(1);
Task call1 = Task.Run(() => DeadlockMethod(lockA, lockB));
Task call2 = Task.Run(() => DeadlockMethod(lockB, lockA));
Task waitTask = Task.Delay(1000);
await Task.WhenAny(call1, call2, waitTask);
if (!call1.IsCompleted
&& !call2.IsCompleted)
{
// DUMP STACKTRACES to find the deadlock
}
}
I would like to dump all stacktraces, even those not having its thread currently, so that I can find the deadlock.
If line await lock2.WaitAsync(); is changed to lock2.Wait();, then it would be possible by already mentioned get stacktraces of all threads. But how to list all stacktraces without a running thread?
PREVENTION OF MISUNDERSTANDING:
The example is very simplified, it just ilustrates one of potential complications. The original problem is a complex multithreaded application, which runs on a server and many hard-to-investigate parallel-related issues may happen.
We would use the list of async/await stacktraces not only to find deadlocks, but also for other purposes. Therefore please do not advice me how to avoid deadlocks or how to write a multithreaded application - that is not the point of the question.
You can answer this generally, but also solution working at least on .Net Core 3.1 is enough.

I know, how to get stacktraces of all existing threads.
Just gonna give a bit of background here.
In Windows, threads are an OS concept. They're the unit of scheduling. So there's a definite list of threads somewhere, since that's what the OS scheduler uses.
Furthermore, each thread has a call stack. This dates back to the early days of computer programming. However, the purpose of the call stack is often misunderstood. The call stack is used as a sequence of return locations. When a method returns, it pops its call stack arguments off the stack and also the return location, and then jumps to the return location.
This is important to remember because the call stack does not represent how code got into a situation; it represents where code is going it returns from the current method. The call stack is where the code is going to, not where it came from. That is the reason the call stack exists: to direct the future code, not to assist diagnostics. Now, it does turn out that the call stack does have useful information on it for diagnostics since it gives an indication of where the code came from as well as where it's going, so that's why call stacks are on exceptions and are commonly used for diagnostics. But that's not the actual reason why the call stack exists; it's just a happy circumstance.
Now, enter asynchronous code.
In asynchronous code, the call stack still represents where the code is returning to (just like all call stacks). But in asynchronous code, the call stack no longer represents where the code came from. In the synchronous world, these two things were the same, and the call stack (which is necessary) can also be used to answer the question of "how did this code get here?". In the asynchronous world, the call stack is still necessary but only answers the question "where is this code going?" and cannot answer the question "how did this code get here?". To answer the "how did this code get here?" question you need a causality chain.
Furthermore, call stacks are necessary for correct operation (in both the synchronous and asynchronous worlds), and so the compiler/runtime ensures they exist. Causality chains are not necessary, and they are not provided out of the box. In the synchronous world, the call stack just happens to be a causality chain, which is nice, but that happy circumstance doesn't carry over to the asynchronous world.
When a thread is released by await, the stacktrace and all objects along the call stack are stored somewhere.
No; this is not the case. This would be true if async used fibers, but it doesn't. There is no call stack saved anywhere.
Because otherwise the continuation thread would lose context.
When an await resumes, it only needs sufficient context to continue executing its own method, and potentially completing the method. So, there is an async state machine structure that is boxed and placed on the heap; this structure contains references to local variables (including this and method arguments). But that is all that is necessary for program correctness; a call stack is not necessary and so it is not stored.
You can easily see this yourself by setting a breakpoint after an await and observing the call stack. You'll see that the call stack is gone after the first await yields. Or - more properly - the call stack represents the code that is continuing the async method, not the code that originally started the async method.
At the implementation level, async/await is more like callbacks than anything else. When a method hits an await, it sticks its state machine structure on the heap (if it hasn't already) and wires up a callback. That callback is triggered (invoked directly) when the task completes, and that continues executing the async method. When that async method completes, it completes its tasks, and anything awaiting those tasks are then invoked to continue executing. So, if a whole sequence of tasks complete, you actually end up with a call stack that is an inversion of the causality stack.
I would like to dump all stacktraces, even those not having its thread currently, so that I can find the deadlock.
So, there's a couple of problems here. First, there is no global list of all Task objects (or more generally, tasklike objects). And that would be a difficult thing to get.
Second, for each asynchronous method/task, there's no causality chain anyway. The compiler doesn't generate one because it's not necessary for correct operation.
That's not to say either of these problems are insurmountable - just difficult. I've done some work on the causality chain problem with my AsyncDiagnostics library. It's rather old at this point but should upgrade pretty easily to .NET Core. It uses PostSharp to modify the compiler-generated code for each method and manually track causality chains.
However, the goal of AsyncDiagnotics is to get causality chains onto exceptions. Getting a list of all tasklikes and associating causality chains with each one is another problem, likely requiring the use of an attached profiler. I'm aware of other companies who have wanted this solution, but none of them have dedicated the time necessary to create one; all of them have found it more efficient to implement code reviews, auditing, and developer training.

I marked Stephen Cleary's answer as the correct answer. He gave hints and explained deeply why it is so difficult.
I posted this alternative answer to explain, how we finally solved it and what we decided to do.
WORKAROUND SOLVING THE PROBLEM
Assumption: stacktraces including own code are enough.
Based on the assumption we can to this:
Encapsulate all called external Async methods (track their enter and leave)
Implement style check, which will warn about using any Async method out of your project namespaces
Ad 1.: Encapsulation
Suppose an external method Task ExternalObject.ExternalAsync(). We will create encapsulating extension method:
public static async Task MyExternalAsync(this ExternalObject obj)
{
using var disposable = AsyncStacktraces.MethodStarted();
await obj.ExternalAsync();
}
During the AsyncStacktraces.MethodStarted(); static call the current stacktrace will be recorded from Environment.StackTrace property into some static dictionary together with the disposable object. There will be no performance issues, since the async method itself is most probably much more expensive than stacktrace retrieval.
The disposable object will implement IDisposable interface. The .Dispose() method will remove the current stacktrace from the static dictionary at the end of the MyExternalAsync() method.
Usually only few tens of external Async methods are actually called in the solution, therefore the effort is quite low.
Ad 2.: Style check
Custom style check extension will warn when anybody uses external Async method directly. CI can be set-up so that it will not pass when this warning exists. On few places, where we will need a direct external Async method, we will use #pragma warning disable.

Safety of AsyncLocal in ASP.NET Core

For .NET Core, AsyncLocal is the replacement for CallContext. However, it is unclear how "safe" it is to use in ASP.NET Core.
In ASP.NET 4 (MVC 5) and earlier, the thread-agility model of ASP.NET made CallContext unstable. Thus in ASP.NET the only safe way to achieve the behavior of a per-request logical context, was to use HttpContext.Current.Items. Under the covers, HttpContext.Current.Items is implemented with CallContext, but it is done in a way that is safe for ASP.NET.
In contrast, in the context of OWIN/Katana Web API, the thread-agility model was not an issue. I was able to use CallContext safely, after careful considerations of how correctly to dispose it.
But now I'm dealing with ASP.NET Core. I would like to use the following middleware:
public class MultiTenancyMiddleware
{
private readonly RequestDelegate next;
static int random;
private static AsyncLocal<string> tenant = new AsyncLocal<string>();
//This is the new form of "CallContext".
public static AsyncLocal<string> Tenant
{
get { return tenant; }
private set { tenant = value; }
}
//This is the new verion of [ThreadStatic].
public static ThreadLocal<string> LocalTenant;
public MultiTenancyMiddleware(RequestDelegate next)
{
this.next = next;
}
public async Task Invoke(HttpContext context)
{
//Just some garbage test value...
Tenant.Value = context.Request.Path + random++;
await next.Invoke(context);
//using (LocalTenant = new AsyncLocal<string>()) {
// Tenant.Value = context.Request.Path + random++;
// await next.Invoke(context);
//}
}
}
So far, the above code seems to be working just fine. But there is at least one red flag. In the past, it was critical to ensure that CallContext was treated like a resource that must be freed after each invocation.
Now I see there is no self-evident way to "clean up" AsyncLocal.
I included code, commented out, showing how ThreadLocal<T> works. It is IDisposable, and so it has an obvious clean-up mechanism. In contrast, the AsyncLocal is not IDisposable. This is unnerving.
Is this because AsyncLocal is not yet in release-candidate condition? Or is this because it is truly no longer necessary to perform cleanup?
And even if AsyncLocal is being used properly in my above example, are there any kinds of old-school "thread agility" issues in ASP.NET Core that are going to make this middleware unworkable?
Special Note
For those unfamiliar with the issues CallContext has within ASP.NET apps, in this SO post, Jon Skeet references an in-depth discussion about the problem (which in turn references commentary from Scott Hanselman). This "problem" is not a bug - it is just a circumstance that must be carefully accounted for.
Furthermore, I can personally attest to this unfortunate behavior. When I build ASP.NET applications, I normally include load-tests as part of my automation test infrastructure. It is during load tests that I can witness CallContext become unstable (where perhaps 2% to 4% of requests show CallContext being corrupted. I have also seen cases where a Web API GET has stable CallContext behavior, but the POST operations are all unstable. The only way to achieve total stability is to rely on HttpContext.Current.Items.
However, in the case of ASP.NET Core, I cannot rely on HttpContext.Items...there is no such static access point. I'm also not yet able to create load tests for the .NET Core apps I'm tinkering with, which is partly why I've not answered this question for myself. :)
Again: Please understand that the "instability" and "problem" I'm discussing is not a bug at all. CallContext is not somehow flawed. The issue is simply a consequence of the thread dispatch model employed by ASP.NET. The solution is simply to know the issue exists, and to code accordingly (e.g. use HttpContext.Current.Items instead of CallContext, when inside an ASP.NET app).
My goal with this question is to understand how this dynamic applies (or does not) in ASP.NET Core, so that I don't accidentally build unstable code when using the new AsyncLocal construct.

I'm just looking into the source code of the ExecutionContext class for CoreClr:
https://github.com/dotnet/coreclr/blob/775003a4c72f0acc37eab84628fcef541533ba4e/src/mscorlib/src/System/Threading/ExecutionContext.cs
Base on my understanding of the code, the async local values are fields/variables of each ExecutionContext instance. They are not based on ThreadLocal or any thread specific persisted data store.
To verify this, in my testing with thread pool threads, an instance left in async local value is not accessible when the same thread pool thread is reused, and the "left" instance's destructor for cleaning up itself got called on next GC cycle, meaning the instance is GCed as expected.

Adding my two cents if someone lands on this page (like I did) after googling if AsyncLocal is "safe" in ASP.NET classic (non Core) application (some commenters have been asking this, and also I see a deleted answer asking about the same).
I wrote a small test that simulates asp.net's ThreadPool behavior
AsyncLocal is always cleared between requests even if thread pool re-uses an existing thread. So it is "safe" in that regard, no data will be leaked to another thread.
However, AsyncLocal can be cleared even within the same context (for example between code that runs in global.asax and the code that runs in controller). Because MVC-methods sometimes runs on a separate thread from non-MVC code, see this question for example: asp.net mvc 4, thread changed by model binding?
Using ThreadLocal is not safe b/c it preserves the value after the thread from Thread Pool is re-used. Never use ThreadLocal in web-applications. I know the question is not about ThreadLocal I'm just adding this warning to whoever considering using it, sorry.
Tested under ASP.NET MVC 5 .NET 4.7.2.
Overall, AsyncLocal seems like a perfect alternative to short-time caching stuff in HttpContext.Current in cases where you can't access the latter directly. You might end up re-calculating the cached value a bit more often though, but that's not a big problem.

MSpec: How to make static variables thread-safe?

I'm using MSpec for my latest project, and overall I'm really happy with it. However, I do have an issue with concurrency when my tests run in paralel and I'm wondering if anybody has run into this issue or, even better, has a solution?
MSpec heavily relies on static methods and variables to work.
Now it appears when I define static variables in my base classes, that are used by multiple test classes, and I run my tests in paralel, that they share the same static variables and thus interfere with eachother.
I'm using both NCrunch and Resharper as my testrunners and I'm experiencing the problem in both.
Anybody familiar with this problem?

Firstly, I would recommend reading the Thead Safety Guidelines on MSDN. This will give you a good overview of how and why to make methods thread safe in C#.
The following rules outline the design guidelines for implementing threading:
Avoid providing static methods that alter static state. In common server scenarios, static state is shared across requests, which means multiple threads can execute that code at the same time. This opens up the possibility for threading bugs. Consider using a design pattern that encapsulates data into instances that are not shared across requests.
... Adding locks to create thread-safe code decreases performance, increases lock contention, and creates the possibility for deadlock bugs to occur
Be aware of method calls in locked sections. Deadlocks can result when a static method in class A calls static methods in class B and vice versa. If A and B both synchronize their static methods, this will cause a deadlock. You might discover this deadlock only under heavy threading stress.
Be aware of issues with the lock statement (SyncLock in Visual Basic). It is tempting to use the lock statement to solve all threading problems. However, the System.Threading.Interlocked Class is superior for updates that must be atomic ...
As a general note a methodology which I prefer to use (where possible) is to make a method (static or otherwise) immutable. To do this, all variables should be local (created locally on the stack, or passed in as parameters to a method). By ensuring only local variables are used, or member variables are immutable each thread will operate in its own compartment and changes to variables will not affect another thread. This is a methodology I have used extensively in .NET simulation software to allow lock-less and therefore high performance multithreading in C#.
Alternatively, if variables must be member variables and mutable access to them may be protected by lock keywords. Be careful with the use of lock will cause context switching (slow down) and introduces the possibility of a deadlock situation. It also doesn't gaurantee thread safety as the use of lock must protect against the specific scenario you are trying to prevent.
For further reading I would suggest looking these related questions which describe thread safety and immutability in C#:
Designing a Thread Safe class
Achieving Thread Safety
Why are immutable objects thread safe
Best regards,

Static fields are not thread-safe by default. To make them thread-safe you can decorate them with the [ThreadStatic] attribute.
Have a look at ThreadStaticAttribute Class at MSDN for more info.

Which features make a class to be thread-safe?

In MSDN some .NET classes described like this:
"This type is thread safe."
or
"Public static (Shared in Visual Basic) members of this type are thread safe. Instance members are not guaranteed to be thread-safe.".
My question is which features make a class to be thread-safe?
Is there any standard, recommendation or guidelines for thread-safety programming?
When I use lock(C#) keyword, it means my class is thread-safe or not?
How to I evaluate thread-safety of a class? Is there any TESTS to be sure that a class is 100% thread safe?
Example:
public class MyClass
{
public void Method()
{
lock (this)
{
// Now, is my class 100% thread-safe like Microsoft classes?
}
}
type m_member1;
type m_member2;
}
thanks

Is there any standard, recommendation or guidelines for thread-safety
programming?
The most important standard is to ensure that all static members are thread-safe. You will see that all well written APIs including the .NET base class library makes this guarantee across the board. There is a really good reason for this. Since static members are shared across an AppDomain they could be used by many different threads without you even realizing it. It would be awkward at best to provide your own synchronization for every single static member access. Imagine what it would be like if Console.WriteLine were not thread-safe.
As far as recommendations and guidelines there are plenty of well established patterns for doing concurrent programming. The patterns that are out there cover a wide variety of programming problems and use many different synchronization mechanisms. The producer-consumer pattern is one of many well known patterns which happens to solve a large percentage of concurrent programming problems.
Read Threading in C# by Joseph Albahari. It is one of the best and most vetted resources available.
When I use lock(C#) keyword, it means my class is thread-safe or not?
Nope! There is no magic bullet that can make a class thread-safe. The lock keyword is but one of many different tools that can be used to make a class safe for simultaneous access by multiple threads. But, just using a lock will not guarantee anything. It is the correct use of synchronization mechanisms that makes code thread-safe. There are plenty ways to use these mechanisms incorrectly.
How to I evaluate thread-safety of a class? Is there any TESTS to be
sure that a class is 100% thread safe?
This is the million dollar question! It is incredibly difficult to test multithreaded code. The CHESS tool provided by Microsoft Research is one attempt at making life easier for concurrent programmers.

A class is generally considered thread-safe if its methods can be invoked by multiple threads concurrently without corrupting the state of the class or causing unexpected side-effects. There are many reasons why a class may not be thread safe, although some common reasons are that it contains some state that would be corrupted on concurrent access.
There are a number of ways to make a class thread-safe:
Make it immutable, if a class contains no state it is safe to use concurrently from multiple threads.
Employ locking to reduce concurrency. However, this is no guarantee of thread safety, it just ensures that a block of code will not be executed concurrently by multiple threads. If state is stored between method invocations this might still become inconsistent.
How you create a thread-safe class really depends on what you want to do with the class in question.
You also need to ask yourself, do I need to make my class threadsafe? a common model of most UI frameworks is that there is a single UI thread. For example in WinForms, WPF and Silverlight the majority of your code will be executed from the UI thread which means you do not have to build thread-safety into your classes.

First of all, don't use lock(this).
This can cause deadlocks. Because other code can lock that same object from outside the class' scope. You should create a local Object and use it as the class' lock.
Second, thread safety is a complicated issue. There's tons of material about this on the web.
As a rule of thumb, all public methods should be locked and thread safe for the class to be thread-safe.

A Class is considered thread safe if only one thread at a time can modify the state of the objects created from the class OR the class provide such functionality that multiple threads can call various methods of the class at same time.

When I use lock(C#) keyword, it means my class is thread-safe or not?
When you use lock it means that the portion of code inside the lock {} is thread safe. It doesn't guarantee that your class is thread safe. And as Yochai Timmer said it is not a good idea to lock(this)
How to I evaluate thread-safety of a class? Is there any TESTS to be sure that a class is 100% thread safe?
I am not sure there are any tests because it is always possible in multi-threading that you are by chance getting correct results. So in order to be sure you can go through the code of class to see how it is making sure it is thread safe

Very simple explanation:
Thread safe type means you don't need any additional synchronization mechanisms when using your type. Say you can create an instance pass a reference to another thread (or multiple threads) and use methods/properties from both threads without any additional overhead for thread safety.

How does HttpContext.Current work in a multi-threaded environment?

So I'm left wondering how exactly asp.net is able to scope a static property, when (to my knowledge) asp.net is multi-threaded.
One theory goes that the ASP.NET guys maintain a different appdomain for every request ... but that doesn't seem feasible.
Another theory goes that the .Current method looks at the current Thread, and then uses that to look up the http context in some hashtable (or other static storage mechanism).
Either way, it's a technique that seems really useful ... I'd like to utilize it, but definitely don't want to be debugging shared state bugs :-/

It isn't an AppDomain per-request. If you want to use thread-specific state, try:
[ThreadStatic]
private static int foo;
public static int Foo {get {return foo;} set {foo = value;}}
Each thread now gets its own value of Foo (or rather: 'foo').
This is not to be used lightly - it does have costs, but is a valid way of sharing state on a per-thread basis. I've used this once, maybe twice - and I've written a lot of C#. Don't over-use it...
In particular, watch out for initialization issues (i.e. forgetting to do it), and remember to clean up after yourself etc. And be very careful if you use any async code, as any callbacks/workers/etc will have different state.

What Marc says is the easiest most likely for what you are after, however ASP.NET is actually somewhat more complicated than what say ThreadStatic does, because single requests actually can be processed by multiple threads.. what I believe happens with ASP.NET is that the executing thread explicitely is told to switch context, of course the hosting environment is scheduling the threads and it has context of which httpcontext needs executing, so it finds a thread, tells the thread which context it should run in.. then sends it off on its way.
So the solution really isn't all that pretty sadly, where as threadstatic is much simpler and probably suits needs 95% of the time.