I'd like to properly understand the consequences of failing to observe an exception thrown on a Task used in a fire and forget manner without exception handling.
Here's an extract from CLR via C#, Third Edition by Jeffry Richter: "[...] when a Task object is garbage collected, its Finalize method checks to see if the Task experienced an unobserved exception; if it has, Task's Finalize method throws [an exception]. Since you cannot catch an exception thrown by the CLR's finalizer thread, your process is terminated immediately."
I am writing some test code to bring about a termination but am unable to cause one.
Using the test code here, I am able to see the TaskScheduler.UnobservedTaskException handler being called. However, if I comment out the event handler subscription, the exception appears to be swallowed and does not bring about termination of the program.
I've tried this using the .NET Framework on both versions 4 and 4.8 with a Release build.
How do I demonstrate that failing to observe an exception thrown on a Task does indeed cause a crash?
The problem is correctly identified by Jon Skeet in his comment to the original post.
The best resource I found concerning this topic is by Stephen Toub.
tldr:
"To make it easier for developers to write asynchronous code based on Tasks, .NET 4.5 changes the default exception behavior for unobserved exceptions. While unobserved exceptions will still cause the UnobservedTaskException event to be raised (not doing so would be a breaking change), the process will not crash by default. Rather, the exception will end up getting eaten after the event is raised, regardless of whether an event handler observes the exception. This behavior can be configured, though. A new CLR configuration flag may be used to revert back to the crashing behavior of .NET 4, e.g."
<configuration>
<runtime>
<ThrowUnobservedTaskExceptions enabled=”true”/>
</runtime>
</configuration>
I'm using similar code to below for a non critical fire-and-forget operation in ASP.NET
private void SomeMethod()
{
FireAndForgetAsync();
}
private async Task FireAndForgetAsync()
{
// Simulate medium running IO blocking operation
await Task.Delay(sleepTime).ConfigureAwait(false);
// Simulating the rare scenario where an error occurs
throw new Exception("An Error Occurred");
}
The exception is captured in the task being returned from the async method, which prevents an unhandled exception altogether crashing the application..
But would this code cause any memory leaks? Because the returned Task (and its exception) are not assigned / accessed? My thoughts are that it would simply be garbage collected unless some .NET framework code is holding a reference to the task..
There isn't a risk for a memory leak in your code. At least From the snippet you provided.
Though, your application may be terminated by the CLR when the Task is being finalized due to an unhandled Exception.
If you create a Task, and you don't ever call task.Wait() or try to retrieve the result of a Task, when the task is collected by the garbage collector, it will tear down your application during finalization. For details, see MSDN's page on Exception Handling in the TPL.
(Reed Copsey's Answer)
There are more than one solution/workarounds for this. You can find more information about this in this SOF post:
A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was
(As I3arnon mentioned, this behavior is changed in .Net 4.5. )
There's no danger of a memory leak in your code. As long as the task has ended (which it has) there's nothing stopping the Garbage Collector of collecting it. You may use a lot of memory if you fire a unusual amount of those tasks, but they will be GCed.
There would be a chance for memory leaks if those tasks will not end (if they have a while (true) for example) or if you were keeping references to them.
As mentioned in the comments. Using "fire and forget" and async-await is heavily discouraged.
MSDN states that StackOverflowException can't be caught by try-catch block starting with .NET Framework 2.
Starting with the .NET Framework version 2.0, a StackOverflowException object cannot be caught by a try-catch block and the corresponding process is terminated by default.
Are there any other exceptions with the same behavior?
Yes, there are some others:
The ThreadAbortedException is special. It will always be re-raised when caught unless the catch block calls ResetAbort(). It is entirely uncatchable when the CLR performs a rude abort of the thread. Done when the AppDomain gets unloaded for example, typically at program exit.
Any native exceptions thrown by unmanaged code in a thread that got started by native code are uncatchable. The common scenario here is COM components that start their own threads. The CLR is powerless to trap such exceptions, it doesn't know about the thread and can't inject a catch block. If the native code doesn't catch the exception then Windows terminates the process.
Any exceptions thrown by finalizers, unless they are critical finalizers. They'll abort the finalizer thread which terminates the process.
Starting with .NET 4.0, an ExecutionEngineException is uncatchable. It is thrown by the CLR when it detects that its internal data structures are compromised. Most typically by an AccessViolationException that's raised while the garbage collector is busy. Continuing to execute managed code when the GC heap is compromised is a risky proposition, and exploitable, .NET 4 pulled the plug on it entirely.
Starting with the .NET 4.0 version of the CLR, but possibly also present in unmanaged code that you interop with in earlier versions, Microsoft's secure CRT can terminate a program instantly when a security problem is detected. This is not actually an exception under the hood, the process is instantly terminated since the code considers the process compromised and not capable of safely processing exceptions. A common case is where the stack frame of native function is smashed, a common problem in native code and used by viral code to tinker with the return address to run arbitrary code. An attack scenario called "stack buffer overflow". There were a few false alarms in CLR code, early after the .NET 4.0 release but I haven't seen any in quite a while. You can trigger such an abort yourself by writing beyond the bounds of a stackalloc.
Quite infamously, exceptions thrown by Windows message handlers when you run code in 32-bit mode in the WOW64 emulation layer on a 64-bit operating system and you have a debugger attached. Best known for the troublesome Load event in Winforms but also present for other messages and in other runtime environments. The ugly details are in this answer.
Starting with .NET 4.5, exceptions that Microsoft classifies as Corrupted State Exceptions (CSEs). They can be caught, but that should only ever be done by a top-level exception handler that doesn't do anything but generate a diagnostic for the user's benefit and terminates the app unconditionally. Backgrounder is available in this magazine article.
Any exception that is thrown by the jitter before your code can start running cannot be caught or reported. Failure to compile your Main() method is the common case, typically a FileNotFoundException.
I have background threads in my application. When one of the threads gets an exception that is not handled the entire CLR exits.
Is it normal behavior, or is it a bug in the CLR?
I would expect that the thread will exit but the CLR will continue working.
The default behavior in .NET applications is to exit whenever an unhandled exception occurs. When an exception goes unhandled, the program is in an unknown and possibly unsteady state. Just because it happened in a background thread doesn't mean that the error won't affect the rest of the program. The most prudent course for the runtime in that situation is to dump the program.
You might look into AppDomain.CurrentDomain.UnhandledException, which will allow you to catch unhandled exceptions and react accordingly. A better solution is to wrap your thread proc with a try...catch. But only have it handle those exceptions it knows how to handle. Doing this:
void MyThreadProc()
{
try
{
// ...
}
catch
{
// handle all exceptions
// This is a BAD idea
}
}
Is a really bad idea, because it can mask exceptions that you really do want to be propagated to the main program.
Your expected behavior used to be the behavior back in 1.1. It was generally considered to have been a bad idea. When you have an unhandled exception in any thread, your process can be left in an inconsistent state. Updates to shared data may be partially applied, etc. The runtime doesn't have the information to safely handle this scenario, or even know how you want to handle this scenario, so its choice would amount to terminating the thread and leaving your program in a strange state. This could lead to resource leaks, hangs, corruption of data, etc. By terminating the process given an unhandled exception you know exactly what happens, the process ends.
This is a normal behavior of the CLR from v2.0. Here is a MSDN post on this. To avoid process from terminating you could use something like this
<legacyUnhandledExceptionPolicy enabled="1"/>
which is not advisable.
It is normal behavior. Perhaps you want to catch the exception to prevent the application from exiting.
I got a thread that is just banishing.. i'd like to know who is killing my thread and why.
It occurs to me my thread is being killed by the OS, but i'd like to confirm this and if possible to know why it's killing it.
As for the thread, i can assert it has at least 40 min of execution before dying, but it suddenly dies around 5 min.
public void RunWorker()
{
Thread worker = new Thread(delegate()
{
try
{
DoSomethingForALongLongTime();
}
catch(Exception e)
{
//Nothing is never logged :(
LogException(e);
throw e;
}
});
worker.IsBackground = true;
worker.SetApartmentState(System.Threading.ApartmentState.STA);
worker.Start();
}
EDIT: Addressing answers
Try/Catch Possible exceptions:
It's implemented and it catches nothing :(
Main Thread dying:
This thread is created by the web server, which continues to run
Work completion:
The work is not completed, as it finally affects the database, i can check whether it's done or not when the thread dies.
Having thought of these things brought me to this question, who is killing my threads??
ps. It's not Lady Goldent in the living room with the candle stick :)
Various people (including myself, here) pointed out that hosting a long-running thread in IIS is a bad idea. Your thread will being running inside an IIS 'worker process'. These processes are periodically terminated (recycled) by IIS, which will cause your thread to die.
I suggest that you try turning-off IIS worker process recycling to see if that makes a difference. You can find more information here.
Your thread probably just threw an exception. Try putting a try/catch block around DoSomethingForALongLongTime and see what it picks up.
Update: I didn't notice before that you were starting this from a web server. That can be a very bad idea. In particular, is the separate thread using any information derived from HttpContext.Current? That would include Request, Response, Session, etc., as well as any information from the page.
This is bad because these things only last as long as the request lasts. Once the request is over, they become invalid, to say the very least.
If you need to kick off a long-running thread from within a web application or web service, then you should create a simple Windows Service and host a WCF service within it. Have the web page then send all the information needed to perform the task to the service. The service can even use MSMQ as a transport, which will ensure that no messages are lost, even if the service gets busy.
A potential way to get more information: attach a debugger and break on thread termination. Depending on how your thread is being terminated, this might not work.
Download Debugging Tools for Windows if you don't already have it
Run windbg.exe, attach to your process
Break into windbg, type sxe et to enable breaking on thread exit
When the debugger breaks, inspect the state of the system, other threads, etc.
To get the managed stack, load sos.dll (.loadby sos mscorsvr, .loadby sos mscorwks, or .loadby sos clr should work), then run !clrstack (see !help for other sos commands)
If you get a lot of noise from other threads exiting, script windbg to continue after breaking if it's not the thread ID you care about.
Edit: If you think the thread is being terminated from within your process, you can also set a breakpoint on TerminateThread (bp kernel32!TerminateThread) and ExitThread (bp kernel32!ExitThread) to catch the stack of the killer.
I don't know the answer, but some thoughts:
Could it be throwing an exception? Have you tried putting a try/catch around the DoSomethingForALongLongTime() call?
Are there any points where it exits normally? Try putting some logging on them.
Do you get the same behaviour in and out of the debugger? Does the output window in the debugger provide any hints?
UPDATE
You said:
This thread is created by the web
server, which continues to run
If the thread is running inside asp.net then it may be that the thread is being killed when the asp.net worker process recycles, which it will do periodically. You could try turning off worker process recycling and see if that makes any difference.
Your edit reveals the answer:
It's the butler web server.
How exactly do you host these threads? A webserver environment isn't exactly designed to host long living processes. In fact, it is probably configured to halt runaway sites, every 40 minutes maybe?
Edit:
For a quick fix, your best chance is to set worker.IsBackground = false; because your current setting of true allows the system to kill the parent-thread w/o waiting for your bgw.
On another note, there is little point in using a BackgroundWorker in an ASP.NET application, it is intended for WinForms and WPF. It would be better to create a separate thread for this, since you are changing some of the Threads properties. That is not advised for a ThreadPool (Bgw) thread.
The process might be terminating. That would be what worker.IsBackground = true; is designed to do, kill your thread when the main thread exits.
A background thread will only run as long there are foreground threads runnnig.
As soon that all foreground threads end, any background thread still running will aborted.
If checking for an exception doesn't show anything useful, get your thread code to write to a log file at key points. You'll then be able to see exactly when it stops working and hopefully why.
A simple answer would be: "The killer doesn't leave a name card" ;)
If your thread is hosted in IIS, probably the thread is killed by the app pool process which recycles. The server might continue running but the process which hosts your item is stopped untill a new request fires everything up again.
If your thread is hosted in an executable, the only way it can be killed is by killing the thread yourself, throwing an exception in the thread or terminating the host process
Hope this helps.
You can try to increase executionTimeout value of configuration\system.web\httpRuntime in web.config (default value is 110 seconds in .NET 4.0 and 90 in corresponds to http://msdn.microsoft.com/en-us/library/e1f13641.aspx). You can try to change it dynamically Server.ScriptTimeout = 300 (see http://www.beansoftware.com/ASP.NET-Tutorials/Long-Operations.aspx). It this parameter will not helps, then I think you have a problem other as thread recycling from IIS. How you can see default value of this parameter is much less as typical live time of your thread. I think, that your problem has another nature, but to be sure...
Why you set apartment state for the thread? Which COM objects you use in the working thread? Do you have an unmanaged code which do the most of work where you can also insert some code? I think you should have more information about SomethingForALongLongTime to be able to solve the problem.
And one more a little suggestion. Could you insert a line of code after calling SomethingForALongLongTime(); to be sure, that SomethingForALongLongTime not end without an exception?
UPDATED: To be absolutely sure that your thread will be not killed by IIS, you can try to create a process which do SomethingForALongLongTime(); instead of using threads.
When you call RunWorker(), you can add a reference to your thread to a list. Once you have detected that your thread has died, you can inspect the state of the thread, perhaps it will reveal how it died. Or, perhaps it hasn't died, its just waiting on some resource (like the connection to the database).
List runningThreads = ...
public void RunWorker() {
Thread worker = new Thread(delegate()
..
runningThreads.add(worker);
worker.Start();
}
public void checkThreads() {
for (Thread t : runningThreads) {
Console.WriteLine("ThreadState: {0}", t.ThreadState);
}
}
It could be throwing one of the various uncatcheable exceptions including Stack Overflow or Out of Memory. These are the hardest exceptions to track down.
What does memory consumption look like while this thread is running? Can you use a memory profiler on it to see if it's out of control? Can you add some logging in inner loops? If you have a recursive method, add a counter and throw an exception if it recurses an impossible number of times. Are you using large objects that could be causing large object heap fragmentation (causes out of memory errors even when you aren't really out).
You should instrument DoSomethingForALongLongTime() with lots of debug logs, so you can find out at what spot does the code stop executing. Or attach a debugger and break on all first chance exceptions.
use AsyncTasks to achieve your long running work in asp.net
Try use app domain UnhandledException event: http://msdn.microsoft.com/en-us/library/system.appdomain.unhandledexception.aspx
it may give you some information if you miss some exceptions