We have a .NET console app that has many foreground threads.
If we kill the process using Task Manager or issuing killjob, kill from the command line in windows, is there a way by which we can gracefully shut down the application (adding manged code within the .net console app), something like having a function being called say TodoBeforeShutdown() that disposes objects, closes any open connections, etc.
P.S. - I read the other threads and they all suggested different ways to kill the process, rather than my specific question, what is the best way we can handle a terminate process, within the .NET managed code.
Thanks in advance.
Unfortunately, there is no event raised that you can handle whenever a process is killed.You can think of killing a process like cutting off the power to the computer—no matter what code you have designed to run on system shutdown, if the computer doesn't shut down gracefully or properly, that code is not going to run.
When you kill a process using Task Manager, it calls the Win32 TerminateProcess function, which unconditionally forces the process (including all of its owned threads) to exit. The execution of all threads/processes is halted, and all pending I/O requests are canceled. Your program is effectively dead. The TerminateProcess function does not invoke the shutdown sequence provided by the CLR, so your managed app would not even have any idea that is was being shut down.
You suggest that you're concerned about disposing objects whenever your application's process is terminated, but there are a couple of things worth pointing out here:
Always strive to minimize the amount of damage that could be done. Dispose of your objects as early as possible, whenever you are finished with them. Don't wait until later. At any given time, when your program's process is terminated, you should only be keeping the bare minimum number of objects around, which will leave fewer possibilities for leaks.
The operating system will generally clean up and free most of these resources (i.e., handles, etc.) upon termination.
Finally, it should go without saying that process termination in this way is truly an exceptional condition—even if some resources leak, that's to be expected. You're not supposed to shut an app down this way any more than you're supposed to kill necessary Windows system processes (even though you can when running as an Administrator).
If this is your regular plan to shut down your console application, you need to find another plan.
In short: You can't!Killing a process is exactly the opposite of a gracefull exit.If you are running Foreground Threads, sending a wm_exit won't shut down your app. Since you have a console app, you could simply redirect the console input to send an "exit" to your process.Further I think you could change the app to service (instead of a console application), this would offer you exactly what you are looking for -> interface for gracefull exit based on windows build-in tools/commands.
I'll start by saying I do not have a lot of experience in troubleshooting multi-threading problems. So a lot of what I've read about debugging race conditions, dead locks, live locks, etc. are strictly theoretical to me.
I have this .NET application that is making use of a dynamically loaded native win32 dll. If the dll is never loaded the application terminates without a problem. However, if the dll is loaded then when the user exits the application the UI disappears but the process never terminates.
I've turned on native code debugging in the project settings so I can see all the threads that are running. When the user closes the application window the main thread appears to terminate. I know this because if I perform a Break All in the Threads windows in Visual Studio the main thread is re-categorized as a worker thread and there is no call stack available for it. There are 20 other threads still active, all with call stacks. I've looked through the call stacks for all of these threads and nothing sticks out at me (there's no mention of the dll in any of the call stacks).
What steps can I take to narrow down the cause of this problem? Are there any additional tools I should be using to help pin point the problem?
This means that some of your Foreground Threads are still alive. Unlike Background Threads, Foreground Threads keeps the process alive.
You should either use Background Threads or Stop your Foregrounds Threads to be able to exit the process gracefully.
Windows application will automatically exit when all of its thread(s) are stopped.
As you said
If the dll is never loaded the application terminates without a
problem
I'm assuming all the running threads are unmanaged threads(not created by clr). There is no concept of background threads in native code. All threads must be terminated to terminate the process.
You must find a way to signal all threads to exit in that library. See if you have any api exposed for that.
If you don't find anything, you've got a "Sledge Hammer" approach. i.e Terminate the process with Environment.Exit which works almost always. Be sure you use this approach as a last resort as this can corrupt the process state.
I am developing a high performance App that is causing the main process to seemingly stop responding and crash - from time to time - due to overload (I get the Close Application Dialog so the App never exits per se, just freezes, which is annoying)
I welcome any clean way to pro grammatically detect when the App is frozen, so I can use a BAT (or else) to automatically Kill the process and Restart.
Of course, this is a temporary fix while auditing the App, but it comes very handy in the meanwhile.
TieBreaker : BTW, Is there a way to override windows' Exception Screen and just quit the App ???
This is mostly annoying feature most of the time.
EDIT :
FOR GOD'S SAKE : the app IS freaking freezing, THOUGH every single task run in BG Workers and threads !!! And I specified that in comments. Come'on, I am not that dumb. Just that your app runs BG workers does not imply it never freezes ! And As I said, please JUST answer my question, I am not looking for lessons on how to design my App, that I am already working on and I know what has to be done. As specified MANY times, I just need a fix on the server in the meantime.
Thank you.
I'll say it if noone else will :) Make a separate forms app that does-
Process[] prs = Process.GetProcesses();
foreach (Process pr in prs)
{
if (!pr.Responding)
{
try
{
pr.Kill();
}
catch { }
}
}
//then to restart-
var process = new Process
{
StartInfo = new ProcessStartInfo
{
FileName = #"C:\yourapp.exe"
}
};
process.Start();
Obviously oversimplified.
We handle this in a service by having the main service exe be little more than a shell that launches the child threads.
The child threads are then responsible for reporting back to the parent thread whenever they can to record the last date/time they were "seen".
On a regular interval, the service app checks the list of child processes and, if one hasn't been seen for a predetermined period of time (i.e. 2 minutes), but did not report that it had closed down, the parent process will first try to join the thread and shut it down gracefully (usually fails) and then, if that didn't work, aborts the thread.
We have used this approach successfully for many years, starting when an OCR service that we were running would hang constantly due to bugs in the OCR software.
Move all CPU instensive tasks off of the GUI and invoke back to the gui to report any statuses.
The main app should never freeze. I wrote an app that generated over 500 threads (all at once) and managed them as they asynchronously (.net 2) processed multiple database calls. I believe you need to systematically move all processes to a thread safe situation and remove any direct GUI calls.
ADDENDUM How did I run 500+ threads:
Usage of smart locks done from the beginning on all shared data locations. See my blog articlesSmart Resource Locking in C# .Net for Thread Safe CodeC# MultiThreading Using ThreadPool, Anonymous Delegates and Locks
Interface created which specified an action operation with data, error status as a properties.
Threading base class created (for classes in #4 and #5) which created, started and cleaned up thread opertions. No business logic, just a method of handling threading in one location.
Class created based on interface in #2 and also derived from step #3. Class via the interface was required to get data and place data (thread safe via localized lock) and report its status. Class also designed to give up thread cycle if no work done to be done via thread.Sleep(0) instead of busy waiting.
Manager class created (which ran on its own thread and derived from #3).It launched 1-N #4 classes to do work. Each of those instances was placed into a working list.
Manager class simply browsed the working list for instances which reported their work was done if it was done it moved the instance to the done work. Manager class also kept records of the status (thread safe via locks) to exposed properties (as well as any reported errors by the children instances). Manager gave up thread cycle after each run and slept for 250 milleseconds before starting again.
GUI thread had working timers which dealt with getting specific status from the manager. They would extract the data from the manager's properties and invoke back the GUI to target a control (gridview) which reported all statuses and errors.
By doing that, I was able to achieve individual threads doing specific work and if a process ran into a problem it reported it, or it reported success or failure. Those messages bubbled up to the manager which bubbled up to the timers which bubbled up to the GUI. The gui handled no business logic except to start the mananager and timers.
I get that this doesn't help your situation right now and I feel your pain. But until you are able to seperate out the business logic from the GUI and handle error situations in the threads (background workers) and bubble them up to the GUI, you will still have this frustration with the current code.
If something is locking that is a sign that business logic is too tightly coupled with GUI operation and that has to be divorced before you will get the performance you want from the GUI.
HTH
I got a thread that is just banishing.. i'd like to know who is killing my thread and why.
It occurs to me my thread is being killed by the OS, but i'd like to confirm this and if possible to know why it's killing it.
As for the thread, i can assert it has at least 40 min of execution before dying, but it suddenly dies around 5 min.
public void RunWorker()
{
Thread worker = new Thread(delegate()
{
try
{
DoSomethingForALongLongTime();
}
catch(Exception e)
{
//Nothing is never logged :(
LogException(e);
throw e;
}
});
worker.IsBackground = true;
worker.SetApartmentState(System.Threading.ApartmentState.STA);
worker.Start();
}
EDIT: Addressing answers
Try/Catch Possible exceptions:
It's implemented and it catches nothing :(
Main Thread dying:
This thread is created by the web server, which continues to run
Work completion:
The work is not completed, as it finally affects the database, i can check whether it's done or not when the thread dies.
Having thought of these things brought me to this question, who is killing my threads??
ps. It's not Lady Goldent in the living room with the candle stick :)
Various people (including myself, here) pointed out that hosting a long-running thread in IIS is a bad idea. Your thread will being running inside an IIS 'worker process'. These processes are periodically terminated (recycled) by IIS, which will cause your thread to die.
I suggest that you try turning-off IIS worker process recycling to see if that makes a difference. You can find more information here.
Your thread probably just threw an exception. Try putting a try/catch block around DoSomethingForALongLongTime and see what it picks up.
Update: I didn't notice before that you were starting this from a web server. That can be a very bad idea. In particular, is the separate thread using any information derived from HttpContext.Current? That would include Request, Response, Session, etc., as well as any information from the page.
This is bad because these things only last as long as the request lasts. Once the request is over, they become invalid, to say the very least.
If you need to kick off a long-running thread from within a web application or web service, then you should create a simple Windows Service and host a WCF service within it. Have the web page then send all the information needed to perform the task to the service. The service can even use MSMQ as a transport, which will ensure that no messages are lost, even if the service gets busy.
A potential way to get more information: attach a debugger and break on thread termination. Depending on how your thread is being terminated, this might not work.
Download Debugging Tools for Windows if you don't already have it
Run windbg.exe, attach to your process
Break into windbg, type sxe et to enable breaking on thread exit
When the debugger breaks, inspect the state of the system, other threads, etc.
To get the managed stack, load sos.dll (.loadby sos mscorsvr, .loadby sos mscorwks, or .loadby sos clr should work), then run !clrstack (see !help for other sos commands)
If you get a lot of noise from other threads exiting, script windbg to continue after breaking if it's not the thread ID you care about.
Edit: If you think the thread is being terminated from within your process, you can also set a breakpoint on TerminateThread (bp kernel32!TerminateThread) and ExitThread (bp kernel32!ExitThread) to catch the stack of the killer.
I don't know the answer, but some thoughts:
Could it be throwing an exception? Have you tried putting a try/catch around the DoSomethingForALongLongTime() call?
Are there any points where it exits normally? Try putting some logging on them.
Do you get the same behaviour in and out of the debugger? Does the output window in the debugger provide any hints?
UPDATE
You said:
This thread is created by the web
server, which continues to run
If the thread is running inside asp.net then it may be that the thread is being killed when the asp.net worker process recycles, which it will do periodically. You could try turning off worker process recycling and see if that makes any difference.
Your edit reveals the answer:
It's the butler web server.
How exactly do you host these threads? A webserver environment isn't exactly designed to host long living processes. In fact, it is probably configured to halt runaway sites, every 40 minutes maybe?
Edit:
For a quick fix, your best chance is to set worker.IsBackground = false; because your current setting of true allows the system to kill the parent-thread w/o waiting for your bgw.
On another note, there is little point in using a BackgroundWorker in an ASP.NET application, it is intended for WinForms and WPF. It would be better to create a separate thread for this, since you are changing some of the Threads properties. That is not advised for a ThreadPool (Bgw) thread.
The process might be terminating. That would be what worker.IsBackground = true; is designed to do, kill your thread when the main thread exits.
A background thread will only run as long there are foreground threads runnnig.
As soon that all foreground threads end, any background thread still running will aborted.
If checking for an exception doesn't show anything useful, get your thread code to write to a log file at key points. You'll then be able to see exactly when it stops working and hopefully why.
A simple answer would be: "The killer doesn't leave a name card" ;)
If your thread is hosted in IIS, probably the thread is killed by the app pool process which recycles. The server might continue running but the process which hosts your item is stopped untill a new request fires everything up again.
If your thread is hosted in an executable, the only way it can be killed is by killing the thread yourself, throwing an exception in the thread or terminating the host process
Hope this helps.
You can try to increase executionTimeout value of configuration\system.web\httpRuntime in web.config (default value is 110 seconds in .NET 4.0 and 90 in corresponds to http://msdn.microsoft.com/en-us/library/e1f13641.aspx). You can try to change it dynamically Server.ScriptTimeout = 300 (see http://www.beansoftware.com/ASP.NET-Tutorials/Long-Operations.aspx). It this parameter will not helps, then I think you have a problem other as thread recycling from IIS. How you can see default value of this parameter is much less as typical live time of your thread. I think, that your problem has another nature, but to be sure...
Why you set apartment state for the thread? Which COM objects you use in the working thread? Do you have an unmanaged code which do the most of work where you can also insert some code? I think you should have more information about SomethingForALongLongTime to be able to solve the problem.
And one more a little suggestion. Could you insert a line of code after calling SomethingForALongLongTime(); to be sure, that SomethingForALongLongTime not end without an exception?
UPDATED: To be absolutely sure that your thread will be not killed by IIS, you can try to create a process which do SomethingForALongLongTime(); instead of using threads.
When you call RunWorker(), you can add a reference to your thread to a list. Once you have detected that your thread has died, you can inspect the state of the thread, perhaps it will reveal how it died. Or, perhaps it hasn't died, its just waiting on some resource (like the connection to the database).
List runningThreads = ...
public void RunWorker() {
Thread worker = new Thread(delegate()
..
runningThreads.add(worker);
worker.Start();
}
public void checkThreads() {
for (Thread t : runningThreads) {
Console.WriteLine("ThreadState: {0}", t.ThreadState);
}
}
It could be throwing one of the various uncatcheable exceptions including Stack Overflow or Out of Memory. These are the hardest exceptions to track down.
What does memory consumption look like while this thread is running? Can you use a memory profiler on it to see if it's out of control? Can you add some logging in inner loops? If you have a recursive method, add a counter and throw an exception if it recurses an impossible number of times. Are you using large objects that could be causing large object heap fragmentation (causes out of memory errors even when you aren't really out).
You should instrument DoSomethingForALongLongTime() with lots of debug logs, so you can find out at what spot does the code stop executing. Or attach a debugger and break on all first chance exceptions.
use AsyncTasks to achieve your long running work in asp.net
Try use app domain UnhandledException event: http://msdn.microsoft.com/en-us/library/system.appdomain.unhandledexception.aspx
it may give you some information if you miss some exceptions
HI,
I have a .NET application which uses threads excessively. At the time of exit the process does not kill itself. Is there is any tool that can show what is causing the problem? although I have checked thoroughly but was unable to find the problem.
Abdul Khaliq
See: Why doesn't my process terminate even after main thread terminated
That means you have some foreground thread running. A .net application will not terminate unless all the foreground threads complete execution.
You can mark threads as Background Threads and then check (Thread.IsBackground property). Please note that all background threads terminate immediately when application exits. If you are doing some important work in those threads like serializing data to database then you should keep them as foreground threads only. Background threads are good for non critical things like spell cheker etc.
Normally, attaching with a debugger should tell you what threads are still active and what code is running on them.