Develop Reference

Application.DoEvents Never Exits

I'm working on a legacy application which has sprinklings of Application.DoEvents here and there. I'm fully aware that this is frowned upon due to re-entrancy issues, but re-factoring isn't currently an option.
An issue has started to occur where DoEvents never exits. The UI is responsive (I can see UI thread activity in the user logs) so DoEvents seems to be pumping the messages, but for some reason it never completes. Unfortunately this DoEvents is in the main data-processing component, which means this stops processing server responses as it's stuck on the DoEvents line.
We have Stopwatch trace which tells how long the DoEvents ran for - staggeringly I got a log where it said it was running for 188267770 milliseconds, which is 52 hours (gulp). It seemed to get into this state at about 3am on a Saturday, until the user came in on Monday and shut the app down (not killing the process, I can see the GUI thread trace closing things gracefully), at which point the DoEvents completes and the timer data is logged (so something which happens during shutdown must convince DoEvents to complete).
Of course, this only happens on the production user's machines, and not on my dev box :)
Has anyone ever seen a similar problem to this?
I've decompiled DoEvents and also how Conrol.BeginInvoke pushes method delegates onto the GUI thread using the Windows message queue, but I cannot see how DoEvents can get stuck like this, and keep the UI responsive.
Source control diff is also not an option since there's been around 30 versions since the last 'good' version the users had, and this new version with the problem - so about 200 files have changed.
Many thanks
Paul

For the loop to keep running there must be messages on the message queue. So I assume that there is a message that when dispatched then causes another message to be put on the queue. And so forth forever.
Do you have any background processing that causes this type of behaviour? Posting another message to continue processing? Is there an event in the system that can occur that when processed could simply occur immediately again?
The other alternative is that one of the messages is itself creating a nested message loop. For example, showing a dialog would cause a nested message loop that does not finish until the dialog is removed. Does your app try and show a dialog that will then not be dismissed for some reason?
Impossible for us to tell you the answer given the number of possibilities.

After much digging I finally found the cause - System.Windows.Forms.Timer.
Basically two or more Timers can cause a DoEvents() call to never end. Whilst processing one timer WM_TIMER message, another timer can post its WM_TIMER message, which is then processed by DoEvents, as that is being processed the first Timer will post, and so on.
The app I'm working on has about 8 Timers I've found so far....
But really, DoEvents is the real culprit so I plan to re-factor to get rid of it.

C# how should I go about creating this threading application?

Alright I will attempt to explain every aspect of why I need to do this a certain way. Basically, I need an application to execute a certain .exe multiple times asynchronously.
Specs:
I need to be able to restrict the amount of executions going at one time.
It has to use threading because my program has a GUI and simply launching the .exe's and monitoring them will lock up the .GUI AND the console for other things.
How should I go about doing this? (examples help me a lot)

I've already told you multiple times how you should go about this. The launcher program has a single thread. It monitors the child processes. If a process ends and there is a free processor, it starts up a new process and affinitizes the process to that processor. When it's not doing any of those things it yields control back to its UI. Since each of those operations is of short duration, the UI never appears to block.

UPDATE
Actually this wasn't a great answer. As Henk pointed out in my comments, when you call Process.Start() that's not a blocking call. You have to explicitly set Process.EnableRaisingEvents to true, and handle the Exited event. I'm not sure if the Exited event is fired in the calling thread (I doubt it, but you should check), but the point is starting a process isn't a blocking call, so you don't need more threads doing the waiting.
See this similar answer for more details: Async process start and wait for it to finish
PREVIOUS ANSWER
Fire off your threads (limited to your max number of threads), and have them run the external exe using the Process.Start() method. Make sure you set them to wait for the process to finish. When the processes finish, have the threads use something like Interlocked.Increment() to increment a counter variable that you can read from your main form code. Better still, have those threads call a callback delegate (e.g. Action<T>), which will in turn check for this.InvokeRequired before doing the actual work.

Monitoring C# Threads - Which does what/when

As everybody, I am used to debugging my code in VS in step-by-step mode. Well, now that I have an application with many Background Workers everywhere, I am not in Kansas anymore.
What is the most efficient way to debug threaded applications and be able to monitor each and every thread to keep track of what's happening all over the code?
As of now, I stick to good ol' debugging using separate logger instances for each Thread, but this is slowly becoming a nightmare and I'll soon be drowning into my own logs.

Don't try to debug everything all at once. Narrow your focus to a particular behavior in one thread or pair of threads that interact around some mutex lock. If accessing a shared resource is the problem, set breakpoints around use of that resource (which should be in common code, not all over the place).
If you just want to see that thread 3 completed before thread 1, or that thread 2 used up all its work items and is sitting idle, use logs for that.
You can also use the VS Threads view to see what each thread is doing whenever the process is stopped at any breakpoint on any thread. This can give you some insight into what all the threads are doing at any given instant.

A small tip that might ease your pain is to use Visual Studio to freeze threads that you are not interested in. Then when you tell the debugger to continue, the frozen threads will never execute and will not hit breakpoints and confuse you.
Maybe you can use this method to allow only the threads you are debugging to work. E.g. keep one thread that enqueues and one thread that dequeues active, but freeze everything else.
You can freeze/thaw threads from Visual Studio's Threads window, by right-clicking on a thread.

Write it correctly the first time.
Joking aside, the trick to debugging is to break it down into manageable parts. Tackle one worker task at a time, make absolutely sure it does what it's supposed to.
Once you've done that, debugging issues in the main thread is a lot easier, because you can pretty much ignore the background workers and just presume they're yielding correct results when they should be.
The only place left that's harder to debug than a single-threaded application is the interconnect between the threads, which shouldn't be much more difficult if you're using the libraries the way you should be.

I Stumbled upon a detailed MSDN article about Debugging Multithreaded Applications
wich was of great help. Thanx for all the previous answers that guided me towards the right track.

How to know who kills my threads

I got a thread that is just banishing.. i'd like to know who is killing my thread and why.
It occurs to me my thread is being killed by the OS, but i'd like to confirm this and if possible to know why it's killing it.
As for the thread, i can assert it has at least 40 min of execution before dying, but it suddenly dies around 5 min.
public void RunWorker()
{
Thread worker = new Thread(delegate()
{
try
{
DoSomethingForALongLongTime();
}
catch(Exception e)
{
//Nothing is never logged :(
LogException(e);
throw e;
}
});
worker.IsBackground = true;
worker.SetApartmentState(System.Threading.ApartmentState.STA);
worker.Start();
}
EDIT: Addressing answers
Try/Catch Possible exceptions:
It's implemented and it catches nothing :(
Main Thread dying:
This thread is created by the web server, which continues to run
Work completion:
The work is not completed, as it finally affects the database, i can check whether it's done or not when the thread dies.
Having thought of these things brought me to this question, who is killing my threads??
ps. It's not Lady Goldent in the living room with the candle stick :)

Various people (including myself, here) pointed out that hosting a long-running thread in IIS is a bad idea. Your thread will being running inside an IIS 'worker process'. These processes are periodically terminated (recycled) by IIS, which will cause your thread to die.
I suggest that you try turning-off IIS worker process recycling to see if that makes a difference. You can find more information here.

Your thread probably just threw an exception. Try putting a try/catch block around DoSomethingForALongLongTime and see what it picks up.
Update: I didn't notice before that you were starting this from a web server. That can be a very bad idea. In particular, is the separate thread using any information derived from HttpContext.Current? That would include Request, Response, Session, etc., as well as any information from the page.
This is bad because these things only last as long as the request lasts. Once the request is over, they become invalid, to say the very least.
If you need to kick off a long-running thread from within a web application or web service, then you should create a simple Windows Service and host a WCF service within it. Have the web page then send all the information needed to perform the task to the service. The service can even use MSMQ as a transport, which will ensure that no messages are lost, even if the service gets busy.

A potential way to get more information: attach a debugger and break on thread termination. Depending on how your thread is being terminated, this might not work.
Download Debugging Tools for Windows if you don't already have it
Run windbg.exe, attach to your process
Break into windbg, type sxe et to enable breaking on thread exit
When the debugger breaks, inspect the state of the system, other threads, etc.
To get the managed stack, load sos.dll (.loadby sos mscorsvr, .loadby sos mscorwks, or .loadby sos clr should work), then run !clrstack (see !help for other sos commands)
If you get a lot of noise from other threads exiting, script windbg to continue after breaking if it's not the thread ID you care about.
Edit: If you think the thread is being terminated from within your process, you can also set a breakpoint on TerminateThread (bp kernel32!TerminateThread) and ExitThread (bp kernel32!ExitThread) to catch the stack of the killer.

I don't know the answer, but some thoughts:
Could it be throwing an exception? Have you tried putting a try/catch around the DoSomethingForALongLongTime() call?
Are there any points where it exits normally? Try putting some logging on them.
Do you get the same behaviour in and out of the debugger? Does the output window in the debugger provide any hints?
UPDATE
You said:
This thread is created by the web
server, which continues to run
If the thread is running inside asp.net then it may be that the thread is being killed when the asp.net worker process recycles, which it will do periodically. You could try turning off worker process recycling and see if that makes any difference.

Your edit reveals the answer:
It's the butler web server.
How exactly do you host these threads? A webserver environment isn't exactly designed to host long living processes. In fact, it is probably configured to halt runaway sites, every 40 minutes maybe?
Edit:
For a quick fix, your best chance is to set worker.IsBackground = false; because your current setting of true allows the system to kill the parent-thread w/o waiting for your bgw.
On another note, there is little point in using a BackgroundWorker in an ASP.NET application, it is intended for WinForms and WPF. It would be better to create a separate thread for this, since you are changing some of the Threads properties. That is not advised for a ThreadPool (Bgw) thread.

The process might be terminating. That would be what worker.IsBackground = true; is designed to do, kill your thread when the main thread exits.

A background thread will only run as long there are foreground threads runnnig.
As soon that all foreground threads end, any background thread still running will aborted.

If checking for an exception doesn't show anything useful, get your thread code to write to a log file at key points. You'll then be able to see exactly when it stops working and hopefully why.

A simple answer would be: "The killer doesn't leave a name card" ;)
If your thread is hosted in IIS, probably the thread is killed by the app pool process which recycles. The server might continue running but the process which hosts your item is stopped untill a new request fires everything up again.
If your thread is hosted in an executable, the only way it can be killed is by killing the thread yourself, throwing an exception in the thread or terminating the host process
Hope this helps.

You can try to increase executionTimeout value of configuration\system.web\httpRuntime in web.config (default value is 110 seconds in .NET 4.0 and 90 in corresponds to http://msdn.microsoft.com/en-us/library/e1f13641.aspx). You can try to change it dynamically Server.ScriptTimeout = 300 (see http://www.beansoftware.com/ASP.NET-Tutorials/Long-Operations.aspx). It this parameter will not helps, then I think you have a problem other as thread recycling from IIS. How you can see default value of this parameter is much less as typical live time of your thread. I think, that your problem has another nature, but to be sure...
Why you set apartment state for the thread? Which COM objects you use in the working thread? Do you have an unmanaged code which do the most of work where you can also insert some code? I think you should have more information about SomethingForALongLongTime to be able to solve the problem.
And one more a little suggestion. Could you insert a line of code after calling SomethingForALongLongTime(); to be sure, that SomethingForALongLongTime not end without an exception?
UPDATED: To be absolutely sure that your thread will be not killed by IIS, you can try to create a process which do SomethingForALongLongTime(); instead of using threads.

When you call RunWorker(), you can add a reference to your thread to a list. Once you have detected that your thread has died, you can inspect the state of the thread, perhaps it will reveal how it died. Or, perhaps it hasn't died, its just waiting on some resource (like the connection to the database).
List runningThreads = ...
public void RunWorker() {
Thread worker = new Thread(delegate()
..
runningThreads.add(worker);
worker.Start();
}
public void checkThreads() {
for (Thread t : runningThreads) {
Console.WriteLine("ThreadState: {0}", t.ThreadState);
}
}

It could be throwing one of the various uncatcheable exceptions including Stack Overflow or Out of Memory. These are the hardest exceptions to track down.
What does memory consumption look like while this thread is running? Can you use a memory profiler on it to see if it's out of control? Can you add some logging in inner loops? If you have a recursive method, add a counter and throw an exception if it recurses an impossible number of times. Are you using large objects that could be causing large object heap fragmentation (causes out of memory errors even when you aren't really out).

You should instrument DoSomethingForALongLongTime() with lots of debug logs, so you can find out at what spot does the code stop executing. Or attach a debugger and break on all first chance exceptions.

use AsyncTasks to achieve your long running work in asp.net

Try use app domain UnhandledException event: http://msdn.microsoft.com/en-us/library/system.appdomain.unhandledexception.aspx
it may give you some information if you miss some exceptions

How to detect when main thread terminates?

What I need to know:
I would like to detect when a the main thread (process?) terminates so that I can ensure certain actions are performed before it is terminated.
What I have found myself:
I found the events AppDomain.DomainUnload and AppDomain.ProcessExit. AppDomain.DomainUnload seems to work with non-applications like MbUnit. AppDomain.ProcessExit seems to work with applications but there is a 3 second time limit which I really don't like. Is there more ways to detect when an AppDomain / process terminates?
Background:
I am looking for such an event to ensure my log is persistet to file when the application terminates. The actual logging runs on another thread using a producer-consumer pattern where it is very likely that log entries might queue up in memory and I need to ensure this queue is saved to file when the application terminates.
Is there anything else I should be aware of?
Update:
Changed the above to reflect what I have found out myself. I am not happy with the 3 second time limit during ProcessExit. The MSDN documentation does say though that it can be extended:
The total execution time of all
ProcessExit event handlers is limited,
just as the total execution time of
all finalizers is limited at process
shutdown. The default is three
seconds, which can be overridden by an
unmanaged host.
Does anyone know how to override the default?
More ideas are also highly appreciated!
Follow up:
I have posted a follow up question to this.

You should have an entry point for your application. Normally you can do there some logging when all tasks are terminated:
static void Main()
{
try
{
Application.Run( .... );
}
finally
{
// logging ...
}
}

What exactly do you want to find out?
When the process terminates? (Just because the AppDomain is unloaded doesn't necessarily mean that the entire process is terminating)
When the main thread terminates (If there are other non-background threads, the main thread can terminate without the process terminating (or AppDomain unloading)
So they're not quite the same thing.
Anyway, it is generally dangerous to have log messages buffered in memory at all. What happens if someone turns off the power? Or if I terminate your process through Task Manager? All your log messages are gone. So often, you'll want unbuffered writes in your log, to get messages pushed to disk immediately.
Anyway, another (more robust) approach might be to run the logger itself in a non-background thread. That way, even if the rest of the application terminates, the logger won't, so the process is kept alive. Then you just have to set some flag when the rest of the app terminates, to let the logger know that it too should close once it has written out all pending log messages.
It still won't save you from the case where the system loses power or someone forcibly termianates the process on the OS-level, but it will handle all cases where the application closes normally, and gives you unlimited time to perform clean-up actions (since the process isn't actually terminating yet, it's still got one live thread)

ie. guaranteed to be called and have unlimited time to finish?
Unfortunately, NO option is going to have unlimited time, and be guaranteed. There is no way to enforce this, as many things can happen. Somebody tripping over the power cord or a forced termination of your program will prevent any option from giving you adequate time to handle things.
In general, putting your logic at the end of the Main routine is probably the most reasonable option, since that gives you complete freedom in handling your termination events. You have no time constraints there, and can have the processing take as much time as needed.
There are no guarantees that this will run, though, since a forceful termination of your program may bypass this entirely.

Based on the documentation, it looks like the default application domain (the one your Main method is probably running in) will not receive the DomainUnload event.
I don't know a built-in event that would do what you expect.
You could define your own custom event, have interested parties register with it, and fire off the event just before you return from Main().

I don't know how old this thread is, but I've had a similar problem whcih was a little tough for me to solve.
I had a WinForms application that was not firing any of the above forementioned events when a user logged out. Wraaping the Application.Run() in a try finally didn't work either.
Now to get around this you would have to using PInvoke into Win32 API's to achieve this. Well you did prior to .NET 2.0 anyways. Luckly MS introduced a new class called SystemEvents. With this class you can catch a SessionEnd event. This event allows you to cleanup when the OS want to terminate your app. There is no .NET time limit o this event it appears, although the OS will eventually kill your app if you take too long. This is a little more than 3 seconds, although 3 seconds should be plenty of time to cleanup.
Secondly my other problem was I wanted my worker thread to terminate the main thread once it was finished its work. With an Application.Run() this was hard to achieve. What I ended up doing was calling Application.Run() with a shared Application context. The thread is then able to call ApplicationContext.ThreadExit() to force the Application.Run to return. This seems to work quite nicely.
Hope this helps someone.
Regards
NozFX