When closing an application, what is the best way to close foreground threads at the same time?
From all my research, it seems to be that the best way is to set IsBackground to true..
Is this correct or is there a better way? I'm skeptical as it doesn't seem to follow Microsofts normally descriptive method name pattern.
It will certainly work, the CLR will abort a thread like that when your main startup exits. Pretty similar to using Thread.Abort(), minus the drastic failure modes you'd normally suffer from using Abort(). It is a rude abort, there's nothing that the thread itself can do to stop it. And there will be no more code that runs afterwards that could fail due to the typical problems you'd get from Abort(), like a deadlock. Other than finalizers.
Calling it "correct" is however a bit of a stretch, there's also nothing that the thread can do to terminate cleanly. Which might be detrimental if it has externally observable behavior. Like updating a dbase, talking over a socket or writing a file. That's rudely interrupted as well, potentially leaving a confused server or a half-written file that can cause trouble later. A mitigating circumstance is that this will also happen when your program dies on an unhandled exception, you'd expect it to not cause trouble either. It depends :)
Related
I have a somewhat unusual scenario where I need to be able to outright slaughter "hung", self-hosted WorkflowInstance's after a given timeout threshold. I tried the Abort(), Terminate() and Cancel() methods but these are all too "nice". They all appear to require a response from the WorkflowInstance before they are honored.
In my scenario, a workflow entered an infinite loop and was therefore unresponsive. Calls to the normal methods mentioned above would simply hang since the workflow was completely unresponsive. I was surprised to learn the WorkflowRuntime does not appear have a mechanism for dealing with this scenario, or that Abort() and Terminate() are merely suggestions as opposed to violent directives.
I scoured google/msdn/stackoverflow/etc trying to find out what to do when Terminate() simply isn't going to get the job done and came up dry. I considered creating my own base activity and giving it a timeout value so my "root" activity can kill itself if one of its child activities hangs. This approach seems like I'd be swatting at flies with a sledge hammer...
Is there a technique I overlooked?
The only true solution is to consider this a bug, fix whatever went wrong, and consider the matter closed.
The only way to forcibly abort any code that is locked in an infinite loop is to call Abort() on the thread. Of course, this is considered bad juju, and should only be done when the state of the application can be ensured after the call.
So, you must supply the WorkflowApplication an implementation of SynchronizationContext that you write which can call Abort() on the thread that the workflow Post()s to.
I am not sure if this will work, but have you tried the WorkflowInstance.TryUnload() function? I remember this to fire off a few events inside of the workflow (been a while since I did this), so you might be able to have an event handler in your workflow that catches this and does a kill switch on itself.
I'm working on a ASP.NET website that on some requests will run a very lengthy caching process. I'm wondering what happens exactly if the execution timeout is reached while it is still running in terms of how the code handles it.
Particularly I am wondering about things like if the code is in the try of a try/finally block will the finally still be run?
Also given I am not sure I want the caching to terminate even if it goes on that long is there a way with spawning new threads, etc. that I can circumvent this execution timeout? I am thinking it would be much nicer to return to the user immediately and say "a cache build is happening" rather than just letting them time out. I have recently started playing with some locking code to make sure only one cache build happens at a time but am thinking about extending this to make it run out of sync.
I've not really played with creating threads and such like myself so am not sure exactly how they work, particularly in terms of interacting with ASP.NET. eg if the parent thread that launched it is terminated will that have any effect on the spawned thread?
I know there is kind of a lot of different questions in here and I can split them if that is deemed best but they all seem to go together... I'll try to summarise the questions though:
Will a finally block still be executed if a thread is terminated by ASP.NET while in the try block
Would newly created threads be subject to the same timeouts as the original thread?
Would newly created threads die at the same time as the parent thread that created them?
And the general one of what is the best way to do long running background processes on an ASP.NET site?
Sorry for some noobish questions, I've never really played with threads and they still intimidate me a bit (my brain says they are hard). I could probably test the answer to a lot of tehse questions but I wouldn't be confident enough of my tests. :)
Edit to add:
In response to Capital G:
The problem I have is that the ASp.NET execution timeout is currently set to one hour which is not always long enough for some of these processes I reckon. I've put some stuff in with locks to prevent more than one person setting off these long processes and I was worried the locks might not be released (which if finally blocks aren't always run might happen I guess).
Your comments on not running long processes in ASP.NET is why I was thinking of moving them to other threads rather than blocking the request thread but I don't know if that still counts as running within the ASP.NET architecture that you said was bad.
The code is not actually mine so I'm not allowed (and not sure I 100% understand it enough) to rework it into a service though that is certainly where it would best live.
Would using a BackgroundWorker process for something that could take an hour be feasible in this situation (with respect to comments on long running processes in ASP.NET). I would then make request return a "Cache is building" page until its finished and then go back to serving normally... Its all a bit of a nightmare but its my job so I've got to find a way to improve it. :)
Interesting question, just tested and no it's not guaranteed to execute the code in the finally block, if a thread is aborted it could stop at any point in the processing. You can design some sanity checking and other mechanisms to handle special cleanup routines and such but it has a lot to do with your thread handling as well.
Not necessarily, it depends on how your implementing your threads. If you are working with threads yourself, then you can easily get into situations where the parent thread is killed while it's child threads are still out there processing, you generally want to do some cleanup in the parent thread that ends the child threads as well. Some objects might do a lot of this for you as well, so it's a tough call to say one way or the other. Never assume this at the very least.
No, not necessarily, don't assume this at least, again has to do with your design and whether your doing threading yourself or using some higher level threading object/pattern. I would never assume this regardless.
I don't recommend long running processes within the ASP.NET architecture, unless its within the typical timeout, if it's 10-20s okay but if it's minutes, no, the reason is resource usage within ASP.NET and it's awfully bad on a user. That being said you could perform asynchronous operations where you hand off the work to the server, then you return back to the user when the processing is finished, (this is great for those 10-20s+ processes), the user can be given a little animation or otherwise not have their browser all stuck for that long waiting for whatever is happening on the server to happen.
If it is a long running process, things that take greater than 30-60s+, unless it absolutely has to be done in ASP.NET due to the nature of the process, I suggest moving it to a windows service and schedule it in some way to occur when required.
Note: Threading CAN be complicated, it's not that it's hard so much as that you have to be very aware of what your doing, which requires a firm understanding of what threads are and how they work, I'm no expert, but I'm also not completely new and I'll tell you that in most situations you don't need to get into the realm of threading, even when it seems like you do, if you must however, I would suggest looking into the BackgroundWorker object as they are simplified for the purposes of doing batched processing etc. (honestly for many situations that DO need threads, this is usually a very simple solution).
http://msdn.microsoft.com/en-us/library/system.componentmodel.backgroundworker.aspx
Long or time consuming processes to be started behind the web-page; it should not hit the ASP.NET execution time out; the user page should be freed; running the requests under lock etc. All these situation points towards using async services. In one of the products, where I architected, used services for such scenarios. The service exposes some async method to initiate. The status of the progress can be queried using another method. Every request is given some id and no duplicate requests are fired ever. The progress proceeds even if the user logs out. The user can see the results at a later time.
If you have looked at such options already, let me know if there is any issue. Or if you are yet to look in this direction, please get it this way. For any help, just send in your comments.
I was hoping to get some good ideas as to what might be causing a really nasty bug.
This is a program which is transmitting data over a socket, and also receives messages back.
I could explain lots more, but I don't think this will help here.
I'm just searching for hypothetical problems which can cause the following behaviour:
program runs
processor time slowly accumulates (till around 60%)
all of a sudden (could be after 30 but also after 60 seconds) the processor time shoots to 100%. the program halts completely
In my syslog it always ends on one line with a memory allocation (something similar to: myArray = new byte[16384]) in the same thread.
now here is the weird part: if I set the debugger anywhere...it immediately stops on that line. So just the act of setting a breakpoint, made the thread continue (it wasn't running since I saw no log output anymore)
I was thinking 'deadlock' but that would not cause 100% processor power. If anything, the opposite. Also, setting a breakpoint would not cause a deadlock to end.
anyone else a theoretical suggestion as to what kind of 'construct' might cause this effect?
(apart from 'bad programming') ;^)
thanks
EDIT:
I just noticed.... by setting the sendspeed slower, the problem shows itself much later than expected. I would think around the same amount of packets send...but no the amount of packets send is much higher this way before it has the same problem.
I can only guess, but the opposite of a deadlock would be a livelock. This means two threads who react to each other in an infinite loop. This could also be possibly interrupted by setting a break point, as livelocks generally depend on the right timing.
Other than this I had once a similar issue with the Java nio classes which are non-blocking which caused the main thread to busy wait for input. Although the CPU usage rose instantaneously, not just after a few seconds.
Maybe if you can provide a bit more information like the programming language or even a code sample there might be more ideas.
Anything that involves repetitive processing (looping, recursion, etc) can cause this.
What's interesting is that if the program is doing anything that normally slows down performance (such as disk IO or network access), then the processor is less likely to peg . The processor pegs at 100% only if the program is using the processor. If you have to wait for disk or network IO, then the processor thread has to wait.
So in the code, I'd check for loops where a lot of work is going on, but little IO.
Also, if you're debugging in Visual Studio, you can hit the pause button to stop the app at the current point and see what your code is doing when it locks.
I'm guessing an infinite loop in the socket receiving end. It keeps trying to allocate a buffer to receive the data that is coming in, but the buffer is never big enough so it keeps allocating. But it is really hard to say without code. I'd advise you to add more logging and/or single step the code if you don't want to share it.
Without seeing code, I only can say your program is probably infinite looping and the call that should block is not blocking correctly as you're expecting
You can also try profiling (EQUATEC free profiler, for example). If will show you how much of your processor time was spent in each method.
I found the answer... quite silly actually (it always is). The thread which is sending/receiving messages is doing this via asynchronous methods. However, the asynchronous callbacks never seem to be able to come through while the thread is also pumping messages in the sendqueue. I notice when I put a thread.sleep every second, all asynchronous callbacks are pumped through. So the solution it turns out is to have a separate thread for sending/receiving, done purely on async, and another one for filling the sendqueue.
why this would have resulted in 100% processor power is beyond me. But it does actually explain why setting a breakpoint allowed the async callbacks to catch up.
Because the program fails while allocating memory I would guess that the incoming message rate is too high for it to handle.
I imagine that your program has some thread that it's only job is to listen to the socket and send the incoming messages to some other threads to handle (maybe you have some thread pool there). Imagine a situation where the incoming message rate is too high so all the worker threads are busy handling previous messages and the thread that listen to the socket have to put the new messages into some kind of queue until one of the worker threads will be free to handle them. this queue will grow and grow until you won't have additional memory. so that could be the reason for your program's termination.
now, about the 100% CPU. I guess that the thread the uses the CPU must be one of the worker threads. this will explain why the listening thread is queuing the messages. the reason can be a corrupted message or something else that causes it to run into an infinite loop. "frenetisch applaudierend" suggested in his answer that two or more of the worker threads can cause "livelock" on each other which could also be the reason for your problem.
What I need to know:
I would like to detect when a the main thread (process?) terminates so that I can ensure certain actions are performed before it is terminated.
What I have found myself:
I found the events AppDomain.DomainUnload and AppDomain.ProcessExit. AppDomain.DomainUnload seems to work with non-applications like MbUnit. AppDomain.ProcessExit seems to work with applications but there is a 3 second time limit which I really don't like. Is there more ways to detect when an AppDomain / process terminates?
Background:
I am looking for such an event to ensure my log is persistet to file when the application terminates. The actual logging runs on another thread using a producer-consumer pattern where it is very likely that log entries might queue up in memory and I need to ensure this queue is saved to file when the application terminates.
Is there anything else I should be aware of?
Update:
Changed the above to reflect what I have found out myself. I am not happy with the 3 second time limit during ProcessExit. The MSDN documentation does say though that it can be extended:
The total execution time of all
ProcessExit event handlers is limited,
just as the total execution time of
all finalizers is limited at process
shutdown. The default is three
seconds, which can be overridden by an
unmanaged host.
Does anyone know how to override the default?
More ideas are also highly appreciated!
Follow up:
I have posted a follow up question to this.
You should have an entry point for your application. Normally you can do there some logging when all tasks are terminated:
static void Main()
{
try
{
Application.Run( .... );
}
finally
{
// logging ...
}
}
What exactly do you want to find out?
When the process terminates? (Just because the AppDomain is unloaded doesn't necessarily mean that the entire process is terminating)
When the main thread terminates (If there are other non-background threads, the main thread can terminate without the process terminating (or AppDomain unloading)
So they're not quite the same thing.
Anyway, it is generally dangerous to have log messages buffered in memory at all. What happens if someone turns off the power? Or if I terminate your process through Task Manager? All your log messages are gone. So often, you'll want unbuffered writes in your log, to get messages pushed to disk immediately.
Anyway, another (more robust) approach might be to run the logger itself in a non-background thread. That way, even if the rest of the application terminates, the logger won't, so the process is kept alive. Then you just have to set some flag when the rest of the app terminates, to let the logger know that it too should close once it has written out all pending log messages.
It still won't save you from the case where the system loses power or someone forcibly termianates the process on the OS-level, but it will handle all cases where the application closes normally, and gives you unlimited time to perform clean-up actions (since the process isn't actually terminating yet, it's still got one live thread)
ie. guaranteed to be called and have unlimited time to finish?
Unfortunately, NO option is going to have unlimited time, and be guaranteed. There is no way to enforce this, as many things can happen. Somebody tripping over the power cord or a forced termination of your program will prevent any option from giving you adequate time to handle things.
In general, putting your logic at the end of the Main routine is probably the most reasonable option, since that gives you complete freedom in handling your termination events. You have no time constraints there, and can have the processing take as much time as needed.
There are no guarantees that this will run, though, since a forceful termination of your program may bypass this entirely.
Based on the documentation, it looks like the default application domain (the one your Main method is probably running in) will not receive the DomainUnload event.
I don't know a built-in event that would do what you expect.
You could define your own custom event, have interested parties register with it, and fire off the event just before you return from Main().
I don't know how old this thread is, but I've had a similar problem whcih was a little tough for me to solve.
I had a WinForms application that was not firing any of the above forementioned events when a user logged out. Wraaping the Application.Run() in a try finally didn't work either.
Now to get around this you would have to using PInvoke into Win32 API's to achieve this. Well you did prior to .NET 2.0 anyways. Luckly MS introduced a new class called SystemEvents. With this class you can catch a SessionEnd event. This event allows you to cleanup when the OS want to terminate your app. There is no .NET time limit o this event it appears, although the OS will eventually kill your app if you take too long. This is a little more than 3 seconds, although 3 seconds should be plenty of time to cleanup.
Secondly my other problem was I wanted my worker thread to terminate the main thread once it was finished its work. With an Application.Run() this was hard to achieve. What I ended up doing was calling Application.Run() with a shared Application context. The thread is then able to call ApplicationContext.ThreadExit() to force the Application.Run to return. This seems to work quite nicely.
Hope this helps someone.
Regards
NozFX
I have an app that starts 10 threads.
Each thread does it's work and waits (using the producer consumer model).
So when more work comes along, one of the threads is unblocked and it does the work.
A colleague at work insists I should write some code that monitors the threads "just in case" they freeze/do not respond or die.
So far in my testing, they work fine and close down correctly.
My question is "should I put code in to do this"? If so, "how do I monitor a thread and check it's status"?
Thanks.
JD
It really depends on the situation. First of all, you should focus on correctness so that it doesn't freeze or die but if you need more reliability, you should first think about how you can gracefully recover from such a situation. You should think about why it might freeze or die and if it did, now what you can do. If you can't do anything good that reliably recovers from such a situation, you shouldn't even try. If can do it without making the situation worse, then you can go and try doing so.
Obviously, if you made such a choice, you have to be careful not to mess things up and introduce some bugs that actually make the bad stuff happen yourself.
Ideally no, your threads should be able to finish properly. Monitoring them is not worth the complexity and processing time. If you do it correctly you won't need monitoring.
Assuming you're coding against the .NET framework 2 or above, an unhandled exception in a worker thread will kill the entire app.
Unless you're absolutely sure your thread code won't throw an exception (or reserve memory, or grab a handle, or call framework code you didn't write...) the first thing you'll want to write in your thread entry method is a try/catch.
If your threads spend a lot of time waiting for work, you could consider refactoring your app to use the System.Threading.ThreadPool class. Doing that would offload much of your thread management to the framework.
Threads don't just hang or die unless there is a bug in the user code. The chances of the OS/Framework screwing up is so negligible that it's, for practical purposes, impossible. It's like worry about the hard disk not writing a file even though File.WriteAllBytes() succeeds. It just doesn't happen.
That said, wildcard brings up a good point that an unhandled exception in a worker thread populates up as an unhandled exception through the entire AppDomain.
Best way to do is use "ThreadPool" and call ThreadPool.QueueUserWorkItem and .Net manages the ThreadPool very efficiently.
You can set maximum number of threads to allocate and you dont need to do any wait operation etc. You can also get WaitHandle to wait upon completion. And you dont need to manage the blocking/unblocking of threads.
You should always code for the chance that the thread may hang or die, even if all you do is forcibly restart the thread.