Does anyone know why sometimes a Directory.Move() operation in C# hangs/waits instead of throwing an exception immediately?
For example:
If I use the Directory.Move() method inside a try block, then navigate to that folder in File Explorer, Windows creates some handles locking it.
Then, I expect the catch block to be executed immediately, but instead it's like the application just hangs for 10-15 seconds before it throws an exception.
The funny thing is, that if I go outside of the folder in File Explorer during these 10-15 seconds, then the application actually completes the Move() operation.
It's like: instead of throwing an exception immediately, Windows hangs for 10-15 seconds to see if the people who are responsible for the handles (locks) closes these handles by themselves.
Is there a way to make the application throw the exception immediately?
The answer to your question "Does anyone know why sometimes a move operation in C# hangs/waits instead of throwing an exception immediately?" is probably that the .net framework is issued with a Pending state from its request for an NTFS lock, and eventually gives up.
The System.IO.Directory.Move maps directly to a Kernel32 function; I guess that eventually this finds itself calling LockFileEx (https://msdn.microsoft.com/en-us/library/windows/desktop/aa365203(v=vs.85).aspx) which allows the caller to specify whether to immediately fail if the lock cannot be obtained, or wait for a specified time. I guess that Kernel32 uses the variant of this which allows for the setting of a timeout. It doesn't seem that the .net framework has any influence on what timeout it used.
I'll start by saying I do not have a lot of experience in troubleshooting multi-threading problems. So a lot of what I've read about debugging race conditions, dead locks, live locks, etc. are strictly theoretical to me.
I have this .NET application that is making use of a dynamically loaded native win32 dll. If the dll is never loaded the application terminates without a problem. However, if the dll is loaded then when the user exits the application the UI disappears but the process never terminates.
I've turned on native code debugging in the project settings so I can see all the threads that are running. When the user closes the application window the main thread appears to terminate. I know this because if I perform a Break All in the Threads windows in Visual Studio the main thread is re-categorized as a worker thread and there is no call stack available for it. There are 20 other threads still active, all with call stacks. I've looked through the call stacks for all of these threads and nothing sticks out at me (there's no mention of the dll in any of the call stacks).
What steps can I take to narrow down the cause of this problem? Are there any additional tools I should be using to help pin point the problem?
This means that some of your Foreground Threads are still alive. Unlike Background Threads, Foreground Threads keeps the process alive.
You should either use Background Threads or Stop your Foregrounds Threads to be able to exit the process gracefully.
Windows application will automatically exit when all of its thread(s) are stopped.
As you said
If the dll is never loaded the application terminates without a
problem
I'm assuming all the running threads are unmanaged threads(not created by clr). There is no concept of background threads in native code. All threads must be terminated to terminate the process.
You must find a way to signal all threads to exit in that library. See if you have any api exposed for that.
If you don't find anything, you've got a "Sledge Hammer" approach. i.e Terminate the process with Environment.Exit which works almost always. Be sure you use this approach as a last resort as this can corrupt the process state.
I'm working on about 35 batches updating many databases as a part of our daily process at work. Every batch of them was developed in a single web app. Due to database issues, i have collected all of them in one windows application to make use of DB connection pooling and i have assigned a single backgroundworker for each batch. Reaching to 20 batch in the application, every thing is working good. But when i add any other backgroundworker for any other batch, the application hangs.I think this is because i'm running too many threads in one process. Is there a solution for this problem, for example, making the application working with many processes ??!!!.
Regards,
Note,
I have assigned a single machine for this application (Core i7 cpu, 8 gb ram).
How many databases you have to update?
I think it is more recommended to have the number of Threads as the number of Databases
If your UI is freezing while many background workers are active, but recovers when those background workers are finished processing, then likely the UI thread is executing a method which waits for a result or signal from one of the background worker threads.
To fix your problem, you will have to look for UI-related code that deals with synchronization / multi-threading. This might be places where one of the many synchronization objects of .NET are being used (including the lock statement), but it could also involve "dumb" polling loops a-ka while(!worker.IsFinished) Thread.Sleep();.
Another possible reason for the freeze might be that you are running a worker (or worker-related method) accidentally in the UI thread instead in a background thread.
But you will find out when you use the debugger.
To keep the scope of your hunt for problematic methods managable, let your program run in the debugger until the UI freezes. At that moment, pause the program execution in the debugger. Look which code the UI thread is processing then, and you will have found one instance of offending code. (Whatever there is wrong, i can't tell you - because i don't know your code.)
It is quite possible that different UI-related methods in your code will suffer from the same issue. So, if you found the offending code (and were able to fix it) you would want to check on for other problematic methods, but that should be rather easy since at that point of time you will know what to look for...
I got a thread that is just banishing.. i'd like to know who is killing my thread and why.
It occurs to me my thread is being killed by the OS, but i'd like to confirm this and if possible to know why it's killing it.
As for the thread, i can assert it has at least 40 min of execution before dying, but it suddenly dies around 5 min.
public void RunWorker()
{
Thread worker = new Thread(delegate()
{
try
{
DoSomethingForALongLongTime();
}
catch(Exception e)
{
//Nothing is never logged :(
LogException(e);
throw e;
}
});
worker.IsBackground = true;
worker.SetApartmentState(System.Threading.ApartmentState.STA);
worker.Start();
}
EDIT: Addressing answers
Try/Catch Possible exceptions:
It's implemented and it catches nothing :(
Main Thread dying:
This thread is created by the web server, which continues to run
Work completion:
The work is not completed, as it finally affects the database, i can check whether it's done or not when the thread dies.
Having thought of these things brought me to this question, who is killing my threads??
ps. It's not Lady Goldent in the living room with the candle stick :)
Various people (including myself, here) pointed out that hosting a long-running thread in IIS is a bad idea. Your thread will being running inside an IIS 'worker process'. These processes are periodically terminated (recycled) by IIS, which will cause your thread to die.
I suggest that you try turning-off IIS worker process recycling to see if that makes a difference. You can find more information here.
Your thread probably just threw an exception. Try putting a try/catch block around DoSomethingForALongLongTime and see what it picks up.
Update: I didn't notice before that you were starting this from a web server. That can be a very bad idea. In particular, is the separate thread using any information derived from HttpContext.Current? That would include Request, Response, Session, etc., as well as any information from the page.
This is bad because these things only last as long as the request lasts. Once the request is over, they become invalid, to say the very least.
If you need to kick off a long-running thread from within a web application or web service, then you should create a simple Windows Service and host a WCF service within it. Have the web page then send all the information needed to perform the task to the service. The service can even use MSMQ as a transport, which will ensure that no messages are lost, even if the service gets busy.
A potential way to get more information: attach a debugger and break on thread termination. Depending on how your thread is being terminated, this might not work.
Download Debugging Tools for Windows if you don't already have it
Run windbg.exe, attach to your process
Break into windbg, type sxe et to enable breaking on thread exit
When the debugger breaks, inspect the state of the system, other threads, etc.
To get the managed stack, load sos.dll (.loadby sos mscorsvr, .loadby sos mscorwks, or .loadby sos clr should work), then run !clrstack (see !help for other sos commands)
If you get a lot of noise from other threads exiting, script windbg to continue after breaking if it's not the thread ID you care about.
Edit: If you think the thread is being terminated from within your process, you can also set a breakpoint on TerminateThread (bp kernel32!TerminateThread) and ExitThread (bp kernel32!ExitThread) to catch the stack of the killer.
I don't know the answer, but some thoughts:
Could it be throwing an exception? Have you tried putting a try/catch around the DoSomethingForALongLongTime() call?
Are there any points where it exits normally? Try putting some logging on them.
Do you get the same behaviour in and out of the debugger? Does the output window in the debugger provide any hints?
UPDATE
You said:
This thread is created by the web
server, which continues to run
If the thread is running inside asp.net then it may be that the thread is being killed when the asp.net worker process recycles, which it will do periodically. You could try turning off worker process recycling and see if that makes any difference.
Your edit reveals the answer:
It's the butler web server.
How exactly do you host these threads? A webserver environment isn't exactly designed to host long living processes. In fact, it is probably configured to halt runaway sites, every 40 minutes maybe?
Edit:
For a quick fix, your best chance is to set worker.IsBackground = false; because your current setting of true allows the system to kill the parent-thread w/o waiting for your bgw.
On another note, there is little point in using a BackgroundWorker in an ASP.NET application, it is intended for WinForms and WPF. It would be better to create a separate thread for this, since you are changing some of the Threads properties. That is not advised for a ThreadPool (Bgw) thread.
The process might be terminating. That would be what worker.IsBackground = true; is designed to do, kill your thread when the main thread exits.
A background thread will only run as long there are foreground threads runnnig.
As soon that all foreground threads end, any background thread still running will aborted.
If checking for an exception doesn't show anything useful, get your thread code to write to a log file at key points. You'll then be able to see exactly when it stops working and hopefully why.
A simple answer would be: "The killer doesn't leave a name card" ;)
If your thread is hosted in IIS, probably the thread is killed by the app pool process which recycles. The server might continue running but the process which hosts your item is stopped untill a new request fires everything up again.
If your thread is hosted in an executable, the only way it can be killed is by killing the thread yourself, throwing an exception in the thread or terminating the host process
Hope this helps.
You can try to increase executionTimeout value of configuration\system.web\httpRuntime in web.config (default value is 110 seconds in .NET 4.0 and 90 in corresponds to http://msdn.microsoft.com/en-us/library/e1f13641.aspx). You can try to change it dynamically Server.ScriptTimeout = 300 (see http://www.beansoftware.com/ASP.NET-Tutorials/Long-Operations.aspx). It this parameter will not helps, then I think you have a problem other as thread recycling from IIS. How you can see default value of this parameter is much less as typical live time of your thread. I think, that your problem has another nature, but to be sure...
Why you set apartment state for the thread? Which COM objects you use in the working thread? Do you have an unmanaged code which do the most of work where you can also insert some code? I think you should have more information about SomethingForALongLongTime to be able to solve the problem.
And one more a little suggestion. Could you insert a line of code after calling SomethingForALongLongTime(); to be sure, that SomethingForALongLongTime not end without an exception?
UPDATED: To be absolutely sure that your thread will be not killed by IIS, you can try to create a process which do SomethingForALongLongTime(); instead of using threads.
When you call RunWorker(), you can add a reference to your thread to a list. Once you have detected that your thread has died, you can inspect the state of the thread, perhaps it will reveal how it died. Or, perhaps it hasn't died, its just waiting on some resource (like the connection to the database).
List runningThreads = ...
public void RunWorker() {
Thread worker = new Thread(delegate()
..
runningThreads.add(worker);
worker.Start();
}
public void checkThreads() {
for (Thread t : runningThreads) {
Console.WriteLine("ThreadState: {0}", t.ThreadState);
}
}
It could be throwing one of the various uncatcheable exceptions including Stack Overflow or Out of Memory. These are the hardest exceptions to track down.
What does memory consumption look like while this thread is running? Can you use a memory profiler on it to see if it's out of control? Can you add some logging in inner loops? If you have a recursive method, add a counter and throw an exception if it recurses an impossible number of times. Are you using large objects that could be causing large object heap fragmentation (causes out of memory errors even when you aren't really out).
You should instrument DoSomethingForALongLongTime() with lots of debug logs, so you can find out at what spot does the code stop executing. Or attach a debugger and break on all first chance exceptions.
use AsyncTasks to achieve your long running work in asp.net
Try use app domain UnhandledException event: http://msdn.microsoft.com/en-us/library/system.appdomain.unhandledexception.aspx
it may give you some information if you miss some exceptions
What I need to know:
I would like to detect when a the main thread (process?) terminates so that I can ensure certain actions are performed before it is terminated.
What I have found myself:
I found the events AppDomain.DomainUnload and AppDomain.ProcessExit. AppDomain.DomainUnload seems to work with non-applications like MbUnit. AppDomain.ProcessExit seems to work with applications but there is a 3 second time limit which I really don't like. Is there more ways to detect when an AppDomain / process terminates?
Background:
I am looking for such an event to ensure my log is persistet to file when the application terminates. The actual logging runs on another thread using a producer-consumer pattern where it is very likely that log entries might queue up in memory and I need to ensure this queue is saved to file when the application terminates.
Is there anything else I should be aware of?
Update:
Changed the above to reflect what I have found out myself. I am not happy with the 3 second time limit during ProcessExit. The MSDN documentation does say though that it can be extended:
The total execution time of all
ProcessExit event handlers is limited,
just as the total execution time of
all finalizers is limited at process
shutdown. The default is three
seconds, which can be overridden by an
unmanaged host.
Does anyone know how to override the default?
More ideas are also highly appreciated!
Follow up:
I have posted a follow up question to this.
You should have an entry point for your application. Normally you can do there some logging when all tasks are terminated:
static void Main()
{
try
{
Application.Run( .... );
}
finally
{
// logging ...
}
}
What exactly do you want to find out?
When the process terminates? (Just because the AppDomain is unloaded doesn't necessarily mean that the entire process is terminating)
When the main thread terminates (If there are other non-background threads, the main thread can terminate without the process terminating (or AppDomain unloading)
So they're not quite the same thing.
Anyway, it is generally dangerous to have log messages buffered in memory at all. What happens if someone turns off the power? Or if I terminate your process through Task Manager? All your log messages are gone. So often, you'll want unbuffered writes in your log, to get messages pushed to disk immediately.
Anyway, another (more robust) approach might be to run the logger itself in a non-background thread. That way, even if the rest of the application terminates, the logger won't, so the process is kept alive. Then you just have to set some flag when the rest of the app terminates, to let the logger know that it too should close once it has written out all pending log messages.
It still won't save you from the case where the system loses power or someone forcibly termianates the process on the OS-level, but it will handle all cases where the application closes normally, and gives you unlimited time to perform clean-up actions (since the process isn't actually terminating yet, it's still got one live thread)
ie. guaranteed to be called and have unlimited time to finish?
Unfortunately, NO option is going to have unlimited time, and be guaranteed. There is no way to enforce this, as many things can happen. Somebody tripping over the power cord or a forced termination of your program will prevent any option from giving you adequate time to handle things.
In general, putting your logic at the end of the Main routine is probably the most reasonable option, since that gives you complete freedom in handling your termination events. You have no time constraints there, and can have the processing take as much time as needed.
There are no guarantees that this will run, though, since a forceful termination of your program may bypass this entirely.
Based on the documentation, it looks like the default application domain (the one your Main method is probably running in) will not receive the DomainUnload event.
I don't know a built-in event that would do what you expect.
You could define your own custom event, have interested parties register with it, and fire off the event just before you return from Main().
I don't know how old this thread is, but I've had a similar problem whcih was a little tough for me to solve.
I had a WinForms application that was not firing any of the above forementioned events when a user logged out. Wraaping the Application.Run() in a try finally didn't work either.
Now to get around this you would have to using PInvoke into Win32 API's to achieve this. Well you did prior to .NET 2.0 anyways. Luckly MS introduced a new class called SystemEvents. With this class you can catch a SessionEnd event. This event allows you to cleanup when the OS want to terminate your app. There is no .NET time limit o this event it appears, although the OS will eventually kill your app if you take too long. This is a little more than 3 seconds, although 3 seconds should be plenty of time to cleanup.
Secondly my other problem was I wanted my worker thread to terminate the main thread once it was finished its work. With an Application.Run() this was hard to achieve. What I ended up doing was calling Application.Run() with a shared Application context. The thread is then able to call ApplicationContext.ThreadExit() to force the Application.Run to return. This seems to work quite nicely.
Hope this helps someone.
Regards
NozFX