I'm writing a windows service with c#. It reads continuously from a MSMQ by calling Read() from a thread :
public string Read()
{
try
{
if (!readMutex.WaitOne(100))
{
return null;
}
var message = queue.Receive();
return (string)message.Body;
}
catch (Exception ex)
{
logger.Error("Exception:" + ex);
}
finally
{
readMutex.ReleaseMutex();
}
return null;
}
The mutex is created in the class constructor and disposed in the destructor.
The problem is that, after I stop and restart the service, I always get a AbandonedMutexException at the if (!readMutex.WaitOne(100)) the first time Read() is called.
After attaching a debugger and adding breakpoints, I found that when the service is stopped, the finally block is never entered, I'm not sure if that would be the problem.
It's maybe not a big problem because the next time Read() is called, the exception is no longer raised. But I'm wondering is there's a simple way to solve this?
Append 1 :
I found that destructor is always called when the service is stopped, so I tried release mutex in the destructor. But found that I'm not allowed to release it ,because the mutex seems to be acquired in a different thread context.
Append 2:
For those who are interested in this question, I'll add what I've found after examining what's going on.
I tested that if I create a program which acquires a mutex without releasing it, and then close the program, the next time the program run, it can still acquire the mutex successfully without any exception. This is contradictory to this problem's symptom, and also contradictory to what I used to think.
I think the truth is that the OS close the mutex for me when the program quits, so next time I can acquire it.
But why I failed with this service? Finally I found that I had another second service which also created this path's mutex. That second service just kept a mutex handle, without doing anything to it(e.g. waiting on it ). In this case, when my first service restart and try acquire the Mutex again, it gets the exception.
In conclusion, when the program terminates with an unreleased mutex:
1) if the mutex is also referenced by any other services/applications, then next time the mutex is acquired, an exception will be raised.
2) if it's the only program referencing this mutex, then the os will handle this gracefully for me, and no error will be reported at next acquisition.
readMutex.ReleaseMutex(); is most likely never being called when the service shuts down
More so, Receive blocks until it receives a message, so its likely that when the service is shutting down it times out, thinks that is has hang, and kills the process without closing gracefully.
There are several approaches here,
You are probably better to call MessageQueue.Receive Method (TimeSpan) with a short timeout, and adjust your logic accordingly, this way (on shut down) the receive will hopefully timeout before the service timesout.
The other approach is running this in a thread or task and killing it on the shutdown and make sure you call readMutex.ReleaseMutex()
Anyway you have enough information now, and you should be able to solve this in a way that suits you
Related
Why would the following structure cause an AbandonedMutexException. Even if there is an error or method returns. The mutex is being released.
static Mutex WriteMutex = new Mutex(false, #"Global\mutex2203");
public static void Demo()
{
try
{
WriteMutex.WaitOne();
//rest of coding stuff here
}
finally
{
WriteMutex.ReleaseMutex();
}
}
Receives reports cant regenerate the bug.
Edit: The exception occurs at WriteMutex.WaitOne(); no other code. And only this method touches that mutex.
An AbandonedMutexException is thrown when one thread acquires a Mutex object that another thread has abandoned by exiting without releasing it (see AbandonedMutexException). The code you cite in your question would not necessarily be the code that is causing the exception, only "receiving" it (i.e. detecting the situation that throws the exception).
That is, code in another thread (could be the same method but is likely not) acquires the Mutex but does not release it and permits its thread to exit without the Mutex ever being released. Then the thread running the code you show above throws the exception when it attempts to acquire the Mutex.
Where is the exception occurring? Does it happen when you do WriteMutex.WaitOne();?
If so, there must be something (presumably not in the code you posted) that takes ownership of it, then exits happening before you get the exception.
Using async methods could also be a problem with code using Mutexes due to swapping the threads around. Make sure you aren't using any of that stuff in a non-compatible way.
Also, be aware that named mutexts are not local to your application: other processes could be screwing with it (and the problem could be there). If you want something local, don't give it a name, or even better use something more efficient and less error prone like the lock keyword for such cases.
Some nice details about using Mutex properly (and avoiding issues like you seem to have) are here: What is a good pattern for using a Global Mutex in C#?
You must also call WriteMutex.Dispose() in the finally block, but it is better to use a using block.
Try to use this pattern:
https://stackoverflow.com/a/229567/2185689
For me, I was getting "The wait completed due to an abandoned mutex." warning because the console app was running under the Task Scheduler and the task scheduler was terminating it. That was not behavior I wanted.
I resolved it by going to the task in question, Editing the Trigger and and unchecking the "Stop task if it runs longer than:" option.
Note that there are other options that can cause the task to termination as well.
Conditions Tab : Power -> "Stop if the computer switches to battery power"
Settings Tab : "Stop the task if it runs longer than:"
I am using the following code for a critical section of a web page
if(Monitor.TryEnter(lockObj,60000))
{
try{
//write some things to a file
}
finally
{
Monitor.Exit(lockObj);
}
}
Here, lockObj is a static member of the class. My question is, what happens if the user closes the web page or the browser while the critical section is being executed? Does lockObj remain locked for future page requests?
Nothing automatically occurs when the user closes a browser window or navigates to another page, if the request is still processing. This is why the HttpResponse.IsClientConnected property exists - so your code can perform appropriate checks as and when you can do something useful.
If a timeout occurs on the server side, then the finally block should operate at around that time and the lock will be released. Whether it is safe for the lock to be released at this time is something only you can determine - the lock must have been put there for a reason, and if a timeout occurs, that may mean that the (shared, lock requiring state) isn't stable for whoever next acquires the lock.
But these are two very different sets of issues.
If you somehow manage to kill the executing thread inside of
try {
//write some things to a file
}
you might have some issues. But for all realistic cases with exceptions the code will function.
In effect, just avoid Thread.Abort.
I've got a simple .NET program, which checks to see if another instance has been started:
Mutex mutex = new Mutex(false,"MyMutexName");
if (!mutex.WaitOne(1))
return;
try{
//do stuff
}
catch{ //exceptions}
finally
{
mutex.ReleaseMutex();
}
My question is, what exactly happens to the mutex if you forget to release it when the program ends? Is it visible in some windows control panel component? Where does it live?
It is a named mutex so it is visible and can be opened in other processes. Windows uses a simple reference count on the handle. If you don't Dispose() it explicitly yourself then the finalizer will close the handle for you. If your program bombs hard and never executes the finalizer then Windows will do it when it cleans up the resources used by your program.
That will automatically decrement the reference count. If that counts down to zero (no other processes have a handle open on it) then the kernel object is released.
In other words: you don't have a problem no matter how badly things turn out. The actual mutant object lives in the kernel memory pool. You can see it with the SysInternals' WinObj tool.
From MSDN
If a thread terminates while owning a
mutex, the mutex is said to be
abandoned. The state of the mutex is
set to signaled and the next waiting
thread gets ownership. If no one owns
the mutex, the state of the mutex is
signaled. Beginning in version 2.0 of
the .NET Framework, an
AbandonedMutexException is thrown in
the next thread that acquires the
mutex. Prior to version 2.0 of the
.NET Framework, no exception was
thrown.
Caution
An abandoned mutex often indicates a
serious error in the code. When a
thread exits without releasing the
mutex, the data structures protected
by the mutex might not be in a
consistent state. The next thread to
request ownership of the mutex can
handle this exception and proceed, if
the integrity of the data structures
can be verified.
In the case of a system-wide mutex, an
abandoned mutex might indicate that an
application has been terminated
abruptly (for example, by using
Windows Task Manager).
Mutexes are OS-level handles. They'll get closed when your process does (if you don't close them sooner, that is.)
edit
Ok, I clearly misunderstood the example and the question. If you're just trying to detect whether another instance exists, you'd create a named mutex (or similar object) and simply check for its existence without ever locking on it.
The call to WaitOne locks on it, taking ownership, while ReleaseMutex gets rid of it (so long as there are no additional calls to WaitOne). If you end the thread without releasing the mutex fully, it does leave the object in a bad state, as explained in the text Micah quotes.
I took your question as being about whether you close the handle before the process completes, which is another thing entirely.
additional
At the SDK [API][1] level, you can call CreateMutex with the expectation of failing when a mutex of the same name has already been created. In .NET (well, in 4.0, at least), there's a [constructor][2] that fills a createdNew bool.
[1]: http://msdn.microsoft.com/en-us/library/ms682411(VS.85).aspx CreateMutex
[2]: http://msdn.microsoft.com/en-us/library/bwe34f1k(v=VS.90).aspx Mutex
I got a thread that is just banishing.. i'd like to know who is killing my thread and why.
It occurs to me my thread is being killed by the OS, but i'd like to confirm this and if possible to know why it's killing it.
As for the thread, i can assert it has at least 40 min of execution before dying, but it suddenly dies around 5 min.
public void RunWorker()
{
Thread worker = new Thread(delegate()
{
try
{
DoSomethingForALongLongTime();
}
catch(Exception e)
{
//Nothing is never logged :(
LogException(e);
throw e;
}
});
worker.IsBackground = true;
worker.SetApartmentState(System.Threading.ApartmentState.STA);
worker.Start();
}
EDIT: Addressing answers
Try/Catch Possible exceptions:
It's implemented and it catches nothing :(
Main Thread dying:
This thread is created by the web server, which continues to run
Work completion:
The work is not completed, as it finally affects the database, i can check whether it's done or not when the thread dies.
Having thought of these things brought me to this question, who is killing my threads??
ps. It's not Lady Goldent in the living room with the candle stick :)
Various people (including myself, here) pointed out that hosting a long-running thread in IIS is a bad idea. Your thread will being running inside an IIS 'worker process'. These processes are periodically terminated (recycled) by IIS, which will cause your thread to die.
I suggest that you try turning-off IIS worker process recycling to see if that makes a difference. You can find more information here.
Your thread probably just threw an exception. Try putting a try/catch block around DoSomethingForALongLongTime and see what it picks up.
Update: I didn't notice before that you were starting this from a web server. That can be a very bad idea. In particular, is the separate thread using any information derived from HttpContext.Current? That would include Request, Response, Session, etc., as well as any information from the page.
This is bad because these things only last as long as the request lasts. Once the request is over, they become invalid, to say the very least.
If you need to kick off a long-running thread from within a web application or web service, then you should create a simple Windows Service and host a WCF service within it. Have the web page then send all the information needed to perform the task to the service. The service can even use MSMQ as a transport, which will ensure that no messages are lost, even if the service gets busy.
A potential way to get more information: attach a debugger and break on thread termination. Depending on how your thread is being terminated, this might not work.
Download Debugging Tools for Windows if you don't already have it
Run windbg.exe, attach to your process
Break into windbg, type sxe et to enable breaking on thread exit
When the debugger breaks, inspect the state of the system, other threads, etc.
To get the managed stack, load sos.dll (.loadby sos mscorsvr, .loadby sos mscorwks, or .loadby sos clr should work), then run !clrstack (see !help for other sos commands)
If you get a lot of noise from other threads exiting, script windbg to continue after breaking if it's not the thread ID you care about.
Edit: If you think the thread is being terminated from within your process, you can also set a breakpoint on TerminateThread (bp kernel32!TerminateThread) and ExitThread (bp kernel32!ExitThread) to catch the stack of the killer.
I don't know the answer, but some thoughts:
Could it be throwing an exception? Have you tried putting a try/catch around the DoSomethingForALongLongTime() call?
Are there any points where it exits normally? Try putting some logging on them.
Do you get the same behaviour in and out of the debugger? Does the output window in the debugger provide any hints?
UPDATE
You said:
This thread is created by the web
server, which continues to run
If the thread is running inside asp.net then it may be that the thread is being killed when the asp.net worker process recycles, which it will do periodically. You could try turning off worker process recycling and see if that makes any difference.
Your edit reveals the answer:
It's the butler web server.
How exactly do you host these threads? A webserver environment isn't exactly designed to host long living processes. In fact, it is probably configured to halt runaway sites, every 40 minutes maybe?
Edit:
For a quick fix, your best chance is to set worker.IsBackground = false; because your current setting of true allows the system to kill the parent-thread w/o waiting for your bgw.
On another note, there is little point in using a BackgroundWorker in an ASP.NET application, it is intended for WinForms and WPF. It would be better to create a separate thread for this, since you are changing some of the Threads properties. That is not advised for a ThreadPool (Bgw) thread.
The process might be terminating. That would be what worker.IsBackground = true; is designed to do, kill your thread when the main thread exits.
A background thread will only run as long there are foreground threads runnnig.
As soon that all foreground threads end, any background thread still running will aborted.
If checking for an exception doesn't show anything useful, get your thread code to write to a log file at key points. You'll then be able to see exactly when it stops working and hopefully why.
A simple answer would be: "The killer doesn't leave a name card" ;)
If your thread is hosted in IIS, probably the thread is killed by the app pool process which recycles. The server might continue running but the process which hosts your item is stopped untill a new request fires everything up again.
If your thread is hosted in an executable, the only way it can be killed is by killing the thread yourself, throwing an exception in the thread or terminating the host process
Hope this helps.
You can try to increase executionTimeout value of configuration\system.web\httpRuntime in web.config (default value is 110 seconds in .NET 4.0 and 90 in corresponds to http://msdn.microsoft.com/en-us/library/e1f13641.aspx). You can try to change it dynamically Server.ScriptTimeout = 300 (see http://www.beansoftware.com/ASP.NET-Tutorials/Long-Operations.aspx). It this parameter will not helps, then I think you have a problem other as thread recycling from IIS. How you can see default value of this parameter is much less as typical live time of your thread. I think, that your problem has another nature, but to be sure...
Why you set apartment state for the thread? Which COM objects you use in the working thread? Do you have an unmanaged code which do the most of work where you can also insert some code? I think you should have more information about SomethingForALongLongTime to be able to solve the problem.
And one more a little suggestion. Could you insert a line of code after calling SomethingForALongLongTime(); to be sure, that SomethingForALongLongTime not end without an exception?
UPDATED: To be absolutely sure that your thread will be not killed by IIS, you can try to create a process which do SomethingForALongLongTime(); instead of using threads.
When you call RunWorker(), you can add a reference to your thread to a list. Once you have detected that your thread has died, you can inspect the state of the thread, perhaps it will reveal how it died. Or, perhaps it hasn't died, its just waiting on some resource (like the connection to the database).
List runningThreads = ...
public void RunWorker() {
Thread worker = new Thread(delegate()
..
runningThreads.add(worker);
worker.Start();
}
public void checkThreads() {
for (Thread t : runningThreads) {
Console.WriteLine("ThreadState: {0}", t.ThreadState);
}
}
It could be throwing one of the various uncatcheable exceptions including Stack Overflow or Out of Memory. These are the hardest exceptions to track down.
What does memory consumption look like while this thread is running? Can you use a memory profiler on it to see if it's out of control? Can you add some logging in inner loops? If you have a recursive method, add a counter and throw an exception if it recurses an impossible number of times. Are you using large objects that could be causing large object heap fragmentation (causes out of memory errors even when you aren't really out).
You should instrument DoSomethingForALongLongTime() with lots of debug logs, so you can find out at what spot does the code stop executing. Or attach a debugger and break on all first chance exceptions.
use AsyncTasks to achieve your long running work in asp.net
Try use app domain UnhandledException event: http://msdn.microsoft.com/en-us/library/system.appdomain.unhandledexception.aspx
it may give you some information if you miss some exceptions
What I need to know:
I would like to detect when a the main thread (process?) terminates so that I can ensure certain actions are performed before it is terminated.
What I have found myself:
I found the events AppDomain.DomainUnload and AppDomain.ProcessExit. AppDomain.DomainUnload seems to work with non-applications like MbUnit. AppDomain.ProcessExit seems to work with applications but there is a 3 second time limit which I really don't like. Is there more ways to detect when an AppDomain / process terminates?
Background:
I am looking for such an event to ensure my log is persistet to file when the application terminates. The actual logging runs on another thread using a producer-consumer pattern where it is very likely that log entries might queue up in memory and I need to ensure this queue is saved to file when the application terminates.
Is there anything else I should be aware of?
Update:
Changed the above to reflect what I have found out myself. I am not happy with the 3 second time limit during ProcessExit. The MSDN documentation does say though that it can be extended:
The total execution time of all
ProcessExit event handlers is limited,
just as the total execution time of
all finalizers is limited at process
shutdown. The default is three
seconds, which can be overridden by an
unmanaged host.
Does anyone know how to override the default?
More ideas are also highly appreciated!
Follow up:
I have posted a follow up question to this.
You should have an entry point for your application. Normally you can do there some logging when all tasks are terminated:
static void Main()
{
try
{
Application.Run( .... );
}
finally
{
// logging ...
}
}
What exactly do you want to find out?
When the process terminates? (Just because the AppDomain is unloaded doesn't necessarily mean that the entire process is terminating)
When the main thread terminates (If there are other non-background threads, the main thread can terminate without the process terminating (or AppDomain unloading)
So they're not quite the same thing.
Anyway, it is generally dangerous to have log messages buffered in memory at all. What happens if someone turns off the power? Or if I terminate your process through Task Manager? All your log messages are gone. So often, you'll want unbuffered writes in your log, to get messages pushed to disk immediately.
Anyway, another (more robust) approach might be to run the logger itself in a non-background thread. That way, even if the rest of the application terminates, the logger won't, so the process is kept alive. Then you just have to set some flag when the rest of the app terminates, to let the logger know that it too should close once it has written out all pending log messages.
It still won't save you from the case where the system loses power or someone forcibly termianates the process on the OS-level, but it will handle all cases where the application closes normally, and gives you unlimited time to perform clean-up actions (since the process isn't actually terminating yet, it's still got one live thread)
ie. guaranteed to be called and have unlimited time to finish?
Unfortunately, NO option is going to have unlimited time, and be guaranteed. There is no way to enforce this, as many things can happen. Somebody tripping over the power cord or a forced termination of your program will prevent any option from giving you adequate time to handle things.
In general, putting your logic at the end of the Main routine is probably the most reasonable option, since that gives you complete freedom in handling your termination events. You have no time constraints there, and can have the processing take as much time as needed.
There are no guarantees that this will run, though, since a forceful termination of your program may bypass this entirely.
Based on the documentation, it looks like the default application domain (the one your Main method is probably running in) will not receive the DomainUnload event.
I don't know a built-in event that would do what you expect.
You could define your own custom event, have interested parties register with it, and fire off the event just before you return from Main().
I don't know how old this thread is, but I've had a similar problem whcih was a little tough for me to solve.
I had a WinForms application that was not firing any of the above forementioned events when a user logged out. Wraaping the Application.Run() in a try finally didn't work either.
Now to get around this you would have to using PInvoke into Win32 API's to achieve this. Well you did prior to .NET 2.0 anyways. Luckly MS introduced a new class called SystemEvents. With this class you can catch a SessionEnd event. This event allows you to cleanup when the OS want to terminate your app. There is no .NET time limit o this event it appears, although the OS will eventually kill your app if you take too long. This is a little more than 3 seconds, although 3 seconds should be plenty of time to cleanup.
Secondly my other problem was I wanted my worker thread to terminate the main thread once it was finished its work. With an Application.Run() this was hard to achieve. What I ended up doing was calling Application.Run() with a shared Application context. The thread is then able to call ApplicationContext.ThreadExit() to force the Application.Run to return. This seems to work quite nicely.
Hope this helps someone.
Regards
NozFX