Develop Reference

Get an exception when acquiring mutex after a service restart

I'm writing a windows service with c#. It reads continuously from a MSMQ by calling Read() from a thread :
public string Read()
{
try
{
if (!readMutex.WaitOne(100))
{
return null;
}
var message = queue.Receive();
return (string)message.Body;
}
catch (Exception ex)
{
logger.Error("Exception:" + ex);
}
finally
{
readMutex.ReleaseMutex();
}
return null;
}
The mutex is created in the class constructor and disposed in the destructor.
The problem is that, after I stop and restart the service, I always get a AbandonedMutexException at the if (!readMutex.WaitOne(100)) the first time Read() is called.
After attaching a debugger and adding breakpoints, I found that when the service is stopped, the finally block is never entered, I'm not sure if that would be the problem.
It's maybe not a big problem because the next time Read() is called, the exception is no longer raised. But I'm wondering is there's a simple way to solve this?
Append 1 :
I found that destructor is always called when the service is stopped, so I tried release mutex in the destructor. But found that I'm not allowed to release it ,because the mutex seems to be acquired in a different thread context.
Append 2:
For those who are interested in this question, I'll add what I've found after examining what's going on.
I tested that if I create a program which acquires a mutex without releasing it, and then close the program, the next time the program run, it can still acquire the mutex successfully without any exception. This is contradictory to this problem's symptom, and also contradictory to what I used to think.
I think the truth is that the OS close the mutex for me when the program quits, so next time I can acquire it.
But why I failed with this service? Finally I found that I had another second service which also created this path's mutex. That second service just kept a mutex handle, without doing anything to it(e.g. waiting on it ). In this case, when my first service restart and try acquire the Mutex again, it gets the exception.
In conclusion, when the program terminates with an unreleased mutex:
1) if the mutex is also referenced by any other services/applications, then next time the mutex is acquired, an exception will be raised.
2) if it's the only program referencing this mutex, then the os will handle this gracefully for me, and no error will be reported at next acquisition.

readMutex.ReleaseMutex(); is most likely never being called when the service shuts down
More so, Receive blocks until it receives a message, so its likely that when the service is shutting down it times out, thinks that is has hang, and kills the process without closing gracefully.
There are several approaches here,
You are probably better to call MessageQueue.Receive Method (TimeSpan) with a short timeout, and adjust your logic accordingly, this way (on shut down) the receive will hopefully timeout before the service timesout.
The other approach is running this in a thread or task and killing it on the shutdown and make sure you call readMutex.ReleaseMutex()
Anyway you have enough information now, and you should be able to solve this in a way that suits you

AbandonedMutexException: The wait completed due to an abandoned mutex

Why would the following structure cause an AbandonedMutexException. Even if there is an error or method returns. The mutex is being released.
static Mutex WriteMutex = new Mutex(false, #"Global\mutex2203");
public static void Demo()
{
try
{
WriteMutex.WaitOne();
//rest of coding stuff here
}
finally
{
WriteMutex.ReleaseMutex();
}
}
Receives reports cant regenerate the bug.
Edit: The exception occurs at WriteMutex.WaitOne(); no other code. And only this method touches that mutex.

An AbandonedMutexException is thrown when one thread acquires a Mutex object that another thread has abandoned by exiting without releasing it (see AbandonedMutexException). The code you cite in your question would not necessarily be the code that is causing the exception, only "receiving" it (i.e. detecting the situation that throws the exception).
That is, code in another thread (could be the same method but is likely not) acquires the Mutex but does not release it and permits its thread to exit without the Mutex ever being released. Then the thread running the code you show above throws the exception when it attempts to acquire the Mutex.

Where is the exception occurring? Does it happen when you do WriteMutex.WaitOne();?
If so, there must be something (presumably not in the code you posted) that takes ownership of it, then exits happening before you get the exception.
Using async methods could also be a problem with code using Mutexes due to swapping the threads around. Make sure you aren't using any of that stuff in a non-compatible way.
Also, be aware that named mutexts are not local to your application: other processes could be screwing with it (and the problem could be there). If you want something local, don't give it a name, or even better use something more efficient and less error prone like the lock keyword for such cases.
Some nice details about using Mutex properly (and avoiding issues like you seem to have) are here: What is a good pattern for using a Global Mutex in C#?

You must also call WriteMutex.Dispose() in the finally block, but it is better to use a using block.
Try to use this pattern:
https://stackoverflow.com/a/229567/2185689

For me, I was getting "The wait completed due to an abandoned mutex." warning because the console app was running under the Task Scheduler and the task scheduler was terminating it. That was not behavior I wanted.
I resolved it by going to the task in question, Editing the Trigger and and unchecking the "Stop task if it runs longer than:" option.
Note that there are other options that can cause the task to termination as well.
Conditions Tab : Power -> "Stop if the computer switches to battery power"
Settings Tab : "Stop the task if it runs longer than:"

Is ReaderWriterLockSlim resistant to ThreadAbortException?

I would like to check whether following code is resistant against ThreadAbortException and will not lead into orphan lock. If it is not, what is the best pattern to avoid orphan locks here?
ReaderWriterLockSlim _lock = new ReaderWriterLockSlim();
public void DoStaff()
{
_lock.EnterWriteLock();
//Is this place where ThreadAbotException can corrupt my code, or is there JIT optimalization which prevent this from happening???
try
{
...
}
finally
{
_lock.ExitWriteLock();
}
}
According following link http://chabster.blogspot.cz/2013/07/a-story-of-orphaned-readerwriterlockslim.html, there is (or at least there was) possible way how to create orphan locks but I was running sample code for a while without any luck.
I am using .NET 4.0
Is there any difference between behavior in Debug and Release?

Yes, ThreadAbortException could occur there, in which case the try wouldn't be entered and therefore you would never exit the write lock.
There's no good general solution to the problem. Which is why Eric Lippert (among others) says that Locks and exceptions do not mix.
You're asking specifically about ThreadAbortException, which leads me to believe that you're contemplating using Thread.Abort for some kind of threading control in your application. I urge you to reconsider. If you want the ability to cancel your threads, you should use Cancellation or something similar. Using Thread.Abort in any other than the most dire circumstances is a horrifically bad idea. It certainly should not be part of your program's overall design.

In order for code which uses a locking primitive to be robust in the face of thread aborts, it is necessary that every lock-acquisition and lock-release request pass, or be performed through, an unshared token which can be given "ownership" the lock. Depending upon the design of the locking API, the token may be an object of some specific type, an arbitrary Object, or a variable passed as a ref parameter. It's imperative, however, that the token be created and stored by some means before the lock is acquired, so that if the token gets created but the store fails, the token may be abandoned without difficulty. Unfortunately, although monitor locks have added (in .NET 4.0) overloads of Monitor.Enter and Monitor.TryEnter which use ref bool as a token, I know of no equivalent for reader-writer locks.
If one wants abort-safe reader-writer lock functionality, I would suggest one would need a class which was designed around that; it should keep track of what threads hold reader or writer access and, rather than relying upon threads to release locks, it should, when waiting for a lock to be released, make sure the thread holding it is still alive. If a thread dies while holding read access, it should be released. If a thread dies while holding right access, any pending or future attempts to acquire the lock should throw an immediate exception.
Otherwise, there are some tricks via which a block of code can be protected against Thread.Abort(). Unfortunately, I don't know any clean way to bracket the code around a lock-acquisition request in such a way that Abort will work when the request itself can be cleanly aborted without having succeeded, but will be deferred if the request succeeds.
There are ways via which a framework could safely allow a thread which is in an endless loop to be killed by another thread, but designing mechanisms which could be used safely would require more effort than was put into Thread.Abort().

Best way to handle multi threaded applications from running code concurrently

I have an automatic betting BOT.
I use a Windows Service and timers to set off a job every 30 seconds in its own thread that takes bets from the DB, loops through and places them.
However in certain occurrences when the job is too long (over 30 seconds) I can get the same bet being placed twice using the same BetPK (unique ID) as the job for placing it runs at the same time as a previously started thread.
I am using C#, NET 4, VS 2012.
At the moment I set a "locked" flag in a table when the job to place bets runs and then unset it on finishing. So if another job runs and the job is locked it will return ASAP. However this is relying on the DB and network traffic.
What would be the best way in C# to prevent a job started by a timer thread from clashing with a previously started thread. I am thinking I could set a flag IN the service controller that spawns the threads so if a job is running another one won't spawn.
However I would like to learn the correct way to handle multi threaed clashes like this. I just lost a couple of hundred pounds today due to 2 LAY bets being placed at exactly the same time. As only one record existed for the Bet, the last bet placed had the Betfair ID updated so I had no clue about the duplicate until I checked Betfairs own page.
I do already do checks to see if the bet has already been placed before trying to place it but in cases where the "placebet" method is running on the same Bet record at exactly the same time then this is no good.
Any help much appreciated.
Thanks

No, the best solution is to keep the locks in the database. The app should be as stateless as possible. You already have a great solution.
Locking inside of your app is error prone and the errors are catastrophic (deadlock, the app stops to work until manually restarted). Locking using the database is much easier, and errors are recoverable.
Just get the locking with the database right. Ask a new question where you post details on what you're doing. I recommend that you XLOCK any betting jobs that you're working on. That way they can only be executed once. Use the power of database locks and transactions to make this work. This is by far easier than app-level threading.

You could always try implementing a db like Redis (redis.io) that offers built in POP functions (http://redis.io/commands/lpop). Redis has a C# client and is super useful for any kind of app where speed is crucial as it keeps the entire db in memory. It's also single threaded which makes it easy to implement distributors for multi-consumer type applications.
I'd also recommend checking out http://kkovacs.eu/cassandra-vs-mongodb-vs-couchdb-vs-redis as it lays out the pros and cons for Redis and other dbs. Might help you make future db decisions.

Old question, I know, but I wanted to throw this out there for anybody that stumbles across it.
C# (and presumably VB.NET) offers a couple of nice options for handling thread synchronization. You can use the lock keyword to block execution until a given lock is available, or Monitor.TryEnter() if you want to specify a timeout (possibly immediately) for taking the lock.
For either of these approaches, you need an object to use for locking. Pretty much any object will do; if you aren't synchronizing access to some object itself (collection, database connection, whatever), you can even just instantiate a throwaway object. For a polling timer, the latter is typical.
First, make sure you have an object to use for synchronization:
public class DatabasePollingClass {
object PollingTimerLock = new object();
...
Now, if you want the polling threads to block indefinitely waiting for their turn, use the lock keyword:
public class DatabasePollingClass {
object PollingTimerLock = new object();
...
protected void PollingTimerCallback() {
lock (PollingTimerLock) {
//Useful stuff here
}
}
}
Only a single thread will be allowed within the lock (PollingTimerLock) block of code at a time. All other threads will wait indefinitely, then resume executing as soon as they can acquire the lock for themselves.
However, you probably don't want that behavior. If you'd rather have the subsequent threads abort immediately (or after a short wait) if another polling thread is still running, you can use Monitor.TryEnter() when taking the lock. This does require slightly more caution, however:
public class DatabasePollingClass {
object PollingTimerLock = new object();
...
protected void PollingTimerCallback() {
if (Monitor.TryEnter(PollingTimerLock)) { //Acquires lock on PollingTimerLock object
try {
//Useful stuff here
} finally {
//Releases lock.
//You MUST do this in a finally block! (See below.)
Monitor.Exit(PollingTimerLock);
}
} else {
Console.WriteLine("Warning: Polling timer overlap. Skipping.");
}
}
}
The additional caution stems from the fact that, unlike the lock keyword, Monitor.TryEnter() requires you to manually release the lock when you're finished with it. In order to guarantee that this happens, you need to wrap your whole critical section in a try block, and release the lock in the finally block. This is to ensure that the lock will be released, even if the polling method fails or returns early. If the method returned without releasing the lock, your program would effectively be hung, as no further threads would be able to acquire the lock.
Another option, which doesn't use locking mechanisms, would be to configure your Timer without a repeat period, i.e. a one-shot Timer. At the end of your polling method, you would dispose the old Timer, and set a new one (you would also need to do this within a finally block to guarantee that the Timer gets reset by the end of the method). This approach would be useful if you want to poll the database at a certain interval since the end of the previous polling. It's a subtle distinction, but it also solves the problem of concurrent polling attempts.
Note that this is a really simple thread concurrency example. As long as all of your locking is happening on threads separate from your UI thread (the message pump itself can become a point of contention), and you're only ever locking a single object, you shouldn't have to worry too much about deadlocks. Those can be really unpleasant to debug; the symptom is usually "application stops responding, and now you get to guess which threads are waiting on what".

How to detect when main thread terminates?

What I need to know:
I would like to detect when a the main thread (process?) terminates so that I can ensure certain actions are performed before it is terminated.
What I have found myself:
I found the events AppDomain.DomainUnload and AppDomain.ProcessExit. AppDomain.DomainUnload seems to work with non-applications like MbUnit. AppDomain.ProcessExit seems to work with applications but there is a 3 second time limit which I really don't like. Is there more ways to detect when an AppDomain / process terminates?
Background:
I am looking for such an event to ensure my log is persistet to file when the application terminates. The actual logging runs on another thread using a producer-consumer pattern where it is very likely that log entries might queue up in memory and I need to ensure this queue is saved to file when the application terminates.
Is there anything else I should be aware of?
Update:
Changed the above to reflect what I have found out myself. I am not happy with the 3 second time limit during ProcessExit. The MSDN documentation does say though that it can be extended:
The total execution time of all
ProcessExit event handlers is limited,
just as the total execution time of
all finalizers is limited at process
shutdown. The default is three
seconds, which can be overridden by an
unmanaged host.
Does anyone know how to override the default?
More ideas are also highly appreciated!
Follow up:
I have posted a follow up question to this.

You should have an entry point for your application. Normally you can do there some logging when all tasks are terminated:
static void Main()
{
try
{
Application.Run( .... );
}
finally
{
// logging ...
}
}

What exactly do you want to find out?
When the process terminates? (Just because the AppDomain is unloaded doesn't necessarily mean that the entire process is terminating)
When the main thread terminates (If there are other non-background threads, the main thread can terminate without the process terminating (or AppDomain unloading)
So they're not quite the same thing.
Anyway, it is generally dangerous to have log messages buffered in memory at all. What happens if someone turns off the power? Or if I terminate your process through Task Manager? All your log messages are gone. So often, you'll want unbuffered writes in your log, to get messages pushed to disk immediately.
Anyway, another (more robust) approach might be to run the logger itself in a non-background thread. That way, even if the rest of the application terminates, the logger won't, so the process is kept alive. Then you just have to set some flag when the rest of the app terminates, to let the logger know that it too should close once it has written out all pending log messages.
It still won't save you from the case where the system loses power or someone forcibly termianates the process on the OS-level, but it will handle all cases where the application closes normally, and gives you unlimited time to perform clean-up actions (since the process isn't actually terminating yet, it's still got one live thread)

ie. guaranteed to be called and have unlimited time to finish?
Unfortunately, NO option is going to have unlimited time, and be guaranteed. There is no way to enforce this, as many things can happen. Somebody tripping over the power cord or a forced termination of your program will prevent any option from giving you adequate time to handle things.
In general, putting your logic at the end of the Main routine is probably the most reasonable option, since that gives you complete freedom in handling your termination events. You have no time constraints there, and can have the processing take as much time as needed.
There are no guarantees that this will run, though, since a forceful termination of your program may bypass this entirely.

Based on the documentation, it looks like the default application domain (the one your Main method is probably running in) will not receive the DomainUnload event.
I don't know a built-in event that would do what you expect.
You could define your own custom event, have interested parties register with it, and fire off the event just before you return from Main().

I don't know how old this thread is, but I've had a similar problem whcih was a little tough for me to solve.
I had a WinForms application that was not firing any of the above forementioned events when a user logged out. Wraaping the Application.Run() in a try finally didn't work either.
Now to get around this you would have to using PInvoke into Win32 API's to achieve this. Well you did prior to .NET 2.0 anyways. Luckly MS introduced a new class called SystemEvents. With this class you can catch a SessionEnd event. This event allows you to cleanup when the OS want to terminate your app. There is no .NET time limit o this event it appears, although the OS will eventually kill your app if you take too long. This is a little more than 3 seconds, although 3 seconds should be plenty of time to cleanup.
Secondly my other problem was I wanted my worker thread to terminate the main thread once it was finished its work. With an Application.Run() this was hard to achieve. What I ended up doing was calling Application.Run() with a shared Application context. The thread is then able to call ApplicationContext.ThreadExit() to force the Application.Run to return. This seems to work quite nicely.
Hope this helps someone.
Regards
NozFX