Develop Reference

Monitor.TryEnter and Threading.Timer race condition

I have a Windows service that every 5 seconds checks for work. It uses System.Threading.Timer for handling the check and processing and Monitor.TryEnter to make sure only one thread is checking for work.
Just assume it has to be this way as the following code is part of 8 other workers that are created by the service and each worker has its own specific type of work it needs to check for.
readonly object _workCheckLocker = new object();
public Timer PollingTimer { get; private set; }
void InitializeTimer()
{
if (PollingTimer == null)
PollingTimer = new Timer(PollingTimerCallback, null, 0, 5000);
else
PollingTimer.Change(0, 5000);
Details.TimerIsRunning = true;
}
void PollingTimerCallback(object state)
{
if (!Details.StillGettingWork)
{
if (Monitor.TryEnter(_workCheckLocker, 500))
{
try
{
CheckForWork();
}
catch (Exception ex)
{
Log.Error(EnvironmentName + " -- CheckForWork failed. " + ex);
}
finally
{
Monitor.Exit(_workCheckLocker);
Details.StillGettingWork = false;
}
}
}
else
{
Log.Standard("Continuing to get work.");
}
}
void CheckForWork()
{
Details.StillGettingWork = true;
//Hit web server to grab work.
//Log Processing
//Process Work
}
Now here's the problem:
The code above is allowing 2 Timer threads to get into the CheckForWork() method. I honestly don't understand how this is possible, but I have experienced this with multiple clients where this software is running.
The logs I got today when I pushed some work showed that it checked for work twice and I had 2 threads independently trying to process which kept causing the work to fail.
Processing 0-3978DF84-EB3E-47F4-8E78-E41E3BD0880E.xml for Update Request. - at 09/14 10:15:501255801
Stopping environments for Update request - at 09/14 10:15:501255801
Processing 0-3978DF84-EB3E-47F4-8E78-E41E3BD0880E.xml for Update Request. - at 09/14 10:15:501255801
Unloaded AppDomain - at 09/14 10:15:10:15:501255801
Stopping environments for Update request - at 09/14 10:15:501255801
AppDomain is already unloaded - at 09/14 10:15:501255801
=== Starting Update Process === - at 09/14 10:15:513756009
Downloading File X - at 09/14 10:15:525631183
Downloading File Y - at 09/14 10:15:525631183
=== Starting Update Process === - at 09/14 10:15:525787359
Downloading File X - at 09/14 10:15:525787359
Downloading File Y - at 09/14 10:15:525787359
The logs are written asynchronously and are queued, so don't dig too deep on the fact that the times match exactly, I just wanted to point out what I saw in the logs to show that I had 2 threads hit a section of code that I believe should have never been allowed. (The log and times are real though, just sanitized messages)
Eventually what happens is that the 2 threads start downloading a big enough file where one ends up getting access denied on the file and causes the whole update to fail.
How can the above code actually allow this? I've experienced this problem last year when I had a lock instead of Monitor and assumed it was just because the Timer eventually started to get offset enough due to the lock blocking that I was getting timer threads stacked i.e. one blocked for 5 seconds and went through right as the Timer was triggering another callback and they both somehow made it in. That's why I went with the Monitor.TryEnter option so I wouldn't just keep stacking timer threads.
Any clue? In all cases where I have tried to solve this issue before, the System.Threading.Timer has been the one constant and I think its the root cause, but I don't understand why.

I can see in log you've provided that you got an AppDomain restart over there, is that correct? If yes, are you sure that you have the one and the only one object for your service during the AppDomain restart? I think that during that not all the threads are being stopped right in the same time, and some of them could proceed with polling the work queue, so the two different threads in different AppDomains got the same Id for work.
You probably could fix this with marking your _workCheckLocker with static keyword, like this:
static object _workCheckLocker;
and introduce the static constructor for your class with initialization of this field (in case of the inline initialization you could face some more complicated problems), but I'm not sure is this be enough for your case - during AppDomain restart static class will reload too. As I understand, this is not an option for you.
Maybe you could introduce the static dictionary instead of object for your workers, so you can check the Id for documents in process.
Another approach is to handle the Stopping event for your service, which probably could be called during the AppDomain restart, in which you will introduce the CancellationToken, and use it to stop all the work during such circumstances.
Also, as #fernando.reyes said, you could introduce heavy lock structure called mutex for a synchronization, but this will degrade your performance.

TL;DR
Production stored procedure has not been updated in years. Workers were getting work they should have never gotten and so multiple workers were processing update requests.
I was able to finally find the time to properly set myself up locally to act as a production client through Visual Studio. Although, I wasn't able to reproduce it like I've experienced, I did accidentally stumble upon the issue.
Those with the assumptions that multiple workers were picking up the work was indeed correct and that's something that should have never been able to happen as each worker is unique in the work they do and request.
It turns out that in our production environment, the stored procedure to retrieve work based on the work type has not been updated in years (yes, years!) of deploys. Anything that checked for work automatically got updates which meant when the Update worker and worker Foo checked at the same time, they both ended up with the same work.
Thankfully, the fix is database side and not a client update.

Xamarin.Mac NSThread.Start() stack overflow

I've got a rather complex Xamarin.Mac application. In fact, it's a windows forms application, but we're using Mono for Mac compatibility with a native Mac GUI. One of our business logic components involves watching the filesystem for changes using FSWatcher. Unfortunately, FSWatcher on Mac is horribly broken, leaving us to use the native FSEvents API via Xamarin.Mac.
Deep down in business logic, I've got a custom class called CBFileSystemWatcher which wraps the .NET FSWatcher, and on mac provides an adapter between the FSWatcher-expecting business logic and FSEvents on mac. INSIDE this compatibility class, I've got
private FSEventStream eventStream;
//...
this.eventStream.ScheduleWithRunLoop (NSRunLoop.Main);
which schedules the filesystem events on the main run loop. Unfortunately, this means the GUI blocks FS event handling, so suddenly if a modal dialog is open, for example, fs events stop getting processed.
My thought is to create a new runloop for the FS event scheduling, which I figure looks like
NSThread.Start(()=>{
// Some other code
this.eventStream.ScheduleWithRunLoop (NSRunLoop.Current);
});
The snag is, I think, that this code runs inside maybe two other layers of thread starts. For testing purposes, I've got the following code where I NEED the above code:
NSThread.Start(()=>{
int i = 0;
});
with a breakpoint on the middle line to determine whether it was hit. 9 times out of ten I get the following stack overflow:
Stack overflow in unmanaged: IP: 0x261ba35, fault addr: 0xb02174d0
Stack overflow in unmanaged: IP: 0x261ba35, fault addr: 0xb02174d0
(the addresses change, though often recur)
One time out of ten the code works exactly as expected and I break on i=0
To test this further, I placed the above test inside my main AppDelegate.cs FinishedLaunching method. There, the code reliably works.
To further confuse matters, I placed the following code at the start of FinishedLaunching:
var fooThread = new Thread(() =>
{
var barThread = new Thread(()=>{
NSThread.Start(() =>
{
int i = 4;
});
});
barThread.Start();
});
fooThread.Start();
With breakpoints on fooThread.Start();, barThread.Start();, and int i = 4; the code works exactly as expected, where the points are hit in reverse order.
My question is, does anyone have any ideas on how to even begin deubgging this? The SO is so out of the blue I don't even know where to start.

A year later, I have this answer for you:
http://forums.xamarin.com/discussion/37451/more-modal-problems-nsmenu-nsstatusitem

Test succeeds when stepping through the test; othwerwise not

I have written a test with WatiN and when a step through the code with F10, the test succeeds, but when I execute the 'Run Test' command from the context menu, the test fails.
Here's my test:
[TestMethod]
[STAThread]
public void Should_show_captcha_after_three_invalid_login_attempts_with_invalid_username()
{
// Given
int numberOfLoginAttempts = 3;
// When
for (int loginAttempt = 1; loginAttempt <= numberOfLoginAttempts; loginAttempt++)
{
EnterUsername(LoginSettings.ValidUserName);
EnterPassword(loginAttempt.ToString());
ClickLoginButton();
// Check we are still on the loginpage
Assert.IsTrue(_browser.Title.Contains("Inloggen"));
}
bool isCaptchaVisible = _browser.Page<LoginPage>().Captcha.Exists;
// Then
Assert.IsTrue(isCaptchaVisible);
// Make sure to clear the login attempts for next test cases
RemoveLoginAttempts();
}
FYI: In the DB we keep track of the loginAttempts based on the username. When the number of loginattempts is > 2, the captcha is shown. The problem I encounter is that the counter in the DB stays 1. When I manually step through the test, the counter is increased.
How is this possible?

Well you are right it's got to be timing. However part of the problem is this is not a unit test, and the other is there's a great deal of asynchronous stuff going on that you are assuming will have completed before you execute another line of test code which has been written on the assumption that it has.
For instance the count in your loop for login attempts is definitely not the one in the db. All things being equal it should match up, but..
So to me you should have a test of the login function.
That logins up the colunt in the db if unsucessful, and reset that count if it succeeds.
Then an other test to see that when login attempts in the db has gone over the limit, login response detects that and shows the the correct response.
If you want to join all this up for an end to end / whitebox test. Then an automation test of some description should be used.
I suspect that windows, the browser, your webserver or even your dbms didn't get time to finish processing the first login attempt, before you'd queued up another two, and then done the test. Whereas in debug mode inbetwen you stepping through, they have plenty of time.

How to determine if a build is queued but not started in the TFS2010 API?

Im using the enum Microsoft.TeamFoundation.Build.Client.BuildStatus for each and it works great for the statuses contained in it.
But is there a blatantly obvious way of determining if a build is queued, but not yet in progress? I know the build explorer in visual studio can see this, im having problems getting that data programmatically.
Do i have to check something on the teamproject instead of the IBuildDetail itself? Any tips appreciated.

Unfortunately it doesn't appear that IBuildDetailSpec queries ever return queued builds. However, you can use the IQueuedBuildSpec interface to query for queued builds.
For example:
public IEnumerable<IQueuedBuild> getQueuedBuilds(TfsTeamProjectCollection tfsCollection, string teamProject)
{
// Get instance of build server service from TFS
IBuildServer buildServer = tfsCollection.GetService<IBuildServer>();
// Set up query
IQueuedBuildSpec spec = buildServer.CreateBuildQueueSpec(teamProject);
spec.Status = QueueStatus.Queued;
// Execute query
IQueuedBuildQueryResult result = buildServer.QueryQueuedBuilds(spec);
// Array of queued builds will be in the result.QueuedBuilds property
return result.QueuedBuilds;
}

You should be able to see the build status as BuildStatus.NotStarted once the build actually reaches the queue. There is a time before it reaches the queue that it really doesn't have a status, though.
If you're submitting your build programmatically, you can do:
bool success = queue.WaitForBuildStart(pollingIntervalInMilliseconds, timeOutInMilliseconds);
As long as the agent is listening, this should actually return very quickly. If you have a slow network, this could potentially take a couple seconds for handshaking.
Once it passes the WaitForBuildStart, the status is set to BuildStatus.NotStarted until it moves to InProgress and the rest of the way down the line.

#Greg Case
First thank you for the code , you gave me the solution that I was searching for. Second , I want to suggest using the same code snippet of yours to receive notifications from TFS for specific Queued build. I already used it to get notified that the build I queued is completed and it even returns exception details for failure case . To do that you need to connect() to the queued build returned from code and then register event , I wanted to add it as a comment but couldn't for long text
IQueuedBuild CurrentBuild = result.QueuedBuilds.First();
CurrentBuild.Connect();
CurrentBuild.PollingCompleted += CurrentBuild_PollingCompleted;

Why do I get an InteropServices.COMException when attempting to launch QTP from C#?

This question is a follow-up to the one at Can I use a language other than VBScript to programmatically execute QTP Tests?. I have a C# (.Net 2.0) program which creates an instance of QuickTest.Application and launches QuickTest Professional (version 10.0). All my development testing worked well.
This morning, I attempted to execute the program from a test machine without the development environment (SharpDevelop 2.2.1). I received an error attempting to execute the program when I double-clicked the Windows icon. The console window flashed too quickly to see what it was, so I dropped down to a command prompt and executed the program from there. Everything worked fine. On the second attempted program launch, and all subsequent ones, I receive a System.Runtime.InteropServices.COMException which seems to be caused by the COM object throwing an RPC_E_SERVERFAULT. The function in question is
virtual public QuickTest.Application LaunchQuickTestPro()
{
QuickTest.Application qtpApp = new QuickTest.Application();
qtpApp.Launch();
qtpApp.Visible = false;
return qtpApp;
}
and the qtpApp.Launch(); line is throwing the exception.
I'm at a complete loss as to what could be wrong. It works fine on the dev machine and worked fine once on the test machine. Rebooting between attempts seems to do no good. I'm fairly new to C#, .NET, and COM, so was hoping someone more experienced here might have seen this before. I'm probably missing something simple.
UPDATE: I have discovered this morning, after a reboot, that the Debug build works fine on the test machine (no development environment), but the Release build does not. I am going to try rebuilding and redeploying. Anyone have a suggestion for build options to examine for the Release build?
UPDATE2: It appears that both releases (Debug and Release) work correctly after a fresh reboot. If I try and launch either a second time, I get the error. I've added part of my Main() method and my ExitQTP() method below.
I'm wondering if part of the problem is my misunderstanding how ref should be used. However, the code does work every time when run in the IDE (SharpDevelop 2.2.1).
It does appear that something is not being properly cleaned up after the first run, but I don't know what. Looking at the task monitor, the QTP* processes go away as I expect them to. I think there may be a third process that is causing the problem, but haven't been able to isolate what that is,
//Program starts here
[STAThread]
public static void Main(string[] args)
{
string configFileName =
ConfigurationManager.AppSettings.Get("TestPathsConfigFile");
TextReader configFile = new StreamReader(configFileName);
QTPLauncher launcher = new QTPLauncher();
string testName = null;
try
{
Debug.WriteLine("Creating QuickTest.Application object...");
QuickTest.Application qtpApp = launcher.LaunchQuickTestPro();
Debug.WriteLine("Successfully created QuickTest.Application object...");
while((testName = configFile.ReadLine()) != null)
{
if((testName != string.Empty) &&
(!(testName.TrimStart()).StartsWith(COMMENT_START)))
{
Debug.WriteLine(testName);
launcher.ExecuteQTPTest(testName, ref qtpApp);
}
}
configFile.Close();
... //code unrelated to problem removed.
configFile = null;
launcher.ExitQTP(ref qtpApp);
}
catch(System.Runtime.InteropServices.COMException ce)
{
Console.Error.WriteLine(ce.StackTrace);
}
}
//Exits QTP
virtual public void ExitQTP(ref QuickTest.Application qtpApp)
{
qtpApp.Quit();
qtpApp = null;
}

I suspect the issue is that you are not properly closing (quitting) your QT app instance (if you check your task manager you may see it running) so the subsequent runs fail to initialize properly.
There is a decent blog post where Grant Holliday automates QT for running under Team Build. Many of the same principles would apply.
http://ozgrant.com/2008/02/28/running-hp-quicktest-professional-ui-tests-with-team-build/
If that's not the issue you'll need to provide more details about what you do with the QT application object.

Use the following:
Object oQTPapp;
oQTPapp = Server.CreateObject("QuickTest.Application");
Application qtpApp = (Application) oQTPapp;
Hopefully it will resolve your problem.