Develop Reference

How to properly set up a multithreaded Asp.Net MVC + COM server on IIS

Update 2
The queueing problem was probably solved already, as we've been able to run multiple requests concurrently and the lib nicely reported progress for each operation. Other issues we're still facing about concurrency were likely the reason for this apparent behaviour, but that's a design matter. To solve this however, it'd be helpful to have some knowledge of the inner workings of classes, modules and variables as used in VB6. A question arise: would encapsulating everything (connections, components etc.) in classes ensure that every created object does not share any data with other instances?
Update 1
We've refactored the application a bit more to cope with resource disposing, especially when dealing with OCXs. Apparently that solved the out of memory issue. What still bothers me is that I don't understand what is happening beneath the surface. In this regard, is there a way to see what objects are currently in memory and how many references they have? I know the reference counting model is different from garbage collector-based systems. Still I would suppose the RCW wrapping our com objects would keep things clean for us. In the model given, is that a safe assumption or there's something we're missing?
So, I've probably read the most variegated kind of articles and docs about the topic of COM multithreading, but I still cannot get how that's supposed to work exactly, especially when interacting with .Net technologies such as ASP.Net MVC. That could be considered a simple fancy of mine, except for the fact that we've got this quite critical project and we're experiencing severe issues in trying to tie everything up. We're getting out of memory errors (in VB6) and apparently we got wrong how objects are created and data shared between these in COM. Continue reading to know how the story goes...
How things came to be
Not much to say here. We have a legacy VB6 Desktop application made up of a number of ActiveX DLLs. These are configured to use Apartment as the threading model, and all classes are set as MultiUse. All worked well and nice until the time came when we was requested to transpose the app on the mighty web :O
The problem we faced and how we (thought we) solved it
Since we haven't got the resources to design and develop a solution from scratch, we used a third party java(script)-based framework to quickly build a web app. However, much of the real work is done by the legacy library, so we needed a way to interface these two components. The easiest way we could think of was to build a very basic (w/o auth and w/o UI) Asp.Net MVC website to use as the middle layer. This would receive requests from the web app and translate them for the COM lib to crunch data.
To this end, and since the libs were never meant to be used as a server, we tried to refactor the whole thing a bit so that most classes can now be used in a standalone manner: this included separating logic from the UI and eliminating all module and public vars where possible; unfortunately, some of the former are still present, in particular some ComponentOne OCXs to handle reports and prints. All in all, this seemed to work just fine, until we had to deal with the COM threading model :O
Making sense of nonsense
Long story short, after a lot of digging and headaches we devised the current solution, which is outlined below:
we install the legacy app as usual, so that it register its dlls in the registry;
in our MVC solution, we use System.Threading.Tasks, one per every request, to start the requested operation in an asynchronous manner. We assign the operation an id and return this id to the client. To start the task we call this method:
protected Task<TReturn> StartSTATask<TReturn>(Func<TReturn> function)
{
var task = Task.Factory.StartNew(
function,
System.Threading.CancellationToken.None,
TaskCreationOptions.None,
STATaskScheduler // property to store the scheduler instance
);
return task;
}
the task is run using the STATaskScheduler. We modified it so that it spawns a new thread if the number of threads in the pool is set to 0.
/// <summary>Initializes a new instance of the StaTaskScheduler class with the specified concurrency level.</summary>
/// <param name="numberOfThreads">The number of threads that should be created and used by this scheduler.</param>
public StaTaskScheduler(int numberOfThreads)
{
// Validate arguments
//if (numberOfThreads < 1) throw new ArgumentOutOfRangeException("concurrencyLevel");
// Initialize the tasks collection
_tasks = new BlockingCollection<Task>();
if (numberOfThreads > 0)
{
// Create the threads to be used by this scheduler
_threads = Enumerable.Range(0, numberOfThreads).Select(i =>
{
var thread = new Thread(() =>
{
// Continually get the next task and try to execute it.
// This will continue until the scheduler is disposed and no more tasks remain.
foreach (var t in _tasks.GetConsumingEnumerable())
{
TryExecuteTask(t);
}
});
thread.Name = "sta_thread_" + i;
thread.IsBackground = true;
thread.SetApartmentState(ApartmentState.STA);
return thread;
}).ToList();
// Start all of the threads
_threads.ForEach(t => t.Start());
}
}
/// <summary>Queues a Task to be executed by this scheduler.</summary>
/// <param name="task">The task to be executed.</param>
protected override void QueueTask(Task task)
{
if (_threads != null)
// Push it into the blocking collection of tasks
_tasks.Add(task);
else
{
var thread = new Thread(() => TryExecuteTask(task));
thread.Name = "sta_thread_task_" + task.Id;
thread.IsBackground = true;
thread.SetApartmentState(ApartmentState.STA);
thread.Start();
}
}
And in our base controller's OnActionExecuting method we initiliaze it so
STATaskScheduler = HttpContext.Application["STATaskScheduler"] as TaskScheduler;
if (null == STATaskScheduler)
{
STATaskScheduler = new StaTaskScheduler(0);
HttpContext.Application["STATaskScheduler"] = STATaskScheduler;
}
we use a thin wrapper to instantiate and call our COM libs through reflection:
// Libraries is a Dictionary containing the names of the registered dlls
protected object InitCom(Libraries lib)
{
return InitCom(lib, true);
}
protected virtual object InitCom(Libraries lib, bool setOperation)
{
var comObj = GetComInstance(lib);
var success = SetUpConnection(comObj);
if (!success)
throw new LeafOperationException(lib, "Errore durante la connessione: {1}".Printf(connectionString));
if(setOperation)
return InitOperation(comObj);
return comObj;
}
protected object GetComInstance(Libraries lib)
{
var comType = Type.GetTypeFromProgID(MALib[lib]);
var comObj = Activator.CreateInstance(comType);
return comObj;
}
protected virtual bool DisposeCom(object comObj)
{
var success = CloseConnection(comObj);
if(!success)
throw new LeafOperationException("Errore durante la chiusura della connessione: {1}".Printf(connectionString));
//Marshal.FinalReleaseComObject(comObj);
//comObj = null;
return success;
}
protected bool SetUpConnection(object comObj)
{
var serverName = connectionString.ServerName();
var catalogName = connectionString.CatalogName();
return Convert.ToBoolean(comObj.InvokeMethod("Set_ConnectionWeb", serverName, catalogName));
}
protected bool CloseConnection(object comObj)
{
return Convert.ToBoolean(comObj.InvokeMethod("Close_ConnectionWeb"));
}
protected object InitOperation(object comObj)
{
comObj.GetType().InvokeMember("OperationID", BindingFlags.SetProperty, null, comObj, new object[] { OperationId });
comObj.GetType().InvokeMember("OperationHash", BindingFlags.SetProperty, null, comObj, new object[] { OperationHash });
return comObj;
}
The rationale behind this is that we create a new instance of the class with each request, eventually releasing it when done. Read here to know why we commented out the ReleaseComObject part. Basically, we were trading out of memory for a lot of COM object that has been separated from its underlying RCW cannot be used exceptions.
The object is then used like this within methods of various classes:
public bool ChiusuraMese()
{
try
{
PulisciMessaggi();
var comObj = InitCom(Libraries.Chiusura);
var byRefArgs = new int[] { 2 };
var oReturn = comObj.InvokeMethodByRef("ChiusuraMese", byRefArgs, IdDitta, PeriodoGiornaliera, IdDipendenti.PadLeft(), IdGruppoInstallazione, CodGruppoGestione);
DisposeCom(comObj);
return Convert.ToInt32(oReturn) == 0;
}
catch (Exception ex)
{
using (ErrorLog Log = new ErrorLog(System.Reflection.Assembly.GetExecutingAssembly().FullName, ex)) { }
aErrorMessage = ex.Message;
return false;
}
}
where InvokeMethodByRef is an extension method defined this way:
public static object InvokeMethodByRef(this object comObj, string methodName, int[] byRefArgs, params object[] args)
{
var modifiers = new ParameterModifier(args.Length);
byRefArgs.ToList().ForEach(index => { modifiers[index] = true; });
return comObj.GetType().InvokeMember(methodName, BindingFlags.InvokeMethod, null, comObj, args, new ParameterModifier[] { modifiers }, null, null);
}
Left out of the apartment
For what I understood, this whole apartment stuff is really hard to get right, with its cross-thread marshalling, message loop, yadda yadda whatnot. Add to that we're using and old, unsupported technology used to develop an application that was not architected for the purpose we're forcing it into. All that said, and taken for grant that the .Net side of things is working correctly, a couple of thoughts still wander in our minds. In particular:
is this the correct way to get advantage of multithreading with COM? Sometimes, multiple requests for the same object get stuck as if queued. This makes us wonder whether COM is actually sharing some instances between threads;
are we really creating and disposing of objects with each request, or under the hood COM handles things differently? Apparently, we're getting public vars overwritten, so there's probably some resource contention and reentering somewhere we wouldn't expect;
is the setup correct? Are there alternatives which are easier to maintain and debug? Please keep in mind we don't have neither the time nor the resources to rewrite anything in great extent. We could probably try something like creating an exe ActiveX, but I wouldn't count on that.
what's the "least worse" way to use OCXs in a project of this kind (not using them is not an option at the moment)? Should we dispose of them in some particular way? We already checked we set them to nothing when finished, but maybe some other thread is still using them;
should we be aware of any particular COM limit related to our out of memory issue? We encountered the problem before when the form had more than 256 unique controls displayed. Maybe the same is happening here somehow? The error seems to be especially related to classes using UI components.
Things I've already read (and probably did not understand)
Before you point to resources online I should read, I add here some topics I've encountered, in random order:
About SingleUse/MultiUse
http://www.vb-helper.com/howto_activex_dll.html
https://msdn.microsoft.com/en-us/library/aa242108(v=vs.60).aspx
Not really much choice here, if we want to stick with ActiveX DLLs with forms.
About (apartment) threading
https://msdn.microsoft.com/en-us/library/aa716297(v=vs.60).aspx
https://msdn.microsoft.com/en-us/library/aa716228(v=vs.60).aspx. By the way, this one probably hints that calls to objects are being serialized for access by other threads.
https://msdn.microsoft.com/en-us/library/windows/desktop/ms680112%28v=vs.85%29.aspx?f=255&MSPPError=-2147217396
About debugging
https://msdn.microsoft.com/en-us/library/aa241684(v=vs.60).aspx
https://msdn.microsoft.com/en-us/library/aa716193%28v=vs.60%29.aspx?f=255&MSPPError=-2147217396
Could a stack dump be of any help when we face the error? I don't even know how to use WinDbg, so I'd like at least to know if that would be a total waste of time :D
We're kinda stuck here, as we've got no clue as to where or what to look for, so any kind of help would be really appreciated.
Comments
So I've been pointed out I should read more about COM's threading model. I kind of expected that. Anyhow, to elaborate further, let me write some comments.
First, I don't have any control over CoInitialize or whatever, I'm just instantiating some VB6 dlls. I guess COM is doing such and such under the hood. Fact is, I could not find anywhere what that is (edit - apparently, .Net is already taking care of that for me, see the answer to this question: Do i need to call CoInitialize before interacting with COM in .NET?).
To recap:
I'm using STA threads from the client app
I'm using Activator.CreateInstance supposing it is actually creating a new object every time it is called. The call is done within a new STA thread.
Let's set aside for a moment questions about thread-safety in the actual DLLs. What I'm mainly interested in understanding here is if the described solution is a correct way (possibly not the best way, I'm aware of that) to exploit multithreading with COM libraries.
To cite some sources, to the best of my current knowledge I should be in the situation depicted in Figure 8.5 here: https://msdn.microsoft.com/en-us/library/aa716228(v=vs.60).aspx
I can't find any reason why this should not work, since as I said I'm supposing each object resides in its own apartment and has its own variables, plus a copy of global vars (see here: https://msdn.microsoft.com/en-us/library/aa261361(v=vs.60).aspx).

How to do unit testing on a static method that uses database access?

Here's the sample static method, say
public static void UpdateSchedule(int selectedScheduleId)
{
using (var dc = new MyDataContext())
{
var selectedSchedule = dc.Schedules.SingleOrDefault(p => p.ScheduleId == selectedScheduleId)
if selectedSchedule != null)
{
selectedSchedule.Name = name;
//and update other properties...
}
dc.SubmitChanges();
}
}
So what would be the correct approach to test on methods like this? Is there a way to avoid calling new MyDataContext() as it might increase the execution time of the unit test.
Also, I am using MsTest test framework in VS2012.

Static functions interfere with testing primarily because they:
Are difficult (sometimes impossible) to substitute from within a consumer
Tend to have "hidden" dependencies
Since you want to test the function itself, number 1 isn't an issue. However, number 2 is problematic because the static function is tightly coupled to the MyDataContext class.
If you want to test the static function without MyDataContext (that is, in isolation) you need to introduce a code seam. This requires some refactoring work, but we can do it fairly painlessly.
Consider if you had the following additional method:
public static void UpdateScheduleWithContext(int selectedScheduleId, IDataContext dc)
{
var selectedSchedule = dc.Schedules.SingleOrDefault(p => p.ScheduleId == selectedScheduleId)
if selectedSchedule != null)
{
selectedSchedule.Name = name;
//and update other properties...
}
dc.SubmitChanges();
}
This new function gives the consumer (i.e., the test) the ability to supply a test double for the data context. This is the seam.
Obviously, though, you don't want all consumers to explicitly provide a data context (which is why you had the original static function to begin with). So you can keep that function around, and just modify it:
public static void UpdateSchedule(int selectedScheduleId)
{
using (var dc = new MyDataContext())
{
UpdateScheduleWithDataContext(selectedScheduleId, dc);
}
}
If you don't have a seam like this, it will be impossible to test the function in isolation. As a result, an integration test would be the best you could hope for.

As Simon Whitehead said, it is impossible to do effective unit testing on static methods and objects. However, using the unit test framework in Visual Studio, you can create a 'unit test' function that is effectively a integration test. Tell Visual Studio to generate a Unit Test for UpdateSchedule, and then modify the function generated to set up the environment/state of the program such that the UpdateSchedule method can execute (create a database connection, instantiate classes, etc.). This includes ensuring that your database has records to update.
Once that is done, you can execute your integration test in the same manner that you would a unit test.

Unit Test Garbage Collection

Question
Do unit tests automatically dispose resources via garbage collection (System.IO.Stream in my case) once the test(s) have completed, or are things left open/in-use, requiring the disposing of IDisposable objects?
Context / Information
I'm currently making a unit test for a file uploader which uses a System.IO.Stream.
I've Moq'd out the HttpPostedFileBase with the InputStream of the file powered by a System.IO.MemoryStream, which all works as expected.
I currently have (altered for brevity):
[TestMethod]
public void TestUpload()
{
var stream = FunctionCreatingTheMemoryStream();
try
{
var file = new Mock<HttpPostedFileBase>();
file.Setup(f => f.FileName).Returns("test.txt");
file.Setup(f => f.InputStream).Returns(stream);
MethodThatUsesTheStream(file.Object)
// rest of test code with Assert
}
finally
{
stream.Dispose();
}
}
The question is with the MemoryStream instance that is created:
var stream = new FunctionCreatingTheMemoryStream();
Is it worthwhile placing any code after this in a try catch and then disposing of the stream in the finally statement, or with it being a unit test, will the memory stream be disposed of automatically?
So is it necessary to do this, or could it simply just be:
[TestMethod]
public void TestUpload()
{
var stream = FunctionCreatingTheMemoryStream();
var file = new Mock<HttpPostedFileBase>();
file.Setup(f => f.FileName).Returns("test.txt");
file.Setup(f => f.InputStream).Returns(stream);
MethodThatUsesTheStream(file.Object)
// rest of test code with Assert
}

The answer ultimately depends on the unit testing framework you use, but in .NET, none of the three major test frameworks (MSTest, NUnit, xUnit.net) automatically dispose of things. You have to manually ask them to do so.
You could argue that when executing a test suite, the test runner in principle launches a new process, runs all the tests, and then the process exits.
For some implementations of IDisposable, like MemoryStream, that simply means that the memory is reclaimed when the process exits. However, you can't always rely on that, because some disposable types may access out-of-process resources. You could theoretically have objects holding on to memory-mapped-files, named pipes, SQL Server connections, etc. Even if the test process exits, you may leave behind such resources. They'll probably time out sooner or later (like SQL Server connections returning to the pool), but it may slow down your system.
Furthermore, some test runners are attempting very hard to be clever these days, so they reuse one or more processes to be able to run faster, changing your test suite in and out of AppDomains.
So, in the end, unless you have something like a MemoryStream, where you're absolutely certain that it's not a big deal to leave it behind, you should deterministically dispose of your objects in your tests.
However, if you're doing Test-Driven Development, you should adopt the GOOS attitude of listening to your tests. If you write a lot of tests that involve IDisposable objects, you should consider if you can simplify your SUT's API. It depends on what you're doing, but if you write mostly managed code, you shouldn't need IDisposable much, because it's a Leaky Abstraction that leaks that the SUT depends on unmanaged resources..

In the past I have written a "Dustcart" class that implements IDisposable, and contains a collection of objects to dispose. These are disposed in the opersite order to how they are added (use a stack to implement it).
Then the test code looks like.
[Setup]
Public void Setup()
{
_dustcart = new Dustcart()
}
[TearDown]
public void TearDown ()
{
_dustcart.Dispose();
}
[TestMethod]
public void TestUpload()
{
var stream = _dustcart.DisposeOnTearDown(FunctionCreatingTheMemoryStream());
var file = new Mock<HttpPostedFileBase>();
file.Setup(f => f.FileName).Returns("test.txt");
file.Setup(f => f.InputStream).Returns(stream);
MethodThatUsesTheStream(file.Object)
// rest of test code with Assert
}
DisposeOnTearDown() is a generic method. You could put this this in a super class that all your tests inherent from, that also includes object mothers for the classes you need to mock etc.
However if you are not very careful your test code becomes harder to understand with no real benefit to the quality of the software you are testing. The tests are there to do a job and they have to be no better than is needed to do that job.

Totally agree with everyone that you need to dispose such objects. You can simplify your code a bit by using "using" statements, like this:
[TestMethod]
public void TestUpload()
{
using (var stream = FunctionCreatingTheMemoryStream())
{
var file = new Mock<HttpPostedFileBase>();
file.Setup(f => f.FileName).Returns("test.txt");
file.Setup(f => f.InputStream).Returns(stream);
MethodThatUsesTheStream(file.Object)
// rest of test code with Assert
}
}

How to unit test methods that use threads?

How to write unit tests for methods that use threads.
In example below how to test someMethod method ?
public class SomeClass {
private final SomeOtherClass someOtherClassInstance = IoC.getInstance(SomeOtherClass.class);
private final ExecutorService executorService = Executors.newCachedThreadPool();
public void someMethod() {
executorService.execute(new Runnable() {
#Override
public void run() {
someOtherClassInstance.someOtherMethod();
}
});
}
}
Are there any solutions in java and .net for this purpose?

Your question begs 'what is a unit test' (UT). UTs are just one form of automated tests. In this case your question implies that the code calls the OS to start a thread, so by definition, a test that tested it would not be a unit test but probably be an integration test.
Why I bother you with what seems like semantics is that understanding the intent of the test makes it so easy to write the test and structure the code. The amount of working code out there is larger than the amount of testable code out there.
So, how can this code be changed to be unit testable (I will skip the 'why bother stuff') ...
A unit test in C# tests a single type (your class) ... importantly (by definition) nothing else. So your code needs to reflect this. What you want to test is that when I call ABC it does this stuff. Something that stuff includes launching a thread. So you want to test that the method to launch a thread a called. The fundamental here is that your application requires a thread so that is what your asserting.
So how? Create a proxy, and perhaps a factory, for the thread creation. Then you can assert in a unit test that it was called and how it was treated. Sounds hard, but is really easy when you get into the habit.
BTW, Resharper makes creating a proxy for an OS types (e.g. thread). Check out the delegating members stuff in their help. If your not using Resharper you should be.
Yea I know this is not the answer you had hopped for, but believe me it is the answer you need.
I find it useful to think of tests in the categories:
Unit tests
Integration tests (or perhaps subsystem tests)
Functional test (e.g. UI/application/UAT tests)

The jMock Team has described an brilliant approach for Java in the 'jMock Cookbook: Test Multithreaded Code'. The concept of executing "thread" functionality on the same thread as the test can also be applied to C# code (see this blog post). The C# test would look like this:
[Test]
public void TestMultiThreadingCodeSynchronously()
{
// Arrange
SomeOtherClass someOtherClassMock = MockRepository.GenerateMock<SomeOtherClass>();
DeterministicTaskScheduler taskScheduler = new DeterministicTaskScheduler();
SomeClass systemUnderTest = new SomeClass(taskScheduler, someOtherClassMock);
// Act
systemUnderTest.SomeMethod();
// Now execute the new task on
// the current thread.
taskScheduler.RunTasksUntilIdle();
// Assert
someOtherClassMock.AssertWasCalled(x=>x.SomeOtherMethod());
}
Your "system under test" would look like this:
public class SomeClass
{
private readonly TaskScheduler taskScheduler;
private readonly SomeOtherClass instance;
public SomeClass(
TaskScheduler taskScheduler,
SomeOtherClass instance)
{
this.taskScheduler = taskScheduler;
this.instance = instance;
}
public void SomeMethod()
{
Task.Factory.StartNew(
instance.SomeOtherMethod,
new CancellationToken(),
TaskCreationOptions.None,
taskScheduler);
}
}
The C# solution is described in detail in this blog post.

If there is another thread, you have wait until the method has been called. You have to decide how long you should wait before considering the method wasn't called, there is no tool which can do that for you.
Of course you can mock the ExecutorService to run in the current thread and avoid the issue.

Mocking a blocking call with Rhino Mocks

I'm currently building a class using TDD. The class is responsible for waiting for a specific window to become active, and then firing some method.
I'm using the AutoIt COM library (for more information about AutoIt look here) since the behavior I want is actually a single method in AutoIt.
The code is pretty much as the following:
public class WindowMonitor
{
private readonly IAutoItX3 _autoItLib;
public WindowMonitor(IAutoItX3 autoItLib)
{
_autoItLib = autoItLib;
}
public void Run() // indefinitely
{
while(true)
{
_autoItLib.WinWaitActive("Open File", "", 0);
// Do stuff now that the window named "Open File" is finally active.
}
}
}
As you can see the AutoIt COM library implements an interface wich I can mock (Using NUnit and Rhino Mocks):
[TestFixture]
public class When_running_the_monitor
{
WindowMonitor subject;
IAutoItX3 mockAutoItLibrary;
AutoResetEvent continueWinWaitActive;
AutoResetEvent winWaitActiveIsCalled;
[SetUp]
public void Setup()
{
// Arrange
mockAutoItLibrary = MockRepository.GenerateStub<IAutoItX3>();
mockAutoItLib.Stub(m => m.WinWaitActive("", "", 0))
.IgnoreArguments()
.Do((Func<string, string, int, int>) ((a, b, c) =>
{
winWaitActiveIsCalled.Set();
continueWinWaitActive.WaitOne();
return 1;
}));
subject = new Subject(mockAutoItLibrary)
// Act
new Thread(new ThreadStart(subject.Run)).Start();
winWaitActiveIsCalled.WaitOne();
}
// Assert
[Test]
[Timeout(1000)]
public void should_call_winWaitActive()
{
mockAutoItLib.AssertWasCalled(m => m.WinWaitActive("Bestand selecteren", "", 0));
}
[Test]
[Timeout(1000)]
public void ensure_that_nothing_is_done_while_window_is_not_active_yet()
{
// When you do an "AssertWasCalled" for the actions when the window becomes active, put an equivalent "AssertWasNotCalled" here.
}
}
The problem is, the first test keeps timing out. I have already found out that when the stub "WinWaitActive" is called, it blocks (as intended, on the seperate thread), and when the "AssertWasCalled" is called after that, execution never returns.
I'm at a loss how to proceed, and I couldn't find any examples of mocking out a blocking call.
So in conclusion:
Is there a way to mock a blocking call without making the tests timeout?
(P.S. I'm less interested in changing the design (i.e. "Don't use a blocking call") since it may be possible to do that here, but I'm sure there are cases where it's a lot harder to change the design, and I'm interested in the more general solution. But if it's simply impossible to mock blocking calls, suggestions like that are more that welcome!)

Not sure if I understand the problem.
Your code is just calling a method on the mock (WinWaitActive). Of course, it can't proceed before the call returns. This is in the nature of the programming language and nothing you need to test.
So if you test that WinWaitActive gets called, your test is done. You could test if WinWaitActive gets called before anything else, but this requires ordered expectations, which requires the old style rhino mocks syntax and is usually not worth to do.
mockAutoItLibrary = MockRepository.GenerateStub<IAutoItX3>();
subject = new Subject(mockAutoItLibrary)
subject.Run()
mockAutoItLib.AssertWasCalled(m => m.WinWaitActive("Open File", "", 0));
You don't do anything else then calling a method ... so there isn't anything else to test.
Edit: exit the infinite loop
You could make it exit the infinite loop by throwing an exception from the mocks. This is not very nice, but it avoids having all this multi-threading stuff in the unit test.
mockAutoItLibrary = MockRepository.GenerateStub<IAutoItX3>();
// make loop throw an exception on second call
// to exit the infinite loop
mockAutoItLib
.Stub(m => m.WinWaitActive(
Arg<string>.Is.Anything,
Arg<string>.Is.Anything,
Arg<int>.Is.Anything));
.Repeat.Once();
mockAutoItLib
.Stub(m => m.WinWaitActive(
Arg<string>.Is.Anything,
Arg<string>.Is.Anything,
Arg<int>.Is.Anything));
.Throw(new StopInfiniteLoopException());
subject = new Subject(mockAutoItLibrary)
try
{
subject.Run()
}
catch(StopInfiniteLoopException)
{} // expected exception thrown by mock
mockAutoItLib.AssertWasCalled(m => m.WinWaitActive("Open File", "", 0));

Your Test only contains a call to the mocked method. Therefore it tests only your mock instead of any real code, which is an odd thing to do. We might need a bit more context to understand the problem.
Use Thread.Sleep() instead of AutoResetEvents: Since you are mocking the COM object that does the blocking window-active check, you can just wait for some time to mimick the behavior, and then make sure that the window is indeed active by making it active programmatically. How you block should not be important in the test, only that you block for some significant time.
Although from your code it is not clear how winWaitActiveIsCancelled and continueWinWaitActive contribute, I suspect they should be left out of the WinWaitActive mock. Replace them with a Thread.Sleep(500).