I'm trying to unit test the cancel execution scenario in a class conceptually similar to the following:
public class ContextExecutor
{
public ContextExecutor(IContextRunner runner, IExecutionCanceler canceler)
{
this.runner = runner;
this.canceler = canceler;
}
public void Execute(IEnumerable<IContext> contexts)
{
foreach (var ctx in contexts)
{
if (canceler.IsCanceled)
{
break;
}
runner.Run(ctx);
}
}
readonly IContextRunner runner;
readonly IExecutionCanceler canceler;
}
public interface IContextRunner
{
void Run(IContext context);
}
public interface IExecutionCanceler
{
bool IsCanceled { get; }
}
The test case I was after should go through the following steps:
start ContextExecutor.Execute() asynchronously somehow;
put that method execution on hold until something unlocks it from unit test code;
unlock execution and let it perform 1 (..or 2, or..) loop runs, anyway less than full enumerable length;
invoke canceling by setting canceler.IsCanceled = true;
unlock loop execution free;
wait synchronously for method completion;
assert that loop has been invoked the expected nr of times.
I got tangled up with controlling loop execution locking/unlocking from unit test code. Apart from starting Execute() in a new thread, I avoided using threading synchronization primitives (e.g. semaphores and locks). I also had to discard a Task-based approach, as I could not change signatures to apply async/await constructs. I tried to play with the following, but with no luck:
inject a yield-powered function as IEnumerable<IContext> input parameter to hold loop on foreach() line, to release loop everytime another yield is hit, and try to control that from unit test code.
inject a IContextRunner runner powered by a Reactive Extension Subject to hold loop on runner.Run line, to release loop everytime another Subject.OnNext is hit, and try to control that from unit test code.
For that matters, unit testing framework is NUnit, while NSubstitute is the mocking framework and FluentAssertion is the assertion library of choice. I know how to arrange/act/assert with those.
What is so evident that I missing? Thanks
EDIT
To provide an example of what has been tried, this is a Task-based approach made after posting question and reading #Peter Duniho helpful comment:
// in unit test class
ContextExecutor executor;
IContextRunner runner;
IExecutionCanceler canceler;
IRunnableContext[] runnableContexts;
int totalNrOfContexts;
int nrOfContextToRun; // this will be < totalNrOfContexts
int actualNrOfContextRan;
[SetUp]
public virtual void before_each()
{
// create instance under test, mock dependencies, dummy input data
Initialize();
RunScenarioAsync().Wait();
}
async Task RunScenarioAsync()
{
// prepare mock IContextRunner so that for each input context:
// * there's a related TaskCompletionSource<Object>
// * Run() increments actualNrOfContextRan
// * Run() performs taskSource.Task.Wait();
List<TaskCompletionSource<Object>> runTaskSources = PrepareMockContextRunner();
canceler.IsCanceled.Returns(false); // let execution go initially
// queue up method under test to be processed asynchronously
var executeTask = Task.Run(() =>
{
executor.Execute(runnableContexts);
};
// "unlock" some IContextRunner.Run() invocations,
// for when they will be invoked
for (int i = 0; i < nrOfContextToRun; i++)
{
runTaskSources[i].SetResult(null);
await Task.Delay(0); // tried also with Delay(1) and without this line at all
}
// flag to cancel execution
canceler.IsCanceled.Returns(true);
// unlock all remaining IContextRunner.Run() invocations,
// again for when/if they will be invoked
for (int i = nrOfContextToRun; i < totalNrOfContexts; i++)
{
runTaskSources[i].SetResult(null);
await Task.Delay(0);
}
// wait until method under test completes
await executeTask;
}
[Test]
public void it_should_only_run_until_cancel()
{
int expected = nrOfContextToRun;
int actual = actualNrOfContextRan;
actual.Should().Be(expected);
}
The problem I have here (and similar to other approaches tried) is about giving and regain control to/from the method under test in a predictable way (that is, synchronizing).
Here, if there's no await Task.Delay() or if delay is 0ms, only 1 context is actually ran: the method under test has no chance to run the 2nd and 3rd one, it finds the canceling flag too soon. If delay is 1ms, method executes more context than expected before actually detecting the flag. Also tried with ticks instead of ms, but in my experience playing with delays usually means you're doing something wrong.
Related
I am actually reading some topics about the Task Parallel Library and the asynchronous programming with async and await. The book "C# 5.0 in a Nutshell" states that when awaiting an expression using the await keyword, the compiler transforms the code into something like this:
var awaiter = expression.GetAwaiter();
awaiter.OnCompleted (() =>
{
var result = awaiter.GetResult();
Let's assume, we have this asynchronous function (also from the referred book):
async Task DisplayPrimeCounts()
{
for (int i = 0; i < 10; i++)
Console.WriteLine (await GetPrimesCountAsync (i*1000000 + 2, 1000000) +
" primes between " + (i*1000000) + " and " + ((i+1)*1000000-1));
Console.WriteLine ("Done!");
}
The call of the 'GetPrimesCountAsync' method will be enqueued and executed on a pooled thread. In general invoking multiple threads from within a for loop has the potential for introducing race conditions.
So how does the CLR ensure that the requests will be processed in the order they were made? I doubt that the compiler simply transforms the code into the above manner, since this would decouple the 'GetPrimesCountAsync' method from the for loop.
Just for the sake of simplicity, I'm going to replace your example with one that's slightly simpler, but has all of the same meaningful properties:
async Task DisplayPrimeCounts()
{
for (int i = 0; i < 10; i++)
{
var value = await SomeExpensiveComputation(i);
Console.WriteLine(value);
}
Console.WriteLine("Done!");
}
The ordering is all maintained because of the definition of your code. Let's imagine stepping through it.
This method is first called
The first line of code is the for loop, so i is initialized.
The loop check passes, so we go to the body of the loop.
SomeExpensiveComputation is called. It should return a Task<T> very quickly, but the work that it'd doing will keep going on in the background.
The rest of the method is added as a continuation to the returned task; it will continue executing when that task finishes.
After the task returned from SomeExpensiveComputation finishes, we store the result in value.
value is printed to the console.
GOTO 3; note that the existing expensive operation has already finished before we get to step 4 for the second time and start the next one.
As far as how the C# compiler actually accomplishes step 5, it does so by creating a state machine. Basically every time there is an await there's a label indicating where it left off, and at the start of the method (or after it's resumed after any continuation fires) it checks the current state, and does a goto to the spot where it left off. It also needs to hoist all local variables into fields of a new class so that the state of those local variables is maintained.
Now this transformation isn't actually done in C# code, it's done in IL, but this is sort of the morale equivalent of the code I showed above in a state machine. Note that this isn't valid C# (you cannot goto into a a for loop like this, but that restriction doesn't apply to the IL code that is actually used. There are also going to be differences between this and what C# actually does, but is should give you a basic idea of what's going on here:
internal class Foo
{
public int i;
public long value;
private int state = 0;
private Task<int> task;
int result0;
public Task Bar()
{
var tcs = new TaskCompletionSource<object>();
Action continuation = null;
continuation = () =>
{
try
{
if (state == 1)
{
goto state1;
}
for (i = 0; i < 10; i++)
{
Task<int> task = SomeExpensiveComputation(i);
var awaiter = task.GetAwaiter();
if (!awaiter.IsCompleted)
{
awaiter.OnCompleted(() =>
{
result0 = awaiter.GetResult();
continuation();
});
state = 1;
return;
}
else
{
result0 = awaiter.GetResult();
}
state1:
Console.WriteLine(value);
}
Console.WriteLine("Done!");
tcs.SetResult(true);
}
catch (Exception e)
{
tcs.SetException(e);
}
};
continuation();
}
}
Note that I've ignored task cancellation for the sake of this example, I've ignored the whole concept of capturing the current synchronization context, there's a bit more going on with error handling, etc. Don't consider this a complete implementation.
The call of the 'GetPrimesCountAsync' method will be enqueued and executed on a pooled thread.
No. await does not initiate any kind of background processing. It waits for existing processing to complete. It is up to GetPrimesCountAsync to do that (e.g. using Task.Run). It's more clear this way:
var myRunningTask = GetPrimesCountAsync();
await myRunningTask;
The loop only continues when the awaited task has completed. There is never more than one task outstanding.
So how does the CLR ensure that the requests will be processed in the order they were made?
The CLR is not involved.
I doubt that the compiler simply transforms the code into the above manner, since this would decouple the 'GetPrimesCountAsync' method from the for loop.
The transform that you shows is basically right but notice that the next loop iteration is not started right away but in the callback. That's what serializes execution.
When I analyse code coverage in Visual Studio 2012, any of the await lines in async methods are showing as not covered even though they are obviously executing since my tests are passing. The code coverage report says that the uncovered method is MoveNext, which is not present in my code (perhaps it's compiler-generated).
Is there a way to fix code coverage reporting for async methods?
Note:
I just ran coverage using NCover, and the coverage numbers make a lot more sense using that tool. As a workaround for now, I'll be switching to that.
This can happen most commonly if the operation you're awaiting is completed before it's awaited.
I recommend you test at least synchronous and asynchronous success situations, but it's also a good idea to test synchronous and asynchronous errors and cancellations.
The reason the code is not shown as being covered has to do with how async methods are implemented. The C# compiler actually translates the code in async methods into a class that implements a state machine, and transforms the original method into a stub that initialized and invokes that state machine. Since this code is generated in your assembly, it is included in the code coverage analysis.
If you use a task that is not complete at the time the code being covered is executing, the compiler-generated state machine hooks up a completion callback to resume when the task completes. This more completely exercises the state machine code, and results in complete code coverage (at least for statement-level code coverage tools).
A common way to get a task that is not complete at the moment, but will complete at some point is to use Task.Delay in your unit test. However, that is generally a poor option because the time delay is either too small (and results in unpredictable code coverage because sometimes the task is complete before the code being tests runs) or too large (unnecessarily slowing the tests down).
A better option is to use "await Task.Yield()". This will return immediately but invoke the continuation as soon as it is set.
Another option - though somewhat absurd - is to implement your own awaitable pattern that has the semantics of reporting incomplete until a continuation callback is hooked up, and then to immediately complete. This basically forces the state machine into the async path, providing the complete coverage.
To be sure, this is not a perfect solution. The most unfortunate aspect is that it requires modification to production code to address a limitation of a tool. I would much prefer that the code coverage tool ignore the portions of the async state machine that are generated by the compiler. But until that happens, there aren’t many options if you really want to try to get complete code coverage.
A more complete explanation of this hack can be found here: http://blogs.msdn.com/b/dwayneneed/archive/2014/11/17/code-coverage-with-async-await.aspx
There are situations where I don't care about testing the async nature of a method but just want to get rid of the partial code coverage. I use below extension method to avoid this and it works just fine for me.
Warning "Thread.Sleep" used here!
public static IReturnsResult<TClass> ReturnsAsyncDelayed<TClass, TResponse>(this ISetup<TClass, Task<TResponse>> setup, TResponse value) where TClass : class
{
var completionSource = new TaskCompletionSource<TResponse>();
Task.Run(() => { Thread.Sleep(200); completionSource.SetResult(value); });
return setup.Returns(completionSource.Task);
}
and the usage is similar to the Moq's ReturnsAsync Setup.
_sampleMock.Setup(s => s.SampleMethodAsync()).ReturnsAsyncDelayed(response);
I created a test runner that runs a block of code multiple times and varies the task that is delayed using a factory. This is great for testing the different paths through simple blocks of code. For more complex paths you may want to create a test per path.
[TestMethod]
public async Task ShouldTestAsync()
{
await AsyncTestRunner.RunTest(async taskFactory =>
{
this.apiRestClient.GetAsync<List<Item1>>(NullString).ReturnsForAnyArgs(taskFactory.Result(new List<Item1>()));
this.apiRestClient.GetAsync<List<Item2>>(NullString).ReturnsForAnyArgs(taskFactory.Result(new List<Item2>()));
var items = await this.apiController.GetAsync();
this.apiRestClient.Received().GetAsync<List<Item1>>(Url1).IgnoreAwait();
this.apiRestClient.Received().GetAsync<List<Item2>>(Url2).IgnoreAwait();
Assert.AreEqual(0, items.Count(), "Zero items should be returned.");
});
}
public static class AsyncTestRunner
{
public static async Task RunTest(Func<ITestTaskFactory, Task> test)
{
var testTaskFactory = new TestTaskFactory();
while (testTaskFactory.NextTestRun())
{
await test(testTaskFactory);
}
}
}
public class TestTaskFactory : ITestTaskFactory
{
public TestTaskFactory()
{
this.firstRun = true;
this.totalTasks = 0;
this.currentTestRun = -1; // Start at -1 so it will go to 0 for first run.
this.currentTaskNumber = 0;
}
public bool NextTestRun()
{
// Use final task number as total tasks.
this.totalTasks = this.currentTaskNumber;
// Always return has next as turn for for first run, and when we have not yet delayed all tasks.
// We need one more test run that tasks for if they all run sync.
var hasNext = this.firstRun || this.currentTestRun <= this.totalTasks;
// Go to next run so we know what task should be delayed,
// and then reset the current task number so we start over.
this.currentTestRun++;
this.currentTaskNumber = 0;
this.firstRun = false;
return hasNext;
}
public async Task<T> Result<T>(T value, int delayInMilliseconds = DefaultDelay)
{
if (this.TaskShouldBeDelayed())
{
await Task.Delay(delayInMilliseconds);
}
return value;
}
private bool TaskShouldBeDelayed()
{
var result = this.currentTaskNumber == this.currentTestRun - 1;
this.currentTaskNumber++;
return result;
}
public async Task VoidResult(int delayInMilliseconds = DefaultDelay)
{
// If the task number we are on matches the test run,
// make it delayed so we can cycle through them.
// Otherwise this task will be complete when it is reached.
if (this.TaskShouldBeDelayed())
{
await Task.Delay(delayInMilliseconds);
}
}
public async Task<T> FromResult<T>(T value, int delayInMilliseconds = DefaultDelay)
{
if (this.TaskShouldBeDelayed())
{
await Task.Delay(delayInMilliseconds);
}
return value;
}
}
I have a process that goes through a loop. During each iteration of the loop, it calls out to an external web service and then adds a object to an EntityFramework repository. The call to the external service is wrapped in a static method. Typically the loop only has one or two iterations but up to 4 is currently possible with the UI. (Each iteration represents an insurance quote).
It seems that this would benefit from being refactored as an asynchronous process. How do I set this up so that each iteration occurs in a seperate thread, and the commit waits until all threads are completed?
public class ProcessRequest
{
private IUnitOfWork = unitOfWork;
public ProcessRequest(IUnitOfWork uow)
{
unitOfWork = uow;
}
public void Execute(MyRequestParams p)
{
foreach (Quote q in p.Quotes)
{
q.Premium = QuoteService.GetQuote(q);
unitOfWork.GetRepository<Quote>().Add(q);
}
unitOfWork.Commit();
}
}
public static class QuoteService
{
public static decimal GetQuote(Quote quote)
{
//I've simplified proprietary code to single line that calls an external service
return ExternalWebService.GetQuote(quote.Deductible);
}
}
You're asking two different things: one is how to execute the loop in parallel, where each iteration occurs (potentially) on a separate thread; this is completely different to executing the entire loop as an asynchronous process which means the thread that initiates it won't wait for it to complete. I assume you meant the first, i.e. that you want to parallelize the iterations in the loop but still block until all of them are done.
Without knowing anything about the context in which this runs, one straightforward way would be to use Parallel Extensions, specifically Parallel Foreach:
public void Execute(MyRequestParams p)
{
Parallel.ForEach(p.Quotes, q => {
q.Premium = QuoteService.GetQuote(q);
unitOfWork.GetRepository<Quote>().Add(q);
});
unitOfWork.Commit();
}
Or maybe something like:
public void Execute(MyRequestParams p)
{
Parallel.ForEach(p.Quotes, q => {
q.Premium = QuoteService.GetQuote(q);
});
unitOfWork.GetRepository<Quote>().AddAll(p.Quotes);
unitOfWork.Commit();
}
This depends heavily on the thread-safety what you're dealing with.
If most of your work is I/O, im not sure i'd go for spinning up a new thread, as you are wasting most of your time idle waiting for your service/DB to reply.
i'd try to go with a async approach:
public async Task Execute(MyRequestParams p)
{
foreach (var quote in p.Quotes)
{
//Of course, you'll need an async endpoint.
var q.Premium = await QuoteService.GetQuoteAsync(q);
}
unitOfWork.GetRepository<Quote>().AddAll(p.Quotes);
await unitOfWork.SaveChangesAsync();
}
With this approach, you save the overhead of spinning up new threads and letting them be idle most of the time.
Hope this makes sense, of course you'd have to have access to an async endpoint of the webservice, and use Entity Framework v6.
I'm trying to create an async unit test for the project, but cannot understand how to wait for the async subject to complete:
[Test]
public async void MicroTest()
{
var value = 2;
var first = new AsyncSubject<int>();
var second = new AsyncSubject<int>();
first.Subscribe(_ =>
{
value = _;
second.OnCompleted();
});
first.OnNext(1);
// how to wait for the second subject to complete?
Assert.AreEqual(value, 1);
}
Sync version of this test is works well:
[Test]
public void MicroTest()
{
var value = 2;
var first = new Subject<int>();
var second = new Subject<int>();
first.Subscribe(_ =>
{
value = _;
second.OnCompleted();
});
first.OnNext(1);
Assert.AreEqual(value, 1);
}
AsyncSubject versus Subject
First off, it's worth pointing out that AsyncSubject<T> is not an asynchronous version of Subject<T>. Both are in fact free-threaded* (see footnote).
AsyncSubject is a specialization of Subject intended to be used to model an operation that completes asynchronously and returns a single result. It has two noteworthy features:
Only the last result is published
The result is cached and is available to observers subscribing after it has completed.
It is used internally in various places, including by the ToObservable() extension method defined on Task and Task<T>.
The issue with the test
Recall AsyncSubject<T> will only return the final result received. It does this by waiting for OnCompleted() so it knows what the final result is. Because you do not call OnCompleted() on first your test is flawed as the OnNext() handler - the lambda function passed in your Subscribe call - will never be invoked.
Additionally, it is invalid not to call OnNext() at least once on an AsyncSubject<T>, so when you call await second; you will get an InvalidOperationException if you haven't done this.
If you write your test as follows, all is well:
[Test]
public async void MicroTest()
{
var value = 2;
var first = new AsyncSubject<int>();
var second = new AsyncSubject<int>();
first.Subscribe(_ =>
{
// won't be called until an OnCompleted() has
// been invoked on first
value = _;
// you must send *some* value to second
second.OnNext(_);
second.OnCompleted();
});
first.OnNext(1);
// you must do this for OnNext handler to be called
first.OnCompleted();
// how to wait for the second subject to complete
await second;
Assert.AreEqual(value, 1);
}
About asynchronous tests
As a general rule I would avoid writing asynchronous tests that could wait forever. This gets particularly annoying when it causes resource drains on build servers. Use some kind of timeout e.g:
await second.Timeout(TimeSpan.FromSeconds(1));
No need to handle the exception since that is enough for the test to fail.
**I've borrowed this term from the COM lexicon. In this sense I mean that they, as with most of the Rx framework components, will generally run on whatever thread you happen to invoke their methods on. Being free-threaded doesn't necessarily mean being fully thread safe though. In particular, unlike AsyncSubject<T>, Subject<T> doesn't protect you from the Rx grammar violation of making overlapping calls to OnNext. Use Subject.Synchronize or Observable.Synchronize for this protection.*
I have a method that looks like this:
protected void OnBarcodeScan(BarcodeScannerEventArgs e)
{
// We need to do this on a seperate thread so we don't block the main thread.
ThreadStart starter = () => SendScanMessage(e, _scanDelegates);
Thread scanThread = new Thread(starter);
scanThread.Start();
}
Then the thread goes off and does some logic (and ends up calling a delegate in my test).
My problem is that my unit test finishes before the thread does. So my test fails.
I can just add in System.Threading.Thread.Sleep(1000); and hope that the logic never takes more than a second (it should not). But that seems like a hack.
The problem is that I don't want to expose that thread to the outside world or even to the rest of the class.
Is there some cool way to find that thread again and wait for it in my unit test?
Something like this:
[TestMethod]
[HostType("Moles")]
public void AddDelegateToScanner_ScanHappens_ScanDelegateIsCalled()
{
// Arrange
bool scanCalled = false;
MCoreDLL.GetTopWindow = () => (new IntPtr(FauxHandle));
// Act
_scanner.AddDelegateToScanner(_formIdentity, ((evnt) => { scanCalled = true; }));
_scanner.SendScan(new BarcodeScannerEventArgs("12345678910"));
// This line is fake!
System.Threading.Thread.CoolMethodToFindMyThread().Join();
// Assert
Assert.IsTrue(scanCalled);
}
I obviously made up the CoolMethodToFindMyThread method. But is there some why to do that?
So if I understand how this works, then the delegates you register are the ones being called on the second thread, right? In that case, you can use thread synchronization in your test and the delegate that gets called. I do this kind of thing in my unit tests all the time.
Something like this:
[TestMethod]
[HostType("Moles")]
public void AddDelegateToScanner_ScanHappens_ScanDelegateIsCalled()
{
// Arrange
var scanCalledEvent = new ManualResetEvent(false);
MCoreDLL.GetTopWindow = () => (new IntPtr(FauxHandle));
// Act
_scanner.AddDelegateToScanner(_formIdentity, ((evnt) => { scanCalledEvent.Set(); }));
_scanner.SendScan(new BarcodeScannerEventArgs("12345678910"));
// Wait for event to fire
bool scanCalledInTime = scanCalledEvent.WaitOne(SOME_TIMEOUT_IN_MILLISECONDS);
// Assert
Assert.IsTrue(scanCalledInTime);
}
It's important to have some sort of timeout in there, otherwise if something goes wrong your test just locks up and that's kind of hard to debug. WaitOne will block until the event gets set or the timeout expires, the return value tells you which happened.
(WARNING: I may have the return value backwards - I don't remember off the top of my head if true means the event got set or if true means the timeout expired. Check the docs.)
There are several sync primitives you can use here, which one depends on what you want to do. ManualResetEvent usually works pretty well for me.
There's another way of doing things:
First, have an AutoResetEvent (or ManualResetEvent, if you feel like) in your test class.
In your test method:
//set up stuff
testEvent.WaitOne();
//ensure everything works
In your callback
testEvent.Set();
Your test method will then stop until the callback gets called.
Presumably you'll want some sort of timeout on that wait call as well.