Develop Reference

Integration test only works with a Thread.Sleep

We currently have an integration test that tests a front-end api call. The call is async therefore we force it until the call is ready to compare the result with what we expect.
However, the problem is that the code contains also an another async call which is not awaited. So we have nested async calls. This also has the cause that the test sometimes succeeds and sometimes it fails (race condition).
For now we have fixed the issue by added a Thread.Sleep (10000) to the code. However, this is not a good solution.
The code likes like this:
[TestMethod]
public void Integration_Test_Example()
{
// Arrange
...
// Act
var request = CreateRequest(#"testfile.xml");
var task = target.SendMessage(request);
Thread.Sleep(10000); // dirty fix
task.Wait();
// Assert
...
}
Does anyone have advice on how we can best solve this?

If I understand correctly SendMessage returns task, so you should be able to await it before moving on with your test.
[TestMethod]
public async Task Integration_Test_Example()
{
var request = CreateRequest(#"testfile.xml");
var result = await target.SendMessage(request);
}

You can change the test to be awaitable
[TestMethod]
public async Task Integration_Test_Example()
{
// Arrange
...
// Act
var request = CreateRequest(#"testfile.xml");
await target.SendMessage(request);
// Assert
...
}

Partial code coverage of await [duplicate]

When I analyse code coverage in Visual Studio 2012, any of the await lines in async methods are showing as not covered even though they are obviously executing since my tests are passing. The code coverage report says that the uncovered method is MoveNext, which is not present in my code (perhaps it's compiler-generated).
Is there a way to fix code coverage reporting for async methods?
Note:
I just ran coverage using NCover, and the coverage numbers make a lot more sense using that tool. As a workaround for now, I'll be switching to that.

This can happen most commonly if the operation you're awaiting is completed before it's awaited.
I recommend you test at least synchronous and asynchronous success situations, but it's also a good idea to test synchronous and asynchronous errors and cancellations.

The reason the code is not shown as being covered has to do with how async methods are implemented. The C# compiler actually translates the code in async methods into a class that implements a state machine, and transforms the original method into a stub that initialized and invokes that state machine. Since this code is generated in your assembly, it is included in the code coverage analysis.
If you use a task that is not complete at the time the code being covered is executing, the compiler-generated state machine hooks up a completion callback to resume when the task completes. This more completely exercises the state machine code, and results in complete code coverage (at least for statement-level code coverage tools).
A common way to get a task that is not complete at the moment, but will complete at some point is to use Task.Delay in your unit test. However, that is generally a poor option because the time delay is either too small (and results in unpredictable code coverage because sometimes the task is complete before the code being tests runs) or too large (unnecessarily slowing the tests down).
A better option is to use "await Task.Yield()". This will return immediately but invoke the continuation as soon as it is set.
Another option - though somewhat absurd - is to implement your own awaitable pattern that has the semantics of reporting incomplete until a continuation callback is hooked up, and then to immediately complete. This basically forces the state machine into the async path, providing the complete coverage.
To be sure, this is not a perfect solution. The most unfortunate aspect is that it requires modification to production code to address a limitation of a tool. I would much prefer that the code coverage tool ignore the portions of the async state machine that are generated by the compiler. But until that happens, there aren’t many options if you really want to try to get complete code coverage.
A more complete explanation of this hack can be found here: http://blogs.msdn.com/b/dwayneneed/archive/2014/11/17/code-coverage-with-async-await.aspx

There are situations where I don't care about testing the async nature of a method but just want to get rid of the partial code coverage. I use below extension method to avoid this and it works just fine for me.
Warning "Thread.Sleep" used here!
public static IReturnsResult<TClass> ReturnsAsyncDelayed<TClass, TResponse>(this ISetup<TClass, Task<TResponse>> setup, TResponse value) where TClass : class
{
var completionSource = new TaskCompletionSource<TResponse>();
Task.Run(() => { Thread.Sleep(200); completionSource.SetResult(value); });
return setup.Returns(completionSource.Task);
}
and the usage is similar to the Moq's ReturnsAsync Setup.
_sampleMock.Setup(s => s.SampleMethodAsync()).ReturnsAsyncDelayed(response);

I created a test runner that runs a block of code multiple times and varies the task that is delayed using a factory. This is great for testing the different paths through simple blocks of code. For more complex paths you may want to create a test per path.
[TestMethod]
public async Task ShouldTestAsync()
{
await AsyncTestRunner.RunTest(async taskFactory =>
{
this.apiRestClient.GetAsync<List<Item1>>(NullString).ReturnsForAnyArgs(taskFactory.Result(new List<Item1>()));
this.apiRestClient.GetAsync<List<Item2>>(NullString).ReturnsForAnyArgs(taskFactory.Result(new List<Item2>()));
var items = await this.apiController.GetAsync();
this.apiRestClient.Received().GetAsync<List<Item1>>(Url1).IgnoreAwait();
this.apiRestClient.Received().GetAsync<List<Item2>>(Url2).IgnoreAwait();
Assert.AreEqual(0, items.Count(), "Zero items should be returned.");
});
}
public static class AsyncTestRunner
{
public static async Task RunTest(Func<ITestTaskFactory, Task> test)
{
var testTaskFactory = new TestTaskFactory();
while (testTaskFactory.NextTestRun())
{
await test(testTaskFactory);
}
}
}
public class TestTaskFactory : ITestTaskFactory
{
public TestTaskFactory()
{
this.firstRun = true;
this.totalTasks = 0;
this.currentTestRun = -1; // Start at -1 so it will go to 0 for first run.
this.currentTaskNumber = 0;
}
public bool NextTestRun()
{
// Use final task number as total tasks.
this.totalTasks = this.currentTaskNumber;
// Always return has next as turn for for first run, and when we have not yet delayed all tasks.
// We need one more test run that tasks for if they all run sync.
var hasNext = this.firstRun || this.currentTestRun <= this.totalTasks;
// Go to next run so we know what task should be delayed,
// and then reset the current task number so we start over.
this.currentTestRun++;
this.currentTaskNumber = 0;
this.firstRun = false;
return hasNext;
}
public async Task<T> Result<T>(T value, int delayInMilliseconds = DefaultDelay)
{
if (this.TaskShouldBeDelayed())
{
await Task.Delay(delayInMilliseconds);
}
return value;
}
private bool TaskShouldBeDelayed()
{
var result = this.currentTaskNumber == this.currentTestRun - 1;
this.currentTaskNumber++;
return result;
}
public async Task VoidResult(int delayInMilliseconds = DefaultDelay)
{
// If the task number we are on matches the test run,
// make it delayed so we can cycle through them.
// Otherwise this task will be complete when it is reached.
if (this.TaskShouldBeDelayed())
{
await Task.Delay(delayInMilliseconds);
}
}
public async Task<T> FromResult<T>(T value, int delayInMilliseconds = DefaultDelay)
{
if (this.TaskShouldBeDelayed())
{
await Task.Delay(delayInMilliseconds);
}
return value;
}
}

How to use a breakpoint after await in unit test?

I am surprised my breakpoint after awaiting an async method that is on a line that references the awaited Task<T> is never hit:
[Test]
public async void GetSomethingTest()
{
var service = SimpleIoc.Default.GetInstance<IService>();
var result = await service.TryGetSomethingAsync(20);
Assert.IsTrue(result.IsSuccess);
Assert.IsNotNull(result.ReturnValue);
}
Putting a breakpoint on the first Assert line is never hit but the test passes.
How to I break when await returns?
UPDATE: I guess it is because the test framework is not awaiting the invocation of the test method, I am using NUnit 2.6.3 and that claims async support, however whether that entails breaking after an await like I am trying to do, I am not sure...

The problem is that your method is async void. That has fire-and-forget semantics.
Conceptually what your method is doing with the async-await usage looks like this:
[Test]
public void GetSomethingTest()
{
var service = SimpleIoc.Default.GetInstance<IService>();
service.TryGetSomethingAsync(20).ContinueWith(t =>
{
var result = t.Result;
Assert.IsTrue(result.IsSuccess);
Assert.IsNotNull(result.ReturnValue);
});
}
Now it should be clear what the problem is. Your test method immediately returns as soon as TryGetSomethingAsync returns its Task. So the test immediately finishes. Since no exceptions were thrown, it is a success.
If your test framework supports Task-returning tests, you can fix your test to do what you want by simply changing its return type to Task instead of void.
[Test]
public async Task GetSomethingTest()
{
var service = SimpleIoc.Default.GetInstance<IService>();
var result = await service.TryGetSomethingAsync(20);
Assert.IsTrue(result.IsSuccess);
Assert.IsNotNull(result.ReturnValue);
}
This will conceptually translate to the following.
[Test]
public Task GetSomethingTest()
{
var service = SimpleIoc.Default.GetInstance<IService>();
return service.TryGetSomethingAsync(20).ContinueWith(t =>
{
var result = t.Result;
Assert.IsTrue(result.IsSuccess);
Assert.IsNotNull(result.ReturnValue);
});
}
Notice how the Task continuation is returned, so that the test framework can now wait on it, ensuring that all the test's code has time to run before the test is considered finished.
(Technically a framework could be made to work in the async void case as well, but I don't know of any reason why that would be a good feature, so I expect most don't handle it.)
If your test framework does not support Task-returning tests, you can fix your test by using .Result instead of await.
[Test]
public void GetSomethingTest()
{
var service = SimpleIoc.Default.GetInstance<IService>();
var result = service.TryGetSomethingAsync(20).Result;
Assert.IsTrue(result.IsSuccess);
Assert.IsNotNull(result.ReturnValue);
}
This will simply block the current thread until the Task returned by TryGetSomethingAsync is completed.

Unexpected behavior when passing async Actions around

I'm pretty familiar with the async/await pattern, but I'm bumping into some behavior that strikes me as odd. I'm sure there's a perfectly valid reason why it's happening, and I'd love to understand the behavior.
The background here is that I'm developing a Windows Store app, and since I'm a cautious, conscientious developer, I'm unit testing everything. I discovered pretty quickly that the ExpectedExceptionAttribute doesn't exist for WSAs. Weird, right? Well, no problem! I can more-or-less replicate the behavior with an extension method! So I wrote this:
public static class TestHelpers
{
// There's no ExpectedExceptionAttribute for Windows Store apps! Why must Microsoft make my life so hard?!
public static void AssertThrowsExpectedException<T>(this Action a) where T : Exception
{
try
{
a();
}
catch (T)
{
return;
}
Assert.Fail("The expected exception was not thrown");
}
}
And lo, it works beautifully.
So I continued happily writing my unit tests, until I hit an async method that I wanted to confirm throws an exception under certain circumstances. "No problem," I thought to myself, "I can just pass in an async lambda!"
So I wrote this test method:
[TestMethod]
public async Task Network_Interface_Being_Unavailable_Throws_Exception()
{
var webManager = new FakeWebManager
{
IsNetworkAvailable = false
};
var am = new AuthenticationManager(webManager);
Action authenticate = async () => await am.Authenticate("foo", "bar");
authenticate.AssertThrowsExpectedException<LoginFailedException>();
}
This, surprisingly, throws a runtime error. It actually crashes the test-runner!
I made an overload of my AssertThrowsExpectedException method:
public static async Task AssertThrowsExpectedException<TException>(this Func<Task> a) where TException : Exception
{
try
{
await a();
}
catch (TException)
{
return;
}
Assert.Fail("The expected exception was not thrown");
}
and I tweaked my test:
[TestMethod]
public async Task Network_Interface_Being_Unavailable_Throws_Exception()
{
var webManager = new FakeWebManager
{
IsNetworkAvailable = false
};
var am = new AuthenticationManager(webManager);
Func<Task> authenticate = async () => await am.Authenticate("foo", "bar");
await authenticate.AssertThrowsExpectedException<LoginFailedException>();
}
I'm fine with my solution, I'm just wondering exactly why everything goes pear-shaped when I try to invoke the async Action. I'm guessing because, as far as the runtime is concerned, it's not an Action, I'm just cramming the lambda into it. I know the lambda will happily be assigned to either Action or Func<Task>.

It is not surprising that it may crash the tester, in your second code fragment scenario:
Action authenticate = async () => await am.Authenticate("foo", "bar");
authenticate.AssertThrowsExpectedException<LoginFailedException>();
It's actually a fire-and-forget invocation of an async void method, when you call the action:
try
{
a();
}
The a() returns instantly, and so does the AssertThrowsExpectedException method. At the same time, some activity started inside am.Authenticate may continue executing in the background, possibly on a pool thread. What's exactly going on there depends on the implementation of am.Authenticate, but it may crash your tester later, when such async operation is completed and it throws LoginFailedException. I'm not sure what is the synchronization context of the unit test execution environment, but if it uses the default SynchronizationContext, the exception may indeed be thrown unobserved on a different thread in this case.
VS2012 automatically supports asynchronous unit tests, as long as the test method signatures are async Task. So, I think you've answered your own question by using await and Func<T> for your test.

Thread.Sleep in Task.Run

I have been trying to get my head around C#'s new async/await and Task.Run functionality recently. In doing so I wrote some simple test code that writes to the console so that I can see in what order things happen in, throwing in a Thread.Sleep here and there to make sure things really happens in the order I expected.
Here is one of my tests
[Test]
public void TaskRun()
{
Console.WriteLine("Before");
Task.Run(() => Console.WriteLine(_terminator.IWillBeBack()));
Console.WriteLine("After");
}
with the following Terminator class:
public class Terminator
{
public string IWillBeBack()
{
return "I will be back";
}
}
From this I expected the result to most likely be Before, After, and then I will be back. This is indeed what happens. Then I make a change to the IWillBeBack-method to allow it to sleep for 3 seconds. The code for this:
public class Terminator
{
public string IWillBeBack()
{
Thread.Sleep(3000);
return "I will be back";
}
}
I expected the result to still be Before, After, and then after 3 seconds again I will be back. This is not what happens though. I instead get Before and After, but never I will be back.
Why is this?
When I debug it clearly goes through the code, sleeping and returning. I don't need Thread.Sleep in this way in any production code, but I would like to have a vague understanding of why I will be back is not written to the console in the second case.

Your test is completing before it tries to print "I will be back". You're not doing anything to wait until that's finished. Depending on the test runner, however it's redirecting Console.WriteLine may have been disconnected as soon as the test has finished.
If you want to see it finish, change your test to:
[Test]
public void TaskRun()
{
Console.WriteLine("Before");
var task = Task.Run(() => Console.WriteLine(_terminator.IWillBeBack()));
Console.WriteLine("After");
task.Wait();
}

If you use the MS Test framework in VS 2012 you can also write the test like this
[TestMethod]
public async Task TaskRun()
{
Console.WriteLine("Before");
Task t = Task.Run(() => Console.WriteLine(IWillBeBack()));
Console.WriteLine("After");
await t;
}
private string IWillBeBack()
{
Thread.Sleep(3000);
return "I will be back";
}