Develop Reference

How correctly add my tests' errors to a stack so the tests don't stop executing?

I use SeleniumWebdriver C# to build automated tests.
I also use NUnit to mark test methods (so I can run them).
In each of the tests there are several verification and when the first verification fails then the test stops executing (an exception is thrown).
I want the test to continue executing so more errors could be found!
Please guys, give me a clue how to do it right.
I think about something like that:
Instead of throwing Exception in verification I'll add an error to a stack
When a test ends I check if my stack is not empty
If the stack is not empty I push to console all errors and fail the test
If the stack is empty then the test passed successfully.
Are these steps good? Is there a better way?

I think this should work for you.
class Asserts
{
private static StringBuilder _stack;
[SetUp]
public void SetUp()
{
_stack = new StringBuilder();
}
[TearDown]
public void TearDown()
{
if (_stack.Lenght != 0) Assert.Fail(_stack.ToString());
}
[Test]
public void Test()
{
AssertHelper(() => Assert.AreEqual(0, 0));
AssertHelper(() => Assert.IsNotNull(null));
AssertHelper(() => Assert.AreEqual(3, 4));
AssertHelper(() => Assert.AreEqual(1, 1));
}
private static void AssertHelper(Action assert)
{
try
{
assert();
}
catch (AssertionException e)
{
_stack.Append(e.Message);
}
}
}
But the good idea is to keep one verification per one test.

Generally you want your tests to only be verifying one thing at a time. If there's 2 steps to an action that require asserting, considering having tests like:
UserForm_FirstNameMissing_ThrowsException
UserForm_SecondNameMissing_ThrowsException
UserForm_AgeTooLarge_ThrowsException
instead of having a single test called something like:
UserForm_TestValidation
This means that - like you wanted - if one stage of validation fails, you can continue to test the rest. Your best bet for this may be a singular test with a dataset, however keeping the answer simple yet relevant, a test per failure state is good enough for what you want.

I solved my problem in the next way:
I've created a set of methods that are performed once prior to executing any of the tests ([TestFixtureSetUpAttribute] in NUnit);
In this set of methods I do the main actions results of which I'll check (in my case I buy a lottery ticket);
In each test ([Test] in NUnit) I check one assertion.
So now I have a main action that doesn't include any verification. After this actions is completed I check all needed assertions.

Unit Test Garbage Collection

Question
Do unit tests automatically dispose resources via garbage collection (System.IO.Stream in my case) once the test(s) have completed, or are things left open/in-use, requiring the disposing of IDisposable objects?
Context / Information
I'm currently making a unit test for a file uploader which uses a System.IO.Stream.
I've Moq'd out the HttpPostedFileBase with the InputStream of the file powered by a System.IO.MemoryStream, which all works as expected.
I currently have (altered for brevity):
[TestMethod]
public void TestUpload()
{
var stream = FunctionCreatingTheMemoryStream();
try
{
var file = new Mock<HttpPostedFileBase>();
file.Setup(f => f.FileName).Returns("test.txt");
file.Setup(f => f.InputStream).Returns(stream);
MethodThatUsesTheStream(file.Object)
// rest of test code with Assert
}
finally
{
stream.Dispose();
}
}
The question is with the MemoryStream instance that is created:
var stream = new FunctionCreatingTheMemoryStream();
Is it worthwhile placing any code after this in a try catch and then disposing of the stream in the finally statement, or with it being a unit test, will the memory stream be disposed of automatically?
So is it necessary to do this, or could it simply just be:
[TestMethod]
public void TestUpload()
{
var stream = FunctionCreatingTheMemoryStream();
var file = new Mock<HttpPostedFileBase>();
file.Setup(f => f.FileName).Returns("test.txt");
file.Setup(f => f.InputStream).Returns(stream);
MethodThatUsesTheStream(file.Object)
// rest of test code with Assert
}

The answer ultimately depends on the unit testing framework you use, but in .NET, none of the three major test frameworks (MSTest, NUnit, xUnit.net) automatically dispose of things. You have to manually ask them to do so.
You could argue that when executing a test suite, the test runner in principle launches a new process, runs all the tests, and then the process exits.
For some implementations of IDisposable, like MemoryStream, that simply means that the memory is reclaimed when the process exits. However, you can't always rely on that, because some disposable types may access out-of-process resources. You could theoretically have objects holding on to memory-mapped-files, named pipes, SQL Server connections, etc. Even if the test process exits, you may leave behind such resources. They'll probably time out sooner or later (like SQL Server connections returning to the pool), but it may slow down your system.
Furthermore, some test runners are attempting very hard to be clever these days, so they reuse one or more processes to be able to run faster, changing your test suite in and out of AppDomains.
So, in the end, unless you have something like a MemoryStream, where you're absolutely certain that it's not a big deal to leave it behind, you should deterministically dispose of your objects in your tests.
However, if you're doing Test-Driven Development, you should adopt the GOOS attitude of listening to your tests. If you write a lot of tests that involve IDisposable objects, you should consider if you can simplify your SUT's API. It depends on what you're doing, but if you write mostly managed code, you shouldn't need IDisposable much, because it's a Leaky Abstraction that leaks that the SUT depends on unmanaged resources..

In the past I have written a "Dustcart" class that implements IDisposable, and contains a collection of objects to dispose. These are disposed in the opersite order to how they are added (use a stack to implement it).
Then the test code looks like.
[Setup]
Public void Setup()
{
_dustcart = new Dustcart()
}
[TearDown]
public void TearDown ()
{
_dustcart.Dispose();
}
[TestMethod]
public void TestUpload()
{
var stream = _dustcart.DisposeOnTearDown(FunctionCreatingTheMemoryStream());
var file = new Mock<HttpPostedFileBase>();
file.Setup(f => f.FileName).Returns("test.txt");
file.Setup(f => f.InputStream).Returns(stream);
MethodThatUsesTheStream(file.Object)
// rest of test code with Assert
}
DisposeOnTearDown() is a generic method. You could put this this in a super class that all your tests inherent from, that also includes object mothers for the classes you need to mock etc.
However if you are not very careful your test code becomes harder to understand with no real benefit to the quality of the software you are testing. The tests are there to do a job and they have to be no better than is needed to do that job.

Totally agree with everyone that you need to dispose such objects. You can simplify your code a bit by using "using" statements, like this:
[TestMethod]
public void TestUpload()
{
using (var stream = FunctionCreatingTheMemoryStream())
{
var file = new Mock<HttpPostedFileBase>();
file.Setup(f => f.FileName).Returns("test.txt");
file.Setup(f => f.InputStream).Returns(stream);
MethodThatUsesTheStream(file.Object)
// rest of test code with Assert
}
}

How to unit test methods that use threads?

How to write unit tests for methods that use threads.
In example below how to test someMethod method ?
public class SomeClass {
private final SomeOtherClass someOtherClassInstance = IoC.getInstance(SomeOtherClass.class);
private final ExecutorService executorService = Executors.newCachedThreadPool();
public void someMethod() {
executorService.execute(new Runnable() {
#Override
public void run() {
someOtherClassInstance.someOtherMethod();
}
});
}
}
Are there any solutions in java and .net for this purpose?

Your question begs 'what is a unit test' (UT). UTs are just one form of automated tests. In this case your question implies that the code calls the OS to start a thread, so by definition, a test that tested it would not be a unit test but probably be an integration test.
Why I bother you with what seems like semantics is that understanding the intent of the test makes it so easy to write the test and structure the code. The amount of working code out there is larger than the amount of testable code out there.
So, how can this code be changed to be unit testable (I will skip the 'why bother stuff') ...
A unit test in C# tests a single type (your class) ... importantly (by definition) nothing else. So your code needs to reflect this. What you want to test is that when I call ABC it does this stuff. Something that stuff includes launching a thread. So you want to test that the method to launch a thread a called. The fundamental here is that your application requires a thread so that is what your asserting.
So how? Create a proxy, and perhaps a factory, for the thread creation. Then you can assert in a unit test that it was called and how it was treated. Sounds hard, but is really easy when you get into the habit.
BTW, Resharper makes creating a proxy for an OS types (e.g. thread). Check out the delegating members stuff in their help. If your not using Resharper you should be.
Yea I know this is not the answer you had hopped for, but believe me it is the answer you need.
I find it useful to think of tests in the categories:
Unit tests
Integration tests (or perhaps subsystem tests)
Functional test (e.g. UI/application/UAT tests)

The jMock Team has described an brilliant approach for Java in the 'jMock Cookbook: Test Multithreaded Code'. The concept of executing "thread" functionality on the same thread as the test can also be applied to C# code (see this blog post). The C# test would look like this:
[Test]
public void TestMultiThreadingCodeSynchronously()
{
// Arrange
SomeOtherClass someOtherClassMock = MockRepository.GenerateMock<SomeOtherClass>();
DeterministicTaskScheduler taskScheduler = new DeterministicTaskScheduler();
SomeClass systemUnderTest = new SomeClass(taskScheduler, someOtherClassMock);
// Act
systemUnderTest.SomeMethod();
// Now execute the new task on
// the current thread.
taskScheduler.RunTasksUntilIdle();
// Assert
someOtherClassMock.AssertWasCalled(x=>x.SomeOtherMethod());
}
Your "system under test" would look like this:
public class SomeClass
{
private readonly TaskScheduler taskScheduler;
private readonly SomeOtherClass instance;
public SomeClass(
TaskScheduler taskScheduler,
SomeOtherClass instance)
{
this.taskScheduler = taskScheduler;
this.instance = instance;
}
public void SomeMethod()
{
Task.Factory.StartNew(
instance.SomeOtherMethod,
new CancellationToken(),
TaskCreationOptions.None,
taskScheduler);
}
}
The C# solution is described in detail in this blog post.

If there is another thread, you have wait until the method has been called. You have to decide how long you should wait before considering the method wasn't called, there is no tool which can do that for you.
Of course you can mock the ExecutorService to run in the current thread and avoid the issue.

Preventing catch of NUnit AssertionException?

I'm working on a project at the moment where I need to inter-operate with code that swallows exceptions. In particular, I'm writing NUnit unit tests. There are some places where I want to embed assertions within code that gets passed as a delegate, as part of mocking a particular behavior. The problem I'm having is that the AssertionException gets swallowed by the code calling the delegate, which means the test passes, even though the test Assert failed.
Is there any way to inform NUnit that a test should fail that can't be circumvented by catching AssertionException? I can't modify the code that swallows the exceptions, as I don't have full ownership and it's already in semi-production use. I'm hoping there's a clean way to accomplish this.
The best I've come up with is something like this:
private static string _assertionFailure;
public static void AssertWrapper(Action action)
{
try
{
action();
}
catch (AssertionException ex)
{
_assertionFailure = ex.Message;
throw;
}
}
[Test]
[ExpectedException(typeof(AssertionException))]
public void TestDefeatSwallowing()
{
Action failure = () => AssertWrapper(() => Assert.Fail("This is a failure"));
EvilSwallowingMethod(failure);
if (_assertionFailure != null)
Assert.Fail(_assertionFailure);
}
private void EvilSwallowingMethod(Action action)
{
try
{
action();
}
catch
{
}
}
It works, but it's pretty ugly. I have to wrap every Assert call and I have to check at the end of every test if an assertion was swallowed.

So you're doing something like this? (this is using Moq syntax)
var dependency1 = new Mock<IDependency1>();
dependency1.Setup(d => d.CalledMethod([Args])
.Callback(TestOutArgsAndPossiblyThrow);
var objectUnderTest = new TestedObject(dependency1.Object);
objectUnderTest.MethodThatCallsIDependency1dotCalledMethod();
And you've got TestOutArgsAndPossiblyThrow encapsulated in your AssertWrapper class?
Unless that's way off, I'd say you're doing it just about right. You have execution re-entering your test at a point where you can record the state of the call to the dependency. Whether that's done via catching exceptions and analyzing them or just directly inspecting the values of the method parameters, you've just gotta do the work. And if you're swallowing exceptions inside the black box, you're going to have to monitor them before they get back into the black box.
I still say you'd be much better off with appropriate logging and notification (you don't have to notify the end users, necessarily). To #TrueWill's point - what do you do when there's an IOException or the database isn't available?
DISCUSSION EDIT
Is your scenario structured like this?
TEST -> TESTED CODE -> SWALLOWING CODE -> THROWING MOCK

In the teardown event during an NUnit test how can I get to the attribute applied to the method that was just tested?

I have a test method that is run. When the method generates an exception I want to know what the name of the test was and the exception content.
In the teardown for the test I want to get access to this information. How would I get access to it from the [TearDown] attributed method?

You can access text context objects in test tear down method
[TearDown]
public void TestTearDown()
{
// inc. class name
var fullNameOfTheMethod = NUnit.Framework.TestContext.CurrentContext.Test.FullName;
// method name only
var methodName = NUnit.Framework.TestContext.CurrentContext.Test.Name;
// the state of the test execution
var state = NUnit.Framework.TestContext.CurrentContext.Result.State; // TestState enum
}
I don't know which version was first to support it, but mine is 24.

I don't think there's a good way built in to nunit, but it's not a hard problem to resolve. Just wrap your tests in a try/catch block, catch any exceptions, and save them (and the test name) to a private member variable in your test class. Then you've got access from your TearDown method.
Not particularly elegant, but it works.

Another solution would be to use a template method and run all tests using this method. For example:
// template method
void Execute(Action test)
{
try
{
test();
}
catch (Exception e)
{
// handle exception here
throw;
}
}
[Test]
public void Test()
{
Execute(() =>
{
// your test here
});
}
This pattern is particularly useful when your test uses some resources that must be initialized before test and disposed after test (e.g. temporary file). In that case, you can use a type parameter in test delegate.
Another advantage is that you can easily let the test run on different thread, using different culture etc.
Disadvantage is clear: it forces you to use lambda method in every test.

OPTION 1: I don't think you can. Or rather, I don't know that you can. How I approach this need is to use a try/catch on the specific tests, do what I want with the exception and then throw again within the catch block so that the test could fail.
try{
// do something that can potentially throw;
}
catch(Exception ex){
// do something interesting with the ex;
throw;
}
OPTION 2: If you've not gone too far along, you may want to use xUnit which has a different exception expectation model and may provide some of the control you are looking for.