Develop Reference

C# Unit test that fails randomly

I'm working on a project that has some unit test that fails randomly when executed on Team City. And nobody can reproduce the same behavior on local machines.
As almost all tests executes the tested method inside a TestDelegate action and executes the Asserts and Verifies outside the actions I belive it could be a concurrency problem.
But before I do any change on test code I'd like to have more information about the execution of TestDelegate actions in .Net Framework 4.0.
This is a example of a unit test that fails randomly.
[TestFixture]
public class MyClassTest
{
private Mock<IAnyService> _anyService;
private MyClass _myClass;
[SetUp]
public void Setup()
{
_anyService = new Mock<IAnyService>();
_myClass.AnyService = new MyClass() { AnyService = _anyService.Object };
}
[Test]
public void MyClass_Should_Call_MockClassMethod()
{
TestDelegate action = () => _myClass.MyMethod();
Assert.DoesNotThrow(action);
_anyService.Verify(_ => _.MockClassMethod(), Times.Once);
}
}
My class method
public override void MyMethod()
{
...DoALotOfStuff
AnyService.MockClassMethod();
}
This is the error on Team City
Expected invocation on the mock once, but was 0 times: _ => _.MockClassMethod()
No setups configured.
No invocations performed.
What could be the reason of it fails randomly?
Is possible that the action is executed in separate thread only on Team City?
EDIT:
I was able to do this test fail randomly using this command to execute the test. But still don't know why.
nunit3-console (ls -r ./tests/**/bin/**/*.Tests.dll -exclude *.Product* | % FullName | sort-object -Unique) --teamcity --x86 --framework=v4.0

Are you running your tests with any of NUnit's Parallelization options enabled? I don't see any evidence of parallelization in your example code, but it could be enabled elsewhere--for example, there could be a [Parallelizable] attribute at the assembly level in AssemlyInfo.cs.
If the tests are running in parallel, it's possible that you're leaking state between tests. NUnit re-uses a single instance of a test fixture for all test methods within that fixture (source). Since your mocks are are fixture-scoped, there could be a race condition between the setup and assertion logic of two tests running concurrently.
The TestDelegate itself doesn't look concerning to me. Calling a delegate isn't much different from calling any other method, except that you're doing it indirectly through a variable. In the example code you provided _myClass.MyMethod() will still run synchronously--unless you've introduced threading or concurrency elsewhere.
Edit:
Additional information provided from the question author in a comment, added here for visibility at their request:
After looking into the documentantion [https://github.com/nunit/docs/wiki/Console-Command-Line] I realized that nunit-console has a parameter --agents that is used to control running the assemblies in parallel and if not specified, all agent processes run tests at the same time, whatever the number of assemblies. When I added --agents=1 on the command line all tests run successfuly.

Nunit not calling setup once per text fixture

I'm new to Nunit and am trying to run 2 Test Fixtures, A & B. Within each Fixture I have a unique Setup method for each. However, when I click "Run All" in the "Test Explorer" in Visual Studio, the test setup for Fixture A is called (it was executed first) and Setup for Fixture B is ignored. I also get the same behavior when running all tests via command line. Below is my code:
Fixture A
[TestFixture]
public class A
{
[SetUp]
public void SetupTest()
{
// ...Setup for Fixture A
}
[Test, Order(1)]
public void TestForFixtureA()
{
// ...perform test
}
}
Fixture B
[TestFixture]
public class B
{
[SetUp]
public void SetupTest()
{
// ...Setup for Fixture B
}
[Test]
public void TestForFixtureB()
{
// ...perform test
}
}
What is the correct way to get Setup methods to execute per Fixture?

You are using the incorrect attribute for setup at the test fixture level. The attribute you should be using is [SetUpFixture]. Information about this can be found in the nunit documentation
Here is a list of all the setup attributes and their usages taken from the documentation:
SetUpAttribute is now used exclusively for per-test setup.
TearDownAttribute is now used exclusively for per-test teardown.
OneTimeSetUpAttribute is used for one-time setup per test-run. If you
run n tests, this event will only occur once.
OneTimeTearDownAttribute is used for one-time teardown per test-run.
If you run n tests, this event will only occur once
SetUpFixtureAttribute continues to be used as at before, but with
changed method attributes.
This doesn't seem to explain the bizzare behaviour you are seeing, as setup should be ran per-test, but using the correct attributes couldn't hurt.

If you intend your setup to be run once per fixture, use [OneTimeSetUp]. But if you intend it to run once per test within the fixture, then [SetUp] is correct. We can't tell what you intend from the code.
Whichever one you use, the setups should all run. The only situation in which [OneTimeSetUp] will run but [SetUp] will not is when no individual tests are found within the fixture. Are all the tests being recognized?
I suggest you verify very clearly that the setup is not being run. The easiest way is to temporarily create some output from the setup method.

Prevent NUnit tests to run depending on environment

I'm struggle with this one quite a long time now. Some background: I created my automated test framework using Selenium. With one project being pure NUnit tests and second one that does all the work for the tests. Now, in test project I keep the directories to all environments I run my tests against. So far many of my tests were read-only and didn't bother much about if tests did not run on environment that they supposed to. This has changed as I started to run some 'write' tests.
I need to prevent this 'Write' tests to run on any other environment then localhost. So far I tried to use method attributes and getting test method names on run time and do work then but this is not quite efficient. Can you guys point me a good solution? Thanks!

I would tag the tests to exclude with a particular category name then define a SetUp function which will stop the tests from running if they are tagged with that name and if you are on a particular environment such as Production. Place the SetUp function in a BaseClass and have all your test fixtures inherit it. The SetUp function will run before every test and prevent it from running if it needs to.
Something like this:
public class BaseSetup
{
protected const string CategoryToExclude = "Write";
[SetUp]
public void Init()
{
string env = ConfigurationManager.GetEnvironment();
if ( env == Constants.Environments.PROD && (TestContext.CurrentContext.Test.Properties["Categories"].Contains(CategoryToExclude)))
{
Assert.Inconclusive(String.Format("Cannot run this test on environment: {0}", env));
}
}
}
[TestFixture]
public class UnitTests : BaseSetup
{
[Test]
[Category(CategoryToExclude)]
public void TestMethod1()
{
Console.WriteLine("TestMethod1");
}
[Test]
public void TestMethod2()
{
Console.WriteLine("TestMethod2");
}
}
Hope this helps!

NUnit have category attribute.
You can group tests by categories and run only wanted categories.

NUnit test cases not run from inherited class

I have a base test class containing some test cases and some ordinary tests:
[TestFixture]
public abstract class TestBase
{
[TestCase(1)]
[TestCase(2)]
[TestCase(3)]
public void TestA(int value)
{
// Perform test
}
[Test]
public void TestB()
{
// Perform test
}
}
These tests are run from child classes which set up the environment in different ways. The child classes only contains setup methods, no tests.
[TestFixture]
public class LocalDatabaseTest : TestBase
{
[SetUp]
public void SetUp()
{
// Set up environment to use local db
}
}
I'm using ReSharper 6.1.1000.82 to run all tests in LocalDatabaseTest, but only the ordinary tests are run. The tests using TestCase does not get any result. If I select Run All on TestA in the TestBase class, all test cases are run (including the other child classes). I'm using NUnit 2.6.2.12296. Any ideas on what I've done wrong?

You've done nothing wrong.
If you open your test dll via NUnit test runner you will see all test are running successfully.
(I just verified your code with NUnit 2.6.2).
Regarding the reason of ignoring parameterized tests on Resharper: It seems there is some issue with Resharper test runner which causes such behavior.
So, my suggestion is to use NUnit to run parameterized tests.
Btw, Resharper 7 has better support NUnit parameterized tests. And probably this issue won't appear in the latest Resharper version.

Multiple [SetupTest] for different configs

Is it possible to have multiple [SetupTest]'s in a fixture?
I am using Selenium and nUnit and would like to be able to specify the Browser on which the user wants to test.
I have a simple user GUI for selecting tests to run but, I am aware that in the future we want to hook this up to cruise control to run the tests automatically. Ideally I want tests that can be run on both our GUI and the NUnit GUI.

Is it possible to have multiple [SetupTest]'s in a fixture? No.
It is possible to define all your tests in a base class, have multiple fixtures inherit the tests, and then select an environment dependent fixture type at runtime.
Here’s the stock example that I have for [TestFixtureSetup]. The same principle works for all the setup attributes. Notice that I’m only putting [TestFixture] on the child classes. Since the base “TestClass” doesn’t have complete setup code, you don’t want to run the tests directly.
public class TestClass
{
public virtual void TestFixtureSetUp()
{
// environment independent code...
}
[Test]
public void Test1() { Console.WriteLine("Test1 pass."); }
// More Environment independent tests...
}
[TestFixture]
public class BrowserFixture : TestClass
{
[TestFixtureSetUp]
public override void TestFixtureSetUp()
{
base.TestFixtureSetUp();
// environment dependent code...
}
}
[TestFixture]
public class GUIFixture : TestClass
{
[TestFixtureSetUp]
public override void TestFixtureSetUp()
{
base.TestFixtureSetUp();
// environment dependent code...
}
}

I suspect you can use the parameterized tests introduced in NUnit 2.5 to do what you want, but I'm not totally clear what you want to do here. However, you could define the fixture and have it take a Browser variable in its constructor, then use parameterized TestFixture attributes like
TextFixture["Firefox"]
TestFixture["Chrome"]
public class ParameterizedTestFixture {
//Constructor
public ParameterizedTestFixture( string Browser) {
//set fixture variables relating to browser treatment
}
//rest of class
}
See NUnit Documentation for more info.
The Setup attribute identifies a method that is run before each test. It only makes sense to have one Setup per test fixture - think of it as a 'reset' or 'preparation' before each test is run.