Develop Reference

C# Unit test that fails randomly

I'm working on a project that has some unit test that fails randomly when executed on Team City. And nobody can reproduce the same behavior on local machines.
As almost all tests executes the tested method inside a TestDelegate action and executes the Asserts and Verifies outside the actions I belive it could be a concurrency problem.
But before I do any change on test code I'd like to have more information about the execution of TestDelegate actions in .Net Framework 4.0.
This is a example of a unit test that fails randomly.
[TestFixture]
public class MyClassTest
{
private Mock<IAnyService> _anyService;
private MyClass _myClass;
[SetUp]
public void Setup()
{
_anyService = new Mock<IAnyService>();
_myClass.AnyService = new MyClass() { AnyService = _anyService.Object };
}
[Test]
public void MyClass_Should_Call_MockClassMethod()
{
TestDelegate action = () => _myClass.MyMethod();
Assert.DoesNotThrow(action);
_anyService.Verify(_ => _.MockClassMethod(), Times.Once);
}
}
My class method
public override void MyMethod()
{
...DoALotOfStuff
AnyService.MockClassMethod();
}
This is the error on Team City
Expected invocation on the mock once, but was 0 times: _ => _.MockClassMethod()
No setups configured.
No invocations performed.
What could be the reason of it fails randomly?
Is possible that the action is executed in separate thread only on Team City?
EDIT:
I was able to do this test fail randomly using this command to execute the test. But still don't know why.
nunit3-console (ls -r ./tests/**/bin/**/*.Tests.dll -exclude *.Product* | % FullName | sort-object -Unique) --teamcity --x86 --framework=v4.0

Are you running your tests with any of NUnit's Parallelization options enabled? I don't see any evidence of parallelization in your example code, but it could be enabled elsewhere--for example, there could be a [Parallelizable] attribute at the assembly level in AssemlyInfo.cs.
If the tests are running in parallel, it's possible that you're leaking state between tests. NUnit re-uses a single instance of a test fixture for all test methods within that fixture (source). Since your mocks are are fixture-scoped, there could be a race condition between the setup and assertion logic of two tests running concurrently.
The TestDelegate itself doesn't look concerning to me. Calling a delegate isn't much different from calling any other method, except that you're doing it indirectly through a variable. In the example code you provided _myClass.MyMethod() will still run synchronously--unless you've introduced threading or concurrency elsewhere.
Edit:
Additional information provided from the question author in a comment, added here for visibility at their request:
After looking into the documentantion [https://github.com/nunit/docs/wiki/Console-Command-Line] I realized that nunit-console has a parameter --agents that is used to control running the assemblies in parallel and if not specified, all agent processes run tests at the same time, whatever the number of assemblies. When I added --agents=1 on the command line all tests run successfuly.

Rider NUnit Test problem: Program does not contain main

I can't start my NUnit testing on Rider (JetBrains). I have a Console Application project named ISDI and I'm tryin to Test it with a NUnit testing project named ISDITest in the same solution.
This is my code:
using System;
using ISDI;
using NUnit.Framework;
namespace ISDITest {
[TestFixture]
public class TestNome
{
[Test]
public void TestRoom()
{
IRoom r = new Room(0);
IEntity p = new Player();
r.InsertEntity(p);
Assert.Equals(r.GetEntities().Count, 1);
Assert.True(r.GetEntities().Contains(p));
}
}
}
When I try to run the test I get a build error:
Program does not contain a static 'Main' method suitable for an entry point
I think testing methods in a testing class would not need a Main and I don't know how to solve this as I already specified that this is a Testing project when I created it.
I'm sorry if this is a stupid question but I'm just getting started with C# and testing.

Solved this putting an empty Main in the project I wanted to test. Still, it does not make any sense to me.

When running a program, you need an entry point - a place for the code to start. Usually, Main is used for this, but when you have NUnit, you can use a [Test] as an entry point.
When you want to run tests, you need to use the [Test] flag as the point of entry for the program. You do not need a Main method for this.
I recommend reading the Rider / Unit Testing documentation for more information on how to run your [Test] code without implementing a Main method.
https://www.jetbrains.com/help/rider/Unit_Testing__Index.html

How can I skip an xunit theory if DotMemory Unit is not available?

In an XUnit test project, I wish to skip an theory that must be run with DotMemory Unit when the test is not run under DotMemory Unit.
Currently I am using dotMemoryApi to force the theory to fail, as per the following snippet:
[Theory]
[MemberData(nameof(SomeTestData)]
public void MyTheory(object someData)
{
if (!dotMemoryApi.IsEnabled)
{
Assert.True(false, "DotMemory API not enabled");
}
// otherwise, proceed with dotMemory unit test calls
}
This works, but I would prefer that the theory be skipped when not ran with DotMemory Unit, rather than fail.
I've tried the SkippableFact nuget package, but it fails when running under DotMemory Unit with the following exception:
DotMemoryUnitException : dotMemory Unit methods were called from outside a >test method:
- If you work with a unit test runner that is not supported out of the box, >then probably calls to dotMemory Unit were made from outside the >DotMemoryUnitController TestStart() and TestEnd() methods or these methods >were called in the wrong order. Learn more about the DotMemoryUnitController >class: https://www.jetbrains.com/help/dotmemory-unit/3.0/Working_with_Unsupported_Unit_Testing_Frameworks.html
I've also tried extending the XUnit.TheoryAttribute class, as below, which does cause the theory to be skipped, but unfortunately it skips the test even when run with dotMemoryUnit.
public sealed class IgnoreOnDotMemoryNotEnabledTheory : TheoryAttribute
{
public IgnoreOnDotMemoryNotEnabledTheory() {
if(!dotMemoryApi.IsEnabled)
{
Skip = "Ignored due to DotMemory API not enabled"
}
}
}
Any thoughts?

In this situation when I have "memory only" tests which do not make any sense w/o dotMemory Unit I mark them with a category (e.g. [Category("MemoryOnly")] and exclude the category when run all tests.
I use NUnit but sure that there is no much difference.

It turns out that there is a SkippableTheory attribute included in the SkippableFact NuGet library. I must have missed it because it wasn't in the project Readme.
Example of usage:
[SkippableTheory]
[MemberData(nameof(SomeTestData)]
public void MyTheory(object someData)
{
skip.IfNot(dotMemoryApi.IsEnabled);
// otherwise, proceed with dotMemory unit test calls
.
.
.

NUnit 3 Assert.That throws NullReferenceException when values are not null

This is driving me crazy! I am using NUnit version 3.6.0 and I am getting unexpected results from my tests. I have created a really simple test to demonstrate the problem I have:
[TestFixture]
public class NunitTest
{
[Test]
public void TestIt()
{
string x = "x";
string y = "y";
// this fails (expected) but with NullReferenceException (unexpected)
Assert.That(x, Is.EqualTo(y));
}
}
When the test runs I get a NullReferenceException when I am expecting something like "expected is 'X', actual is 'Y'"
I have furthered the asserts and they all pass
[TestFixture]
public class NunitTest
{
[Test]
public void TestIt()
{
string x = "x";
string y = "y";
// this passes
bool atest = x.Equals(y);
Assert.IsFalse(atest);
// this passes
Assert.IsNotNull(x);
// this passes
Assert.IsNotNull(y);
// this fails (expected) but with NullReferenceException (unexpected)
Assert.That(x, Is.EqualTo(y));
}
}
The other weird thing is that when the values are equal the test passed correctly. Also worth noting I get exactly the same problem when I use Assert.AreEqual(x, y)
Stack Trace:
at NUnit.Framework.Assert.ReportFailure(String message)
at NUnit.Framework.Assert.ReportFailure(ConstraintResult result, String message, Object[] args)
at NUnit.Framework.Assert.That[TActual](TActual actual, IResolveConstraint expression, String message, Object[] args)
at NUnit.Framework.Assert.That[TActual](TActual actual, IResolveConstraint expression)
at VSI.Tests.NunitTest.TestIt() in c:\Users\DavidBecker\Documents\Visual Studio 2013\Projects\VSI.Tests\VSI.Tests\NunitTest.cs:line 30
Tools:
.NET Framework version 4.5.1
Visual Studio 2013 Professional
Resharper Version 9
EDIT: I am running the tests from Resharper.

You're right that ReSharper 9 is the problem.
ReSharper didn't support NUnit 3 until ReSharper version 10. There are however are a number of similarities between NUnit versions 2 and 3, which mean that some older NUnit 2 runners will appear to run NUnit 3 tests - however, this won't be done accurately, and is neither supported or advised.
In the final version of NUnit 2 (2.6.4) - an exception was put in, such that a runner would error if attempting to load NUnit 3 tests would fail immediately, instead of attempting to run tests in an unsupported manner. It seems likely however that ReSharper 9 is using a runner older than NUnit 2.6.4, which doesn't throw this error.
The reason this has only become apparent to you with NUnit 3.6 is due to an internal change with how assertions are registered within NUnit. The coincidental previous success now no longer succeeds - although in many ways, this is an advantage, as it gets rid of this right-side failure, where tests appear to be passing, but aren't actually running correctly.
ReSharper supports NUnit 3 from version 10, although that was the point I believe when they switched from a one-off purchase model to a subscription model. As another option - consider the NUnit 3 Test Adapter (NuGet package or VSIX extension) - which allows you to run NUnit tests individually, using Visual Studio's own test window.
(If your interested in further info, this issue came up recently over on GitHub also: https://github.com/nunit/nunit/issues/1992)

Running NUnit.3.6.0 tests in Resharper (9) Unit Test Runner has unexpected results. Use NUnit.ConsoleRunner.3.6.0 instead.
(this worked for me, if anyone has better way or can make it work in RS9 then please let me know).

Ncrunch all test pass first run, but fails after code change and when run all button is pressed

I'm running ncrunch, in a new MVC 4 solution in VS2012 using nunit and ninject.
When I first open the solution all 50 or so test run and pass successfully.
After I make any code change (even just a added empty space) ncrunch reports that most of my unit test fail. The same thing happens if I press the 'run all tests' in the ncrunch window.
But if you hit the 'Run all tests visible here' button all 50 test pass again and ncrunch reports everything is fine.
Also when you run each test individually they pass every time.
When they do fail they seem to be failing in my ninject setup code
Error: TestFixtureSetUp failed in ControllerTestSetup
public class ControllerTestSetup
{
[SetUp]
public void InitIntegrationTest()
{
var context = IntegrationTestContext.Instance;
context.Init();
context.NinjectKernel.Load<MediGapWebTestModule>();
}
[TearDown]
public void DisposeIntegrationTest()
{
IntegrationTestContext.Instance.Dispose();
}
}
public class IntegrationTestContext : IDisposable
{
private static IntegrationTestContext _instance = null;
private static readonly object _monitor = new object();
private IntegrationTestContext() { }
public static IntegrationTestContext Instance
{
get
{
if (_instance == null)
{
lock (_monitor)
{
if (_instance == null)
{
_instance = new IntegrationTestContext();
}
}
}
return _instance;
}
}
}
All the test also run in the resharper test runner without problems every time.
Does anyone know what could be causing this?
I guessing its something to do with the singleton lock code inside the Instance property but I'm not sure.
==============================================================================
Progress:
I was able to track this down to a error in the ninject setup method above by wrapping it in a try catch statement, and writing the error to the output window.
The exception was caused by trying to load a module more than once, even tho I definitely haven't and I don't use any type of automatic module loading.
This happen on the lines
LocalSessionFactoryModule.SetMappingAssemblies(() => new[] { typeof(ProviderMap).Assembly });
_kernel.Load<LocalSessionFactoryModule>();
_sessionFactory = _kernel.Get<ISessionFactory>();
where LocalSessionFactoryModule is the ninject module class derived for NinjectModule class.
Why is this only happening with ncrunch and what can I do to solve this issue? Is there a way to check if a module has already been loaded?

NCrunch will never execute tests concurrency within the same process, so unless you have multi-threaded behaviour inside your test logic, then it should be safe to say that this isn't an issue caused by the locking or threading over the singleton.
As you've already tried disabling parallel execution and this hasn't made a difference, I'm assuming that the problem wouldn't be caused by concurrent use of resources outside the test runner process (i.e. files on disk).
This means that the problem is almost certainly related to the sequence in which the tests are being executed. Almost all manual test runners (including Resharper) will run tests in a defined sequence from start to finish. This is good for consistency, but it can mask problems that may surface when the tests are run in an inconsistent/random order. NCrunch will execute tests in order of priority and can also reuse test processes between test runs, which can make the runtime behaviour of your tests different if they haven't been designed with this in mind.
A useful way to surface (and thus debug) sequence related issues is to try running your tests in a manually defined order by using NCrunch. If you right-click a test inside the NCrunch Tests Window, under the 'Advanced' menu you'll find an option to run a test using an existing task runner process. Try this action against several of your tests to see if you can reproduce the sequence that surfaces the problem. When it happens, you should easily be able to get a debugger onto the test and find out why it is failing.
Most sequence related problems are caused by uncleared static members, so make sure each of your tests is written in the assumption that existing state may be left behind by another test that has been run within the process. Another option is to ensure all state is fully cleared by tests on tear down (although in my opinion, this is often a less pragmatic approach).