I'm working on a project that has some unit test that fails randomly when executed on Team City. And nobody can reproduce the same behavior on local machines.
As almost all tests executes the tested method inside a TestDelegate action and executes the Asserts and Verifies outside the actions I belive it could be a concurrency problem.
But before I do any change on test code I'd like to have more information about the execution of TestDelegate actions in .Net Framework 4.0.
This is a example of a unit test that fails randomly.
[TestFixture]
public class MyClassTest
{
private Mock<IAnyService> _anyService;
private MyClass _myClass;
[SetUp]
public void Setup()
{
_anyService = new Mock<IAnyService>();
_myClass.AnyService = new MyClass() { AnyService = _anyService.Object };
}
[Test]
public void MyClass_Should_Call_MockClassMethod()
{
TestDelegate action = () => _myClass.MyMethod();
Assert.DoesNotThrow(action);
_anyService.Verify(_ => _.MockClassMethod(), Times.Once);
}
}
My class method
public override void MyMethod()
{
...DoALotOfStuff
AnyService.MockClassMethod();
}
This is the error on Team City
Expected invocation on the mock once, but was 0 times: _ => _.MockClassMethod()
No setups configured.
No invocations performed.
What could be the reason of it fails randomly?
Is possible that the action is executed in separate thread only on Team City?
EDIT:
I was able to do this test fail randomly using this command to execute the test. But still don't know why.
nunit3-console (ls -r ./tests/**/bin/**/*.Tests.dll -exclude *.Product* | % FullName | sort-object -Unique) --teamcity --x86 --framework=v4.0
Are you running your tests with any of NUnit's Parallelization options enabled? I don't see any evidence of parallelization in your example code, but it could be enabled elsewhere--for example, there could be a [Parallelizable] attribute at the assembly level in AssemlyInfo.cs.
If the tests are running in parallel, it's possible that you're leaking state between tests. NUnit re-uses a single instance of a test fixture for all test methods within that fixture (source). Since your mocks are are fixture-scoped, there could be a race condition between the setup and assertion logic of two tests running concurrently.
The TestDelegate itself doesn't look concerning to me. Calling a delegate isn't much different from calling any other method, except that you're doing it indirectly through a variable. In the example code you provided _myClass.MyMethod() will still run synchronously--unless you've introduced threading or concurrency elsewhere.
Edit:
Additional information provided from the question author in a comment, added here for visibility at their request:
After looking into the documentantion [https://github.com/nunit/docs/wiki/Console-Command-Line] I realized that nunit-console has a parameter --agents that is used to control running the assemblies in parallel and if not specified, all agent processes run tests at the same time, whatever the number of assemblies. When I added --agents=1 on the command line all tests run successfuly.
Related
In an XUnit test project, I wish to skip an theory that must be run with DotMemory Unit when the test is not run under DotMemory Unit.
Currently I am using dotMemoryApi to force the theory to fail, as per the following snippet:
[Theory]
[MemberData(nameof(SomeTestData)]
public void MyTheory(object someData)
{
if (!dotMemoryApi.IsEnabled)
{
Assert.True(false, "DotMemory API not enabled");
}
// otherwise, proceed with dotMemory unit test calls
}
This works, but I would prefer that the theory be skipped when not ran with DotMemory Unit, rather than fail.
I've tried the SkippableFact nuget package, but it fails when running under DotMemory Unit with the following exception:
DotMemoryUnitException : dotMemory Unit methods were called from outside a >test method:
- If you work with a unit test runner that is not supported out of the box, >then probably calls to dotMemory Unit were made from outside the >DotMemoryUnitController TestStart() and TestEnd() methods or these methods >were called in the wrong order. Learn more about the DotMemoryUnitController >class: https://www.jetbrains.com/help/dotmemory-unit/3.0/Working_with_Unsupported_Unit_Testing_Frameworks.html
I've also tried extending the XUnit.TheoryAttribute class, as below, which does cause the theory to be skipped, but unfortunately it skips the test even when run with dotMemoryUnit.
public sealed class IgnoreOnDotMemoryNotEnabledTheory : TheoryAttribute
{
public IgnoreOnDotMemoryNotEnabledTheory() {
if(!dotMemoryApi.IsEnabled)
{
Skip = "Ignored due to DotMemory API not enabled"
}
}
}
Any thoughts?
In this situation when I have "memory only" tests which do not make any sense w/o dotMemory Unit I mark them with a category (e.g. [Category("MemoryOnly")] and exclude the category when run all tests.
I use NUnit but sure that there is no much difference.
It turns out that there is a SkippableTheory attribute included in the SkippableFact NuGet library. I must have missed it because it wasn't in the project Readme.
Example of usage:
[SkippableTheory]
[MemberData(nameof(SomeTestData)]
public void MyTheory(object someData)
{
skip.IfNot(dotMemoryApi.IsEnabled);
// otherwise, proceed with dotMemory unit test calls
.
.
.
I'm new to Nunit and am trying to run 2 Test Fixtures, A & B. Within each Fixture I have a unique Setup method for each. However, when I click "Run All" in the "Test Explorer" in Visual Studio, the test setup for Fixture A is called (it was executed first) and Setup for Fixture B is ignored. I also get the same behavior when running all tests via command line. Below is my code:
Fixture A
[TestFixture]
public class A
{
[SetUp]
public void SetupTest()
{
// ...Setup for Fixture A
}
[Test, Order(1)]
public void TestForFixtureA()
{
// ...perform test
}
}
Fixture B
[TestFixture]
public class B
{
[SetUp]
public void SetupTest()
{
// ...Setup for Fixture B
}
[Test]
public void TestForFixtureB()
{
// ...perform test
}
}
What is the correct way to get Setup methods to execute per Fixture?
You are using the incorrect attribute for setup at the test fixture level. The attribute you should be using is [SetUpFixture]. Information about this can be found in the nunit documentation
Here is a list of all the setup attributes and their usages taken from the documentation:
SetUpAttribute is now used exclusively for per-test setup.
TearDownAttribute is now used exclusively for per-test teardown.
OneTimeSetUpAttribute is used for one-time setup per test-run. If you
run n tests, this event will only occur once.
OneTimeTearDownAttribute is used for one-time teardown per test-run.
If you run n tests, this event will only occur once
SetUpFixtureAttribute continues to be used as at before, but with
changed method attributes.
This doesn't seem to explain the bizzare behaviour you are seeing, as setup should be ran per-test, but using the correct attributes couldn't hurt.
If you intend your setup to be run once per fixture, use [OneTimeSetUp]. But if you intend it to run once per test within the fixture, then [SetUp] is correct. We can't tell what you intend from the code.
Whichever one you use, the setups should all run. The only situation in which [OneTimeSetUp] will run but [SetUp] will not is when no individual tests are found within the fixture. Are all the tests being recognized?
I suggest you verify very clearly that the setup is not being run. The easiest way is to temporarily create some output from the setup method.
I'm struggle with this one quite a long time now. Some background: I created my automated test framework using Selenium. With one project being pure NUnit tests and second one that does all the work for the tests. Now, in test project I keep the directories to all environments I run my tests against. So far many of my tests were read-only and didn't bother much about if tests did not run on environment that they supposed to. This has changed as I started to run some 'write' tests.
I need to prevent this 'Write' tests to run on any other environment then localhost. So far I tried to use method attributes and getting test method names on run time and do work then but this is not quite efficient. Can you guys point me a good solution? Thanks!
I would tag the tests to exclude with a particular category name then define a SetUp function which will stop the tests from running if they are tagged with that name and if you are on a particular environment such as Production. Place the SetUp function in a BaseClass and have all your test fixtures inherit it. The SetUp function will run before every test and prevent it from running if it needs to.
Something like this:
public class BaseSetup
{
protected const string CategoryToExclude = "Write";
[SetUp]
public void Init()
{
string env = ConfigurationManager.GetEnvironment();
if ( env == Constants.Environments.PROD && (TestContext.CurrentContext.Test.Properties["Categories"].Contains(CategoryToExclude)))
{
Assert.Inconclusive(String.Format("Cannot run this test on environment: {0}", env));
}
}
}
[TestFixture]
public class UnitTests : BaseSetup
{
[Test]
[Category(CategoryToExclude)]
public void TestMethod1()
{
Console.WriteLine("TestMethod1");
}
[Test]
public void TestMethod2()
{
Console.WriteLine("TestMethod2");
}
}
Hope this helps!
NUnit have category attribute.
You can group tests by categories and run only wanted categories.
I'm running ncrunch, in a new MVC 4 solution in VS2012 using nunit and ninject.
When I first open the solution all 50 or so test run and pass successfully.
After I make any code change (even just a added empty space) ncrunch reports that most of my unit test fail. The same thing happens if I press the 'run all tests' in the ncrunch window.
But if you hit the 'Run all tests visible here' button all 50 test pass again and ncrunch reports everything is fine.
Also when you run each test individually they pass every time.
When they do fail they seem to be failing in my ninject setup code
Error: TestFixtureSetUp failed in ControllerTestSetup
public class ControllerTestSetup
{
[SetUp]
public void InitIntegrationTest()
{
var context = IntegrationTestContext.Instance;
context.Init();
context.NinjectKernel.Load<MediGapWebTestModule>();
}
[TearDown]
public void DisposeIntegrationTest()
{
IntegrationTestContext.Instance.Dispose();
}
}
public class IntegrationTestContext : IDisposable
{
private static IntegrationTestContext _instance = null;
private static readonly object _monitor = new object();
private IntegrationTestContext() { }
public static IntegrationTestContext Instance
{
get
{
if (_instance == null)
{
lock (_monitor)
{
if (_instance == null)
{
_instance = new IntegrationTestContext();
}
}
}
return _instance;
}
}
}
All the test also run in the resharper test runner without problems every time.
Does anyone know what could be causing this?
I guessing its something to do with the singleton lock code inside the Instance property but I'm not sure.
==============================================================================
Progress:
I was able to track this down to a error in the ninject setup method above by wrapping it in a try catch statement, and writing the error to the output window.
The exception was caused by trying to load a module more than once, even tho I definitely haven't and I don't use any type of automatic module loading.
This happen on the lines
LocalSessionFactoryModule.SetMappingAssemblies(() => new[] { typeof(ProviderMap).Assembly });
_kernel.Load<LocalSessionFactoryModule>();
_sessionFactory = _kernel.Get<ISessionFactory>();
where LocalSessionFactoryModule is the ninject module class derived for NinjectModule class.
Why is this only happening with ncrunch and what can I do to solve this issue? Is there a way to check if a module has already been loaded?
NCrunch will never execute tests concurrency within the same process, so unless you have multi-threaded behaviour inside your test logic, then it should be safe to say that this isn't an issue caused by the locking or threading over the singleton.
As you've already tried disabling parallel execution and this hasn't made a difference, I'm assuming that the problem wouldn't be caused by concurrent use of resources outside the test runner process (i.e. files on disk).
This means that the problem is almost certainly related to the sequence in which the tests are being executed. Almost all manual test runners (including Resharper) will run tests in a defined sequence from start to finish. This is good for consistency, but it can mask problems that may surface when the tests are run in an inconsistent/random order. NCrunch will execute tests in order of priority and can also reuse test processes between test runs, which can make the runtime behaviour of your tests different if they haven't been designed with this in mind.
A useful way to surface (and thus debug) sequence related issues is to try running your tests in a manually defined order by using NCrunch. If you right-click a test inside the NCrunch Tests Window, under the 'Advanced' menu you'll find an option to run a test using an existing task runner process. Try this action against several of your tests to see if you can reproduce the sequence that surfaces the problem. When it happens, you should easily be able to get a debugger onto the test and find out why it is failing.
Most sequence related problems are caused by uncleared static members, so make sure each of your tests is written in the assumption that existing state may be left behind by another test that has been run within the process. Another option is to ensure all state is fully cleared by tests on tear down (although in my opinion, this is often a less pragmatic approach).
Using VS 2010, NUnit, Typemock, Entity Framework...
I am having a bit of an odd problem using NUnit/Typemock. I am relative new to the testing world so it could be a beginners mistake. This is the problem I am having.
Test Project is compiled.
Launch NUnit from within Visual Studio
Run tests from within NUnit client app. First run, first test always fails.
Rerun and test passes.
It does not matter what test is first. I can select a particular test. If it is the first one to run, it fails on first execution. It passes on rerun.
This is an example class having the problem. But the class doesn't matter. Whatever test is run first has this problem. The exception code was just something that was there to test Typemock being loaded. It fails on the WhenCalled.
[TestClass, Isolated]
public class FirstTest
{
[TestMethod]
public void TestMe()
{
Exception e = new TypeMock.ArrangeActAssert.NestedCallException();
Isolate.WhenCalled(() => UnitOfWorkManager.GetCurrentSession(null)).WillReturn(null);
Assert.IsTrue(true);
}
}
The following is the error message.
HCSO.ESL.Test.Fakes.FirstTest.TestMe:
TypeMock.ArrangeActAssert.NestedCallException :
* WhenCalled does not support using a property call as an argument.
- To fix this pass false instead of AssemblyReader.IsDotNetFile
Example - this would work:
MyObj argument = Something.Other().GetStuff();
Isolate.WhenCalled(() => ObjUnderTest.MethodUnderTest(argument))...;
Example - this would not work:
Isolate.WhenCalled(() => ObjUnderTest.MethodUnderTest(Something.Other().GetStuff()))...;
(End error message)
Anyone have an idea why the first test always fails but runs fine on rerun? Something with how the assemblies are being loaded?
(Edit)Additional Details:
Versions:
Typemock Isolator: 6.0.10.0
Visual Studio: 10.0.30319.1
In addition, I added simplified test code. This way you can see the code being tested. And yes, this test fails first time, passes on every run after that.
[TestClass, Isolated]
public class FirstTest
{
public static int DummyCall(int i)
{
return 0;
}
[TestMethod]
public void TestMe()
{
Exception e = new TypeMock.ArrangeActAssert.NestedCallException();
//Isolate.WhenCalled(() => UnitOfWorkManager.GetCurrentSession(null)).WillReturn(null);
Isolate.WhenCalled(() => FirstTest.DummyCall(-1)).WillReturn(1);
Assert.IsTrue(true);
}
}
I work at Typemock,
It seems very strange, as this is definitely not a nested call from the looks of it.
Could you please try and email us a small solution demonstrating the problem to support#typemock.com?
What is UnitOfWorkManager? Is this a class belonging to EF, or is it your code?
Also, what version of Isolator are you using?
I resolved the issue. As I expected, it was partially a newbie mistake. Inside of NUnit there is a setting to determine how the assembly is isolated. The default option is to run the tests in the same process as NUnit. I tried changing the isolation in a seperate process per assembly and the problem goes away.
To reproduce error.
* Make sure NUnit option for "Run tests directly in NUnit process" is selected.
* Close NUnit (just to make sure setting is used)
* Launch NUnit from within VS.
* Select a test containing Isolate.WhenCalled()
* Run that test first.
Thanks for the help.
[EDIT: Update]
Updating this in the event someone else has this issue.
I found that in the NUnit client if I set the following options everything works great.
Under Settings:
Test Loader -> Assembly Isolation -> Default Process Model -> Run test directly in the NUnit process.
Test Loader -> Assembly Isolation -> Default Domain Usage -> Use a seperate AppDomain per Assembly