Develop Reference

xunit skip tests if other tests failed

The problem: I have a bunch of xunit tests that connect to an external server. If that server is up - they fly through very quickly. If that server is down, everything goes into timeouts and retries and the tests all take a long time. The thing is, if I have 100 tests - if the first one fails, I still want all the rest to fail, but it's completely pointless running them... What I'd like is to be able to make my own fact attribute something like:
[FactInFailureGroup( "ConectsToBlah", typeof(ConnectionException)]
public void TestOne()
{
...
}
[FactInFailureGroup( "ConnectsToBlah", typeof(ConnectionException)]
public void TestTwo()
{
...
}
I've looked at before/after attributes, but they don't seem to be able to kill the test or see the result - and I've looked at creating my own fact attribute, which looks like it can prevent the test from running, but not put in a trap for the result. I really want to somehow make code along these lines:
class FactInFailureGroupAttribute : FactAttribute
{
private static _failedGroups = new HashSet<String>();
...
void BeforeTest( TestManipulator manipulator )
{
if (_failedGroups.contains( _thisGroup ))
manipulator.SkipWithMessage( f"all tests in group {_thisGroup} failed because of {_exceptionType}");
}
void AfterTest( TestResult test )
{
if (test.Failed && test.Exception.GetType() == _exceptionType)
_failedGroups.add( _thisGroup );
}
}

You can abuse constructors and statics to accomplish your task.
The constructor of your test class will get called at the beginning of every test but your static should exist for your entire test run.
Try something like this:
private static object Lock = new object();
private static bool? IsServerUp;
public FactAssertionTests()
{
lock (Lock)
{
if (!IsServerUp.HasValue)
{
// check server connection and set IsServerUp
}
else if (!IsServerUp.Value)
{
throw new Exception("Failed to connect to server");
}
}
}
That way whatever test gets into the Lock first will check the server's up-ness. If it fails, every test coming in next will throw the exception.
This isn't quite a decorator attribute and requires you to put all the tests into one class, but it's simple.
Edit:
I tested the following approach, it relies on the fact that xUnit runs all tests in a single class in serial. If you've got tests spanning multiple classes this won't work:
public class ConnectionTests
{
[Fact]
public void Test1()
{
RunServerTest(() =>
{
var svc = new ServiceConnection();
svc.Connect();
});
}
[Fact]
public void Test2()
{
RunServerTest(() =>
{
var svc = new ServiceConnection();
svc.Connect();
});
}
private static bool? ServerIsDown;
public void RunServerTest(Action testAction)
{
var exception = new Exception("Server is down, test will not run.");
if (ServerIsDown.GetValueOrDefault())
{
throw exception;
}
try
{
testAction.Invoke();
}
catch (ServerMissingException ex)
{
ServerIsDown = true;
throw;
}
}
}

You can accomplish what you want by calling your first test explicitly in the constructor of the class. If it fails due to server missing it should throw an exception. This will prevent the class from being instantiated, and none of the other tests in the class will run. They will be marked as failed in the test explorer.
public class SO74607887Tests
{
public SO74607887Tests()
{
TestOne();
}
[Fact]
public void TestOne()
{
// Arrange
// Act
// Oops server is down
throw new Exception("Server is down");
}
[Fact]
public void TestTwo()
{
// Will not run
}
}
If your tests are distributed over a lot of different classes, you can follow #JeremyLakeman 's suggestion to use a fixture. Proceed as follows:
using System;
using Xunit;
namespace SO74607887_XunitCancelTestsIfOneFails
{
// This class will never actually be instantiated, it is only present to provide information to Xunit.
[CollectionDefinition("SO74607887")]
public class SO74607887CollectionDefinition : ICollectionFixture<SO74607887Base>
{
}
// This class creates a single object injected into the constructors of all the other classes in the collection.
public class SO74607887Base : IDisposable
{
public bool serverOK;
public SO74607887Base()
{
// Check server, if it is missing, set the flag
serverOK = false;
}
public void Dispose()
{
// Clean up
}
}
[Collection("SO74607887")]
public class SO74607887Tests
{
public SO74607887Tests(SO74607887Base basis)
{
if (!basis.serverOK)
{
throw new Exception($"Server is down, tests in {nameof(SO74607887Tests)} will not run.");
}
}
[Fact]
public void TestOne()
{
// Arrange
// Act
// Assert
}
[Fact]
public void TestTwo()
{
// Arrange
// Act
// Assert
}
}
}
The check on the server is only done once in the fixture. All the other classes only need to check if it is available. The fixture constructor is guaranteed to run before the test classes are instantiated.
Instead of a boolean to check, you could also simply make a method checkServer(string forClass) in the fixture class to call which itself throws the exception, then you'd only have to call
basis.checkServer(nameof(SO74607887Tests));
in the test classes instead of throwing in each test class.
All the tests in the classes will be marked as failed in the Test Excplorer window:

Recursive class fixtures in C# xUnit

I want to do something like this to provide methods from test classes to other test classes using composition.
public class SomeTestClass : IClassFixture<SomeService>
{
private readonly SomeService SomeService;
public SomeTestClass(SomeService someService)
{
SomeService = someService;
}
[Fact]
private void Test()
{
//....
}
public SomeData CreateSomeData()
{
// populate fields with something based on internal/service state
return new SomeData();
}
public void DoSomeAction(....)
{
// does action which modifies internal/service state
}
}
public class SomeConsumingClass : IClassFixture<SomeTestClass>
{
private readonly SomeTestClass SomeTestClass;
public SomeConsumingClass(SomeTestClass someTestClass)
{
SomeTestClass = someTestClass;
}
[Fact]
private void Test()
{
var data = SomeTestClass.CreateSomeData();
// ....
SomeTestClass.DoSomeAction(...)
}
}
The test in SomeTestClass passes but the test in SomeConsumingClass fails with a message
Class fixture type 'Requirements.SomeTestClass' had one or more unresolved constructor arguments: SomeService someService) (The following constructor parameters did not have matching fixture data: SomeTestClass someTestClass
It seems like a feature like this is not directly supported as it seems to be looking for a parameterless constructor. I intended to use this pattern for each test class, therefore I am looking for some good way to do something similar without too much boilerplate code. Any suggestions on how I could provide methods from the other test classes without inheritance?
EDIT:
Added some additional examples how I imagine myself using this

From the xUnit documentation on Class Fixtures (emphasis added):
Note that you cannot control the order that fixture objects are created, and fixtures cannot take dependencies on other fixtures. If you have need to control creation order and/or have dependencies between fixtures, you should create a class which encapsulates the other two fixtures, so that it can do the object creation itself.
One solution would be to move the CreateSomeData method to SomeService, and then change SomeConsumingClass so that it also implements IClassFixture<SomeService>.
However, it's worth pointing out this line from the documentation regarding when IClassFixture is appropriate:
When to use: when you want to create a single test context and share it among all the tests in the class, and have it cleaned up after all the tests in the class have finished.
From the description you've provided, it doesn't seem clear to me that IClassFixture is necessary here, since all you really want is the ability to call CreateSomeData from different test classes. A dead-simple alternative would be to just move GetSomeData to a utility class that can be directly called from any test fixture that needs it.

So I played around with possible solutions and was able to come up with a solution which allows for identical behavior
public class SomeTestClass
{
private readonly SomeService SomeService;
public SomeTestClass(SomeService someService)
{
SomeService = someService;
}
[Fact]
private void Test()
{
//....
}
public SomeData CreateSomeData()
{
// populate fields with something based on internal/service state
return new SomeData();
}
public void DoSomeAction(....)
{
// does action which modifies internal/service state
}
}
public class SomeDerivingTestClass : SomeTestClass
{
public SomeDerivingTestClass() : base(CreateSomeService())
{
}
private static SomeService CreateSomeService()
{
return new SomeService();
}
}
public class SomeConsumingClass : IClassFixture<SomeDerivingTestClass>
{
private readonly SomeTestClass SomeTestClass;
public SomeConsumingClass(SomeDerivingTestClass someTestClass)
{
SomeTestClass = someTestClass;
}
[Fact]
private void Test()
{
var data = SomeTestClass.CreateSomeData();
// ....
SomeTestClass.DoSomeAction(...)
}
}

how to return to the system initial state between several tests executions

I'm using Nunit and I wonder if there is a way to return to the real initialization state including all the singletons that constructed already.
currently I've solved it with a workaround of adding to the singletons a reset method and I don't like it because now my production code is not clean.
as far as I understand the single tones will be initialized once and be kept in the heap which is available during all the tests executions.
Is there a way to clean the heap at the teardown function of the Nunit testfixture?
I've managed to reproduce the issue with the following (ugly) code sample, you can see that when both tests executed one after the other the second one failed...
[TestFixture]
public class SingletonCleanups
{
private MySingleTonsHolder _holder;
[SetUp]
public void Setup()
{
_holder = new MySingleTonsHolder();
}
[Test]
public void DoWork_FirstExecution_SingleCalledOnce()
{
_holder.DoWork();
Assert.AreEqual(1, MySingleTonsHolder.MySingleTon.CalledCount);
}
[Test]
public void DoWork_SecondExecution_SingleCalledOnce()
{
_holder.DoWork();
Assert.AreEqual(1, MySingleTonsHolder.MySingleTon.CalledCount);
}
}
public class MySingleTonsHolder
{
public static MySingleTon MySingleTon => MySingleTon.Instance();
public void DoWork()
{
MySingleTon.Instance().CalledCount++;
}
}
public class MySingleTon
{
private static MySingleTon _instance;
public static MySingleTon Instance()
{
if (_instance==null)
{
_instance = new MySingleTon();
}
return _instance;
}
public int CalledCount { get; set; }
private MySingleTon()
{
CalledCount = 0;
}
}

The thing here that unit tests shows to you very useful stuff: static fields in code is EVIL. now my production code is not clean=> It wasnt clean, because of statics.
If you want to introduce singletone, do this please using DI framework, which you use, or without DI framework, but with IoC.
For more information, pls go to anwser from here Dependency Injection & Singleton Design pattern

MsTest ClassInitialize and Inheritance

I have a base class for my tests which is composed in the following way:
[TestClass]
public abstract class MyBaseTest
{
protected static string myField = "";
[ClassInitialize]
public static void ClassInitialize(TestContext context)
{
// static field initialization
myField = "new value";
}
}
Now I am trying to create a new test that inherits from the base, with the following signature:
[TestClass]
public class MyTest : MyBaseTest
{
[TestMethod]
public void BaseMethod_ShouldHave_FieldInitialized()
{
Assert.IsTrue(myField == "new value");
}
}
The ClassInitialize is never called by the child tests ... What is the real and correct way of using test initialization with inheritance on MsTest?

Unfortunately you cannot achieve this that way because the ClassInitializeAttribute Class cannot be inherited.
An inherited attribute can be used by the sub-classes of the classes that use it. Since the ClassInitializeAttribute cannot not be inherited, when the MyTest class is initialized the ClassInitialize method from the MyBaseTest class cannot be called.
Try to solve it with another way. A less efficient way is to define again the ClassInitialize method in MyTest and just call the base method instead of duplicating the code.

A potential workaround is to define a new class with AssemblyInitializeAttribute instead. It has a different scope, obviously, but for me it meets my needs (cross-cutting concerns, which just so happen to require exactly the same settings for every test class and test method.)
using Microsoft.VisualStudio.TestTools.UnitTesting;
namespace MyTests
{
[TestClass]
public sealed class TestAssemblyInitialize
{
[AssemblyInitialize]
public static void Initialize(TestContext context)
{
...
}
}
}

Use a static constructor on a base class? It's executed only once, by design, and it doesn't have the weird limitation on inheritance, like the ClassInitializeAttribute.

There is a parameter for the ClassInitialize and ClassCleanup attributes:
[ClassInitialize(InheritanceBehavior.BeforeEachDerivedClass)]
public static void ClassInitialize(TestContext context)
{
// gets called once for each class derived from this class
// on initialization
}
[ClassCleanup(InheritanceBehavior.BeforeEachDerivedClass)]
public static void Cleanup()
{
// gets called once for each class derived from this class
// on cleanup
}
which will actually do what you want.

UPDATE: Added lock to avoid multi-threading issues...
We know that a new instance of the class is constructed for every [TestMethod] in the class as it gets run. The parameter-less constructor of the base class will be called each time this happens. Couldn't you simply create a static variable in the base class and test it when constructor runs?
This helps you to not forget to put the initialization code in the sub-class.
Not sure if there's any drawback to this approach...
Like so:
public class TestBase
{
private static bool _isInitialized = false;
private object _locker = new object();
public TestBase()
{
lock (_locker)
{
if (!_isInitialized)
{
TestClassInitialize();
_isInitialized = true;
}
}
}
public void TestClassInitialize()
{
// Do one-time init stuff
}
}
public class SalesOrderTotals_Test : TestBase
{
[TestMethod]
public void TotalsCalulateWhenThereIsNoSalesTax()
{
}
[TestMethod]
public void TotalsCalulateWhenThereIsSalesTax()
{
}
}

How to mark a whole class as "Inconclusive"?

I've a test class named MyClass. MyClass has a TestFixtureSetUp that loads some initial data. I want to mark whole class as Inconclusive when loading initial data fails. Just like when somebody marks a test method Inconclusive by calling Assert.Inconclusive().
Is there any solution?

You can work around it using Setup by signaling it when a data loading failed.
For example:
[TestFixture]
public class ClassWithDataLoad
{
private bool loadFailed;
[TestFixtureSetUp]
public void FixtureSetup()
{
// Assuming loading failure throws exception.
// If not if-else can be used.
try
{
// Try load data
}
catch (Exception)
{
loadFailed = true;
}
}
[SetUp]
public void Setup()
{
if (loadFailed)
{
Assert.Inconclusive();
}
}
[Test] public void Test1() { }
[Test] public void Test2() { }
}
Nunit does not support Assert.Inconclusive() in the TestFixtureSetUp. If a call to Assert.Inconclusive() is done there all the tests in the fixture appear as failed.

Try this:
In your TestFixtureSetUp, store a static value in the class to indicate whether the data has yet to be loaded, was successfully loaded, or was attempted but unsuccessfully loaded.
In your SetUp for each test, check the value.
If it indicates an unsuccessful load, immediately bomb out by calling Assert.Inconclusive().