I've a test class named MyClass. MyClass has a TestFixtureSetUp that loads some initial data. I want to mark whole class as Inconclusive when loading initial data fails. Just like when somebody marks a test method Inconclusive by calling Assert.Inconclusive().
Is there any solution?
You can work around it using Setup by signaling it when a data loading failed.
For example:
[TestFixture]
public class ClassWithDataLoad
{
private bool loadFailed;
[TestFixtureSetUp]
public void FixtureSetup()
{
// Assuming loading failure throws exception.
// If not if-else can be used.
try
{
// Try load data
}
catch (Exception)
{
loadFailed = true;
}
}
[SetUp]
public void Setup()
{
if (loadFailed)
{
Assert.Inconclusive();
}
}
[Test] public void Test1() { }
[Test] public void Test2() { }
}
Nunit does not support Assert.Inconclusive() in the TestFixtureSetUp. If a call to Assert.Inconclusive() is done there all the tests in the fixture appear as failed.
Try this:
In your TestFixtureSetUp, store a static value in the class to indicate whether the data has yet to be loaded, was successfully loaded, or was attempted but unsuccessfully loaded.
In your SetUp for each test, check the value.
If it indicates an unsuccessful load, immediately bomb out by calling Assert.Inconclusive().
Related
The problem: I have a bunch of xunit tests that connect to an external server. If that server is up - they fly through very quickly. If that server is down, everything goes into timeouts and retries and the tests all take a long time. The thing is, if I have 100 tests - if the first one fails, I still want all the rest to fail, but it's completely pointless running them... What I'd like is to be able to make my own fact attribute something like:
[FactInFailureGroup( "ConectsToBlah", typeof(ConnectionException)]
public void TestOne()
{
...
}
[FactInFailureGroup( "ConnectsToBlah", typeof(ConnectionException)]
public void TestTwo()
{
...
}
I've looked at before/after attributes, but they don't seem to be able to kill the test or see the result - and I've looked at creating my own fact attribute, which looks like it can prevent the test from running, but not put in a trap for the result. I really want to somehow make code along these lines:
class FactInFailureGroupAttribute : FactAttribute
{
private static _failedGroups = new HashSet<String>();
...
void BeforeTest( TestManipulator manipulator )
{
if (_failedGroups.contains( _thisGroup ))
manipulator.SkipWithMessage( f"all tests in group {_thisGroup} failed because of {_exceptionType}");
}
void AfterTest( TestResult test )
{
if (test.Failed && test.Exception.GetType() == _exceptionType)
_failedGroups.add( _thisGroup );
}
}
You can abuse constructors and statics to accomplish your task.
The constructor of your test class will get called at the beginning of every test but your static should exist for your entire test run.
Try something like this:
private static object Lock = new object();
private static bool? IsServerUp;
public FactAssertionTests()
{
lock (Lock)
{
if (!IsServerUp.HasValue)
{
// check server connection and set IsServerUp
}
else if (!IsServerUp.Value)
{
throw new Exception("Failed to connect to server");
}
}
}
That way whatever test gets into the Lock first will check the server's up-ness. If it fails, every test coming in next will throw the exception.
This isn't quite a decorator attribute and requires you to put all the tests into one class, but it's simple.
Edit:
I tested the following approach, it relies on the fact that xUnit runs all tests in a single class in serial. If you've got tests spanning multiple classes this won't work:
public class ConnectionTests
{
[Fact]
public void Test1()
{
RunServerTest(() =>
{
var svc = new ServiceConnection();
svc.Connect();
});
}
[Fact]
public void Test2()
{
RunServerTest(() =>
{
var svc = new ServiceConnection();
svc.Connect();
});
}
private static bool? ServerIsDown;
public void RunServerTest(Action testAction)
{
var exception = new Exception("Server is down, test will not run.");
if (ServerIsDown.GetValueOrDefault())
{
throw exception;
}
try
{
testAction.Invoke();
}
catch (ServerMissingException ex)
{
ServerIsDown = true;
throw;
}
}
}
You can accomplish what you want by calling your first test explicitly in the constructor of the class. If it fails due to server missing it should throw an exception. This will prevent the class from being instantiated, and none of the other tests in the class will run. They will be marked as failed in the test explorer.
public class SO74607887Tests
{
public SO74607887Tests()
{
TestOne();
}
[Fact]
public void TestOne()
{
// Arrange
// Act
// Oops server is down
throw new Exception("Server is down");
}
[Fact]
public void TestTwo()
{
// Will not run
}
}
If your tests are distributed over a lot of different classes, you can follow #JeremyLakeman 's suggestion to use a fixture. Proceed as follows:
using System;
using Xunit;
namespace SO74607887_XunitCancelTestsIfOneFails
{
// This class will never actually be instantiated, it is only present to provide information to Xunit.
[CollectionDefinition("SO74607887")]
public class SO74607887CollectionDefinition : ICollectionFixture<SO74607887Base>
{
}
// This class creates a single object injected into the constructors of all the other classes in the collection.
public class SO74607887Base : IDisposable
{
public bool serverOK;
public SO74607887Base()
{
// Check server, if it is missing, set the flag
serverOK = false;
}
public void Dispose()
{
// Clean up
}
}
[Collection("SO74607887")]
public class SO74607887Tests
{
public SO74607887Tests(SO74607887Base basis)
{
if (!basis.serverOK)
{
throw new Exception($"Server is down, tests in {nameof(SO74607887Tests)} will not run.");
}
}
[Fact]
public void TestOne()
{
// Arrange
// Act
// Assert
}
[Fact]
public void TestTwo()
{
// Arrange
// Act
// Assert
}
}
}
The check on the server is only done once in the fixture. All the other classes only need to check if it is available. The fixture constructor is guaranteed to run before the test classes are instantiated.
Instead of a boolean to check, you could also simply make a method checkServer(string forClass) in the fixture class to call which itself throws the exception, then you'd only have to call
basis.checkServer(nameof(SO74607887Tests));
in the test classes instead of throwing in each test class.
All the tests in the classes will be marked as failed in the Test Excplorer window:
I have a method
using Microsoft.VisualStudio.TestTools.UnitTesting; // using visual studio's test framework
[TestMethod]
public void ATestMethod()
{
// stuff
}
from a public class ATestClass. This test class runs two types of tests :
tests requiring that a certain software is installed on the machine running the test
tests that can run free
To handle this, I added a public class BaseTestClass from which I made ATestClass derive, and in ATestClass I added a :
public bool isTheSoftwareInstalledOnTheMachine()
{
// stuff
}
and I "decorated" all internal scopes of tests from ATestClass as follows :
[TestMethod]
public void ATestMethod()
{
if (isTheSoftwareInstalledOnTheMachine())
{
// stuff
}
}
I find this horrible. I would rather like to be able to write something like :
[TestMethod]
[RunIfTheSoftwareInstalledOnTheMachine]
public void ATestMethod()
{
// stuff
}
but I don't know if one is allowed to define "custom" [characterizer]'s. (I don't even know the right word for them.) If it is, would that be the best design ? (I heard about the decorator pattern, but I don't know if I could make it generic enough in my context, because I would potentially need to use the condition for many other test classes.) Anyway, how would I proceed with characterizer's ?
I know you're using VS test framework but if you can change to NUnit you can accomplish what you want.
Test case:
using NUnit.Framework;
[TestFixture]
public class MyAppTests
{
[Test]
[RunIfTheSoftwareInstalledOnTheMachine]
public void ATestMethod()
{
// Executes if custom attribute is true, otherwise test case is ignored
}
}
Custom attribute:
using NUnit.Framework;
using NUnit.Framework.Interfaces;
public class TestHelper
{
public static bool IsTheSoftwareInstalledOnTheMachine()
{
// Return state of software
return true;
}
}
public class RunIfTheSoftwareInstalledOnTheMachineAttribute : Attribute, ITestAction
{
public ActionTargets Targets { get; private set; }
public void AfterTest(ITest test) {}
public void BeforeTest(ITest test)
{
if (!TestHelper.IsTheSoftwareInstalledOnTheMachine())
{
Assert.Ignore("Omitting {0}. Software is not installed on machine.", test.Name);
}
}
}
If you define your own attribute you surerly have to check for its existance on your own. You can´t expect your framework to guess what the attribute is for.
But I suppose you don´t even need an attribute to do this. You can simply ignore the test by putting the logic inside the test-method anyway:
[Test]
public void MyTest()
{
if(!RunIfTheSoftwareInstalledOnTheMachine)
Assert.Ignore("Test not run because no software was installed");
// your actual test-code
}
Another approach is to use the CategoryAttribute provided by NUnit, with which you can run only those tests that fall within your provided category:
[Test]
[Category("SoftwareInstalled")]
public void MyTest() { /* ... */ }
EDIT: You could also use the TestCaseAttribute with a specific method that returns a TestCase when the condition is met:
[TestCaseSource("ProvideTestcases")]
public void MyTest() { /* ... */ }
private static IEnumerable<TestCaseData> ProvideTestcases()
{
if(RunIfTheSoftwareInstalledOnTheMachine)
yield return new TestCaseData();
}
If the codition is not met no testcase is generated at all.
If the software being installed on the machine is a requirement for any of the tests to pass and any one test failing means the whole suite fails then why bother checking in multiple tests if the software is installed? Just write a single test to fail if the software is not installed and throw a useful exception. Something like:
[Test]
public void EnsureImportantSoftwareIsInstalled()
{
if(!importantSoftwareIsInstalled)
{
Assert.Fail($"Software X must be installed for the tests in {nameof(MyTestClass)} to run, please install it");
}
}
For Nunit 2.6, a slight variation of the HimBromBeere's answer works well for me. The test case is displayed as ignored.
[TestCaseSource("ProvideTestcases")]
public void MyTest() { /* ... */ }
private static IEnumerable<TestCaseData> ProvideTestcases()
{
if(RunIfTheSoftwareInstalledOnTheMachine)
yield return new TestCaseData().Ignore();
}
I am trying to grab a test result in NUnit 3 upon tear down using the internal ITestResult interface. However when I pass an ITestResult object to the tear down method I get "OneTimeSetup: Invalid signature for SetUp or TearDown method: TestFinished" where TestFinished is listed as my teardown method.
If I don't pass the object the tests work fine. I have tried to move my [TearDown] method to the actual class containing the tests instead of my base class but result in the same error. I would like to have my TestFinish function run upon each test complete so I can act accordingly depending on pass/fail or what is in the exception message rather than using my test try/catch with action structure I have now.
Here is my code structure below:
----A file that starts and ends testing and creates a webdriver object to use---
[OneTimeSetUp]
public void Setup()
{
//Do some webdriver setup...
}
----Base Test Class that is used for setup or tear down of testing----
[TestFixture]
public class BaseTestClass
{
//Also use the webdriver object created at [OneTimeSetUp]
protected void TestRunner(Action action)
{
//perform the incoming tests.
try
{
action();
}
//if the test errors out, log it to the file.
catch (Exception e)
{
//Do some logging...
}
}
[TearDown]
public void TestFinished(ITestResult i)
{
//Handle the result of a test using ITestResult object
}
}
----Test file that uses the BaseTestClass----
class AccountConfirmation : BaseTestClass
{
[Test]
public void VerifyAccountData() {
TestRunner(() => {
//Do a test...
});
}
}
Remove the ITestResult from your TearDown method and instead use TestContext.CurrentContext.Result within the method.
For example,
[Test]
public void TestMethod()
{
Assert.Fail("This test failed");
}
[TearDown]
public void TearDown()
{
TestContext.WriteLine(TestContext.CurrentContext.Result.Message);
}
Will output,
=> NUnitFixtureSetup.TestClass.TestMethod
This test failed
Given the following:
public class TestEnvironment
{
private static TransactionScope scope;
[AssemblyInitialize]
public static void Init(TestContext context)
{
/* establish database connection, create Session fake, create Items fake, etc. */
scope = new TransactionScope();
}
[AssemblyCleanup]
public static void Cleanup()
{
scope.Dispose();
}
}
I'm seeing test data show up in the database. And I'm seeing the following in an error in the test output:
A TransactionScope must be disposed in the same thread that created
it.
This occurs only when ALL tests are run. When any given test is run individually, there's no problem.
If I remove the scope.Dispose() call, allowing the scope to be disposed "naturally", the error vanishes, but I still see records accumulate in the database.
Without speaking too much to how TransactionScope works with threads (because I'm ignorant on the matter), the problem has been resolved by creating the scopes during the instantiation of each TestClass.
To save a handful of keystrokes, we created a ScopedTestClass class:
public class ScopedTestClass : IDisposable
{
private TransactionScope TxnScope;
public ScopedTestClass()
{
TxnScope = new TransactionScope();
}
public void Dispose()
{
TsnScope.Dispose();
}
}
And each TestClass inherits from that:
[TestClass]
public class MyTestClass : ScopedTestClass
{
[TestMethod]
public void DoSomething()
{
// sanity at last!
}
}
And everything is good.
I'm planning to use Assert.Fail in this way in my unit testing.
Inside a private helper method inside the Test class (IsFileExist)
Inside the methods of a helper class (LoadData)
So is this ok? or this is out of the unit test framework usage?
If I did like this when the Assert.Fail execute does it unwind the whole stack for the test method or just only unwind the stack for that particular method?
Helper class
public class DataLoader
{
public void LoadData(string file)
{
if (Util.readfile(file)) {
Assert.Fail("Unable to read the file.");
}
}
}
Test class
[TestClass]
public class testFileData
{
[TestMethod]
public void TestData()
{
string file = "C:\\data.txt";
this.IsFileExist(file);
DataLoader dl = new DataLoader();
dl.LoadData(file);
}
private void IsFileExist(string file)
{
if(!Util.IsFileExist(file)) {
Assert.Fail("File not exist");
}
}
}
The fact that Assert is in the Microsoft.VisualStudio.TestTools.UnitTesting name space should serve as a hint that no, you should not be using it out side of a unit test.
If you want to fail based on a condition in your code, throw an exception.