I'm trying to setup my tests using Xunit. I have a requirement to delete all images in a folder start of the tests, and then each method does some image resizing and saves a copy of it's output to the folder. The folder should only be emptied once, and then each method will save their own image into the folder.
When I use IUseFixture<T>, the ClearVisualTestResultFolder function is still being called before every test, so I only end up with one image in the folder.
public class Fixture
{
public void Setup()
{
ImageHelperTest.ClearVisualTestResultFolder();
}
}
public class ImageHelperTest : IUseFixture<EngDev.Test.Fixture>
{
public void SetFixture(EngDev.Test.Fixture data)
{
data.Setup();
}
public static void ClearVisualTestResultFolder()
{
// Logic to clear folder
}
}
If I put the ClearVisualTestResultFolder in the constructor, it is also being called once for every test method. I need this just run once before all test methods are executed, how can I achieve this?
If it matters, I use the ReSharper test runner.
Following the guidance in this xUnit discussion topic, it looks like you need to implement a constructor on the Fixture and also implement IDisposable. Here's a complete sample that behaves the way you want:
using System;
using System.Diagnostics;
using Xunit;
using Xunit.Sdk;
namespace xUnitSample
{
public class SomeFixture : IDisposable
{
public SomeFixture()
{
Console.WriteLine("SomeFixture ctor: This should only be run once");
}
public void SomeMethod()
{
Console.WriteLine("SomeFixture::SomeMethod()");
}
public void Dispose()
{
Console.WriteLine("SomeFixture: Disposing SomeFixture");
}
}
public class TestSample : IUseFixture<SomeFixture>, IDisposable
{
public void SetFixture(SomeFixture data)
{
Console.WriteLine("TestSample::SetFixture(): Calling SomeMethod");
data.SomeMethod();
}
public TestSample()
{
Console.WriteLine("This should be run once before every test " + DateTime.Now.Ticks);
}
[Fact]
public void Test1()
{
Console.WriteLine("This is test one.");
}
[Fact]
public void Test2()
{
Console.WriteLine("This is test two.");
}
public void Dispose()
{
Console.WriteLine("Disposing");
}
}
}
When running this from the console runner, you'll see the following output:
D:\xUnit>xunit.console.clr4.exe test.dll /html foo.htm xUnit.net
console test runner (64-bit .NET 4.0.30319.17929) Copyright (C)
2007-11 Microsoft Corporation.
xunit.dll: Version 1.9.1.1600 Test assembly: test.dll
SomeFixture ctor: This should only be run once
Tests complete: 2 of 2
SomeFixture: Disposing SomeFixture
2 total, 0 failed, 0 skipped, took 0.686 seconds
Then, when you crack open the test output file foo.htm, you'll see the other test output.
The old IUseFixture<T> interface in xUnit.net v1.x has been replaced
by two new interfaces: IClassFixture<T> and ICollectionFixture<T>. In
addition, the mechanism for injecting fixture values into your tests
has changed from property setters to constructor arguments. Class
fixtures are created once and shared amongst all tests in the same
class (much like the old IUseFixture). Collection fixtures work the
same way, except that the single instance is shared amongst all tests
in the same test collection.
NOTE this applies only to xUnit.net v1 - see the accepted answer unless that's your case
IUseFixture<T>.SetFixture gets called once for each test. The Fixture itself is only created once.
In other words, you shouldnt be doing anything in your SetFixture method, but you should be instead be triggering it in the Fixture constructor.
For one-time cleanup, implement an IDisposable.Dispose on the Fixture (it's not required though)
Note that it's a bad idea to be (even potentially) sharing state between tests. Best to use a TemporaryDirectoryFixture like this one.
Related
I have a project where multiple test classes share a base class fixture.
As expected, the fixture's constructor runs before any test in the test class is run.
Is there a way for this class fixture's constructor to know the name of the test class that is about to run?
Background: My project is not doing unit testing, but rather I'm using xUnit.net to run an alternate type of testing. All the classes and tests will run in order, so a class is guaranteed to run completely before another class tries to run.
Here is an example of what I'd like to be able to do:
using Xunit;
using Xunit.Abstractions;
namespace TestingProject
{
// In my project, the classes are guaranteed to run in order, one after the other. Tests won't randomly execute between classes.
// This fixture is used by multiple test classes. It handles logging test class starts and stops to the db.
public class SharedTestingClassFixture : IDisposable
{
// This will run at the start of every testing class.
public SharedTestingClassFixture()
{
// Need to get name of class that is about to execute, i.e. "TestingClassName"
var nameOfTestingClass = "";
// Log name of class that is about to start to the db BEFORE any tests run.
var logMessage = $"{nameOfTestingClass} class starting";
}
// This will run at the end of every testing class.
public void Dispose()
{
// Log class ending to db after all tests in this class have finished.
}
}
public class TestingClassName : TestClassBase, IDisposable, IClassFixture<SharedTestingClassFixture>
{
// This will run before each individual test.
public TestingClassName(ITestOutputHelper testOutputHelper) : base(testOutputHelper)
{
// At this point, I need to already have written the class name to the db; BEFORE this constructor is called.
// The TestClassBase (not shown) constructor then extracts the test name from testOutputHelper and logs to db.
}
// This will run after each individual test.
public void Dispose()
{
}
[Fact]
public void Test1()
{
// Testing code here
}
[Fact]
public void Test2()
{
// Testing code here
}
}
}
I have a method
using Microsoft.VisualStudio.TestTools.UnitTesting; // using visual studio's test framework
[TestMethod]
public void ATestMethod()
{
// stuff
}
from a public class ATestClass. This test class runs two types of tests :
tests requiring that a certain software is installed on the machine running the test
tests that can run free
To handle this, I added a public class BaseTestClass from which I made ATestClass derive, and in ATestClass I added a :
public bool isTheSoftwareInstalledOnTheMachine()
{
// stuff
}
and I "decorated" all internal scopes of tests from ATestClass as follows :
[TestMethod]
public void ATestMethod()
{
if (isTheSoftwareInstalledOnTheMachine())
{
// stuff
}
}
I find this horrible. I would rather like to be able to write something like :
[TestMethod]
[RunIfTheSoftwareInstalledOnTheMachine]
public void ATestMethod()
{
// stuff
}
but I don't know if one is allowed to define "custom" [characterizer]'s. (I don't even know the right word for them.) If it is, would that be the best design ? (I heard about the decorator pattern, but I don't know if I could make it generic enough in my context, because I would potentially need to use the condition for many other test classes.) Anyway, how would I proceed with characterizer's ?
I know you're using VS test framework but if you can change to NUnit you can accomplish what you want.
Test case:
using NUnit.Framework;
[TestFixture]
public class MyAppTests
{
[Test]
[RunIfTheSoftwareInstalledOnTheMachine]
public void ATestMethod()
{
// Executes if custom attribute is true, otherwise test case is ignored
}
}
Custom attribute:
using NUnit.Framework;
using NUnit.Framework.Interfaces;
public class TestHelper
{
public static bool IsTheSoftwareInstalledOnTheMachine()
{
// Return state of software
return true;
}
}
public class RunIfTheSoftwareInstalledOnTheMachineAttribute : Attribute, ITestAction
{
public ActionTargets Targets { get; private set; }
public void AfterTest(ITest test) {}
public void BeforeTest(ITest test)
{
if (!TestHelper.IsTheSoftwareInstalledOnTheMachine())
{
Assert.Ignore("Omitting {0}. Software is not installed on machine.", test.Name);
}
}
}
If you define your own attribute you surerly have to check for its existance on your own. You can´t expect your framework to guess what the attribute is for.
But I suppose you don´t even need an attribute to do this. You can simply ignore the test by putting the logic inside the test-method anyway:
[Test]
public void MyTest()
{
if(!RunIfTheSoftwareInstalledOnTheMachine)
Assert.Ignore("Test not run because no software was installed");
// your actual test-code
}
Another approach is to use the CategoryAttribute provided by NUnit, with which you can run only those tests that fall within your provided category:
[Test]
[Category("SoftwareInstalled")]
public void MyTest() { /* ... */ }
EDIT: You could also use the TestCaseAttribute with a specific method that returns a TestCase when the condition is met:
[TestCaseSource("ProvideTestcases")]
public void MyTest() { /* ... */ }
private static IEnumerable<TestCaseData> ProvideTestcases()
{
if(RunIfTheSoftwareInstalledOnTheMachine)
yield return new TestCaseData();
}
If the codition is not met no testcase is generated at all.
If the software being installed on the machine is a requirement for any of the tests to pass and any one test failing means the whole suite fails then why bother checking in multiple tests if the software is installed? Just write a single test to fail if the software is not installed and throw a useful exception. Something like:
[Test]
public void EnsureImportantSoftwareIsInstalled()
{
if(!importantSoftwareIsInstalled)
{
Assert.Fail($"Software X must be installed for the tests in {nameof(MyTestClass)} to run, please install it");
}
}
For Nunit 2.6, a slight variation of the HimBromBeere's answer works well for me. The test case is displayed as ignored.
[TestCaseSource("ProvideTestcases")]
public void MyTest() { /* ... */ }
private static IEnumerable<TestCaseData> ProvideTestcases()
{
if(RunIfTheSoftwareInstalledOnTheMachine)
yield return new TestCaseData().Ignore();
}
I am writing coded ui tests and I have the application open if it is not already open. Then if one of them fails I close the application the thing is I have multiple tests in multiple projects is there a way to close the application after all of the tests are done executing? Is there maybe something in the testSettings file?
If this helps at all, all of my test classes derive from one codeduiTestBase which is how I set up the settings I do have.
I do not want to have to open and close the application before and after each test runs because it is a big application and it takes too long to load.
Yes it is possible. You can use the AssemblyCleanup Attribute for this purpose:
Identifies a method that contains code to be used after all tests in
the assembly have run and to free resources obtained by the assembly.
Here is an overview of all MSTest methods arranged according to execution time:
using Microsoft.VisualStudio.TestTools.UnitTesting;
using SampleClassLib;
using System;
using System.Windows.Forms;
namespace TestNamespace
{
[TestClass()]
public sealed class DivideClassTest
{
[AssemblyInitialize()]
public static void AssemblyInit(TestContext context)
{
MessageBox.Show("AssemblyInit " + context.TestName);
}
[ClassInitialize()]
public static void ClassInit(TestContext context)
{
MessageBox.Show("ClassInit " + context.TestName);
}
[TestInitialize()]
public void Initialize()
{
MessageBox.Show("TestMethodInit");
}
[TestCleanup()]
public void Cleanup()
{
MessageBox.Show("TestMethodCleanup");
}
[ClassCleanup()]
public static void ClassCleanup()
{
MessageBox.Show("ClassCleanup");
}
[AssemblyCleanup()]
public static void AssemblyCleanup()
{
MessageBox.Show("AssemblyCleanup");
}
[TestMethod()]
[ExpectedException(typeof(System.DivideByZeroException))]
public void DivideMethodTest()
{
DivideClass.DivideMethod(0);
}
}
}
see: MSTest-Methods
IMPORTANT
Someone who is using base class can do assembly cleanup there as well
[TestClass]
public class Page : PageContract, IWindowControlAccess
{
[AssemblyCleanup()]
public static void ApplicationCleanup()
{
Cleanup();
}
}
Two important things here
1.
[TestClass] attribute at the base is MANDATORY
2.
signature of ApplicationCleanup see its a static method
That's it you are done. I had hard time struggling and debugging it.
Perhaps what you're looking for is the TestCleanup attribute?
EDIT: Also, if you want to run something after all tests have been run, the AssemblyCleanupAttribute might be more appropriate.
If you just want to clean up after all tests inside a single class have finished, use the ClassCleanup attribute rather than the AssemblyCleanup attribute
I use [OneTimeSetUp] and [OneTimeTearDown].
What i want to do is this :
Create a bunch of Unit Tests.
Create a variety of different permutations/combinations of initialization of mocks, input variables etc.
Run each given unit test with a against a set of such initializations based on some parameters.
How would i go about doing something like this?
Is there already any framework to handle this (I.e. run a given test multiple times while changing initialization)? Can you suggest any design or ideas with which i could make something to do this?
I am aware of unit testing frame works. i use NUnit and Rhino mocks myself.
Shown below is an example of what i need.
[Test Initialize]
Setup( <-possible parameter-> )
[Test Method]
TestA()
now i want TestA() to be run multiple times. Each time the Test initialize would pick another initialization combination.
More clarification
Lets suppose a test would require variables A, B, C. Each of them are very complex objects with the end result that the a large number of combinations can be formed. So i'm hoping that somehow i could create a test initialize that could possible iterate through a list of such combinations, so it would initialize them, run the TESTA, go back to next initialization in the list, run TESTA again and so on until the list runs out. Next it picks another list for TESTB and once again follows this process.
At the least im hoping for some ability to be able to run a given TEST function n times. The rest i know i can build once this is possible
In nUnit you can use the [TestCase] attribute for simple types:
[Test]
[TestCase("a", "b")]
[TestCase("c", "b")]
[TestCase("a", "d")]
public void TestMethod(string param1, string param2){
// run your test with those parameters
}
Or you can use a TestCaseSource method for complex types:
[Test]
[TestCaseSource("GetTestCases")]
public void TestMethod(MyObject1 param1, MyObject2 param2){
// run your test with those parameters
}
private IEnumerable GetTestCases(){
yield return new TestCaseData( new MyObject1("first test args"),
new MyObject2("first test args"))
.SetName("SomeMeaningfulNameForThisTestCase" );
yield return new TestCaseData( new MyObject1("2nd test args"),
new MyObject2("2nd test args"))
.SetName("SomeMeaningfulNameForThisTestCase2" );
}
You can do something similar in MS-Test using a DataSource:
http://codeclimber.net.nz/archive/2008/01/18/How-to-simulate-RowTest-with-MS-Test.aspx
You might be able to do this without needing any framework-specific addons by creating an abstract base class that contains all your test functions, then inheriting that base class with multiple classes, each with their own setup function.
public abstract class MyTests
{
[Test]
public void TestOne()
{
...
}
[Test]
public void TestTwo()
{
...
}
}
[TestFixture]
public class FirstSetup : MyTests
{
[Setup]
public void Setup()
{
...
}
}
[TestFixture]
public class SecondSetup : MyTests
{
[Setup]
public void Setup()
{
...
}
}
I have done this in other languages, but not sure how the various C# frameworks will handle it.
I'm a big fan of the xUnit.NET framework; I find it light, simple, clean, and extensible.
Now let's say that I have a class like so:
public class AdditionSpecification
{
static int result;
public void Because()
{
result = 2 + 2;
}
public void Result_is_non_zero()
{
Assert.True(result <> 0);
}
public void Result_is_correct()
{
Assert.Equal(4, result);
}
}
With the test class above I want xUnit.NET to see 2 test cases and to run the Because() method before each of them.
Leaving aside any issues you may have with my class or method names, the structure of this test/specification, the xUnit.NET framework, or BDD, here's my question:
How can I tell xUnit.NET that I want to customize how it identifies and executes test methods out of this class without using a custom [Fact]-like attribute on each target test method?
I know that I can derive from BeforeAfterAttribute to decorate each test method with custom before and after execution. How can i do this at the class level? Do i have to write a custom runner?
xUnit.net's IUseFixture allows you to do per fixture setup. You could therefore define your own fixture class:
public class AdditionFixture : IDisposable
{
public int Because()
{
return 2 + 2;
}
public void Dispose()
{
//test tear down code
}
}
Your test class can then implement this (with setFixture requiring implementing) :
public class AdditionSpecification : IUseFixture<AdditionFixture>
{
int result;
public void SetFixture(AdditionFixture Fixture)
{
result = Fixture.Because();
}
[Fact]
public void Result_is_non_zero()
{
Assert.True(result <> 0);
}
[Fact]
public void Result_is_correct()
{
Assert.Equal(4, result);
}
}
The xUnit runner will create a single instance of your fixture, and pass it into SetFixture before running each test. After running all of your tests, the runner will then dispose of the fixture if it implements IDisposable. I hope that helps!
The xUnit wiki on codeplex has more information, including a nice example of how to implement IUseFixture to manage a database connection for you test fixtures.
So it turns out that I was looking for the ITestClassCommand.EnumerateTestMethods() method.
The default xUnit.NET test runner
will iterate over all the classes in
your test assembly.
For each one it will check for a RunWithAttribute;
that's your chance to override the
ITestClassCommand implementation
that is used to identify methods
containing tests. (RunWithNUnit is a good example)
ITestClassCommand.EnumerateTestMethods() is called to process the test class and return an IEnumerable of test methods.
each test IMethodInfo is then passed to ITestClassCommand.EnumerateTestCommands(IMethodInfo testMethod) to get the IEnumerable of ITestCommands
each ITestCommand is then executed and given the opportunity to return a result.
In the case of my example above, I would need something like:
[AttributeUsage(AttributeTargets.Class, AllowMultiple = false)]
public class RunWithMyTestClassCommandAttribute : RunWithAttribute
{
public RunWithMyTestClassCommandAttribute()
: base(typeof(MyTestClassCommand)) {}
}
Then I could decorate my above example with:
[RunWithMyTestClassCommand]
public class AdditionSpecification
{
static int result;
public void Because()
{
result = 2 + 2;
}
public void Result_is_non_zero()
{
Assert.True(result <> 0);
}
public void Result_is_correct()
{
Assert.Equal(4, result);
}
}
Finally, in MyTestClassCommand, I get to opportunity between EnumerateTestMethods() and EnumerateTestCommands(IMethodInfo testMethod) to use whatever logic I want to locate and construct ITestCommand instances that get executed as individual tests.
BTW, in the process of researching this issue, I ran into a small bug in the xUnit.NET framework where a custom IMethodInfo generated by EnumerateTestMethods() never showed up in EnumerateTestCommands(..) because it was being unwrapped and rewrapped by the test runner or one of it's factories.
I submitted this issue to the xUnit project on codeplex and it was corrected on May 30th, 2009 for xUnit.NET 1.5 CTP 2