I googled for this issue but could not find the answer.
My test runs appear to run in parallel and cause each other to fail. They do all pass when run individually. I tried to add thread in the test and put them to sleep but no luck.
Is there a way to run these tests in sequence one after another?
My environment:
Visual Studio 2010
Resharper Jet brains 6.1
I would suggest you have unit tests that are deterministic. That is they don't depend on the order they are run or that other tests be run before or after. Not doing this is a recipe for failure. Most test runners are based on the fact that test methods are completely independent.
This fact is inherently obvious in the way the methods of a test class are invoked. e.g. with MS Test you can have Assembly, Class and Test initialize methods. All of these are invoked for each TestMethod being invoked. For example, with the following class:
[TestClass()]
public class DivideClassTest
{
[AssemblyInitialize()]
public static void AssemblyInit(TestContext context)
{
Console.WriteLine("Assembly Init");
}
[ClassInitialize()]
public static void ClassInit(TestContext context)
{
Console.WriteLine("ClassInit");
}
[TestInitialize()]
public void Initialize()
{
Console.WriteLine("TestMethodInit");
}
[TestCleanup()]
public void Cleanup()
{
Console.WriteLine("TestMethodCleanup");
}
[ClassCleanup()]
public static void ClassCleanup()
{
Console.WriteLine("ClassCleanup");
}
[AssemblyCleanup()]
public static void AssemblyCleanup()
{
Console.WriteLine("AssemblyCleanup");
}
[TestMethod()]
public void Test1()
{
Console.WriteLine("Test1");
}
[TestMethod()]
public void Test2()
{
Console.WriteLine("Test2");
}
}
You'll see output like
Assembly Init
ClassInit
TestMethodInit
Test1
TestMethodCleanup
TestMethodInit
Test2
TestMethodCleanup
ClassCleanup
AssemblyCleanup
Although there is a "Test" class, the TestMethod itself is considered the test. A "test" class can effectively have many tests.
Related
I have a method
using Microsoft.VisualStudio.TestTools.UnitTesting; // using visual studio's test framework
[TestMethod]
public void ATestMethod()
{
// stuff
}
from a public class ATestClass. This test class runs two types of tests :
tests requiring that a certain software is installed on the machine running the test
tests that can run free
To handle this, I added a public class BaseTestClass from which I made ATestClass derive, and in ATestClass I added a :
public bool isTheSoftwareInstalledOnTheMachine()
{
// stuff
}
and I "decorated" all internal scopes of tests from ATestClass as follows :
[TestMethod]
public void ATestMethod()
{
if (isTheSoftwareInstalledOnTheMachine())
{
// stuff
}
}
I find this horrible. I would rather like to be able to write something like :
[TestMethod]
[RunIfTheSoftwareInstalledOnTheMachine]
public void ATestMethod()
{
// stuff
}
but I don't know if one is allowed to define "custom" [characterizer]'s. (I don't even know the right word for them.) If it is, would that be the best design ? (I heard about the decorator pattern, but I don't know if I could make it generic enough in my context, because I would potentially need to use the condition for many other test classes.) Anyway, how would I proceed with characterizer's ?
I know you're using VS test framework but if you can change to NUnit you can accomplish what you want.
Test case:
using NUnit.Framework;
[TestFixture]
public class MyAppTests
{
[Test]
[RunIfTheSoftwareInstalledOnTheMachine]
public void ATestMethod()
{
// Executes if custom attribute is true, otherwise test case is ignored
}
}
Custom attribute:
using NUnit.Framework;
using NUnit.Framework.Interfaces;
public class TestHelper
{
public static bool IsTheSoftwareInstalledOnTheMachine()
{
// Return state of software
return true;
}
}
public class RunIfTheSoftwareInstalledOnTheMachineAttribute : Attribute, ITestAction
{
public ActionTargets Targets { get; private set; }
public void AfterTest(ITest test) {}
public void BeforeTest(ITest test)
{
if (!TestHelper.IsTheSoftwareInstalledOnTheMachine())
{
Assert.Ignore("Omitting {0}. Software is not installed on machine.", test.Name);
}
}
}
If you define your own attribute you surerly have to check for its existance on your own. You can´t expect your framework to guess what the attribute is for.
But I suppose you don´t even need an attribute to do this. You can simply ignore the test by putting the logic inside the test-method anyway:
[Test]
public void MyTest()
{
if(!RunIfTheSoftwareInstalledOnTheMachine)
Assert.Ignore("Test not run because no software was installed");
// your actual test-code
}
Another approach is to use the CategoryAttribute provided by NUnit, with which you can run only those tests that fall within your provided category:
[Test]
[Category("SoftwareInstalled")]
public void MyTest() { /* ... */ }
EDIT: You could also use the TestCaseAttribute with a specific method that returns a TestCase when the condition is met:
[TestCaseSource("ProvideTestcases")]
public void MyTest() { /* ... */ }
private static IEnumerable<TestCaseData> ProvideTestcases()
{
if(RunIfTheSoftwareInstalledOnTheMachine)
yield return new TestCaseData();
}
If the codition is not met no testcase is generated at all.
If the software being installed on the machine is a requirement for any of the tests to pass and any one test failing means the whole suite fails then why bother checking in multiple tests if the software is installed? Just write a single test to fail if the software is not installed and throw a useful exception. Something like:
[Test]
public void EnsureImportantSoftwareIsInstalled()
{
if(!importantSoftwareIsInstalled)
{
Assert.Fail($"Software X must be installed for the tests in {nameof(MyTestClass)} to run, please install it");
}
}
For Nunit 2.6, a slight variation of the HimBromBeere's answer works well for me. The test case is displayed as ignored.
[TestCaseSource("ProvideTestcases")]
public void MyTest() { /* ... */ }
private static IEnumerable<TestCaseData> ProvideTestcases()
{
if(RunIfTheSoftwareInstalledOnTheMachine)
yield return new TestCaseData().Ignore();
}
I know, that in Coded UI there are two methods (MyTestInitialize and MyTestCleanup) which can perform action before and after each tests. I need add some action which launch before and after all tests. For example, if you familiar with rspec there are two methods before() and after(), which take parameter :each (will call before/after each tests) or :all (will call before/after all test).
Create your methods with the [ClassInitialize] and [ClassCleanup] attributes as necessary. This should be within your Test Class. Example:
[CodedUITest]
public class MyTestClass
{
[ClassInitialize]
public void DoSomethingFirst()
{
// your code here that will run at the beginning of each test run.
}
[TestInitialize]
public void RunBeforeEachTest()
{
// your test initialization here
}
[TestMethod]
public void MyTestMethod()
{
}
}
And you would do the same for your [TestCleanup] and [ClassCleanup].
More on this attribute can be found here.
I have got a standard NUnit unit test.
[TestFixture]
public class MyTest
{
[SetUp]
public virtual void Setup()
{
}
[TearDown]
public void CleanUp()
{
}
[Test]
public void Test01()
{
Assert.AreEqual(10, 10);
}
[Test]
public void Test02()
{
Assert.AreEqual(4, 1);
}
[Test]
public void Test03()
{
Assert.AreEqual(1, 51);
}
//......a huge amount of Tests
[Test]
public void TestN()
{
Assert.AreEqual(1, 1);
}
}
As you see, there are a lot of tests here. What I need to do is to handle any error in each test and to get the information about the error without modifying the body of a test method.
So in JUnit there is an interface called TestListener which allows to handle any test error, see the error message, see the name of the test, etc.
I didn't find something similar in NUnit. Does it exist? Or is there is a way to do it?
It sounds like you probably want EventListeners:
EventListeners are able to respond to events that occur in the course of a test run, usually by recording information of some kind. Note that EventListeners called asynchronously with respect to test execution and are not able to affect the actual execution of the test.
It sounds like you'll be interested in this method:
void TestFinished(TestResult result);
(As an aside, it seems odd to me that this doesn't use normal .NET events or provide a no-op interface implementation in an abstract class, but there we go...)
I'm in the process of setting up tests in NUnit and have a newbie question.
Is it possible to have a Test/s that could be used in multiple [TestFixture]s?
So
[Test]ValidateString(string bob)
Could be called in a series of different [TestFixture]?
That doesn't sound like a test to me. Tests are typically parameterless (unless you're using [TestCase]s) and running it within a context of a single fixture would typically be enough -- it either passes once and that's good or it doesn't and it's a broken test.
If you just have a method that does some validation on a string, you could set it up as a static method on some class (e.g. TestHelpers) and call it from whatever tests (in multiple test fixtures) need it.
Here's another idea: inheritance. You can have a base fixture that has all your tests, and then fixtures that inherit from it that set up whatever variables you need. The tests will run for each fixture. I'm not familiar with Selenium RC, but you should be able to adapt the code below to set up whatever variables you need in various fixtures.
[TestFixture]
public class BaseFixtureTests
{
protected IMyClass _myClass;
[TestFixtureSetUp]
public void FixtureSetup()
{
_myClass = ConfigureMyClass();
}
protected virtual IMyClass ConfigureMyClass()
{
// fixtures that inherit from this will set up _myClass here as they see fit.
}
[Test]
public void MyClassTest1()
{
// test something about _myClass;
}
}
[TestFixture]
public class MySpecificFixture1 : BaseFixtureTests
{
protected override IMyClass ConfigureMyClass()
{
return new MySpecificMyClassImplementation();
}
}
public class MySpecificMyClassImplementation : IMyClass
{
//some implementation
}
You can also add extra tests in each fixture as well that don't test common functionality and don't need to be reused across fixtures.
The newer version of NUnit supports generics. This is a great fit if what you are testing doesn’t need to be configured (only created) from your test code. Here is an example copied from http://nunit.net/blogs/:
[TestFixture(typeof(ArrayList))]
[TestFixture(typeof(List<int>))]
public class IList_Tests<TList> where TList : IList, new()
{
private IList list;
[SetUp]
public void CreateList()
{
this.list = new TList();
}
[Test]
public void CanAddToList()
{
list.Add(1); list.Add(2); list.Add(3);
Assert.AreEqual(3, list.Count);
}
}
I’ve also used Anna’s approach of inheritance. One possible refinement to her example (depending on personal preference): Don’t mark the base class as a TestFixture, only the child classes. Each class that you mark as a TestFixture will be displayed as a set of tests in the NUnit client. You will probably never want to run the base class methods directly because the child is providing all of the setup code. If you remove TestFixture from the base class, running invalid tests won’t be an option in the UI. This allows you to run all the tests and see all green… always a nice feeling.
You might be able to achieve what you want with inheritance.
using NUnit.Framework;
namespace ClassLibrary1
{
[TestFixture]
public class TestFixtureBase
{
[SetUp]
public virtual void Setup()
{
// setup code here
}
[Test]
public void CommonTest1()
{
Assert.True(true);
}
[Test]
public void CommonTest2()
{
Assert.False(false);
}
}
public class MyClassTests : TestFixtureBase
{
[SetUp]
public override void Setup()
{
base.Setup();
// additional setup code
}
[Test]
public void MyClassTest1()
{
Assert.True(true);
}
}
}
You can write a method to be called from multiple [Test] methods. But I don't think there is a way to have the same [Test] included in multiple [TestFixture]s.
[TestFixture]
public class TestsOne{
[Test] public void TryOne(){
Helpers.ValidateString("Work?");
}
}
[TestFixture]
public class TestsTwo{
[Test] public void TryTwo(){
Helpers.ValidateString("Work?");
}
}
public static class Helpers{
public static void ValidateString(string s){
Assert.IsNotNull(s);
}
}
I have an issue with NUnit - wondering if anyone has any ideas.
We're using NUnit 2.5.3.9345 and C# 3.5.
Take the following code:
public class UnitTestBase
{
[TestFixtureSetUp]
public void SetUpTestFixture()
{
//Do something in base
}
}
[TestFixture]
public class SomeTestClass : UnitTestBase
{
[TestFixtureSetUp]
public void FixtureSetUp()
{
//Do something in test class
}
[Test]
public void SomeTest()
{
//Some assertion
}
}
According to the documentation, if I run SomeTestClass.SomeTest(), UnitTestBase.SetUpTestFixture() should be called before SomeTestClass.FixtureSetUp().
This isn't the case - the base method will only be called if I don't provide a [TestFixtureSetUp] method in the derived class.
Any ideas please? Has me really puzzled!
Thanks.
I am not having the problem. I tested the outcome with the following:
Derived Test
[TestFixture]
public class DerivedTest : TestBase
{
[TestFixtureSetUp]
public void FixtureSetup()
{
File.AppendAllText("Out.txt", string.Format("TestFixtureSetUp From DerivedTest{0}", Environment.NewLine));
}
[TestFixtureTearDown]
public void FixtureTearDown()
{
File.AppendAllText("Out.txt", string.Format("TestFixtureTearDown Down From DerivedTest{0}", Environment.NewLine));
}
[SetUp]
public void Setup()
{
File.AppendAllText("Out.txt", string.Format("Setup From DerivedTest{0}", Environment.NewLine));
}
[TearDown]
public void Down()
{
File.AppendAllText("Out.txt", string.Format("TearDown From DerivedTest{0}", Environment.NewLine));
}
[Test]
public void DoATest()
{
File.AppendAllText("Out.txt", string.Format("Did a Test{0}", Environment.NewLine));
}
}
TestBase
public class TestBase
{
[TestFixtureSetUp]
public void BaseTestFixtureSetUp()
{
File.AppendAllText("Out.txt", string.Format("TestFixtureSetUp From TestBase{0}", Environment.NewLine));
}
[TestFixtureTearDown]
public void BaseTestFixtureTearDown()
{
File.AppendAllText("Out.txt", string.Format("TestFixtureTearDown From TestBase{0}", Environment.NewLine));
}
[SetUp]
public void BaseSetup()
{
File.AppendAllText("Out.txt", string.Format("Setup From TestBase{0}", Environment.NewLine));
}
[TearDown]
public void TearDown()
{
File.AppendAllText("Out.txt", string.Format("TearDown From TestBase{0}", Environment.NewLine));
}
}
This produces the following output:
TestFixtureSetUp From TestBase
TestFixtureSetUp From DerivedTest
Setup From TestBase
Setup From DerivedTest
Did a Test
TearDown From DerivedTest
TearDown From TestBase
TestFixtureTearDown Down From DerivedTest
TestFixtureTearDown From TestBase
I am was able to test the output with ReSharper 5 beta and the Nunit GUI v 2.5.3.9345 (32-bit)
Edit
While at work the test runner in ReSharper 4.5 did not work properly, however running the built test project in x86 and x64 with the corresponding NUnit.exe/NUnit-86.exe produced valid output.
A workaround / different way of doing it:
Instead of relying on behaviour that is not immediately clear, do something like this instead using the template method pattern to make the ordering explicit using normal language features:
public class UnitTestBase
{
protected abstract void PerFixtureSetUp();
[TestFixtureSetUp]
public void SetUpTestFixture()
{
PerFixtureSetUp();
}
}
[TestFixture]
public class SomeTestClass : UnitTestBase
{
protected override void PerFixtureSetUp()
{
}
[Test]
public void SomeTest()
{
//Some assertion
}
}
Any time I have had reason to use inherited fixtures or test contexts, this way has worked well enough. :)
My problem with relying on the attributes is that since these types are created and invoked via reflection in the runner with no relation between the methods, (no polymorphism) it's harder to reason about the order in which they're called. Using standard language features helps simplify this a little.
Have you tried giving the base class the [TestFixture] attribute? I don't know that that will fix it, but it seems like it might be worth a try... the idea is that NUnit may be ignoring the attributes on the base class if it's not a TestFixture.
Yep, I've been playing around with this for past half hour and it's definitely a bug. I've tried adding TestFixture to all classes as well as having different combinations. I've also tried static and instance methods. It just doesn't seem to want to play nicely! :-(
Anyway, the best workaround I could find is to put the TestFixtureSetUp code in the constructors of your test class and base class. At least then you can be confident of the inheritance and it's more clear to other readers of your code who maybe don't know the inner workings of NUnit :-)
What are you running your tests with? The behavior you are experiencing is not related to NUnit (framework), rather the runner you are using. Are you using Resharper integrated testrunner?