I have a test suite that utilizes a test reporting framework that I need to run through a base class, everything works fine but the [BeforeScenario] is being called 10 times for every scenario I run, 1 time for each feature in my suite, strangely the teardown only runs once.
Here is what my base class looks like:
[Binding]
public class BaseStep : AllureReport
{
public DriverSupport _support;
public BaseStep(DriverSupport support)
{
_support = support;
}
[BeforeScenario]
public void Setup(FeatureContext context)
{
_support.startDriver();
AllureLifecycle.Instance.RunStep(() =>
{
TestContext.Progress.WriteLine(
$"Test \"{TestExecutionContext.CurrentContext.CurrentTest.FullName}\" is starting...");
});
}
[AfterScenario, Order(0)]
private void TearDown()
{
AllureLifecycle.Instance.RunStep(() =>
{
TestContext.Progress.WriteLine(
$"Test {TestExecutionContext.CurrentContext.CurrentTest.FullName}\" is stopping...");
});
}
And this is what my feature step file looks like (there are 10 of these)
[Binding]
public class TestSteps1:BaseStep
{
public TestSteps1(DriverSupport support) : base(suppprt)
{
}
[Given(#"user goes to (.*)")]
public void GivenUserGoesTo(string p0)
{
_support.driver.GoToUrl(p0)
}
When I run just ONE scenario in a feature by themselves, it will print start the driver and print "test xxx is starting..." 10 times, once for each feature I have, I just want it to run once.
I have thought of moving beforescenario to the step classes themselves, but many of my features use steps from multiple step files, so I think that would be an issue. Is there any way to make it so it only runs BeforeScenario one time per scenario?
SpecFlow will execute the BeforeScenario method for each type it finds in your test project. It looks like there are 10 types that define or inherit this method:
The BaseStep class
Each sub class of BaseStep
When I want a BeforeScenario to run only once per test run I will put it into its own "hooks" class.
Since both the before and after hooks should only execute once, just put both of them in something like SpecFlowHooks.cs.
The Selenium web driver object should be registered with the SpecFlow dependency injection framework, and passed directly to each step class as a constructor argument. Given the code you posted, I'm not sure a common base class for your step definitions is beneficial.
See my answer for How to properly manage and access webdriver instances to avoid problems with parallel execution of tests? for more information about properly registering a Selenium web driver object with SpecFlow.
TextFixture and Test
public class TestFixture : IDisposable
{
public TestFixture()
{
var options = new ChromeOptions();
options.AddExcludedArgument("enable-automation");
options.AddAdditionalCapability("useAutomationExtension", true);
WebDriver.Init("https://localhost:44335/", Browser.Chrome, options);
WebDriver.GetDriver.Manage().Window.Maximize();
}
public void Dispose()
{
WebDriver.Close();
}
}
public abstract class Test : IClassFixture<TestFixture>
{
}
AuthTest
public abstract class AuthTest : Test
{
[Fact, Priority(-1)]
public void LoginTest()
{
var home = GetPage<HomePage>();
home.LoginModal.Open();
home.LoginModal.EnterLoginText(new Login("user", "pw"));
home.LoginModal.Login();
GetPage<DashboardPage>().IsAt();
}
}
HomeTest
[TestCaseOrderer(PriorityOrderer.Name, PriorityOrderer.Assembly)]
public sealed class HomeTest : AuthTest
{
}
ProfileTest
[TestCaseOrderer(PriorityOrderer.Name, PriorityOrderer.Assembly)]
public sealed class ProfileTest : AuthTest
{
[Fact, Priority(0)]
public void OpenProfilePageTest()
{
var profile = GetPage<ProfilePage>();
profile.GoTo();
profile.IsAt();
}
[Fact, Priority(1)]
public void LogoutTest() => GetPage<DashboardPage>().Logout();
}
A few days ago I wrote this code and it used to create 1 browser instance. I started the project today again and now suddenly the fixture gets executed twice and it opens two seperate browsers (which causes my tests to fail). I thought the IClassFixture was supposed to only execute once like [OneTimeSetUp] attribute in NUnit. Why is my fixture executing twice?
This happens when I run all tests (all tests runsProfileTest and HomeTest). If I run for example one of the two tests individually, then just 1 browser instance opens and the test passes.
I'm using XUnit 2.4.0.
- EDIT -
When I use:
VS 2019 (run all tests): It opens 2 browsers at same time and fails.
VS 2019 (debug all tests): It opens 2 browsers at same time and fails.
Jetbrain's Rider IDE (run all tests): It opens 2 browsers at same time and fails.
Jetbrain's Rider IDE (debug all tests): It opens 1 browser till HomeTest finishes and then another browser for ProfileTest, and both tests pass (including LoginTest).
This last is how it is supposed to work and it used to be like this when I used NUnit before.
From https://xunit.net/docs/shared-context#class-fixture
You can use the class fixture feature of xUnit.net to share a single
object instance among all tests in a test class
Notice in a test class
In you case you have two separated classes HomeTest and ProfileTest, regardless of both are derived from same abstracted class, they are treated by xUnit as two different test classes.
Consider to use Collection Fixtures instead.
https://xunit.net/docs/shared-context#collection-fixture
You can use the collection fixture feature of xUnit.net to share a
single object instance among tests in several test class
I'm new to Nunit and am trying to run 2 Test Fixtures, A & B. Within each Fixture I have a unique Setup method for each. However, when I click "Run All" in the "Test Explorer" in Visual Studio, the test setup for Fixture A is called (it was executed first) and Setup for Fixture B is ignored. I also get the same behavior when running all tests via command line. Below is my code:
Fixture A
[TestFixture]
public class A
{
[SetUp]
public void SetupTest()
{
// ...Setup for Fixture A
}
[Test, Order(1)]
public void TestForFixtureA()
{
// ...perform test
}
}
Fixture B
[TestFixture]
public class B
{
[SetUp]
public void SetupTest()
{
// ...Setup for Fixture B
}
[Test]
public void TestForFixtureB()
{
// ...perform test
}
}
What is the correct way to get Setup methods to execute per Fixture?
You are using the incorrect attribute for setup at the test fixture level. The attribute you should be using is [SetUpFixture]. Information about this can be found in the nunit documentation
Here is a list of all the setup attributes and their usages taken from the documentation:
SetUpAttribute is now used exclusively for per-test setup.
TearDownAttribute is now used exclusively for per-test teardown.
OneTimeSetUpAttribute is used for one-time setup per test-run. If you
run n tests, this event will only occur once.
OneTimeTearDownAttribute is used for one-time teardown per test-run.
If you run n tests, this event will only occur once
SetUpFixtureAttribute continues to be used as at before, but with
changed method attributes.
This doesn't seem to explain the bizzare behaviour you are seeing, as setup should be ran per-test, but using the correct attributes couldn't hurt.
If you intend your setup to be run once per fixture, use [OneTimeSetUp]. But if you intend it to run once per test within the fixture, then [SetUp] is correct. We can't tell what you intend from the code.
Whichever one you use, the setups should all run. The only situation in which [OneTimeSetUp] will run but [SetUp] will not is when no individual tests are found within the fixture. Are all the tests being recognized?
I suggest you verify very clearly that the setup is not being run. The easiest way is to temporarily create some output from the setup method.
I'm currently running tests through visual studio. Before all the tests are run, I automatically create a set number of users with know credentials, and at the end of the run I delete those users. However, sometimes I need to cancel my tests midway. In these cases the test never gets the chance to clean up, this means that there is left over fake user info from the test run and may causes the next test run to crash (when it attempts to add user info into the DB). Is there anyway to force visual studio/mstest to run a clean up method even if the test is canceled?
I know one option is to have the test check and make sure that the user info doesn't already exist, and if it does remove it before creating the new users. But this still wouldn't solve the issue of the canceled test run leaving unwanted test data.
Update:
Sorry for the miscommunication, however cleaning up the data at the start of the test is not an option. I'm giving a very simplistic view of the issue, but put simply, I have no easy way of making sure that no test data exists at the start of the test. All clean up must occur at the end of the test.
That is impossible. You better find an alternative solution like using separate database for testing and clean all data before each test run, using fixed set of test users or mark test data with some flag. Check Isolating database data in integration tests article by Jimmy Bogard.
There is no built-in way to change MSTest default behavior. In theory you can write MSTest extension that utilizes TestExecution.OnTestStopping event, but that is not an easy process and it requires registry change. Moreover, a lot of people complain that it is not working.
There also MSTest V2, a new version of MSTest with new extensibility points. But it looks like you can't alter cancel behavior with this points, only write attribute decorators. See Extending MSTest V2.
You can't use AppDomain.CurrentDomain.ProcessExit and Process.GetCurrentProcess().Exited events because cancel seems to kill test run process.
NUnit also doesn't support this at the moment. See related NUnit test adapter Run TearDowns on VS Cancel Test Run issue.
Instead of calling the cleanup function at the end of the test, I call mine at the beginning of each test in order to address this exact problem.
Perform the clean up before creating the data as well, this will ensure that you have no leftover data whatever happens. Of course this is only possible if you can identify any leftover data before running the setup.
The idea is that a transaction is initialized before the test begins. In order for the data to be saved in the database, the transaction must be commited, but it is not commited never. It works in case when test a stop, in the case of a successful or unsuccessful completion of the test.
In integration tests we use somthing like this (with NUnit)(It real production code)
public class RollbackAttribute : TestAttribute, ITestAction
{
private TransactionScope _transaction;
public void BeforeTest(ITest test)
{
_transaction = new TransactionScope();
}
public void AfterTest(ITest test)
{
_transaction.Dispose();
}
public ActionTargets Targets => ActionTargets.Test;
}
[TestFixture]
public class SomeTestClass
{
[Rollback] //No need [Test] because Rollback is inherit it.
public void SomeTestMethod()
{
}
}
On MsTest you can make somthing similar, but in this case you should inherit from base class, I hope it works. For example:
public class RollbackTestBase
{
private TransactionScope _transaction;
[TestInitialize]
public virtual void Setup()
{
_transaction = new TransactionScope();
}
[TestCleanup]
public virtual void TearDown()
{
_transaction.Dispose();
}
}
[TestClass]
public class IntegrationTest : RollbackTestBase
{
[TestMethod]
public void TestDataBase()
{
Assert.IsTrue(true);
}
[TestInitialize]
public virtual void Init()
{
}
[TestCleanup]
public virtual void CleanUp()
{
}
}
There are 2 cases which we need to consider while allocating resources in ATPs (Resources might be Creating users, Connection with database). They are
Creation and deletion of resources after each test.
Creation and deletion of resources after set of tests.
Creation and deletion of resources after each test:
If we want to create instance of particular object before execution of a test and want to clean up memory allocated to that object after execution of that test, then we use Test SetUp and Test TearDown attributes of NUnit. In your case object is creation of number of Users.
[SetUp] : Function which is decorated with Test SetUp attribute contains piece of code that executes before the execution of any test.
[TearDown] : Function which is decorated with Test TearDown attributes contains piece of code that executes after execution of any test
Implementation:
[TestClass]
public class UnitTest1
{
[SetUp]
public void SetUP()
{
// Creating Users with proper credentials
}
[TestMethod]
public void TestMethod1()
{
//Write your ATP
}
[TearDown]
public void TearDown()
{
//Clean up
}
}
Creation and deletion of resources after set of tests:
Now If we want to create instance of an object for set of tests and want to clean up memory after execution of all tests then [TestFixtureSetUp] and [TestFixureTearDown] to initialize an object and to clean up memory respectively. Again In your case object can be creation of set of users.
[TestFixtureSetUp] : Function decorated with TestFixtureSetUp will executes once before the execution of group of tests.
[TestFixtureTearDown] : Function decorated with TestFixtureTearDown will executes once after the execution of group of tests.
Implementation
[TestFixture]
public class Tests
{
[TestFixtureSetUp]
public void Setup()
{
//Create users with credentials
}
[Test]
public void _Test1()
{
//Test_1
}
[Test]
public void _Test2()
{
//Test2
}
[TestFixtureTearDown]
public void CleanUp()
{
//Cleanup; Here you need to add code to Delete all users
}
}
Note: I will suggest you, if you are trying to create and delete users for particular ATP then go with SetUp and TearDown. If you are trying same for bunch of ATPs, I would recommend you to go with TestFixtureSetUp and TestFixtureTearDown.
"If your test get pass or fail, SetUp and TearDown functions will execute"
References:
#Shuvra's Answer.
Nunit: SetUp, TearDown, SetUpFixture, TearDownFixture
I think you should open a transaction before you test, create the data and finish test test. But do not commit the transaction. That will ensure that the test can't affect your DB at all.
Update:
The easier approach is use docker container.
You can run a container from your image and remove that container after the test is done. This should definitely reduce the complexity of your test.
Visual studio uses NUNIT therefore, you can use TearDownAttribute. It should run after the test, even if the test is canceled. You can write a function to clean your data.
Please read the reference documentation here: http://nunit.org/docs/2.2/teardown.html
Just to clear more about the NUNIT standrad. Please follow the steps in the Test class:
[TestFixture]
public class _TestClass
{
[TestFixtureSetUp]
public void Setup()
{
//Clearup can be here before start of the tests. But not Recommended
}
[Test]
public void _Test1()
{
}
[Test]
public void _Test2()
{
}
[TestFixtureTearDown]
public void CleanUp()
{
//I will recommend to clean up after all the tests complete
}
}
Reference: http://nunit.org/docs/2.5/fixtureTeardown.html
A better solution to the problem to use what is called "database mocking". In this case you would have your tests run with a different database (or a fake, virtual one).
This article explains how to implement it in C#
https://msdn.microsoft.com/en-us/library/ff650441.aspx
You should begin a transaction and not commit your records to the DB. Thus, all your changes will be automatically rollbacked when the session is over.
I have a base test class containing some test cases and some ordinary tests:
[TestFixture]
public abstract class TestBase
{
[TestCase(1)]
[TestCase(2)]
[TestCase(3)]
public void TestA(int value)
{
// Perform test
}
[Test]
public void TestB()
{
// Perform test
}
}
These tests are run from child classes which set up the environment in different ways. The child classes only contains setup methods, no tests.
[TestFixture]
public class LocalDatabaseTest : TestBase
{
[SetUp]
public void SetUp()
{
// Set up environment to use local db
}
}
I'm using ReSharper 6.1.1000.82 to run all tests in LocalDatabaseTest, but only the ordinary tests are run. The tests using TestCase does not get any result. If I select Run All on TestA in the TestBase class, all test cases are run (including the other child classes). I'm using NUnit 2.6.2.12296. Any ideas on what I've done wrong?
You've done nothing wrong.
If you open your test dll via NUnit test runner you will see all test are running successfully.
(I just verified your code with NUnit 2.6.2).
Regarding the reason of ignoring parameterized tests on Resharper: It seems there is some issue with Resharper test runner which causes such behavior.
So, my suggestion is to use NUnit to run parameterized tests.
Btw, Resharper 7 has better support NUnit parameterized tests. And probably this issue won't appear in the latest Resharper version.