I am looking at moving from NUnit to MbUnit for my unit testing framework as it has a couple of features that I like, one of them being the parallelizable attribute. If I mark tests with this attribute what happens
i, are all instance variables available only to their own thread or are they shared?
ii, how many tests will execute at once? Is it dependant on the number of processors/cores?
Reason for asking the first question is that I have, as a test simply swapped the Nunit framework for th MbUnit framework, and in a particular test class sets of tests tend to fail when run in parallel and pass when run sequentially. These test use variables at the class level and then setup in the [SetUp].
The tests run on a single instance of your test fixture class, so the instance fields will be shared.
By default, the degree of parallelism equals the number of CPUs you have, or 2 at a minimum.
You can use the DegreeOfParallelism attribute at the assembly level to override this.
See this blog post for details and some examples showing you how to use the various attributes.
Related
I have a selenium project written with NUnit in C# .NET 6. I have a folder called 'Tests' where there are multiple sub folders and each folder has a lot of classes. Each class has only one Test method. The reason for this is for structuring the project and each class represents one process in the software I'm testing. However, some processes need to be run after some other processes have already ran.
My question is; is there any way to run the classes in a specific order I want? I have tried using
dotnet test --filter
However this did not work. I also tried using NUnit's
Order
attribute but this works only when a class has multiple test methods.
The Order attribute may be placed on a class or a method. From the NUnit docs:
The OrderAttribute may be placed on a test method or fixture to specify the order in which tests are run within the fixture or other suite in which they are contained.
The bold italics in the citation are mine. In your case, the "other suite" containing the fixture class is the namespace in which it is defined.
There is no "global" ordering facility, but if all your tests are in the same namespace, using the OrderAttribute on the fixtures will cause them to run in the order you specify. If it doesn't interfere with any other use of the namespaces you might consider putting them all in one namespace.
A couple of notes:
The OrderAttribute specifies the order in which the tests start. If you run tests in parallel, multiple tests may run at the same time.
It's not advisable to have the tests depend on one another in most cases.
There are lots of reasons not to control the order of tests, which are covered in the answers quoted by other folks. I'm just answering the specific "how-to" question you posed.
I have got a question regarding test parallelization with parameterized fixtures and methods using NUnit 3.8.0.0, .NET 4.0 and C#.
I am currently developing a test suite that runs tests using each resource in a set of resources. Since initializing these resources is quite time consuming, and the resources do not preserve state, I want to share them between my test cases, in order to improve performance. These resource can not be used simultaneously by multiple tests.
In order to do that, I made a base fixture with a constuctor taking one argument, which makes the fixture point to the right resource. The actual test fixtures subclass from this fixture.
I am having a test fixture that uses both constuctor arguments (passed to a base class) and test case arguments. Based on the parameter, the base class initializes some resources. These resources can't be used at the same time.
Now I am trying to parallelize these test cases in such a way that the tests in the different fixtures generated by the class MyParameterizedTestFixture(V1), MyParameterizedTestFixture(V2). can run simultaniously, but tests in the same fixture can not (since the resources can not be used simultaniously). Additionally, different fixture classes may not run in parallel.
My code looks like this (with details abstracted away):
[TestFixture]
[TestFixtureSource(typeof(TestData), "FixtureParams")]
public class MyParameterizedTestFixture : BaseFixture
{
public MyParameterizedTestFixture(Resource resource) : base (resource) { }
[Test]
public void Test1() { /* run test using Resource */ }
[TestCaseSource("Params")]
public void TestParameterized(object param) { /* run test using Resource and param */ }
}
I tried to add Parallelizable(ParallelScope.Self)] to my derived fixtures, but that results in tests from different fixtures using the same resource (i.e. having the same version) fixture parameter, being run simultaniously (it works when only selecting one fixture).
Using [Parallelizable(ParallelScope.Fixture)] is definately not correct, because it will cause different fixtures to run together. Using [Parallelizable(ParallelScope.Children)] is also not correct, because it will cause different test cases in a fixture to run together, and not the different fixtures from the same class.
Now I was wondering if something like what I want could be archived by using a 'layered' approach (marking fixtures parallelizable some way, and methods in another way), or is it possible to define custom parallel scopes?
NUnit allows you to run any fixture in parallel or not. When a fixture is run in parallel, it may run at the same time as any other parallel fixture. Some way of grouping fixtures that can be run together but not with other fixtures would be a nice feature, but it's not one we have right now
When you have multiple instances of the same fixture - i.e. generic or parameterized fixtures - NUnit treats those instances exactly the same way as it treats any fixture. That is, if the fixture is parallelizable, the instances may run at the same time as any of the other instances as well as instances of different fixtures.
Similarly, you can run test cases (methods) in parallel or not. If a parallel case is contained in a non-parallel fixture, then it only runs in parallel with other methods of that fixture. If it is contained in a parallel fixture, then it can run at the same time as any other parallel method.
In other words, with the features we presently have, parallelism is basically all or nothing for each test, whether it's a suite, a fixture or a test case. This may change with enhancements in a future release.
I understand that ideal unit tests should not share any context between them but I have a dilemma: I am trying to unit test against a piece of software that only has a single license that can be instantiated at a time.
.NET unit tests seem to run in parallel so when I click "Run all tests" the classes are all run simultaneously and most fail because they can't all have a license.
So I see two questions that are mutually exclusive:
How do I share the context between C# unit testing classes?
OR How do I force .NET unit tests to NOT run in parallel?
Clarification: The licensed software is not what I'm trying to test, it's just the tool I need to DO the test
Normally I'd consider Singleton an anti-pattern since it makes unit testing impossible.
But this a good use case to have a Singleton.
A real singleton, with a private constructor and a static constructor, will run only once and will be thread-safe.
This way you can keep your tests running in parallel.
I'm not sure if this is what you might be looking for but would that work if you run all tests at the same time however each of them runs on a separate AppDomain?
For reference I used the cross domain delegates where you could pass your actual test: https://msdn.microsoft.com/en-us/library/system.appdomain.docallback(v=vs.110).aspx
Let me know if that's works!
I am approaching database testing with NUnit. As its time consuming so I don't want to run everytime.
So, I tried creating the base class and every other database testing classes derive from it as I thought if I will decorate the base class with [Ignore] attribute then rest of the derived classes will get ignored, but thats not happening.
I need to know is there any way to Ignore set of the classes with minimal effort?
If you don't want to split out integration and unit tests into separate projects you can also group tests into categories
[Test, Category("Integration")]
Most test runners allow you to filter which categories to run which would give you finer grained control if you need it (e.g. 'quick', 'slow' and 'reaaallyy slow' categories)
A recommended approach is seperating your unit tests that can run in isolation from your integration tests into different projects, then you can choose which project to execute when you run your tests. This will make it easier to run your faster running tests more often, multiple times daily or even hourly (and hopefully without ever having to worry about such things as configuration), while letting your slower running integration tests run on a different schedule.
What mocking frameworks in C# allow for parallel execution? (Thread safety) I was trying RhinoMocks, but it doesn't work well with parallel execution. These tests are not using external resources.
Background: I'm easing other developers into unit testing with MSTest and wanted to use multiple cores. That part appears to run properly.
Update:
Ok, so I originally posted the outline below because I was confident that it might be something within your tests such as shared state -- it couldn't possibly be RhinoMocks, right? right?? Well, to your point I think there's something odd with the framework. My sincere apologies!
Using Reflector to casually look at the RhinoMock source, we can see that MockRepository has an internal static field to represent the current mock repository. Looking at the usages of this field, it appears that this repository is used extensively by the dynamic proxy interceptor which strongly suggests nothing is thread-safe.
Here's an example usage that would use that static field.
LastCall.IgnoreArguments();
So, to answer your question - I would recommend you try out Moq. Based on the source, it appears that the Fluent API is static, but it's marked with [ThreadStatic], so it should be okay. Of course this comes with the caveat that you're doing the initial set up of the mocks on the current thread and not across threads.
Let me know how that works.
How to setup Parallel Test Execution in Visual Studio 2010:
This link outlines the mechanism to enable parallel test execution. In short, you need to manually edit the testsettings file and add the parallelTestCount attribute to the Execution node. Setting the parallelTestCount to 0 means that it will automatically balance the test execution across the available cores on the machine.
<TestSettings>
<!-- etc -->
<Execution parallelTestCount="0">
<!-- etc -->
</Execution>
<!-- etc -->
</TestSettings>
There are a few caveats:
It only works for standard MSTest "unit tests" (there are a variety of different tests you can create (ordered, coded ui, custom test extensions, etc). Standard unit tests are the TestClass/TestMethod variety.
It doesn't support Data Adapters. Thus, no support for parallel database driven tests
It must run locally. (You can't run tests distributed over multiple machines in parallel.)
Lastly, and this is the most important one -- the onus is on you to ensure that your tests are thread safe. This means if you use any kind of static singletons or shared-state, you're going to run into issues.