I want to run my selenium tests in parallel and set the following in my assembly.cs.
[assembly: Parallelizable(ParallelScope.Fixtures)]
Ok, fine. That works.
Here is a short example of the code structure
using NUnit.Framework;
namespace MyExample
{
[TestFixture]
[Category("TestsRunningWithLogin1")]
public class Test_Example1
{
[Test, Order(1)]
public void Test1()
{
}
[Test, Order(2)]
public void Test2()
{
}
}
[TestFixture]
[Category("TestsRunningWithLogin2")]
public class Test_Example2
{
[Test, Order(1)]
public void Test1()
{
}
[Test, Order(2)]
public void Test2()
{
}
}
}
The tests require a username and password and do something in a web page. The login etc. is currently handled in a OneTimeSetUp method. The webpage saves the last used view in user settings.
If I run the tests sequentially, I have no problems, due to the tests do not influence each other. All tests can run with the same username.
If I run it parallel, they could influence each other. For example test1 configures a view, what should not be seen in test2.
My idea was to run the classes (and there are a lot of them) with different users. When the test starts, it should take a username, which is currently not used by the parallel runing tests. Currently I don't know which tests are running in parallel by nunit, so I cannot parameterize them directly.
I did not find anything how to control the parallel tests. I can define, if parallel or not and how many executed in parallel. What I want is, to give the parallel running tests parameters. If I have 3 parallel running test classes, I want to give all 3 different parameters.
Any ideas how to achieve that?
What about using a singleton pattern to allocate from a set of passwords based on the threadid.
A quick explanation,
IThreadCredentials is an interface to describe the credentials, whatever they look like in your case.
ThreadCredentials is a simple class I have written that implements IThreadCredentials.
ICredentialManager is an interface to describe how credentials can be allocated and returned.
CredentialManager.Instance is the singleton that is shared amongst your fixtures to borrow and return the credentials.
public interface IThreadCredentials
{
string UserName { get; }
string Password { get; }
}
public class ThreadCredentials : IThreadCredentials
{
public ThreadCredentials(string userName, string password)
{
this.UserName = userName;
this.Password = password;
}
public string UserName { get; }
public string Password { get; }
}
public interface ICredentialManager
{
IThreadCredentials GetCredentialsFromPool();
void ReturnCredentialsToPool();
}
public sealed class CredentialManager : ICredentialManager
{
private static readonly Lazy<CredentialManager> lazy = new Lazy<CredentialManager>(() => new CredentialManager());
private static readonly object syncRoot = new object ();
private static readonly Queue<IThreadCredentials> availableCredentialQueue = new Queue<IThreadCredentials>();
private static readonly IDictionary<int, IThreadCredentials> credentialsByThread = new Dictionary<int, IThreadCredentials>();
private CredentialManager()
{
IEnumerable<IThreadCredentials> availableCredentials = new[]{new ThreadCredentials("Foo", "FooPassword"), new ThreadCredentials("Bar", "BarPassword")};
foreach (IThreadCredentials availableCredential in availableCredentials)
{
availableCredentialQueue.Enqueue(availableCredential);
}
}
public static CredentialManager Instance => lazy.Value;
public IThreadCredentials GetCredentialsFromPool()
{
return GetCredentialsFromPool(Thread.CurrentThread.ManagedThreadId);
}
public void ReturnCredentialsToPool()
{
ReturnCredentialsToPool(Thread.CurrentThread.ManagedThreadId);
}
private static IThreadCredentials GetCredentialsFromPool(int threadId)
{
lock (syncRoot)
{
IThreadCredentials result;
if (credentialsByThread.TryGetValue(threadId, out result))
{
return result;
}
// This presupposes you have enough credentials for the concurrency you are permitting
result = availableCredentialQueue.Dequeue();
credentialsByThread.Add(threadId, result);
return result;
}
}
private static void ReturnCredentialsToPool(int threadId)
{
lock (syncRoot)
{
if (credentialsByThread.ContainsKey(threadId))
{
IThreadCredentials credentials = credentialsByThread[threadId];
credentialsByThread.Remove(threadId);
availableCredentialQueue.Enqueue(credentials);
}
}
}
}
Usage:
In your test fixture setup, you can do something like:
IThreadCredentials credentials = CredentialManager.Instance.GetCredentialsFromPool();
// Now you can use credentials for whatever
In the teardown, you can then
CredentialManager.Instance.ReturnCredentialsToPool();
// Then promise you stop using those credentials
Obviously, you will need to have at least the number of credentials available as you intend on running threads in parallel or you will get an exception on dequeue.
Use nunit TestCase("data") attribute
Example:
[TestCase("differentUserName", "password")]
public void MyTest(string username, string password)
{
// Test steps
}
One possible answer without nunit would be a little service, which delivers the parameters. So each parallel test should call a webservice and will get its unique parameters. The webservice would return with each call the next parameter set.
If I would provide 10 different parameter sets, and run 3 tests in parallel, I can quite be sure, that the 3 parallel tests never get the same parameters. Assumed that all test cases needs nearly the same time.
I would call this a hack, therefore I'm asking for a nunit solution.
Related
Take this class:
public class LoginPresenter
{
private ILoginView view;
private APIWrapper api;
(...)
public virtual IEnumerator Login(string email, string password)
{
return api.Login(email, password, OnSuccess, OnError);
}
private void OnError(HttpError error)
{
switch (error.statusCode)
{
case 0:
view.ShowMessage("Check your Internet Connection");
break;
default:
view.ShowMessage("Invalid Credentials");
break;
}
}
(...)
}
I need to write a test for when Login fails, but I'm not sure about the right way to do it. This is how I was asked to do it:
public class LoginPresenterTests
{
private LoginPresenter presenter;
private ILoginView view;
(...)
[Test]
public void _03_Test_LoginOnError()
{
//Arrange
Dictionary<int, string> statusCodeMessages = new Dictionary<int, string>()
{
{0, "Check your Internet Connection"},
{401, "Invalid Credentials"}
};
//Act
foreach (var statusCodeMessage in statusCodeMessages)
{
object[] args = { new HttpError(statusCodeMessage.Key, "", "") };
ReflectionUtils.Invoke(presenter, "OnError", args);
//Assert
view.Received().ShowMessage(statusCodeMessage.Value);
}
}
(...)
}
But it seems so wrong to me. This is nothing but a mirror of the original function. A single change to the message string would break the test. Is this really what testing is about? Shouldn't I just be checking whether OnError is called if login fails, nothing more?
Then again, given that OnError is a callback and a private method, I don't know how to check if it was called...
You are right, it is generally not considered good practice to invoke private methods using reflection when testing.
For this answer, I will assume you are writing a unit test for LoginPresenter, and so you should not be concerned with the behaviour of ApiWrapper or the View. The approach would be different if you were writing an integration test.
I assume the ApiWrapper class is part of your codebase? (Sorry I'm not familiar with Unity). If so, you could create an interface for ApiWrapper, then have LoginPresenter reference the interface instead of the concrete class. The following technique may also be possible without the interface, as long as the Login method is virtual.
You're missing the part of the code where the view and api are instantiated or, more likely, injected into the presenter via the constructor. Ideally, the class looks like this:
public class LoginPresenter
{
private ILoginView view;
private IAPIWrapper api;
public LoginPresenter(ILoginView view, IAPIWrapper api)
{
this.api = api;
this.view = view;
}
public virtual IEnumerator Login(string email, string password)
{
return api.Login(email, password, OnSuccess, OnError);
}
// etc
Next, in your test, you can use a mocking framework (such as NSubstitute) to pass a mock of your api and view into the LoginPresenter. Then, you mock the api.Login method, and tell it to call its fourth argument (OnError) when it executes. How exactly you do this will depend on your mocking framework.
Finally, you assert that the view received the ShowMessage call, in the same way as you are doing already.
I have written the xUnit test cases in C#. That test class contains so many methods. I need to run the whole test cases in a sequence. How can I set the test case sequence in xUnit?
In xUnit 2.* this can be achieved using the TestCaseOrderer attribute to designate an ordering strategy, which can be used to reference an attribute that is annotated on each test to denote an order.
For example:
Ordering Strategy
[assembly: CollectionBehavior(DisableTestParallelization = true)]
public class PriorityOrderer : ITestCaseOrderer
{
public IEnumerable<TTestCase> OrderTestCases<TTestCase>(IEnumerable<TTestCase> testCases) where TTestCase : ITestCase
{
var sortedMethods = new SortedDictionary<int, List<TTestCase>>();
foreach (TTestCase testCase in testCases)
{
int priority = 0;
foreach (IAttributeInfo attr in testCase.TestMethod.Method.GetCustomAttributes((typeof(TestPriorityAttribute).AssemblyQualifiedName)))
priority = attr.GetNamedArgument<int>("Priority");
GetOrCreate(sortedMethods, priority).Add(testCase);
}
foreach (var list in sortedMethods.Keys.Select(priority => sortedMethods[priority]))
{
list.Sort((x, y) => StringComparer.OrdinalIgnoreCase.Compare(x.TestMethod.Method.Name, y.TestMethod.Method.Name));
foreach (TTestCase testCase in list)
yield return testCase;
}
}
static TValue GetOrCreate<TKey, TValue>(IDictionary<TKey, TValue> dictionary, TKey key) where TValue : new()
{
TValue result;
if (dictionary.TryGetValue(key, out result)) return result;
result = new TValue();
dictionary[key] = result;
return result;
}
}
Attribute
[AttributeUsage(AttributeTargets.Method, AllowMultiple = false)]
public class TestPriorityAttribute : Attribute
{
public TestPriorityAttribute(int priority)
{
Priority = priority;
}
public int Priority { get; private set; }
}
Test Cases
[TestCaseOrderer("FullNameOfOrderStrategyHere", "OrderStrategyAssemblyName")]
public class PriorityOrderExamples
{
[Fact, TestPriority(5)]
public void Test3()
{
// called third
}
[Fact, TestPriority(0)]
public void Test2()
{
// called second
}
[Fact, TestPriority(-5)]
public void Test1()
{
// called first
}
}
xUnit 2.* ordering samples here
Testpriority: at the bottom of this page.
[PrioritizedFixture]
public class MyTests
{
[Fact, TestPriority(1)]
public void FirstTest()
{
// Test code here is always run first
}
[Fact, TestPriority(2)]
public void SeccondTest()
{
// Test code here is run second
}
}
BTW, I have the same problem right now. And yes, it is not the clean art.. but QA wanted a manual test.. so an automated test with a specific order already is a big leap for them.. (cough) and yes, it is not really unit testing..
If you really have the need to prioritize your tests (probably not your unit tests) you can use Xunit.Priority.
I have used it for some integration testing and works really well and simple without the overhead of having to write your prioritization classes, for simple case scenarios
For some reason, XUnit.Priority didn't work for me. In my test cases, it wasn't running the tests in the priority order specified.
So I tried XUnitPriorityOrderer, which is similar to use but was working (To quickly test it, save the following code in a text editor as OrderedXUnitTests.linq, then open it with LinqPad 6 and execute it. Alternatively, you can also copy the TestClass to Visual Studio and add XUnit, XUnit.Runner.VisualStudio and XUnitPriorityOrderer):
<Query Kind="Program">
<NuGetReference>XUnitPriorityOrderer</NuGetReference>
<Namespace>Xunit</Namespace>
<Namespace>XUnitPriorityOrderer</Namespace>
</Query>
#load "xunit"
// using XUnitPriorityOrderer
// see: https://github.com/frederic-prusse/XUnitPriorityOrderer
void Main()
{
RunTests(); // Call RunTests() or press Alt+Shift+T to initiate testing.
}
#region private::Tests
[TestCaseOrderer(CasePriorityOrderer.TypeName, CasePriorityOrderer.AssembyName)]
public class TestClass
{
static List<string> Order { get; set; }
public TestClass()
{
Order = Order ?? new List<string>();
}
[Fact, Order(2)]
void Test_Xunit_AnotherTest()
{
Order.Add("Test_Xunit_AnotherTest");
Assert.True(3 + 1 == 4);
}
[Fact, Order(1)]
void Test_Xunit()
{
Order.Add("Test_XUnit");
Assert.True(1 + 1 == 2);
}
[Fact, Order(99)]
void Print_Order()
{
Order.Add("Print_Order");
var strOrder = string.Join(", ", Order.ToArray());
strOrder.Dump("Execution Order");
Assert.True(true);
}
}
#endregion
This will run the tests in given order (Order(1), Order(2) and then Order(99)) and will dump the executed tests finally (test method Print_Order()).
You can't, by design. It's deliberately random in order to prevent anyone getting one of those either by desire or by accident.
The randomness is only for a given Test class, so you may be able to achieve your goals by wrapping items you want to control the order of inside a nested class - but in that case, you'll still end up with random order whenever you have more than two Test Methods in a class.
If you're trying to manage the building up of fixtures or context, the built-in IUseFixture<T> mechanism may be appropriate. See the xUnit Cheat Sheet for examples.
But you really need to tell us more about what you're trying to do or we'll just have to get speculative.
I am writing a testing framework for my system, which allows users to create mocked inputs to the system. The system manipulates that input, and updates some of its members for later post processing.
In order to allow users to mock the input, I need to be able to update a mocked input's members. Furthermore, the input may not be a mock, so I would prefer a solution oblivious to the type of element received.
Simply put, I have a function which receives an object and attempts to set one of its properties:
func(object a)
a.m = 5;
Which I want to test by mocking its input a, using the Moq library. Unfortunately, my test failed, since mocked objects' members need to be set using Mock.SetUpGet, instead of standard member assignment.
What would be a good way to test such a function, without changing it?
In order to be able to mock and test it, the property must be virtual, but if that's the case you can use the SetupSet method, rather than SetupGet:
public class A
{
public virtual int m { get; set; }
}
[TestFixture]
public class Tests
{
public static void SetProperty(A a)
{
a.m = 5;
}
[Test]
public void IsThePropertySet()
{
var x = new Mock<A>();
x.SetupSet<int>(a => a.m = 5).Verifiable();
SetProperty(x.Object);
x.Verify();
}
}
Your function should have getters and setters anyway.
And a good way of testing whether your functions work is to do a small-scale test. Initialize an object and send that in to your method, seeing whether it does what you want.
public class myClass
{
public int number;
public void setNumber(int nNum)
{
number = nNum;
}
public int getNumber()
{
return number;
}
}
class Program
{
static void myMethod(myClass obj, int nNumb)
{
//obj.setNumber(nNumb);
obj.number = nNumb;
}
static void Main(string[] args)
{
myClass myObj = new myClass();
myMethod(myObj, 3);
//Console.WriteLine(myObj.getNumber());
Console.WriteLine(myObj.number);
Console.ReadLine();
}
}
}
Output: 3
I'm planning to store all test cases in a excel file with columns indicate test method names, parameters and expected results; however, I found TestCaseSource simply assigns all test cases to every test method. I'm wondering that is there any way I can make NUnit select test cases for methods base on method names I put in the spreadsheet?
Thanks.
There is a way to do this.
For example, as you mentioned, you can create a custom attribute.
The idea is to pass name of test to TestCaseSource.
You can do it by creating TestCaseSource as separate class.
First, TestCaseSource class:
public class SpreadSheetTestCaseSource
{
[ThreadStatic]
public static string TestName = String.Empty;
public static IEnumerable TestCases
{
get
{
SpreadSheetTestCaseProvider.GetTestCases()
.Where(testCase => testCase.TestName == TestName);
}
}
}
Then attribute:
public class MyTestCaseSourceAttribute : TestCaseSourceAttribute
{
public MyTestCaseSourceAttribute(Type sourceType, string sourceName,
[CallerMemberName] string name = null)
: base(sourceType, sourceName)
{
SpreadSheetTestCaseSource.TestName = name;
}
//Another two members impl.
}
And test:
[TestFixture]
public class TestClass
{
[MyTestCaseSource(typeof(SpreadSheetTestCaseSource), "TestCases")]
public void TestMethod()
{
//Test logic
}
}
SpeadSheetTestCaseSource.TestName is thread static. So you can run tests parallel.
This isn't a feature that NUnit supports directly. The various TestCaseSource type attributes have no ability to feed a test method based on the input.
An option would be to create a TestCaseSource for each of your test methods. Each of these would be a simple wrapper that passes in the method name to a single internal method. That internal method would read in the Excel file and only return the rows for the given method name.
PseudoCode;
[TestCaseSource(nameof(TestMethodOneSource))]
public void TestMethodOne(int x, int y, int expected)
{
Assert.That(x + y, Is.EqualTo(expected));
}
public static IEnumerable<object[]> TestMethodOneSource() =>
ReadExcel(nameof(TestMethodOne));
private static IEnumerable<object[]> ReadExcel(string method)
{
// Open and start reading Excel
for(var row in rows)
{
if(row[0] == method)
{
// Return objects minus the method
yield return new [] {row[1], ..., row[n]};
}
}
}
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 8 years ago.
Improve this question
So I have been working in an organization that puts a fair amount of pressure on devs to write and maintain unit tests. While it's not something I've done a whole lot in the past, I like the idea and believe any serious project should have some level of unit testing, especially for self container class libraries that lend themselves to such testing.
However, I've also found that what was once very simple, readable code is made into a monstrosity of factories and interfaces. At the simplest case, a service wrapper:
No unit tests
class ShazamService
{
private string url;
public ShazamService(string url) { this.url = url; }
string IdentifySong(byte [] mp3Data)
{
return HttpHelper.Upload(url, mp3Data).Response;
}
}
class Program
{
public static int Main(string [] args)
{
var svc = new ShazamService("http://www.shazam.com");
Console.Writeline(svc.IdentifySong(args[0].ToByteArray());
}
}
Unit testable version
public interface IShazamService
{
public string IdentifySong(byte [] mp3Data);
}
public class ShazamClassFactory
{
private string url;
public ShazamClassFactory(string url) { this.url = url; }
public IShazamService GetInstance(bool test)
{
if (test)
{
return new ShazamServiceTest(this.url);
}
else
{
return new ShazamService(this.url);
}
}
class ShazamService
{
private string url;
public ShazamService(string url) { this.url = url; }
string IdentifySong(byte [] mp3Data)
{
return HttpHelper.Upload(url, mp3Data).Response;
}
}
class Program
{
public static int Main(string [] args)
{
var factory = new ShazamClassFactory("http://www.shazam.com");
var svc = factory.GetInstance(false);
Console.Writeline(svc.IdentifySong(args[0].ToByteArray());
}
}
Not only is the code significantly longer in the 2nd one, but (to me) it's less clear - from Main I don't even know the type of the return value from CreateInstance if I need to look at an implementation detail, so I can't even F12 through the logic as easily. Also what would have been 1 file for the service now becomes 4 (factory, interface, 2 implementations), with header, documentation, etc. Lastly, if I decide I want to change the constructor from string url to string url, SongGenre genre, I now need to check out, update, and check in 4 separate files, updating constructors, datamembers, documentation, etc for each.
Is this method of promoting unit testing the norm? Are there less intrusive options? And, is it worth it? To my mind, by complicating the code, you increase dev time, and make errors more likely to sneak in, all for unit testing using fake objects that will only sorta-kinda test the code you're using.
The code is unclear because it is badly written.
Dependency injection is done by injecting the class you want in a setter or a constructor, not by hardcoding the different options and using a GetInstance(bool) method to get your testing action.
Instead it should look more like this:
public class ShazamClassFactory
{
private string url;
private IShazamService _shazamService;
public ShazamClassFactory(string url) { this.url = url; }
public void SetShazamService(IShazamService service) {
_shazamService = service;
}
public string GetSong(){
return _shazamService.IdentifySong(url.ToByteArray());
}
}
Now you can use it like this:
var factory = new ShazamClassFactory("http://www.shazam.com");
factory.SetShazamService(new ShazamTestService());
var song = factory.GetSong();
The problem I see here is that it's not immediately clear what you're trying to test.
If you are writing code that uses a ShazamService then you can pass either a concrete implementation or a test implementation, depending on whether it's a unit test or not.
The use of a factory should be used if you need to control when an object gets created, and should not (imo) be the default pattern when passing in dependencies.
For your instance, a better option could be.
Service Interface
public interface IShazamService
{
string IdentifySong(byte [] mp3Data);
}
Actual Live Interface
public class LiveShazamService : IShazamService
{
private readonly string _url;
public LiveShazamService(string url)
{
_url = url;
}
public string IdentifySong(byte [] mp3Data)
{
return HttpHelper.Upload(url, mp3Data).Response;
}
}
Test Interface (probably lives in your test project)
public class MockShazamService : IShazamService
{
private readonly string _testData;
public LiveShazamService(string testData)
{
_testData = testData;
}
public string IdentifySong(byte [] mp3Data)
{
return _testData;
}
}
Test Code
[Test]
public void ShouldParseTitleOfSong()
{
// arrange
var shazamService = new MockShazamService(
"<html><title>Bon Jovi - Shock to the Heart</title></html>");
var parser = new ShazamMp3Parser(shazamService);
// act
// this is just dummy input,
// we're not testing input in this specific test
var result = parser.Parse(new byte[0]);
// assert
Assert.AreEqual("Bon Jovi - Shock to the Heart", result.Title);
}
Production Code
public class ShazamMp3Parser
{
private readonly IShazamService _shazamService;
public ShazamMp3Parser(IShazamService shazamService)
{
_shazamService = shazamService;
}
public ShazamParserResult Parse(byte[] mp3Data)
{
var rawText = _shazamService.IdentifySong(mp3Data);
// bla bla bla (up to the viewer to implement properly)
var title = rawText.SubString(24, 50);
return new ShazamParserResult { Title = title };
}
}
Usage of Production Code
public static int Main(string [] args)
{
var service = new LiveShazamService("http://www.shazam.com");
var parser = new ShazamMp3Parser(service);
var mp3Data = args[0].ToByteArray();
Console.Writeline(parser.Parse(mp3Data).Title);
}
In this example, I am showing how to test code that depends upon the IShazamService (the ShazamMp3Parser), this lets you unit test the parsing of the title without having to make an internet connection and pull live data. The mock service lets you simulate data and unit test how your parsing code works.
I did not implement the factory as I don't feel it's necessary in this scenario, but if you wanted to control when the service is instantiated, you could write a factory interface, followed by two implementations, one that constructs the live service and one that constructs the test one.
If you get brave later on, or you get sick of writing mock classes all over the place, you can use a mocking framework (like moq) to make your unit test writing faster.
[Test]
public void ShouldParseTitleOfSong()
{
// arrange
var mockShazamService = new Mock<IShazamService>();
mockShazamService.Setup(x => x.IdentifySong(It.IsAny<byte[]>()))
.Returns("<html><title>Bon Jovi - Shock to the Heart</title></html>");
var parser = new ShazamMp3Parser(mockShazamService.Object);
// act
var result = parser.Parse(new byte[0]);
// assert
Assert.AreEqual("Bon Jovi - Shock to the Heart", result.Title);
}
I think what you're looking for is an abstract factory. By providing an interface that abstract the factory itself, you can pass around a factory that creates test object or a factory that creates real objects and not have to instrument your code.