Develop Reference

Selenium - disable test and do not show in test explorer [duplicate]

Is it possible to skip all tests from a specific class like in NUnit
[TestFixture]
[Ignore("Reason")]
public class TestClass {
}

No - there is no such facility at present, and the last time it was requested it was considered too low value to add,
One quick way of achieving the effect in xUnit is to comment out the public - private classes are not reflected over (obviously it won't appear on the skip list that way though).
UPDATE: Another way is to put a TraitAttribute on the class and then (assuming you're using the xunit.console runner) filter it out by running with /-trait traitName. (e.g. you can achieve ExplicitAttribute, some aspects of the BDD frameworky technique of Pending tests and similar semantics that way - of course the big problem is they don't show up in any reports when using any of these filtering techniques)
UPDATE 2: You can do
const string skip = "Class X disabled";
[Fact(Skip=skip)]
void Test() {}
Then you can change to to const string skip = null to undo the skip. The (dis)advantage of this is that the test is still shown as a Skipped test in the test list, generally with a reason included in the test run report (vs making it private which makes it likely to be forgotten)

Here is my hack to avoid error xUnit1000: Test classes must be public (checked on single Fact, I think Theories can be hacked this way, too).
// Uncomment to enable tests
//public class FactSwitch : FactAttribute { } // public! ahh, a bug!
// Uncomment to disable tests
internal class FactSwitch : Attribute { }
public class MyTests
{
[FactSwitch]
public void MyTest1()
{
"it".ShouldBe("it");
}
}
(3 years later)
While searching for the same solution I found there are better ways to do the same.
Let's rewrite the example above in a way Ruben Bartelink suggested (continuation of his idea).
public class MyTests
{
//const string SkipOrNot = null; // Run all tests
const string SkipOrNot = "reason"; // Skip all tests
[Fact(Skip = SkipOrNot)]
public void MyTest1()
{
"it".ShouldBe("it");
}
}
Nathan Cooper suggested a good improvement for my idea:
public class MyTests
{
// Uncomment to disable tests
//private class FactAttribute : Attribute { }
[Fact]
public void MyTest1()
{
"it".ShouldBe("it");
}
}
So I like both ideas from Ruben and Nathan. There is a subtle difference between using Skip="something" (Ruben) and not using Skip at all. Using "Skip" will put all your tests in a "Skipped tests" warning zone, while "FactAttribute : Attribute" will hide them.

I've found yet another way of temporary disabling entire class without compiler warning.
Disabled:
[System.Diagnostics.CodeAnalysis.SuppressMessage("Usage", "xUnit1000:Test classes must be public", Justification = "Disabled")]//*/
/*
public /**/class DatabaseTests
{
}
to enable move the /* one line up (i.e. using alt+up):
/*
[System.Diagnostics.CodeAnalysis.SuppressMessage("Usage", "xUnit1000:Test classes must be public", Justification = "Disabled")]//*/
public /**/class DatabaseTests
{
}
Note that using full namespace path for SupressMessage does not mess up with your usings.

You need to set the your class access level as as internal and surpress message as #Miq did:
[System.Diagnostics.CodeAnalysis.SuppressMessage("Usage", "xUnit1000:Test classes must be public", Justification = "Disabled")]
internal class MyClassThatIsNotATestClass
{ ... }

You can create LocalOnlyFactAttribute
public class LocalOnlyFactAttribute : FactAttribute
{
//uncomment to run on local
//const string skip = null;
//keep this to avoid slow running tests on other env
const string skip = "Disabled slow running tests.";
public override string Skip { get => skip; set => base.Skip = value; }
}

As far as I know, the simplest way to dynamically skip a whole xUnit test class at runtime is to use the TestFrameworkAttribute at the assembly level, to point to a class that implements the ITestFramework interface (or inherits from XunitTestFramework, which is simpler) and which overrides the CreateDiscoverer() method to return another class, that implements the ITestFrameworkDiscoverer interface (or inherits from XunitTestFrameworkDiscoverer, which is simpler), where you can finally override the IsValidTestClass() method, to decide whether a class should be skipped or not.
Here is some sample code:
[assembly: TestFramework("MyNamespace.Xunit.MyTestFramework", "MyAssembly")]
namespace MyNamespace.Xunit
{
public class MyTestFramework : XunitTestFramework
{
public MyTestFramework(IMessageSink messageSink)
: base(messageSink)
{
}
protected override ITestFrameworkDiscoverer CreateDiscoverer(
IAssemblyInfo assemblyInfo)
=> new MyTestFrameworkDiscoverer(
assemblyInfo,
SourceInformationProvider,
DiagnosticMessageSink);
}
public class MyTestFrameworkDiscoverer : XunitTestFrameworkDiscoverer
{
public MyTestFrameworkDiscoverer(
IAssemblyInfo assemblyInfo,
ISourceInformationProvider sourceProvider,
IMessageSink diagnosticMessageSink,
IXunitTestCollectionFactory collectionFactory = null)
: base(
assemblyInfo,
sourceProvider,
diagnosticMessageSink,
collectionFactory)
{
}
protected override bool IsValidTestClass(ITypeInfo type)
=> base.IsValidTestClass(type) &&
FilterType(type);
protected virtual bool FilterType(ITypeInfo type)
{
// Insert your custom filter conditions here.
return true;
}
}
}
Tested with xUnit 2.4.1.
We are using it in Pomelo.EntityFrameworkCore.MySql (see AssemblyInfo.cs and MySqlXunitTestFrameworkDiscoverer.cs) (a bit more complex than the sample code here).

You could achieve this through a custom ITestClassCommand.
See http://mariangemarcano.blogspot.be/2010/12/xunitnet-running-tests-testcategory.html

Here's another hack that requires minimal changes to code
using FactAttribute = System.Runtime.CompilerServices.CompilerGeneratedAttribute;
using TheoryAttribute = System.Runtime.CompilerServices.CompilerGeneratedAttribute;
Any compatible attribute can be used for the replacement.
If you also use the InlineDataAttribute then you'll need to define a replacement as I don't think there's an existing compatible attribute.
using InlineDataAttribute = DummyDataAttribute;
[AttributeUsage(AttributeTargets.Method, AllowMultiple = true)]
internal class DummyDataAttribute : Attribute
{
public DummyDataAttribute(params object[] data)
{
}
}

Adding a reason almost after one year after the initial question. I have a set of tests which are calling real server apis, and I would like to run then on demand. With nUnit, it has Ignore attribute : with that set, test runner will skip those tests, but I can still manually run it.
xUnit has no such feature. The nearest one is setting such a class level attribute, and comment it out when I want to run it.

Consider creating LocalOnlyFactAttribute, which can be reused across multiple test files.
public class LocalOnlyFactAttribute : FactAttribute
{
//uncomment to run on local
//const string skip = null;
//keep this to avoid slow running tests on other env
const string skip = "Disabled slow running tests.";
public override string Skip { get => skip; set => this.Skip = value; }
}

Run nunit tests parallel with parameters (nunit 3.8.x)

I want to run my selenium tests in parallel and set the following in my assembly.cs.
[assembly: Parallelizable(ParallelScope.Fixtures)]
Ok, fine. That works.
Here is a short example of the code structure
using NUnit.Framework;
namespace MyExample
{
[TestFixture]
[Category("TestsRunningWithLogin1")]
public class Test_Example1
{
[Test, Order(1)]
public void Test1()
{
}
[Test, Order(2)]
public void Test2()
{
}
}
[TestFixture]
[Category("TestsRunningWithLogin2")]
public class Test_Example2
{
[Test, Order(1)]
public void Test1()
{
}
[Test, Order(2)]
public void Test2()
{
}
}
}
The tests require a username and password and do something in a web page. The login etc. is currently handled in a OneTimeSetUp method. The webpage saves the last used view in user settings.
If I run the tests sequentially, I have no problems, due to the tests do not influence each other. All tests can run with the same username.
If I run it parallel, they could influence each other. For example test1 configures a view, what should not be seen in test2.
My idea was to run the classes (and there are a lot of them) with different users. When the test starts, it should take a username, which is currently not used by the parallel runing tests. Currently I don't know which tests are running in parallel by nunit, so I cannot parameterize them directly.
I did not find anything how to control the parallel tests. I can define, if parallel or not and how many executed in parallel. What I want is, to give the parallel running tests parameters. If I have 3 parallel running test classes, I want to give all 3 different parameters.
Any ideas how to achieve that?

What about using a singleton pattern to allocate from a set of passwords based on the threadid.
A quick explanation,
IThreadCredentials is an interface to describe the credentials, whatever they look like in your case.
ThreadCredentials is a simple class I have written that implements IThreadCredentials.
ICredentialManager is an interface to describe how credentials can be allocated and returned.
CredentialManager.Instance is the singleton that is shared amongst your fixtures to borrow and return the credentials.
public interface IThreadCredentials
{
string UserName { get; }
string Password { get; }
}
public class ThreadCredentials : IThreadCredentials
{
public ThreadCredentials(string userName, string password)
{
this.UserName = userName;
this.Password = password;
}
public string UserName { get; }
public string Password { get; }
}
public interface ICredentialManager
{
IThreadCredentials GetCredentialsFromPool();
void ReturnCredentialsToPool();
}
public sealed class CredentialManager : ICredentialManager
{
private static readonly Lazy<CredentialManager> lazy = new Lazy<CredentialManager>(() => new CredentialManager());
private static readonly object syncRoot = new object ();
private static readonly Queue<IThreadCredentials> availableCredentialQueue = new Queue<IThreadCredentials>();
private static readonly IDictionary<int, IThreadCredentials> credentialsByThread = new Dictionary<int, IThreadCredentials>();
private CredentialManager()
{
IEnumerable<IThreadCredentials> availableCredentials = new[]{new ThreadCredentials("Foo", "FooPassword"), new ThreadCredentials("Bar", "BarPassword")};
foreach (IThreadCredentials availableCredential in availableCredentials)
{
availableCredentialQueue.Enqueue(availableCredential);
}
}
public static CredentialManager Instance => lazy.Value;
public IThreadCredentials GetCredentialsFromPool()
{
return GetCredentialsFromPool(Thread.CurrentThread.ManagedThreadId);
}
public void ReturnCredentialsToPool()
{
ReturnCredentialsToPool(Thread.CurrentThread.ManagedThreadId);
}
private static IThreadCredentials GetCredentialsFromPool(int threadId)
{
lock (syncRoot)
{
IThreadCredentials result;
if (credentialsByThread.TryGetValue(threadId, out result))
{
return result;
}
// This presupposes you have enough credentials for the concurrency you are permitting
result = availableCredentialQueue.Dequeue();
credentialsByThread.Add(threadId, result);
return result;
}
}
private static void ReturnCredentialsToPool(int threadId)
{
lock (syncRoot)
{
if (credentialsByThread.ContainsKey(threadId))
{
IThreadCredentials credentials = credentialsByThread[threadId];
credentialsByThread.Remove(threadId);
availableCredentialQueue.Enqueue(credentials);
}
}
}
}
Usage:
In your test fixture setup, you can do something like:
IThreadCredentials credentials = CredentialManager.Instance.GetCredentialsFromPool();
// Now you can use credentials for whatever
In the teardown, you can then
CredentialManager.Instance.ReturnCredentialsToPool();
// Then promise you stop using those credentials
Obviously, you will need to have at least the number of credentials available as you intend on running threads in parallel or you will get an exception on dequeue.

Use nunit TestCase("data") attribute
Example:
[TestCase("differentUserName", "password")]
public void MyTest(string username, string password)
{
// Test steps
}

One possible answer without nunit would be a little service, which delivers the parameters. So each parallel test should call a webservice and will get its unique parameters. The webservice would return with each call the next parameter set.
If I would provide 10 different parameter sets, and run 3 tests in parallel, I can quite be sure, that the 3 parallel tests never get the same parameters. Assumed that all test cases needs nearly the same time.
I would call this a hack, therefore I'm asking for a nunit solution.

Does NUnit provide a multiple enumeration helper?

In my company code base we have a bunch of extension method for IEnumerable object.
Some of this method enumerate multiple time the parameter given in entry.
Of course, we don't want the so I'm going to fix those methods.
But first I would like to write some unit test to detect the multiple enumeration.
I come to an implementation of IEnumerable that provide enumeration count information :
public class TestableEnumerable<T> : IEnumerable<T>
{
readonly T[] values;
public TestableEnumerable(T[] values)
{
this.values = values;
}
IEnumerator IEnumerable.GetEnumerator() => GetEnumerator();
public IEnumerator<T> GetEnumerator()
{
EnumerationCount++;
foreach (var value in values)
{
yield return value;
}
}
public int EnumerationCount { get; private set; }
}
The test look like this:
[TestFixture]
public class EnumerableExtensionsTests
{
TestableEnumerable<int> sut;
[SetUp]
public void SetUp()
{
sut = new TestableEnumerable<int>(new[] { -10, 0, 10 });
}
[Test]
public void Append_Enumerate_Once()
{
var result = sut.Append(1).ToArray();
Assert.That(sut.EnumerationCount, Is.EqualTo(1));
}
}
Does NUnit provide any mechanism to accomplish this more lightly ?

If you looks only for GetEnumerator method calls count, you can use any mock framework to create fake enumerable. There is NUnit.Mock framework. But now it's no longer being developed and NUnit project uses NSubstitute.
var fakeEnumerable = Substitute.For<IEnumerable<int>>();
fakeEnumerable.GetEnumerator().Returns(Substitute.For<IEnumerator<int>>());
fakeEnumerable.ToArray();
fakeEnumerable.ToArray();
fakeEnumerable.Received(2).GetEnumerator();
Also there is another way to enumerate collection twice. It can be done by using IEnumerator.Reset metod.
So it may worth to check Reset method calls too.

Testing member assignment using mocks in c#

I am writing a testing framework for my system, which allows users to create mocked inputs to the system. The system manipulates that input, and updates some of its members for later post processing.
In order to allow users to mock the input, I need to be able to update a mocked input's members. Furthermore, the input may not be a mock, so I would prefer a solution oblivious to the type of element received.
Simply put, I have a function which receives an object and attempts to set one of its properties:
func(object a)
a.m = 5;
Which I want to test by mocking its input a, using the Moq library. Unfortunately, my test failed, since mocked objects' members need to be set using Mock.SetUpGet, instead of standard member assignment.
What would be a good way to test such a function, without changing it?

In order to be able to mock and test it, the property must be virtual, but if that's the case you can use the SetupSet method, rather than SetupGet:
public class A
{
public virtual int m { get; set; }
}
[TestFixture]
public class Tests
{
public static void SetProperty(A a)
{
a.m = 5;
}
[Test]
public void IsThePropertySet()
{
var x = new Mock<A>();
x.SetupSet<int>(a => a.m = 5).Verifiable();
SetProperty(x.Object);
x.Verify();
}
}

Your function should have getters and setters anyway.
And a good way of testing whether your functions work is to do a small-scale test. Initialize an object and send that in to your method, seeing whether it does what you want.
public class myClass
{
public int number;
public void setNumber(int nNum)
{
number = nNum;
}
public int getNumber()
{
return number;
}
}
class Program
{
static void myMethod(myClass obj, int nNumb)
{
//obj.setNumber(nNumb);
obj.number = nNumb;
}
static void Main(string[] args)
{
myClass myObj = new myClass();
myMethod(myObj, 3);
//Console.WriteLine(myObj.getNumber());
Console.WriteLine(myObj.number);
Console.ReadLine();
}
}
}
Output: 3

NUnit: TestCaseSource assigns tests to specific test methods

I'm planning to store all test cases in a excel file with columns indicate test method names, parameters and expected results; however, I found TestCaseSource simply assigns all test cases to every test method. I'm wondering that is there any way I can make NUnit select test cases for methods base on method names I put in the spreadsheet?
Thanks.

There is a way to do this.
For example, as you mentioned, you can create a custom attribute.
The idea is to pass name of test to TestCaseSource.
You can do it by creating TestCaseSource as separate class.
First, TestCaseSource class:
public class SpreadSheetTestCaseSource
{
[ThreadStatic]
public static string TestName = String.Empty;
public static IEnumerable TestCases
{
get
{
SpreadSheetTestCaseProvider.GetTestCases()
.Where(testCase => testCase.TestName == TestName);
}
}
}
Then attribute:
public class MyTestCaseSourceAttribute : TestCaseSourceAttribute
{
public MyTestCaseSourceAttribute(Type sourceType, string sourceName,
[CallerMemberName] string name = null)
: base(sourceType, sourceName)
{
SpreadSheetTestCaseSource.TestName = name;
}
//Another two members impl.
}
And test:
[TestFixture]
public class TestClass
{
[MyTestCaseSource(typeof(SpreadSheetTestCaseSource), "TestCases")]
public void TestMethod()
{
//Test logic
}
}
SpeadSheetTestCaseSource.TestName is thread static. So you can run tests parallel.

This isn't a feature that NUnit supports directly. The various TestCaseSource type attributes have no ability to feed a test method based on the input.
An option would be to create a TestCaseSource for each of your test methods. Each of these would be a simple wrapper that passes in the method name to a single internal method. That internal method would read in the Excel file and only return the rows for the given method name.
PseudoCode;
[TestCaseSource(nameof(TestMethodOneSource))]
public void TestMethodOne(int x, int y, int expected)
{
Assert.That(x + y, Is.EqualTo(expected));
}
public static IEnumerable<object[]> TestMethodOneSource() =>
ReadExcel(nameof(TestMethodOne));
private static IEnumerable<object[]> ReadExcel(string method)
{
// Open and start reading Excel
for(var row in rows)
{
if(row[0] == method)
{
// Return objects minus the method
yield return new [] {row[1], ..., row[n]};
}
}
}