Develop Reference

Selenium - disable test and do not show in test explorer [duplicate]

Is it possible to skip all tests from a specific class like in NUnit
[TestFixture]
[Ignore("Reason")]
public class TestClass {
}

No - there is no such facility at present, and the last time it was requested it was considered too low value to add,
One quick way of achieving the effect in xUnit is to comment out the public - private classes are not reflected over (obviously it won't appear on the skip list that way though).
UPDATE: Another way is to put a TraitAttribute on the class and then (assuming you're using the xunit.console runner) filter it out by running with /-trait traitName. (e.g. you can achieve ExplicitAttribute, some aspects of the BDD frameworky technique of Pending tests and similar semantics that way - of course the big problem is they don't show up in any reports when using any of these filtering techniques)
UPDATE 2: You can do
const string skip = "Class X disabled";
[Fact(Skip=skip)]
void Test() {}
Then you can change to to const string skip = null to undo the skip. The (dis)advantage of this is that the test is still shown as a Skipped test in the test list, generally with a reason included in the test run report (vs making it private which makes it likely to be forgotten)

Here is my hack to avoid error xUnit1000: Test classes must be public (checked on single Fact, I think Theories can be hacked this way, too).
// Uncomment to enable tests
//public class FactSwitch : FactAttribute { } // public! ahh, a bug!
// Uncomment to disable tests
internal class FactSwitch : Attribute { }
public class MyTests
{
[FactSwitch]
public void MyTest1()
{
"it".ShouldBe("it");
}
}
(3 years later)
While searching for the same solution I found there are better ways to do the same.
Let's rewrite the example above in a way Ruben Bartelink suggested (continuation of his idea).
public class MyTests
{
//const string SkipOrNot = null; // Run all tests
const string SkipOrNot = "reason"; // Skip all tests
[Fact(Skip = SkipOrNot)]
public void MyTest1()
{
"it".ShouldBe("it");
}
}
Nathan Cooper suggested a good improvement for my idea:
public class MyTests
{
// Uncomment to disable tests
//private class FactAttribute : Attribute { }
[Fact]
public void MyTest1()
{
"it".ShouldBe("it");
}
}
So I like both ideas from Ruben and Nathan. There is a subtle difference between using Skip="something" (Ruben) and not using Skip at all. Using "Skip" will put all your tests in a "Skipped tests" warning zone, while "FactAttribute : Attribute" will hide them.

I've found yet another way of temporary disabling entire class without compiler warning.
Disabled:
[System.Diagnostics.CodeAnalysis.SuppressMessage("Usage", "xUnit1000:Test classes must be public", Justification = "Disabled")]//*/
/*
public /**/class DatabaseTests
{
}
to enable move the /* one line up (i.e. using alt+up):
/*
[System.Diagnostics.CodeAnalysis.SuppressMessage("Usage", "xUnit1000:Test classes must be public", Justification = "Disabled")]//*/
public /**/class DatabaseTests
{
}
Note that using full namespace path for SupressMessage does not mess up with your usings.

You need to set the your class access level as as internal and surpress message as #Miq did:
[System.Diagnostics.CodeAnalysis.SuppressMessage("Usage", "xUnit1000:Test classes must be public", Justification = "Disabled")]
internal class MyClassThatIsNotATestClass
{ ... }

You can create LocalOnlyFactAttribute
public class LocalOnlyFactAttribute : FactAttribute
{
//uncomment to run on local
//const string skip = null;
//keep this to avoid slow running tests on other env
const string skip = "Disabled slow running tests.";
public override string Skip { get => skip; set => base.Skip = value; }
}

As far as I know, the simplest way to dynamically skip a whole xUnit test class at runtime is to use the TestFrameworkAttribute at the assembly level, to point to a class that implements the ITestFramework interface (or inherits from XunitTestFramework, which is simpler) and which overrides the CreateDiscoverer() method to return another class, that implements the ITestFrameworkDiscoverer interface (or inherits from XunitTestFrameworkDiscoverer, which is simpler), where you can finally override the IsValidTestClass() method, to decide whether a class should be skipped or not.
Here is some sample code:
[assembly: TestFramework("MyNamespace.Xunit.MyTestFramework", "MyAssembly")]
namespace MyNamespace.Xunit
{
public class MyTestFramework : XunitTestFramework
{
public MyTestFramework(IMessageSink messageSink)
: base(messageSink)
{
}
protected override ITestFrameworkDiscoverer CreateDiscoverer(
IAssemblyInfo assemblyInfo)
=> new MyTestFrameworkDiscoverer(
assemblyInfo,
SourceInformationProvider,
DiagnosticMessageSink);
}
public class MyTestFrameworkDiscoverer : XunitTestFrameworkDiscoverer
{
public MyTestFrameworkDiscoverer(
IAssemblyInfo assemblyInfo,
ISourceInformationProvider sourceProvider,
IMessageSink diagnosticMessageSink,
IXunitTestCollectionFactory collectionFactory = null)
: base(
assemblyInfo,
sourceProvider,
diagnosticMessageSink,
collectionFactory)
{
}
protected override bool IsValidTestClass(ITypeInfo type)
=> base.IsValidTestClass(type) &&
FilterType(type);
protected virtual bool FilterType(ITypeInfo type)
{
// Insert your custom filter conditions here.
return true;
}
}
}
Tested with xUnit 2.4.1.
We are using it in Pomelo.EntityFrameworkCore.MySql (see AssemblyInfo.cs and MySqlXunitTestFrameworkDiscoverer.cs) (a bit more complex than the sample code here).

You could achieve this through a custom ITestClassCommand.
See http://mariangemarcano.blogspot.be/2010/12/xunitnet-running-tests-testcategory.html

Here's another hack that requires minimal changes to code
using FactAttribute = System.Runtime.CompilerServices.CompilerGeneratedAttribute;
using TheoryAttribute = System.Runtime.CompilerServices.CompilerGeneratedAttribute;
Any compatible attribute can be used for the replacement.
If you also use the InlineDataAttribute then you'll need to define a replacement as I don't think there's an existing compatible attribute.
using InlineDataAttribute = DummyDataAttribute;
[AttributeUsage(AttributeTargets.Method, AllowMultiple = true)]
internal class DummyDataAttribute : Attribute
{
public DummyDataAttribute(params object[] data)
{
}
}

Adding a reason almost after one year after the initial question. I have a set of tests which are calling real server apis, and I would like to run then on demand. With nUnit, it has Ignore attribute : with that set, test runner will skip those tests, but I can still manually run it.
xUnit has no such feature. The nearest one is setting such a class level attribute, and comment it out when I want to run it.

Consider creating LocalOnlyFactAttribute, which can be reused across multiple test files.
public class LocalOnlyFactAttribute : FactAttribute
{
//uncomment to run on local
//const string skip = null;
//keep this to avoid slow running tests on other env
const string skip = "Disabled slow running tests.";
public override string Skip { get => skip; set => this.Skip = value; }
}

Share variable value between tests in Xunit test [duplicate]

I have written the xUnit test cases in C#. That test class contains so many methods. I need to run the whole test cases in a sequence. How can I set the test case sequence in xUnit?

In xUnit 2.* this can be achieved using the TestCaseOrderer attribute to designate an ordering strategy, which can be used to reference an attribute that is annotated on each test to denote an order.
For example:
Ordering Strategy
[assembly: CollectionBehavior(DisableTestParallelization = true)]
public class PriorityOrderer : ITestCaseOrderer
{
public IEnumerable<TTestCase> OrderTestCases<TTestCase>(IEnumerable<TTestCase> testCases) where TTestCase : ITestCase
{
var sortedMethods = new SortedDictionary<int, List<TTestCase>>();
foreach (TTestCase testCase in testCases)
{
int priority = 0;
foreach (IAttributeInfo attr in testCase.TestMethod.Method.GetCustomAttributes((typeof(TestPriorityAttribute).AssemblyQualifiedName)))
priority = attr.GetNamedArgument<int>("Priority");
GetOrCreate(sortedMethods, priority).Add(testCase);
}
foreach (var list in sortedMethods.Keys.Select(priority => sortedMethods[priority]))
{
list.Sort((x, y) => StringComparer.OrdinalIgnoreCase.Compare(x.TestMethod.Method.Name, y.TestMethod.Method.Name));
foreach (TTestCase testCase in list)
yield return testCase;
}
}
static TValue GetOrCreate<TKey, TValue>(IDictionary<TKey, TValue> dictionary, TKey key) where TValue : new()
{
TValue result;
if (dictionary.TryGetValue(key, out result)) return result;
result = new TValue();
dictionary[key] = result;
return result;
}
}
Attribute
[AttributeUsage(AttributeTargets.Method, AllowMultiple = false)]
public class TestPriorityAttribute : Attribute
{
public TestPriorityAttribute(int priority)
{
Priority = priority;
}
public int Priority { get; private set; }
}
Test Cases
[TestCaseOrderer("FullNameOfOrderStrategyHere", "OrderStrategyAssemblyName")]
public class PriorityOrderExamples
{
[Fact, TestPriority(5)]
public void Test3()
{
// called third
}
[Fact, TestPriority(0)]
public void Test2()
{
// called second
}
[Fact, TestPriority(-5)]
public void Test1()
{
// called first
}
}
xUnit 2.* ordering samples here

Testpriority: at the bottom of this page.
[PrioritizedFixture]
public class MyTests
{
[Fact, TestPriority(1)]
public void FirstTest()
{
// Test code here is always run first
}
[Fact, TestPriority(2)]
public void SeccondTest()
{
// Test code here is run second
}
}
BTW, I have the same problem right now. And yes, it is not the clean art.. but QA wanted a manual test.. so an automated test with a specific order already is a big leap for them.. (cough) and yes, it is not really unit testing..

If you really have the need to prioritize your tests (probably not your unit tests) you can use Xunit.Priority.
I have used it for some integration testing and works really well and simple without the overhead of having to write your prioritization classes, for simple case scenarios

For some reason, XUnit.Priority didn't work for me. In my test cases, it wasn't running the tests in the priority order specified.
So I tried XUnitPriorityOrderer, which is similar to use but was working (To quickly test it, save the following code in a text editor as OrderedXUnitTests.linq, then open it with LinqPad 6 and execute it. Alternatively, you can also copy the TestClass to Visual Studio and add XUnit, XUnit.Runner.VisualStudio and XUnitPriorityOrderer):
<Query Kind="Program">
<NuGetReference>XUnitPriorityOrderer</NuGetReference>
<Namespace>Xunit</Namespace>
<Namespace>XUnitPriorityOrderer</Namespace>
</Query>
#load "xunit"
// using XUnitPriorityOrderer
// see: https://github.com/frederic-prusse/XUnitPriorityOrderer
void Main()
{
RunTests(); // Call RunTests() or press Alt+Shift+T to initiate testing.
}
#region private::Tests
[TestCaseOrderer(CasePriorityOrderer.TypeName, CasePriorityOrderer.AssembyName)]
public class TestClass
{
static List<string> Order { get; set; }
public TestClass()
{
Order = Order ?? new List<string>();
}
[Fact, Order(2)]
void Test_Xunit_AnotherTest()
{
Order.Add("Test_Xunit_AnotherTest");
Assert.True(3 + 1 == 4);
}
[Fact, Order(1)]
void Test_Xunit()
{
Order.Add("Test_XUnit");
Assert.True(1 + 1 == 2);
}
[Fact, Order(99)]
void Print_Order()
{
Order.Add("Print_Order");
var strOrder = string.Join(", ", Order.ToArray());
strOrder.Dump("Execution Order");
Assert.True(true);
}
}
#endregion
This will run the tests in given order (Order(1), Order(2) and then Order(99)) and will dump the executed tests finally (test method Print_Order()).

You can't, by design. It's deliberately random in order to prevent anyone getting one of those either by desire or by accident.
The randomness is only for a given Test class, so you may be able to achieve your goals by wrapping items you want to control the order of inside a nested class - but in that case, you'll still end up with random order whenever you have more than two Test Methods in a class.
If you're trying to manage the building up of fixtures or context, the built-in IUseFixture<T> mechanism may be appropriate. See the xUnit Cheat Sheet for examples.
But you really need to tell us more about what you're trying to do or we'll just have to get speculative.

Is there a way to utilize a variable defined in an nUnit [Test] in subsequent tests in a [TestFixture]?

I am converting a JMeter TestPlan into nUnit using RestSharp to consume API calls. My goal is to merge these tests into my visual studio project solution.
My problem is, many of the API calls in the JMeter's Test Plan are extracting key/values from their JSON response which are then used in subsequent test's requests.
My question is, is it possible in nUnit/RestSharp to define variables from a test's response that can be used in subsequent tests within a [TestFixture]? Or will these variables have to be redefined under every [Test]?

Use an [Order(n)] attribute to run the tests in order, and I'd just use a private object to store the variables between tests. So something like:
[TestFixture]
public class Tests
{
private int valueBeingPassed;
[OneTimeSetUp]
public void Setup()
{
valueBeingPassed = 1;
}
[Test, Order(1)]
public void Test1()
{
valueBeingPassed += 2;
Assert.AreEqual(valueBeingPassed, 3);
}
[Test, Order(2)]
public void Test2()
{
var doubleValue = valueBeingPassed * 2;
Assert.AreEqual(doubleValue, 6);
}
}

How would I achieve a range based test in MSTest [duplicate]

This question already has answers here:
How can we run a test method with multiple parameters in MSTest?
(9 answers)
Closed 6 years ago.
Using MSTest I want to run a test like this ...
var range = Enumerable.Range(0, 9);
foreach(var i in range)
{
Test(i);
}
... one theory I had was to create a new Test attribute like this ...
[TestClass]
public class CubeTests
{
[TestMethod]
[TestRange(0, 9)]
public void Test(int i)
{
// Test stuff
}
}
...
The key here is that I have some quite memory intensive code that I would like MSTest to clean up between tests for me.
For something so simple I really don't want to be relying on files and using Datasource and Deployment items.
Can this be done, if so, is anyone prepared to offer up an idea of how?

Maybe that's what you're looking for. Some years ago, Microsoft made ​​available an extension for visual studio called PEX.
PEX generate unit tests from a single parametric test, Pex finds
interesting input-output values of your methods, which you can save as
a small test suite with high code coverage.
You can use assumption and precondition for the parameters of your test, which ensure the best control of the tests generation.
Pex is no longer available(it was a research project), but is now available instead Intellitest, which still uses the same static analysis engine.
Intellitest generates a parameterized test that is modifiable and
general/global assertions can be added there. It also generates the
minimum number of inputs that maximize the code coverage; stores the
inputs as individual unit tests, each one calling the parameterized
test with a crafted input.
[PexMethod]
public void Test(int i)
{
PexAssume.IsTrue(i >= 0);
PexAssume.IsTrue(i < 10);
// Test stuff
}

You dont have to resort to built-in test runner magic. Simply add your range as a property of your test class:
private static IEnumerable<int> TestRange
{
get
{
int i = 0;
while(i < 10)
yield return i++;
}
}
now in your testmethod, you can do the for-loop as usual, using your uniquely defined testrange:
[TestMethod]
public void DoStuff_RangeIsValid_NoExceptions(){
// Act
foreach(var i in TestRange){
// do the unit test here
}
}

Implement test in separate method and call method from Test method
I think you can do this in NUnit, but I am pretty sure you can't do it in MS test.
If you want to do clean up then you can call the GC after every call, or create a TestCleanUpImpl method ( did this in snippet calling GC.Collect() to show how to force GC ).
Would suggest something like the following:
public void TestSetup()
{
//Setup tests
}
public void TestCleanUpImpl()
{
//unassign variables
//dispose disposable object
GC.Collect();
}
public void TestImpl(int i)
{
// Test stuff
// Do assert statements here
}
[TestMethod]
public void Test()
{
int fromNum = 0;
int untilNum = 9;
for(int i=fromNum;i<=untilNum;i++)
{
TestSetup();
TestImpl(i);
TestCleanUpImpl();
}
}
If you have complicated setup and clean up could possibly implement a class that handles disposing and creating, handle setup in constructor, disposal in Dispose method
I wouldn't use this as my first choice, prefer to keep my tests as simple as possible, even if my tests do violate DRY it makes them much easier to follow, which means less debugging, which is a good trade off in my opinion
public class TestImplObj : IDisposable
{
public TestImplObj()
{
//Setup test
}
public void TestImpl(int i)
{
//Do the actual test
}
public void Dispose()
{
Dispose(true);
GC.SuppressFinalize(this);
}
protected virtual void Dispose(bool disposing)
{
if (disposing)
{
// Do the clean up here
}
}
}

How do I Reset a Static Field (i.e. trigger it to be created again the next time the class is created)?

So here's a sample singleton:
public class Singleton
{
private static Singleton _instance = new Singleton();
private Singleton()
{
//do stuff
}
public static Singleton Get()
{
return _instance;
}
}
I'm trying to run multiple tests on this singleton class, and the first one always works, naturally. Basically, I'm able to see that when I create a new Singleton() the constructor is called. Or, when I call Singleton.Get() the constructor is called. The tests work fine individually, but then the static field has been set, and even if I manually set it to null, it still won't reset itself for the next test. This is a problem for me as automated testing is important here. So how can I get it to reset itself?
So far, I researched AppDomain and found that I could put each test into its own assembly and run it manually. But of course, VS already uses a unique AppDomain for each assembly which would allow for that to happen automatically, but that's a LOT of test assemblies and seems a bit silly. I'm looking for an alternative to that.
Note: I don't want advice about code quality. This is code I'm testing, not that I've written. I will not change the code until I've tested it first. I researched this thoroughly and did not find a question about this that was not answered with "You shouldn't use singletons". I am not interested in code advice. This is StackOverflow, not CodeReview.
Additional Information Requested
I am running tests currently using Visual Studio's Built-In Testing framework. I would prefer that this still worked when it is moved to MSBuild. That doesn't rule out manually triggering an external application for the tests, but it does make it harder to do.

You're misunderstanding AppDomains.
You can simply create a new AppDomain for each test, load all of the test assemblies into it, and invoke only one test method in each domain.
It would probably be rather slow, though.

You can design your Singleton class like this:
public class Singleton
{
private static Singleton _instance;
private static object _instanceLock = new object();
private Singleton()
{
//do stuff
}
public static Singleton Get()
{
if (_instance == null)
{
lock(_instanceLock)
{
if (_instance == null)
{
_instance = new Singleton();
}
}
}
return _instance;
}
public static void Clear()
{
if (_instance != null)
{
lock(_instanceLock)
{
if (_instance != null)
{
_instance = null;
}
}
}
}
}
Then you'd have to call Singleton.Clear() before you begin each test, like this:
[TestInitialize]
public void Initialize()
{
Singleton.Clear();
}

Write a console program that has a command-line argument to determine which of those singleton tests to run. Then call that multiple time from a batch file (or bash or powershell, whichever you prefer). It's extra work but it will let you test this code in a fresh new environment every time. Or you could try to figure out whether there's some option in your current tool to do that.
Perhaps something like that:
static int Main(string[] args) {
try {
int testcase = (Int32.Parse(args[0]));
RunTest(testcase);
} catch (Exception x) {
Console.WriteLine("test failed: "+x.Message);
return 1;
}
Console.WriteLine("test passed.");
return 0;
}

After numerous bits of advice from #redtuna #drch and #SLaks and lots of googling we have determined a way to do this.
Step 1: Your test class needs to inherit from MarshalByRefObject
public class UnitTest1 : MarshalByRefObject
Step 2: This next piece is a custom method to run each test in it's own AppDomain.
private void RunTestInCustomDomain(string methodName)
{
// selecting the dll from the debug directory using relative directories
var testDll = #"..\..\..\UnitTests\bin\Debug\UnitTests.dll";
// first verify the dll exists
Assert.IsTrue(File.Exists(testDll));
var assemblyName = AssemblyName.GetAssemblyName(testDll).FullName;
var domain = AppDomain.CreateDomain(methodName, null, new AppDomainSetup()
{
// This is important, you need the debug directory as your application base
ApplicationBase = Path.GetDirectoryName(testDll)
});
// create an instance of your test class
var tests = domain.CreateInstanceAndUnwrap(assemblyName, typeof(UnitTest1).FullName) as UnitTest1;
var type = tests.GetType();
var method = type.GetMethod(methodName);
// invoke the method inside custom AppDomain
method.Invoke(tests, new object[]{});
// Unload the Domain
AppDomain.Unload(domain);
}
Step 3: The next trick is that your Test Method is calling this custom method, so your tests are written into public methods without the [TestMethod] attribute.
[TestMethod]
[DeploymentItem("UnitTests.dll")]
public void TestMethod1()
{
RunTestInCustomDomain("actual_TestMethod1");
}
public void actual_TestMethod1()
{
// Assert Stuff
}
For Completeness: If you need to run initialization or cleanup for each test, they need to be called manually because the TestMethod is running in a different AppDomain from the actual_TestMethod
public void actual_TestMethod1()
{
// Assert Stuff
TestCleanup();
}
Edit: I should note that because these methods are being run under a separate AppDomain, Code Coverage doesn't work :(. If anyone finds a way to get that functionality in addition to this feature, please let me know.

Just an idea:
Setting it to null just sets your variable to null instead of what it's pointing at, of course. But what if you could get the Singleton itself... Well, it seems that that can be done using Expressions. See Getting the object out of a MemberExpression?