We have some UI integration tests that can't be run on the build server as launching test GUI app requires running the build agent as an user (instead of a service which is how it's currently setup).
This causes the build pipeline to get stuck. So I'd like to run these tests locally but not on the build server.
Is there a way to achieve this using xUnit or MSTests and Azure DevOps build pipeline?
You sure can.
Setup an Environment variable to indicate if it's running on the build server in your build.yml file.
variables:
- name: IsRunningOnBuildServer
value: true
Answer 1: Using xUnit
Now create a custom fact attribute to use this:
// This is taken from this SO answer: https://stackoverflow.com/a/4421941/8644294
public class IgnoreOnBuildServerFactAttribute : FactAttribute
{
public IgnoreOnBuildServerFactAttribute()
{
if (IsRunningOnBuildServer())
{
Skip = "This integration test is skipped running in the build server as it involves launching an UI which requires build agents to be run as non-service. Run it locally!";
}
}
/// <summary>
/// Determine if the test is running on build server
/// </summary>
/// <returns>True if being executed in Build server, false otherwise.</returns>
public static bool IsRunningOnBuildServer()
{
return bool.TryParse(Environment.GetEnvironmentVariable("IsRunningOnBuildServer"), out var buildServerFlag) ? buildServerFlag : false;
}
}
Now use this FactAttribute on your test methods that you want to skip running on build server. For eg:
[IgnoreOnBuildServerFact]
public async Task Can_Identify_Some_Behavior_Async()
{
// Your test code...
}
Answer 2: Using MSTests
Create a custom test method attribute to override the Execute method:
public class SkipTestOnBuildServerAttribute : TestMethodAttribute
{
public override TestResult[] Execute(ITestMethod testMethod)
{
if (!IsRunningOnBuildServer())
{
return base.Execute(testMethod);
}
else
{
return new TestResult[] { new TestResult { Outcome = UnitTestOutcome.Inconclusive } };
}
}
public static bool IsRunningOnBuildServer()
{
return bool.TryParse(Environment.GetEnvironmentVariable("IsRunningOnBuildServer"), out var buildServerFlag) ? buildServerFlag : false;
}
}
Now use this TestMethodAttribute on your test methods that you want to skip running on build server. For eg:
[SkipTestOnBuildServer]
public async Task Can_Identify_Some_Behavior_Async()
{
// Your test code...
}
You can filter out tests based on namespace etc.
dotnet test --filter FullyQualifiedName!~IntegrationTests
This will run all tests NOT containing "IntetegrationTests" in it's namespace.
You can read more about test filtering here: https://learn.microsoft.com/en-us/dotnet/core/testing/selective-unit-tests
Related
I'm currently writing a suite of integration tests for a mature ASP.NET application.
As part of the application's supporting services, it uses an Azure blob storage container which I need to make sure is accessible and existing prior to running the tests. I want to add what effectively amounts to a check that the configured Azure blob container for the application (be it on the local emulator or Azure itself when running in CI) is up and ready to handle the requests made by the test suite. A large amount of the tests will straight up fail if the backend is inaccessible and they take several minutes to fail as the Azure library waits through several timeouts.
When throwing or asserting in a collection fixture, it doesn't seem to actually prevent any test in the collection from running nor does the fixture exception appear in the resulting logs.
Is there a way to prevent tests from running if any of their associated fixtures throw during instantiation?
My current code is as follows:
The fixture
using Azure.Storage.Blobs;
using Newtonsoft.Json.Linq;
using System.IO;
using Xunit;
namespace product.test.integration
{
/// <summary>
/// This fixture ensures the configured azure blob container used by the FileStorage system is accessible.
/// A concern mostly in local environments where the Azurite emulator may not be running, but also can alert to a misconfigured testing environment.
/// </summary>
public class FileStorageFixture
{
public FileStorageFixture()
{
//get the configuration
string appdataLocation = Path.Combine(Directory.GetCurrentDirectory(), "testappsettings.json");
string appdataContent = File.ReadAllText(appdataLocation);
var json = JObject.Parse(appdataContent);
string connectionString = json["Settings"]["FileStorage"]["AzureStorageKey"].Value<string>();
string containerName = json["Settings"]["FileStorage"]["ContainerName"].Value<string>();
//check if the container exists
var container = new BlobContainerClient(connectionString, containerName);
try
{
Assert.True(container.Exists()); //this line will throw if a timeout occurs
} catch
{
throw new System.Exception("Configured FileStorage backend is not accessible!");
}
}
}
}
The base test class (IntegrationTest) which all tests extend, and the collection that contains them:
//...
[CollectionDefinition("Integration Tests", DisableParallelization = true)]
public class IntegrationTestCollection :
//...
ICollectionFixture<FileStorageFixture> { }
[Collection("Integration Tests")]
public abstract class IntegrationTest : IClassFixture<WebApplicationFactory<product.api.Startup>>
{
//...
One way to accomplish what you desire is to have the fixture set a flag if its initialization was not successful and query the flag in the constructor of your base test class.
In FileStorageFixture add the flag as a property
public bool initOK { get; private set; } = false;
and set the property depending on whether the initialization was successful
try
{
Assert.True(container.Exists()); //this line will throw if a timeout occurs
initOK = true;
}
catch
{
initOK = false;
}
It would be even more reliable if you included the rest of the code in the FileStorageFixture constructor inside the try, since an exception anywhere there would also bring your tests down.
I assume the reference to the fixture is being provided as a parameter to the constructor of IntegrationTest, so you should throw an exception there if the fixture wasn't initialized properly:
public IntegrationTest(FileStorageFixture fixture)
{
Assert.True(fixture.initOK, "Configured FileStorage backend is not accessible!");
// ... and do the rest of the constructor stuff
}
Without the IntegrationTest object, none of its tests will run.
So i have this code:
[TestFixture]
[Category("MyTestSet")]
public class MyTests
{
[Test]
public void TestCase12()
{
ExecuteTestCase(12);
}
[Test]
public void TestCase13()
{
ExecuteTestCase(13);
}
[Test]
public void TestCase14()
{
ExecuteTestCase(14);
}
}
The ExecuteTestCase gets test parameters from my web server and executes the test case with these settings.
Each time i add a new test case parameters on my web server i need to add a new test in my C# code and pass the ID of test case parameters i have in my web server database and compile my code.
Is there any way to do it automatically? Like say, C# gets from my server ID's of all test case parameters and creates tests for them on the fly?
What is important, test cases change frequently. I was thinking about running all test cases in one test case on a loop, but than i'd be unable to run my test cases separately for example in Nunit IDE.
So my question is: how to run multiple test cases depending on data i receive on run time.
You can use TestCaseSourceattribute in order to get parameters from web service and have your test cases auto generated
[TestFixture]
[Category("MyTestSet")]
public class MyTests
{
[Test, TestCaseSource(nameof(GetTestParameters))]
public void TestCase(int parameter)
{
ExecuteTestCase(parameter);
}
static int[] GetTestParameters()
{
//call web service and get parameters
return new[] { 1, 2, 3 };
}
}
documentation
We have an open source project for running remote XUnit.Net tests. The feature is that the body of the test is designed to run on a remote process and the results are serialized back to visual studio. The project can be found here.
https://github.com/Weingartner/XUnitRemote
For example a test might be like
[SampleProcessFact]
public void OutOfProcess()
{
_Output.WriteLine("Process name: " + Process.GetCurrentProcess().ProcessName);
Assert.Equal(1,1);
}
SampleProcessFact is a custom attribute declared so.
[AttributeUsage(AttributeTargets.Method)]
[XunitTestCaseDiscoverer("XUnitRemote.Test.SampleProcessFactDiscoverer", "XUnitRemote.Test")]
public class SampleProcessFactAttribute : FactAttribute { }
[AttributeUsage(AttributeTargets.Method)]
[XunitTestCaseDiscoverer("XUnitRemote.Test.ScheduledSampleProcessFactDiscoverer", "XUnitRemote.Test")]
public class ScheduledSampleProcessFactAttribute : FactAttribute { }
[AttributeUsage(AttributeTargets.Method)]
[XunitTestCaseDiscoverer("XUnitRemote.Test.SampleProcessTheoryDiscoverer", "XUnitRemote.Test")]
public class SampleProcessTheoryAttribute : TheoryAttribute { }
See https://github.com/Weingartner/XUnitRemote/blob/master/XUnitRemote.Test/XUnit.cs#L26 for source code.
However we would like an option so that if I tag my test case as SampleProcessFact then the test runner will only run the tests sequentially.
I am aware that I can tag all my test cases with TestCollection(string id) and it should prevent sequential running but this should be encapsulated in the SampleProcessFact tag if possible. This should work across all test cases in all assemblies.
My question only relates to the firing off of tests from visual studio. The remote part works fine but visual studio is calling our remote test case engine in parallel for all the tests.
Any ideas?
We made a checkin to our project that solved the problem.
https://github.com/Weingartner/XUnitRemote/commit/565e1dfb55f65ff0612afa40bc5d076c69bb739c
The general solution was to intercept the ITestMethod instance within XUnitRemoteTestCaseDiscoverer::Discover and rewrite the UniqueId of it's test collection.
public IEnumerable<IXunitTestCase> Discover(ITestFrameworkDiscoveryOptions discoveryOptions, ITestMethod testMethod, IAttributeInfo factAttribute)
{
if(CollectionId.HasValue)
testMethod = WrapTestMethod(testMethod, CollectionId.Value);
if (!Process.GetCurrentProcess().ProcessName.Equals(Path.GetFileNameWithoutExtension(_ExePath), StringComparison.OrdinalIgnoreCase))
{
yield return new XUnitRemoteTestCase(_DiagnosticMessageSink, discoveryOptions.MethodDisplayOrDefault(), testMethod, _Id, _ExePath);
}
else
{
foreach (var testCase in _DefaultTestCaseDiscoverer.Discover(discoveryOptions, testMethod, factAttribute))
{
yield return _TestCaseConverter(testCase);
}
}
}
and
private ITestMethod WrapTestMethod(ITestMethod testMethod, Guid uniqueId)
{
var testClass = testMethod.TestClass;
var testCollection = testClass.TestCollection;
testCollection = new TestCollection
(testCollection.TestAssembly, testCollection.CollectionDefinition, testCollection.DisplayName)
{
UniqueID = uniqueId
};
testClass = new TestClass(testCollection, testClass.Class);
testMethod = new TestMethod(testClass, testMethod.Method);
return testMethod;
}
i've got class as below and i need to test RunMethod. Problem is, that ExternalClass needs some application run on server side (app providing external dll's). My dev environment has that app but my dev environment for test hasn't. My question is, how to shim/fake that ExternalClass in UnitTests to not check if app exist (test always faild in env without that external app)? This class is not important in tests but run automatically if i execute RunMethod.
public class MyExampleClass : ISomeInterface
{
private static ExternalClass = new ExternalClass(string someParam);
public object RunMethod()
{
/* Actuall code hear, doesn't matter */
/* few unimportant (from the view point of tester) operation in ExternalClass (object loggin etc.) */
return someVar;
}
}
MsFakes generate a property named AllInstances to the shim class, through this property you can override the behavior of any instance method:
[TestMethod]
public void TestMethod1()
{
using (ShimsContext.Create())
{
ShimExternalClass.AllInstances.ToString01 = () =>
{
return String.Empty();
};
Assert.IsNull(new ExternalClass().ToString());
}
}
I want to ignore certain tests based on data I pulled from a configuration file during the TestFixtureSetUp. Is there a way to ignore running a test based on parameters?
[TestFixture]
public class MessagesTests
{
private bool isPaidAccount;
[TestFixtureSetUp]
public void Init () {
isPaidAccount = ConfigurationManager.AppSettings["IsPaidAccount"] == "True";
}
[Test]
//this test should run only if `isPaidAccount` is true
public void Message_Without_Template_Is_Sent()
{
//this tests an actual web api call.
}
}
If account we are testing with is a paid account, the test should run fine, if not, the method will throw an exception.
Would there be an extension of the attribute [Ignore(ReallyIgnore = isPaidAccount )]? Or should I write this inside the method and run 2 separate test cases for eg.
public void Message_Without_Template_Is_Sent()
{
if(isPaidAccount)
{
//test for return value here
}
else
{
//test for exception here
}
}
You can use Assert.Ignore() like Matthew states. You could also use Assert.Inconclusive() if you want to categorize the result differently.
This Question/Answer is slightly similar: Programmatically skip an nunit test