Develop Reference

Dynamically create nunit test cases based off config file

I'm looking for a way to dynamically create nunit test cases based off config file. In my App.config, I have a list of filenames and its expected operation. I would like to be able to add a new entry and have a test case be created dynamically based off that. My code is as follows:
App.config
<?xml version="1.0" encoding="utf-8" ?>
<configuration>
<appSettings>
<!--syntax is filename#operation where 0 = Valid File, 1 = Missing Column, 2 = Invalid File-->
<add key="FILENAME_0" value="C:\Users\me\Desktop\file_0.csv#0" />
<add key="FILENAME_1" value="C:\Users\me\Desktop\file_1.csv#1" />
<add key="FILENAME_2" value="C:\Users\me\Desktop\file_2.csv#2" />
</appSettings>
</configuration>
In my test fixture, I initialize a list of test case data, which is created by parsing the App.config file as follows:
[TestFixture]
public class MyTests
{
public enum Operation
{
ValidFile = 0,
MissingColumns = 1,
InvalidFile = 2
};
/// <summary>
/// Gets test case data with the file name and an enum of the expected operation.
/// </summary>
private static IEnumerable<TestCaseData> FilenameCases()
{
int i = 0;
bool moreFilesToTest = true;
var files = new Dictionary<string, Operation>();
while (moreFilesToTest)
{
string filenameAndOp = ConfigurationManager.AppSettings[$"FILENAME_{i}"];
if (filenameAndOp == null)
moreFilesToTest = false;
else
{
string filename = filenameAndOp.Split('#')[0];
int operation = Int32.Parse(filenameAndOp.Split('#')[1]);
Operation op = (Operation)operation;
files.Add(filename, op);
}
i++;
}
foreach (var pair in files)
yield return new TestCaseData(pair.Key, pair.Value);
}
[Test, TestCaseSource("FilenameCases")]
public void ShouldLoadFiles(string FILENAME, Operation expected)
{
// ... run test cases
}
}
This works, however, would need to rebuild the project each time in order for any changes in the App.config to be recognized. The code also only works if the TestCaseData is static. What is the correct way to have new test cases be created dynamically based off the configuration file.

After researching this post, I have found out that the issue is not with the code, but rather the the configuration file. As I was running the test via the command line, the test .dll was not referencing my App.config, but another file.
When I compiled my project, I noticed it produced a config file based off the name of the project. So a test project titled MyTestProject would produce a file called MyTestProject.dll.config. Therefore, when I made changes to that config file, my test cases reflected the changes, without recompiling the project.
Credit to both Charlie and Amittai Shapira, as NUnit doesn't support dynamic test cases and TestCaseSource needs to be static. However, the code I tried was the best work around, as the end result did produced dynamic test cases.

TestCaseSource methods need to be static. There are a few ways around this if you need it but I see no problem with using static in your case. Just because a method is static, that doesn't mean it has to always return the same values.
There is no reason for you to recompile your code when you change the config file. However, you have to reload the test assembly in order for the new values in the config to be read. In some environments, like under VS, it may be easier to just recompile in order to force that to happen.
Essentially, I'd say you have already found the best way to do this, although my personal tendency would be to use a flat file rather than the config file.

Visual Studio: Revoke reading permission of test file for Unit Testing

I am working on a Visual Studio (Community 2015) project in C# that reads metadata from video files. Therefore I want to write a Unit Test that tests if the program handles an inaccessible file the right way. So I want to add a test file to my test data for which the Unit Test does not have read permissions.
But what is the best way to do this? I didn't find any option to set permissions in Visual Studio. The only idea I have at the moment is to revoke reading permissions directly in the properties of the file. Through the Windows File Explorer, thus outside of Visual Studio. But then I wouldn't be able to read the file myself. Also, it doesn't seem to be the right way and I think there should be a more clean way to solve this problem directly in Visual Studio.
Any ideas?

If you are writing a unit test then it should just check if your code is written correcly and not about the external issues which are difficult to replicate or may not be available on other machines. So I think you should read about Dependency Injection and how to do mock dependencies in your test. I can give you an easy to understand example but please do write it in your words.
public interface IFileReader
{
string ReadFile( string filePath );
}
public class FileReader : IFileReader
{
public string ReadFile( string filePath )
{
return System.IO.File.ReadAllText( filePath );
}
}
So Suppose you have a class VideoMetaDataReader then in that class you will inject the interface as a dependency and use the ReadFile method to read. And then in your test
[TestFixture]
public class FileReaderTests
{
[Test]
public void ShouldDisplayANiceMessage_WhenFileIsInaccessible()
{
var moq = new Moq.Mock<IFileReader>();
moq
.Setup( x => x.ReadFile( Moq.It.IsAny<string>() ) )
.Throws<Exception>();
var metaDataReader = new MetaDataReader( moq.Object );
metaDataReader.ReadVideoFile( "video.mp4" );
Assert.AreEqual( 1, meaDataReader.Errors.Count );
}
}
I am assuming there will be a errors property which will return encountered errors but it depends how you want to do it (I didn't think too much on this).
Anyways, the point is don't test what is not covered by your code. What your code needs to do is to display a nice message if there is an exception and that you can do if you mock the interface (which means don't call the actual implementation and instead do something which I want in this test) and then check if your code can handle that. btw i used nunit and moq for test and mocking.

How to access TestRunParameters within RunSettings file

Reading through https://msdn.microsoft.com/en-us/library/jj635153.aspx I have created a .RunSettings files with a few parameters similar to the example:
<TestRunParameters>
<Parameter name="webAppUrl" value="http://localhost" />
<Parameter name="webAppUserName" value="Admin" />
<Parameter name="webAppPassword" value="Password" />
</TestRunParameters>
I plan on having a .RunSettings file for each of our environments with appropriate URLs and credentials for running a CodedUI test on the specified RunSettings file's environment.
I can see that from command line to reference the Settings file I can run:
vstest.console myTestDll.dll /Settings:Local.RunSettings /Logger:trx
vstest.console myTestDll.dll /Settings:QA.RunSettings /Logger:trx
etc...
But I don't see any way that calls out how to actually utilize the TestRunParameters from within the codedUI test.
What I would like to do is set up test initializers that use the TestRunParameters to determine where to log in, and what credentials to use. Something like this:
[TestInitialize()]
public void MyTestInitialize()
{
// I'm unsure how to grab the RunSettings.TestRunParameters below
string entryUrl = ""; // TestRunParameters.webAppUrl
string userName = ""; // TestRunParameters.webAppUserName
string password = ""; // TestRunParameters.webAppPassword
LoginToPage(entryUrl, userName, password);
}
public void LoginToPage(string entryUrl, string userName, string password)
{
// Implementation
}
Information on how to reference the TestRunParameters is greatly appreciated!
EDIT
/// <summary>
/// Summary description for CodedUITest1
/// </summary>
[CodedUITest]
public class CodedUITest1
{
public static string UserName = string.Empty;
[ClassInitialize]
public static void TestClassInitialize(TestContext context)
{
UserName = context.Properties["webAppUserName"].ToString();
Console.WriteLine(UserName);
}
[TestMethod]
public void CodedUITestMethod1()
{
this.UIMap.RecordedMethod1();
// To generate code for this test, select "Generate Code for Coded UI Test" from the shortcut menu and select one of the menu items.
}
// Rest of the default class - TestContext instantiation, UI map instantiation, etc
}
The exception I'm getting when running:
NullReference Exception
#williamfalconeruk I have updated my test class as above, but I am still getting the same error, any idea what I'm doing wrong?

For those that use Resharper with this issue, I discovered the fix (no need to disable Resharper):
Go to Visual Studio top menu -> Resharper -> Options
Find the Tools section, expand "Unit Testing"
Click on "MsTest". The checkbox should be on enabled, but the Test Settings file path below that may be blank. If it is, click on browse and select the runsettings file you want to use.
Click save, rebuild and try to run the tests, the parameters should now work.
Not sure why but simply selecting the Test Settings file from the Tests menu -> Test Settings does not actually work when using Resharper, so this file needs to be explicitly pointed to directly in the Resharper options.

I also came across this recently, as we wanted to move away from legacy environment variable usage. The solution provided below was suitable for our needs, but there may be a better one...
It turns out you can access these in the TestContext of a ClassInitialize method of your Test fixture. There is a Properties dictionary which has these parameters, and you could access the values here:
[ClassInitialize]
public static void TestClassinitialize(TestContext context)
{
var webAppUrl = context.Properties["webAppUrl"].ToString();
//other settings etc..then use your test settings parameters here...
}
Note: these are static so if you need access to this you may need to set up static properties to access within your Test code.
An alternative suggested is using a Data Driven Test approach.
Here's some basic info on data driven tests here which may also help:
https://msdn.microsoft.com/en-us/library/ms182527.aspx
As I said, the above solution suited our needs at a basic level.
UPDATE: see image below in response to test settings returning null...

This works for me (VS2017-pro):
namespace TestApp.Test
{
[TestClass]
public class UnitTest1
{
// This enables the runner to set the TestContext. It gets updated for each test.
public TestContext TestContext { get; set; }
[TestMethod]
public void TestMethod1()
{
// Arrange
String expectedName = "TestMethod1";
String expectedUrl = "http://localhost";
// Act
String actualName = TestContext.TestName;
// The properties are read from the .runsettings file
String actualUrl = TestContext.Properties["webAppUrl"].ToString();
// Assert
Assert.AreEqual(expectedName, actualName);
Assert.AreEqual(expectedUrl, actualUrl);
}
[TestMethod]
public void TestMethod2()
{
// Arrange
String expectedName = "TestMethod2";
// Act
String actualName = TestContext.TestName;
// Assert
Assert.AreEqual(expectedName, actualName);
}
}
}
Make sure to select the runsettings file you wish to use, here: Test -> Test Settings.

An alternative to disable Resharper is to enable MSTest support and select the test setting file on Resharper Options dialog (->Tools->Unit Testing->MsTest).

I was trying to do this exact thing as well. As many of you may know, running tests through MTM exposes some additional properties to the TestContext, including the name of the Run Settings used. I used this property as a "Foreign Key" of sorts for our test data, allowing us to specify environment URLs etc. without hardcoding them or using the incredibly lackluster "Data Driving" tools that come with out of the box testing.
Of course, there's no way to expose any run-time properties when executing tests as part of a BDT or release workflow besides what #kritner is attempting which microsoft describes HERE. However if you read the comments of that link you'll discover what you may be able to infer here:
You need to use VS 2013 R5 or VS 2015 to use this solution
It will only work for Unit Tests!
Those of us who are trying to execute UI or Load tests as part of a CI or CD workflow are completely screwed. You don't get any additional properties in testContext, even when executing a Plan/Suite with certain test configurations (not settings) created in MTM. #Adam may have been able to get this to work when running vs debugging, but that may have only worked with unit tests. Through CodedUI I've been unable to retrieve the properties without getting a NullReferenceException. Here's an example of the janky code I was using to investigate:
if (testContextInstance.Properties["__Tfs_TestConfigurationName__"] != null) //Exposed when run through MTM
{
TFSTestConfigurationName = testContextInstance.Properties["__Tfs_TestConfigurationName__"].ToString();
}
else TFSTestConfigurationName = "Local"; //Local
var configName = testContextInstance.Properties["configurationName"] ?? "No Config Found";
Trace.WriteLine("Property: " + configName);
And the XML of my .runsettings file:
<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
<!-- Parameters used by tests at runtime. These are required as a substitute for TFS/MTM test settings.-->
<!-- File instructions: https://msdn.microsoft.com/en-us/library/jj635153.aspx#example -->
<!-- TFS instructions: https://blogs.msdn.microsoft.com/visualstudioalm/2015/09/04/supplying-run-time-parameters-to-tests/ -->
<TestRunParameters>
<Parameter name="configurationName" value="Local" />
</TestRunParameters>
</RunSettings>
And an excerpt from the .trx produced by the BDT workflow:
Property: No Config Found

For the NullReferenceException issue:
I was also facing the same issue recently and the solution to it is to have the latest update of Visual Studio 2013. Right now the latest update is Update 5. I am not sure which particular update fixes this issue. I applied Update 5 and was able to successfully access the TestRunParameters in the ClassInitialize method.
You can find the updates # https://support.microsoft.com/en-us/kb/2829760
So I had two machines, on one it was all working fine and on the other I was getting the exception. I investigated that the only difference is the Update of VS; applied it and that solved the problem. :)

With NUnit 3, I was able to find the properties in the runsettings file, using
TestContext.Parameters
So in this case it would be:
string entryUrl = TestContext.Parameters["webAppUrl"];
string username = TestContext.Parameters["webAppUserName"];
string password = TestContext.Parameters["webAppPassword"];

I was able to resolve this for Unit tests by disabling Resharper. Wish I could say the same for Coded UI tests.

Why dont you use Application settings?
you can then read them anywhere like
var yesICan= Properties.Settings.Default.IToldYou;
you can create Properties out from them, pretty much you can do a lot.
public string URL_WEBOFFICE
{
get
{
return Properties.Settings.Default.WEBOFFICE_URL.Replace("***", Properties.Settings.Default.SiteName);
}
}

Referencing a folder in my project with a SpecFlow test

I'm trying to write a SpecFlow test where I test what happens when my application reads a certain structure of folders and files. I want to include these folders and files in my project so the tests don't just run on my own computer.
For example, I have two folders in my Specs project. One called 'SimpleTestModel' and the other called 'ComplexTestModel'. How can I reference these folders in my SpecFlow tests?

You want a Test Fixture.
From Wikipedia:
In software testing, a test fixture is a fixed state of the software under test used as a baseline for running tests; also known as the test context. It may also refer to the actions performed in order to bring the system into such a state.
Examples of fixtures:
Loading a database with a specific, known set of data
Erasing a hard disk and installing a known clean operating system installation
Copying a specific known set of files
Preparation of input data and set-up/creation of fake or mock objects
Software used to systematically run reproducible tests on a piece of software under test is known as a test harness; part of its job is to set up suitable test fixtures.
For your specific problem:
Create a Fixtures directory in your SpecFlow test project. Inside that create any numbers of sub directories based on your tests to set up the directory and file structures that you need.
Add an <appSettings> in App.config entry define the root folder for all your test fixtures
<configuration>
...
<appSettings>
<!-- Path relative to the build output directory -->
<add name="FixturesRootDirectory" value="..\..\Fixtures" />
</appSettings>
...
</configuration>
In a [BeforeScenario] hook, set the absolute path to the fixtures directory on the current scenario context (reference: How do I get the path of the assembly the code is in?)
using System.Configuration;
using System.IO;
using TechTalk.SpecFlow;
namespace Foo
{
[Binding]
public class CommonHooks
{
[BeforeScenario]
public void BeforeScenario()
{
InitFixturesPath();
}
private void InitFixturesPath()
{
if (ScenarioContext.Current.ContainsKey("FixturesPath"))
return;
string codeBase = Path.GetDirectoryName(Assembly.GetExecutingAssembly().CodeBase)
+ Path.DirectorySeparatorChar
+ ConfigurationManager.AppSettings["FixturesRootDirectory"];
UriBuilder uri = new UriBuilder(codeBase);
string path = Uri.UnescapeDataString(uri.Path);
ScenarioContext.Current.Set<string>("FixturesPath", Path.GetDirectoryName(path));
}
}
}
Now you can use ScenarioContext.Current.Get<string>("FixturesPath") to get the root directory for all of your fixtures. You could even write your own Fixtures helper class:
public static class FixturesHelper
{
public static string Path { get; set; }
// other methods and properties making it easier to use fixtures
}

Text templates for string generation at run-time (like Razor or T4)

Is there any tool on the web that can be used to generate strings from a template, i'm looking for something similar to Razor.
The strings should be able to be generated at run time, and don't depend on Visual Studio (like T4). And the framework should work in Silverlight.
RazorEngine is a framework that meets the requeriments but doesn't work in Silverlight.
Thanks in advance.

Hopefully I understood what you asked for but I would argue that you can make T4 work in SL as well. T4 can be asked to generate what is sometimes called runtime templates. I have defined my template (very simple) and added it to my Silverlight project.
<#
for (var iter = 0; iter < 10; ++iter)
{
#>
This is just a test: #<#=iter#>
<#
}
#>
Normally it would generate an output like this:
This is just a test: #0
This is just a test: #1
This is just a test: #2
This is just a test: #3
This is just a test: #4
This is just a test: #5
This is just a test: #6
This is just a test: #7
This is just a test: #8
This is just a test: #9
But in this case I like to generate the code that generates that output ie a runtime template. In order to do that I switch the custom tool to: TextTemplatingFilePreprocessor
Now the template generates the code that generates that output. If you stay clear of hostspecific=true you don't get Visual Studio dependencies. By extending the partial class with member variables and referencing them from the template file you can modify the behavior on the template in runtime.
The problem in Silverlight is that silverlight lacks the classes: System.CodeDom.Compiler.CompilerError and System.CodeDom.Compiler.CompilerErrorCollection.
I worked around that by creating my own classes for that (just for this purpose):
namespace System.CodeDom.Compiler
{
public class CompilerError
{
public string ErrorText;
public bool IsWarning;
}
public class CompilerErrorCollection : List<CompilerError>
{
}
}
Now my template compiles and I just it like this from my Silverlight app to produce the output:
var runtimeTemplate = new MyRuntimeTemplate();
string output = runtimeTemplate.TransformText();