I'm looking for a way to dynamically create nunit test cases based off config file. In my App.config, I have a list of filenames and its expected operation. I would like to be able to add a new entry and have a test case be created dynamically based off that. My code is as follows:
App.config
<?xml version="1.0" encoding="utf-8" ?>
<configuration>
<appSettings>
<!--syntax is filename#operation where 0 = Valid File, 1 = Missing Column, 2 = Invalid File-->
<add key="FILENAME_0" value="C:\Users\me\Desktop\file_0.csv#0" />
<add key="FILENAME_1" value="C:\Users\me\Desktop\file_1.csv#1" />
<add key="FILENAME_2" value="C:\Users\me\Desktop\file_2.csv#2" />
</appSettings>
</configuration>
In my test fixture, I initialize a list of test case data, which is created by parsing the App.config file as follows:
[TestFixture]
public class MyTests
{
public enum Operation
{
ValidFile = 0,
MissingColumns = 1,
InvalidFile = 2
};
/// <summary>
/// Gets test case data with the file name and an enum of the expected operation.
/// </summary>
private static IEnumerable<TestCaseData> FilenameCases()
{
int i = 0;
bool moreFilesToTest = true;
var files = new Dictionary<string, Operation>();
while (moreFilesToTest)
{
string filenameAndOp = ConfigurationManager.AppSettings[$"FILENAME_{i}"];
if (filenameAndOp == null)
moreFilesToTest = false;
else
{
string filename = filenameAndOp.Split('#')[0];
int operation = Int32.Parse(filenameAndOp.Split('#')[1]);
Operation op = (Operation)operation;
files.Add(filename, op);
}
i++;
}
foreach (var pair in files)
yield return new TestCaseData(pair.Key, pair.Value);
}
[Test, TestCaseSource("FilenameCases")]
public void ShouldLoadFiles(string FILENAME, Operation expected)
{
// ... run test cases
}
}
This works, however, would need to rebuild the project each time in order for any changes in the App.config to be recognized. The code also only works if the TestCaseData is static. What is the correct way to have new test cases be created dynamically based off the configuration file.
After researching this post, I have found out that the issue is not with the code, but rather the the configuration file. As I was running the test via the command line, the test .dll was not referencing my App.config, but another file.
When I compiled my project, I noticed it produced a config file based off the name of the project. So a test project titled MyTestProject would produce a file called MyTestProject.dll.config. Therefore, when I made changes to that config file, my test cases reflected the changes, without recompiling the project.
Credit to both Charlie and Amittai Shapira, as NUnit doesn't support dynamic test cases and TestCaseSource needs to be static. However, the code I tried was the best work around, as the end result did produced dynamic test cases.
TestCaseSource methods need to be static. There are a few ways around this if you need it but I see no problem with using static in your case. Just because a method is static, that doesn't mean it has to always return the same values.
There is no reason for you to recompile your code when you change the config file. However, you have to reload the test assembly in order for the new values in the config to be read. In some environments, like under VS, it may be easier to just recompile in order to force that to happen.
Essentially, I'd say you have already found the best way to do this, although my personal tendency would be to use a flat file rather than the config file.
Related
I have a static C# library project which relies on configuration. This library may be used in two scenarios:
From managed C# applications. Easy - the application's app.config file will be used, via ConfigurationManager
From unmanaged C++ applications (using COM)
In case 2. there is no app.config. I would like the library to still be able to use ConfigurationManager but it should explicitly load an XML config file with the same structure as app.config.
This question talks about how to manually load a config file: Loading custom configuration files
But how can the library detect which case 1/2 its in? I would be happy with an InitLib method which passes a config name, or a static initializer, but I can't see how to put the pieces together.
The config file name to use in case 2 could either be passed in directly, or hardcoded as MyAssembly.Lib.config or similar.
So to clarify: you have a class in a class library, the library references System.Configuration.ConfigurationManager.dll, and the class looks does something like this:
using System.Configuration;
namespace FooLibrary
{
public class Foo
{
public Foo()
{
var bar = ConfigurationManager.AppSettings("FooLibrary.Foo.Bar");
}
}
}
Now when you call this class from a console application, say, Baz.exe, there will exist a Baz.exe.config:
<?xml version="1.0"?>
<configuration>
<appSettings>
<add key="FooLibrary.Foo.Bar" value="Baz" />
</appSettings>
</configuration>
And all works.
It doesn't work when the library runs in a context without a configuration file. While I think it's possible to give an unmanaged application, say, Qux.exe a configuration file Qux.exe.config which .NET will then read for assemblies loaded from that executable, that situation isn't ideal. Even if that'd work (I think, but am not sure, that it's just a file name convention, not something the runtime does for executables on startup). It's possible that the executable running the assembly is not under your control anyway (e.g. somewhere in System32).
While you could let the library load a configuration file relative to its own location, and to answer your question:
But how can the library detect which case 1/2 its in?
You could just test for the AppSettings key anyway, and if not found, assume you've got no configuration file, and open that of the DLL instead:
public class Foo
{
private readonly string _bar;
public Foo()
{
// Read from .exe.config or Web.config
_bar = ConfigurationManager.AppSettings("FooLibrary.Foo.Bar");
if (string.IsNullOrEmpty(_bar))
{
// Assume that if not present, own config must be loaded:
var dllConfigPath = Assembly.GetExecutingAssembly().Location;
// Will just append ".config" despite its name
Configuration config = ConfigurationManager.OpenExeConfiguration(dllConfigPath);
_bar = config.AppSettings.Settings["test"]?.Value;
if (string.IsNullOrEmpty(_bar))
{
// Not found in both configs, now what?
}
}
}
}
And you should refactor it a bit to read multiple variables. And now you want to unit test this code, so you'll have to provide a configuration file there as well.
These problems are solved by not reading the configuration in class libraries, but in application startup, and passing the appropriate values to the constructor of the classes from the library:
public Foo(string bar) { ... }
// And then from your executable:
var foo = new Foo(ConfigurationManager.AppSettings["FooLibrary.Foo.Bar"));
// And your unit test
var foo = new Foo("FromTest");
But you can't pass constructor parameters through COM, which the compiler will tell you as well. So you'll have to provide a parameterless constructor, and something like an InitLib(string bar) method for COM only.
And if you, despite all the above, still insist on reading the config there, then it would look something like this:
public class Foo
{
private string _bar;
public Foo()
{
// Read from .exe.config or Web.config
_bar = ConfigurationManager.AppSettings["FooLibrary.Foo.Bar"];
// Can't throw here if _bar is empty, maybe we're called from COM
}
public void ReadDllConfig()
{
// Assume that if not present, own config must be loaded:
var dllConfigPath = Assembly.GetExecutingAssembly().Location;
// Will just append ".config" despite its name
Configuration config = ConfigurationManager.OpenExeConfiguration(dllConfigPath);
_bar = config.AppSettings.Settings["test"]?.Value;
// Can throw here if _bar is empty
}
public void FooForCom()
{
// TODO: test _bar again, it can still be null.
}
}
I'm working on a C# class library that needs to be able to read settings from the web.config or app.config file (depending on whether the DLL is referenced from an ASP.NET web application or a Windows Forms application).
I've found that
ConfigurationSettings.AppSettings.Get("MySetting")
works, but that code has been marked as deprecated by Microsoft.
I've read that I should be using:
ConfigurationManager.AppSettings["MySetting"]
However, the System.Configuration.ConfigurationManager class doesn't seem to be available from a C# Class Library project.
What is the best way to do this?
For a sample app.config file like below:
<?xml version="1.0" encoding="utf-8" ?>
<configuration>
<appSettings>
<add key="countoffiles" value="7" />
<add key="logfilelocation" value="abc.txt" />
</appSettings>
</configuration>
You read the above application settings using the code shown below:
using System.Configuration;
You may also need to also add a reference to System.Configuration in your project if there isn't one already. You can then access the values like so:
string configvalue1 = ConfigurationManager.AppSettings["countoffiles"];
string configvalue2 = ConfigurationManager.AppSettings["logfilelocation"];
You'll need to add a reference to System.Configuration in your project's references folder.
You should definitely be using the ConfigurationManager over the obsolete ConfigurationSettings.
Update for .NET Framework 4.5 and 4.6; the following will no longer work:
string keyvalue = System.Configuration.ConfigurationManager.AppSettings["keyname"];
Now access the Setting class via Properties:
string keyvalue = Properties.Settings.Default.keyname;
See Managing Application Settings for more information.
Right click on your class library, and choose the "Add References" option from the Menu.
And from the .NET tab, select System.Configuration. This would include the System.Configuration DLL file into your project.
I'm using this, and it works well for me:
textBox1.Text = ConfigurationManager.AppSettings["Name"];
Read From Config:
You'll need to add a reference to the configuration:
Open "Properties" on your project
Go to "Settings" Tab
Add "Name" and "Value"
Get Value with using following code:
string value = Properties.Settings.Default.keyname;
Save to the configuration:
Properties.Settings.Default.keyName = value;
Properties.Settings.Default.Save();
You must add a reference to the System.Configuration assembly to the project.
You might be adding the App.config file to a DLL file. App.Config works only for executable projects, since all the DLL files take the configuration from the configuration file for the EXE file being executed.
Let's say you have two projects in your solution:
SomeDll
SomeExe
Your problem might be related to the fact that you're including the app.config file to SomeDLL and not SomeExe. SomeDll is able to read the configuration from the SomeExe project.
Try this:
string keyvalue = System.Configuration.ConfigurationManager.AppSettings["keyname"];
In the web.config file this should be the next structure:
<configuration>
<appSettings>
<add key="keyname" value="keyvalue" />
</appSettings>
</configuration>
Step 1: Right-click on references tab to add reference.
Step 2: Click on Assemblies tab
Step 3: Search for 'System.Configuration'
Step 4: Click OK.
Then it will work.
string value = System.Configuration.ConfigurationManager.AppSettings["keyname"];
I had the same problem. Just read them this way:
System.Configuration.ConfigurationSettings.AppSettings["MySetting"]
web.config is used with web applications. web.config by default has several configurations required for the web application. You can have a web.config for each folder under your web application.
app.config is used for Windows applications. When you build the application in Visual Studio, it will be automatically renamed to <appname>.exe.config and this file has to be delivered along with your application.
You can use the same method to call the app settings values from both configuration files:
System.Configuration.ConfigurationSettings.AppSettings["Key"]
As I found the best approach to access application settings variables in a systematic way by making a wrapper class over System.Configuration as below
public class BaseConfiguration
{
protected static object GetAppSetting(Type expectedType, string key)
{
string value = ConfigurationManager.AppSettings.Get(key);
try
{
if (expectedType == typeof(int))
return int.Parse(value);
if (expectedType == typeof(string))
return value;
throw new Exception("Type not supported.");
}
catch (Exception ex)
{
throw new Exception(string.Format("Config key:{0} was expected to be of type {1} but was not.",
key, expectedType), ex);
}
}
}
Now we can access needed settings variables by hard coded names using another class as below:
public class ConfigurationSettings:BaseConfiguration
{
#region App setting
public static string ApplicationName
{
get { return (string)GetAppSetting(typeof(string), "ApplicationName"); }
}
public static string MailBccAddress
{
get { return (string)GetAppSetting(typeof(string), "MailBccAddress"); }
}
public static string DefaultConnection
{
get { return (string)GetAppSetting(typeof(string), "DefaultConnection"); }
}
#endregion App setting
#region global setting
#endregion global setting
}
Also, you can use Formo:
Configuration:
<appSettings>
<add key="RetryAttempts" value="5" />
<add key="ApplicationBuildDate" value="11/4/1999 6:23 AM" />
</appSettings>
Code:
dynamic config = new Configuration();
var retryAttempts1 = config.RetryAttempts; // Returns 5 as a string
var retryAttempts2 = config.RetryAttempts(10); // Returns 5 if found in config, else 10
var retryAttempts3 = config.RetryAttempts(userInput, 10); // Returns 5 if it exists in config, else userInput if not null, else 10
var appBuildDate = config.ApplicationBuildDate<DateTime>();
If your needing/wanting to use the ConfigurationManager class...
You may need to load System.Configuration.ConfigurationManager by Microsoft via NuGet Package Manager
Tools->NuGet Package Manager->Manage NuGet Packages for Solution...
Microsoft Docs
One thing worth noting from the docs...
If your application needs read-only access to its own configuration,
we recommend that you use the GetSection(String) method. This method
provides access to the cached configuration values for the current
application, which has better performance than the Configuration
class.
I strongly recommend you to create a wrapper for this call. Something like a ConfigurationReaderService and use dependency injection to get this class. This way you will be able to isolate this configuration files for test purposes.
So use the ConfigurationManager.AppSettings["something"]; suggested and return this value. With this method you can create some kind of default return if there isn't any key available in the .config file.
Just for completeness, there's another option available for web projects only:
System.Web.Configuration.WebConfigurationManager.AppSettings["MySetting"]
The benefit of this is that it doesn't require an extra reference to be added, so it may be preferable for some people.
I always create an IConfig interface with typesafe properties declared for all configuration values. A Config implementation class then wraps the calls to System.Configuration. All your System.Configuration calls are now in one place, and it is so much easier and cleaner to maintain and track which fields are being used and declare their default values. I write a set of private helper methods to read and parse common data types.
Using an IoC framework you can access the IConfig fields anywhere your in application by simply passing the interface to a class constructor. You're also then able to create mock implementations of the IConfig interface in your unit tests so you can now test various configuration values and value combinations without needing to touch your App.config or Web.config file.
Please check the .NET version you are working on. It should be higher than 4. And you have to add the System.Configuration system library to your application.
You can use the below line. In my case it was working:
System.Configuration.ConfigurationSettings.AppSettings["yourKeyName"]
You must take care that the above line of code is also the old version and it's deprecated in new libraries.
The ConfigurationManager is not what you need to access your own settings.
To do this you should use
{YourAppName}.Properties.Settings.{settingName}
I was able to get the below approach working for .NET Core projects:
Steps:
Create an appsettings.json (format given below) in your project.
Next create a configuration class. The format is provided below.
I have created a Login() method to show the usage of the Configuration Class.
Create appsettings.json in your project with content:
{
"Environments": {
"QA": {
"Url": "somevalue",
"Username": "someuser",
"Password": "somepwd"
},
"BrowserConfig": {
"Browser": "Chrome",
"Headless": "true"
},
"EnvironmentSelected": {
"Environment": "QA"
}
}
public static class Configuration
{
private static IConfiguration _configuration;
static Configuration()
{
var builder = new ConfigurationBuilder()
.AddJsonFile($"appsettings.json");
_configuration = builder.Build();
}
public static Browser GetBrowser()
{
if (_configuration.GetSection("BrowserConfig:Browser").Value == "Firefox")
{
return Browser.Firefox;
}
if (_configuration.GetSection("BrowserConfig:Browser").Value == "Edge")
{
return Browser.Edge;
}
if (_configuration.GetSection("BrowserConfig:Browser").Value == "IE")
{
return Browser.InternetExplorer;
}
return Browser.Chrome;
}
public static bool IsHeadless()
{
return _configuration.GetSection("BrowserConfig:Headless").Value == "true";
}
public static string GetEnvironment()
{
return _configuration.GetSection("EnvironmentSelected")["Environment"];
}
public static IConfigurationSection EnvironmentInfo()
{
var env = GetEnvironment();
return _configuration.GetSection($#"Environments:{env}");
}
}
public void Login()
{
var environment = Configuration.EnvironmentInfo();
Email.SendKeys(environment["username"]);
Password.SendKeys(environment["password"]);
WaitForElementToBeClickableAndClick(_driver, SignIn);
}
Another possible solution:
var MyReader = new System.Configuration.AppSettingsReader();
string keyvalue = MyReader.GetValue("keyalue",typeof(string)).ToString();
I have been trying to find a fix for this same issue for a couple of days now. I was able to resolve this by adding a key within the appsettings tag in the web.config file. This should override the .dll file when using the helper.
<configuration>
<appSettings>
<add key="loginUrl" value="~/RedirectValue.cshtml" />
<add key="autoFormsAuthentication" value="false"/>
</appSettings>
</configuration>
extra : if you are working on a Class Library project you have to embed the settings.json file.
A class library shouldn't really be directly referencing anything in
app.config - the class doesn't have an app.config, because it's not an
application, it's a class.
Go to the JSON file's properties.
Change Build Action -> Embedded resource.
Use the following code to read it.
var assembly = Assembly.GetExecutingAssembly();
var resourceStream = assembly.GetManifestResourceStream("Assembly.file.json");
string myString = reader.ReadToEnd();
now we have a JSON string we can Deserialize it using JsonConvert
if you didn't embed the file inside the assembly you can't use only the DLL file without the file
I'm using Visual Studio for Mac version 17.0.6.
As you can see on this screenshot it is not possible to add a reference to System.Configuration.
Solution:
install NuGet Package - System.Configuration.ConfigurationManager.
Create app.config file and set "Build action" to "EmbeddedResource"
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<appSettings>
<add key="name" value="Joe"/>
</appSettings>
</configuration>
using System.Configuration;
enjoy)
string name = ConfigurationManager.AppSettings["name"];
BTW: Do not add an app.config for a library
I found the answer in this link https://stackoverflow.com/a/1836938/1492229
It's not only necessary to use the namespace System.Configuration. You have also to add the reference to the assembly System.Configuration.dll , by
Right-click on the References / Dependencies
Choose Add Reference
Find and add System.Configuration.
This will work for sure.
Also for the NameValueCollection you have to write:
using System.Collections.Specialized;
Here's an example: App.config
<applicationSettings>
<MyApp.My.MySettings>
<setting name="Printer" serializeAs="String">
<value>1234 </value>
</setting>
</MyApp.My.MySettings>
</applicationSettings>
Dim strPrinterName as string = My.settings.Printer
I've searched around and no answers have worked for me. I have 2 ASP.NET projects, one of them being my project, the other being a project containing unit tests.
# Projects
Project.Core
...
secretConnectionStrings.config
Web.config
Project.Tests
In Project.Core, I have a method that requests both a connection string (from secretConnectionStrings.config and an app setting from the web.config file. Executing on it's own, the correct values are pulled from both of these config files. However, I want to test this within my unit test.
My test looks like this (within Project.Tests):
[TestClass]
public class TestUserController
{
[TestMethod]
public void Post_ShouldReturnNewUser()
{
var controller = new UserController();
var result = controller.Post() as User;
Assert.AreNotEqual(result, null);
}
}
The line it bombs out on is this
string cacheConnection = ConfigurationManager.ConnectionStrings["RedisCache"].ConnectionString;
System.NullReferenceException: Object reference not set to an instance of an object
I've tried adding web.config and secretConnectionStrings.config as a link in my Project.Tests solution. That hasn't worked, and I do not want to interface to my web.config (I want the values as they are saved in Project.Core). Is there a solution where I can pull values from Project.Core's web.config into my unit test?
The project containing the unit tests will need its own config file for the ConfigurationManager to pull in the connection strings. The best way to set this up and the keep only 1 copy of the config file is to add the same file into each project that needs it, but add it as a linked file:
Don't forget to set the file properties so that it deploys with the executable code:
And add it as a deployment item to your test fixture(s):
[TestClass]
[DeploymentItem(#"db.config")]
public class MyTestFixture { ... }
Now, if you want to have centralized connection strings in two differently-named config files (app.config & web.config), then you can place them in a third file db.config and import that file into each of the other config files:
db.config
<connectionStrings>
<add name="testDb" connectionString="data source=(localdb)\v11.0;..."
providerName="System.Data.SqlClient" />
</connectionStrings>
app.config / web.config
<connectionStrings configSource="db.config" />
Make sure to set Copy To Output Directory for the imported config files in addition to your app.config or web.config and add them as deployment items to the test fixture with the attribute.
Reading through https://msdn.microsoft.com/en-us/library/jj635153.aspx I have created a .RunSettings files with a few parameters similar to the example:
<TestRunParameters>
<Parameter name="webAppUrl" value="http://localhost" />
<Parameter name="webAppUserName" value="Admin" />
<Parameter name="webAppPassword" value="Password" />
</TestRunParameters>
I plan on having a .RunSettings file for each of our environments with appropriate URLs and credentials for running a CodedUI test on the specified RunSettings file's environment.
I can see that from command line to reference the Settings file I can run:
vstest.console myTestDll.dll /Settings:Local.RunSettings /Logger:trx
vstest.console myTestDll.dll /Settings:QA.RunSettings /Logger:trx
etc...
But I don't see any way that calls out how to actually utilize the TestRunParameters from within the codedUI test.
What I would like to do is set up test initializers that use the TestRunParameters to determine where to log in, and what credentials to use. Something like this:
[TestInitialize()]
public void MyTestInitialize()
{
// I'm unsure how to grab the RunSettings.TestRunParameters below
string entryUrl = ""; // TestRunParameters.webAppUrl
string userName = ""; // TestRunParameters.webAppUserName
string password = ""; // TestRunParameters.webAppPassword
LoginToPage(entryUrl, userName, password);
}
public void LoginToPage(string entryUrl, string userName, string password)
{
// Implementation
}
Information on how to reference the TestRunParameters is greatly appreciated!
EDIT
/// <summary>
/// Summary description for CodedUITest1
/// </summary>
[CodedUITest]
public class CodedUITest1
{
public static string UserName = string.Empty;
[ClassInitialize]
public static void TestClassInitialize(TestContext context)
{
UserName = context.Properties["webAppUserName"].ToString();
Console.WriteLine(UserName);
}
[TestMethod]
public void CodedUITestMethod1()
{
this.UIMap.RecordedMethod1();
// To generate code for this test, select "Generate Code for Coded UI Test" from the shortcut menu and select one of the menu items.
}
// Rest of the default class - TestContext instantiation, UI map instantiation, etc
}
The exception I'm getting when running:
NullReference Exception
#williamfalconeruk I have updated my test class as above, but I am still getting the same error, any idea what I'm doing wrong?
For those that use Resharper with this issue, I discovered the fix (no need to disable Resharper):
Go to Visual Studio top menu -> Resharper -> Options
Find the Tools section, expand "Unit Testing"
Click on "MsTest". The checkbox should be on enabled, but the Test Settings file path below that may be blank. If it is, click on browse and select the runsettings file you want to use.
Click save, rebuild and try to run the tests, the parameters should now work.
Not sure why but simply selecting the Test Settings file from the Tests menu -> Test Settings does not actually work when using Resharper, so this file needs to be explicitly pointed to directly in the Resharper options.
I also came across this recently, as we wanted to move away from legacy environment variable usage. The solution provided below was suitable for our needs, but there may be a better one...
It turns out you can access these in the TestContext of a ClassInitialize method of your Test fixture. There is a Properties dictionary which has these parameters, and you could access the values here:
[ClassInitialize]
public static void TestClassinitialize(TestContext context)
{
var webAppUrl = context.Properties["webAppUrl"].ToString();
//other settings etc..then use your test settings parameters here...
}
Note: these are static so if you need access to this you may need to set up static properties to access within your Test code.
An alternative suggested is using a Data Driven Test approach.
Here's some basic info on data driven tests here which may also help:
https://msdn.microsoft.com/en-us/library/ms182527.aspx
As I said, the above solution suited our needs at a basic level.
UPDATE: see image below in response to test settings returning null...
This works for me (VS2017-pro):
namespace TestApp.Test
{
[TestClass]
public class UnitTest1
{
// This enables the runner to set the TestContext. It gets updated for each test.
public TestContext TestContext { get; set; }
[TestMethod]
public void TestMethod1()
{
// Arrange
String expectedName = "TestMethod1";
String expectedUrl = "http://localhost";
// Act
String actualName = TestContext.TestName;
// The properties are read from the .runsettings file
String actualUrl = TestContext.Properties["webAppUrl"].ToString();
// Assert
Assert.AreEqual(expectedName, actualName);
Assert.AreEqual(expectedUrl, actualUrl);
}
[TestMethod]
public void TestMethod2()
{
// Arrange
String expectedName = "TestMethod2";
// Act
String actualName = TestContext.TestName;
// Assert
Assert.AreEqual(expectedName, actualName);
}
}
}
Make sure to select the runsettings file you wish to use, here: Test -> Test Settings.
An alternative to disable Resharper is to enable MSTest support and select the test setting file on Resharper Options dialog (->Tools->Unit Testing->MsTest).
I was trying to do this exact thing as well. As many of you may know, running tests through MTM exposes some additional properties to the TestContext, including the name of the Run Settings used. I used this property as a "Foreign Key" of sorts for our test data, allowing us to specify environment URLs etc. without hardcoding them or using the incredibly lackluster "Data Driving" tools that come with out of the box testing.
Of course, there's no way to expose any run-time properties when executing tests as part of a BDT or release workflow besides what #kritner is attempting which microsoft describes HERE. However if you read the comments of that link you'll discover what you may be able to infer here:
You need to use VS 2013 R5 or VS 2015 to use this solution
It will only work for Unit Tests!
Those of us who are trying to execute UI or Load tests as part of a CI or CD workflow are completely screwed. You don't get any additional properties in testContext, even when executing a Plan/Suite with certain test configurations (not settings) created in MTM. #Adam may have been able to get this to work when running vs debugging, but that may have only worked with unit tests. Through CodedUI I've been unable to retrieve the properties without getting a NullReferenceException. Here's an example of the janky code I was using to investigate:
if (testContextInstance.Properties["__Tfs_TestConfigurationName__"] != null) //Exposed when run through MTM
{
TFSTestConfigurationName = testContextInstance.Properties["__Tfs_TestConfigurationName__"].ToString();
}
else TFSTestConfigurationName = "Local"; //Local
var configName = testContextInstance.Properties["configurationName"] ?? "No Config Found";
Trace.WriteLine("Property: " + configName);
And the XML of my .runsettings file:
<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
<!-- Parameters used by tests at runtime. These are required as a substitute for TFS/MTM test settings.-->
<!-- File instructions: https://msdn.microsoft.com/en-us/library/jj635153.aspx#example -->
<!-- TFS instructions: https://blogs.msdn.microsoft.com/visualstudioalm/2015/09/04/supplying-run-time-parameters-to-tests/ -->
<TestRunParameters>
<Parameter name="configurationName" value="Local" />
</TestRunParameters>
</RunSettings>
And an excerpt from the .trx produced by the BDT workflow:
Property: No Config Found
For the NullReferenceException issue:
I was also facing the same issue recently and the solution to it is to have the latest update of Visual Studio 2013. Right now the latest update is Update 5. I am not sure which particular update fixes this issue. I applied Update 5 and was able to successfully access the TestRunParameters in the ClassInitialize method.
You can find the updates # https://support.microsoft.com/en-us/kb/2829760
So I had two machines, on one it was all working fine and on the other I was getting the exception. I investigated that the only difference is the Update of VS; applied it and that solved the problem. :)
With NUnit 3, I was able to find the properties in the runsettings file, using
TestContext.Parameters
So in this case it would be:
string entryUrl = TestContext.Parameters["webAppUrl"];
string username = TestContext.Parameters["webAppUserName"];
string password = TestContext.Parameters["webAppPassword"];
I was able to resolve this for Unit tests by disabling Resharper. Wish I could say the same for Coded UI tests.
Why dont you use Application settings?
you can then read them anywhere like
var yesICan= Properties.Settings.Default.IToldYou;
you can create Properties out from them, pretty much you can do a lot.
public string URL_WEBOFFICE
{
get
{
return Properties.Settings.Default.WEBOFFICE_URL.Replace("***", Properties.Settings.Default.SiteName);
}
}
I have an integration testing solution. I have my tests described in XML files. In order to capitalize on Visual Studio 2010 testing infrastructure, I have a C# class where every XML test file has an associated method that loads the XML file and executes its content. It looks like this:
[TestClass]
public class SampleTests
{
[TestMethod]
public void Test1()
{
XamlTestManager.ConductTest();
}
[TestMethod]
public void Test2()
{
XamlTestManager.ConductTest();
}
...
[TestMethod]
public void TestN()
{
XamlTestManager.ConductTest();
}
}
Each method name corresponds to an XML file name. Hence, I have to have the following files in my test directory:
Test1.xml
Test2.xml
...
TestN.xml
XamlTestManager.ConductTest() uses the StackTrace class to get the name of the calling method and this way it can find the correct XML test file to load.
I would like to get rid of the extra administration of adding/removing/renaming test methods whenever I change my tests, adding/removing/renaming an XML test file. How can I automagically generate this class or its methods during the compilation process based on the actual XML files in my test directory?
Option 1:
I have considered PostSharp, but it does not allow me to look up the XML files and generate methods on the fly (or was I superficial?).
Option 2:
The other idea was to build a Visual Studio custom tool that generates my code whenever it is executed. The downside here is the deployment. The custom tool needs to be registered to VS. I want a solution that can be committed into a repository, check it out to another computer and use it right away.
(I believe in simplicity. "Check out and run" just simplifies the life of new developers soooooo much, if they do not need to go through a list of thing to install before they can compile run the application.)
Do you have any recommendation, how to get rid of the unnecessary maintenance issue?
EDIT:
For the request of Justin, I add more details. We use Bizunit (fantastic!!!) as the basis of our framework with a truckload of custom made high level test steps. From these steps we can build our test like from lego blocks in a declarative manner. Our steps include like FileDrop, WebService invokation or even polling, firing up a full blown web server to simulate a partner web application, random data generator, data comparing steps etc. Here is an example test xml (in fact XAML):
<TestCase BizUnitVersion="4.0.154.0" Name="StackOverflowSample" xmlns="clr-namespace:BizUnit.Xaml;assembly=BizUnit" xmlns:nib="clr-namespace:MyCompany.IntegrationTest;assembly=BizUnit.MyCustomSteps" xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml">
<TestCase.SetupSteps>
<nib:ClearStep FailOnError="True" RunConcurrently="False" />
<nib:LaunchSimulatedApp AppKernelCacheKey="provider" FailOnError="True" FireWakeUpCall="False" PortNumber="4000" RepresentedSystem="MyProviderService" RunConcurrently="False" />
<nib:HttpGetStep FailOnError="True" RunConcurrently="False" Url="http://localhost:10000/Home/StartSvgPolling">
<nib:HttpGetStep.Parameters>
<x:String x:Key="PolledAddress">http://localhost:4000/SvgOutputPort.asmx</x:String>
<x:String x:Key="PollingInterval">10</x:String>
<x:String x:Key="FilterFile"></x:String>
</nib:HttpGetStep.Parameters>
</nib:HttpGetStep>
</TestCase.SetupSteps>
<TestCase.ExecutionSteps>
<nib:DocumentMergeStep FailOnError="True" OutputCacheKey="inputDocument" RunConcurrently="False">
<nib:DocumentMergeStep.InputDocuments>
<nib:RandomLoader BoundingBox="Europe" LinkbackUrlPattern="http://MyProviderService/id={0}" MaxAmount="10" MaxID="100" MinAmount="10" MinID="0" NamePattern="EuropeanObject_{0}" NativeFormat="Svg" RepeatableRandomness="False" UriPrefix="European" />
<nib:RandomLoader BoundingBox="PacificIslands" LinkbackUrlPattern="http://MyProviderService/id={0}" MaxAmount="10" MaxID="100" MinAmount="10" MinID="0" NamePattern="PacificObject_{0}" NativeFormat="Svg" RepeatableRandomness="False" UriPrefix="Pacific" />
</nib:DocumentMergeStep.InputDocuments>
</nib:DocumentMergeStep>
<nib:PushToSimulatedApp AppKernelCacheKey="provider" ContentFormat="Svg" FailOnError="True" RunConcurrently="False">
<nib:PushToSimulatedApp.InputDocument>
<nib:CacheLoader SourceCacheKey="inputDocument" />
</nib:PushToSimulatedApp.InputDocument>
</nib:PushToSimulatedApp>
<nib:GeoFilterStep FailOnError="True" OutputCacheKey="filteredDocument" RunConcurrently="False" SelectionBox="Europe">
<nib:GeoFilterStep.InputDocument>
<nib:CacheLoader SourceCacheKey="inputDocument" />
</nib:GeoFilterStep.InputDocument>
</nib:GeoFilterStep>
<nib:DeepCompareStep DepthOfComparision="ID, Geo_2MeterAccuracy, PropertyBag, LinkbackUrl" FailOnError="True" RunConcurrently="False" Timeout="30000" TolerateAdditionalItems="False">
<nib:DeepCompareStep.ReferenceSource>
<nib:CacheLoader SourceCacheKey="filteredDocument" />
</nib:DeepCompareStep.ReferenceSource>
<nib:DeepCompareStep.InvestigatedSource>
<nib:SvgWebServiceLoader GeoFilter="Europe" NvgServiceUrl="http://localhost:10000/SvgOutputPort.asmx"/>
</nib:DeepCompareStep.InvestigatedSource>
</nib:DeepCompareStep>
</TestCase.ExecutionSteps>
<TestCase.CleanupSteps>
<nib:HttpGetStep FailOnError="True" RunConcurrently="False" Url="http://localhost:10000/Home/StopSvgPolling">
<nib:HttpGetStep.Parameters>
<x:String x:Key="PolledAddress">http://localhost:4000/SvgOutputPort.asmx</x:String>
</nib:HttpGetStep.Parameters>
</nib:HttpGetStep>
<nib:KillSimulatedApp AppKernelCacheKey="provider" FailOnError="True" PortNumber="4000" RunConcurrently="False" />
</TestCase.CleanupSteps>
</TestCase>
This is what it does:
Invokes a Clear operation on the test subject
Launches a webserver on port 4000 as a simulated partner app under the name MyProviderService
Invokes the test subject via HTTP Get to poll the simulated partner
Creates a new document containing geo info from two random generated content
Pushes the document to the simulated partner - hence the test subject will pick it up via polling
The test applies a geo filter on the document
The deep compare step loads the filtered document as base of comparision, and loads the content of the test subject via a web service
As clean-up, it stops the polling via an HTTP GET step and kills the simulated partner's web server.
The power of Bizunit is that merges the ease of creating tests in C# with intellisense and ease of maintaining/duplicating it in XAML files. For a quick easy read on how it works: http://kevinsmi.wordpress.com/2011/03/22/bizunit-4-0-overview/
As #GeorgeDuckett said, T4 templates are probably the way to go. In the application I am working on, we use them for a lot, including generating Repositories, Services, ViewModels, Enums and recently unit tests.
They are basically code generating scripts written in either VB or C#, looking at a directory for XML files would be no problem for these kinds of templates.
If you do choose to go the T4 route, the Tangible T4 Editor is definitely a must have, it is a free download.
Here is a quick example of a T4 script which should do or be pretty close to what you want:
<## template language="C#" debug="true" hostspecific="true"#>
<## output extension="g.cs"#>
[TestClass]
public class SampleTests
{
<#
string[] files = Directory.GetFiles(#"C:\TestFiles", "*.xml");
foreach(string filePath in files)
{
string fileName = Path.GetFileNameWithoutExtension(filePath);
#>
[TestMethod]
public void <#=fileName#>()
{
XamlTestManager.ConductTest();
}
<#
}
#>
}
Make sure this is placed in a file with the .tt extension, then on the property windows for this file, ensure the Build Action is None, Custom Tool is TextTemplatingFileGenerator.
Edit: Accessing output directory from T4 template
Add the following two lines to the top of your T4 template, under the <## template ... #> line:
<## assembly name="EnvDTE" #>
<## import namespace="EnvDTE" #>
Then inside your template, you can access and use the visual studio API like so:
IServiceProvider serviceProvider = this.Host as IServiceProvider;
DTE dte = serviceProvider.GetService(typeof(DTE)) as DTE;
object[] activeSolutionProjects = dte.ActiveSolutionProjects as object[];
if(activeSolutionProjects != null)
{
Project project = activeSolutionProjects[0] as Project;
if(project != null)
{
Properties projectProperties = project.Properties;
Properties configurationProperties = project.ConfigurationManager.ActiveConfiguration.Properties;
string projectDirectory = Path.GetDirectoryName(project.FullName);
string outputPath = configurationProperties.Item("OutputPath").Value.ToString();
string outputFile = projectProperties.Item("OutputFileName").Value.ToString();
string outDir = Path.Combine(projectDirectory, outputPath);
string targetPath = Path.Combine(outDir, outputFile);
}
}
outDir and targetPath contain the output directory and the full path to the output file.
Instead of creating a separate test for each set of test data you can create a single test that is repeatedly run for each set of test data:
[TestClass]
public class SampleTests
{
[TestMethod]
public void Test()
{
for (var i = 0; i < 10; ++i)
XamlTestManager.ConductTest(i);
}
}
You can also perform data-driven tests by using the DataSource attribute. This will perform your test for each row in your data set.
[TestClass]
public class SampleTests
{
public TestContext Context { get; set; }
[TestMethod]
[DataSource(...)]
public void Test()
{
var someData = Context.DataRow["SomeColumnName"].ToString();
...
}
}
I actually don't think this is a job for build-time code generation, I think you should use data attributes to drive the tests in this case.
If you used xunit you could do that like this:
public class SampleTests
{
[Theory]
[InlineData(1)]
[InlineData(2)]
[InlineData(...)]
[InlineData(N)]
public void Test(int x)
{
XamlTestManager.ConductTest(x);
}
}
And it will run the test once per InlineData attribute. Also I believe there is another attribute that you can pass a path to a file and it will populate your parameters with values from that file...
I think NUnit has a similar feature but XUnit is much better, I would recommend using XUnit instead.
Just answered alike "code generation from XML with T4" question.
https://stackoverflow.com/a/8554949/753110
Your requirement matches exactly what we did initially (and what lead to discovery of the ADM described on that answer).
We are currently working on test-case based generation, where the test-cases are actually built by the testing-staff, yet the complete integrationtests through code are generated to support them.
Added custom XML based generation demo for that other example, if you want to see:
https://github.com/abstractiondev/DemoSOCase8552428ABS