Develop Reference

FsCheck integration with NUnit in C#

TL;DR: I'm not able to successfully use FsCheck with NUnit in C#: either:
it tells me on stdout that the test failed but the test still appears green
it tells me that it doesn't find any test to run
or I don't understand how to apply in C# the doc I read
I think a dummy but complete example would help...
(More details)
1st step: test remains green
I installed the Nuget package FsCheck.NUnit (2.10.4), and naively tried:
[NUnit.Framework.Test]
public void SomeTest()
{
// Note the subtle bug: I Reverse only once, because I want the test to fail
Func<int[],bool> revRevIsOrig = xs => xs.Reverse().SequenceEqual( xs );
Prop.ForAll(revRevIsOrig).QuickCheck();
}
When I run it as I would run any NUnit test, it ends up green, even though I see on stdout that
Falsifiable, after 3 tests (1 shrink) (StdGen (2129798881,296376481)):
Original:
[|-1; 0|]
Shrunk:
[|1; 0|]
2nd step: test inconclusive
So I went on, found some doc, and noticed that I should use Property instead of Test. So I changed my code to
[FsCheck.NUnit.Property] // <-- the line that changed
public void SomeTest()
{
Func<int[],bool> revRevIsOrig = xs => xs.Reverse().SequenceEqual( xs );
Prop.ForAll(revRevIsOrig).QuickCheck();
}
I launched the test from Visual, and it ended up with the status inconclusive. And some log was telling me:
Cannot run tests: No suitable tests found in 'xxx.exe'. Either assembly contains no tests or proper test driver has not been found
3rd step: I notice I didn't understood the doc
When I re-read the doc, I notice that it says that my test method can take argument and should return a property. I'm obviously not doing it since I return nothing.
The sad thing is that I don't understand what I'm actually supposed to do (and I'm not familiar enough with F# to understand the example below...)... (I've blindly tried some random stuff that looked they could make sense, but I never ended up with a red test)
I would really appreciate any pointer to help me get this test red!

Try using the QuickCheckThrowOnFailure function
the QuickThrowOnFailure ensures that if the test fails, an exception with the necessary information is raised so the runner know the test failed.
[Test]
public void SomeTest() {
// Note the subtle bug: I Reverse only once, because I want the test to fail
Func<int[],bool> revRevIsOrig = xs => xs.Reverse().SequenceEqual( xs );
Prop.ForAll(revRevIsOrig).QuickCheckThrowOnFailure();
}

The other way to use FsCheck with NUnit (from FsCheck github examples) is
[FsCheck.NUnit.Property(Verbose = true)]
public void SomeTest()
{
Func<int[],bool> revRevIsOrig = xs => xs.Reverse().SequenceEqual( xs );
return Prop.ForAll(revRevIsOrig);
}
But this way doesn't report you why your test fails. So the Nkosi's answer now is better but I think it will be fixed somedays.

Besides other mentioned options you can turn this test to a Property Based Test by using the Property attribute and correct return type (I've included namespaces for clarity).
Note the .ToProperty call to turn it into a Property.
[FsCheck.NUnit.Property]
public FsCheck.Property RevRev_IsOriginal(int[] xs)
{
Func<bool> revRevIsOrig = () => xs.Reverse().Reverse().SequenceEqual(xs);
return revRevIsOrig.ToProperty();
}
The framework will generate testdata via the int[] xs method parameter.
This requires nuget packages fscheck and fscheck.nunit (besides NUnit3TestAdapter and Microsoft.NET.Test.Sdk in my .NET Core 3.1 testproject).

How to use ITestEventListener in NUnit 3?

I want to add a custom test reporter to NUnit. I already did it with NUnit2, but I now need to use NUnit3.
To implement the reporter, I need to get various events from the framework, like start, end and failure of tests.
In NUnit2 I used NUnitHook to register my EventListener and it worked pretty good.
In NUnit3 I need to use the extension point mechanism, but when I add the extension point to the project, VisualStudio (2012 ultimate) immediately fails to discover the NUnit tests.
[TypeExtensionPoint(Description = "Test Reporter Extension")]
public class MyTestEventListener : ITestEventListener
{
public void OnTestEvent(string report)
{
Console.WriteLine(report);
}
}
If I remove the ITestEventListener implementation declaration from the class, it rediscovers the tests perfectly.
[TypeExtensionPoint(Description = "Test Reporter Extension")]
public class MyTestEventListener //: ITestEventListener
{
public void OnTestEvent(string report)
{
Console.WriteLine(report);
}
}
Am I doing something wrong? is there a better way to achieve it?

You don't say where you are putting this code, but I am suspecting it's in your test assembly. If so, that's not where it belongs. NUnit engine extensions get installed into the NUnit engine, so they need to be in a separate assembly. Once you have a separate assembly, you need to tell the engine where it is. Currently, you do this by creating a file of type .addins in the same directory as the engine. (You could modify the existing file, but that introduces maintenance problems in the future)
A future release will have an easier way to install addins, but they will continue to be entirely separate from your tests.
A further problem is that you are using TypeExtensionPointAttribute. I didn't notice this originally in your code and it's probably the biggest error so I'm adding this info now.
An "ExtensionPoint" is the thing you are extending. NUnit defines ExtensionPoints, while you create Extenisons to extend them. TypeExtensionPointAttribute is used inside NUnit to define extension points. It's not used by you. You use the ExtensionAttribute to define your extension.
Your extension should be defined something like this:
[Extension(Description = "Test Reporter Extension", EngineVersion="3.4")]
public class MyTestEventListener : ITestEventListener
{
public void OnTestEvent(string report)
{
Console.WriteLine(report);
}
}
You don't say what version of NUnit you are running. Test Listeners are only supported beginning with version 3.4. The EngineVersion property above is purely documentary at this point, because 3.4 is also the first version to recognize it.
There is a new writeup in the NUnit docs that may be helpful: https://github.com/nunit/docs/wiki/Writing-Engine-Extensions

How to access TestRunParameters within RunSettings file

Reading through https://msdn.microsoft.com/en-us/library/jj635153.aspx I have created a .RunSettings files with a few parameters similar to the example:
<TestRunParameters>
<Parameter name="webAppUrl" value="http://localhost" />
<Parameter name="webAppUserName" value="Admin" />
<Parameter name="webAppPassword" value="Password" />
</TestRunParameters>
I plan on having a .RunSettings file for each of our environments with appropriate URLs and credentials for running a CodedUI test on the specified RunSettings file's environment.
I can see that from command line to reference the Settings file I can run:
vstest.console myTestDll.dll /Settings:Local.RunSettings /Logger:trx
vstest.console myTestDll.dll /Settings:QA.RunSettings /Logger:trx
etc...
But I don't see any way that calls out how to actually utilize the TestRunParameters from within the codedUI test.
What I would like to do is set up test initializers that use the TestRunParameters to determine where to log in, and what credentials to use. Something like this:
[TestInitialize()]
public void MyTestInitialize()
{
// I'm unsure how to grab the RunSettings.TestRunParameters below
string entryUrl = ""; // TestRunParameters.webAppUrl
string userName = ""; // TestRunParameters.webAppUserName
string password = ""; // TestRunParameters.webAppPassword
LoginToPage(entryUrl, userName, password);
}
public void LoginToPage(string entryUrl, string userName, string password)
{
// Implementation
}
Information on how to reference the TestRunParameters is greatly appreciated!
EDIT
/// <summary>
/// Summary description for CodedUITest1
/// </summary>
[CodedUITest]
public class CodedUITest1
{
public static string UserName = string.Empty;
[ClassInitialize]
public static void TestClassInitialize(TestContext context)
{
UserName = context.Properties["webAppUserName"].ToString();
Console.WriteLine(UserName);
}
[TestMethod]
public void CodedUITestMethod1()
{
this.UIMap.RecordedMethod1();
// To generate code for this test, select "Generate Code for Coded UI Test" from the shortcut menu and select one of the menu items.
}
// Rest of the default class - TestContext instantiation, UI map instantiation, etc
}
The exception I'm getting when running:
NullReference Exception
#williamfalconeruk I have updated my test class as above, but I am still getting the same error, any idea what I'm doing wrong?

For those that use Resharper with this issue, I discovered the fix (no need to disable Resharper):
Go to Visual Studio top menu -> Resharper -> Options
Find the Tools section, expand "Unit Testing"
Click on "MsTest". The checkbox should be on enabled, but the Test Settings file path below that may be blank. If it is, click on browse and select the runsettings file you want to use.
Click save, rebuild and try to run the tests, the parameters should now work.
Not sure why but simply selecting the Test Settings file from the Tests menu -> Test Settings does not actually work when using Resharper, so this file needs to be explicitly pointed to directly in the Resharper options.

I also came across this recently, as we wanted to move away from legacy environment variable usage. The solution provided below was suitable for our needs, but there may be a better one...
It turns out you can access these in the TestContext of a ClassInitialize method of your Test fixture. There is a Properties dictionary which has these parameters, and you could access the values here:
[ClassInitialize]
public static void TestClassinitialize(TestContext context)
{
var webAppUrl = context.Properties["webAppUrl"].ToString();
//other settings etc..then use your test settings parameters here...
}
Note: these are static so if you need access to this you may need to set up static properties to access within your Test code.
An alternative suggested is using a Data Driven Test approach.
Here's some basic info on data driven tests here which may also help:
https://msdn.microsoft.com/en-us/library/ms182527.aspx
As I said, the above solution suited our needs at a basic level.
UPDATE: see image below in response to test settings returning null...

This works for me (VS2017-pro):
namespace TestApp.Test
{
[TestClass]
public class UnitTest1
{
// This enables the runner to set the TestContext. It gets updated for each test.
public TestContext TestContext { get; set; }
[TestMethod]
public void TestMethod1()
{
// Arrange
String expectedName = "TestMethod1";
String expectedUrl = "http://localhost";
// Act
String actualName = TestContext.TestName;
// The properties are read from the .runsettings file
String actualUrl = TestContext.Properties["webAppUrl"].ToString();
// Assert
Assert.AreEqual(expectedName, actualName);
Assert.AreEqual(expectedUrl, actualUrl);
}
[TestMethod]
public void TestMethod2()
{
// Arrange
String expectedName = "TestMethod2";
// Act
String actualName = TestContext.TestName;
// Assert
Assert.AreEqual(expectedName, actualName);
}
}
}
Make sure to select the runsettings file you wish to use, here: Test -> Test Settings.

An alternative to disable Resharper is to enable MSTest support and select the test setting file on Resharper Options dialog (->Tools->Unit Testing->MsTest).

I was trying to do this exact thing as well. As many of you may know, running tests through MTM exposes some additional properties to the TestContext, including the name of the Run Settings used. I used this property as a "Foreign Key" of sorts for our test data, allowing us to specify environment URLs etc. without hardcoding them or using the incredibly lackluster "Data Driving" tools that come with out of the box testing.
Of course, there's no way to expose any run-time properties when executing tests as part of a BDT or release workflow besides what #kritner is attempting which microsoft describes HERE. However if you read the comments of that link you'll discover what you may be able to infer here:
You need to use VS 2013 R5 or VS 2015 to use this solution
It will only work for Unit Tests!
Those of us who are trying to execute UI or Load tests as part of a CI or CD workflow are completely screwed. You don't get any additional properties in testContext, even when executing a Plan/Suite with certain test configurations (not settings) created in MTM. #Adam may have been able to get this to work when running vs debugging, but that may have only worked with unit tests. Through CodedUI I've been unable to retrieve the properties without getting a NullReferenceException. Here's an example of the janky code I was using to investigate:
if (testContextInstance.Properties["__Tfs_TestConfigurationName__"] != null) //Exposed when run through MTM
{
TFSTestConfigurationName = testContextInstance.Properties["__Tfs_TestConfigurationName__"].ToString();
}
else TFSTestConfigurationName = "Local"; //Local
var configName = testContextInstance.Properties["configurationName"] ?? "No Config Found";
Trace.WriteLine("Property: " + configName);
And the XML of my .runsettings file:
<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
<!-- Parameters used by tests at runtime. These are required as a substitute for TFS/MTM test settings.-->
<!-- File instructions: https://msdn.microsoft.com/en-us/library/jj635153.aspx#example -->
<!-- TFS instructions: https://blogs.msdn.microsoft.com/visualstudioalm/2015/09/04/supplying-run-time-parameters-to-tests/ -->
<TestRunParameters>
<Parameter name="configurationName" value="Local" />
</TestRunParameters>
</RunSettings>
And an excerpt from the .trx produced by the BDT workflow:
Property: No Config Found

For the NullReferenceException issue:
I was also facing the same issue recently and the solution to it is to have the latest update of Visual Studio 2013. Right now the latest update is Update 5. I am not sure which particular update fixes this issue. I applied Update 5 and was able to successfully access the TestRunParameters in the ClassInitialize method.
You can find the updates # https://support.microsoft.com/en-us/kb/2829760
So I had two machines, on one it was all working fine and on the other I was getting the exception. I investigated that the only difference is the Update of VS; applied it and that solved the problem. :)

With NUnit 3, I was able to find the properties in the runsettings file, using
TestContext.Parameters
So in this case it would be:
string entryUrl = TestContext.Parameters["webAppUrl"];
string username = TestContext.Parameters["webAppUserName"];
string password = TestContext.Parameters["webAppPassword"];

I was able to resolve this for Unit tests by disabling Resharper. Wish I could say the same for Coded UI tests.

Why dont you use Application settings?
you can then read them anywhere like
var yesICan= Properties.Settings.Default.IToldYou;
you can create Properties out from them, pretty much you can do a lot.
public string URL_WEBOFFICE
{
get
{
return Properties.Settings.Default.WEBOFFICE_URL.Replace("***", Properties.Settings.Default.SiteName);
}
}

How to cover multiple test cases in one automation script

In many cases I find it redundant to write a script for each little test case. How can I use Microsoft VS to write a script that can test more than one test case and report the result to each associated test case in Microsoft MTM without running each test cases separately. Say, for example, I have a yes/No/Cancel dialog that pops up and there is a test case to verify for each of the three cases. All three cases can be verified in one script. Would it be possible to associate each test case to the same script and get the results reported to each one by running the script only once?

You can use MSTest.exe or VSTest.console.exe to run TestMethods via command line. One can call MSTest.exe or VSTest.console.exe in batch file.
Assign user defined testcategory attribute to group your tests. Please refer Defining Test Categories to Group Your Tests. For Example
[TestCategory("Nightly"),
TestCategory("Weekly"),
TestCategory("ShoppingCart"),
TestMethod()]
public Void DebitTest()
{
}
[TestCategory("Nightly"),
TestCategory("Weekly"),
TestCategory("ShoppingCart"),
TestMethod()]
public Void CreditTest()
{
}
[TestCategory("Nightly"),
TestCategory("Daily"),
TestCategory("ShoppingCart"),
TestMethod()]
public Void BVTTest1()
{
}
[TestCategory("Nightly"),
TestCategory("Daily"),
TestCategory("ShoppingCart"),
TestMethod()]
public Void BVTTest2()
{
}
Running Tests via VSTest.Console.exe group by TestCategory
Vstest.console.exe myTestProject.dll /TestCaseFilter:”TestCategory=Nightly"
Running Tests by MSTest.exe group by Test Category
mstest /testcontainer:MyTestprojectName.dll /category:"Nightly"
mstest /testcontainer:MyTestprojectName.dll /category:"Daily"
Please refer MSDN Link for more command line options in this topic.

What does `DeploymentItem` attribute mean?

Let's say we have a short program:
namespace ConsoleTryIt
{
static class Program
{
static void Main(string[] args)
{
var sum = Add(1, 2);
}
private static int Add(int p, int p2)
{
return p + p2;
}
}
}
When create the unit test class for this class, Visual Studio create a test method with the attribute DeploymentItem. I read MSDN about this attribute but still don't get what it means.
/// <summary>
///A test for Add
///</summary>
[TestMethod()]
[DeploymentItem("ConsoleTryIt.exe")]
public void AddTest()
{
var expected = 122;
var actual = Program_Accessor.Add(1, 121);
Assert.AreEqual(expected, actual);
}
If you get the idea, please share!
Edit
Thanks everyone for your answers. So the idea is to copy the item given in the argument to the testing evironment's folder. My next question is: why does this method need this attribute while others don't?
I guess it's related to the private members on the tested class but nothing clear to me.
Please continue to discuss.

This specifies files that are required by the specific test. The test system creates a new directory where the tests are run from. With this attribute, you can make the test system copy specific files to that new directory.

Is used deploy files that are not necessary present in the Output directory to the folder used for that particular TestRun.
in the example you posted above, the test environment makes sure that "consoleTryIt.exe" is copied(and therefore present) in the test folder.
If the file is not found, the test is not even run, and a FileNotFound Exception is returned.

This means that the item is copied to the 'TestResults\Out' folder and is basically an artifact/necessary item for the test run. Thus, it is stored apart from the bin directory and does not get overwritten.
This is especially useful when running the tests in different environments (build server, no hard-coded paths...) and of course it is necessary to make the tests repeatable.
HTH.
Thomas

Ensures the files required for the test are copied over to the folder where MSTest runs its tests TestResults\Out.
Files in your solution should have "Copy Always" set for the files to be first copied to bin folder and then to MSTest's folder.
Make sure you have "Enable deployment" checked in testrunconfig.