I have a question about the NUnit retry attribute
We are using
NUnit 3.12.0
NUnit3TestAdapter 3.17.0
C# and Selenium
This is a typical feature file
Test.feature
Scenario Template: 01 Test footer options - Homeowner
Given we set the destination to the homepage
When we select footer option "1"
Then we should see the url "tester" and title "test page"
This is the hook
Hooks.cs
[BeforeFeature(Order = 1)]
[Test, Retry(2)]
public static void BeforeFeatureTab(FeatureContext featureContext)
{
etc
}
And here we have were the fail is being captured
public void AssertUrlContains(string comparing, IWebDriver driver)
{
var currentUrl = driver.Url;
try
{
WriteToReport(Status.Pass, "Pass: URL is correct");
Assert.That(currentUrl.Contains(comparing));
}
catch (Exception ex)
{
WriteToReport(Status.Fail, "Fail error: " + ex);
Assert.That(false);
}
}
Now I thought the Assert.That(false) would trigger a retry, but it is now. Can anyone see what I got wrong or missed pleased.
Thanks
Kev
Although you didn't tag it, I'm assuming you are using SpecFlow. While I'm not that knowledgeable about it, I've learned a bit from helping folks who use NUnit with it. That said, here's what I think...
The BeforeFeatureAttribute translates into an NUnit OneTimeSetUpAttribute (formerly TestFixtureSetUpAttribute, called before any tests are run. OTOH, the TestAttribute indicates that the method is a test case. IOW, you are saying that the same method should be called by NUnit to initialize the fixture and as a test. That's probably leading to an exception being thrown.
In NUnit, unexpected exceptions are not treated as failures but as errors. An error is typically something that is wrong with your test rather than the system you are testing, so it is handled differently. In particular, only failures are retried, while errors are not.
Note also that NUnit knows nothing about SpecFlow, so if it's a specflow exception, that will be treated as an error like any other exception.
Your best bet is to stop using the same method as both a test and for initializing a fixture.
Related
I am using the testing framework that comes with Visual Studio, along with NSubstitute to unit test a method that takes a system ID, and throws an exception if the system can't be found in the database...
public VRTSystem GetSystem(int systemID)
{
VRTSystem system = VrtSystemsRepository.GetVRTSystemByID(systemID);
if (system == null)
{
throw new Exception("System not found");
}
return system;
}
(In case this seems odd, there is a specific business case for this method that requires it to throw an exception, as returning a null system is not acceptable for its usage)
I want to write a test to check that an exception is thrown if the system doesn't exist. I currently have the following...
[TestMethod]
public void LicensingApplicationServiceBusinessLogic_GetSystem_SystemDoesntExist()
{
var bll = new LicensingApplicationServiceBusinessLogic();
try
{
VRTSystem systemReturned = bll.GetSystem(613);
Assert.Fail("Should have thrown an exception, but didn't.);
}
catch () { }
}
By not mocking the repository, the system returned by VrtSystemsRepository.GetVRTSystemByID() will be null, and the exception thrown. Whilst this works, it looks wrong to me. I wouldn't have expected to need a try/catch block in a test.
The NSubstitute docs have an example that implies I should be able to test this as follows...
[TestMethod]
public void GetSystem_SystemDoesntExist()
{
var bll = new LicensingApplicationServiceBusinessLogic();
Assert.Throws<Exception>(() => bll.GetSystem(613));
}
However, if I try this in my test code, I get Throws highlighted in red, with the error message "Assert does not contain definition for Throws"
Now, I'm not actually sure that the sample on that page covers my scenario, as the test code specifies that the method under test throws an exception, which I don't really understand, as I thought the idea of testing was to leave the method under test alone, and test what happens under various scenarios. However, even without that, I don't understand why the Assert.Throws method doesn't exist.
Anyone any ideas?
Edit: DavidG pointed out that Assert.Throws is probably part of NUnit, not the MS framework, which would explain why it's not recognised. If so, is the way I'm currently testing the right way to do it?
As mentioned by DavidG The referenced documentation is using NUnit for assertions.
If not using that framework you can use the ExpectedExceptionAttribute Class
[TestMethod]
[ExpectedException(typeof(<<Your expected exception here>>))]
public void GetSystem_SystemDoesntExist() {
var bll = new LicensingApplicationServiceBusinessLogic();
bll.GetSystem(613);
}
which would fail if the expected exception is not thrown.
I'm trying to figure out how to have tests fail as 'Inconclusive' if some required setup that also exists as a test fails.
This is an overly simplified example of what I'm trying to do, but hopefully it illustrates the point:
[Test] public void TestSetter() {
Assert.That(() => myInstance.Property = "test", Throws.Nothing);
}
[Test] public void TestGetter() {
try { TestSetter(); } catch (AssertionException) {
Assert.Inconclusive("Unable to test, prereq failed");
}
Assert.That(myInstance.Property, Is.EqualTo("test"));
}
This doesn't seem to work though, if I force the TestSetter test to fail, both still show the 'Failed' result, instead of the TestGetter resulting in 'Inconclusive'. I stepped through the code, it's definitely hitting the Assert.Inconclusive call, but it seems the earlier AssertionException is still getting precedence.
Is there any way of getting this to correctly report 'Inconclusive'?
Using C#7, with NUnit 3.6.0
Try upgrading to NUnit Framework v3.6.1, and see if that runs as you expect.
There was a change in v3.6.0 relating to catching AssertionExceptions, which was later reverted.
See https://github.com/nunit/nunit/issues/2043 for details. To summarise - you should be aware that catching AssertionException's isn't strictly a supported interface to write tests to, and it may be worthwhile refactoring, to test for setup success in a different way.
I use SeleniumWebdriver C# to build automated tests.
I also use NUnit to mark test methods (so I can run them).
In each of the tests there are several verification and when the first verification fails then the test stops executing (an exception is thrown).
I want the test to continue executing so more errors could be found!
Please guys, give me a clue how to do it right.
I think about something like that:
Instead of throwing Exception in verification I'll add an error to a stack
When a test ends I check if my stack is not empty
If the stack is not empty I push to console all errors and fail the test
If the stack is empty then the test passed successfully.
Are these steps good? Is there a better way?
I think this should work for you.
class Asserts
{
private static StringBuilder _stack;
[SetUp]
public void SetUp()
{
_stack = new StringBuilder();
}
[TearDown]
public void TearDown()
{
if (_stack.Lenght != 0) Assert.Fail(_stack.ToString());
}
[Test]
public void Test()
{
AssertHelper(() => Assert.AreEqual(0, 0));
AssertHelper(() => Assert.IsNotNull(null));
AssertHelper(() => Assert.AreEqual(3, 4));
AssertHelper(() => Assert.AreEqual(1, 1));
}
private static void AssertHelper(Action assert)
{
try
{
assert();
}
catch (AssertionException e)
{
_stack.Append(e.Message);
}
}
}
But the good idea is to keep one verification per one test.
Generally you want your tests to only be verifying one thing at a time. If there's 2 steps to an action that require asserting, considering having tests like:
UserForm_FirstNameMissing_ThrowsException
UserForm_SecondNameMissing_ThrowsException
UserForm_AgeTooLarge_ThrowsException
instead of having a single test called something like:
UserForm_TestValidation
This means that - like you wanted - if one stage of validation fails, you can continue to test the rest. Your best bet for this may be a singular test with a dataset, however keeping the answer simple yet relevant, a test per failure state is good enough for what you want.
I solved my problem in the next way:
I've created a set of methods that are performed once prior to executing any of the tests ([TestFixtureSetUpAttribute] in NUnit);
In this set of methods I do the main actions results of which I'll check (in my case I buy a lottery ticket);
In each test ([Test] in NUnit) I check one assertion.
So now I have a main action that doesn't include any verification. After this actions is completed I check all needed assertions.
Hi all I'm making a Chess AI as hobby project with Test Driven Development. But for some reason the only message my tests give is "test has thrown an exception: ...". And therefore omitting the only thing that matters. Instead of directly reading the error I now have to right-click and view test result details. I have tried adding and removing columns, but I cannot get the whole message to be shown directly.
Can VS2010 be setup so the exception message gets shown directly for each unit test?
edit: I am using standaard VS unit tests:
[TestClass]
public class MiniMaxTest
{
[TestMethod]
public void TestConstructor()
{
throw new Exception("Must I click view details to see this?");
}
}
Why these questions? You guys can reproduce these things. Debug or Run tests give the same messages:
No, I don't believe you can configure VS to show it differently. The IDE shows the first line of the exception message, and there's a newline character in the full message text, so you'll need to click through to the details to view the whole thing.
What you can do however, is abuse the MSTest framework to show your own messages.
MS Test Assertions are all implemented by throwing exceptions. All the MS Test Assert functions throw exceptions which are derived from UnitTestAssertException. Visual studio has special handling for these kinds of exceptions.
For example: If you write this test:
[TestMethod]
public void AllAboard()
{
throw new AssertFailedException("Failboat");
}
AssertFailedException is the standard base class for most other assertion failures.
You'll note that VS2010 does not print the generic "test threw an exception" message, instead it just prints "Failboat".
Now what you can do is surround your tests in things that convert normal exceptions into AssertFailedException and then you can print whatever messages you like.
[TestMethod]
public void TestStuff()
{
try
{
string o = null; // pretend this came from some real code
var x = o.Split(new[] { ',' }); // this will throw NullRefException
}
catch (Exception e)
{
throw new AssertFailedException(e.Message, e);
}
}
Of course, I wouldn't recommend actually doing this... it's verbose, and more importantly, you lose the call stack... but hey, now you've got one more tool in your toolbelt
In the past I have tested for expected exceptions like this:
[TestMethod]
public void TestThrowsException() {
try {
Foo();
Assert.Fail();
} catch (MyException ex){//good
}
}
However I notice that there is a (cleaner?) way to test this using the ExpectedException attribute. Why does this test method pass when the exception is not thrown? Surely this defeats the purpose of the attribute.
[TestMethod]
[ExpectedException(typeof(MyException))]
public void TestThrowsException() {
}
[Edit] I am running this test using Silverlight 2
I've never seen that pass - is that really all you've got? Are you absolutely sure you have marked it as a TestMethod? Does the test runner show it passing? Have you definitely got the most recent code?
I'll double check, but I'm sure that will fail...
Jon Skeet was in fact right, I did have an old version of the testing framework. I updated to the Dec 08 release here http://code.msdn.microsoft.com/silverlightut/Release/ProjectReleases.aspx?ReleaseId=1913 and got the expected behaviour when tagging with ExpectedException.
I've actually experienced that ReSharper 4.5 testrunner does not work with ExpectedException in NUnit 2.5. ...but this looks like MSTest ... can you elaborate on which test framework you are using and which test runner you are using to execute the tests?
You should use 'ExpectedException' with an additional assert.fail, so that the test fails if the exception is not thrown (still in VS 2010 with .net 4 and Microsoft.VisualStudio.QualityTools.UnitTestFramework):
[TestMethod]
[ExpectedException(typeof(MyException))]
public void TestThrowsException() {
Do.SomethingThatThrowsAnException();
assert.fail("No exception ;-(");
}