This is driving me crazy! I am using NUnit version 3.6.0 and I am getting unexpected results from my tests. I have created a really simple test to demonstrate the problem I have:
[TestFixture]
public class NunitTest
{
[Test]
public void TestIt()
{
string x = "x";
string y = "y";
// this fails (expected) but with NullReferenceException (unexpected)
Assert.That(x, Is.EqualTo(y));
}
}
When the test runs I get a NullReferenceException when I am expecting something like "expected is 'X', actual is 'Y'"
I have furthered the asserts and they all pass
[TestFixture]
public class NunitTest
{
[Test]
public void TestIt()
{
string x = "x";
string y = "y";
// this passes
bool atest = x.Equals(y);
Assert.IsFalse(atest);
// this passes
Assert.IsNotNull(x);
// this passes
Assert.IsNotNull(y);
// this fails (expected) but with NullReferenceException (unexpected)
Assert.That(x, Is.EqualTo(y));
}
}
The other weird thing is that when the values are equal the test passed correctly. Also worth noting I get exactly the same problem when I use Assert.AreEqual(x, y)
Stack Trace:
at NUnit.Framework.Assert.ReportFailure(String message)
at NUnit.Framework.Assert.ReportFailure(ConstraintResult result, String message, Object[] args)
at NUnit.Framework.Assert.That[TActual](TActual actual, IResolveConstraint expression, String message, Object[] args)
at NUnit.Framework.Assert.That[TActual](TActual actual, IResolveConstraint expression)
at VSI.Tests.NunitTest.TestIt() in c:\Users\DavidBecker\Documents\Visual Studio 2013\Projects\VSI.Tests\VSI.Tests\NunitTest.cs:line 30
Tools:
.NET Framework version 4.5.1
Visual Studio 2013 Professional
Resharper Version 9
EDIT: I am running the tests from Resharper.
You're right that ReSharper 9 is the problem.
ReSharper didn't support NUnit 3 until ReSharper version 10. There are however are a number of similarities between NUnit versions 2 and 3, which mean that some older NUnit 2 runners will appear to run NUnit 3 tests - however, this won't be done accurately, and is neither supported or advised.
In the final version of NUnit 2 (2.6.4) - an exception was put in, such that a runner would error if attempting to load NUnit 3 tests would fail immediately, instead of attempting to run tests in an unsupported manner. It seems likely however that ReSharper 9 is using a runner older than NUnit 2.6.4, which doesn't throw this error.
The reason this has only become apparent to you with NUnit 3.6 is due to an internal change with how assertions are registered within NUnit. The coincidental previous success now no longer succeeds - although in many ways, this is an advantage, as it gets rid of this right-side failure, where tests appear to be passing, but aren't actually running correctly.
ReSharper supports NUnit 3 from version 10, although that was the point I believe when they switched from a one-off purchase model to a subscription model. As another option - consider the NUnit 3 Test Adapter (NuGet package or VSIX extension) - which allows you to run NUnit tests individually, using Visual Studio's own test window.
(If your interested in further info, this issue came up recently over on GitHub also: https://github.com/nunit/nunit/issues/1992)
Running NUnit.3.6.0 tests in Resharper (9) Unit Test Runner has unexpected results. Use NUnit.ConsoleRunner.3.6.0 instead.
(this worked for me, if anyone has better way or can make it work in RS9 then please let me know).
Related
Visual Studio 2019 Version 16.8.4
Resharper Version 2020.3.2
Nunit 3.13.1
if I wasn't wrong, i should be using the test runner from Resharper.
I put my test code below. It has no logic but creating a large test case set.
The test runner just freeze for a really long time (as you can see at screendump) and only start running test after 20-30 mins wait.
In actual development, for the sake of 95%+ unit test coverage, we do have some test case that has more than 80,000 cases that cannot be split or remove.
Is there any other alternative way to make the test run work ?
[TestFixture]
public class Test_LargeTestCaseSet
{
private static IEnumerable<TestCaseData> LargeTestCases()
{
for(var i = 0; i < 100000; i++)
{
yield return new TestCaseData(i);
}
}
[TestCaseSource(nameof(LargeTestCases))]
public void LargeTest(int i)
{
Assert.AreEqual(i, i);
}
}
Yes, there is! Do not declare your test as 100k separate test-cases. Obviously, it would not work well with UI that has a visual representation for each individual test-case.
Instead, just write the loop inside of the test method itself. Use Assert.Multiple if you want all of "test-cases" to run on each execution of the test. If knowing the first encountered failure is enough, simple Assert.XXX would suffice.
I have a question about the NUnit retry attribute
We are using
NUnit 3.12.0
NUnit3TestAdapter 3.17.0
C# and Selenium
This is a typical feature file
Test.feature
Scenario Template: 01 Test footer options - Homeowner
Given we set the destination to the homepage
When we select footer option "1"
Then we should see the url "tester" and title "test page"
This is the hook
Hooks.cs
[BeforeFeature(Order = 1)]
[Test, Retry(2)]
public static void BeforeFeatureTab(FeatureContext featureContext)
{
etc
}
And here we have were the fail is being captured
public void AssertUrlContains(string comparing, IWebDriver driver)
{
var currentUrl = driver.Url;
try
{
WriteToReport(Status.Pass, "Pass: URL is correct");
Assert.That(currentUrl.Contains(comparing));
}
catch (Exception ex)
{
WriteToReport(Status.Fail, "Fail error: " + ex);
Assert.That(false);
}
}
Now I thought the Assert.That(false) would trigger a retry, but it is now. Can anyone see what I got wrong or missed pleased.
Thanks
Kev
Although you didn't tag it, I'm assuming you are using SpecFlow. While I'm not that knowledgeable about it, I've learned a bit from helping folks who use NUnit with it. That said, here's what I think...
The BeforeFeatureAttribute translates into an NUnit OneTimeSetUpAttribute (formerly TestFixtureSetUpAttribute, called before any tests are run. OTOH, the TestAttribute indicates that the method is a test case. IOW, you are saying that the same method should be called by NUnit to initialize the fixture and as a test. That's probably leading to an exception being thrown.
In NUnit, unexpected exceptions are not treated as failures but as errors. An error is typically something that is wrong with your test rather than the system you are testing, so it is handled differently. In particular, only failures are retried, while errors are not.
Note also that NUnit knows nothing about SpecFlow, so if it's a specflow exception, that will be treated as an error like any other exception.
Your best bet is to stop using the same method as both a test and for initializing a fixture.
I'm trying to figure out how to have tests fail as 'Inconclusive' if some required setup that also exists as a test fails.
This is an overly simplified example of what I'm trying to do, but hopefully it illustrates the point:
[Test] public void TestSetter() {
Assert.That(() => myInstance.Property = "test", Throws.Nothing);
}
[Test] public void TestGetter() {
try { TestSetter(); } catch (AssertionException) {
Assert.Inconclusive("Unable to test, prereq failed");
}
Assert.That(myInstance.Property, Is.EqualTo("test"));
}
This doesn't seem to work though, if I force the TestSetter test to fail, both still show the 'Failed' result, instead of the TestGetter resulting in 'Inconclusive'. I stepped through the code, it's definitely hitting the Assert.Inconclusive call, but it seems the earlier AssertionException is still getting precedence.
Is there any way of getting this to correctly report 'Inconclusive'?
Using C#7, with NUnit 3.6.0
Try upgrading to NUnit Framework v3.6.1, and see if that runs as you expect.
There was a change in v3.6.0 relating to catching AssertionExceptions, which was later reverted.
See https://github.com/nunit/nunit/issues/2043 for details. To summarise - you should be aware that catching AssertionException's isn't strictly a supported interface to write tests to, and it may be worthwhile refactoring, to test for setup success in a different way.
Using VS 2010, NUnit, Typemock, Entity Framework...
I am having a bit of an odd problem using NUnit/Typemock. I am relative new to the testing world so it could be a beginners mistake. This is the problem I am having.
Test Project is compiled.
Launch NUnit from within Visual Studio
Run tests from within NUnit client app. First run, first test always fails.
Rerun and test passes.
It does not matter what test is first. I can select a particular test. If it is the first one to run, it fails on first execution. It passes on rerun.
This is an example class having the problem. But the class doesn't matter. Whatever test is run first has this problem. The exception code was just something that was there to test Typemock being loaded. It fails on the WhenCalled.
[TestClass, Isolated]
public class FirstTest
{
[TestMethod]
public void TestMe()
{
Exception e = new TypeMock.ArrangeActAssert.NestedCallException();
Isolate.WhenCalled(() => UnitOfWorkManager.GetCurrentSession(null)).WillReturn(null);
Assert.IsTrue(true);
}
}
The following is the error message.
HCSO.ESL.Test.Fakes.FirstTest.TestMe:
TypeMock.ArrangeActAssert.NestedCallException :
* WhenCalled does not support using a property call as an argument.
- To fix this pass false instead of AssemblyReader.IsDotNetFile
Example - this would work:
MyObj argument = Something.Other().GetStuff();
Isolate.WhenCalled(() => ObjUnderTest.MethodUnderTest(argument))...;
Example - this would not work:
Isolate.WhenCalled(() => ObjUnderTest.MethodUnderTest(Something.Other().GetStuff()))...;
(End error message)
Anyone have an idea why the first test always fails but runs fine on rerun? Something with how the assemblies are being loaded?
(Edit)Additional Details:
Versions:
Typemock Isolator: 6.0.10.0
Visual Studio: 10.0.30319.1
In addition, I added simplified test code. This way you can see the code being tested. And yes, this test fails first time, passes on every run after that.
[TestClass, Isolated]
public class FirstTest
{
public static int DummyCall(int i)
{
return 0;
}
[TestMethod]
public void TestMe()
{
Exception e = new TypeMock.ArrangeActAssert.NestedCallException();
//Isolate.WhenCalled(() => UnitOfWorkManager.GetCurrentSession(null)).WillReturn(null);
Isolate.WhenCalled(() => FirstTest.DummyCall(-1)).WillReturn(1);
Assert.IsTrue(true);
}
}
I work at Typemock,
It seems very strange, as this is definitely not a nested call from the looks of it.
Could you please try and email us a small solution demonstrating the problem to support#typemock.com?
What is UnitOfWorkManager? Is this a class belonging to EF, or is it your code?
Also, what version of Isolator are you using?
I resolved the issue. As I expected, it was partially a newbie mistake. Inside of NUnit there is a setting to determine how the assembly is isolated. The default option is to run the tests in the same process as NUnit. I tried changing the isolation in a seperate process per assembly and the problem goes away.
To reproduce error.
* Make sure NUnit option for "Run tests directly in NUnit process" is selected.
* Close NUnit (just to make sure setting is used)
* Launch NUnit from within VS.
* Select a test containing Isolate.WhenCalled()
* Run that test first.
Thanks for the help.
[EDIT: Update]
Updating this in the event someone else has this issue.
I found that in the NUnit client if I set the following options everything works great.
Under Settings:
Test Loader -> Assembly Isolation -> Default Process Model -> Run test directly in the NUnit process.
Test Loader -> Assembly Isolation -> Default Domain Usage -> Use a seperate AppDomain per Assembly
In the past I have tested for expected exceptions like this:
[TestMethod]
public void TestThrowsException() {
try {
Foo();
Assert.Fail();
} catch (MyException ex){//good
}
}
However I notice that there is a (cleaner?) way to test this using the ExpectedException attribute. Why does this test method pass when the exception is not thrown? Surely this defeats the purpose of the attribute.
[TestMethod]
[ExpectedException(typeof(MyException))]
public void TestThrowsException() {
}
[Edit] I am running this test using Silverlight 2
I've never seen that pass - is that really all you've got? Are you absolutely sure you have marked it as a TestMethod? Does the test runner show it passing? Have you definitely got the most recent code?
I'll double check, but I'm sure that will fail...
Jon Skeet was in fact right, I did have an old version of the testing framework. I updated to the Dec 08 release here http://code.msdn.microsoft.com/silverlightut/Release/ProjectReleases.aspx?ReleaseId=1913 and got the expected behaviour when tagging with ExpectedException.
I've actually experienced that ReSharper 4.5 testrunner does not work with ExpectedException in NUnit 2.5. ...but this looks like MSTest ... can you elaborate on which test framework you are using and which test runner you are using to execute the tests?
You should use 'ExpectedException' with an additional assert.fail, so that the test fails if the exception is not thrown (still in VS 2010 with .net 4 and Microsoft.VisualStudio.QualityTools.UnitTestFramework):
[TestMethod]
[ExpectedException(typeof(MyException))]
public void TestThrowsException() {
Do.SomethingThatThrowsAnException();
assert.fail("No exception ;-(");
}