I'm working on a project at the moment where I need to inter-operate with code that swallows exceptions. In particular, I'm writing NUnit unit tests. There are some places where I want to embed assertions within code that gets passed as a delegate, as part of mocking a particular behavior. The problem I'm having is that the AssertionException gets swallowed by the code calling the delegate, which means the test passes, even though the test Assert failed.
Is there any way to inform NUnit that a test should fail that can't be circumvented by catching AssertionException? I can't modify the code that swallows the exceptions, as I don't have full ownership and it's already in semi-production use. I'm hoping there's a clean way to accomplish this.
The best I've come up with is something like this:
private static string _assertionFailure;
public static void AssertWrapper(Action action)
{
try
{
action();
}
catch (AssertionException ex)
{
_assertionFailure = ex.Message;
throw;
}
}
[Test]
[ExpectedException(typeof(AssertionException))]
public void TestDefeatSwallowing()
{
Action failure = () => AssertWrapper(() => Assert.Fail("This is a failure"));
EvilSwallowingMethod(failure);
if (_assertionFailure != null)
Assert.Fail(_assertionFailure);
}
private void EvilSwallowingMethod(Action action)
{
try
{
action();
}
catch
{
}
}
It works, but it's pretty ugly. I have to wrap every Assert call and I have to check at the end of every test if an assertion was swallowed.
So you're doing something like this? (this is using Moq syntax)
var dependency1 = new Mock<IDependency1>();
dependency1.Setup(d => d.CalledMethod([Args])
.Callback(TestOutArgsAndPossiblyThrow);
var objectUnderTest = new TestedObject(dependency1.Object);
objectUnderTest.MethodThatCallsIDependency1dotCalledMethod();
And you've got TestOutArgsAndPossiblyThrow encapsulated in your AssertWrapper class?
Unless that's way off, I'd say you're doing it just about right. You have execution re-entering your test at a point where you can record the state of the call to the dependency. Whether that's done via catching exceptions and analyzing them or just directly inspecting the values of the method parameters, you've just gotta do the work. And if you're swallowing exceptions inside the black box, you're going to have to monitor them before they get back into the black box.
I still say you'd be much better off with appropriate logging and notification (you don't have to notify the end users, necessarily). To #TrueWill's point - what do you do when there's an IOException or the database isn't available?
DISCUSSION EDIT
Is your scenario structured like this?
TEST -> TESTED CODE -> SWALLOWING CODE -> THROWING MOCK
Related
as title suggests, i have a selenium project, sometimes for various reasons i get an ivalid unique id on the driver.close() as a solution i enclosed this in the try block, if it fails i want to catch the exception execute driver.quit() and print the exception to the test log.
As the one time tear down executes regardless of exception or not, its failing the driver.close and moving straight to onetime tear down. Any one any ideas round this, i may not run every test every run. I cant guarantee that the extent.flush will be in the right place.
I've added the code below, i think i can handle it with better logic but i cant think of how!
try
{
TearDown();
}
catch (Exception e)
{
test.Info(e.StackTrace.ToString());
test.Info(e.ToString());
Console.WriteLine("hello" + e.ToString());
ForceTearDown();
}
[TearDown]
public void TearDown()
{
driver.Close();
}
public void ForceTearDown()
{
driver.Quit();
}
[OneTimeTearDown]
public void ExtentClose()
{
extent.Flush();
}
The entire point of NUnit's teardowns - both [TearDown] and [OneTimeTearDown] is that they execute whether the test passed or failed and whether an exception is thrown or not. You should not try to defeat that feature and you should not call the methods with those attributes yourself... let NUnit do the calling.
OTOH, with proper checking, both your TearDown and your test code may call driver.Close() or driver.Quit if you need to do it.
I'm writing unit tests for some legacy code and have come across a situation where I'm getting false positives for my unit tests. A simplified example is below.
If I forget to set up an implementation for DoSomething, I would want the test below to fail with a MockException. However, the exception is actually caught by the try/catch in the business logic, and so the test appears to pass. VerifyAll likewise does not cause a failure as the method has been 'called'.
I found a request for change related to this issue, but there doesn't seem to be a response to it.
Any ideas on how I can ensure this test works?
public class SUT
{
public ISystem iSystem { get; set; } // complicated web service
public void Foo()
{
try
{
iSystem .DoSomething();
}
catch (Exception ex)
{
//Report error
}
}
}
[TestClass]
public class Test
{
[TestMethod]
public void TestDoSomething()
{
var mr = new MockRepository(MockBehavior.Strict);
var iSystem = mr.Create<ISystem>();
var foo = new Foo { Interface = interface.Object };
foo.DoSomething();
mr.VerifyAll();
}
}
Your unit tests should isolate the tested class, SUT is behaving correctly by logging (and swallowing) the error - if you wanted to test the throw, that belongs in an iSystem test. You could have the test still fail by detecting the unexpected error logging call.
I believe the problem here is not really related to moq, or to testing, but more of a design / exception handling practice.
I would imagine that the constructor of Foo is not the proper place to handle all unexpected errors.
You probably want to have unexpected exceptions such as OutOfMemoryException, StackOverflowException and other unexpected exception to bubble up to a centralized place where all fatal errors will be handled.
Typically, that would be:
For WPF : the AppDomain.UnhandledException event (https://msdn.microsoft.com/en-us/library/system.appdomain.unhandledexception.aspx)
For CommandLine applications: UnhandledExceptionEventHandler (https://msdn.microsoft.com/en-us/library/system.appdomain.unhandledexception.aspx)
For ASP.Net, the Application_Error method in global.asax (https://msdn.microsoft.com/en-us/library/24395wz3(v=vs.140).aspx)
Bottom line is. In order to solve your case, first, write a brand new test where you set-up your DoSomething() method to throw an exception
Use
mr.Setup(o => o.DoSomething().Throws<MyUnexpectedException>();
Then, assert that instantiating Foo does not swallow MyUnexpectedException and is actually thrown.
By doing so, you will be forced to change your catch statement and use more specific types (it is a very bad practice to use Exception in a catch block)
Then, by changing your architecture, MockException, which inherits from Exception will not be handled by your catch block.
I have an application for smoke testing several key services. I want these tests to be easily written and self discovering. To this end, I have created an attribute for each method in the style of [TestMethod] from MSTest. I have also created a object that will find all these methods and execute them within a try-catch. If the method throws an exception I report as a failure, otherwise as a success.
This is all very unit test familiar and that is the intention. So a test would look like...
[MyTestAttribute]
public void TestingTimesAhead()
{
var d = DateTime.MaxValue.AddDays(1);
}
The magic happens in the test object, this has an Action property and a Run method...
public Action TestAction { get; private set; }
public override ITestResult RunTest()
{
try
{
this.TestAction.Invoke();
return new BaseTestResult() { Result = TestResultStatus.Success };
}
catch(Exception ex)
{
return new BaseTestResult() { Result = TestResultStatus.Failure, FailureException = ex};
}
}
When ran in isolation the above test, being wrapped as the Action, will cause an exception and the test fails. Perfect.
However, when I wrap the attributes and test running object up into a dll and reference from a new project, VS Debugger breaks for the exception.
I am presented with an option in that dialog to toggle 'break when this exception type is user-unhandled'.
I can suppress exceptions on a type by type basis, but that isn't going to work for reuse. Also, if I run the release build from the BIN output, the expected behaviour is there - it's just a really awkward development experience.
Is it possible to manage this exception setting from my code instead? Or if anyone can suggest a way around the problem, I'm open to ideas.
I need to call several methods from an external framework - or rather I am writing a wrapper around it for other users to call methods from this framework in a non-predetermined order. Now some methods of the framework will throw exceptions, even though no "real" error occured. Basically they are supposed to be internal exceptions just to notify whoever that the action to be performed has already been performed before. For example: that a file has been loaded. It wont hurt to load the file another time, so for all I care this "error" is no error at all. So I need to continue on this exception, but I also need to catch other, real exceptions, such as when the framework, which connects to clients and stuff, cannot do so.
Below I have some (extremely simplified) example code. Obviously that code wont compile because the code for the custom exceptions is missing. Also in real life the code is spread over three assemblies. This means, that I cannot wrap the exception handler around those framework methods which will throw InternalFrameworkException() only. I can only wrap it around the whole SomeMethod(). As I wrote, this is an extremely simplified example.
Is there any way to handle the RealException()s but continue the InternalFrameworkException()s without using PostSharp as mentioned here? Note that this is not about letting the InternalFrameworkException() fall through, but they should actually not break out of the try{} block at all.
namespace ExceptionTest
{
using System;
internal class Program
{
private static void Main(string[] args)
{
try
{
SomeMethod();
}
catch (InternalFrameworkException exception)
{
// Do not actually catch it - but also dont break the
// execution of "SomeMethod()".
// Actually I never want to end up here...
}
catch (RealException exception)
{
// Break the execution of SomeMethod() as usual.
throw;
}
catch (Exception exception)
{
// Again, break the execution of SomeMethod() as usual.
throw;
}
finally
{
// Clean up.
}
}
#region == Method is actually contained in another assembly referencing this assembly ===
private static void SomeMethod()
{
// Should break if uncommented.
// MethodThrowingProperException();
// Should not break.
MethodThrowingInternalExceptionOrRatherContinuableError();
// Should execute, even if previously an internal framework error happened.
MethodNotThrowingException();
}
#endregion
#region ===== Framework methods, they are contained in a foreign dll =====
private static void MethodThrowingProperException()
{
// Something happened which should break execution of the
// application using the framework
throw new RealException();
}
private static void MethodThrowingInternalExceptionOrRatherContinuableError()
{
// Perform some stuff which might lead to a resumable error,
// or rather an error which should not break the continuation
// of the application. I.e. initializing a value which is
// already initialized. The point is to tell the application using
// this framework that the value is already initialized, but
// as this wont influence the execution at all. So its rather
// a notification.
throw new InternalFrameworkException();
}
private static void MethodNotThrowingException()
{
// Well, just do some stuff.
}
#endregion
}
}
Edit: I did try the example in the post I already linked above, and it works like a charm ... when using it in SomeMethod() only. I could theoretically implement this as I am wrapping all the methods that are called in SomeMethod() before exposing them to the final assembly, but I dislike this approach, because it will give my code unnessessary complexity.
When an exception is thrown, the execution flow is broken. You can catch the exception or not but you cannot "continue" after the exception is thrown.
You can split your logic into parts and continue to the next part when one throws an exception, though.
I'm not sure of a way apart from an AOP approach in this case. Given that you are unable to change SomeMethod() or any of the methods it calls, you will need to look at adorning the called methods like MethodThrowingInternalExceptionOrRatherContinuableError() with an aspect that catches the 'continuable' exceptions.
The aspect would effectively wrap the method call in a try{...} catch(InternalFrameworkException) (or similar catchable exception) block.
As you have already noted, you are unable to drop back into a method once it has thrown an exception, even if the caller catches the exception in a catch() block, so you need to inject into the methods you are calling, which an AOP framework like PostSharp will allow you to do.
I have solved similar problem by wrapping the calls to InternalFrameworkMethod() in try-catch(InternalFrameworkException) blocks and calling it somethig like InternalFrameworkMethodSafe() and then in SomeMethod call the treated InternalFrameworkMethodSafe().
void InternalFrameworkMethodSafe()
{
try
{
InternalFrameworkMethod();
}
catch(InternalFrameworkException e)
{
Trace.Write("error in internal method" + e);
}
}
void SomeMethod()
{
...
InternalFrameworkMethodSafe();
...
}
It may not work in your case if the internal framework is in wrong state and not able to continue.
I have a test method that is run. When the method generates an exception I want to know what the name of the test was and the exception content.
In the teardown for the test I want to get access to this information. How would I get access to it from the [TearDown] attributed method?
You can access text context objects in test tear down method
[TearDown]
public void TestTearDown()
{
// inc. class name
var fullNameOfTheMethod = NUnit.Framework.TestContext.CurrentContext.Test.FullName;
// method name only
var methodName = NUnit.Framework.TestContext.CurrentContext.Test.Name;
// the state of the test execution
var state = NUnit.Framework.TestContext.CurrentContext.Result.State; // TestState enum
}
I don't know which version was first to support it, but mine is 24.
I don't think there's a good way built in to nunit, but it's not a hard problem to resolve. Just wrap your tests in a try/catch block, catch any exceptions, and save them (and the test name) to a private member variable in your test class. Then you've got access from your TearDown method.
Not particularly elegant, but it works.
Another solution would be to use a template method and run all tests using this method. For example:
// template method
void Execute(Action test)
{
try
{
test();
}
catch (Exception e)
{
// handle exception here
throw;
}
}
[Test]
public void Test()
{
Execute(() =>
{
// your test here
});
}
This pattern is particularly useful when your test uses some resources that must be initialized before test and disposed after test (e.g. temporary file). In that case, you can use a type parameter in test delegate.
Another advantage is that you can easily let the test run on different thread, using different culture etc.
Disadvantage is clear: it forces you to use lambda method in every test.
OPTION 1: I don't think you can. Or rather, I don't know that you can. How I approach this need is to use a try/catch on the specific tests, do what I want with the exception and then throw again within the catch block so that the test could fail.
try{
// do something that can potentially throw;
}
catch(Exception ex){
// do something interesting with the ex;
throw;
}
OPTION 2: If you've not gone too far along, you may want to use xUnit which has a different exception expectation model and may provide some of the control you are looking for.