I'm writing unit tests for some legacy code and have come across a situation where I'm getting false positives for my unit tests. A simplified example is below.
If I forget to set up an implementation for DoSomething, I would want the test below to fail with a MockException. However, the exception is actually caught by the try/catch in the business logic, and so the test appears to pass. VerifyAll likewise does not cause a failure as the method has been 'called'.
I found a request for change related to this issue, but there doesn't seem to be a response to it.
Any ideas on how I can ensure this test works?
public class SUT
{
public ISystem iSystem { get; set; } // complicated web service
public void Foo()
{
try
{
iSystem .DoSomething();
}
catch (Exception ex)
{
//Report error
}
}
}
[TestClass]
public class Test
{
[TestMethod]
public void TestDoSomething()
{
var mr = new MockRepository(MockBehavior.Strict);
var iSystem = mr.Create<ISystem>();
var foo = new Foo { Interface = interface.Object };
foo.DoSomething();
mr.VerifyAll();
}
}
Your unit tests should isolate the tested class, SUT is behaving correctly by logging (and swallowing) the error - if you wanted to test the throw, that belongs in an iSystem test. You could have the test still fail by detecting the unexpected error logging call.
I believe the problem here is not really related to moq, or to testing, but more of a design / exception handling practice.
I would imagine that the constructor of Foo is not the proper place to handle all unexpected errors.
You probably want to have unexpected exceptions such as OutOfMemoryException, StackOverflowException and other unexpected exception to bubble up to a centralized place where all fatal errors will be handled.
Typically, that would be:
For WPF : the AppDomain.UnhandledException event (https://msdn.microsoft.com/en-us/library/system.appdomain.unhandledexception.aspx)
For CommandLine applications: UnhandledExceptionEventHandler (https://msdn.microsoft.com/en-us/library/system.appdomain.unhandledexception.aspx)
For ASP.Net, the Application_Error method in global.asax (https://msdn.microsoft.com/en-us/library/24395wz3(v=vs.140).aspx)
Bottom line is. In order to solve your case, first, write a brand new test where you set-up your DoSomething() method to throw an exception
Use
mr.Setup(o => o.DoSomething().Throws<MyUnexpectedException>();
Then, assert that instantiating Foo does not swallow MyUnexpectedException and is actually thrown.
By doing so, you will be forced to change your catch statement and use more specific types (it is a very bad practice to use Exception in a catch block)
Then, by changing your architecture, MockException, which inherits from Exception will not be handled by your catch block.
Related
I'm writing API tests and am using the mstest framework. My setup is pretty simple, my cleanup and initialize functions look like below -- stand up and shut down logging instaces. My problem is that some tests throw exceptions that are not caught (this is usually due to test code that is still in development). This causes the test to fail, which I would expect, but I can't manage to get the actual exception object and log it which makes debugging a little difficult. The obvious answer is "wrap your test in a try catch block that handles the logging and gracefully fails the test", but I don't really like that because it just adds another responsibility that future test authors (probably me) poorly implementing the catch, log, and rethrow logic.
TL;DR : How can I, using TestContext (or another solution) get access to an exception that was thrown but not caught by the actual testing logic in the test cleanup method?
public TestContext TestContext { get; set; }
[TestInitialize]
public void TestInitialize()
{
Log.RegisterLogger(new TestLog() {
TestCaseName = TestContext.TestName,
TestCastStart = DateTime.UtcNow
});
}
[TestCleanup]
public void TestCleanup()
{
/*I want to be able to get the details of an uncaught exception here*/
Log.FinalizeLogging();
}
I am using the testing framework that comes with Visual Studio, along with NSubstitute to unit test a method that takes a system ID, and throws an exception if the system can't be found in the database...
public VRTSystem GetSystem(int systemID)
{
VRTSystem system = VrtSystemsRepository.GetVRTSystemByID(systemID);
if (system == null)
{
throw new Exception("System not found");
}
return system;
}
(In case this seems odd, there is a specific business case for this method that requires it to throw an exception, as returning a null system is not acceptable for its usage)
I want to write a test to check that an exception is thrown if the system doesn't exist. I currently have the following...
[TestMethod]
public void LicensingApplicationServiceBusinessLogic_GetSystem_SystemDoesntExist()
{
var bll = new LicensingApplicationServiceBusinessLogic();
try
{
VRTSystem systemReturned = bll.GetSystem(613);
Assert.Fail("Should have thrown an exception, but didn't.);
}
catch () { }
}
By not mocking the repository, the system returned by VrtSystemsRepository.GetVRTSystemByID() will be null, and the exception thrown. Whilst this works, it looks wrong to me. I wouldn't have expected to need a try/catch block in a test.
The NSubstitute docs have an example that implies I should be able to test this as follows...
[TestMethod]
public void GetSystem_SystemDoesntExist()
{
var bll = new LicensingApplicationServiceBusinessLogic();
Assert.Throws<Exception>(() => bll.GetSystem(613));
}
However, if I try this in my test code, I get Throws highlighted in red, with the error message "Assert does not contain definition for Throws"
Now, I'm not actually sure that the sample on that page covers my scenario, as the test code specifies that the method under test throws an exception, which I don't really understand, as I thought the idea of testing was to leave the method under test alone, and test what happens under various scenarios. However, even without that, I don't understand why the Assert.Throws method doesn't exist.
Anyone any ideas?
Edit: DavidG pointed out that Assert.Throws is probably part of NUnit, not the MS framework, which would explain why it's not recognised. If so, is the way I'm currently testing the right way to do it?
As mentioned by DavidG The referenced documentation is using NUnit for assertions.
If not using that framework you can use the ExpectedExceptionAttribute Class
[TestMethod]
[ExpectedException(typeof(<<Your expected exception here>>))]
public void GetSystem_SystemDoesntExist() {
var bll = new LicensingApplicationServiceBusinessLogic();
bll.GetSystem(613);
}
which would fail if the expected exception is not thrown.
I'm working on a project at the moment where I need to inter-operate with code that swallows exceptions. In particular, I'm writing NUnit unit tests. There are some places where I want to embed assertions within code that gets passed as a delegate, as part of mocking a particular behavior. The problem I'm having is that the AssertionException gets swallowed by the code calling the delegate, which means the test passes, even though the test Assert failed.
Is there any way to inform NUnit that a test should fail that can't be circumvented by catching AssertionException? I can't modify the code that swallows the exceptions, as I don't have full ownership and it's already in semi-production use. I'm hoping there's a clean way to accomplish this.
The best I've come up with is something like this:
private static string _assertionFailure;
public static void AssertWrapper(Action action)
{
try
{
action();
}
catch (AssertionException ex)
{
_assertionFailure = ex.Message;
throw;
}
}
[Test]
[ExpectedException(typeof(AssertionException))]
public void TestDefeatSwallowing()
{
Action failure = () => AssertWrapper(() => Assert.Fail("This is a failure"));
EvilSwallowingMethod(failure);
if (_assertionFailure != null)
Assert.Fail(_assertionFailure);
}
private void EvilSwallowingMethod(Action action)
{
try
{
action();
}
catch
{
}
}
It works, but it's pretty ugly. I have to wrap every Assert call and I have to check at the end of every test if an assertion was swallowed.
So you're doing something like this? (this is using Moq syntax)
var dependency1 = new Mock<IDependency1>();
dependency1.Setup(d => d.CalledMethod([Args])
.Callback(TestOutArgsAndPossiblyThrow);
var objectUnderTest = new TestedObject(dependency1.Object);
objectUnderTest.MethodThatCallsIDependency1dotCalledMethod();
And you've got TestOutArgsAndPossiblyThrow encapsulated in your AssertWrapper class?
Unless that's way off, I'd say you're doing it just about right. You have execution re-entering your test at a point where you can record the state of the call to the dependency. Whether that's done via catching exceptions and analyzing them or just directly inspecting the values of the method parameters, you've just gotta do the work. And if you're swallowing exceptions inside the black box, you're going to have to monitor them before they get back into the black box.
I still say you'd be much better off with appropriate logging and notification (you don't have to notify the end users, necessarily). To #TrueWill's point - what do you do when there's an IOException or the database isn't available?
DISCUSSION EDIT
Is your scenario structured like this?
TEST -> TESTED CODE -> SWALLOWING CODE -> THROWING MOCK
I've got a method that does some IO that generally looks like this:
public bool Foo()
{
try
{
// bar
return true;
}
catch (FileNotFoundException)
{
// recover and complete
}
catch (OtherRecoverableException)
{
// recover and complete
}
catch (NonRecoverableException ex)
{
ExceptionPolicy.HandleException(ex, "LogException");
return false;
}
}
The method isn't mission critical to be completed, there are external recovery steps - and it's relatively common for NonRecoverableException to be thrown - it's in the spec for it to return false, report 'cannot be completed at this time' and processing moves along. A NonRecoverableException does not put the program in an invalid state.
When I'm unit testing, and one of these exceptions is thrown, I get the error that
Activation error occured while trying to get instance of type ExceptionPolicyImpl
And I'd like to suppress that in favor of getting the actual/original exception information instead of EntLib not being able to log (and, indeed to force the NonRecoverableException and have an [ExpectedException(typeof(NonRecoverableException))] unit test to ensure that this method complies with the spec.
How might I go about that?
edit
Using preprocessor directives is not ideal as I hate seeing test-specific code in the codebase.
Testability of your code using the Entlib static facades is difficult. Without changing your code a little, your only answer is to add an app.config file to your test assembly and set up the Entlib exception block with an innocuous policy that does nothing.
However, in Entlib 4 (and 5, which I see you're using) there's another way. We added an instance entry point to all the blocks specifically to improve the testability story. For the exception block, that instance is the ExceptionManager. Using it is pretty simple. Get an exception manager instance into your type, and then call it instead of ExceptionPolicy. Something like this:
public class Whatever {
private ExceptionManager exm;
public Whatever(ExceptionManager exm) { this.exm = exm; }
public bool Foo() {
try {
... do whatever ...
}
catch(NonRecoverableException ex) {
exm.HandleException(ex, "LogException");
return false;
}
}
}
Now that you've got that in there, you can mock out the ExceptionManager (it's an abstract base class) to essentially no-op it during test, either manually or using a mock object framework.
If you don't want to force your users to use a DI container, you can add a default constructor that gets the current exception manager:
public class Whatever {
private ExceptionManager exm;
public Whatever() : this(EnterpriseLibraryContainer.Current.GetInstance<ExceptionManager>()) { }
public Whatever(ExceptionManager exm) { this.exm = exm; }
}
End users use the default constructor, your tests use the one that takes in an explicit ExceptionManager, and you have your hook to mock out anything Entlib uses.
All the blocks now have these "Manager" classes (where they make sense, anyway).
Hmm, you could refactor the code to place everything in the try block in a separate method and configure you tests to call that instead of the existing method?
I have a test method that is run. When the method generates an exception I want to know what the name of the test was and the exception content.
In the teardown for the test I want to get access to this information. How would I get access to it from the [TearDown] attributed method?
You can access text context objects in test tear down method
[TearDown]
public void TestTearDown()
{
// inc. class name
var fullNameOfTheMethod = NUnit.Framework.TestContext.CurrentContext.Test.FullName;
// method name only
var methodName = NUnit.Framework.TestContext.CurrentContext.Test.Name;
// the state of the test execution
var state = NUnit.Framework.TestContext.CurrentContext.Result.State; // TestState enum
}
I don't know which version was first to support it, but mine is 24.
I don't think there's a good way built in to nunit, but it's not a hard problem to resolve. Just wrap your tests in a try/catch block, catch any exceptions, and save them (and the test name) to a private member variable in your test class. Then you've got access from your TearDown method.
Not particularly elegant, but it works.
Another solution would be to use a template method and run all tests using this method. For example:
// template method
void Execute(Action test)
{
try
{
test();
}
catch (Exception e)
{
// handle exception here
throw;
}
}
[Test]
public void Test()
{
Execute(() =>
{
// your test here
});
}
This pattern is particularly useful when your test uses some resources that must be initialized before test and disposed after test (e.g. temporary file). In that case, you can use a type parameter in test delegate.
Another advantage is that you can easily let the test run on different thread, using different culture etc.
Disadvantage is clear: it forces you to use lambda method in every test.
OPTION 1: I don't think you can. Or rather, I don't know that you can. How I approach this need is to use a try/catch on the specific tests, do what I want with the exception and then throw again within the catch block so that the test could fail.
try{
// do something that can potentially throw;
}
catch(Exception ex){
// do something interesting with the ex;
throw;
}
OPTION 2: If you've not gone too far along, you may want to use xUnit which has a different exception expectation model and may provide some of the control you are looking for.