I have read in several book and articles about TDD and BDD that one should avoid multiple assertions or expectations in a single unit test or specification. And I can understand the reasons for doing so. Still I am not sure what would be a good way to verify a complex result.
Assuming a method under test returns a complex object as a result (e.g. deserialization or database read) how do I verify the result correctly?
1.Asserting on each property:
Assert.AreEqual(result.Property1, 1);
Assert.AreEqual(result.Property2, "2");
Assert.AreEqual(result.Property3, null);
Assert.AreEqual(result.Property4, 4.0);
2.Relying on a correctly implemented .Equals():
Assert.AreEqual(result, expectedResult);
The disadvantage of 1. is that if the first assert fails all the following asserts are not run, which might have contained valuable information to find the problem. Maintainability might also be a problem as Properties come and go.
The disatvantage of 2. is that I seem to be testing more than one thing with this test. I might get false positives or negatives if .Equals() is not implemented correctly. Also with 2. I do not see, what properties are actually different if the test fails but I assume that can often be addressed with a decent .ToString() override. In any case I think I should avoid to be forced to throw the debugger at the failing tests to see the difference. I should see it right away.
The next problem with 2. is that it compares the whole object even though for some tests only some properties might be significant.
What would be a decent way or best practise for this in TDD and BDD.
Don't take TDD advice literally. What the "good guys" mean is that you should test one thing per test (to avoid a test failing for multiple reasons and subsequently having to debug the test to find the cause).
Now test "one thing" means "one behavior" ; NOT one assert per test IMHO.
It's a guideline not a rule.
So options:
For comparing whole data value objects
If the object already exposes a usable production Equals, use it.
Else do not add a Equals just for testing (See also Equality Pollution). Use a helper/extension method (or you could find one in an assertion library) obj1.HasSamePropertiesAs(obj2)
For comparing unstructured parts of objects (set of arbitrary properties - which should be rare),
create a well named private method AssertCustomerDetailsInOrderEquals(params) so that the test is clear in what part you're actually testing. Move the set of assertions into the private method.
With the context present in the question I'd go for option 1.
It likely depends on context. If I'm using some sort of built in object serialization within the .NET framework, I can be reasonably assured that if no errors were encountered then the entire object was appropriately marshaled. In that case, asserting a single field in the object is probably fine. I trust MS libraries to do the right thing.
If you are using SQL and manually mapping results to domain objects I feel that option 1 makes it quicker to diagnose when something breaks than option 2. Option 2 likely relies on toString methods in order to render the assertion failure:
Expected <1 2 null 4.0> but was <1 2 null null>
Now I am stuck trying to figure out what field 4.0/null was. Of course I could put the field name into the method:
Expected <Property1: 1, Property2: 2, Property3: null, Property4: 4.0>
but was <Property1: 1, Property2: 2, Property3: null, Property4: null>
This is fine for small numbers of properties, but begins to break down larger numbers of properties due to wrapping, etc. Also, the toString maintenance could become an issue as it needs to change at the same rate as the equals method.
Of course there is no correct answer, at the end of the day, it really boils down to your team's (or your own) personal preference.
Hope that helps!
Brandon
I would use the second approach by default. You're right, this fails if Equals() is not implemented correctly, but if you've implemented a custom Equals(), you should have unit-tested it too.
The second approach is in fact more abstract and consise and allows you to modify the code easier later, allowing in the same way to reduce code duplication. Let's say you choose the first approach:
You'll have to compare all properties in several places,
If you add a new property to a class, you'll have to add a new assertion in unit tests; modifying your Equals() would be much easier. Of course you have still to add a value of the property in expected result (if it's not the default value), but it would be shorter to do than adding a new assertion.
Also, it is much easier to see what properties are actually different with the second approach. You just run your tests in debug mode, and compare the properties on break.
By the way, you should never use ToString() for this. I suppose you wanted to say [DebuggerDisplay] attribute?
The next problem with 2. is that it compares the whole object even though for some tests only some properties might be significant.
If you have to compare only some properties, than:
ether you refactor your code by implementing a base class which contains only those properties. Example: if you want to compare a Cat to another Cat but only considering the properties common to Dog and other animals, implement Cat : Animal and compare the base class.
or you do what you've done in your first approach. Example: if you do care only about the quantity of milk the expected and the actual cats have drunk and their respective names, you'll have two assertions in your unit test.
Try "one aspect of behavior per test" rather than "one assertion per test". If you need more than one assertion to illustrate the behavior you're interested in, do that.
For instance, your example might be ShouldHaveSensibleDefaults. Splitting that up into ShouldHaveADefaultNameAsEmptyString, ShouldHaveNullAddress, ShouldHaveAQuantityOfZero etc. won't read as clearly. Nor will it help to hide the sensible defaults in another object then do a comparison.
However, I would separate examples where the values had defaults with any properties derived from some logic somewhere, for instance, ShouldCalculateTheTotalQuantity. Moving small examples like this into their own method makes it more readable.
You might also find that different properties on your object are changed by different contexts. Calling out each of these contexts and looking at those properties separately helps me to see how the context relates to the outcome.
Dave Astels, who came up with the "one assertion per test", now uses the phrase "one aspect of behavior" too, though he still finds it useful to separate that behavior. I tend to err on the side of readability and maintainability, so if it makes pragmatic sense to have more than one assertion, I'll do that.
Related
I have an unit test, where I want to test, that under some circumstances my class hasn't been changed.
[Test]
public void ShouldNotRecalculate()
{
var subject = new Invoice();
//Create50InvoiceItems()
sut.Recalculate(subject);
Assert.That(subject.Items[0].NetAmount.Equals());
Assert.That(subject.Items[0].VatRate.Equals());
...
Assert.That(subject.Items[49].NetAmount.Equals());
Assert.That(subject.Items[49].VatRate.Equals());
}
Without creating variables and saving there old values of those all 50 invoice items, is there any other solution? Should I create a shallow copy of this object? Maybe using AutoMapper?
Thank in advance!
I sometimes implemented the IEquatable interface so that I can compare objects and all it's properties.
It's quite handy in general, and also allows you to create a "reference object" or an object and compare and check if it has been changed.
Depending on the situation you may also just create multiple assertions and check all values.
For those cases I like to use Assert.Multiple() to signify that the values belong together.
In your case I would go for the IEquatable interface, this makes the Assertions super concise.
On top IClonable will make the preparation of the test data easy.
I wonder though - how many objects do you have to test to "proof" your code is working correctly?
Sure, you may have to make a test for each property, but per property, I guess only a few tests should be necessary.
Besides - you can use [TestCaseSource] and potentially [Combinatorical] attributes to generate the test cases necessary.
Then you don't need to create a huge list with test case data yourself.
Meaning - instead of creating a huge list of test case data yourself and test them all in a huge test, let NUnit generate the data for you and run it in many test cases.
This ought to be possible - and I think it might be possible somehow using Constraints, but they seem to only take one variable as input, which fails.
Here's the simplest example I've run into:
I have an object with size (i.e. a 2D variable)
I need to show it's partly off-screen
This happens if x is outside a range OR y is outside a range
... I can't see how to achieve that in NUnit (without throwing away critical info)
It seems that NUnit fundamentally doesn't support anything other than 1-dimensional problems, which would be absurd, so I must be missing something here. But the only methods I can find all only work with 1-d inputs.
Things I've thought of ... but don't work:
i. Assert.True( A | B ) - useless: it throws away all the "expected" info and generates such weak messages that there's essentially no point using a testing framework here
ii. Assert.Or - doesn't allow multiple variables, so you can test "X is 3, or 4, or 5", but you cannot test "X is 3, or Y is 4"
iii. Writing custom assertions for a Tuple<A,B> - but then it goes horribly wrong when you get to "greater than" ... is "<A,B> greater than <C,D>" is a non-trivial question and ends up with code that's vastly confusing and I can guarantee someone will misunerstand and misapply it in future (probably me :))
1-dimensional problems
To be frank I don't really get it. If you think of it, actually all members in Assert are tools for boolean problems (zero dimension). Which perfectly makes sense for raising a red or green flag in case of all assertions.
Your problem is also a yes/no question: "Is my object off-screen?".
i. Assert.True( A | B ) - useless: it throws away all the "expected" info and generates such weak messages
Nothing stops you to specify the message, which is displayed if the assertion fails:
Assert.IsTrue(myObject.IsOnScreen(), $"Object is off-screen: {myObject.Bounds}")
But in this particular case you can easily turn it into an equality assertion:
// here also the default message may be clear enough
Assert.AreEqual(expected: screenBounds, actual: screenBounds.UnionWith(myObject.Bounds));
And voila, multiple (four) properties were compared at once...
To be honest, you haven't done a great job of explaining what you want. I re-read several times and I think I have guessed right, but I shouldn't have to so please try code for the next problem.
I take it that you wish it were possible to change the actual value in the course of an NUnit constraint expression... something like this...
// NOT WORKING CODE
Assert.That(X, Is.OnScreen.And.That(Y, Is.OnScreen));
(Where "OnScreen" makes the test you need)
But you can't do that. To work within the existing features of NUnit, each Assertion needs to deal with one actual value.
Simplest thing I can think of is to use multiple assertions:
Assert.Multiple(() =>
{
Assert.That(X.IsOnScreen);
Assert.That(Y.IsOnScreen);
});
I say this is simple because NUnit already does it. It's not very expressive, however.
If you are doing a lot of such tests, the simplest thing is to create a custom constraint, which deals with your objects as an entity. Then you would be able to write something like
Assert.That(myObject, Is.OnScreen);
Not just as pseudo-code, but as actual code. Basically, you would be creating a set of tests on rectangles, representing the object bounds on the one hand and the screen dimensions on the other.
Is there anything wrong with checking so many things in this unit test?:
ActualModel = ActualResult.AssertViewRendered() // check 1
.ForView("Index") // check 2
.WithViewData<List<Page>>(); // check 3
CollectionAssert.AreEqual(Expected, ActualModel); // check 4
The primary goals of this test are to verify the right view is returned (check 2) and it contains the right data (check 4).
Would I gain anything by splitting this into multiple tests? I'm all about doing things right, but I'm not going to split things up if it doesn't have practical value.
I'm pretty new to unit testing, so be gentle.
As others have pointed out, it's best to stick with one assert in each test in order to avoid losing information - if the first assert fails, you don't know whether the later ones would fail also. You have to fix the problem with less information - which might (probably will) be harder.
A very good reference is Roy Osherove's Art of Unit Testing - if you want to get started right with Unit Testing, you could do a lot worse than starting here.
Noting for future readers that this question and its duplicate Is it bad practice to have more than one assertion in a unit test? have opposite majority opinions, so read them both and decide for yourself.
My experience is that the most useful quantum of testing is not the assertion, but the scenario -- that is, there should be one unit test for a given set of initial conditions and method call, with as many assertions as are necessary to assert the expected final conditions. Having one unit test per assertion leads to duplicate setup or tortuous workarounds to avoid the duplication (such as the awful deeply nested rspec contexts that I'm seeing more and more of lately). It also multiplies tests, drastically slowing your suite.
As long as each assertion has a unique and identifying failure message you should be good and will sidestep any Assertion Roulette issues because it won't be hard to tell which test failed. Use common sense.
I have found a different argument to this question (at least for myself):
Use of multiple asserts is OK if they are testing the same thing.
For example, it's OK to do:
Assert.IsNotNull(value);
Assert.AreEqual(0, value.Count);
Why? - because these two asserts are not hiding the intention of the test. If the first assert fails, it means the second would fail too. And indeed, if we remove the first assert, the second assert fails (with null reference exception - !!!) anyway when the value is null. If this was not the case, then we should not put these two asserts together.
So, this is wrong:
Assert.IsNotNull(value1);
Assert.IsNotNull(value2);
As I've mentioned above, if the first assert fails, there is no clear indication about the second assert - we would still want to know what happens to the second one (even if the first one failed). So, for this reason, these two asserts belong to two different unit tests.
Conclusion: putting one or more asserts, when done properly, becomes the matter of the preference - whether we want to see assertion exceptions in the test results, or do we want to see some other exceptions as well in particular cases.
It's best to stick with only one assert in each test to avoid Assertion Roulette.
If you need to set up the same scenario to test multiple conditions based on identical premises, it's better to extract the setup code to a shared helper method, and then write several tests that call this helper method.
This ensures that each test case tests only one thing.
As always, there are exceptions to this rule, but since you are new to unit testing, I would recommend that you stick with the one assertion per unit test rule until you have learned when it's okay to deviate.
I know that this is an old question but I thought I'd add my bit.
Generally I'd have as few assertions as possible in each test case. Tests can be often be written to save having lots of different assertions.
Suppose I have unit tests for a method which creates a salutation (eg Mr Smith) from the components of a name. I want to check this with a large number of different scenarios, it is unnecessary to have a separate test for each.
Given the following code, there are many different assertions. When they fail you can fix them one at a time until the assertions stop.
Assert.AreEqual("Mr Smith", GetSalutation("Mr", "J", "Smith"));
Assert.AreEqual("Mr Smith", GetSalutation("Mr", "John", "Smith"));
Assert.AreEqual("Sir/Madam", GetSalutation("", "John", "Smith"));
Assert.AreEqual("Sir/Madam", GetSalutation("", "J", "Smith"));
An alternative to this is to keep a count of issues and assert this at the end.
int errorCount = 0;
string result;
result = GetSalutation("Mr", "J", "Smith");
if (result == "Mr Smith")
errorCount++;
result = GetSalutation("Mr", "John", "Smith");
if (result == "Mr Smith")
errorCount++;
Assert.AreEqual(0, errorCount);
In a real world situation I'd probably add some Trace commands to write the detail of the individual tests which failed to the output window
So I see that Assert has dozens of methods that seem to do essentially the same thing.
Assert.IsFalse( a == b );
Assert.IsTrue( a != b );
Assert.AreNotEqual( a, b );
Why? Is it just to be more explicit? When should the various methods be used? Is there an offical best practices document?
The difference between IsFalse and IsTrue is readability. AreNotEqual allows a better error message to be presented when the test fails. IsTrue for example will just tell you that the answer was supposed to be true and was really false. AreNotEqual will show the two values that were compared in its error message.
Short answer: For readability.
Slightly longer answer:
Your tests are also code, and in terms of intent are as important as the code you are testing. As such, you want to make the intent of the test as clear as possible. Sometimes that means you use IsFalse, sometimes it means using IsTrue.
Those three methods have three different, specific goals. The goal of testing is to provide clear verification and validation of your code. By using the most clear and specific method possible, you're making your test the smallest test possible, with the most specific, clear meaning possible.
This helps because it adds clarity - you can see, specifically, what a test is supposed to do in a more declarative nature, where using the same method for multiple testing scenarios, which each have a different meaning, requires more understanding of the code itself, instead of the nature of the test.
In this case, the third is the (only) appropriate one to use. However, if you had this situation, you'd use a different one, for example:
Assert.IsTrue( myClass.MethodThatReturnsTrue() );
You should use the method that provides the most clarity as to your goal - if you're checking two values for equality, use Assert.IsEqual, if you're checking a boolean to verify that it's false, use Assert.IsFalse. This makes the error reports meaningful and understandable.
It makes the errors easiest to read; use the one that matches your code most closely.
In this case, use #3.
Because you can overload the == and != , thats why.
It's called a "Fluent Interface"... and makes things more readable in the opinion of many.
I need to derive an important value given 7 potential inputs. Uncle Bob urges me to avoid functions with that many parameters, so I've extracted the class. All parameters now being properties, I'm left with a calculation method with no arguments.
“That”, I think, “could be a property, but I'm not sure if that's idiomatic C#.”
Should I expose the final result as a property, or as a method with no arguments? Would the average C# programmer find properties confusing or offensive? What about the Alt.Net crowd?
decimal consumption = calculator.GetConsumption(); // obviously derived
decimal consumption = calculator.Consumption; // not so obvious
If the latter: should I declare interim results as [private] properties, also? Thanks to heavy method extraction, I have several interim results. Many of these shouldn't be part of the public API. Some of them could be interesting, though, and my expressions would look cleaner if I could access them as properties:
decimal interim2 = this.ImportantInterimValue * otherval;
Happy Experiment Dept.:
While debugging my code in VS2008, I noticed that I kept hovering my mouse over the method calls that compute interim results, expecting a hover-over with their return value. After turning all methods into properties, I found that exposing interim results as properties greatly assisted debugging. I'm well pleased with that, but have lingering concerns about readability.
The interim value declarations look messier. The expressions, however, are easier to read without the brackets. I no longer feel compelled to start the method name with a verb. To contrast:
// Clean method declaration; compulsive verby name; callers need
// parenthesis despite lack of any arguments.
decimal DetermineImportantInterimValue() {
return this.DetermineOtherInterimValue() * this.SomeProperty;
}
// Messier property declaration; clean name; clean access syntax
decimal ImportantInterimValue {
get {
return this.OtherInterimValue * this.SomeProperty;
}
}
I should perhaps explain that I've been coding in Python for a decade. I've been left with a tendency to spend extra time making my code easier to call than to write. I'm not sure the Python community would regard this property-oriented style as acceptably “Pythonic”, however:
def determineImportantInterimValue(self):
"The usual way of doing it."
return self.determineOtherInterimValue() * self.someAttribute
importantInterimValue = property(
lambda self => self.otherInterimValue * self.someAttribute,
doc = "I'm not sure if this is Pythonic...")
The important question here seems to be this:
Which one produces more legible, maintainable code for you in the long run?
In my personal opinion, isolating the individual calculations as properties has a couple of distinct advantages over a single monolothic method call:
You can see the calculations as they're performed in the debugger, regardless of the class method you're in. This is a boon to productivity while you're debugging the class.
If the calculations are discrete, the properties will execute very quickly, which means (in my opinion), they observe the rules for property design. It's absurd to think that a guideline for design should be treated as a straightjacket. Remember: There is no silver bullet.
If the calculations are marked private or internal, they do not add unnecessary complexity to consumers of the class.
If all of the properties are discrete enough, compiler inlining may resolve the performance issues for you.
Finally, if the final method that returns your final calculation is far and away easier to maintain and understand because you can read it, that is an utterly compelling argument in and of itself.
One of the best things you can do is think for yourself and dare to challenge the preconceived One Size Fits All notions of our peers and predecessors. There are exceptions to every rule. This case may very well be one of them.
Postscript:
I do not believe that we should abandon standard property design in the vast majority of cases. But there are cases where deviating from The Standard(TM) is called for, because it makes sense to do so.
Personally, I would prefer if you make your public API as a method instead of property. Properties are supposed to be as 'fast' as possible in C#. More details on this discussion: Properties vs Methods
Internally, GetConsumption can use any number of private properties to arrive at the result, choice is yours.
I usually go by what the method or property will do. If it is something that is going to take a little time, I'll use a method. If it's very quick or has a very small number of operations going on behind the scenes, I'll make it a property.
I use to use methods to denote any action on the object or which changes the state of an object. so, in this case I would name the function as CalculateConsumption() which computes the values from other properties.
You say you are deriving a value from seven inputs, you have implemented seven properties, one for each input, and you have a property getter for the result. Some things you might want to consider are:
What happens if the caller fails to set one or more of the seven "input" properties? Does the result still make sense? Will an exception be thrown (e.g. divide by zero)?
In some cases the API may be less discoverable. If I must call a method that takes seven parameters, I know that I must supply all seven parameters to get the result. And if some of the parameters are optional, different overloads of the method make it clear which ones.
In contrast, it may not be so clear that I have to set seven properties before accessing the "result" property, and could be easy to forget one.
When you have a method with several parameters, you can more easily have richer validation. For example, you could throw an ArgumentException if "parameter A and parameter B are both null".
If you use properties for your inputs, each property will be set independently, so you can't perform the validation when the inputs are being set - only when the result property is being dereferenced, which may be less intuitive.