Can we have a loop inside a unit test?
My method returns an IEnumerable<IEnumerable>, I would like to unit test this logic where the IEnumerable<IEnumerable> is created. Basically I wanna test if the count of elements in the IEnumerable are as expected.
I cannot figure out an alternate way to test the inner IEnumerable without having a looping statement. Please let me know if this is a good practice.
There is no technical reason you can't do it. You can have multiple Assert statements in a unit test. Having an Assert statement in a loop is simply a shorthand way of having multiple Assert statements in a test.
However, some people think there should only be a single Assert statement in a unit test.
I personally don't agree - I think a test should test a single thing - and in order to do that sometimes you may need more than one Assert statement.
If your method returns an IEnumerable of Product's, and each Product contains an IEnumerable of Color's, then I think the following test is is fine:
[Test]
public void All_products_should_have_some_colors()
{
var products = GetProducts();
foreach (var product in products)
{
Assert.IsNotEmpty(product.Colors);
}
}
However, you do need to be aware that if the IEnumerable contains 0 elements, the loop will never execute any of the Assert statements and your unit test will "pass" - whereas you probably would have intended it to fail.
To remedy that situation, you could have a separate test making sure that there are more than 0 elements in the IEnumerable (i.e. that GetProducts does actually return some Product's):
Assert.IsNotEmpty(products);
One reason to avoid writing a loop in a test would be to keep the test concise and readable at a glance. Since you have tagged the question with NUnit and you say you just want to test that the element count is as expected, consider making your Asserts using the NUnit Constraints.
For example,
IEnumerable<IEnumerable<char>> someStrings = new[] { "abc", "cat", "bit", "hat" };
Assert.That(someStrings, Has.All.With.Length.EqualTo(3).And.All.Contains("a"));
fails with the message:
Expected: all items property Length equal to 3 and all items String containing "a"
But was: < "abc", "cat", "bit", "hat" >
but passes if you change "bit" to "bat".
The book xUnit Test Patterns: Refactoring Test Code
By Gerard Meszaros
has many great answers to questions such as yours.
Yes you can have loops in unit test, but with caution. As mentioned by Alex York, loops are acceptable if you test one thing; i.e. one expectation.
If you use loops, then I recommend that you must do two things:
As mentioned above, test for a non-empty iteration set. Iterating over an empty set is a false positive. False positive results are bane of all automated testing because nobody double checks a green result.
Include a test description that describes the current iteration. At a minimum include the iteration index.
Here is an example from my testing of the Greater Than property of an object.
[Test]
public void TestCompare_XtoY_GreaterThan()
{
int numObjects = mOrderedList.Count;
for (int i = 1; i < numObjects; ++i)
{
for (int j = 0; j < i; ++j)
{
string testDescription = string.Format("{0} is greater than {1} which implies\n {2}\n is greater than\n {3}"
, i, j
, mOrderedList[i], mOrderedList[j]
);
Assert.IsTrue(0 < mOrderedList[i].CompareTo(mOrderedList[j]), testDescription);
Assert.IsTrue(0 < mOrderedList[i].Compare(mOrderedList[i], mOrderedList[j]), testDescription);
Assert.IsTrue(0 < mOrderedList[j].Compare(mOrderedList[i], mOrderedList[j]), testDescription);
Assert.Greater(mOrderedList[i], mOrderedList[j], testDescription);
}
}
}
I test that my ordered list is non-empty in the test setup using:
[SetUp]
public void GeneralTestSetup()
{
// verify the iterated sources are not empty
string testDescription = string.Format("The ordered list of objects must have at least 3 elements, but instead has only {0}.", mOrderedList.Count);
Assert.IsTrue(2 < mOrderedList.Count, testDescription);
}
I have multiple Asserts even within my loop, but all of the asserts are testing the single expectation:
if i > j then mOrderedList[i] > mOrderedList[j]
The test description at the iteration level is so you get Failures such as:
10 is greater than 9 which implies
TestActionMethodInfo: [Actions.File, Version=1.0.446.0, File, VerifyReadOnly]
is greater than
TestActionMethodInfo: [Actions.File, Version=1.0.446.0, File, Write]
Expected: True
But was: False
instead of just:
Expected: True
But was: False
The question/debate about my code above:
Is am I testing one thing?
I am asserting on 4 different comparison methods within the object which could be argued as testing 4 things not one. The counter is greater than is greater than is greater than and that all of the methods which make that evaluation should be consistent.
Related
I have a method I verify as part of my test. One of its parameters is an array that can potentially be very large (over 100 bytes). How can I easily find the point of failure, without needing to go through some serious debugging?
The tested line is:
mockDependeny.verify(x=>x.callMethod(expectedModel, expectedModel.Length, It.IsAny<otherKindOfParam>()));
expectedModel is passed in the method (a theory) and is an array.
One set of data/expected in the theory works, but the next one says that it failed. The message it gives me is not very helpful -
Expected invocation on the mock at least once, but was never performed: x=>x.callMethod([1,2,3,4,5,6,7,8,9,10,...], 175 , It.IsAny())
Perfomed invocations:
Mock(x): MyDependency.callMethod([1,2,3,4,5,6,7,8,9,10,...], 175 , instanceOfOtherParam)
All I can glean from this is that somewhere in the next n-10 items in the array, there is something that does not match up ( the first ten items are the same, the overall length is the same)
Is there a way to get better feedback from the test, so I do not have to debug and manually compare the contents of the expceted vs actual arrays?
Replace the expectedModel array parameter with an It.Is<> and implement anything you want. E.g.:
mockDependency.Verify(x => x.callMethod(It.Is<byte[]>(m => VerifyThisEnumerableParam(m, expectedModel)), expectedModel.Length, It.IsAny<object>()));
...
private bool VerifyThisEnumerableParam<T>(IEnumerable<T> received, IEnumerable<T> expected)
{
if (received != expected)
{
var receivedArray = received.ToArray();
var expectedArray = expected.ToArray();
if (receivedArray.Length != expectedArray.Length ||
receivedArray.Where((t, idx) => !Object.Equals(t, expectedArray[idx])).Any())
{
// now let's visualize the two thing
throw new AssertFailedException($#"received != expected
expected: {String.Join(", ", expected.Select(t=>t.ToString()))}
received: {String.Join(", ", received.Select(t => t.ToString()))}");
}
}
return true;
}
The above isn't fool proof (no additional null, etc. checks), but hope you got the idea. If you prefer just the first differing index or something else just implement it in the method.
I've tested List<string> vs IEnumerable<string>
iterations with for and foreach loops , is it possible that the List is much faster ?
these are 2 of few links I could find that are publicly stating that performance is better iterating IEnumerable over List.
Link1
Link2
my tests was loading 10K lines from a text file that holds a list of URLs.
I've first loaded it in to a List , then copied List to an IEnumerable
List<string> StrByLst = ...method to load records from the file .
IEnumerable StrsByIE = StrByLst;
so each has 10k items Type <string>
looping on each collection for 100 times , meaning 100K iterations, resulted with
List<string> is faster by amazing 50 x than the IEnumerable<string>
is that predictable ?
update
this is the code that is doing the tests
string WorkDirtPath = HostingEnvironment.ApplicationPhysicalPath;
string fileName = "tst.txt";
string fileToLoad = Path.Combine(WorkDirtPath, fileName);
List<string> ListfromStream = new List<string>();
ListfromStream = PopulateListStrwithAnyFile(fileToLoad) ;
IEnumerable<string> IEnumFromStream = ListfromStream ;
string trslt = "";
Stopwatch SwFr = new Stopwatch();
Stopwatch SwFe = new Stopwatch();
string resultFrLst = "",resultFrIEnumrable, resultFe = "", Container = "";
SwFr.Start();
for (int itr = 0; itr < 100; itr++)
{
for (int i = 0; i < ListfromStream.Count(); i++)
{
Container = ListfromStream.ElementAt(i);
}
//the stop() was here , i was doing changes , so my mistake.
}
SwFr.Stop();
resultFrLst = SwFr.Elapsed.ToString();
//forgot to do this reset though still it is faster (x56??)
SwFr.Reset();
SwFr.Start();
for(int itr = 0; itr<100; itr++)
{
for (int i = 0; i < IEnumFromStream.Count(); i++)
{
Container = IEnumFromStream.ElementAt(i);
}
}
SwFr.Stop();
resultFrIEnumrable = SwFr.Elapsed.ToString();
Update ... final
taking out the counter to outside of the for loops ,
int counter = ..countfor both IEnumerable & List
then passed counter(int) as a count of total items as suggested by #ScottChamberlain .
re checked that every thing is in place, now the results are 5 % faster IEnumerable.
so that concludes , use by scenario - use case... no performance difference at all ...
You are doing something wrong.
The times that you get should be very close to each other, because you are running essentially the same code.
IEnumerable is just an interface, which List implements, so when you call some method on the IEnumerable reference it ends up calling the corresponding method of List.
There is no code implemented in the IEnumerable - this is what interfaces are - they only specify what functionality a class should have, but say nothing about how it's implemented.
You have a few problems with your test, one is the IEnumFromStream.Count() inside the for loop, every time it want to get that value it must enumerate over the entire list to get the count and the value is not cached between loops. Move that call outside of the for loop and save the result in a int and use that value for the for loop, you will see a shorter time for your IEnumerable.
Also the IEnumFromStream.ElementAt(i) behaves similarly to Count() it must iterate over the whole list up to i (eg: first time it goes 0, second time 0,1, third 0,1,2, and so on...) every time where List can jump directly to the index it needs. You should be working with the IEnumerator returned from GetEnumerator() instead.
IEnumerable's and for loop's don't mix well. Use the correct tool for the job, either call GetEnumerator() and work with that or use it in a foreach loop.
Now, I know a lot of you may be saying "But it is a interface it will be just mapping the calls and it should make no difference", but there is a key thing, IEnumerable<T> Does not have a Count() or ElementAt() method!. Those methods are extension methods added by LINQ, and the LINQ classes do not know the underlying collection is a List, so it does what it knows the underlying object can do, and that is iterating over the list every time the method is called.
IEnumerable using IEnumerator
using(var enu = IEnumFromStream.GetEnumerator())
{
//You have to call "MoveNext()" once before getting "Current" the first time,
// this is done so you can have a nice clean while loop like this.
while(enu.MoveNext())
{
Container = enu.Current;
}
}
The above code is basically the same thing as
foreach(var enu in IEnumFromStream)
{
Container = enu;
}
The important thing to remember is IEnumerable's do not have a length, in fact they can be infinitely long. There is a whole field of computer science on detecting a infinitely long IEnumerable
Based on the code you posted I think the problem is with your use of the Stopwatch class.
You declare two of these, SwFr and SwFe, but only use the former. Because of this, the last call to SwFr.Elapsed will get the total amount of time across both sets of for loops.
If you are wanting to reuse that object in this way, place a call to SwFr.Reset() right after resultFrLst = SwFr.Elapsed.ToString();.
Alternatively, you could use SwFe when running the second test.
I'm playing around with Code Contracts at the moment and I'm not completely sure whether the static methods of the Contract class are powerful enough to compete with mathematical notation of conditions.
Let's assume we got a simple factorial method
int Factorial(int n);
I would express the following conditions:
Precondition:
n >= 0
Postconditions:
Factorial(n) = 1, in case n = 0
Factorial(n) = n*(n-1)*...*1, in case n > 0
These conditions clearly specify the behavior of Factorial in a short and clean way. My question is, whether they can be expressed through Code Contracts.
The precondition is trivial:
Contract.Requires(n >= 0)
The conditional post condition might be expresses using
if(n==0)
Contract.Ensures(Contract.Result<int>() == 1);
if(n > 0)
...
But I don't like the way I need the "if" statement here as it makes the plain list of pre- and postconditions harder to read. I hoped we would have something like
Contract.Ensures(...).InCase(...);
And last but not least, I do not have any idea how to express this, which is a common notation regarding math:
n*(n-1)*...*1
Guess I would need some kind of loop, but this would copy the whole implementation. Is there any smart way to express such notations?
Thank you in advance.
What you are looking for are Unit Tests, not Code Contracts.
Tipically, checks like if n=0, then f(n) = 1 and if n=3, then f(n) = 6 are Test Cases that should be expressed as Unit Tests.
In your case, I think a suitable post condition would be something like "The result is always >= 1". And nothing more than that.
Assuming that your factorial class looks something like this:
public class Factorial
{
public int Compute(int n)
{
if (n == 0)
return 1;
return n * Compute(n - 1);
}
}
a suitable Unit Test written with the NUnit Framework would be:
[TestFixture]
public class FactorialTests
{
[TestCase(0, 1)]
[TestCase(1, 1)]
[TestCase(2, 2)]
[TestCase(7, 5040)]
[TestCase(10, 3628800)]
public void Compute_ReturnsCorrectResult(int n, int expectedResult)
{
var sut = new Factorial();
Assert.AreEqual(expectedResult, sut.Compute(n));
}
}
Update (after the comments)
Stating result >= 1 does not fully specify the algorithm.
I don't think the Code Contract's job is to specify the algorithm in detail. the algorithm is specified by the method.
If the Code Contract was a complex piece of logic like the method itself, then I guess we would need a Code Contract Contract to verify that the Code Contract performs the correct checks. This obviously leads to infinite recursion.
I didn't expect n*(n-1)*...*1 to be accepted by the compiler. But some generic range operator in a LINQ-flavoured way would surely be a gread addition, e.g. From(n).To(1).Product() or From(n).To(m).Sum()
If there was such a form of expressing factorials (and probably there is) you could certainly use it in your code, rather than the Code Contracts.
Update 2
Just for fun, I found a LINQ way of computing Factorials:
Enumerable.Range(1, n == 0 ? 1 : n).Aggregate((a, i) => a * i);
You could try to the following:
Contract.Ensures(Contract.Result<int>() == AlternativeFactorial(n));
where AlternativeFactorial is:
[Pure]
public static int AlternativeFactorial(int n)
{
if(n==0)
return 1;
if(n > 0)
{
//Alternative implementation.
}
}
Of course anything you use in a contract should be side-effect free (pure).
Now as far as the factorial implementation, I cannot come up with a more compact "alternative" implementation than w0lf's. What you should consider though is changing the return value of your method from int to BigInteger. Factorials can get very large very quickly. Also note that by adding a post-condition on the factorial value, you will pretty much double the time your method will take to return a result. This can be resolved by building CodeContracts only on the debug configuration.
Again at the Rhino Mocks Noob Wall
mockUI.Expect( x => x.Update( new Frame[] {Frame.MakeIncompleteFrame(1, 5)} ) );
This is the exact argument that I need to match. Via trace statements, I have verified that is the actual output as well i.e. the code behaves as intended but the test disagrees. RhinoMocks responds with
TestBowlingScorer.TestGamePresenter.TestStart:
Rhino.Mocks.Exceptions.ExpectationViolationException : IScoreObserver.Update([Frame# 1, Score = 0 Rolls [ 5, PENDING, ]]); Expected #1, Actual #0.
A Frame object contains few properties but doesn't override Equals() yet (overridden ToString() seen above). Update receives an array of Frames;
How do I setup this expectation? I see an Is.Matching constraint.. not sure how to use it or rather am concerned with the verbose nature of it.
I have a helper NUnit style custom Assert
public static void AssertFramesAreEqual(Frame[] expectedFrames, Frame[] actualFrames)
{
// loop over both collections
// compare attributes
}
#Gishu,
Yeah, that's it. I just also learned about the Arg<> static class which should allow you to do something like this:
mockUI.Expect( x => x.Update(Arg<Frame[]>
.Matches(fs=>HelperPredicates.CheckFrames ( expected, fs)) ));
There is also the Arg<>.List configuration starting point which I have not explored yet but might be even better for what you want
Verified works.. don't know if this is THE RhinoMocks way
var expectedFrames = new Frame[] { Frame.MakeIncompleteFrame(1, 5) };
mockUI.Expect( x => x.Update(null) )
.IgnoreArguments()
.Constraints( Is.Matching<Frame[]>( frames => HelperPredicates.CheckFramesMatch(expectedFrames, frames) ) );
The helper predicate is just a function that returns a boolean value - True on an exact match else false.
public static bool CheckFramesMatch(Frame[] expectedFrames, Frame[] actualFrames)
{
// return false if array lengths differ
// loop over corresponding elements
// return false if any attribute differs
// return true
}
I have been asked to write a testing application that needs to test a new stored procedure on multiple rows in a database, in essence I want to do something like this:
[Test]
public void TestSelect()
{
foreach(id in ids)
{
DataTable old = Database.call("old_stored_proc",id);
DataTable new_ = Database.call("new_stored_proc",id);
Assert.AreEqual(old.Rows[0]["column"],ne_.Rows[0]["column"]);
}
}
When I run this test, if 1 row doesn't match the other, the entire test fails; instead I would like to count how many times the assertion was passed and how many times it has failed. Is there a way to do this with NUnit?
I realize that NUnit might be overkill and this is a simple task without it...I just wanted to learn it. ;)
Seems like you are just Asserting the wrong thing. If you want to check all the values and then assert that there are no errors (or show the number of errors) then try this:
[Test]
public void TestSelect()
{
int errors = 0;
foreach(id in ids)
{
DataTable old = Database.call("old_stored_proc",id);
DataTable new_ = Database.call("new_stored_proc",id);
if (old.Rows[0]["column"] != new_.Rows[0]["column"])
{
errors++;
}
}
Assert.AreEqual(0, errors, "There were " + errors + " errors.");
}
1) If the id's are constant and not looked up at test run time, create a separate unit test fixture for each id. That way you will know which id's are actually failing. See here for a write up on the problems with data driven tests:
http://googletesting.blogspot.com/2008/09/tott-data-driven-traps.html
2) If you need to dynamically look up the id's making it impossible to create a fixture for each id, use akmad's suggestion with one change. Keep a list of id's where the values are not equal and add the list to the error message. It will be extremely difficult to diagnose a failing test that only states the number of errors, as you won't know what id's cause the errors.
3) I don't know how difficult it would be to do in NUnit, but in PyUnit, when we need to run tests on dynamically generated data, we dynamically create tests fixtures and attach them to the TestCase class so that we have a failed test for each piece of data that does not pass. Though I imagine this would be much more difficult without python's dynamic abilities.
I know that the question is specifically about NUnit, but interestingly enough, Gallio/MbUnit has a feature which allows to run and catch several assertions at once.
[Test]
public void MultipleTest()
{
Assert.Multiple(() =>
{
Assert.IsTrue(blabla);
Assert.AreEqual(pik, pok);
// etc.
}
}
The Assert.Multiple is catching all the failing assertions and is going to report them at the end of the test.
I would count the number of rows which do not match and then would write an assertion which will compare this number with 0 and would return the number of non matching strings in the message.
you could also use Assert.Greater for this.
P.S. In principal you should try to do one assertion per unit test. That's the gist of it.
Well you could declare a counter and then assert the value of the counter to determine pass/fail
Also, you could do the bulk of the work in the test setup, and then just create multiple tests.
I'm not clear as to why you need all the assert stmts in the same test.
Based on the objective you laid out, the entire test should fail if one row doesn't match another. Counting the number of times an assertion passes or fails gives you less information than a comparison of the outcome you expected with the outcome you actually got.
I recently had the same issue. I combined the idea of counting errors with Yann Trevin's mention of Assert.Multiple into an extension method for IEnumberable that lets me do things like:
[Test]
public void TestEvenNumbers()
{
int[] numbers = new int[] { 2, 4, 12, 22, 13, 42 };
numbers.AssertAll((num) => Assert.That((num % 2) == 0, "{0} is an odd number", num));
}
Which results in the NUnit output:
TestEvenNumbers:
5 of 6 tests passed; 0 inconclusive
FAILED: 13: 13 is an odd number
Expected: True
But was: False
Expected: 6
But was: 5
And the solution to the OP's problem would be:
[Test]
public void TestSelect()
{
ids.AssertAll(CheckStoredProcedures);
}
private void CheckStoredProcedures(Id id)
{
DataTable old = Database.call("old_stored_proc",id);
DataTable new_ = Database.call("new_stored_proc",id);
Assert.AreEqual(old.Rows[0]["column"], new_.Rows[0]["column"]);
}
Here is the extension method (note that I used "All" instead of "Multiple" for consistency with Linq terminology):
using System;
using System.Text;
using System.Collections.Generic;
using NUnit.Framework;
public static class NUnitExtensions
{
public static void AssertAll<T>(this IEnumerable<T> objects, Action<T> test)
{
int total = 0;
int passed = 0;
int failed = 0;
int inconclusive = 0;
var sb = new StringBuilder();
foreach (var obj in objects)
{
total++;
try
{
test(obj);
passed++;
}
catch (InconclusiveException assertion)
{
inconclusive++;
string message = string.Format("INCONCLUSIVE: {0}: {1}", obj.ToString(), assertion.Message);
Console.WriteLine(message);
sb.AppendLine(message);
}
catch (AssertionException assertion)
{
failed++;
string message = string.Format("FAILED: {0}: {1}", obj.ToString(), assertion.Message);
Console.WriteLine(message);
sb.AppendLine(message);
}
}
if (passed != total)
{
string details = sb.ToString();
string message = string.Format("{0} of {1} tests passed; {2} inconclusive\n{3}", passed, total, inconclusive, details);
if (failed == 0)
{
Assert.Inconclusive(message);
}
else
{
Assert.AreEqual(total, passed, message);
}
}
}
}
You can use [TestCase()] attribute if a simple hard coded list of IDs.
[Test]
[TestCase(1234)]
[TestCase(5678)]
[TestCase(7654)]
public void TestSelect(int id)
{
DataTable old = Database.call("old_stored_proc", id);
DataTable new_ = Database.call("new_stored_proc", id);
Assert.AreEqual(old.Rows[0]["column"], new_.Rows[0]["column"]);
}
This will generate three separate tests for each ID and whatever nunit test runner you use will display pass/fail counts.
If need to generate a dynamic list of IDs then recommend using [TestCaseSource()] attribute.