Nunit: TestCaseSource does not expect to generate all test cases? - c#

Hi everyone I have a problem with generate test cases for TestCaseSource. I wrote this code for tests:
using System;
using System.Collections.Generic;
using System.Linq;
using NUnit.Framework;
namespace HeapSort.Tests
{
[TestFixture]
public class Tests
{
[Test, TestCaseSource(typeof(TestsGenerator),"TestCases")]
public void IsEqualCollections(int[] received, int[] expected)
{
CollectionAssert.AreEqual(received, expected);
}
}
public class TestsGenerator
{
public static IEnumerable<TestCaseData> TestCases
{
get
{
for (var i = 0; i < 25; i++)
{
int[] t1 = GenerateCollection(i), t2 = t1.ToArray();
HeapSort.Sort(t1);
Array.Sort(t2);
yield return new TestCaseData(t1, t2);
}
}
}
private static int[] GenerateCollection(int seed)
{
var rnd = new Random(seed+DateTime.Now.Millisecond);
int size = rnd.Next(100, 10000);
int[] array = new int[size];
for (var i = 0; i < size; i++)
array[i] = rnd.Next(-100, 100);
return array;
// return Enumerable
// .Repeat(100, rnd.Next(100, 10000))
// .Select(i => rnd.Next(-100, 100))
// .ToArray();
}
}
}
Where is the problem? Rather than get 25 tests, I get from 1 to 8. And often at the starting point of testing it shows that the tests are 7/8 and in the end there is only one test case.
How can I solve this problem?
UPD1: What's interesting is when I run tests through the console I handle all 25 tests how do I achieve the same results through the GUI!?
P.S. sorry for my bad english.
Perhaps I should mention that I'm working under Ubuntu in Rider

DateTime.Now is normally not very accurate. Your loop is generating many identical tests because they are all starting from the same seed. Why are you using a seed rather than simply letting Random work on its own?
Different runners will handle identical test in different ways. If you indicate what runner you are using to execute your tests, I can edit this answer with more information. However, in general, you most certainly don't want to generate a bunch of tests with the same data. They don't do anything for you!

The reason is TestExplorer looks at the name of test case and groups ones with equal names. So you don't see all test cases results. As reason is TestExplorer you don't have the same problem running test using console. Look at this issue for details. According to the issue there are two solutions:
create class for your test case and override ToString() method
making it return unique values for each test case.
use TestCaseData class and its method SetName to set unique name for each test.

Related

Fastest way of incrementing an element of a List<(Guid, int)>

I have a List<(Guid, int)> (a list of value-tuples), and I want to increment the Item2 field of an element at a specified index. Based on the answers in this question, there are two ways to do it:
The first is to get a copy of the existing (Guid, int) at the specified index, increment the Item2 field of the copy, and replace the existing element with the copy.
The second is to use the CollectionsMarshal.AsSpan API (.NET 5), get the Span<(Guid, int)> representation of the backing array of the list, and update in-place the Item2 of the desirable element.
static void Increment1(List<(Guid, int)> list, int index)
{
(Guid, int) copy = list[index];
copy.Item2++;
list[index] = copy;
}
static void Increment2(List<(Guid, int)> list, int index)
{
Span<(Guid, int)> span = CollectionsMarshal.AsSpan(list);
span[index].Item2++;
}
Which of these two approaches is the most performant? I am interested about a benchmark on the newest .NET platform (.NET 7), in release mode.
I'm going to treat this as a more general question about benchmarking so this answer is more generally useful. I'll use your proposed algorithms as an example though.
A naive approach to benchmarking in .NET would be to execute your algorithm wrapped in a StopWatch, however, except for the most crude of estimations, (talk order of magnitude levels of accuracy), this is pretty useless. So please don't do this. Use BenchmarkDotNet instead.
For simple tests such as this, it's very simple.
Create a console app.
Install the package through NuGet.
Create a test class
Create a method for each of your tests and annotate with [Benchmark]
Create a setup method to get your data ready and annotate it with either [GlobalSetup] or [IterationSetup] depending on whether or not you want it resetting between each test.
Add a single-line top-level statement to run your tests.
using System.Runtime.InteropServices;
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Reports;
using BenchmarkDotNet.Running;
Summary? summary = BenchmarkRunner.Run<IncrementBenchmark>();
public class IncrementBenchmark
{
private List<(Guid, int)> _testData = new();
[GlobalSetup]
public void Setup()
{
_testData = new();
for (int i = 0; i < 10000; i++)
{
_testData.Add((Guid.NewGuid(), Random.Shared.Next()));
}
}
[Benchmark]
public void IndexCopyBasedIncrement()
{
for (int i = 0; i < _testData.Count; i++)
{
(Guid, int) copy = _testData[i];
copy.Item2++;
_testData[i] = copy;
}
}
[Benchmark]
public void SpanBasedIncrement()
{
Span<(Guid, int)> span = CollectionsMarshal.AsSpan(_testData);
for (int i = 0; i < _testData.Count; i++)
{
span[i].Item2++;
}
}
}
Then run your console app and wait for the results.
In this case, the span-based implementation is an order of magnitude faster.
Method
Mean
Error
StdDev
IndexCopyBasedIncrement
70.743 us
0.3260 us
0.3049 us
SpanBasedIncrement
4.679 us
0.0419 us
0.0391 us
Given the massive improvements made in .NET 7 to LINQ, you might be able to get further improvements by replacing my loop with something that can make use of those improvements, but that's beyond the scope of this answer.

Generating random numbers with yield changes its values during debug mode

I had a complex scenario when I was using a yield with random numbers and faced some strange behavior.
I tried to simplify my case with a simpler code example and managed to reproduce it with a very simple code.
In the following code, I would expect to get "Equal!" printed, but actually, it is not.
using System;
using System.Collections.Generic;
using System.Linq;
namespace ConsoleAppSample
{
static class Program
{
public static void Main(string[] args)
{
IEnumerable<int> randomNumbers = GetRandomNumbers();
int firstRandomnumber = randomNumbers.First();
if (firstRandomnumber == randomNumbers.First())
{
Console.WriteLine("Equal!");
}
}
private static readonly Random random = new Random();
private static readonly object syncLock = new object();
public static int RandomNumber(int min, int max)
{
lock (syncLock)
{
return random.Next(min, max);
}
}
private static IEnumerable<int> GetRandomNumbers(int loopCount = 2)
{
for (int i = 0; i < loopCount; i++)
{
yield return RandomNumber(int.MinValue, int.MaxValue);
}
}
}
}
I tried to debug it and found even something weirder. While holding the breakpoint at the same place the values of the randomNumbers variable change every moment.
In the following image, you can see the watch shows different numbers for the same variable.
If I change the GetRandomNumbers() method implementation to insert values into a list and return the list (without using yield), it will work fine.
I try to understand why it works this way? I guess it is related to the way random generates numbers based on the computer clock and the way yield works, but I am not sure.
yield means that the collection is not saved but essentially calculated when needed, namely each time you iterate the collection. Thus, each time new Random().Next() is called which may or may not give you a different result, depending on the time between those calls.
Thus, I would suggest the following changes:
If you want to see the same collection contents everytime, save the collection (for example, into a List as you suggested)
If you want to have multiple random numbers, keep the Random instance, if necessary in a static field. Creating a new instance will calculate a new seed based on the current system time and that may be the same, if you create two instances before Windows refreshes the system time.

Unit testing in C# how to?

I am new in the world of unit testing in C#.
I have a piece of code in my Main.cs
public static string Generate(int length)
{
char[] chars = "$%##!*abcdefghijklmnopqrstuvwxyz1234567890?;:ABCDEFGHIJKLMNOPQRSTUVWXYZ^&".ToCharArray();
string password = string.Empty;
Random random = new Random();
for (int i = 0; i < length; i++)
{
int x = random.Next(1, chars.Length);
if (!password.Contains(chars.GetValue(x).ToString()))
password += chars.GetValue(x);
else
i--;
}
return password;
}
now i don't know how to test this code with unit tests can someone give me a example?
EDIT:
I have make a test code
[TestMethod]
[Timeout(1000)]
public void RenderingPasswordShouldHaveMaximumSize()
{
var amountOfCharacters = Int32.MaxValue;
var generator = new PasswordGenerator();
var target = generator.Generate(amountOfCharacters);
// TODO
Assert.Fail("This method should throw an exception if you try to create a password with too many characters");
}
But it gives me the following error: Message: Test 'RenderingPasswordShouldHaveMaximumSize' exceeded execution timeout period can someone help me with this needs to have a max size of 74!
the idea of unit testing is to put something in a small method of you and check if the result is ok.
Visual Studio there is a project template for that. and there are also other tools like NUnit oderXUnit for tests in C#
There is a great pluralsight course:
and a webcast of Uncle Bob in which he demonstrates test driven development http://cleancoders.com/codecast/clean-code-episode-6-part-1/show
See also "Verifying Code by using Unit Tests" at msdn
A simple example using NUnit. Here I'm testing that when I pass 0 as an argument, nothing is generated (perhaps you should throw an Exception?)
[TextFixture]
public class Tests {
[Test]
public void Test_Length0_ReturnsNothing() {
string result = Generate(0);
Assert.IsTrue(string.IsNullOrEmpty(result));
}
}
You can write then similar tests (eg. to make sure it include the characters you want, etc).
There are much nuances in unit testing. I would recommend you to read book about unit testing "The Art of Unit Testing: With Examples in .Net".
There are described a lot of techniques and approach in unit testing. You can also find many examples here.
var amountOfCharacters = Int32.MaxValue;
var generator = new PasswordGenerator();
var target = generator.Generate(amountOfCharacters);
You're specifying the number of characters the password should contain as 2,147,483,647 characters...
chars.Length
There's only 74 possible values in your array.
No doubt it times out because the loop takes longer to iterate 2.2 billion times trying to find the last few values in your array.
for (int i = 0; i < length; i++)
Your logic changes too since you're not specifying the length of the password, but the number of iterations you want to do.

Visual Studio Unit Testing Assert.AreEqual Fails with Identical Expected and Actual Values

I am a total newb when it comes to Unit Testing, so please pardon me if this is total ignorance. But I can't get this method to pass a unit test even if I provide the exact same values for the expected and actual. When I set breakpoints and step through, I have confirmed that both expected and actual variables are a string array containing two items, blah and blah. But the test still fails every time stating that "Assert.AreEqual failed. Expected: System.String[] Actual:System.String[]"
namespace TestProject1
{
public class Testing
{
public string[] PMDate(string lastPm)
{
if (string.IsNullOrEmpty(lastPm))
return new []{"blah", "blah"};
string[] results = lastPm.Split(new[] {" "}, StringSplitOptions.None);
if (results.Length < 2)
return new[] {"blah", "blah"};
return results;
}
}
[TestClass()]
public class UnitTest1
{
[TestMethod()]
public void PmDateTest()
{
Testing test = new Testing();
string[] expected = test.PMDate("01/01/1900");
string[] actual = test.PMDate("01/01/1900");
Assert.AreEqual(expected, actual);
}
}
}
Your test will be using object.Equals, which isn't overridden for arrays. In other words, this will print false:
var x = new[] { 0 };
var y = new[] { 0 };
Console.WriteLine(x.Equals(y));
You should use CollectionAssert instead for collections:
CollectionAssert.AreEqual(expected, actual);
This is failing because the string arrays are two different objects, even if they contain the same values, they are different objects. You can simply verify this by checking expected.GetHashcode() and actual.GetHashCode() at runtime and you will see that the hash codes are different

NUnit: Running multiple assertions in a single test

I have been asked to write a testing application that needs to test a new stored procedure on multiple rows in a database, in essence I want to do something like this:
[Test]
public void TestSelect()
{
foreach(id in ids)
{
DataTable old = Database.call("old_stored_proc",id);
DataTable new_ = Database.call("new_stored_proc",id);
Assert.AreEqual(old.Rows[0]["column"],ne_.Rows[0]["column"]);
}
}
When I run this test, if 1 row doesn't match the other, the entire test fails; instead I would like to count how many times the assertion was passed and how many times it has failed. Is there a way to do this with NUnit?
I realize that NUnit might be overkill and this is a simple task without it...I just wanted to learn it. ;)
Seems like you are just Asserting the wrong thing. If you want to check all the values and then assert that there are no errors (or show the number of errors) then try this:
[Test]
public void TestSelect()
{
int errors = 0;
foreach(id in ids)
{
DataTable old = Database.call("old_stored_proc",id);
DataTable new_ = Database.call("new_stored_proc",id);
if (old.Rows[0]["column"] != new_.Rows[0]["column"])
{
errors++;
}
}
Assert.AreEqual(0, errors, "There were " + errors + " errors.");
}
1) If the id's are constant and not looked up at test run time, create a separate unit test fixture for each id. That way you will know which id's are actually failing. See here for a write up on the problems with data driven tests:
http://googletesting.blogspot.com/2008/09/tott-data-driven-traps.html
2) If you need to dynamically look up the id's making it impossible to create a fixture for each id, use akmad's suggestion with one change. Keep a list of id's where the values are not equal and add the list to the error message. It will be extremely difficult to diagnose a failing test that only states the number of errors, as you won't know what id's cause the errors.
3) I don't know how difficult it would be to do in NUnit, but in PyUnit, when we need to run tests on dynamically generated data, we dynamically create tests fixtures and attach them to the TestCase class so that we have a failed test for each piece of data that does not pass. Though I imagine this would be much more difficult without python's dynamic abilities.
I know that the question is specifically about NUnit, but interestingly enough, Gallio/MbUnit has a feature which allows to run and catch several assertions at once.
[Test]
public void MultipleTest()
{
Assert.Multiple(() =>
{
Assert.IsTrue(blabla);
Assert.AreEqual(pik, pok);
// etc.
}
}
The Assert.Multiple is catching all the failing assertions and is going to report them at the end of the test.
I would count the number of rows which do not match and then would write an assertion which will compare this number with 0 and would return the number of non matching strings in the message.
you could also use Assert.Greater for this.
P.S. In principal you should try to do one assertion per unit test. That's the gist of it.
Well you could declare a counter and then assert the value of the counter to determine pass/fail
Also, you could do the bulk of the work in the test setup, and then just create multiple tests.
I'm not clear as to why you need all the assert stmts in the same test.
Based on the objective you laid out, the entire test should fail if one row doesn't match another. Counting the number of times an assertion passes or fails gives you less information than a comparison of the outcome you expected with the outcome you actually got.
I recently had the same issue. I combined the idea of counting errors with Yann Trevin's mention of Assert.Multiple into an extension method for IEnumberable that lets me do things like:
[Test]
public void TestEvenNumbers()
{
int[] numbers = new int[] { 2, 4, 12, 22, 13, 42 };
numbers.AssertAll((num) => Assert.That((num % 2) == 0, "{0} is an odd number", num));
}
Which results in the NUnit output:
TestEvenNumbers:
5 of 6 tests passed; 0 inconclusive
FAILED: 13: 13 is an odd number
Expected: True
But was: False
Expected: 6
But was: 5
And the solution to the OP's problem would be:
[Test]
public void TestSelect()
{
ids.AssertAll(CheckStoredProcedures);
}
private void CheckStoredProcedures(Id id)
{
DataTable old = Database.call("old_stored_proc",id);
DataTable new_ = Database.call("new_stored_proc",id);
Assert.AreEqual(old.Rows[0]["column"], new_.Rows[0]["column"]);
}
Here is the extension method (note that I used "All" instead of "Multiple" for consistency with Linq terminology):
using System;
using System.Text;
using System.Collections.Generic;
using NUnit.Framework;
public static class NUnitExtensions
{
public static void AssertAll<T>(this IEnumerable<T> objects, Action<T> test)
{
int total = 0;
int passed = 0;
int failed = 0;
int inconclusive = 0;
var sb = new StringBuilder();
foreach (var obj in objects)
{
total++;
try
{
test(obj);
passed++;
}
catch (InconclusiveException assertion)
{
inconclusive++;
string message = string.Format("INCONCLUSIVE: {0}: {1}", obj.ToString(), assertion.Message);
Console.WriteLine(message);
sb.AppendLine(message);
}
catch (AssertionException assertion)
{
failed++;
string message = string.Format("FAILED: {0}: {1}", obj.ToString(), assertion.Message);
Console.WriteLine(message);
sb.AppendLine(message);
}
}
if (passed != total)
{
string details = sb.ToString();
string message = string.Format("{0} of {1} tests passed; {2} inconclusive\n{3}", passed, total, inconclusive, details);
if (failed == 0)
{
Assert.Inconclusive(message);
}
else
{
Assert.AreEqual(total, passed, message);
}
}
}
}
You can use [TestCase()] attribute if a simple hard coded list of IDs.
[Test]
[TestCase(1234)]
[TestCase(5678)]
[TestCase(7654)]
public void TestSelect(int id)
{
DataTable old = Database.call("old_stored_proc", id);
DataTable new_ = Database.call("new_stored_proc", id);
Assert.AreEqual(old.Rows[0]["column"], new_.Rows[0]["column"]);
}
This will generate three separate tests for each ID and whatever nunit test runner you use will display pass/fail counts.
If need to generate a dynamic list of IDs then recommend using [TestCaseSource()] attribute.

Categories

Resources