I'm curious to know if there's any built-in mechanism to retry tests in the Visual Studio 2008 unit testing framework for C#.
Case in point, I have a C# unit test which looks something like:
[TestMethod]
public void MyMethod() {
DoSomething();
Assert.Something();
}
Now, occasionally DoSomething() performs badly; in that case I would like to rerun the DoSomething() method before reaching the assertion. Obviously I can do something like:
...
do {
Initialization();
DoSomething();
} while (PerformedOK() == false);
Assert.Something();
...
Though this is a bit cumbersome because of the added loop and repeating the test initialization which would otherwise be completely handled by other methods / class constructor.
My question is whether there is a more convenient mechanism for retrying a test, something like:
DoSomething();
if (PerformedOK() == false) Retry();
else Assert.Something();
which will automatically retry the test without registering it as a failure, while performing all the regular initialization code as usual.
Seriously...
occasionally DoSomething() performs
badly
A test should be green every time. If the tested code sometimes perform "badly", then you need to fix your code, isolating the different behavior. You should have two test, one where it Asserts correct when DoSomething fails (and is supposed to fail), and one where it Asserts correct when DoSomething is ok (and is supposed to be ok).
Having retry logic in a test is just wrong imo. You should always Assert on the expected outcome, and you should be able to isolate and instrument your code to return what you expect.
[Edit - added some code which could be used for a retry loop]
You could create a loop wrapper which takes whatever method in and calls it X number of times, or until it succeeds. You could also have the Loop function call your init, or pass it as a separate argument. The method could also return a bool if successful. Change the signature to fit your needs.
[TestMethod]
public void something()
{
Loop.LoopMe(TestMethod,3);
Assert.Something();
}
class Loop
{
public static void LoopMe(Action action, int maxRetry)
{
Exception lastException = null;
while (maxRetry > 0)
{
try
{
action();
return;
}
catch (Exception e)
{
lastException = e;
maxRetry--;
}
}
throw lastException;
}
}
Your second example is almost the same lines of code and same complexity. There are tons of ways to skin it, you could not that I am advocating it use recursion.
[TestMethod]
public void MyMethod() {
bool success = DoSomething();
Assert.IsTrue(success);
}
public boolean DoSomething(){
//Do whatever
if(performedOk){
return true;
}else{
//find a way to stop it.
}
}
But the point is it is a unit test. If something is causing it to go wrong, you need to find a way isolate your test, so that it is in a controlled environment.
Unless you have a requirement that says, test should pass eventually. The best retry logic you should use, is after it fails. Click the test and hit run again.
Related
I've got some C# Unit Tests that must run serially because they clean the whole database and setting up a specified state. Doing this in parallel causes unpredictable behaviour. For that reason I tried to set the number of parallel tests to 1 in the XML, but it still did parallel tests. So my next approach was to use [TestInitialize()] and [TestCleanup()] Methods/Extensions to force serial processing.
Here is my code:
static Object exclusiveDbAccess = new Object();
//Use TestInitialize to run code before running each test
[TestInitialize()]
public void MyTestInitialize()
{
lock (exclusiveDbAccess)
{
Monitor.Enter(exclusiveDbAccess);
}
}
//
//Use TestCleanup to run code after each test has run
[TestCleanup()]
public void MyTestCleanup()
{
lock (exclusiveDbAccess)
{
Monitor.Exit(exclusiveDbAccess);
}
}
This seems to work most of the time, but I had (even this is very seldom) effects that looked like there were still some parallel tests running.
As this happened always when there was a lot of load on the computer, I was wondering if this wasn't caused by a time out of the "lock()" (e.g. after 10 or 30 seconds). For example, if the lock () block would be skipped after unsuccessfully trying to acquire the lock for x seconds, this could cause these problems.
So I would like some expert to tell me the exact behavior of that "lock" statement. Please do not just post any "guesses". In case lock() is known to time out experience reports are of course welcome...
that seems like a wrong usage to me.
lock(syncObject) { somecode }
basically does the same as
Monitor.Enter(syncObject);
try { somecode }
finally { Monitor.Exit(syncObject); }
so it does not seem right to do lock and Monitor.Enter/Monitor.Exit on the same object
neither of those should timeout, unless you explicitly set some timeout (like Monitor.TryEnter(syncObject, timeout))
see msdn+
They don't time out - but you shouldn't be using lock and Monitor.Enter().
You should do it like this:
static Object exclusiveDbAccess = new Object();
//Use TestInitialize to run code before running each test
[TestInitialize()]
public void MyTestInitialize()
{
Monitor.Enter(exclusiveDbAccess);
}
//
//Use TestCleanup to run code after each test has run
[TestCleanup()]
public void MyTestCleanup()
{
Monitor.Exit(exclusiveDbAccess);
}
A lock statement of the form
lock (x) ...
is implemented exactly like this:
System.Threading.Monitor.Enter(x);
try {
...
}
finally {
System.Threading.Monitor.Exit(x);
}
(From the C# Language specification, section 8.12)
In a nunit test, can we an assert that if the test fails at a certain circumstance then it is a pass (i.e. it is expected to try and fail).
But it should pass under all other circumstances.
Thing is that the test could fall apart before it can reach it's assertions sections.
I mean something in the lines of
[TestSetup]
void init()
{
if (condition==true)
{
Assert.That(this test fails); /*any feature to do this?*/
}
}
If the test can fail and it's classed as a pass under some circumstances, it sounds a bad test. Split it out into individual tests with clear method names detailing what it's achieving.
So instead of having one test and having a conditional inside it, split it out into each scenario. That way you can have the one scenario, where it is supposed to fail under something like
// do stuff
bool success = DoStuff();
Assert.IsFalse(success);
It's a little hard to understand your question. Are you wanting Assert.Fail() to force a failure? Like so...
[TestSetup]
public void Init()
{
if (condition==true)
{
Assert.Fail();
}
}
If instead you want to check for failure instead of cause one, you should follow Arran's advice and check for a specific fact about the work you're validating - such as a method's return value.
You can also use the "Action" object to invoke the code in an action and test that action if it is an exception. Look at FluentAssertions they have lots of examples.
int number1 = 1;
int number0 = 0;
Action someAction = () => { int j = number1 / number0; };
someAction.ShouldThrow<DivideByZeroException>();
I'm going to provide a simple example of what I'm trying to do -- hopefully it is possible?
I basically have a class that does a whole ton of formatting/analyzing to the data. As a result, there a lot of things that can go wrong with this. The problem I have is handling the class when things go wrong. I want all execution of this class to stop once an error has occurred.
This class (AnalyzingStuff) is called from a parent form that does various things based on the result of this classes execution.
Ideally, I would fire an event named say "ABORT".
So in this code here I do the following:
Class AnalyzingStuff{
public event EventHandler ABORT;
public AnalyzingStuff(){
for(int i = 0; i < 999999; i ++){
AnalyzeSomeStuff();
AnalyzerSomeOtherStuff();
}
MoreStuff();
OtherStuff();
}
private void AnalyzeSomeStuff(){
if(someconditionNotMet){
//EXIT OUT OF THIS CLASS, STOP EXECUTION!!!
this.ABORT.Invoke(this, null);
}
}
}
Calling this 'ABORT' event, I would stop the execution of this class (stop the loop and not do anything else). I could also catch this event handler in some other parent form. Unfortunately, I can't find any way of stopping the execution of this class.
Ideas so far:
The obvious answer is to simply set a flag and constantly check this flag over and over in multiple places, but I really don't like this approach (my current implementation). Having to check this after every single method call (there are MANY) is ugly codewise.
I thought maybe a background worker or something where you could cancel the execution of the DoWork?
Use a form as a base class for the AnalyzingStuff so I can simply call "this.Close();".
What do you think is the best approach to this situation? Are these the best solutions? Are there any other elegant solutions to what I want here or am I going completely in the wrong direction?
EDIT: I have a series of try/catch blocks used throughout this code that is used to handle different errors that can occur. Unfortunately, not all of them call for an Abort to occur so they need to be caught immediately. Therefore, try/catch not the most ideal approach.. or is it?
Don't do analysys in the constructor. Do it in a main Analyze() method.
Use exceptions. If you want to abort because of a fatal error, throw a fatal exception. That is, throw an exception that you don't catch within the scope of the main analysis method.
class Analyzer
{
public Analyzer()
{
// initialize things
}
public void Analyze()
{
// never catch a fatal exception here
try
{
AnalyzeStuff();
... optionally call more methods here ...
}
catch (NonFatalException e)
{
// handle non fatal exception
}
... optionally call more methods (wrapped in try..catch) here ...
}
private void AnalyzeStuff()
{
// do stuff
if (something nonfatal happens)
throw new NonFatalException();
if (something fatal happens)
throw new FatalException();
}
}
outside:
{
var analyzer = new Analyzer();
try
{
analyzer.Analyze();
}
catch (FatalException)
{
Console.WriteLine("Analysis failed");
}
}
If you don't like using exception this way, you can accomplish the same thing by having every analysis method return a bool:
if (!AnalyzeStuff())
return false;
if (!AnalyzeMoreStuff())
return false;
...
return true;
But you end up with a lot of return statements or a lot of braces. It's a matter of style and preference.
Could you throw an Exception if things go wrong, and run a try catch around where you call the method in the loop?
if you do this you could do stuff if the class fails (which you will put in the catch), and stuff you could do to close connections to database ++ when it is done.
or you could make the methods return an int, to tell if the execution of the method was valid. ex. return 0; is valid execution, return 1-500 would then might be different error codes. Or you might go for the simple version of passing a bool. If you need to return values from methods other than the error code you could pass these as OUT variables. example following:
Class AnalyzingStuff{
public AnalyzingStuff(){
for(int i = 0; i < 999999; i ++){
if (!AnalyzeSomeStuff() || !AnalyzerSomeOtherStuff())
break;
}
MoreStuff();
OtherStuff();
}
private bool AnalyzeSomeStuff(){
if(someconditionNotMet){
return false;
}
return true;
}
}
You can of course use your event. I just removed it for the simplicity of it.
is there a MsTest Equivalent of Assert.Warning in MbUnit ?
The closest match is Assert.Inconclusive() - it doesn't make the test fail as such, but it doesn't succeed either. It fall into a third stage called Inconclusive.
A single Inconclusive test will cause an entire test suite to be Inconclusive.
There are overloads that supports custom messages as well:
Assert.Inconclusive("Ploeh");
I have a similar issue as I use NUnit for some projects. Try using
Console.Write("Some Warning");
You may want to use a custom exception.
The trouble with Assert.Inconclusive is that Test Explorer states the test wasn't even run. This may be misleading when running the test in the future, particularly if the test is run by other developers:
The way I've come to prefer is as follows. Firstly, define a custom UnitTestWarningException. I've given mine an additional constructor so I can pass my warning message String.Format-style with arguments:
public class UnitTestWarningException : Exception
{
public UnitTestWarningException(string Message) : base(Message) { }
public UnitTestWarningException(string Format, params object[] Args) : base(string.Format(Format, Args)) { }
}
Then, at the point where you want to end a unit test with a warning, throw a UnitTestWarningException instead:
[TestMethod]
public void TestMethod1()
{
.
.
.
try
{
WorkflowInvoker.Invoke(workflow1, inputDictionary);
}
catch (SqlException ex)
{
if (ex.Errors.Count > 0
&& ex.Errors[0].Procedure == "proc_AVAILABLEPLACEMENTNOTIFICATIONInsert")
{
//Likely to occur if we try to repeat an insert during development/debugging.
//Probably not interested--the mail has already been sent if we got as far as that proc.
throw new UnitTestWarningException("Note: after sending the mail, proc_AVAILABLEPLACEMENTNOTIFICATIONInsert threw an exception. This may be expected depending on test conditions. The exception was: {0}", ex.Message);
}
}
}
The result: Test Explorer then shows that the test has been executed, but failed with a UnitTestWarningException that shows your warning:
Here is my hack on how to have warnings with nunit ( i know this question was about mstest, but this should work too). As always, I am interested in any improvements. This method is working for me.
Background: I have code which checks the tests themselves for correct comments and has logic to detect if someone has copied and pasted another test without changing comments. These are warnings I want to be shown to the developer without normal Assert.Inconclusive blocking the actual test from running. Some are focused on the test and the cleanup refactorings phase is to remove the warnings.
Mission: to have warnings after all other asserts are run. This means even showing the warnings after Assert.Fail that normally occur in tests during development.
Implementation: (best to create a base class for all test files):
public class BaseTestClass
{
public static StringBuilder Warnings;
[SetUp]
public virtual void Test_SetUp()
{
Warnings = new StringBuilder();
}
[TearDown]
public virtual void Test_TearDown()
{
if (Warnings.Length > 0)
{
string warningMessage = Warnings.ToString();
//-- cleared if there is more than one test running in the session
Warnings = new StringBuilder();
if (TestContext.CurrentContext.Result.Status == TestStatus.Failed)
{
Assert.Fail(warningMessage);
}
else
{
Assert.Inconclusive(warningMessage);
}
}
}
Testing Usage
[Test]
public void Sample_Test()
{
if (condition) Warning.AppendLine("Developer warning");
Assert.Fail("This Test Failed!");
}
Actual Result:
"This Test Failed!"
"Developer warning"
Status of test is failed - RED
If the test passed and there was a warning, you will then get the status of Inconclusive - YELLOW.
I would like to have my TearDown method check whether the previous test was a success before it applies some logic. Is there an easy way to do this?
This has been already solved in Ran's answer to similar SO question. Quoting Ran:
Since version 2.5.7, NUnit allows Teardown to detect if last test failed.
A new TestContext class allows tests to access information about themselves including the TestStauts.
For more details, please refer to http://nunit.org/?p=releaseNotes&r=2.5.7
[TearDown]
public void TearDown()
{
if (TestContext.CurrentContext.Result.Status == TestStatus.Failed)
{
PerformCleanUpFromTest();
}
}
If you want to use TearDown to detect status of last test with NUnit 3.5 it should be:
[TearDown]
public void TearDown()
{
if (TestContext.CurrentContext.Result.Outcome.Status == TestStatus.Failed)
{
//your code
}
}
sounds like a dangerous idea unless it's an integration test, with say data to remove say. Why not do it in the test itself?
Obviously a private flag in the class could be set.
This is what Charlie Poole himself has suggested if you must
Only if you do this manually. In fact you even won't know which tests are intend to run. In NUnit IDE one can enable some tests and disable some other. If you want to know if some specific test has run you could include code like this in your test class:
enum TestStateEnum { DISABLED, FAILED, SUCCEDED };
TestStateEnum test1State = TestStateEnum.DISABLED;
[Test]
void Test1()
{
test1State = TestStateEnum.FAILED; // On the beginning of your test
...
test1State = TestStateEnum.SUCCEDED; // On the End of your Test
}
Then you can check the test1State variable. If the test throws an exception it won't set the SUCCEDED. you can also put this in a try catch finally block in your tests with a slightly different logic:
[Test]
void Test1()
{
test1State = TestStateEnum.SUCCEDED; // On the beginning of your test
try
{
... // Your Test
}
catch( Exception )
{
test1State = TestStateEnum.FAILED;
throw; // Rethrows the Exception
}
}
[OneTimeTearDown]
public void AfterEachTest()
{
if (TestContext.CurrentContext.Result.Outcome.Status.Equals(TestStatus.Failed))
{
Console.WriteLine("FAILS");
}
else if (TestContext.CurrentContext.Result.Outcome.Equals(ResultState.Success))
{
Console.WriteLine("SUCESS");
}
}
IMHO tear down logic should be independent of test results.
Ideally you should avoid using setup and teardown completely, a al xunit.net. See here for more info.