is there a MsTest Equivalent of Assert.Warning in MbUnit ?
The closest match is Assert.Inconclusive() - it doesn't make the test fail as such, but it doesn't succeed either. It fall into a third stage called Inconclusive.
A single Inconclusive test will cause an entire test suite to be Inconclusive.
There are overloads that supports custom messages as well:
Assert.Inconclusive("Ploeh");
I have a similar issue as I use NUnit for some projects. Try using
Console.Write("Some Warning");
You may want to use a custom exception.
The trouble with Assert.Inconclusive is that Test Explorer states the test wasn't even run. This may be misleading when running the test in the future, particularly if the test is run by other developers:
The way I've come to prefer is as follows. Firstly, define a custom UnitTestWarningException. I've given mine an additional constructor so I can pass my warning message String.Format-style with arguments:
public class UnitTestWarningException : Exception
{
public UnitTestWarningException(string Message) : base(Message) { }
public UnitTestWarningException(string Format, params object[] Args) : base(string.Format(Format, Args)) { }
}
Then, at the point where you want to end a unit test with a warning, throw a UnitTestWarningException instead:
[TestMethod]
public void TestMethod1()
{
.
.
.
try
{
WorkflowInvoker.Invoke(workflow1, inputDictionary);
}
catch (SqlException ex)
{
if (ex.Errors.Count > 0
&& ex.Errors[0].Procedure == "proc_AVAILABLEPLACEMENTNOTIFICATIONInsert")
{
//Likely to occur if we try to repeat an insert during development/debugging.
//Probably not interested--the mail has already been sent if we got as far as that proc.
throw new UnitTestWarningException("Note: after sending the mail, proc_AVAILABLEPLACEMENTNOTIFICATIONInsert threw an exception. This may be expected depending on test conditions. The exception was: {0}", ex.Message);
}
}
}
The result: Test Explorer then shows that the test has been executed, but failed with a UnitTestWarningException that shows your warning:
Here is my hack on how to have warnings with nunit ( i know this question was about mstest, but this should work too). As always, I am interested in any improvements. This method is working for me.
Background: I have code which checks the tests themselves for correct comments and has logic to detect if someone has copied and pasted another test without changing comments. These are warnings I want to be shown to the developer without normal Assert.Inconclusive blocking the actual test from running. Some are focused on the test and the cleanup refactorings phase is to remove the warnings.
Mission: to have warnings after all other asserts are run. This means even showing the warnings after Assert.Fail that normally occur in tests during development.
Implementation: (best to create a base class for all test files):
public class BaseTestClass
{
public static StringBuilder Warnings;
[SetUp]
public virtual void Test_SetUp()
{
Warnings = new StringBuilder();
}
[TearDown]
public virtual void Test_TearDown()
{
if (Warnings.Length > 0)
{
string warningMessage = Warnings.ToString();
//-- cleared if there is more than one test running in the session
Warnings = new StringBuilder();
if (TestContext.CurrentContext.Result.Status == TestStatus.Failed)
{
Assert.Fail(warningMessage);
}
else
{
Assert.Inconclusive(warningMessage);
}
}
}
Testing Usage
[Test]
public void Sample_Test()
{
if (condition) Warning.AppendLine("Developer warning");
Assert.Fail("This Test Failed!");
}
Actual Result:
"This Test Failed!"
"Developer warning"
Status of test is failed - RED
If the test passed and there was a warning, you will then get the status of Inconclusive - YELLOW.
Related
I've noticed that NUnit's TestContext.CurrentContext.Result.Outcome.Status is always Inconclusive at the end of a test run. The CurrentContext is also unaware that any assertions have taken place.
Is it possible to get the status of a test before the [TearDown]?
I was hoping to use the value during the Dispose() of my test management class to capture metrics and other data for post-test diagnosis.
Code example from a new .NET Framework 4.6.1 project that only has the NuGet packages NUnit and FluentAssertions:
namespace TestProject
{
using FluentAssertions;
using NUnit.Framework;
[TestFixture]
public class Class1
{
[Test]
public void test1()
{
var a = 1;
var b = 2;
var c = 1;
var context = TestContext.CurrentContext;
a.Should().Be(c);
Assert.AreEqual(a, c);
}
}
}
A test result starts out as Inconclusive. If the test is skipped or ignored, then the result changes, but of course it is never executed.
If it is executed, the Outcome is Inconclusive until the test is over. Clearly, while you are still executing the test, it is not yet finished. When teardown begins, the outcome of the test is known, so it will vary according to whether the test method itself was successful. Of course, an exception in teardown may change the result to an Error state.
Bottom line, the Outcome field is not useful while the test method itself is still running. In any case, if you are executing code in the test method, the test has not yet failed. Otherwise, you would not have continued execution!
You say you can't use TearDown but the link you provide doesn't deal with the issue of accessing the test result. Can you explain further? What exactly do you want to do after checking the test result?
You could try using AssertionScope along with the AssertionScope.Succeeded flag. Although the intelisense on that property specifies:
Gets a value indicating whether or not the last assertion executed through this scope succeeded.
Example usage of AssertionScope
[Test]
public void Test()
{
var a = 1;
var b = 2;
var c = 1;
var context = new AssertionScope();
try
{
throw new Exception("Test");
}
catch (Exception e)
{
context.FailWith(e.ToString());
}
var strings = context.Discard();
Console.WriteLine(strings.StringJoin(","));
context.Succeeded.Should().BeFalse();
var someOtherContext = new AssertionScope();
try
{
c.Should().Be(a);
}
catch (Exception e)
{
someOtherContext.FailWith(e.ToString());
}
var discard = someOtherContext.Discard();
Console.WriteLine(discard.StringJoin(","));
someOtherContext.Succeeded.Should().BeTrue();
}
I am trying to add a testcase name as a folder name. I am adding it in Teardown method to detect first whether to testcase passed or fail then accordingly it adds Folder in Pass or Fail Folder which already exist.
[Test]
public void TestCase12345()
{
string Name = methodBase.Name;
teardown(Name);
}
[TearDown]
public void teardown(string testcase)
{
if (TestContext.CurrentContext.Result.Status == TestStatus.Passed)
{
string sourcepath = #"sourcepath";
string timestamp = DateTime.Now.ToString("yy-MM-dd hh-mm");
string destpath = (#"destinationlocation" + "Test Case - " + testcase + " " + timestamp);
...
}
Error:
Invalid signature for SetUp or TearDown method: teardown
What am I missing over here?
You can't pass parameters to [TearDown] method, NUnit doesn't support it. For example, to pass parameters to [Test] you do something like this
[Test]
[TestCase("abc", 123)]
public void TestCase12345(string str, int number)
{
}
But as I said, NUnit doesn't support it in [TearDown].
As a side note, the check if the test succeeded should be in the test method (I find Assert very useful for that purpose). TearDown should be used only for the "cleaning", i.e. dispose of the WebDriver and any other things you created for the test and doesn't close automatically when the code is finished.
Edit
"Then what is the solution. how can add function name which I am calling it to create folder?"
You can implement an EventListener interface.
EventListeners are able to respond to events that occur in the course of a test run, usually by recording information of some kind.
For example
public class TestEventListaener : EventListener
{
// The test collection started/finished to run.
void RunStarted(string name, int testCount);
void RunFinished(TestResult result );
void RunFinished(Exception exception );
void TestStarted(TestName testName)
{
// create folder with the test name
}
void TestFinished(TestResult result)
{
// if test succeeded insert data to the folder otherwise delete it
}
// more `EventListener` methods
}
In addition to KjetilNodin's answer I would try removing the test case parameter from your tear down function since it is most likely expecting a function without parameters to be used for TearDown.
If you need this functionality I'd remove the TearDown attribute from the function and just call it at the end of your test case as in your example.
Two separate issues here:
1= As others have said, NUnit doesn't allow a teardown method to take an argument.
2 = If you call your method from the test, then it isn't functioning as a TearDown method but as part of the test code. The test has not yet finished, so it has no final result. What's more, if you should have a failed assert, you'll never reach the call.
You should remove the argument and the call. Use TestContext to get the test name as suggested in one of the comments.
I've a method test that does not have a clear assert expression. The returned tested value is a very long string that has to be inspected by a programmer to check if it is correct or not. For this reason, if the code executes without exceptions, I'm calling 'Assert.Inconclusive'.
However, if some kind of Exception is thrown, I want to call 'Assert.Fail' with the exception message. Something like this:
[TestMethod()]
public void Test()
{
try {
string toBeChecked = MethodToBeTested();
//the string is so particular that no clear
//assertion can be made about it.
Console.WriteLine(toBeChecked);
Assert.Inconclusive("Check the console output.");
} catch(Exception e) {
Assert.Fail(e.Message);
}
}
The problem with this code is that if no regular exception is thrown, the Assert.Inconclusive method also throws an exception that is catched, so Assert.Fail is called, and from the IDE test result pane it seems that the test has failed. This is not what I want.
Is there a way to filter the exceptions, such as catching every exception but the Assert-like ones?
(I'm using .NET framework 3.5SP1)
Why not just leave out the Assert.Inconclusive()? In fact, why catch any exceptions at all - if the code throws an exception the unit test framework will mark it as failed. Less is more:
[TestMethod()]
public void Test()
{
string toBeChecked = MethodToBeTested();
Console.WriteLine(toBeChecked);
}
But this is a poor unit test if we can not automatically check the result. All we are checking is that no exception is thrown.
Are there no asserts you can make about the resulting string?
For example, that is is not null or empty? Do we expect it to contain a certain sub-string that we can test?
At least give the test a helpful name, which includes something like: ManualAssertRequired
The Assert.Inconclusive method should throw a AssertInconclusiveException so you can either mark the Test as ExcpectedException(typeof(AssertInconclusiveExcpetion)) or use something like this:
[TestMethod()]
public void Test()
{
try {
string toBeChecked = MethodToBeTested();
//the string is so particular that no clear
//assertion can be made about it.
Console.WriteLine(toBeChecked);
Assert.Inconclusive("Check the console output.");
} catch(AsssertInconclusiveException) {
/* Do nothing */
}
} catch(Exception e) {
Assert.Fail(e.Message);
}
}
Try to catch specific exception type instead of Exception or add another catch for nunit exception that is caused by Assert.Inconclusivemethod which is AssertInconclusiveException...
For example modify it like this:
[TestMethod()]
public void Test()
{
try {
string toBeChecked = MethodToBeTested();
//the string is so particular that no clear
//assertion can be made about it.
Console.WriteLine(toBeChecked);
Assert.Inconclusive("Check the console output.");
} catch(AssertInconclusiveException e) {
// do nothing...
} catch(Exception e) {
Assert.Fail(e.Message);
}
}
I'm curious to know if there's any built-in mechanism to retry tests in the Visual Studio 2008 unit testing framework for C#.
Case in point, I have a C# unit test which looks something like:
[TestMethod]
public void MyMethod() {
DoSomething();
Assert.Something();
}
Now, occasionally DoSomething() performs badly; in that case I would like to rerun the DoSomething() method before reaching the assertion. Obviously I can do something like:
...
do {
Initialization();
DoSomething();
} while (PerformedOK() == false);
Assert.Something();
...
Though this is a bit cumbersome because of the added loop and repeating the test initialization which would otherwise be completely handled by other methods / class constructor.
My question is whether there is a more convenient mechanism for retrying a test, something like:
DoSomething();
if (PerformedOK() == false) Retry();
else Assert.Something();
which will automatically retry the test without registering it as a failure, while performing all the regular initialization code as usual.
Seriously...
occasionally DoSomething() performs
badly
A test should be green every time. If the tested code sometimes perform "badly", then you need to fix your code, isolating the different behavior. You should have two test, one where it Asserts correct when DoSomething fails (and is supposed to fail), and one where it Asserts correct when DoSomething is ok (and is supposed to be ok).
Having retry logic in a test is just wrong imo. You should always Assert on the expected outcome, and you should be able to isolate and instrument your code to return what you expect.
[Edit - added some code which could be used for a retry loop]
You could create a loop wrapper which takes whatever method in and calls it X number of times, or until it succeeds. You could also have the Loop function call your init, or pass it as a separate argument. The method could also return a bool if successful. Change the signature to fit your needs.
[TestMethod]
public void something()
{
Loop.LoopMe(TestMethod,3);
Assert.Something();
}
class Loop
{
public static void LoopMe(Action action, int maxRetry)
{
Exception lastException = null;
while (maxRetry > 0)
{
try
{
action();
return;
}
catch (Exception e)
{
lastException = e;
maxRetry--;
}
}
throw lastException;
}
}
Your second example is almost the same lines of code and same complexity. There are tons of ways to skin it, you could not that I am advocating it use recursion.
[TestMethod]
public void MyMethod() {
bool success = DoSomething();
Assert.IsTrue(success);
}
public boolean DoSomething(){
//Do whatever
if(performedOk){
return true;
}else{
//find a way to stop it.
}
}
But the point is it is a unit test. If something is causing it to go wrong, you need to find a way isolate your test, so that it is in a controlled environment.
Unless you have a requirement that says, test should pass eventually. The best retry logic you should use, is after it fails. Click the test and hit run again.
I would like to have my TearDown method check whether the previous test was a success before it applies some logic. Is there an easy way to do this?
This has been already solved in Ran's answer to similar SO question. Quoting Ran:
Since version 2.5.7, NUnit allows Teardown to detect if last test failed.
A new TestContext class allows tests to access information about themselves including the TestStauts.
For more details, please refer to http://nunit.org/?p=releaseNotes&r=2.5.7
[TearDown]
public void TearDown()
{
if (TestContext.CurrentContext.Result.Status == TestStatus.Failed)
{
PerformCleanUpFromTest();
}
}
If you want to use TearDown to detect status of last test with NUnit 3.5 it should be:
[TearDown]
public void TearDown()
{
if (TestContext.CurrentContext.Result.Outcome.Status == TestStatus.Failed)
{
//your code
}
}
sounds like a dangerous idea unless it's an integration test, with say data to remove say. Why not do it in the test itself?
Obviously a private flag in the class could be set.
This is what Charlie Poole himself has suggested if you must
Only if you do this manually. In fact you even won't know which tests are intend to run. In NUnit IDE one can enable some tests and disable some other. If you want to know if some specific test has run you could include code like this in your test class:
enum TestStateEnum { DISABLED, FAILED, SUCCEDED };
TestStateEnum test1State = TestStateEnum.DISABLED;
[Test]
void Test1()
{
test1State = TestStateEnum.FAILED; // On the beginning of your test
...
test1State = TestStateEnum.SUCCEDED; // On the End of your Test
}
Then you can check the test1State variable. If the test throws an exception it won't set the SUCCEDED. you can also put this in a try catch finally block in your tests with a slightly different logic:
[Test]
void Test1()
{
test1State = TestStateEnum.SUCCEDED; // On the beginning of your test
try
{
... // Your Test
}
catch( Exception )
{
test1State = TestStateEnum.FAILED;
throw; // Rethrows the Exception
}
}
[OneTimeTearDown]
public void AfterEachTest()
{
if (TestContext.CurrentContext.Result.Outcome.Status.Equals(TestStatus.Failed))
{
Console.WriteLine("FAILS");
}
else if (TestContext.CurrentContext.Result.Outcome.Equals(ResultState.Success))
{
Console.WriteLine("SUCESS");
}
}
IMHO tear down logic should be independent of test results.
Ideally you should avoid using setup and teardown completely, a al xunit.net. See here for more info.