When I run my tests in Visual Studio individually, they all pass without a problem. However, when I run all of them at once some pass and some fail. I tried putting in a pause of 1 second in between each test method with no success.
Any ideas? Thanks in advance for your help...
It's possible that you have some shared data. Check for static member variables in the classes in use that means one test sets a value that causes a subsequent test to fail.
You can also debug unit tests. Depending on the framework you're using, you should be able to run the framework tool as a debug start application passing the path to the compiled assembly as a parameter.
It's very possible that some modifications/instantiations done in one test affect the others. That indicates poor test design and lack of proper isolation.
Everyone is probably right, some shared date is being modified between tests. But note the order of MS Test execution. Simply pausing between tests is not a solution. Each test is executed in it's own instance of the test class on a separate thread.
as per the other responses. It sounds like you have a singleton or a global variable that is causing the interaction.
Other unit test frameworks that I have used work hard to ensure that a test produces identical results whether the test is run individually or is run as part of the 'run them all' alternative. The goal is to prevent one test from having an effect on another due to side effects such as (for example) having one test leave the static state of a class in a configuration that another test is not expecting. The VS unit test framework does not appear to provide this isolation. I have 2 suggestions for minimizing the kinds of problems that the question implies. First, use non-static class in preference to a static class if the class has state (has anything other than static methods). Create a single instance of this class and have it keep the state info that was being kept in the static class. Second, if you do elect to have static class(es) with static state, have a static method that sets the static state back to 'empty' (e.g., a method that sets all static properties to null/zero/etc.). Call this at the end of each unit test to undo any effects that the test has imposed on the static state. (This is admittedly less than elegant but can be workable if done in moderation). Or do what I plan to do - find a unit test framework that provides isolation across tests.
I also ran into this problem although my issue ended up being a threading problem. In my case I was faking the HttpContext object since the tests relied on it's existence. However, I was setting this in the ClassInitialize method thinking this would be used for each method like below:
[ClassInitialize]
public static void ClassInit(TestContext testContext)
{
HttpContext.Current = new HttpContext(new HttpRequest(null, "http://tempuri.org", null), new HttpResponse(null));
}
However, it turns out that each test method in the class runs in a separate thread. So I had to add this code to every test method that relied upon it to fix the issue.
[TestMethod]
public void TestMethod1()
{
HttpContext.Current = new HttpContext(new HttpRequest(null, "http://tempuri.org", null), new HttpResponse(null));
...
}
[TestMethod]
public void TestMethod2()
{
HttpContext.Current = new HttpContext(new HttpRequest(null, "http://tempuri.org", null), new HttpResponse(null));
...
}
See link for more information on this.
http://blogs.msdn.com/b/nnaderi/archive/2007/02/17/explaining-execution-order.aspx
I faced a similar issue here how I solved it:
Copy the code of the second test inside the first test (after it).
Try to test the first test. The first test will probably fails, and then you can debug the first test (step by step) to find the static/shared variable or logic that makes the problem.
In my case, I had an Environment variable set with:
Environment.SetEnvironmentVariable("KEY", "value");
The implementation code for a few other tests was assuming a default value, and if the test(s) with the above line were executed first, those failed. The solution is to clean up with the following (at the end of each unit test, or in a special method for the same purpose - TestCleanup in MS/VSTest):
Environment.SetEnvironmentVariable("KEY", null);
Though redundant, it is best practice to also set the environment variable to any value previously assumed by (the failing) tests to be default. Do this at the top of those unit tests (the Arrange step).
Related
I wrote [Test,OneTimeSetup] above a test case is that running the OneTimeSetup method I defined again?
We have a heavy setup process that requires us to use a [OneTimeSetup] for Nunit. There was a problem that some of the data would be changed and cause a test case to fail when run in the Test Fixture but not as a Individual test. So above the test I wrote [Test, OneTimeSetup] and that solved the problem.
[OneTimeSetUp]
public void Initialize()
{
//Setup Code
}
[Test]
public void TestName1()
{
...
}
[Test, OneTimeSetUp]
public void TestName2()
{
//Test Code
}
I think your "fix" changed behavior in a way that no failure is seen but that you did't actually fix a dependency that is messing up your tests.
Here's what happens with the attributes you use.
Because you tricked NUnit to think that TestName2 is a one-time setup method, it runs first thing, before any other tests.
Then TestName1 runs, as the first test found. Although the order is not guaranteed, that's probably the order that's being used, based on your report.
Then TestName2 runs again, this time as a test, because it is marked as a test as well.
However, what you are doing should really be an error in NUnit. A method is either a Test method, SetUp method, TearDown method, etc. It's not intended to be more than one of those things. It would be unsurprising if the next release of NUnit detected what you are doing and gave an error.
A better solution is to analyze your tests to find the ordering dependency and remove it.
I am abstracting the history tracking portion of a class of mine so that it looks like this:
private readonly Stack<MyObject> _pastHistory = new Stack<MyObject>();
internal virtual Boolean IsAnyHistory { get { return _pastHistory.Any(); } }
internal virtual void AddObjectToHistory(MyObject myObject)
{
if (myObject == null) throw new ArgumentNullException("myObject");
_pastHistory.Push(myObject);
}
internal virtual MyObject RemoveLastObject()
{
if(!IsAnyHistory) throw new InvalidOperationException("There is no previous history.");
return _pastHistory.Pop();
}
My problem is that I would like to unit test that Remove will return the last Added object.
AddObjectToHistory
RemoveObjectToHistory -> returns what was put in via AddObjectToHistory
However, it isn't really a unit test if I have to call Add first? But, the only way that I can see to do this in a true unit test way is to pass in the Stack object in the constructor OR mock out IsAnyHistory...but mocking my SUT is odd also. So, my question is, from a dogmatic view is this a unit test? If not, how do I clean it up...is constructor injection my only way? It just seems like a stretch to have to pass in a simple object? Is it ok to push even this simple object out to be injected?
There are two approaches to those scenarios:
Interfere into design, like making _pastHistory internal/protected or injecting stack
Use other (possibly unit tested) methods to perform verification
As always, there is no golden rule, although I'd say you generally should avoid situations where unit tests force design changes (as those changes will most likely introduce ambiguity/unnecessary questions to code consumers).
Nonetheless, in the end it is you who has to weigh how much you want unit test code interfere into design (first case) or bend the perfect unit test definition (second case).
Usually, I find second case much more appealing - it doesn't clutter original class code and you'll most likely have Add already tested - it's safe to rely on it.
I think it's still a unit test, assuming MyObject is a simple object. I often construct input parameters to unit test methods.
I use Michael Feather's unit test criteria:
A test is not a unit test if:
It talks to the database
It communicates across the network
It touches the file system
It can't run at the same time as any of your other unit tests
You have to do special things to your environment (such as editing config files) to run it.
Tests that do these things aren't bad. Often they are worth writing, and they can be written in a unit test harness. However, it is important to be able to separate them from true unit tests so that we can keep a set of tests that we can run fast whenever we make our changes.
My 2 cents... how would the client know if remove worked or not ? How is a 'client' supposed to interact with this object? Are clients going to push in a stack to the history tracker? Treat the test as just another user/consumer/client of the test subject.. using exactly the same interaction as in real production.
I haven't heard of any rule stating that you're not allowed to call multiple methods on the object under test.
To simulate, stack is not empty. I'd just call Add - 99% case. I'd refrain from destroying the encapsulation of that object.. Treat objects like people (I think I read that in Object Thinking). Tell them to do stuff.. don't break-in and enter.
e.g. If you want someone to have some money in their wallet,
the simple way is to give them the money and let them internally put it into their wallet.
throw their wallet away and stuff in a wallet in their pocket.
I like Option1. Also see how it frees you from implementation details (which induce brittleness in tests). Let's say tomorrow the person decides to use an online wallet. The latter approach will break your tests - they will need to be updated for pushing in an online wallet now - even though the object behavior is not broken.
Another example I've seen is for testing Repository.GetX() where people break-in to the DB to inject records with SQL now in the unit test.. where it would have be considerably cleaner and easier to call Repository.AddX(x) first. Isolation is desired but not to the extent that it overrides pragmatism.
I hope I didn't come on too strong here.. it just pains me to see object APIs being 'contorted for testability' to the point where it no longer resembles the 'simplest thing that could work'.
I think you're trying to be a little overly specific with your definition of a unit test. You should be testing the public behavior of your class, not the minute implementation details.
From your code snippet, it looks like all you really need to care about is whether a) calling AddObjectToHistory causes IsAnyHistory to return true and b) RemoveLastObject eventually causes IsAnyHistory to return false.
As stated in the other answers I think your options can be broken down like so.
You take a dogmatic approach to your testing methodology and add constructor injection for the stack object so you can inject your own fake stack object and test your methods.
You write a separate test for add and remove, the remove test will use the add method but consider it a part of the test setup. As long as your add test passes, your remove should be too.
I had visual studio create a test for each member in my class. Here's one example:
/// <summary>
///A test for CloseCurrentTextLogFile
///</summary>
[TestMethod()]
public void CloseCurrentTextLogFileTest()
{
Logger.CloseCurrentTextLogFile();
Assert.Inconclusive( "A method that does not return a value cannot be verified." );
}
Based on the assert string, I'm wondering how to test this... Any ideas?
Static state methods naturally make themselves fairly untestable, so my suggestion is based around refactoring your code away from static methods.
I would turn Logger into an instance class that takes an IO object in the constructor. That will allow you to stub the IO object and you can Assert that your IO object's Close method was called.
This is only if you want to make your code 100% testable. Otherwise, I would agree with Mo that if it is not testable, then do not write a forced test...those tend to be very brittle. In the end, you need to be pragmatic about your code. Often a logger is useful to keep static, however as I already mentioned, these tend to be very untestable....so just be pragmatic about your work and don't write tests in the mere effort to get 100% code coverage...that 100% will come with a price...
UPDATE
Here is why this is not truly testable from a dogmatic POV of unit testing. You are not testing a unit of work, but instead you are testing the Logger AND the Logger's dependencies (the IO object in this case). It also makes your tests slower and requiring environmental setup and state (you must have first opened an actual file to close it, right?). These are all bad for unit testing, but ok for integration testing...so it depends on what kind of tests you are writing, also.
I would posit that if it's truly not testable, then it doesn't actually do anything at all, and shouldn't exist ;)
Something along the lines of this might work...
Assert.IsNotNull( Logger.File );
Logger.CloseCurrentTextLogFile();
Assert.IsNull( Logger.File );
Or check the status of Logger.FileOpenStatus or check that Logger.OpenFile(fname) throws an exception before closing, but not after. There's got to be something in Logger whose behaviour depends on whatever action CloseCurrentTextLogFile() performs.
You can check the state of Logger or you could call some other method on logger that will not produce an error because you called this method, which should succeeded if you had not called the method.
Am not sure either, but you could try the following:
A function is supposed to do something (write a file, set some variables, etc)
Maybe you can check if the variables have been writen, or file has been created.
You could mock up the Logger class and assert that the CloseCurrentTextLogFile is being called. Some might argue that you need to check that any open log files are closed, I personally do not agree with that as that would test the Logger itself not your method.
This is the kind of questions developers should ask themselves when they start designing their systems, how can I test my application.
I have a unit test called TestMakeAValidCall(). It tests my phone app making a valid call.
I am about to write another test called TestShowCallMessage() that needs to have a valid call made for the test. Is it bad form to just call TestMakeAValidCall() in that test?
For reference this is my TestMakeAValidCall() test.
[TestMethod]
public void TestMakeAValidCall()
{
//Arrange
phone.InCall = false;
phone.CurrentNumber = "";
// Stub the call to the database
data.Expect(x => x.GetWhiteListData()).
Return(FillTestObjects.GetSingleEntryWhiteList());
// Get some bogus data
string phoneNumber = FillTestObjects.GetSingleEntryWhiteList().
First().PhoneNumber;
// Stub th call to MakeCall() so that it looks as if a call was made.
phone.Expect(x => x.MakeCall(phoneNumber)).
WhenCalled(invocation =>
{
phone.CurrentNumber = phoneNumber;
phone.InCall = true;
});
//Act
// Select the phone number
deviceControlForm.SelectedNumber = phoneNumber;
// Press the call button to make a call.
deviceMediator.CallButtonPressed();
//Assert
Assert.IsTrue(phone.InCall);
Assert.IsTrue(phone.CurrentNumber == phoneNumber);
}
Refactor the setup to another method and call that method from both tests. Tests should not call other tests.
IMHO, you should do one of the following:
Create a method that returns a valid call, and use it separately for both tests (not one calling the other)
Mock the valid call for the ShowCallMessageTest
To offer a counter point:
I strongly believe that well designed unit test should depend on one another!
Of course, that makes sense only if the testing framework is aware of these dependencies such that it can stop running dependent test when a dependency fails. Even better, such a framework can pass the fixture from test to test, such that can build upon a growing and extending fixture instead of rebuilding it from scratch for each single test. Of course, caching is done to take care no side-effects are introduced when more than one test depends from the same example.
We implemented this idea in the JExample extension for JUnit. There is no C# port yet, though there are ports for Ruby and Smalltalk and ... the most recent release of PHPUnit picked up both our ideas: dependencies and fixture reuse.
PS: folks are also using it for Groovy.
I think its a bad idea. You want your unit test to test one thing and one thing only. Instead of creating a call through your other test, mock out a call and pass it in as an argument.
A unit test should test one unit/function of your code by definition. Having it call other unit tests makes it test more than one unit. I break it up in to individual tests.
Yes - unit tests should be separate and should aim to test only one thing (or at least a small number of closely-related things). As an aside, the calls to data.Expect and phone.Expect in your test method are creating expectations rather than stub calls, which can make your tests brittle if you refactor...
A unit vs. module....we also think tests should depend on reusable methods as well and should test at an api level testing integration of classes. Many just tests a single class but many bugs are at that integration between the class level. We also use verifydesign to guarantee the api does not depend on implementation. This allows you to refactor the whole component/module without touching a test(and we went through that once actually and it worked great). Of course, any architectural changes force you to refactor the tests but at least design changes in the module don't cause test refactor work(unless you change the behavior of the api of course implicitly like firing more events than you used to but that "would" be an api change anyways).
"Could someone ellaborate on how the refactoring would look like in this case? – Philip Bergström Nov 28 '15 at 15:33"
I am currently doing something like this and this is what i came up with:
Notice that ProcessorType and BuildProcessors both call TestLevels
the actual content besides that fact is unimportant
its using XUnit, and Shouldly NuGet package
private static void TestLevels(ArgProcessor incomingProcessor)
{
Action<ProcessorLevel, int> currentLevelIteration = null;
currentLevelIteration = (currentProcessor, currentLevel) =>
{
currentProcessor.CurrentLevel.ShouldBeEquivalentTo(currentLevel);
ProcessorLevel nextProcessor = currentProcessor.CurrentProcessor;
if (nextProcessor != null)
currentLevelIteration(nextProcessor, currentLevel + 1);
};
currentLevelIteration(incomingProcessor, 0);
}
[Theory]
[InlineData(typeof(Build), "Build")]
public void ProcessorType(Type ProcessorType, params string[] args)
{
ArgProcessor newCLI = new OriWeb_CLI.ArgProcessor(args);
IncomingArgumentsTests.TestLevels(newCLI);
newCLI.CurrentProcessor.ShouldBeOfType(ProcessorType);
}
[Theory]
[InlineData(typeof(Build.TypeScript), "TypeScript")]
[InlineData(typeof(Build.CSharp), "CSharp")]
public void BuildProcessors(Type ProcessorType, params string[] args)
{
List<string> newArgs = new List<string> {"Build"};
foreach(string arg in args) newArgs.Add(arg);
ArgProcessor newCLI = new OriWeb_CLI.ArgProcessor(newArgs.ToArray());
IncomingArgumentsTests.TestLevels(newCLI);
newCLI.CurrentProcessor.CurrentProcessor.ShouldBeOfType(ProcessorType);
}
I would like to unit test a method on a class I have made, but this method requires another method to be called first. Example:
// This would work
MyClass myClass1 = new MyClass(mockDevice);
myClass1.Run(myDatastructure);
myClass1.Stop();
// This would throw an InvalidOperationException
MyClass myClass2 = new MyClass(mockDevice);
myClass2.Stop();
Run is starting an operation on a hardware device, and Stop is of course trying to stop that operation (sending a reset-command and starting a timeout-timer).
Anyway I would like to test various post-conditions of calling Stop, but I would like NOT to have to call Run, because I am testing Stop - not Run! I would like something like this:
MyClass myClass = new MyClass(mockDevice);
myClass.Stop();
Assert.IsTrue(mockDevice.ResetCalled);
So far I only see one possible solution, and that is to create a TestableMyClass that inherits from MyClass, that makes it possible to set the right internal state of the MyClass instance before calling Stop. The problem with this solution is that I have to change my MyClass-implementation to have protected members instead of private members, and I don't like the idea of having to change the implementation in order to test it!
Should I use this solution, is there an error in my design, or is there a smarter way of doing this?
As far as I see it, you are already testing Stop in the two ways in which it can be used (with and without a running operation). As long as the mockDevice is doing its job, it seem to me that you're testing it reasonably. Ideally you should be able to verify the commands sent to the device etc (which most mock frameworks will make simple).
In this situation, personally, I would have two tests for this:
Test without Run() being called first. I would test if it really throws the exception. I would also test if the post conditions are what I expect them to be.
Test with Run() being called first. I would test only the post conditions, that I expect.
Those are the only two important uses of the method that have different behaviors - therefor I would test them both.
EDIT:
I understand, why you don't want to call run, before stop - you think that if run fails, the test, that is supposed to only test stop method will most likely fail as well.
However, I would assume, that you also have test for the run method. This means, that when the tests, that test the behavior of run method pass - stop method tests must pass as well. If the run method tests fail, then the results of run method tests are undefined - they may or may not fail.
So, I'd say, don't be afraid to call other dependent methods in your tests, but make sure you test those dependent methods in separate tests.
It won't help your immediate problem, but I tend to favour mapping state to type rather than having objects with different modes.
IdleDevice idle = new IdleDevice(mockDevice);
RunningDevice running = idle.Run(myDatastructure);
running.Stop();
// This would not compile, as IdleDevice has no Stop method
IdleDevice idle = new IdleDevice(mockDevice);
idle.Stop();
If you can't compile impossible situations, you don't need to test them.