I recently encountered an odd issue when performing unit tests. My solution contains a helper class with a property for getting the directory of the executing assembly. It looks like this:
public static class DirectoryHelper
{
public static string ExecutingAssemblyDirectory
{
get
{
var codeBase = Assembly.GetExecutingAssembly().CodeBase;
var uri = new UriBuilder(codeBase);
var path = Uri.UnescapeDataString(uri.Path);
return Path.GetDirectoryName(path);
}
}
}
This method is called through various test classes to get relative file paths to dependent resources.
Take the following contrived projects as examples:
TestProject1.dll - TestFixture1.cs
[TestFixture]
public class TestFixture1
{
[Test]
public void VerifyExecutingAssemblyDirectory1()
{
StringAssert.Contains(#"\TestProject1\bin\Debug",
DirectoryHelper.ExecutingAssemblyDirectory);
}
}
TestProject2.dll - TestFixture2.cs
[TestFixture]
public class TestFixture2
{
[Test]
public void VerifyExecutingAssemblyDirectory1()
{
StringAssert.Contains(#"TestProject2\bin\Debug",
DirectoryHelper.ExecutingAssemblyDirectory);
}
}
When these tests are ran individually they pass and the location of the returned assembly is the debug folder of the test class.
However, when ran together, TestFixture2.VerifyExecutingAssemblyDirectory2() is actually returning the path to the bin folder of TestProject1, rather than TestProject2.
I'm trying to determine why this behavior is happening and understand a better way of going about this.
I've found that using .GetCallingAssembly will resolve this problem, but it doesn't seem like I should have to do this.
I've created an example to reproduce this issue and posted to GitHub. TylerNielsen/NUnitExecutingAssemblyExample
Note: I'm aware of the TestContext.TestDirectory in NUnit, however this library is currently not dependent on NUnit and I'd prefer to keep it that way.
UPDATE
I'm running the NUnit tests through both Resharper in Visual Studio and via NUnit3-Console. When I run using NUnit3-Console, I'm only specifying the two individual .dlls and not providing any other arguments.
Both TestProject1 and TestProject2 reference the assembly containing DirectoryHelper. I'm assuming that your references cause the assembly to be copied to the individual (separate) output directories.
When you run both test assemblies together, one of them causes it's "personal" copy of that assembly to be loaded. The second one finds that the assembly is already in memory.
Of course, this behavior will depend on how you run the assemblies, which you haven't said. In the case where you use nunit3-console, it will also depend on your command-line arguments, especially whether you use a separate process for each assembly.
Related
I want to add a custom test reporter to NUnit. I already did it with NUnit2, but I now need to use NUnit3.
To implement the reporter, I need to get various events from the framework, like start, end and failure of tests.
In NUnit2 I used NUnitHook to register my EventListener and it worked pretty good.
In NUnit3 I need to use the extension point mechanism, but when I add the extension point to the project, VisualStudio (2012 ultimate) immediately fails to discover the NUnit tests.
[TypeExtensionPoint(Description = "Test Reporter Extension")]
public class MyTestEventListener : ITestEventListener
{
public void OnTestEvent(string report)
{
Console.WriteLine(report);
}
}
If I remove the ITestEventListener implementation declaration from the class, it rediscovers the tests perfectly.
[TypeExtensionPoint(Description = "Test Reporter Extension")]
public class MyTestEventListener //: ITestEventListener
{
public void OnTestEvent(string report)
{
Console.WriteLine(report);
}
}
Am I doing something wrong? is there a better way to achieve it?
You don't say where you are putting this code, but I am suspecting it's in your test assembly. If so, that's not where it belongs. NUnit engine extensions get installed into the NUnit engine, so they need to be in a separate assembly. Once you have a separate assembly, you need to tell the engine where it is. Currently, you do this by creating a file of type .addins in the same directory as the engine. (You could modify the existing file, but that introduces maintenance problems in the future)
A future release will have an easier way to install addins, but they will continue to be entirely separate from your tests.
A further problem is that you are using TypeExtensionPointAttribute. I didn't notice this originally in your code and it's probably the biggest error so I'm adding this info now.
An "ExtensionPoint" is the thing you are extending. NUnit defines ExtensionPoints, while you create Extenisons to extend them. TypeExtensionPointAttribute is used inside NUnit to define extension points. It's not used by you. You use the ExtensionAttribute to define your extension.
Your extension should be defined something like this:
[Extension(Description = "Test Reporter Extension", EngineVersion="3.4")]
public class MyTestEventListener : ITestEventListener
{
public void OnTestEvent(string report)
{
Console.WriteLine(report);
}
}
You don't say what version of NUnit you are running. Test Listeners are only supported beginning with version 3.4. The EngineVersion property above is purely documentary at this point, because 3.4 is also the first version to recognize it.
There is a new writeup in the NUnit docs that may be helpful: https://github.com/nunit/docs/wiki/Writing-Engine-Extensions
Description
I am writing unit tests for a method, which copies a file from a source to a destination. Basically it includes this code:
public void MyMethod()
{
// ...
File.Copy(source, destination, true);
// ...
}
In my unit test project, I have a test file: (test.png), which is located in the Resources folder of my unit test project. And I've set the Copy to Output property to Always.
I have 3 unit tests which are testing this method.
When they hit the line of the code which copies the file: source = "Resources\\test.png".
Issue
When I run the unit test individually, they all pass and everything is fine.
However, when I run All Tests in Visual Studio, I get this run time error and unit tests fail:
System.IO.DirectoryNotFoundException
Could not find a part of the path 'Resources\test.png'.
My Thoughts...(UPDATED)
Probably because Visual Studio runs each unit tests simultaneously in a separate thread and they all accessing the same file at the same time?
I think for every unit test, Visual Studio is cleaning bin/Debug and bin/Release folders. Then it copies all the required project files in that folder. This causes sometimes the file actually does not exist?
Question
How can I fix this problem?
Is there any settings of configurations to resolve this?
How can I run all unit tests in Visual Studio (and Team City) when multiple unit tests are accessing the same file?
You could try to rule out the multi-threading issue by following the instructions from MSDN: Executing Unit Tests in parallel on a multi-CPU/core machine, setting parallelTestCount to 1. If the tests now pass, you've narrowed down the problem.
However, if your tests are still failing when you run them in a group - and I think this is the more likely scenario -, then my advice would be to check for any state those tests are sharing. The pattern you describe (i.e. passes in isolation; fails when not in isolation) is a symptom typically exhibited by tests that are (incorrectly) sharing state, and that state is being modified by those tests, causing one or more tests to fail.
Accessing the same file should not be a problem. Make sure you don't have a cleanUp Fixture(TestSuite level) to delete the file. Because from exception it looks like the file is being deleted after running a test.
Also concurrent read operation is fine and perfectly legal. If your unit tests are overwriting the file then it's a problem.
What happened was, since I was using relative path to the testing files, for some reason when running the unit tests in batch, test runner working directory is different than when running individual tests, hence it couldn't find the directory.
So I used this function to build the absolute path to the testing files:
private string GetFilePath([CallerFilePath] string path = "")
{
return path;
}
Then:
string projectDir = Path.GetDirectoryName(GetFilePath());
string testFile = Path.Combine(projectDir, #"Resources\test.png";
I'd speculate that your problem is that one of the tested methods changes directory, given the explicit "Directory Not Found" exception. It's improbable that file locking or any concurrency problems would cause the behaviour described.
If unit testing you shouldn't really be testing whether File.Copy (or any of the File class methods) worked since you didn't write that code. Instead you should test whether your code interacts correctly with the File type (i.e. did it pass the correct source file name, desination file name and overwrite value when you called "Copy"). First create an interface for the File class and a wrapper for it that implements the interface;
public interface IFileWrapper
{
void Copy(string sourceFileName,string destFileName,bool overwrite);
//Other required file system methods and properties here...
}
public class FileWrapper : IFileWrapper
{
public void Copy(string sourceFileName, string destFileName, bool overwrite)
{
File.Copy(sourceFileName, destFileName, overwrite);
}
}
You should then make the class you are testing include an IFileWrapper parameter (dependency injection). In your unit tests you can then use a mocking framework such as Moq or you could write your own mock;
public class MockFileWrapper : IFileWrapper
{
public string SoureFileName { get; set; }
public string DestFileName { get; set; }
public bool Overwrite { get; set; }
public void Copy(string sourceFileName, string destFileName, bool overwrite)
{
SoureFileName = sourceFileName;
DestFileName = destFileName;
Overwrite = overwrite;
}
}
In real implementations pass in FileWrapper as the IFileWrapper parameter but in your unit tests pass in MockFileWrapper. By checking the properties of mockFileWrapper in your unit tests you can now determine whether you class calls Copy and how it is called. Since you are no longer sharing a real file between your unit tests you will avoid the chance of the tests sharing state or potentially locking the file.
As you mentioned in your answer, the test framework does not always run tests with the working directory set to your build output folder.
To instruct the test framework to place build artifacts or other files from your build output into the test directory, you need to use DeploymentItemAttribute. For your case, you would do something like:
const string destination = "Destination.txt";
const string source = "MyData.txt";
[DeploymentItem(source)]
[TestMethod]
public void MyMethod()
{
// …
File.Copy(source, destination, true);
// …
}
[TestCleanup]
public void Cleanup()
{
// Clean up the destination so that subsequent tests using
// the same deploy don’t collide.
File.Delete(destination);
}
Also ensure that your files are marked with a Build Action of Contents and Always Copy. Otherwise, they won’t be in the build output directory and won’t be able to be copied to the test directory.
I'm trying to write a SpecFlow test where I test what happens when my application reads a certain structure of folders and files. I want to include these folders and files in my project so the tests don't just run on my own computer.
For example, I have two folders in my Specs project. One called 'SimpleTestModel' and the other called 'ComplexTestModel'. How can I reference these folders in my SpecFlow tests?
You want a Test Fixture.
From Wikipedia:
In software testing, a test fixture is a fixed state of the software under test used as a baseline for running tests; also known as the test context. It may also refer to the actions performed in order to bring the system into such a state.
Examples of fixtures:
Loading a database with a specific, known set of data
Erasing a hard disk and installing a known clean operating system installation
Copying a specific known set of files
Preparation of input data and set-up/creation of fake or mock objects
Software used to systematically run reproducible tests on a piece of software under test is known as a test harness; part of its job is to set up suitable test fixtures.
For your specific problem:
Create a Fixtures directory in your SpecFlow test project. Inside that create any numbers of sub directories based on your tests to set up the directory and file structures that you need.
Add an <appSettings> in App.config entry define the root folder for all your test fixtures
<configuration>
...
<appSettings>
<!-- Path relative to the build output directory -->
<add name="FixturesRootDirectory" value="..\..\Fixtures" />
</appSettings>
...
</configuration>
In a [BeforeScenario] hook, set the absolute path to the fixtures directory on the current scenario context (reference: How do I get the path of the assembly the code is in?)
using System.Configuration;
using System.IO;
using TechTalk.SpecFlow;
namespace Foo
{
[Binding]
public class CommonHooks
{
[BeforeScenario]
public void BeforeScenario()
{
InitFixturesPath();
}
private void InitFixturesPath()
{
if (ScenarioContext.Current.ContainsKey("FixturesPath"))
return;
string codeBase = Path.GetDirectoryName(Assembly.GetExecutingAssembly().CodeBase)
+ Path.DirectorySeparatorChar
+ ConfigurationManager.AppSettings["FixturesRootDirectory"];
UriBuilder uri = new UriBuilder(codeBase);
string path = Uri.UnescapeDataString(uri.Path);
ScenarioContext.Current.Set<string>("FixturesPath", Path.GetDirectoryName(path));
}
}
}
Now you can use ScenarioContext.Current.Get<string>("FixturesPath") to get the root directory for all of your fixtures. You could even write your own Fixtures helper class:
public static class FixturesHelper
{
public static string Path { get; set; }
// other methods and properties making it easier to use fixtures
}
Let's say we have a short program:
namespace ConsoleTryIt
{
static class Program
{
static void Main(string[] args)
{
var sum = Add(1, 2);
}
private static int Add(int p, int p2)
{
return p + p2;
}
}
}
When create the unit test class for this class, Visual Studio create a test method with the attribute DeploymentItem. I read MSDN about this attribute but still don't get what it means.
/// <summary>
///A test for Add
///</summary>
[TestMethod()]
[DeploymentItem("ConsoleTryIt.exe")]
public void AddTest()
{
var expected = 122;
var actual = Program_Accessor.Add(1, 121);
Assert.AreEqual(expected, actual);
}
If you get the idea, please share!
Edit
Thanks everyone for your answers. So the idea is to copy the item given in the argument to the testing evironment's folder. My next question is: why does this method need this attribute while others don't?
I guess it's related to the private members on the tested class but nothing clear to me.
Please continue to discuss.
This specifies files that are required by the specific test. The test system creates a new directory where the tests are run from. With this attribute, you can make the test system copy specific files to that new directory.
Is used deploy files that are not necessary present in the Output directory to the folder used for that particular TestRun.
in the example you posted above, the test environment makes sure that "consoleTryIt.exe" is copied(and therefore present) in the test folder.
If the file is not found, the test is not even run, and a FileNotFound Exception is returned.
This means that the item is copied to the 'TestResults\Out' folder and is basically an artifact/necessary item for the test run. Thus, it is stored apart from the bin directory and does not get overwritten.
This is especially useful when running the tests in different environments (build server, no hard-coded paths...) and of course it is necessary to make the tests repeatable.
HTH.
Thomas
Ensures the files required for the test are copied over to the folder where MSTest runs its tests TestResults\Out.
Files in your solution should have "Copy Always" set for the files to be first copied to bin folder and then to MSTest's folder.
Make sure you have "Enable deployment" checked in testrunconfig.
Banging my head against a wall trying to get a really simple testing scenario working.
I'm sure I'm missing something really simple!
Whatever I do, I seem to get the following error from the NUnit gui when running a test against my DLL:
System.TypeLoadException : Type 'Castle.Proxies.ITestProxy' from assembly 'DynamicProxyGenAssembly2, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null' is attempting to implement an inaccessible interface.
Now I've seen reference to this error in heaps of places when looking in Stack Overflow and elsewhere, but the solution I keep finding doesn't seem to help. And I'm not even using an internal interface at this stage! The solution I see around the place is too put the following line in AssemblyInfo.cs
[assembly: InternalsVisibleTo("DynamicProxyGenAssembly2")]
I'm using:
Visual Studio 2010 Professional
c# 4.0
Moq 4.10810.8 Beta (bin deployed)
NUnit 2.5.5 (Installed in GAC)
To recreate this error, all I need to do is:
Create a new class library project
Reference Moq and Unit (as above)
Create an interface. I've called my interface ITest, made it public, and it has one method which is 'string TestMethod();'. Am doing this in the local project for simplicity.
Create a class called 'Testing', decorated with [TextFixture] and a test method called 'TestMethod' decorated with [Test]
Build the project, then run NUnit against the resulting dll in the Debug folder.
Here's the contents of my test class
namespace MoqTest {
[TestFixture]
public class Testing {
[Test]
public void TestMethod() {
var testMock = new Mock<ITest>();
testMock.Setup(x => x.TestMethod()).Returns("String val");
var xyz = testMock.Object;
Assert.AreEqual(1, 1);
}
}
}
---- UPDATE ---
After changing Moq version from 4.10810.8 to 4.0.10501.6 everything works fine!
The following test passes for me:
public interface ITest { string TestMethod(); }
public class Testing
{
[Test]
public void TestMethod()
{
var testMock = new Mock<ITest>();
testMock.Setup(x => x.TestMethod()).Returns("String val");
var xyz = testMock.Object;
Assert.AreEqual(1, 1);
}
}
If your interface is public and in the same assembly, there really should be no problem. I suspect that you just missed an accessibility keyword somewhere, as a non-public interface does provoke a runtime error because the proxying assembly will be unable to instantiate a type based on it.
Probably the best thing to do is start with the code I've provided and change one thing at a time until it matches your failing code. If you run your test in between each change, I presume you'll find what was missing.
If you do go back to an internal interface, note that your InternalsVisibleTo statement must be in the same assembly as your internal interface, not your test assembly. Also note that if your assembly is strongly named you may need to add a public key fingerprint to your InternalsVisibleTo statement as described in MSDN.
Yeah I had the same issue with Moq.4.0.10810.8 for NET40... When I downgraded to version 4.0.10531.7 everything went green again!
I'm using 4.10.1 now, and I got this same issue. I tried downgrading to 4.10.0, but to no avail.
I finally found that, although the interface I was using was marked as public, it was in a class without a modifier. I found 2 things got it to work:
1) Pull the interface outside of the class. Because the class was no longer making the interface internal, it became accessible to the assembly.
2) Mark the class public. With all parts of the path to the interface being marked public, the assembly had no trouble accessing.
These strategies worked in both 4.10.0 and 4.10.1.