Good afternoon/morning/evening folks,
I was wondering is it possible for me to "execute" a SpecFlow test via some sort of test harness (not NUnit)?
Previously my test harness I built ran MS Unit tests by calling methods from within the DLL that was created when I compiled the tests.
I'm assuming the same is possible in theory since a DLL is created, but im wondering how it will get all of the arguments etc.
So in short, is this possible if so is there a straight forward way to do this or am I barking up the wrong tree?
It's possible, but I'm not clear why you would want to.
Specflow is basically just a clever way of generating tests. Normally these are nUnit tests, but they can also be switched to use mstest. When you save your edits to the .feature file then VS runs a Custom Tool that converts your plaintext into a .feature.cs file that contains a code version of what you wrote with nUnit attributes applied to the methods.
Later, an nUnit runner (nUnit, resharper, gallio, teamcity etc) loads the dll and looks for all public methods marked with [Test] inside public classes marked with [TestFixture]. These methods get called.
There is nothing to stop you writing your own runner, however I'm not sure why you would do that. nUnit provides extensive reporting of the success of your test run in xml format, so its probably faster just to write something to parse that.
So I decided to invest some time on this and figured that using reflection was the way to do this task.
Here is some of my code:
TestRunner.TestDLLString = getDLL(project);
var TestDLL = Assembly.LoadFrom(TestDLLString);
Type myClassType = TestDLL.GetType("SeleniumDPS." + testname);
var instance = Activator.CreateInstance(myClassType);
MethodInfo myInitMethod = myClassType.GetMethod("Initialize");
try
{
myInitMethod.Invoke(instance, null);
}
catch (Exception ex)
{
//Error logging etc
}
I then repeat that for the "[TestMethod]" etc. I understand some people dislike reflection but in this instance the performance isnt critical so it works quite well for us.
So essentially what im doing is reading on the name of the test from an XML file then searching the DLL for that test method, then executing the Intitialize method, and later on executing the test method itself. After the test is run I then execute the cleanup method.
It might seem a bit hacky and NUnit might seem the logical choice for some, but as I mentioned earlier I needed a customizable approach. Thanks for all the suggestions though.
Related
I'm trying to run a UnitTest in C#, but only if another UnitTest passes? Can't quite get it to work, does anybody know how I can do this?
Edit: It's in NUnit
I encountered this problem a while back. A solution I came up with may not be the most elegant but it works. What I did to get around the ordering issue is I created my own unit testing framework. It was propriety to the company I worked for so I can't share it with you.
In the testing framework, there existed
A template for each type of test & a generic template to aggregate all the tests
A utility to execute each template type
For example, if I was doing an integration test, I would have a "http utility" and the template would contain the endpoint & payload.
The tests I wanted to run would need to be stored into an intermediate data structure such as json. This allowed me to serialize the tests into templates.
Now this is where it gets tricky... Using some fancy T4 templating, I would get the json data & serialize it to a list of templates. Then I would order the tests by execution order and dependency (one test could be dependent on another for chaining integration tests). I would then generate a unit test for every template. The generated unit tests would then run on build
For your question about canceling test execution if one fails, you can build that into your templating using some fancy logic
static List<ITestTemplate> requiredTests = new List<ITestTemplate>();
...
if(requiredTests.Any(t => t.Failed))
Assert.IsTrue(false) //fail subsequent tests
You could accomplish this, though it may not be the cleanest solution, by using the [Order()] attribute. (docs)
This will allow you to run the dependency-test as [Order(1)] and the test relying on the first test as [Order(2)]. You can share your driver across the tests, and if the first test fails, close the driver, causing the other tests relying on the first test passing to fail.
I want to run C90 tests in NUnit. I have written a short adapter that actually works, but i want to get rid of writing that adapter and instead make a tool that opens nunit and passes in loadable data so that nunit can run the tests.
I spare you the codeload i wrote so far and just summarize its usage:
I wrtie my C90 tests as callable dll functions with annotations above them like comments. Example /TEST/. As of now the test returns 0 or 1 for pas fail information.
A C# Pathinspector inspects all *.c files for annotations and extracts the loadinformation.
A C# Executer manages the execution of the tests and returns PASS/FAIL information.
What i want to do instead of my Executer is to create either the Launchinformation for NUnit and pass that into a freshly created gui, or create C# code to pass into the freshly created gui.
I know how to create an NUnit gui runner, but i dont know how to pass information into it.
Nunit source is online available, however it uses a lot of singletons during initialisation so i would argue against it.
One way could be to actually write and compile cs files and pass those compiled into nunit.
The best approach seems to be writing an xUnit compliant framework from scratch.
I've written a class and want to test if it works well. For now I think the best way to do it is to create new console application referencing main project, then make new instance of my class and mess with it. This approach unlike others enables IntelliSense, using keywords (no full names for classes) and Debugging.
Anyone knows how to do it in more convenient way to do this without making new console app?
Using a console app to test your class is what I would call a "poor man's unit test."
You are on the right track in wanting to do this sort of testing and I (and most others on SO) would suggest using a unit testing framework of some sort to help you out. (Mocking may be important and useful to you as well.)
Here's the thing though. Regardless of what framework you use, or if you go with a console app to test your code, you do have to create a separate project, or a separate, significant chunk of code of some sort, to be able to execute tests properly and independently. That's just part of the process. It is an investment but don't let the extra work keep you from doing it. It will save a lot time, and your skin, a little while in the future. Maybe even next week.
Also, while you're looking up unit testing make sure to also study up on test-driven development (TDD.)
Unit testing is absolutely the way to go. Depending on what version of VS you are running, there may be unit testing functionality built in, or you may have to use an additional tool such as NUnit. Both options are good and will allow you to fully test your classes.
Bear in mind also, that a comprehensive suite of unit tests will make refactoring much easier in the long run as well. If you refactor and break your unit tests, you know you've made a boo-boo somewhere. :)
Unit testing is the way forward here> this is a good introductory article.
The basic concept of a unit test is that you isolate and invoke a specific portion of code and assert that the results are expected and within reason. For example lets say you have a simple method to return the square of a floating point number:
public float Square(float value)
{
return value * value;
}
A reasonable unit test would be that the method returns the correct value, handles negetive values etc:
Assert.AreEqual(25, Square(5));
Assert.AreEqual(100, Square(-10));
Unit tests are also a good way to see how your code handles edge cases:
Assert.Throws<OverflowException>(Square(float.MaxValue));
If you are using VS 2010, check out Pex and Moles...
http://research.microsoft.com/en-us/projects/pex/
The Console App approach is more of a test harness for your class, which is fine.
But you can also use a unit testing framework to test your class. Visual Studio has one built in or you can leverage something like NUnit.
Also, try the Object Test Bench in Visual Studio. It should allow you to create a new instance, modify and view properties, and call some methods. It usually only works with very simple apps, though.
If you use Visual Studio 2008 or higher you will be able to test your code using MSTest framework:
1.Open Test View window: Test/Windows/Test View;
2.Add new unit test project: right click on Solution in Solution Explorer/Add/New
Project/Test Project;
3.Remove all files apart from UnitTest.cs file in created test project;
4.Write your unit test in method under [TestMethod] attribute:
[TestClass]
public class UnitTest1
{
[TestMethod]
public void TestMethod1()
{
var ranges = new Ranges();
int sum = ranges.CountOccurrences(11);
Assert.AreEqual(128, sum);
}
}
5.Run your test from Test View window added in p.1
6.See test results in Test/Windows/Test Results window
I've created a test suite in NUnit that references several distinct unit test fixtures in various assemblies.
I've pretty much used the example code from NUnit's docs:
namespace NUnit.Tests
{
using System;
using NUnit.Framework;
using System.Collections;
public class AllTests
{
[Suite]
public static IEnumerable Suite
{
get
{
ArrayList suite = new ArrayList();
suite.Add(new VisionMap.DotNet.Tests.ManagedInteropTest.DotNetUtilsTest());
return suite;
}
}
}
}
My goal is to add several tests to the list above so I can run them all in a batch.
But when I try to load the DLL in NUnit's GUI I get this:
What am I doing wrong?
I'm aware that the docs say the GUI won't run suites, but I've tried the console as well. Can somebody please tell me what Suites are good for and how I can use them to achieve my goal?
I'm using nunit 2.5.0.9122.
Edit
Well, no answers are forthcoming. I found an alternative solution in the end: Categories. I group test fixtures by giving them appropriate categories and then I can run a subset of them in batch, while still ignoring another subset.
Still, very odd that this Suite feature seems to be completely broken.
Suites aren't really needed for anything much at all these days. If you only wanted to use them to specify which tests do and don't get run this is much better achieved with Category attributes. This is what you ended up doing, and sounds like the best solution to your problem.
However, for others' and future reference, you can still use Suites in Nunit. You have to run them from the console, and only using the /fixture option. For example, to run the suite you specified above, you'd run (assuming your class was compiled into an assembly AllTests.dll):
nunit-console /fixture:AllTests.Suite AllTests.dll
You won't see any evidence of or way to run suites in the GUI - this is noted in the documentation. You can however run them from the console that is built into the GUI using commands like the above.
I use suites in some of my testing because I have some odd use cases that require me to sometimes need to pass an argument to my test methods. I do this by creating a suite such as the below. So there are some uses for them, just none needed in your case.
[Suite]
public static IEnumerable MySuite
{
get
{
var suite = new ArrayList{new TestClass1(arg1), TestClass2(arg2)};
return suite;
}
}
Is there any reason why you are returning "IEnumerable" instead of "TestSuite"?
[Suite]
public static TestSuite Suite
Update
Reading the small-print at the bottom of the page at NUnit site, it looks like Suite type tests will not show in in the Gui runner, so I guess that's the issue :)
Suites are currently not displayed in the Gui or run automatically by either runner when they are encountered. The historical purpose of the Suite mechanism was to provide a way of aggregating tests at the top level of each run. Hence, they are only supported when used with the /fixture option on the console or gui command line.
Update 2
I'm not sure what you are trying to achieve with the "Suite" feature, but if you are trying to find a way of configuring a set of test assemblies to be run together, I have used "NUnit Test Projects" to do this in the past (it's just an xml config file listing test dlls). This allows a fixed set of test assembly references to be configured and then loaded into the GUI or run by the console runner:
http://www.nunit.org/index.php?p=multiAssembly&r=2.5.5
I've got a LOT of tests written for a piece of software (which is a GREAT thing) but it was built essentially as a standalone test in C#. While this works well enough, it suffers from a few shortcomings, not the least of which is that it isn't using a standard testing framework and ends up requiring the person running the test to comment out calls to tests that shouldn't be run (when it isn't desired to run the entire test 'suite'). I'd like to incorporate it into my automated testing process.
I saw that the Test Edition of VS 2008 has the notion of a 'Generic Test' that might do what I want, but we're not in a position to spend the money on that version currently. I recently started using the VS 2008 Pro version.
These test methods follow a familiar pattern:
Do some setup for the test.
Execute the test.
Reset for the next test.
Each of them returns a bool (pass/fail) and a string ref to a fail reason, filled in if it fails.
On the bright side, at least the test methods are consistent.
I am sitting here tonight contemplating the approach I might take tomorrow morning to migrate all this test code to a testing framework and, frankly, I'm not all that excited about the idea of poring over 8-9K lines of test code by hand to do the conversion.
Have you had any experience undertaking such a conversion? Do you have any tips? I think I might be stuck slogging through it all doing global search/replaces and hand-changing the tests.
Any thoughts?
If you use NUnit (which you should), you'll need to create a new test method for each of your current test methods. NUnit uses reflection to query the test class for methods marked with the [Test] attribute, which is how it builds its list of the tests that show up in the UI, and the test classes use the NUnit Assert method to indicate whether they've passed or failed.
It seems to me that if your test methods are as consistent as you say, all of those NUnit methods would look something like this:
[Test]
public void MyTest()
{
string msg;
bool result = OldTestClass.MyTest(out msg);
if (!result)
{
Console.WriteLine(msg);
}
Assert.AreEqual(result, true);
}
Once you get that to work, your next step is to write a program that uses reflection to get all of the test method names on your old test class and produces a .cs file that has an NUnit method for each of your original test methods.
Annoying, maybe, but not extremely painful. And you'll only need to do it once.
You're about to live through the idiom of "An Ounce of Prevention is worth a pound of cure". Its especially true in programming.
You make no mention of NUnit(which I think was bought by Microsoft for 2008, but don't hold me to that). Is there a paticular reason you didn't just use NUnit in the first place?