I was browsing for a while the internet and this site and instead of finding some ways to unit test my existing code the only finding was to separate logic and interaction with the user (MVC approach). Although this is great for new projects it is time-consuming and as a result too expensive to invest for existing ones. Is there a way to create specific unit tests, ideally automated, for existing GUI projects that unfortunately connect directly to databases or other systems to get data and the data are manipulated before it is shown? Currently we have two projects the one being MFC, the other C# .net 2.0 Thanks a lot.
Unit testing won't cut in here considering you can't change your existing code (not to mention you don't really unit test UI). You should look for some kind of GUI testing automation/scripting tools. Like Sikuli. Quoting literally the first paragraph from their website:
Sikuli is a visual technology to automate and test graphical user interfaces (GUI) using images (screenshots).
It doesn't get any simplier than that. You "tell" the tool which parts of your UI it should observe/interact, it records and replays it. Skimming through this presentation will give you idea of what exactly you can do (might also check their video). Probably won't solve all your problems, but might be alternative worth considering.
Unit testing an already existing project is always a challenge. however i point you to some open source tools that will help you to automate unit testing
C++
Boost unit test framework
Google Mock
C#
NUnit
NMock
It is possible to produce some level of automated testing, but not unit tests.
Unit tests, by definition, test small units of logic decoupled from the system as a whole. I'd recommend new code be written in the way you described (mvc etc) to be unit testable.
With your existing code, unit testing will obviously require refactoring, which I appreciate is not in your timeframe. You will need to work with what you've got an look at a way to perform more whole-system automated testing, probably driven through the UI. The fact these are not Unit tests is by-the-by, there are helpful tests to have even if you have unit tests. Its helpful to know the distinction though when you are searching for resources.
You are probably best searching for automated ui testing. With the .net apps, you may find something like White useful
If you are lucky enough to have a Premium (at least) version of Visual Studio 2010, then you could consider writing Coded UI Tests.
A UI Test is basically an automated sequence of actions (mouse, keyboard...) on a GUI. These are very high level tests (or functional tests), not unit tests, but it can help testing a GUI application that is already existing.
For instance you can easily automate CRUD actions (which imply a database) and check (assert) that actions have produce the expected result in the UI (new created item in a list...).
Write UI testing can be very time consuming because there are various aspect you had to test. Thanks God there are a lot of frameworks to achieve that result but you always have to write some code.
I assume you already did unit testing (Visual Studio itself comes with a not so bad unit test framework) so what you want to check is not algorithms but UI automation/results. What does it mean? Everything that is code must be tested by code (database operations and algorithms, for example). Even some UI controls can be somehow tested by code (example: if I simulate a user click I'll get that event fired when this condition is true). Trust me, UI testing is Black Art and often you'll get failed tests even if everything is OK.
Simple stress scenario
For a simple scenario, for example to stress your application to reproduce a bug repeating the same operation many times, you can use a macro recorder (such as WinMacro). You register user inputs and then you run that macro in a loop. If there's a subtle bug you have many chances to reproduce (and/or to find) it when that actions are repeated 5000 times in a night. That done you'll get data from your logs.
Simple scenario
If your application can be somehow automated (it may be easy for .NET application using VSA) you can prepare some "good" macro to automate an operation, put results in a file and compare them with a known good results data file.
Simple tip: for MFC application you can write your own "macro" with a text file where each line is a Windows message with its parameter; read it, parse it and SendMessage() them to your application to simulate user inputs, menu clicks and so on. Grab - for example - text box value and compare with something known. WinSpy++ is your friend.
Complex scenario
For anything else (is my custom control drawing everything in the right way? when user click that button then UI colors changes?) you have to use a more complex tool. There are several tools to automate UI testing, built-in in Visual Studio 2010 (not in every edition) what you need to create coded UI tests. What does it mean? You write code to automate your application and then you write more code to check its results (sometimes even comparing bitmaps with known results. It may be tedious and a lot but virtually you can test everything, even if the application hasn't been designed for UI testing.
Start reading this from MSDN.
There is a plethora of commercial tools too (even if I never used it in any project) so I not write any link, I guess you'll have a lot of results in Google.
Mocking is usually the best approach for simulating integration points, but unfortunately most Mocking frameworks fall short if the code is too interconnected and bury dependencies in private methods etc.
It might be worth looking into a framework called Moles (made by Microsoft) that have very few limitations for what you can Mock. It even handles private methods!
Perhaps you could use it to mock your db calls to test your data manipulation?
There are several tutorials online.
Here's one that might get you started:
http://www.unit-testing.net/CurrentArticle/How-To-Mock-Code-Dependencies-Using-Moles.html
In order to reduce code duplication I would like to generate unit tests programatically from various sources. One simple way I could think of was to generate a whole bunch of delegates from within a method that parses some configuration information and tags all those delegates with the [TestMethod] attribute which are then run by the visual studio test framework.
My motivation is to use as much of the visual studio's test reporting facilities as possible because I could write my own reporting layer for the tests by using some of C#'s reflection facilities but I rather not. My solution seems pretty elegant and simple but I can't get visual studio's test framework to understand what exactly I'm trying to do so does anyone know how to go about doing what I would like?
You may want to consider Pex (and possibly Moles for legacy code) from Microsoft Research. Pex is a research project that automatically generates unit tests with high code coverage, and supposedly picks interesting input and outputs for the tests. I guess there's some intelligence there. :)
I have not personally used it, but I have heard from some peers that Pex and Moles is pretty interesting and has helped them out. May be worth a look.
Hope this helps!
I think if you actually want to generate tests, you'd be best off literally generating the tests, using something like T4 templates. In other words, use code-generation and create the test fixtures and methods such that you run them as you would any other test case.
I'm sorry if this does not answer you question exactly but I think that auto-generating unit tests is the wrong way to tackle this problem.
Unit tests are an effective way to help avoid regression and provide documentation to how your code should be used. When you automatically generate your tests you get a bunch of tests but if they fail you don't really know if a real bug exist and you end up with a lot of failing tests that you have no idea why they fail or how to make them pass. At the end you would probably delete all of the tests - because they "don't work".
Having said that - try out Armadillo from Typemock - it's a tool that writes the tests according to the user's actions
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
I am going to be starting up a new project at work and want to get into unit testing. We will be using Visual Studio 2008, C#, and the ASP.NET MVC stuff. I am looking at using either NUnit or the built-in test projects that Visual Studio 2008 has, but I am open to researching other suggestions. Is one system better than the other or perhaps easier to use/understand than the other?
I am looking to get this project set up as kind of the "best practice" for our development efforts going forward.
Daok named all the pro's of Visual Studio 2008 test projects. Here are the pro's of NUnit.
NUnit has a mocking framework.
NUnit can be run outside of the
IDE. This can be useful if you want
to run tests on a non-Microsoft build server,
like CruiseControl.NET.
NUnit has more versions coming out
than visual studio. You don't have
to wait years for a new version.
And you don't have to install a new version of the IDE to
get new features.
There are extensions being developed
for NUnit, like row-tests, etc.
Visual Studio tests take a long time
to start up for some reason. This is
better in Visual Studio 2008,
but it is still too slow
for my taste. Quickly running a test
to see if you didn't break something
can take too long. NUnit with
something like Testdriven.Net to run
tests from the IDE is actually much
faster. Especially when running
single tests.
According to Kjetil Klaussen, this is caused by the Visual Studio testrunner. Running MSTest tests in TestDriven.Net makes MSTest performance comparable to NUnit.
The unit-testing framework doesn't actually matter much, because you can convert test classes with separate project files and conditional compilation (like this, Visual Studio → NUnit):
#if !NUNIT
using Microsoft.VisualStudio.TestTools.UnitTesting;
#else
using NUnit.Framework;
using TestClass = NUnit.Framework.TestFixtureAttribute;
using TestMethod = NUnit.Framework.TestAttribute;
using TestInitialize = NUnit.Framework.SetUpAttribute;
using TestCleanup = NUnit.Framework.TearDownAttribute;
using TestContext = System.String;
using DeploymentItem = NUnit.Framework.DescriptionAttribute;
#endif
The TestDriven.Net plugin is nice and not very expensive... With only plain Visual Studio 2008 you have to find the test from your test class or test list. With TestDriven.Net you can run your test directly from the class that you are testing. After all, unit tests should be easy to maintain and near the developer.
Benefits/changes of the Visual Studio 2008 built-in unit testing framework:
The 2008 version now is available in
professional editions (before it
required expensive versions of Visual Studio, and
this is just for developer unit
testing) that left a lot of
developers with the only choice of
open/external testing frameworks.
Built-in API supported by a single company.
Use the same tools to to run and create tests (you may run them using the command line also MSTest).
Simple design (granted without a mock framework, but this is a great starting point for many programmers).
Long term support granted (I still remember what happened to NDoc, and I don't want to commit to a testing framework that might not be supported in five years, but I still consider NUnit a great framework).
If using Team Foundation Server as your backend, you can create work items or bugs with the failed test data in a simple fashion.
I have been using NUnit for two years. All is fine, but I have to say that the unit testing system in Visual Studio is pretty nice, because it's inside the GUI and can more easily do a test for private function without having to mess around.
Also, the unit testing of Visual Studio lets you do covering and other stuff that NUnit alone can't do.
One slight annoyance of Visual Studio's testing framework is that it will create many test run files that tend to clutter your project directory - though this isn't that big of a deal.
Also, if you lack a plugin such as TestDriven.NET, you cannot debug your NUnit (or MbUnit, xUnit, etc.) unit tests within the Visual Studio environment, as you can with the Microsoft Visual Studio testing framework, which is built in.
Slightly off-topic, but if you go with NUnit I can recommend using ReSharper - it adds some buttons to the Visual Studio UI that make it a lot easier to run and debug tests from within the IDE.
This review is slightly out-of-date, but explains this in more detail:
Using ReSharper as an essential part of your TDD toolkit
xUnit is another possibility for a greenfield project. It's got perhaps a more intuitive syntax, but it is not really compatible with the other frameworks.
My main beef with Visual Studio unit tests over NUnit is the Visual Studio test creation tends to inject a bunch of generated code for private member access.
Some might want to test their private methods, and some may not. That's a different topic.
My concern is when I'm writing unit tests they should be extremely controlled so I know exactly what I'm testing and exactly how I'm testing it. If there's auto generated code I'm losing some of that ownership.
I have done some TDD using both and (maybe I'm a little dumb) NUnit seems to be a lot faster and simpler to use to me. And when I say a lot, I mean a lot.
In MSTest, there is too many attributes, everywhere - the code that do the real tests is the tiny lines you may read here and there. A big mess. In NUnit, the code that do the test just dominates the attributes, as it should do.
Also, in NUnit, you just have to click on the tests you want to run (only one? All the tests covering a class? An assembly? The solution?). One click. And the window is clear and large. You get clear green and red lights. You really know what happens in one sight.
In VSTS, the test list is jammed in the bottom of the screen, and it's small and ugly. You have to look twice to know what happened. And you cannot run just one test (well, I did not find out yet!).
But I may be wrong, of course - I just read about 21 blog posts about "How to do simple TDD using VSTS". I should have read more; you are right.
For NUnit, I read one. And I was TDDing the same day. With fun.
By the way, I usually love Microsoft products. Visual Studio is really the best tool a developer can buy - but TDD and Work Item management in Visual Studio Team System sucks, really.
First I want to correct a wrong statement: you can run MSTest outside of Visual Studio using the command line. Although several CI tools, such as TeamCity, have better support for NUnit (probably would change as MSTest becomes more popular).
In my current project we use both and the only big difference we found that MSTest always runs as a 32 bit while NUnit runs as either 32 bit or 64 bit tests which only matters if your code uses native code that is 32/64 bit dependent.
I got messages that "NUnit file structure is richer than VSTest"...
Of course, if you prefer the NUnit file structure, you can use this solution to the other way, like this (NUnit → Visual Studio):
#if !MSTEST
using NUnit.Framework;
#else
using Microsoft.VisualStudio.TestTools.UnitTesting;
using TestFixture = Microsoft.VisualStudio.TestTools.UnitTesting.TestClassAttribute;
using Test = Microsoft.VisualStudio.TestTools.UnitTesting.TestMethodAttribute;
using SetUp = Microsoft.VisualStudio.TestTools.UnitTesting.TestInitializeAttribute;
using TearDown = Microsoft.VisualStudio.TestTools.UnitTesting.TestCleanupAttribute;
#endif
Or any other conversion... :-) This use here is just an alias to the compiler.
I started with MSTest, but I switched for one simple reason. MSTest does not support inheritance of test methods from other assemblies.
I hated the idea of writing the same test multiple times. Especially on a large project where test methods can easily run into 100's of tests.
NUnit does exactly what I need. The only thing that is missing with NUnit is a Visual Studio addin which has can display the red/green status (like VSTS) of each test.
If you are considering either MSTest or NUnit, then I recommend you look at MbUnit. My reasons are
TestDriven.Net compatibility. Nothing beats have TestDriven.Net.ReRunWithDebugger bound to a keyboard combination.
The Gallio framework. Gallio is a test runner like NUnit's. The only difference is it doesn't care if you wrote your tests in NUnit, MSTest, xUnit or MbUnit. They all get run.
Compatibility with NUnit. All features in NUnit are supported by MbUnit. I think you don't even need to change your attributes (will have to check that), just your reference and usings.
Collection asserts. MbUnit has more Assert cases, including the CollectionAssert class. Basically you no longer need to write your own tests to see if two collections are the same.
Combinatorial tests. Wouldn't it be cool if you could supply two sets of data and get a test for all the combinations of data? It is in MbUnit.
I originally picked up MbUnit because of its [RowTest ....] functionality, and I haven't found a single reason to go back. I moved all my active test suites over from NUnit and never looked back. Since then I've converted two different development teams over to the benefits.
As far as I know, there are four frameworks available for unit testing with .NET these days:
NUnit
MbUnit
MSTest
xUnit
NUnit has always been out in front, but the gap has closed in the last year or so. I still prefer NUnit myself, especially as they added a fluent interface a while back which makes tests very readable.
If you're just getting started with unit testing it probably doesn't make much difference. Once you're up to speed, you'll be in a better position to judge which framework is best for your needs.
I don't like the Visual Studio's built-in testing framework, because it forces you to create a separate project as opposed to having your tests as part of the project you're testing.
MSTest is essentially NUnit slightly reworked, with a few new features (such as assembly setup and teardown, not just fixture and test level), and missing some of the best bits (such as the new 2.4 constraint syntax). NUnit is more mature, and there is more support for it from other vendors; and of course since it's always been free (whereas MSTest only made it into the Professional version of Visual Studio 2008, and before that it was in way more expensive SKUs), and most ALT.NET projects use it.
Having said that, there are some companies who are incredibly reluctant to use something which does not have the Microsoft label on it, and especially so OSS code. So having an official Microsoft test framework may be the motivation that those companies need to get testing; and let's be honest, it's the testing that matters, not what tool you use (and using Tuomas Hietanen's code, you can almost make your test framework interchangeable).
With the release in .NET 4.0 of the Code Contracts system and the availability of a static checker, you would need to theoretically write fewer test cases and a tool like Pex will help identify those cases. Relating this to the discussion at hand, if you need to do less with your unit tests because your contracts are covering your tail, then why not just go ahead and use the built-in pieces since that is one less dependency to manage. These days, I am all about simplicity. :-)
See also:
Microsoft Pex – Automated Unit Testing
Unit Tests generation with Pex using Visual Studio 2010 and C# 4.0
I would prefer to use MS's little test framework, but for now am sticking with NUnit. The problems with MS's are generally (for me)
Shared "tests" file (pointless) that must be maintained
Tests lists cause conflicts with multiple developers / VCSs
Poor integrated UI - confusing setup, burdensome test selection
No good external runner
Caveats
If I were testing an aspx site, I would definitely use MS's
If I were developing solo, also MS would be fine
If I had limited skill and couldn't configure NUnit :)
I find it much easier to just write my tests and fire up NUnitGUI or one of the other front ends (testDriven is far far far far overpriced). Setting up debugging with the commandline version is also pretty easy.
I'm looking to introduce a unit testing framework into the mix at my job. We're using Visual Studio 2005 (though we may be moving to 2008 within the next six months) and work primarily in C#. If the framework has some kind of IDE integration that would be best, but I'm open to frameworks that don't have integration but are still relatively simple to get set up. I'm going to get resistance to it one way or another, so if I can make sure what I'm pushing isn't a pain in the neck, that would help my case.
The obvious choice from the research I've done so far points to NUnit, but I'd like to get the impressions of someone who's actually used it before recommending it to my team.
Has anyone out there used NUnit? If so, are there any pitfalls or limitations of which I should be aware? Are there other good options out there? If so, if you've used both NUnit at that, I'd greatly appreciate an idea of the strengths and weaknesses of them.
I think NUnit is your best bet. With TestDriven.NET, you get great integration within Visual Studio. (ReSharper also has a unit test runner if you're using it). NUnit is simple to use and follows an established paradigm. You'll also find plenty of projects, tutorials, and guides using it which always helps.
Your other main choice is probably MbUnit, which is more and more positioning itself as the BDD framework of choice (in conjunction with Gallio).
Scott Hanselman had a good podcast about this, entitled:
"The Past, Present and Future of .NET Unit Testing Frameworks"
:
Hanselminutes #112
Visual Studio 2008 has a built-in test project type that works in a similar way to NUnit, but obviously has much tighter integration with Visual Studio (can run on every build and shows the results in a similar way to the conversion results page when upgrading solution files), but it is obviously not as mature as NUnit as it's pretty new and I'm not sure about how it handles mocking.
But it would be worth looking into when your team moves to Visual Studio 2008.
The built-in unit testing in Visual Studio 2008 is all right, but its difficult to integrate with CruiseControl.NET, certainly a lot harder than normal NUnit.
So go with NUnit if you plan to have nice automated tests.
We've been using xUnit.net. It seems to combine all the best of NUnit, MbUnit, and MSTest.
When I started unit testing, I started with NUnit as it is simple to set up and use. Currently I am using the built-in test runner that comes with ReSharper. That way, I can easily flip between code and test results.
Incidentally, NUnit detects when you have compiled your code, so you do not need to do any refresh in NUnit. ReSharper automatically does a build when you choose to run a specific test.
Try also the PEX tool.
It's Microsoft's own, probably soon to be integrated into VSTS.
It does support NUnit, MbUnit and xUnit.net.
I also use a small console application for testing one class or a small library. You could copy paste the code from here.
VSTT 2010 (Visual Studio Team System Test) should be a good bet if you are looking for functional test automation. Web services testing, UI testing, BizTalk testing and data-driven testing support. Please look at VSTT.
MbUnit is worth a look. It has a set of features comparable to NUnit. It has its own GUI, or can be integrated into Visual Studio if you have ReSharper. I would also recommend Rhino Mocks if you are doing any sort of TDD.
I would say MbUnit also. I like being able to run a single test many times just by specifying inputs and the result is right above the test function. It is a horrible description of what I mean, so here is a link that shows you what I mean.