Develop Reference

How to set test category for all tests in the class

I am using MSTest, and I want to set the same test category for all methods in test class at once, without setting TestCategory attribute to each method individually. How can this be done?
The most convenient and obvious way would be to set TestCategory attribute on class, but it can be applied to methods only.
The ultimate goal is to skip integration tests during test run on TFS check-in.

To be able to set the [TestCategory] attribute at the class level, install the “MSTest V2” TestFramework using NuGet.
Ref: https://blogs.msdn.microsoft.com/devops/2016/06/17/taking-the-mstest-framework-forward-with-mstest-v2/

I've been looking to do something similar, and I've arrived at a solution that works really well for my purposes.
This does not solve the problem of applying a TestCategory on a per-class basis, but you can use the /test: command line argument for mstest to specify a search string to match any part of the fully qualified method name of the test. That means you can generally match against the class, or the namespace, or whatever search string you can arrive at that will match the target tests. And if that doesn't do it, you can use the /test: argument multiple times. I.e:
> mstest /testcontainer:My.dll /test:My.FullyQualified.Namespace
/test:My.FullyQualified.OtherNamespace.OtherClass
More Info
Edit:
Adding the TestCategory attribute at the class level is now available with MSTest V2, as noted in NomadeNumerique's answer below. Details

The ultimate goal is to skip integration tests during test run on TFS
check-in.
There are other ways to do this. In your TFS builds, you can set which unit tests you want to run, depending on their assembly name.
As the default behaviour, it will run all unit tests in assemblies which have "test" in their name. A simple fix would be to rename your integration tests to something that does not include "test".
If you do want to use the categories, you could try using AOP. For example, with Postsharp, you can create an aspect in your integration test assembly that puts the attribute on the method. Then enable the aspect for all public methods in your integration assembly if all tests are grouped in one dll or on each integration test class.

One way to get around this limitation is to put the test category in the beginning of each test method. For example, name your unit tests
public void UnitTestDoSomething_ExpectThis()
and your integration test
public void IntegrationTestDoSomething_ExpectThis()
Then when you do your TFS query to get the integration tests you can do
Field[Automated Test Name] with Operator[Contains] and Value[IntegrationTest]
Although this is not a perfect solution, it will help you distinguish your tests in code and TFS. Alternatively, you can look at area and iteration paths.

You can group by "Class name" into Test Explorer panel.
With the test TestCategory attribute you can not solve your issue just because attributes in C# are meta-data and can not be used as dynamic values.

In VS 2017 this is possible (and looks to be part of VS2012 update 1).
You can put [TestCategory("Integration")] on a class in your unit test and have it apply to all tests, and likewise [TestCategory("Unit")] on your unit test classes.
You can then use the Test Explorer's Search bar to filter by Trait name = Unit and the "Run All" will only run the tests matching your search.
When you go to run these tests on your build server, you can use a switch like /category:Unit to only run the unit tests.

NUnit base class

I am approaching database testing with NUnit. As its time consuming so I don't want to run everytime.
So, I tried creating the base class and every other database testing classes derive from it as I thought if I will decorate the base class with [Ignore] attribute then rest of the derived classes will get ignored, but thats not happening.
I need to know is there any way to Ignore set of the classes with minimal effort?

If you don't want to split out integration and unit tests into separate projects you can also group tests into categories
[Test, Category("Integration")]
Most test runners allow you to filter which categories to run which would give you finer grained control if you need it (e.g. 'quick', 'slow' and 'reaaallyy slow' categories)

A recommended approach is seperating your unit tests that can run in isolation from your integration tests into different projects, then you can choose which project to execute when you run your tests. This will make it easier to run your faster running tests more often, multiple times daily or even hourly (and hopefully without ever having to worry about such things as configuration), while letting your slower running integration tests run on a different schedule.

NUnit not running Suite tests

I've created a test suite in NUnit that references several distinct unit test fixtures in various assemblies.
I've pretty much used the example code from NUnit's docs:
namespace NUnit.Tests
{
using System;
using NUnit.Framework;
using System.Collections;
public class AllTests
{
[Suite]
public static IEnumerable Suite
{
get
{
ArrayList suite = new ArrayList();
suite.Add(new VisionMap.DotNet.Tests.ManagedInteropTest.DotNetUtilsTest());
return suite;
}
}
}
}
My goal is to add several tests to the list above so I can run them all in a batch.
But when I try to load the DLL in NUnit's GUI I get this:
What am I doing wrong?
I'm aware that the docs say the GUI won't run suites, but I've tried the console as well. Can somebody please tell me what Suites are good for and how I can use them to achieve my goal?
I'm using nunit 2.5.0.9122.
Edit
Well, no answers are forthcoming. I found an alternative solution in the end: Categories. I group test fixtures by giving them appropriate categories and then I can run a subset of them in batch, while still ignoring another subset.
Still, very odd that this Suite feature seems to be completely broken.

Suites aren't really needed for anything much at all these days. If you only wanted to use them to specify which tests do and don't get run this is much better achieved with Category attributes. This is what you ended up doing, and sounds like the best solution to your problem.
However, for others' and future reference, you can still use Suites in Nunit. You have to run them from the console, and only using the /fixture option. For example, to run the suite you specified above, you'd run (assuming your class was compiled into an assembly AllTests.dll):
nunit-console /fixture:AllTests.Suite AllTests.dll
You won't see any evidence of or way to run suites in the GUI - this is noted in the documentation. You can however run them from the console that is built into the GUI using commands like the above.
I use suites in some of my testing because I have some odd use cases that require me to sometimes need to pass an argument to my test methods. I do this by creating a suite such as the below. So there are some uses for them, just none needed in your case.
[Suite]
public static IEnumerable MySuite
{
get
{
var suite = new ArrayList{new TestClass1(arg1), TestClass2(arg2)};
return suite;
}
}

Is there any reason why you are returning "IEnumerable" instead of "TestSuite"?
[Suite]
public static TestSuite Suite
Update
Reading the small-print at the bottom of the page at NUnit site, it looks like Suite type tests will not show in in the Gui runner, so I guess that's the issue :)
Suites are currently not displayed in the Gui or run automatically by either runner when they are encountered. The historical purpose of the Suite mechanism was to provide a way of aggregating tests at the top level of each run. Hence, they are only supported when used with the /fixture option on the console or gui command line.
Update 2
I'm not sure what you are trying to achieve with the "Suite" feature, but if you are trying to find a way of configuring a set of test assemblies to be run together, I have used "NUnit Test Projects" to do this in the past (it's just an xml config file listing test dlls). This allows a fixed set of test assembly references to be configured and then loaded into the GUI or run by the console runner:
http://www.nunit.org/index.php?p=multiAssembly&r=2.5.5

where should I put my test code for my class?

So I've written a class and I have the code to test it, but where should I put that code? I could make a static method Test() for the class, but that doesn't need to be there during production and clutters up the class declaration. A bit of searching told me to put the test code in a separate project, but what exactly would the format of that project be? One static class with a method for each of the classes, so if my class was called Randomizer, the method would be called testRandomizer?
What are some best practices regarding organizing test code?
EDIT: I originally tagged the question with a variety of languages to which I thought it was relevant, but it seems like the overall answer to the question may be "use a testing framework", which is language specific. :D

Whether you are using a test framework (I highly recommend doing so) or not, the best place for the unit tests is in a separate assembly (C/C++/C#) or package (Java).
You will only have access to public and protected classes and methods, however unit testing usually only tests public APIs.
I recommend you add a separate test project/assembly/package for each existing project/assembly/package.
The format of the project depends on the test framework - for a .NET test project, use VSs built in test project template or NUnit in your version of VS doesn't support unit testing, for Java use JUnit, for C/C++ perhaps CppUnit (I haven't tried this one).
Test projects usually contain one static class init methods, one static class tear down method, one non-static init method for all tests, one non-static tear down method for all tests and one non-static method per test + any other methods you add.
The static methods let you copy dlls, set up the test environment and clear up the test enviroment, the non-static shared methods are for reducing duplicate code and the actual test methods for preparing the test-specific input, expected output and comparing them.

Where you put your test code depends on what you intend to do with the code. If it's a stand-alone class that, for example, you intend to make available to others for download and use, then the test code should be a project within the solution. The test code would, in addition to providing verification that the class was doing what you wanted it to do, provide an example for users of your class, so it should be well-documented and extremely clear.
If, on the other hand, your class is part of a library or DLL, and is intended to work only within the ecosystem of that library or DLL, then there should be a test program or framework that exercises the DLL as an entity. Code coverage tools will demonstrate that the test code is actually exercising the code. In my experience, these test programs are, like the single class program, built as a project within the solution that builds the DLL or library.
Note that in both of the above cases, the test project is not built as part of the standard build process. You have to build it specifically.
Finally, if your class is to be part of a larger project, your test code should become a part of whatever framework or process flow has been defined for your greater team. On my current project, for example, developer unit tests are maintained in a separate source control tree that has a structure parallel to that of the shipping code. Unit tests are required to pass code review by both the development and test team. During the build process (every other day right now), we build the shipping code, then the unit tests, then the QA test code set. Unit tests are run before the QA code and all must pass. This is pretty much a smoke test to make sure that we haven't broken the lowest level of functionality. Unit tests are required to generate a failure report and exit with a negative status code. Our processes are probably more formal than many, though.

In Java you should use Junit4, either by itself or (I think better) with an IDE. We have used three environments : Eclipse, NetBeans and Maven (with and without IDE). There can be some slight incompatibilities between these if not deployed systematically.
Generally all tests are in the same project but under a different directory/folder. Thus a class:
org.foo.Bar.java
would have a test
org.foo.BarTest.java
These are in the same package (org.foo) but would be organized in directories:
src/main/java/org/foo/Bar.java
and
src/test/java/org/foo/BarTest.java
These directories are universally recognised by Eclipse, NetBeans and Maven. Maven is the pickiest, whereas Eclipse does not always enforce strictness.
You should probably avoid calling other classes TestPlugh or XyzzyTest as some (old) tools will pick these up as containing tests even if they don't.
Even if you only have one test for your method (and most test authorities would expect more to exercise edge cases) you should arrange this type of structure.
EDIT Note that Maven is able to create distributions without tests even if they are in the same package. By default Maven also requires all tests to pass before the project can be deployed.

Most setups I have seen or use have a separate project that has the tests in them. This makes it a lot easier and cleaner to work with. As a separate project it's easy to deploy your code without having to worry about the tests being a part of the live system.
As testing progresses, I have seen separate projects for unit tests, integration tests and regression tests. One of the main ideas for this is to keep your unit tests running as fast as possible. Integration & regression tests tend to take longer due to the nature of their tests (connecting to databases, etc...)

I typically create a parallel package structure in a distinct source tree in the same project. That way your tests have access to public, protected and even package-private members of the class under test, which is often useful to have.
For example, I might have
myproject
src
main
com.acme.myapp.model
User
com.acme.myapp.web
RegisterController
test
com.acme.myapp.model
UserTest
com.acme.myapp.web
RegisterControllerTest
Maven does this, but the approach isn't particularly tied to Maven.

This would depend on the Testing Framework that you are using. JUnit, NUnit, some other? Each one will document some way to organize the test code. Also, if you are using continuous integration then that would also affect where and how you place your test. For example, this article discusses some options.

Create a new project in the same solution as your code.
If you're working with c# then Visual Studio will do this for you if you select Test > New Test... It has a wizard which will guide you through the process.

hmm. you want to test random number generator... may be it will be better to create strong mathematical proof of correctness of algorithm. Because otherwise, you must be sure that every sequence ever generated has a desired distribution

Create separate projects for unit-tests, integration-tests and functional-tests. Even if your "real" code has multiple projects, you can probably do with one project for each test-type, but it is important to distinguish between each type of test.
For the unit-tests, you should create a parallel namespace-hierarchy. So if you have crazy.juggler.drummer.Customer, you should unit-test it in crazy.juggler.drummer.CustomerTest. That way it is easy to see which classes are properly tested.
Functional- and integration-tests may be harder to place, but usually you can find a proper place. Tests of the database-layer probably belong somewhere like my.app.database.DatabaseIntegrationTest. Functional-tests might warrant their own namespace: my.app.functionaltests.CustomerCreationWorkflowTest.
But tip #1: be tough about separating the various kind of tests. Especially be sure to keep the collection of unit-tests separate from the integration-tests.

In the case of C# and Visual Studio 2010, you can create a test project from the templates which will be included in your project's solution. Then, you will be able to specify which tests to fire during the building of your project. All tests will live in a separate assembly.
Otherwise, you can use the NUnit Assembly, import it to your solution and start creating methods for all the object you need to test. For bigger projects, I prefer to locate these tests inside a separate assembly.
You can generate your own tests but I would strongly recommend using an existing framework.

MbUnit Parallelizable tests

I am looking at moving from NUnit to MbUnit for my unit testing framework as it has a couple of features that I like, one of them being the parallelizable attribute. If I mark tests with this attribute what happens
i, are all instance variables available only to their own thread or are they shared?
ii, how many tests will execute at once? Is it dependant on the number of processors/cores?
Reason for asking the first question is that I have, as a test simply swapped the Nunit framework for th MbUnit framework, and in a particular test class sets of tests tend to fail when run in parallel and pass when run sequentially. These test use variables at the class level and then setup in the [SetUp].

The tests run on a single instance of your test fixture class, so the instance fields will be shared.
By default, the degree of parallelism equals the number of CPUs you have, or 2 at a minimum.
You can use the DegreeOfParallelism attribute at the assembly level to override this.
See this blog post for details and some examples showing you how to use the various attributes.