I have a stage with the follow tasks:
There are 2 test suites with 8 test cases.
I want to execute only test cases with tag "download" in feature file
When I locally execute command
dotnet test <name>.dll --filter TestCategory=download
everything is fine, but when I try to add command-line options in "Other console options" in task
test cases are not filtered and all are executed.
What am I doing wrong and what can help me to filter tests?
The other console options are not honored when running from a test plan. You either need to change it to select tests using an assembly (which matches your local copy) or you need to filter the tests in the suite using a filter criteria not tied in source code.
If you expand the info of that option:
These options are not supported and will be ignored when running tests
using the ‘Multi agent’ parallel setting of an agent job or when
running tests using ‘Test plan’ option. The options can be specified
using a settings file instead.
Matt is correct. The other console options in Visual Studio test task doesn't support running from a test plan.
To solve this issue, you could use .runsettings file.
Now, versions after VS 16.6 preview 3 can support adding testcasefilter to .runsettings directly.
Here is an example:
.runsettings file
<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
<!-- Configurations that affect the Test Framework -->
<RunConfiguration>
<MaxCpuCount>1</MaxCpuCount>
<!-- Path relative to directory that contains .runsettings file-->
<ResultsDirectory>.\TestResults</ResultsDirectory>
<TestCaseFilter>TestCategory=xxx</TestCaseFilter>
</RunConfiguration>
....
Pipeline Settings:
You could install the VS 16.7.1 in Visual Studio Test Platform Installer task.
Then you could set the Test Platform version and runsettings file in the visual studio test task.
Here is a ticket about testcasefilter in runsettings file.
I want to be able to run tests from the Debugger in Visual Studio using MSTEST or VSTEST. The current behavior is to delete Test Run Results (TRX files) when running locally. The behavior I'm looking for is the equivalent of a build server running this command.
mstest /testcontainer:unittest.dll /resultsfile:results.trx
Because the Test Window already knows the dll, I'm guessing that we'd only need to pass in the /resultsfile switch.
Does anyone know of a way to configure Visual Studio so that the trx files are kept after the run?
I did not find anything within the TestSettings configurations that allows this.
Woops;
May have just found the answeer:
<RunSettings>
<MSTest>
<DeleteDeploymentDirectoryAfterTestRunIsComplete>False</DeleteDeploymentDirectoryAfterTestRunIsComplete>
</MSTest>
</RunSettings>
I have this strange problem. When I use the OpenCover from Visual Studio all tests pass and everything seems fine.
Image from Visual Studio here: http://i.stack.imgur.com/awmWy.png
But when I want to automate the tests using the OpenCover.Console.exe than some tests fail. I have noticed that the 2 failing tests read from config file.
Either the test or the code that is tested reads from config file using ConfigurationManager.AppSettings["someConfigValue"].
I have the same values in the main project and the test project appsettings.
The two failing tests when executing from console image here: http://i.stack.imgur.com/a77Tt.png
In addition I provide the command used for OpenCover.Console.exe.
"OpenCover.Console.exe"
-register
-target:"C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\MSTest.exe"
-targetargs:"/noisolation /testcontainer:\"E:\TEAM\TestProject.Tests\bin\Debug\TestProject.Tests.dll" /resultsfile:TestResults.trx"
-mergebyhash
-output:CodeCoverage.xml
UPDATE [04 September 2015]
Since I did not find any particular solution, I came up with a workaround.
Practically I have taken as granted that configuration file and all its data does not belong to unit tests, but to integration tests.
So in my code I do not use ConfigurationManager.AppSettings["someConfigValue"] any more. What I did is created a ConfigurationService and in the IConfigurationService I have all the methods I need for getting the values from the config file.
Having this injected in the BL of the solution, I use Mocks for unit testing purposes.
I am stuck trying to automate unit tests runs with MSTest and deployment of app.config. I read multiple posts and blogs, tried multiple things and yet still app.config doesn't seem to be picked up during MSTest execution. Having a dll that contains all my unit tests built with msbuild, here is what I've tried...
Attempt 1
Copied app.config to the same location with MyTests.dll as MyTests.dll.config (on one of the msdn forums it was said it'd be picked up automagically)
Added [DeploymentItem("MyTests.dll.config")] attribute to each test
Ran MSTest.exe /noisolation /testcontainer:d:\MyTestTests.dll /test:MyTest
Attempt 2
Created local.testrunconfig file with the following content (below)
Ran mstest with /runconfig and no isolation, but nothing was executed: MSTest.exe /runconfig:d:\local.testrunconfig /testcontainer:d:\MyTestTests.dll /test:MyTest
Result:
Loading d:\local.testrunconfig...
d:\local.testrunconfig
d:\local.testrunconfig
... and nothing happens: no errors, no tests are executed!
EDIT/RESOLUTION: By default, MSTest executes tests in separate processes. In this case, config file is automatically picked up if it is named like "dllname.dll.config". However, it is hard to debug tests running in separate processes if they run outside of VS. /noisolation switch is used to make MSTest run all tests in one process. However, in this case test config file is NOT picked up. Instead, MSTest.exe.config file is used, which is located in the same directory as MSTest. To resolve this issue, configuration file can be loaded pragmatically like this:
ExeConfigurationFileMap map = new ExeConfigurationFileMap();
map.ExeConfigFilename = #"path to config file";
Configuration config =
ConfigurationManager.OpenMappedExeConfiguration(map, ConfigurationUserLevel.None);
Kateroh,
My setup looks like this (I'm using msbuild from a TFSbuild.proj):
Build sln
Copy all from output to %TEMP% (black magic :D)
Don't know why but mstest is looking for references in %TEMP%.
Run mstest with parms:
"C:\Program Files\Microsoft Visual Studio 10.0\Common7\IDE\MSTest.exe" /nologo /testcontainer:$(TestDir)\mylib.dll /resultsfile:$(TestResultFile) /runconfig:$(SlnDir)\AutomaticBuildTest.testrunconfig /searchpathroot:$(TestDir) /publish:mytfsserver /publishbuild:$(BuildDefinition) /flavor:Debug /platform:AnyCPU /teamproject:mytfsproject
where AutomaticBuildTest.testrunconfig is bellow
<?xml version="1.0" encoding="UTF-8"?>
<TestSettings name="AutomaticBuildTest" id="eda99352-93e1-402e-9517-d04fffa66b35" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
<!--<Deployment enabled="false" />-->
<Deployment enabled="true">
<DeploymentItem filename="D:\sa12\78\bin\Debug" />
</Deployment>
<NamingScheme baseName="BC2ALibraryTest" appendTimeStamp="false" useDefault="false" />
<!-- http://blogs.msdn.com/b/vstsqualitytools/archive/2009/12/01/executing-unit-tests-in-parallel-on-a-multi-cpu-core-machine.aspx -->
<Execution location="Local" hostProcessPlatform="MSIL">
<!--http://msdn.microsoft.com/en-us/library/ms404663.aspx-->
<ExecutionThread apartmentState="MTA" />
<Hosts skipUnhostableTests="false" />
<TestTypeSpecific>
<UnitTestRunConfig testTypeId="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b">
<AssemblyResolution applicationBaseDirectory="D:\sa12\78\bin\Debug">
<TestDirectory useLoadContext="false" />
</AssemblyResolution>
</UnitTestRunConfig>
</TestTypeSpecific>
<AgentRule name="LocalMachineDefaultRole">
</AgentRule>
</Execution>
</TestSettings>
Your application doesn't use app.config. It uses application.exe.config. That's what you need to deploy.
Sorry, didn't see you were testing a DLL.
This is not how .NET configuration works. Each DLL does not use its own config file. It can only use the config file of the .exe it is running in (actually, of the AppDomain it's running in).
You would somehow need to get MSTEST to add your .dll.config to its own configuration, assuming it's actually the host process. I don't see how to do that from the command line.
I usually use a Unit Test Project in Visual Studio, so I just deploy the config file from that project. Works fine.
The problem is, as it turns out, with our over-complicated build environment and the fact that we are using and x-copiable version of MSTest (produced locally). The following command succeeded when I ran against VS2008 "proper" MSTest:
"%ProgramFiles%\Microsoft Visual Studio 9.0\Common7\IDE\MSTest.exe" /testcontainer:d:\MyTests.dll /test:MyTests /resultsfile:results.trx
Thanks everyone for the answers! The check goes to you, Marius, you made me learn new things with your tesrunconfig.
I've recently been building a test framework for a bit of C# I've been working on. I have NUnit set up and a new project within my workspace to test the component. All works well if I load up my unit tests from Nunit (v2.4), but I've got to the point where it would be really useful to run in debug mode and set some break points.
I've tried the suggestions from several guides which all suggest changing the 'Debug' properties of the test project:
Start external program: C:\Program Files\NUnit 2.4.8\bin\nunit-console.exe
Command line arguments: /assembly: <full-path-to-solution>\TestDSP\bin\Debug\TestDSP.dll
I'm using the console version there, but have tried the calling the GUI as well. Both give me the same error when I try and start debugging:
Cannot start test project 'TestDSP' because the project does not contain any tests.
Is this because I normally load \DSP.nunit into the Nunit GUI and that's where the tests are held?
I'm beginning to think the problem may be that VS wants to run it's own test framework and that's why it's failing to find the NUnit tests?
Edit: To those asking about test fixtures, one of my .cs files in the TestDSP project looks roughly like this:
namespace Some.TestNamespace
{
// Testing framework includes
using NUnit.Framework;
[TestFixture]
public class FirFilterTest
{
[Test]
public void Test01_ConstructorTest()
{
...some tests...
}
}
}
...I'm pretty new to C# and the NUnit test framework so it's entirely possible I've missed some crucial bit of information ;-)
Final Solution: The big problem was the project I'd used. If you pick Other Languages -> Visual C# -> Test -> Test Project ...when you're choosing the project type, Visual Studio will try and use it's own testing framework as far as I can tell. You should pick a normal C# class library project instead and then the instructions in my selected answer will work.
When I need to debug my NUnit tests, I simply attach to the NUnit GUI application nunit-agent.exe using "Debug|Attach to Process" and run the tests from the GUI. Any breakpoints in my tests (or the code they're testing) are hit. Am I misunderstanding your question, or will that work for you?
I use the same technique as you are trying Jon, without the /assembly flag, i.e.
Start External Program: C:\Program Files\NUnit 2.4.8\bin\nunit.exe
Command line arguments: "<path>\bin\Debug\Quotes.Domain.Tests.dll"
Does TestDSP.dll contain all your TestFixtures?
As my test project is not the startup project in the solution, I run my tests by right-clicking on the test project and choosing Debug --> Start New Instance
Simply remove the line that looks like
<ProjectTypeGuids>
{3AC096D0-A1C2-E12C-1390-A8335801FDAB};{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}
</ProjectTypeGuids>
from your project file.
This line basically tells VS.Net that it's a Test project, thus the "Cannot start test project". FYI here the 1st Guid says "it's a test", the 2nd says "it's C#".
For information on those Guids: http://www.mztools.com/Articles/2008/MZ2008017.aspx
In addition to the answer provided by #Justin here are some more details for NUnit 2.6.
Using NUnit 2.6 attach to nunit.exe or nunit-console.exe and NOT the agent. The configuration noted by #Justin is slightly different. Below is an example from nunit.exe.config (same for nunit-console.exe.config).
<startup useLegacyV2RuntimeActivationPolicy="true">
<!-- Comment out the next line to force use of .NET 4.0 -->
<supportedRuntime version="v2.0.50727" />
<supportedRuntime version="v4.0.30319" />
</startup>
For .NET 4 test project, to get break points to hit, you will have to comment out or remove the v2.0 line as the comment suggests. Once I did that I was able to debug the .NET 4.0 test project.
If you are using NUnit 2.4 or newer you can put the following code in your SetUpFixture class. (You can do this with older versions but you will need to do whatever equivalent that has to the SetUpFixture, or copy it in to the test itself.)
[SetUpFixture]
public class SetupFixtureClass
{
[SetUp]
public void StartTesting()
{
System.Diagnostics.Debugger.Launch();
}
}
What Debugger.Launch() does is cause the following dialog to show up when you click Run inside NUnit.
You then choose your running instance of visual studio with your project open (the 2nd one in my screenshot) then the debugger will be attached and any breakpoints or exceptions will show up in Visual Studio.
In Nunit 3.0.1 (I'm using VS2013), Open from main menu > Test > Windows > Test Explorer. Then in "Test explorer", right-click the test case, you might see:
Hope this helps.
Install TestDriven.NET, which is a plugin for Visual Studio
From there you can right click on your unit test assembly and click Run Tests to run the whole suite, right click on a TestFixture class to run just the tests in that class, or right click on a Test method to run just that method.
You also have the option to Test With Debugger, if you need to breakpoint into your tests in debug mode.
Try NUnitit - a open source Visual Studio Addin for Debugging NUnit Test cases
HomePage - http://nunitit.codeplex.com/
Remove ProjectTypeGuids from the project file.
Now with pictures:
Run NUnit gui (Download 2.6.2 from here) then go to File -> Open Project
Select your test .dll from bin folder (C:\......\[SolutionFolder][ProjectFolder]\bin\Debug\xxxxTests.dll)
Go to Visual Studio Debug -> Attach to process (Attach to process window will open)
From the list scroll down and select nunit-agent.exe then click Attach
At this point breakpoints in your tests should turn ripe red (from hollow).
Click Run on Nunit Gui and you should get your breakpoint hit...
Hope this saves you some time.
If you are able to get the console / or GUI working, but your breakpoints are not being hit, it may be because your app is running a different .NET runtime than NUnit is. Check to see if your nunit-console.exe.config / nunit.exe.config has the runtime specified.(The configurations live in the same directory as the nunit exe's.) Specify the runtime using the startup node:
<configuration>
<startup>
<supportedRuntime version="4.0" />
</startup>
If project path contains spaces e.g. "New Project" in path <path>\bin\Debug\New Project\Quotes.Domain.Tests.dll then enclose the Start Option --> Command Line Arguments project path in double quotes.
I spent a lot of time to figure this out.
Regarding what Mr. Patrick McDonald said
As my test project is not the startup project in the solution, I run my tests by right-clicking on the test project and choosing Debug --> Start New Instance
I tried to apply for my test class library but got some error regarding the path, so I tried to remove the 'Command Line Arguments', and luckily it worked well and as expected.
It sounds like you are trying to use the wrong library. NUnit can only start if the dll you are using contains TestFixtures.
+1 on TestDriven.Net. I've had the chance to use it a number of times.
You can download the personal version for evaluations purposes according the the license at http://testdriven.net/purchase_licenses.aspx.
I got the same error with MSTest. I found that in the Test Output window, some of the tests had duplicate IDs and could not be loaded. I removed all duplicate tests and now I was able to run the tests when i start the project.
There is also an extension now "Visual NUnit" that will allow you to run the tests from within Visual studio much like the build in test framework handles. Check it out its in the extension manager.
Open Visual Studio ---> your Project---> Select 'Properties'---> Select 'Debug' --> Select 'Start external program' and set the path of your NUnit there(Eg: Start external program = C:\Program Files\NUnit 2.6.2\bin\nunit.exe) ---->Save
After setting this just click Debug
For me solution was to adapt nunit configuration file. To use nunit with 4.5-.Net framework and x64 build option, I had to add one line to startup tag (supported runtime-version).
<startup useLegacyV2RuntimeActivationPolicy="true">
<!-- Comment out the next line to force use of .NET 4.0 -->
<supportedRuntime version="v4.0.30319" />
</startup>
Afterwards, I could start by right-click on the Testproject Debug -> Start new instance.
Before, I needed to again manually attach the project to the process.
My Debug properties were,
C:\Program Files (x86)\NUnit 2.6.4\bin\nunit.exe
with argument of the location of the .dll to be tested.
More information: nunit for testing with .NET 4.0
See if this helps..
How to add NUnit in Visual Studio
(RighteousRant)Although personally I don't like this approach.. If you need a debugger while you are test-driving your code, it's a "smell" in that you do not have enough confidence/know how your code works & need the debugger to tell you that. TDD should free you from needing a debugger if done right. Use 'Attach debugger to NUNit' only for rare cases or when you are wading in someone else's code.