How to change Visual Studio's default MSTest, VSTest Parameters - c#

I want to be able to run tests from the Debugger in Visual Studio using MSTEST or VSTEST. The current behavior is to delete Test Run Results (TRX files) when running locally. The behavior I'm looking for is the equivalent of a build server running this command.
mstest /testcontainer:unittest.dll /resultsfile:results.trx
Because the Test Window already knows the dll, I'm guessing that we'd only need to pass in the /resultsfile switch.
Does anyone know of a way to configure Visual Studio so that the trx files are kept after the run?
I did not find anything within the TestSettings configurations that allows this.

Woops;
May have just found the answeer:
<RunSettings>
<MSTest>
<DeleteDeploymentDirectoryAfterTestRunIsComplete>False</DeleteDeploymentDirectoryAfterTestRunIsComplete>
</MSTest>
</RunSettings>

Related

How to select a tests form a test suite

I have a stage with the follow tasks:
There are 2 test suites with 8 test cases.
I want to execute only test cases with tag "download" in feature file
When I locally execute command
dotnet test <name>.dll --filter TestCategory=download
everything is fine, but when I try to add command-line options in "Other console options" in task
test cases are not filtered and all are executed.
What am I doing wrong and what can help me to filter tests?
The other console options are not honored when running from a test plan. You either need to change it to select tests using an assembly (which matches your local copy) or you need to filter the tests in the suite using a filter criteria not tied in source code.
If you expand the info of that option:
These options are not supported and will be ignored when running tests
using the ‘Multi agent’ parallel setting of an agent job or when
running tests using ‘Test plan’ option. The options can be specified
using a settings file instead.
Matt is correct. The other console options in Visual Studio test task doesn't support running from a test plan.
To solve this issue, you could use .runsettings file.
Now, versions after VS 16.6 preview 3 can support adding testcasefilter to .runsettings directly.
Here is an example:
.runsettings file
<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
<!-- Configurations that affect the Test Framework -->
<RunConfiguration>
<MaxCpuCount>1</MaxCpuCount>
<!-- Path relative to directory that contains .runsettings file-->
<ResultsDirectory>.\TestResults</ResultsDirectory>
<TestCaseFilter>TestCategory=xxx</TestCaseFilter>
</RunConfiguration>
....
Pipeline Settings:
You could install the VS 16.7.1 in Visual Studio Test Platform Installer task.
Then you could set the Test Platform version and runsettings file in the visual studio test task.
Here is a ticket about testcasefilter in runsettings file.

When I run the coverage tests, I get the coverage of the code of the tests methods, no the coverage of the methods that I want to test

I have a project with classes and methods that I want to test. I have another project with the test methods that will test the methods of my main project.
I run the tests with opencover and I generate the reports with reportgenerator, with this commands that I have in a .bet file:
..\tools\OpenCover.Console.exe -register:user -target:"C:\myDllWithTests.dll" -output:"c:\coverage\opencovertests.xml"
.\ReportGenerator.exe "-reports:c:\coverage\opencovertests.xml" "-targetdir:c:\coverage\opencovertests.xml\reports"
I am using MSTest for testing.
The problem is that in the html report, I see that the code that is covered is the tests methods, not the methods in my test main project.
How I could add the main methods in the result?
Thanks.
In target argument for OpenCover pass the path to MSTest (e.g. "C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\mstest.exe") and specify your test assemblies (e.g. "C:\myDllWithTests.dll") in targetargs argument.
To remove test assemblies from code coverage statistics, specify them in filter argument.
Below is OpenCover command that works fine for me. Here code under test is placed in SampleApp.dll and test code is placed in SampleApp.Tests.dll.
.\OpenCover.Console.exe -register:user -mergebyhash -target:"c:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\MSTest.exe" -targetargs:"/testcontainer:\"d:\test\SampleApp\SampleApp.Tests\bin\Debug\SampleApp.Tests.dll\"" -output:UTResults.xml -filter:"+[SampleApp*]* -[SampleApp.Tests]*"
Result report contains only stats for SampleApp.dll assembly, SampleApp.Tests.dll is excluded:
Check this answer for some more details.
There is also a great article by Allen Conway on using OpenCover & ReportGenerator for .Net projects.
This might be quite a late answer here, but I've spent an hour or two playing with this and found the following will fix this. It's worth noting that I had the original bat script from another project that I know works, and just changed the DLL file name, so I know the script was OK.
The additional check to make is to:-
Right click the project that has the source code you want visible in the coverage report (not the unit test project) and click Properties
Select Build > Output > Advanced
Set Debugging information to Full
Rebuild solution and re-run bat file.
Works for me in Visual Studio 2019 with .NET Framework 4.7.2 projects.

Visual Studio Test Run can't load test

There is a problem with Visual Studio 2010 test runner. Today I've loaded test results from night build on build server. Exproted this results to *.trx file, removeed from it information about test agent to run test localy. And than imported this file to VS Test Result window. After try of run failed tested, I received message One or more tests could not be found and message Test ... has not been loaded and cannot be added to the test run.
What does it mean? What is the problem?
I've created my project with tests like Class Library Project, not like Test Project. Can it be a reason of error?
PS. I tryed to add image with this error, but I need more reputation.
I can upload image to external resource, if it'll be able to help to solve my problem.
Yes you are right.
If you use MsTest and want to use Visual Studio 2010 test runner, your tests should be placed into a project created like Test Project.
There is a discussion on MSDN Forum on converting project to Test Project.
I suggest you to create new Test Project.
It is the easiest solution.
Otherwise you have to edit *.csproj file and add Guid {3AC096D0-A1C2-E12C-1390-A8335801FDAB} inside ProjectTypeGuids.
<ProjectTypeGuids>{3AC096D0-A1C2-E12C-1390-A8335801FDAB};{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}</ProjectTypeGuids>
Create new project and cut and paste files in new project
restart VS

Prevent MSTest from copying / deploying every dll

When running MSTest from Visual Studio - the unit test execution time is relatively quick.
When running MSTest from the command line, with /testsettings flag - the execution takes forever and that is because it spends 95% of its startup time copying the dll's to its Out folder. Is there a way to prevent this?
The default Local.testsettings in the project has no modifications to it (which also means it is empty). However, if I try to use that same file from the command line, MSTest complains about missing DLL's that the Unit Test reference.
Have you tried disabling deployment in the test settings? When it is disabled, the tests should be run in-place rather than on copied assemblies. (See http://msdn.microsoft.com/en-us/library/ms182475.aspx for details.)
try MSTest.exe /noisolation http://msdn.microsoft.com/en-US/library/ms182489.aspx

How do I run NUnit in debug mode from Visual Studio?

I've recently been building a test framework for a bit of C# I've been working on. I have NUnit set up and a new project within my workspace to test the component. All works well if I load up my unit tests from Nunit (v2.4), but I've got to the point where it would be really useful to run in debug mode and set some break points.
I've tried the suggestions from several guides which all suggest changing the 'Debug' properties of the test project:
Start external program: C:\Program Files\NUnit 2.4.8\bin\nunit-console.exe
Command line arguments: /assembly: <full-path-to-solution>\TestDSP\bin\Debug\TestDSP.dll
I'm using the console version there, but have tried the calling the GUI as well. Both give me the same error when I try and start debugging:
Cannot start test project 'TestDSP' because the project does not contain any tests.
Is this because I normally load \DSP.nunit into the Nunit GUI and that's where the tests are held?
I'm beginning to think the problem may be that VS wants to run it's own test framework and that's why it's failing to find the NUnit tests?
Edit: To those asking about test fixtures, one of my .cs files in the TestDSP project looks roughly like this:
namespace Some.TestNamespace
{
// Testing framework includes
using NUnit.Framework;
[TestFixture]
public class FirFilterTest
{
[Test]
public void Test01_ConstructorTest()
{
...some tests...
}
}
}
...I'm pretty new to C# and the NUnit test framework so it's entirely possible I've missed some crucial bit of information ;-)
Final Solution: The big problem was the project I'd used. If you pick Other Languages -> Visual C# -> Test -> Test Project ...when you're choosing the project type, Visual Studio will try and use it's own testing framework as far as I can tell. You should pick a normal C# class library project instead and then the instructions in my selected answer will work.
When I need to debug my NUnit tests, I simply attach to the NUnit GUI application nunit-agent.exe using "Debug|Attach to Process" and run the tests from the GUI. Any breakpoints in my tests (or the code they're testing) are hit. Am I misunderstanding your question, or will that work for you?
I use the same technique as you are trying Jon, without the /assembly flag, i.e.
Start External Program: C:\Program Files\NUnit 2.4.8\bin\nunit.exe
Command line arguments: "<path>\bin\Debug\Quotes.Domain.Tests.dll"
Does TestDSP.dll contain all your TestFixtures?
As my test project is not the startup project in the solution, I run my tests by right-clicking on the test project and choosing Debug --> Start New Instance
Simply remove the line that looks like
<ProjectTypeGuids>
{3AC096D0-A1C2-E12C-1390-A8335801FDAB};{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}
</ProjectTypeGuids>
from your project file.
This line basically tells VS.Net that it's a Test project, thus the "Cannot start test project". FYI here the 1st Guid says "it's a test", the 2nd says "it's C#".
For information on those Guids: http://www.mztools.com/Articles/2008/MZ2008017.aspx
In addition to the answer provided by #Justin here are some more details for NUnit 2.6.
Using NUnit 2.6 attach to nunit.exe or nunit-console.exe and NOT the agent. The configuration noted by #Justin is slightly different. Below is an example from nunit.exe.config (same for nunit-console.exe.config).
<startup useLegacyV2RuntimeActivationPolicy="true">
<!-- Comment out the next line to force use of .NET 4.0 -->
<supportedRuntime version="v2.0.50727" />
<supportedRuntime version="v4.0.30319" />
</startup>
For .NET 4 test project, to get break points to hit, you will have to comment out or remove the v2.0 line as the comment suggests. Once I did that I was able to debug the .NET 4.0 test project.
If you are using NUnit 2.4 or newer you can put the following code in your SetUpFixture class. (You can do this with older versions but you will need to do whatever equivalent that has to the SetUpFixture, or copy it in to the test itself.)
[SetUpFixture]
public class SetupFixtureClass
{
[SetUp]
public void StartTesting()
{
System.Diagnostics.Debugger.Launch();
}
}
What Debugger.Launch() does is cause the following dialog to show up when you click Run inside NUnit.
You then choose your running instance of visual studio with your project open (the 2nd one in my screenshot) then the debugger will be attached and any breakpoints or exceptions will show up in Visual Studio.
In Nunit 3.0.1 (I'm using VS2013), Open from main menu > Test > Windows > Test Explorer. Then in "Test explorer", right-click the test case, you might see:
Hope this helps.
Install TestDriven.NET, which is a plugin for Visual Studio
From there you can right click on your unit test assembly and click Run Tests to run the whole suite, right click on a TestFixture class to run just the tests in that class, or right click on a Test method to run just that method.
You also have the option to Test With Debugger, if you need to breakpoint into your tests in debug mode.
Try NUnitit - a open source Visual Studio Addin for Debugging NUnit Test cases
HomePage - http://nunitit.codeplex.com/
Remove ProjectTypeGuids from the project file.
Now with pictures:
Run NUnit gui (Download 2.6.2 from here) then go to File -> Open Project
Select your test .dll from bin folder (C:\......\[SolutionFolder][ProjectFolder]\bin\Debug\xxxxTests.dll)
Go to Visual Studio Debug -> Attach to process (Attach to process window will open)
From the list scroll down and select nunit-agent.exe then click Attach
At this point breakpoints in your tests should turn ripe red (from hollow).
Click Run on Nunit Gui and you should get your breakpoint hit...
Hope this saves you some time.
If you are able to get the console / or GUI working, but your breakpoints are not being hit, it may be because your app is running a different .NET runtime than NUnit is. Check to see if your nunit-console.exe.config / nunit.exe.config has the runtime specified.(The configurations live in the same directory as the nunit exe's.) Specify the runtime using the startup node:
<configuration>
<startup>
<supportedRuntime version="4.0" />
</startup>
If project path contains spaces e.g. "New Project" in path <path>\bin\Debug\New Project\Quotes.Domain.Tests.dll then enclose the Start Option --> Command Line Arguments project path in double quotes.
I spent a lot of time to figure this out.
Regarding what Mr. Patrick McDonald said
As my test project is not the startup project in the solution, I run my tests by right-clicking on the test project and choosing Debug --> Start New Instance
I tried to apply for my test class library but got some error regarding the path, so I tried to remove the 'Command Line Arguments', and luckily it worked well and as expected.
It sounds like you are trying to use the wrong library. NUnit can only start if the dll you are using contains TestFixtures.
+1 on TestDriven.Net. I've had the chance to use it a number of times.
You can download the personal version for evaluations purposes according the the license at http://testdriven.net/purchase_licenses.aspx.
I got the same error with MSTest. I found that in the Test Output window, some of the tests had duplicate IDs and could not be loaded. I removed all duplicate tests and now I was able to run the tests when i start the project.
There is also an extension now "Visual NUnit" that will allow you to run the tests from within Visual studio much like the build in test framework handles. Check it out its in the extension manager.
Open Visual Studio ---> your Project---> Select 'Properties'---> Select 'Debug' --> Select 'Start external program' and set the path of your NUnit there(Eg: Start external program = C:\Program Files\NUnit 2.6.2\bin\nunit.exe) ---->Save
After setting this just click Debug
For me solution was to adapt nunit configuration file. To use nunit with 4.5-.Net framework and x64 build option, I had to add one line to startup tag (supported runtime-version).
<startup useLegacyV2RuntimeActivationPolicy="true">
<!-- Comment out the next line to force use of .NET 4.0 -->
<supportedRuntime version="v4.0.30319" />
</startup>
Afterwards, I could start by right-click on the Testproject Debug -> Start new instance.
Before, I needed to again manually attach the project to the process.
My Debug properties were,
C:\Program Files (x86)\NUnit 2.6.4\bin\nunit.exe
with argument of the location of the .dll to be tested.
More information: nunit for testing with .NET 4.0
See if this helps..
How to add NUnit in Visual Studio
(RighteousRant)Although personally I don't like this approach.. If you need a debugger while you are test-driving your code, it's a "smell" in that you do not have enough confidence/know how your code works & need the debugger to tell you that. TDD should free you from needing a debugger if done right. Use 'Attach debugger to NUNit' only for rare cases or when you are wading in someone else's code.

Categories

Resources