I'm currently running dotCover as a part of my MSTest build step in TeamCity. I've configured MSTest to run the assembly files matched by **.Tests.dll, otherwise everything else has default settings.
I have no idea how dotCover decides which files to track (I would like to know), but I'm noticing that a lot of third party libraries and the test projects themselves show up in the report - I would like to exclude this from being analyzed and reported on. How can I do that?
In TeamCity 7 you can specify assembly filters for dotCover. For example you can exclude a referenced assembly by adding a negative filter:
-:TheAssembly.To.Ignore.*
This would exclude any assemblies where the name started with TheAssembly.To.Ignore
Related
I have a solution written on c#, .netcore 2.2 and the testing framework is xunit that looks pretty much like this:
-src
--controllers
--services
-test
--controllers.integrationtest
--services.integratiotest
we are performing some integration test with two distinct dlls (controllers.integrationtest.dll & services.integrationtest) that use the same database (solutionname.test)
now, when I run "dotnet test" on the solution, it seems that the two dlls are trying to access the same resource (database) and one dll gets a deadlock.
The thing is that I want to disable the parallel execution of the dlls when I run "dotnet test" to avoid that deadlock, so I search that up and the documentation says that to disable parallel execution of dlls you have to:
Add a xunit.runner.json in the root of the csproject which I did and works fine. (this is working on visual studio because I tested it with another feature that deletes the "_" character of test names: [methodDisplay])
Configure xunit.runner.json file to copy always or preserve newest in visual studio so that gets copied on bin folder (as any appsetings.json file)
I've read that you can place an assembly attribute in the assemblyInfo.cs file that by the way it seems that was replaced by the plain csproj so I'm a bit confused.
The ultimate goal that I want to achieve is that when devops use dotnet test the integration test doesn't blow up on concurrency problems.
In xUnit the default value for ParallelizeAssemblies is false so I can only assume that a behavior of dotnet test is causing both to be executed together.
Try and run the tests using dotnet vstest. This requires the DLLS to have been built.
dotnet vstest **/*test.dll
I have a .NET solution with bunch of projects. I wan to analyze the project with SonarQube, and show unit tests details. I'm using NUnit for tests and OpenCover to generate coverage reports. I have a simple batch file to run the required SonarQube tasks in sequence.
Now I want some of the assemblies to be excluded from the coverage analysis & calculation, and do the aggregate analysis only on a subset of assemblies. I'm using opencover filters for the purpose, but they do not seem to work.
Let's say I have assemblies like
MyProject.ViewModels
MyProject.DomainModels
MyProject.Core
MyProject.CoreTests
My.Shared.Data
MyProject.Data.Tests
MyProject.Data.Console
MyShared.Console
UIModules.Services
UIModules.Services.UnitTests
UIModules.Web
UIModules.Web.Tests
And to process them I have a batch file like the following. I have added some filter to exclude the Tests, Models & Console assemblies.
Seems like the filters are not working properly. In the code coverage shown on SonarQube, though the Test projects are excluded, still the Models & Console projects are taken into account for overall coverage calculation.
SonarQube.Scanner.MSBuild.exe begin /k:"key" /n:"My Project" /v:"1.0" /d:sonar.cs.nunit.reportsPaths="NUnitResults.xml" /d:sonar.cs.opencover.reportsPaths="opencover.xml"
REM build the project/solution
msbuild MyProject.sln /t:rebuild
REM run NUnit tests
"C:\Program Files (x86)\NUnit 2.6.4\bin\nunit-console.exe" /result=NUnitResults.xml Core\bin\Debug\MyProject.CoreTests.dll Data\bin\Debug\MyProject.Data.Tests.dll Services\bin\Debug\UIModules.ServicesTests.dll Web\bin\Debug\UIModules.Web.Tests.dll
REM run OpenCover coverage
"C:\Users\Me\AppData\Local\Apps\OpenCover\OpenCover.Console.exe" -register:user "-target:C:\Program Files (x86)\NUnit 2.6.4\bin\nunit-console.exe" "-targetargs:Core\bin\Debug\MyProject.CoreTests.dll Data\bin\Debug\MyProject.Data.Tests.dll Services\bin\Debug\UIModules.ServicesTests.dll Web\bin\Debug\UIModules.Web.Tests.dll /noshadow" "-output:opencover.xml" "-filter:+[MyProject*]* +[My.Shared*]* +[UIModules*]* -[*Tests]* -[*.Console]* -[*Models]*"
REM process end analysis
SonarQube.Scanner.MSBuild.exe end
Can someone guide me to the correct direction.
Not sure where you are looking for the rules.
The coverage filter rules in the coverage tool should influence the output of the coverage.
When the coverage gets imported additional rules apply.
Sonar Qube has additional rules. And the runner can have filter rules built in as well.
Some Runners can ignore Sonar settings.
Since you are talking about coverage I assume you want to skip testing data.
For that the Testing projects names need to end in Test or Tests.
This is mentioned here:
https://docs.sonarqube.org/display/SCAN/Miscellaneous+Advanced+Usages
I also tend to exclude AssemblyInfo.cs which I set in my Sonar Host.
In Configuration -> Analysis Scope -> Global Source File Exclusions with my Pattern of **/AssemblyInfo.cs
I hope this gets you started.
I think you should leave the Other projects as not covered and list that.
If you really want to exclude per project you could try to pass additional parameters to the sonar scanner like sonar.exclusions
It is possible that sonar.exclusions is ignored by the runner.
See this answer for inspiration on different exclusion methods:
The root directory used for the resolution of the sonar.exclusions relative path is the location of each .csproj file.
I have a project with classes and methods that I want to test. I have another project with the test methods that will test the methods of my main project.
I run the tests with opencover and I generate the reports with reportgenerator, with this commands that I have in a .bet file:
..\tools\OpenCover.Console.exe -register:user -target:"C:\myDllWithTests.dll" -output:"c:\coverage\opencovertests.xml"
.\ReportGenerator.exe "-reports:c:\coverage\opencovertests.xml" "-targetdir:c:\coverage\opencovertests.xml\reports"
I am using MSTest for testing.
The problem is that in the html report, I see that the code that is covered is the tests methods, not the methods in my test main project.
How I could add the main methods in the result?
Thanks.
In target argument for OpenCover pass the path to MSTest (e.g. "C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\mstest.exe") and specify your test assemblies (e.g. "C:\myDllWithTests.dll") in targetargs argument.
To remove test assemblies from code coverage statistics, specify them in filter argument.
Below is OpenCover command that works fine for me. Here code under test is placed in SampleApp.dll and test code is placed in SampleApp.Tests.dll.
.\OpenCover.Console.exe -register:user -mergebyhash -target:"c:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\MSTest.exe" -targetargs:"/testcontainer:\"d:\test\SampleApp\SampleApp.Tests\bin\Debug\SampleApp.Tests.dll\"" -output:UTResults.xml -filter:"+[SampleApp*]* -[SampleApp.Tests]*"
Result report contains only stats for SampleApp.dll assembly, SampleApp.Tests.dll is excluded:
Check this answer for some more details.
There is also a great article by Allen Conway on using OpenCover & ReportGenerator for .Net projects.
This might be quite a late answer here, but I've spent an hour or two playing with this and found the following will fix this. It's worth noting that I had the original bat script from another project that I know works, and just changed the DLL file name, so I know the script was OK.
The additional check to make is to:-
Right click the project that has the source code you want visible in the coverage report (not the unit test project) and click Properties
Select Build > Output > Advanced
Set Debugging information to Full
Rebuild solution and re-run bat file.
Works for me in Visual Studio 2019 with .NET Framework 4.7.2 projects.
I am getting the following error message when trying to run unit tests in Visual Studio:
NUnit failed to load w:\Repos\trading.tools\Trading.Tools.Test\bin\x64\Debug\Trading.Tools.Test.dll
I am using
Visual Studio Community 2013
NUnit Adapter 3.4.0.0
NUnit 3.4.1
The weird thing is, that I have another project which is set up the same way as this one and it works just fine.
I also downloaded NUnit 3.4.1 and installed it. When I run
nunit3-console.exe Trading.Tools.Test.dll
everything works just fine.
Any ideas what I can do?
Many thanks
Konstantin
Edit #1
Here is the full console output from Visual Studio when trying to run all test.
Test run will use DLL(s) built for framework Framework45 and platform X86. Following DLL(s) will not be part of run:
Trading.Tools.Test.dll, Trading.Tools.dll are built for Framework Framework45 and Platform X64.
Go to http://go.microsoft.com/fwlink/?LinkID=236877&clcid=0x409 for more details on managing these settings.
NUnit Adapter 3.4.0.0: Test discovery starting
NUnit failed to load w:\Repos\trading.tools\Trading.Tools.Test\bin\x64\Debug\Trading.Tools.Test.dll
Assembly contains no NUnit 3.0 tests: w:\Repos\trading.tools\Trading.Tools\bin\x64\Debug\Trading.Tools.dll
NUnit Adapter 3.4.0.0: Test discovery complete
As you can see it is very obvious that NUnit expects a x86 build, but I build for a x64 platform. And again, my x64 build works just fine if I execute it using nunit3-console.exe.
What I see in the csproj file is this:
<Reference Include="nunit.framework, Version=2.6.4.14350, Culture=neutral, PublicKeyToken=96d09a1eb7f44a77, processorArchitecture=MSIL">
<SpecificVersion>False</SpecificVersion>
<HintPath>..\packages\NUnit.3.4.1\lib\net45\nunit.framework.dll</HintPath>
</Reference>
The weird thing here is that it specifies using Version=2.6.4.14350 but referencing a 3.4.1 dll.
So the next question from this point is how can I make NUnit execute my x64 build? Any ideas?
I had a similar issue, the key is the fact that it is the Test Runner in Visual Studio that is stating that only x86 assemblies will be tested. I am assuming from this that it then forces the use of the x86 NUnit runner. To change this (in VS2015 and VS2017 at least), go to Test > Test Settings > Default Processor Architecture > X64.
You can also set the execution target in the runsettings file. You then have to select that file. This should make the solution more stable.
A runsettings file which only set this can look like:
To enable it, do as shown in the figure below:
When you select it from the test menu (1), it will be added as the selected one in the menu (2), and a Rebuild will then make the test appear in the Test Explorer (3)
There is an extra bonus by using a runsettings file, and that is that it will then run properly on the TFS Build system, if you use that. I have written a blog post on that issue, see http://hermit.no/how-to-control-the-selection-of-test-runner-in-tfsvsts-making-it-work-with-x86x64-selected-targets/
I couldn't execute my tests and found that to be one of the issues. It turns out that my TestFixture was internal. Just switching it to public solved my case.
After unsuccessfully trying all other responses above, the following worked for me:
In my case the .NET project and solution is on a mounted drive (I use a MacBook and Parallels for .NET development). The mount also contains the /bin/debug and /bin/release locations where NUnit was attempting to read the "test" DLL from.
The fix was to move the solution/project files to the C: drive of my Windows image. The tests were discovered immediately.
Apparently the shared/mounted location was not to its liking. I don't know why, since the mount is permanent and read/writable to all users on the Windows image. I suspect file permissions problems or maybe somehow the entire mount is not accessible to the user/process running the NUnit discovery logic.
I happened to get this error when writing the unit test method. And noticed the root cause that one of the dependent dll was missing to load. This error("NUnit failed to load the .dll") was shown in the Output("Test") window after modifying the test method code and trying to run it. After updating the nuget package for the dependent dll, nunit started picking the test project dll and test cases got executed.
Today I was running into this issue as well (on a .NET Framework 4.8 based NUnit project). The solution for me was to also install the Microsoft.NET.Test.Sdk package.
To get to the root cause, it may help to attempt to run your tests using the NUnit CLI.
In my case, the CLI logged a bind failure I didn't see in Visual Studio. After I had fixed that, my tests ran correctly via CLI and VS.
I got this error in a .Net 6.0 Asp.Net solution and here's how I solved it.
It was only occurring in one Test project whose tests wouldn't run, the other Test projects were running fine in Test Explorer and in Debug.
The tests that were failing to be detected had "Test Not Run":
In the Output is the error:
NUnit Adapter 4.3.1.0: Test discovery starting NUnit failed to load [dll path]
No test is available in [dll path]
What I did was comment out each class and bring them back one by one until the Tests would STOP to RUN. Then with the failing class put a break point on a [Test] method.
If you can't hit the break point then its failing in this classes [SetUp]. Put a BreakPoint in the [SetUp] and use F11 to step off the edge of the method, ie F11 off the final curly brace..
AND THEN you get a prompt saying which DLL it can't load.
In my case it was “couldn’t load a DLL Microsoft.AspNetCore.Mvc.Core”
Referencing the DLL via Package Manager resolved the problem.
Edit: if this happens in a Unit Test you may want to reference the Microsoft.AspNetCore.App FrameworkReference rather than the Microsoft.AspNetCore.Mvc.Core package reference:
<ItemGroup>
<FrameworkReference Include="Microsoft.AspNetCore.App" />
</ItemGroup>
I am trying to set up unit testing and code coverage for a C# project. I am using sonar runner as the analyzer, Gallio 3.4.14 OpenCover4.5.1
This is the sonar-project.properties file:
sonar.projectKey=Foo
sonar.projectName=Foo-SonarQube Runner
sonar.projectVersion=1.0
# Path to the source directories (required)
sonar.sources=.
sonar.language=cs
sonar.sourceEncoding=UTF-8
sonar.dotnet.visualstudio.solution.file=Foo.sln
sonar.silverlight.4.mscorlib.location=C:\Program Files (x86)\Reference Assemblies
\Microsoft\Framework\Silverlight\v5.0
sonar.dotnet.excludeGeneratedCode=true
sonar.dotnet.4.0.sdk.directory=C:/WIndows/Microsoft.NET/Framework/v4.0.30319
sonar.dotnet.version=4.0
sonar.gallio.runner=IsolatedProcess
sonar.gallio.coverage.tool=OpenCover
sonar.donet.visualstudio.testProjectPattern=**.Tests; **.UnitTests
sonar.opencover.installDirectory=C:/Program Files (x86)/OpenCover
sonar.dotnet.test.assemblies=FooUnitTests\bin\Release\FooUnitTests.dll
It runs successfully but in the logs I see that Gallio didnt execute:
No assembly to check with Gendarme
Skipping the non generated assembly of project : Foo
No assembly to check with NDeps
Gallio won't execute as there are no test projects.
Any help will be appreciated
Double-check your test project patterns:
sonar.donet.visualstudio.testProjectPattern=**.Tests; **.UnitTests
Try this instead:
sonar.donet.visualstudio.testProjectPattern=*Tests;*UnitTests
Also, toward the top of the sonar-runner output (within the first 20 or so lines), you should see some lines like this:
05:40:07.021 INFO - Initializing Hibernate
05:40:13.601 INFO - Load project settings
05:40:13.653 INFO - The following 'sln' file has been found and will be used: C:\TeamCity\buildAgent\work\920b95942ca1a758\Sonar\MyProject-Sonar.sln
05:40:17.276 INFO - The project 'MyProject.Tests' has been qualified as a test project.
05:40:19.235 INFO - The project 'MyOther.Project.Tests' has been qualified as a test project.
If you're not seeing this, try running sonar-runner with -X to enable debugging output. This should give you more details as to what patterns are being used to match test projects.
Additionally, any paths need to use forward slashes (/), so the following need to be changed:
sonar.silverlight.4.mscorlib.location=C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\Silverlight\v5.0
sonar.dotnet.test.assemblies=FooUnitTests\bin\Release\FooUnitTests.dll
To this:
sonar.silverlight.4.mscorlib.location=C:/Program Files (x86)/Reference Assemblies/Microsoft/Framework/Silverlight/v5.0
sonar.dotnet.test.assemblies=FooUnitTests/bin/Release/FooUnitTests.dll