I have divided my source code and tests in separate solutions for C#.
In my tests I create link to the actual code and implement stub-implementations for its dependencies where I can't mock the implementation.
If I run opencover or dotcover to get the coverage of the unit-tests and genreate a report with reportgenerator it groups the coverage per test assembly, however i want to map this coverage to the actual source code/assembly (from which i linked my file) in my resulting report.
Is there any way, in opencover or dotcover, to map my code coverage results with the actual code (and exclude my stub-classes that don't map with the code)?
I fixed this by running a small script before generating the actual report. This script replaces the ModuleName in the XML of the coverage result by OpenCover with the assembly of my production code.
ReportGenerator will then nicely merge all my results into a single report.
Related
I would like to add the code coverage report along with code analysis for my UnitTest project in SonarQube.
I had tried by adding the test files under 'sonar.test.inclusions' but the source files did not show any code coverage % nor the code smells
Project Code Coverage Report
I have a .NET solution with bunch of projects. I wan to analyze the project with SonarQube, and show unit tests details. I'm using NUnit for tests and OpenCover to generate coverage reports. I have a simple batch file to run the required SonarQube tasks in sequence.
Now I want some of the assemblies to be excluded from the coverage analysis & calculation, and do the aggregate analysis only on a subset of assemblies. I'm using opencover filters for the purpose, but they do not seem to work.
Let's say I have assemblies like
MyProject.ViewModels
MyProject.DomainModels
MyProject.Core
MyProject.CoreTests
My.Shared.Data
MyProject.Data.Tests
MyProject.Data.Console
MyShared.Console
UIModules.Services
UIModules.Services.UnitTests
UIModules.Web
UIModules.Web.Tests
And to process them I have a batch file like the following. I have added some filter to exclude the Tests, Models & Console assemblies.
Seems like the filters are not working properly. In the code coverage shown on SonarQube, though the Test projects are excluded, still the Models & Console projects are taken into account for overall coverage calculation.
SonarQube.Scanner.MSBuild.exe begin /k:"key" /n:"My Project" /v:"1.0" /d:sonar.cs.nunit.reportsPaths="NUnitResults.xml" /d:sonar.cs.opencover.reportsPaths="opencover.xml"
REM build the project/solution
msbuild MyProject.sln /t:rebuild
REM run NUnit tests
"C:\Program Files (x86)\NUnit 2.6.4\bin\nunit-console.exe" /result=NUnitResults.xml Core\bin\Debug\MyProject.CoreTests.dll Data\bin\Debug\MyProject.Data.Tests.dll Services\bin\Debug\UIModules.ServicesTests.dll Web\bin\Debug\UIModules.Web.Tests.dll
REM run OpenCover coverage
"C:\Users\Me\AppData\Local\Apps\OpenCover\OpenCover.Console.exe" -register:user "-target:C:\Program Files (x86)\NUnit 2.6.4\bin\nunit-console.exe" "-targetargs:Core\bin\Debug\MyProject.CoreTests.dll Data\bin\Debug\MyProject.Data.Tests.dll Services\bin\Debug\UIModules.ServicesTests.dll Web\bin\Debug\UIModules.Web.Tests.dll /noshadow" "-output:opencover.xml" "-filter:+[MyProject*]* +[My.Shared*]* +[UIModules*]* -[*Tests]* -[*.Console]* -[*Models]*"
REM process end analysis
SonarQube.Scanner.MSBuild.exe end
Can someone guide me to the correct direction.
Not sure where you are looking for the rules.
The coverage filter rules in the coverage tool should influence the output of the coverage.
When the coverage gets imported additional rules apply.
Sonar Qube has additional rules. And the runner can have filter rules built in as well.
Some Runners can ignore Sonar settings.
Since you are talking about coverage I assume you want to skip testing data.
For that the Testing projects names need to end in Test or Tests.
This is mentioned here:
https://docs.sonarqube.org/display/SCAN/Miscellaneous+Advanced+Usages
I also tend to exclude AssemblyInfo.cs which I set in my Sonar Host.
In Configuration -> Analysis Scope -> Global Source File Exclusions with my Pattern of **/AssemblyInfo.cs
I hope this gets you started.
I think you should leave the Other projects as not covered and list that.
If you really want to exclude per project you could try to pass additional parameters to the sonar scanner like sonar.exclusions
It is possible that sonar.exclusions is ignored by the runner.
See this answer for inspiration on different exclusion methods:
The root directory used for the resolution of the sonar.exclusions relative path is the location of each .csproj file.
Is there a way to account for the code coverage of all source code for my solution, even if I'm only including code coverage results that include a few projects? In this case if I add a new project that has no unit tests/code coverage ran against it, it doesn't get included in code coverage % value on sonarqube, so my code coverage stays the same. But I want it to consider new code not being tested. I'm using OpenCover and Xunit for a .net project.
This is addressed in SonarC# 6.6, but not in VB.NET. The plugin is currently in RC and should be released in next day or two.
There is nothing you need to configure, just use 6.6.
We use Jenkins to build C# project, to run unit tests (NUnit) and code coverage (NCover). As output, coverage.nccov and nunit-result.xml files.
Jenkins triggers SonarQube analysis (SonarQube 5.0.1 and up to date C# plugin). The SonarQube dashboard displays unit tests coverage and unit tests results, but list of failed tests cannot be displayed as drilldown.
When user clicks on the metrics, the page displayed is quite empty (no list of files, no drilldown, just the metric).
sonar-project.properties:
sonar.visualstudio.solution=MyProject.sln
sonar.cs.ncover3.reportsPaths=coverage.nccov
sonar.cs.nunit.reportsPaths=nunit-result.xml
Unit Tests Coverage metrics display drilldown as expected.
This indeed is a known limitation of the plugin, which depends on this ticket: https://jira.sonarsource.com/browse/SONARCS-657
For your information, the main difficulty to implement this feature is due to unit test reports not containing links back to the source code files, but only to assemblies/types/methods instead. SonarQube needs to know which files to show in the drilldown.
We have a very large source tree with many solutions/projects. We're attempting to improve our unit test coverage and putting together a CI environment (TFS). However, some of our projects will not generate code coverage when using VS 2015's "Analyze Code Coverage for All Tests". We simply get the "Empty results generated: No binaries were instrumented." message.
This is easily remedied on a development machine by selecting all the tests in Test Explorer, right clicking, and then selecting "Analyze Code Coverage for Selected Tests". Unfortunately, this does nothing to fix the issue in our CI environment.
In both cases the unit tests are run and I can see the pass/fail results.
I've not been able to find any rhyme or reason as to why this happens in some project and not others. We do have a runsettings file to prevent calculating coverage for the test assemblies, however, excluding it only causes the test assemblies to be included in the coverage. Coverage for the code under test is still missing.
Presumably there is a difference in how the underlying commands are run. Is there a reason why code coverage would exist under one and not the other? How can I fix my projects to make sure the coverage is calculated properly when running on the TFS server or using the "Analyze Code Coverage for All Tests" from the VS2015 interface?
Additional Notes:
Projects under test are all .NET 4.0 Class Libraries. There is some WinForms code in these projects, Forms are being excluded from coverage though.
Test projects were created using the Test Project template.
I can replicate the visual studio behavior on my projects using the command line.
Does not provide coverage:
vstest.console.exe /EnableCodeCoverage "TestAssembly.dll"
Does provide coverage:
vstest.console.exe /EnableCodeCoverage /Tests: Test1 Test2 Test3 ... TestN "TestAssembly.dll"
Additional Notes 2
I ran
vstest.console.exe /EnableCodeCoverage "TestAssembly.dll"
All unit tests are run and pass as expected.
Afterwards I ran the CodeCoverage analyze suggested at the end of this blog.
CodeCoverage.exe analyze /include_skipped_modules my.coverage > analysis.xml
"AssemblyUnderTest.dll" does not appear in either the modules section or the skipped_modules section. I know "AssemblyUnderTest.dll" is being run because I can debug the test and break into the code for that assembly and as mentioned the tests ran and passed during this test run. Other assemblies referenced by the code are present in the skipped_modules section with reason "no_symbols" - this is expected.