I am working with a windows application in c#.
Using Jenkins, i created a job that will do the following tasks.
1.Build the application using msbuild
2.Tests Unit Test Cases using nunit-console.exe
3.Calculate Code coverage using NCover. (Issue)
4.Later publish the application using Nant plugin
Tasks 1, 2 and 4 works fine, while 3 having issue.
Can somebody put light on this matter?
This is the batch file that i used to find out the coverage
C:\Program Files\NCover\NCover.Console.exe" "E:\Myapp\test.exe" -h //x "E:\Newfolder\coverage.xml
The batch file is executed in Jenkins and we can see the Test.exe in task manager, what i need is the code coverage in html format when executing the Nunit test cases and no need to run my text.exe
D:\Set Up\Nuint\NUnit-2.6.2\bin\nunit-console.exe" "E:\Myapp\test.sln" /xml="E:\Newfolder\TestResult.xml
This is the batch command that i used to test the test cases, i need to know the code coverage while executing the test case, but in my case my test.exe is executed and NCover console.exe start monitor my test.exe for calculating the coverage
i tried by adding
C:\Program Files\NCover\NCover.Console.exe" infront of "D:\Set Up\Nuint\NUnit-2.6.2\bin\nunit-console.exe" "E:\Myapp\test.sln" /xml="E:\Newfolder\TestResult.xml , build succeded. and in console o/p found some coverage data like
Execution Time: 92.4688 s
Symbol Coverage: 43.72%
Branch Coverage: 22.70%
and a coverage.nccov file is created . but i need to create/show a coverage report in html format.
You can use the NCover plug-in or a post-build-task to start the calculation.
For second variant we mostly use a simple batch-file to start the action (in your case the ncover calculation). This batch-file will be called by the jenkins post-build-task.
Edit:
To get HTML you can do it via (look here):
NCover.Reporting Coverage.xml //or FullCoverageReport:Html //op "C:\Coverage Report"
Related
We use the following command line (more or less) to collect C# code coverage in our pipeline:
dotnet test --no-build -l trx -r TheResultsDir --collect "Code coverage" -s CodeCoverage.runsettings
(We actually use the build-in DotNetCoreCLI#2 task for that)
This produces a bunch of .coverage files. We want to do 2 things with them:
Send to our SonarQube server
Publish on the build itself using PublishCodeCoverageResults#1 task.
As it turns out (surprise, surprise), the produced .coverage files are only understood by VS IDE.
As usual - Internet to the rescue. We figured out that:
Using the Microsoft.CodeCoverage tool we can convert the .coverage files to .xml which are understood by SonarQube, but not the PublishCodeCoverageResults#1 task.
Using the reportgenerator tool we can convert the .xml files from (1) to the Cobertura format understood by the PublishCodeCoverageResults#1 task
And that is what we do and it is fine for small projects. However, now we introduce code coverage to our big monolithic application (sigh, helas - all is true) and the timings are awful. For one solution (out of several):
Conversion from .coverage to .xml takes about 20 minutes.
Conversion from .xml to Cobertura - 2h 33m.
So, this is really bad.
Is there a better solution if I want both send to SQ and publish to the build?
Here is the actual code we use:
From .coverage to .xml
CodeCoverage.exe analyze /output:CoverageResult.xml CoverageResult.coverage
(there are several .coverage files, so the command is applied on each one)
From .xml to Cobertura
reportgenerator.exe -reports:CoverageResults\*.xml -targetdir:CoberturaReport -reporttypes:Cobertura
Publishing to build
- task: PublishCodeCoverageResults#1
displayName: Publish Coverage Results
inputs:
codeCoverageTool: Cobertura
summaryFileLocation: CoberturaReport/Cobertura.xml
failIfCoverageEmpty: true
Publishing to SonarQube is done with the standard SQ tasks:
Prepare SQ Analysis
Run SQ Analysis
Publish SQ Analysis
The prepare task is:
- task: SonarQubePrepare#4
displayName: Prepare CI SQ Analysis
inputs:
SonarQube: SonarQube
scannerMode: MSBuild
projectKey: $(SonarQubeProjectKey)
projectName: $(SonarQubeProjectName)
projectVersion: $(SonarQubeProjectVersion)
extraProperties: |
sonar.cs.vscoveragexml.reportsPaths=$(Common.TestResultsDirectory)\vstest-coverage\*.xml
sonar.cs.nunit.reportsPaths=$(Common.TestResultsDirectory)\tests\*.TestResult.xml
sonar.inclusions=**/*.cs
sonar.branch.name=$(SonarQubeSourceBranch)
sonar.scm.disabled=true
Ideally I would like to find a way to eliminate the Cobertura conversion at all. If only it was possible to publish the .coverage files directly, that would be ideal.
This is a known issue on Azure devops. Now, AFAIK, there is no such out box way to eliminate the Cobertura conversion. The only way is using the reportgenerator tool convert the the test result to the Cobertura format at this moment.
That because the Publish Code Coverage Results task only supports coverage result formats such as Cobertura and JaCoCo. And only support the download link for .coverage files currently.
Besides, this issue has been submitted in this earlier suggestion ticket linked here:
support vstest .coverage "code coverage" build results tab
This feature request is On Roadmap, I believe it will be released soon, you can follow this thread to know its latest feedback.
"Run All" from "Test Explorer" does not complete (VS2017 Enterprise) anymore. It stalls with Passed (411), Not Run (309). The counts vary a little, usually roughly half and half.
The output window (Visual Studio | Output tab | Show output from: Tests) contains the following error message:
"The active test run was aborted. Reason: Unhandled Exception: System.AppDomainUnloadedException: Attempted to access an unloaded AppDomain."
The tests continue to run fine in ReSharper (720 of 720 pass). R# is where I usually run my tests. I jump over to Microsoft's "Test Explorer" when I am trying to Analyze Code Coverage (though the tests stall with or without code coverage). It (Analyze Code Coverage) worked as recently as 5/15/2018 (and at least a half dozen to a dozen times before that).
Check the tests output pane for more details - open the output window and select "Tests" option from the "Show output from" combo box.
In my case, my tests project was targeting a version of .NET core I didn't have installed. I just had to change the project target framework to the correct version and it solved the problem.
The Microsoft test runner was being tripped up by a single unit test class that happened to have Task.Run() calls such as the following:
var task = Task.Run(async () =>
{
<various code>
});
These tests were missing calls to task.Wait() to wait for each task to finish before exiting the test.
(This appears to trip up the Microsoft test runner but not the ReSharper test runner. Specifically, the Microsoft Test Runner aborts the sln test run and skips 300+ tests. ReSharper was able to run all tests without incident.)
Aside: The reason for the varied behavior on Windows 7 versus Windows 10 is because the test class was for a Windows 10 sensitive 3rd party control/library.
Just simple
Build -> Clear Solution
Help for me.
Upon running the vstest.console.exe command mentioned below,
vstest.console.exe C:\Products\Engineering\ACOE\EEDemo\ParallelExecution\ParallelExecution\bin\Debug\ParallelExecution.dll /Settings:C:\Products\Engineering\ACOE\EEDemo\ParallelExecution\Parallel1.runsettings /Parallel /Platform:"x64" /TestCaseFilter:"Name~Test" /Logger:TfsPublisher; Collection=http://rdtfs01:8080/tfs/Engineering; TeamProject="ACoE"; Platform="Any CPU"; Flavor="Release"; RunTitle="Sample Project"; BuildName="20170928.1"
The output obtained is
"Publish completed successfully. Test Results: mtm://rdtfs01:8080/tfs/Engineering/p:ACoE/Testing/testrun/open?id=5360."
As a next step, if we try opening the test result file using the above generated link, it externally opens the report in MTM however does not link with any of the test plan. (Please find the image attached).
However if we next close the report and open
Microsoft Test Manager >> Testing Center >> Test >> Analyze Test Runs
we do not find the report.
This is because of the test run results have not been associated with the TestPlanId in the Database ( Please find the DB screenshot attached ).
Is there a way where we can associate testplan with the output result file generated, so that if an end-user opens the
Microsoft Test Manager >> selects test plan >> Testing Center >> Test Tab >> Analyze Test Runs
will allow user to see the current results?
Seems you are using vstest.console.exe command line and use it with /logger:TfsPublisher which lets you publish the results to the tfs server against a build & test runs show up on the build page.
Published results report can be viewed in Microsoft Test Manager or through build reports from Visual Studio and Web access. Test results can be published through TfsPublisher only for the current test run. More details please refer this blog: Publishing test results through command line test runner
For tests that you run from a test plan by using Microsoft Test
Manager you can review your test results, assign a reason for a
failed test, and assign a resolution.
For tests that you run from the Visual Studio, you can save and then
reopen your test results to analyze them. You can also publish these
test results to Team Foundation Server. However, these test results
are not used in the pre-defined test reports and cannot be associated
with a test plan.
Source Link: Reviewing Test Results
Since you are using the vstest.console.exe to run the tests and publish test results to TFS, it's just using Visual Studio, so it's not able to assoicate the result with Test plan in MTM.
I am using the following batch file to try and generate code coverage results:
#echo off
SET dotnet="C:/Program Files/dotnet/dotnet.exe"
SET opencover=.\packages\OpenCover.4.6.519\tools\OpenCover.Console.exe
SET reportgenerator=.\packages\ReportGenerator.2.5.8\tools\ReportGenerator.exe
SET targetargs="test .\test\IdentityServer.UnitTests\IdentityServer.UnitTests.csproj -f net461"
SET filter="+[*]QuickstartIdentityServer* -[*.Tests]* -[xunit.*]* -[FluentValidation]*"
SET coveragefile=Coverage.xml
SET coveragedir=Coverage
REM Run code coverage analysis
%opencover% -oldStyle -register:user -target:%dotnet% -output:%coveragefile% -targetargs:%targetargs% -filter:%filter% -skipautoprops -hideskipped:All
REM Generate the report
%reportgenerator% -targetdir:%coveragedir% -reporttypes:Html;Badges -reports:%coveragefile% -verbosity:Error
REM Open the report
start "report" "%coveragedir%\index.htm"
The test seem to run fine, but I get the following error message:
Starting test execution, please wait...
Total tests: 12. Passed: 12. Failed: 0. Skipped: 0.
Test Run Successful.
Test execution time: 4.0469 Seconds
Committing...
No results, this could be for a number of reasons. The most common reasons are:
1) missing PDBs for the assemblies that match the filter please review the
output file and refer to the Usage guide (Usage.rtf) about filters.
2) the profiler may not be registered correctly, please refer to the Usage
guide and the -register switch.
Some more background on the projects:
Solution Structure:
test project:SolutionDir\test\IdentityServer.UnitTests\IdentityServer.UnitTests.csproj
project being tested:
path: SolutionDir\src\QuickstartIdentityServer\QuickstartIdentityServer.csproj
target framework: net461
Any help is appreciated!
You may also need to set DebugType to full in your .csproj file:
<DebugType>full</DebugType>
I have not used OpenCover, but successfully used coverlet
You can see a full example in this link: Cross platform code coverage arrives for .NET Core.
Basically you can add the package globally or in you project like below:
dotnet add package coverlet.msbuild
Than simple use the command:
dotnet test /p:CollectCoverage=true
I am using Win Server 2012, Developer Command Prompt for VS2013, SonarQube v5.1.2, SonarQube Scanner for MSBuild 1.1.
I have my Sonar instance running off this same server, but connected to an Oracle database. Using the default sonar runner, i can analyze projects properly and see them appear in the dashboard using this method. That cannot be said for using the recommended MSBuild.SonarQube.Runner.exe approach.
I have tried with the csharp and vbnet examples, as well as our actual code. Oddly, i want to say each of these have worked at least once, but fail to do so now. Today, I tried using the vbnet example project for the first time and discovered and it worked as intended when using the MSBuild runner. I deleted the project from the dashboard, and ran it again. At this point i do not remember it it worked or not, but the third time for sure it stopped working and began producing the error below (wherein it tries to connect to the default h2 database instead of the oracle one that SonarQube is connected to).
I tried deleting the examples folder, extracting it again, giving it a slightly different name, and running the Sonar on it giving THOSE a different name/key/version, but now it is consistently giving me the db error.
Any idea what could be going on here, and why it would go from working to not, without me making any changes to the runner, msbuild, or the code?
Running the begin step:
c:\sonar-examples-master\projects\languages\vbnet>msbuild.sonarqube.runner.exe b
egin /v:vbnet1 /k:vbnet1 /n:vbnet1
SonarQube Scanner for MSBuild 1.1
Default properties file was found at c:\sonarqube\bin\SonarQube.Analysis.xml
Loading analysis properties from c:\sonarqube\bin\SonarQube.Analysis.xml
Pre-processing started.
Preparing working directories...
Checking for updates...
MSBuild SonarQube Runner Pre-processor 1.0.2.0
17:40:49.625 Loading analysis properties from c:\sonarqube\bin\SonarQube.Analys
is.xml
17:40:49.656 Updating build integration targets...
17:40:49.656 Fetching analysis configuration settings...
17:40:50.813 Generating rulesets...
Pre-processing succeeded.
MSBuild ran, but nothing worthwhile to post from it..
Running the end step:
c:\sonar-examples-master\projects\languages\vbnet>msbuild.sonarqube.runner.exe e
nd
SonarQube Scanner for MSBuild 1.1
Default properties file was found at c:\sonarqube\bin\SonarQube.Analysis.xml
Loading analysis properties from c:\sonarqube\bin\SonarQube.Analysis.xml
Post-processing started.
MSBuild SonarQube Runner Post-processor 1.0.2.0
WARNING: File is not under the project directory and cannot currently be analyse
d by SonarQube. File: C:\Users\ts3conusr\AppData\Local\Temp\.NETFramework,Versio
n=v4.5.AssemblyAttributes.vb, project: c:\sonar-examples-master\projects\languag
es\vbnet\ConsoleApplication1\ConsoleApplication1.vbproj
The SONAR_RUNNER_HOME environment variable is not required and will be ignored.
SONAR_RUNNER_OPTS is not configured. Setting it to the default value of -Xmx1024
m
Calling the sonar-runner...
c:\sonar-examples-master\projects\languages\vbnet\.sonarqube\bin\sonar-runner\bi
n\..
SonarQube Runner 2.4
Java 1.7.0_79 Oracle Corporation (32-bit)
Windows Server 2008 R2 6.1 x86
SONAR_RUNNER_OPTS=-Xmx1024m
INFO: Error stacktraces are turned on.
INFO: Runner configuration file: c:\sonar-examples-master\projects\languages\vbn
et\.sonarqube\bin\sonar-runner\bin\..\conf\sonar-runner.properties
INFO: Project configuration file: c:\sonar-examples-master\projects\languages\vb
net\.sonarqube\out\sonar-project.properties
INFO: Default locale: "en_US", source code encoding: "UTF-8"
INFO: Work directory: c:\sonar-examples-master\projects\languages\vbnet\.sonarqu
be\out\.sonar
INFO: SonarQube Server 5.1.2
17:41:04.563 INFO - Load global repositories
17:41:04.751 INFO - Load global repositories (done) | time=188ms
17:41:04.751 INFO - Server id: 20160120154951
17:41:04.751 INFO - User cache: C:\Users\ts3conusr\.sonar\cache
17:41:04.766 INFO - Install plugins
17:41:04.860 INFO - Install JDBC driver
17:41:04.860 INFO - Create JDBC datasource for jdbc:h2:tcp://localhost/sonar
INFO: ------------------------------------------------------------------------
INFO: EXECUTION FAILURE
INFO: ------------------------------------------------------------------------
Total time: 1.047s
Final Memory: 3M/15M
INFO: ------------------------------------------------------------------------
ERROR: Error during Sonar runner execution
org.sonar.runner.impl.RunnerException: Unable to execute Sonar
Of note: I feel that it is worth mentioning that the vbnet report didn't actually have any content in it, but did at least properly get generated in the Sonar dashboard. I have tried running the commands against both that and the csharp example with now neither of them being added to the dashboard.
In the MSBuild.SonarQube.Runner install folder, there is a SonarQube.Analysis.xml. Do you have the correct server details on the below line in there?
<Property Name="sonar.host.url">http://{host}:{port}/{optionalPath}</Property>
This property value should point to your Sonar server.
Couple other things to try. Do you see tables being populated with analysis data in your oracle database? When was the last time it got any record?
Also, are you seeing below warning on your SonarQube application pages (say default dashboard)? If yes, then oracle configuration is not being picked up.
So i upgraded to the latest SonarQube version (5.3) and upgraded a few of the extensions while i was at it and preliminary testing is showing positive results. Ill try it a few more times but im closing the question for now. Thank you all that helped!