Recently, I've been running into a perplexing issue regarding Sonar that I can't seem to figure out. I couldn't really find anything on this specific issue online, so I figured I'd reach out and see if anyone has experienced this as well.
I am using the following:
Sonar 6.0
TFS 2017 Update 1
SonarQube VSTS Extension 2.1.1
Build server with Visual Studio 2017
When I am performing Sonar analysis after a build and after tests complete, it is detecting both the resulting trx file and coverage file.
But once the analysis is done, it only reports code coverage. It does not report the unit test count.
Originally, I was using the following arguments in the "Additional Settings" in the Sonar Begin Analysis step to specifically point it to the trx:
/d:sonar.exclusions=**Scripts**,**Content** /d:sonar.cs.vscoveragexml.reportsPaths="$(Common.TestResultsDirectory)\*.coverage" /d:sonar.cs.vstest.reportsPaths="$(Common.TestResultsDirectory)\*.trx"
But I just tried removing all those arguments as I was told by a sonar rep in the past that you want to specify parameters from the site instead of by command line. When I removed my parameters, nothing changed (which I guess is to be expected). It still was able to find the trx file and still did not report unit test count.
I would try adding the vstest report path in the UI, but the issue is the path and file name are dynamically generated. The path depends on what build server/vsts agent the build is sent to, and the trx file name is generated based on date-time, and as far as I could find, there is no way to force the trx file to have a specific name.
I also checked the trx file to make sure I could open it and it reported test data.
Essentially, as far as I can tell, it should be reporting the unit test count. It did in the past but seems to have been not working as of recently. I'm not sure if this maybe was fixed in a newer version of Sonar? Unfortunately, I can't upgrade because dashboards were removed in newer versions of Sonar, and the organization I work for uses them. Has anyone experienced this issue before, and if so, were you able to fix it? Let me know if you need any additional information!
Try to use /d:sonar.cs.vstest.reportsPaths=..\**\TestResults\**\*.trx instead (Additional Settings).
On the other hand, there is SonarQube 2.1.2 aviable.
My result: https://1drv.ms/i/s!AresBGZVYryjhSpb9taJWFht7YB9
Related
NUnitLite is really useful for Autocad/Bricscad plugin testing, because i can load plugin, and then manually, from inside of an assembly, invoke tests new AutoRun().Execute(nunitArgs); Thanks to CADbloke github repository for tutorial on how to do this.
NUnitLite test results are saved to a xml file and can be converted to HTML (ReportUnit or ExtentReports ) and opened with browser.
Is there a way to connect NUnitLite with Visual Studio 2019 Test Explorer or to NUnit GUI, so i can automatically see results there? Or some non automatic way?
Bonus question:
Is it possible to manually invoke NUnit tests (from assembly) in other way than using NUnitLite?
Thanks
This is one of the key use cases for which NUnitLite is designed. Unfortunately, since NUnitLite is nothing more than a console application, there is no way for it to transmit test results back to TestExplorer. It would have to be enhanced to function as some sort of agent using a communications channel - not impossible but definitely non-trivial, especially when used "inside" your AutoCad plugin.
After some research on the topic, i decided to share information i have found for future readers (and maybe future me :) )
Since NUnitLite outputs .xml file with test results, my idea was to somehow load and show these results to Test Explorer in Visual Studio. I have found some articles which explain how to do it, and basically you need to create Test Adapter - it loads data from .xml file and doesn't run any tests, but returns test results (looks like mocking tests - we can't just load results into Test Explorer).
I decided not to go further for now because: 1) i am not sure how much time will be needed, 2) i will get only readonly test results, without possibility to use features of Test Explorer - and this already exists as html preview.
Here are some blogs that could be helpful:
https://devblogs.microsoft.com/devops/writing-a-visual-studio-2012-unit-test-adapter/
https://blog.dantup.com/2014/02/some-things-i-learned-while-building-my-visual-studio-test-adapter/
http://matthewmanela.com/blog/anatomy-of-the-chutzpah-test-adapter-for-vs-2012-rc/
https://blogs.msdn.microsoft.com/aseemb/2012/03/03/how-to-make-your-extension-visible-to-the-test-explorer-in-visual-studio-11/
Of course check NUnit test adapter, it is adapter that works:
https://github.com/nunit/nunit3-vs-adapter
And some forum posts, with useful information:
How to create and install test adapter in Visual Studio
https://social.msdn.microsoft.com/Forums/vstudio/en-US/0ecdcbff-4c1b-43c0-8988-6071255a4f7e/plug-custom-unit-tests-into-visual-studios-test-explorer?forum=vcgeneral
https://social.msdn.microsoft.com/Forums/vstudio/en-US/0ecdcbff-4c1b-43c0-8988-6071255a4f7e/plug-custom-unit-tests-into-visual-studios-test-explorer?forum=vcgeneral
Very common problem for many users, that SonarQube Code Analysis fails with an error:
[10:06:05]No ProjectInfo.xml files were found. Possible causes:
[10:06:05]1. The project has not been built - the end step was called right
after the begin step, without a build step in between
[10:06:05]2. An unsupported version of MSBuild has been used to build the
project. Currently MSBuild 12.0 upwards are supported
[10:06:05]3. The build step has been launched from a different working folder
[10:06:05]Post-processing failed. Exit code: 1
[10:06:05]Process exited with code 1
A lot of references say that fix is to use full path for MSBuild.exe, but yes, I use full path, but also I use MSBuild 15.0 version with newest C# version, older MSBuild just fails for new C# features in code.
However I can't get rid of this error and I have no idea what can be done about it, so maybe any of you guys have already met this problem and could help me?
EDITED
I am pretty sure that none of these steps should be the reason for the fail, the second one only the closest one, because my build steps are
Team City build steps look like the following.
Step XX:
cd %projectDirectory%
"C:\sonarqube-5.3\bin\MSBuild.SonarQube.Runner\MSBuild.SonarQube.Runner.exe" begin ... params
"C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\MSBuild\15.0\Bin\msbuild.exe" "MyProject.sln" /t:Clean;Rebuild
Step XY:
...
Step XZ: (the last)
cd %projectDirectory%
"C:\sonarqube-5.3\bin\MSBuild.SonarQube.Runner\MSBuild.SonarQube.Runner.exe" end
For me it looks very simple and easy to understand, but still I got that error and I have no idea why, the error is thrown in the last step (XZ).
You need to upgrade to a newer version of the Scanner for MSBuild. I'd suggest upgrading to the newest available version (v4.2 at the time of writing).
The scanner copies a targets file under the following location for each supported version of MBuild: %localappdata%\Microsoft\MSBuild[MSBuild version]\Microsoft.Common.targets\ImportBefore.
From the look of the log output you are using an version of the Scanner for MSBuild that pre-dates the release of MSBuild 15, so the file won't be copied to the MSBuild15-specific location. This will lead to the "no ProjectInfo.xml files were found" state.
It would be better if the warning message in the scanner log explicitly listed the range of MSBuild versions that were supported e.g. "This version of the Scanner supports MSBuild v12.0 to v14.0". I've created issue #502 to track this.
My laptop did a bluescreen when I was running some unit tests, and now nothing I can do will make it able to run tests again.
I am attempting to run tests that are programmed against NUnit 3.2 using Resharper, although Resharper reports that the problem is in NUnit. I pulled the NUnit code down from GitHub and it looks like the error is thrown if the RecentFilesService is unable to find a particular setting from the settings file. I have searched my computer for this file and cannot find it. I have reinstalled all of the NUnit packages in my project as well as reinstalled Resharper.
At this point, I am pretty much DOA. I need to figure out how to reset this file to its correct state, or else simply bypass this error.
I was able to track down the problem to the file Nunit30Settings.xml, which had become corrupted and was full of null characters. I deleted the file and everything ran fine and the file was regenerated. This file is in your [User]\AppData\Local\NUnit folder.
#drz, thank you for above information. It was very helpful. I can not comment or vote so adding your answer here
"Nunit30Settings.xml, becomes corrupted so delete the file at
[User]\AppData\Local\NUnit folder. File will be generated again."
I have been struggling with this issue. I have been able to generate the HTML reports with the help of DotCover. Also, I have given the path of the reports in the Sonar Portal too under Project Settings--> C# --> CodeCoverage --> DotCoverReportPath and also in the Sonar.Properties file for the particular solution (Mine is a DotNet application).
When I run the Sonar-Runner it seems to parse the HTML files and also says that Execution success, but however my dashboard on the Sonar System shows "No Data" for the coverage.
Please help me out if any of you have faced a similar issue and have been able to figure out the error.
SonarQube requires that the absolute paths of the source files found in the coverage HTML reports to be strictly equal to the absolute paths of the files being analyzed.
This implies that you cannot run your tests and collect code coverage on a machine A, transfer the report to machine B, and then run the SonarQube analysis from machine B. From within the same machine, you also cannot checkout the code once to run the tests & collect code coverage, and then check it out another time to run the SonarQube analysis.
This is a common case of why code coverage doesn't get imported into SonarQube, but without the logs, it is hard to tell whether this is your case or not.
We have an ASP.NET MVC 4.5 project. For months it has been compiling and all unit tests passing locally on our development machines and on the TeamCity 7.1.5 machine. Monday we updated to newer versions of some of our dependencies (specifically, to get OData 5.0.0-rc1 for select-expand).
The project compiles fine on our development machines and the build machine. It runs fine on our development machines and on both QA environments it's deployed to. All the unit tests pass on our development machines. However, about half (~300) of the unit tests now fail when run from TeamCity using the MSTest build runner, all with the same, useless error:
Unit Test Adapter threw exception:
Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information.
I've tried everything imaginable. At first I figured there was a legitimate problem. Since there's no way to retrieve the LoaderExceptions property from TeamCity (why doesn't MSTest just display more information!?!?), I logged in to the machine TeamCity is running on, opened up the console, changed into the work directory that our build is running from, and ran the tests manually so that I could retrieve the property. ALL OF THE TESTS PASSED! There goes that theory...
After an entire day of Googling and reading other answers on StackOverflow, I've tried about two dozen different combinations of the following steps:
Changing the build from "Rebuild" to "Clean," running it without tests so that it just cleans, changing it back, and running it again.
Changing the build from "Debug" to "Release," running it, and changing it back again.
Cleaning agent sources from TeamCity admin pages.
Manually deleting entire work directory off of the agents, skipping the recycle bin and going straight to the netherworlds.
Deleting all source control (TFS) caches off the build machine and agents.
Telling TeamCity to clean the work directory before getting latest from source control.
I've reached the end of my troubleshooting capabilities. The tests just won't run anymore on the build machine, for apparently no reason at all.
What can I do? What the heck is going on?
The short answer to this is, "It's a bug."
I reported the bug to JetBrains. It appears that, though there might be some underlying problem with my project, it's only a problem when run from Teamcity, and it's impossible to retrieve the actual error message, so there's no way to know what that problem is. The runner should be catching the exception and printing out the LoaderExceptions property, but it isn't. Instead, it's just calling ToString() on the exception. The runner will have to be updated to look for this specific exception and print the LoaderExceptions property. Once that happens, then we can see the underlying problem, which may result in changing something in the project, or it may result in filing another bug with JetBrains.
Workaround
I spent quite some time debugging our Continuous Integration build failure while experiencing this bug. Since Jetbrains doesn't address the issue, I thought I'd post my workaround here so others have something to work with.
My problem was that the build failed on the step which runs the unit tests. Coincidentally the build which failed added a unit test project. When the newly added unit tests were excluded, everything ran successfully. Because of the reasons stated above and Teamcity not handling the exception properly, debugging options are limited.
I never found a solution which doesn't get your hands dirty, but what you can do is the following. It's basically manually running MSTest versus the test assemblies on the build agent and does require remote desktop access to it. In my case, Teamcity ran the unit tests from the following directory:
C:\BuildAgent\temp\buildTmp[NAME TIMESTAMP]\Out.
Run the following command to initiate the MSTest runner on the assembly which contains failed tests:
C:\BuildAgent\temp\buildTmp\Out>"C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\MSTest.exe" /testcontainer:"C:\BuildAgent\temp\buildTmp[NAME TIMESTAMP]\Assembly.With.Failed.Tests.dll"
The output should give you the problem which actually makes the tests fail. In my case it turned out that the new project had a reference to Ploeh.AutoFixture with a different version than the version used in the other unit test projects.
Unable to load the test container 'C:\BuildAgent\temp\buildTmp\Out[NAME TIMESTAMP]\Assembly.With.Failed.Tests.dll' or one of its dependencies. Error details: System.IO.FileLoadException: Could not load file or assembly 'Ploeh.AutoFixture, Version=3.18.10.0, Culture=neutral, PublicKeyToken=b24654c590009d4f' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040).
Specific problem solution
In case you're wondering how to fix a similar issue, you can do the following. Open the package manager console (View -> Other windows -> NuGet Package Manager) and execute the following command.
Install-Package Ploeh.AutoFixture -Version 3.18.10.0
This adds the NuGet package (to the project selected on the dropdown) with a specific version, just make sure that all the projects are aligned in this sense.