so I am using the MSTest for framework for visual studio 2022 to run a few tests for QA. I want the data in a html format to easily keep track of the tests. It would show the failed test results just fine, but I want it to show which tests were ran as well, which are not displayed.
I found that the data I want is there in the html file, but for some reason MSTest would hide that data using css. How can I get MSTest to show this data? This is what it and what I actually want: https://imgur.com/a/LCW3zyR
Related
NUnitLite is really useful for Autocad/Bricscad plugin testing, because i can load plugin, and then manually, from inside of an assembly, invoke tests new AutoRun().Execute(nunitArgs); Thanks to CADbloke github repository for tutorial on how to do this.
NUnitLite test results are saved to a xml file and can be converted to HTML (ReportUnit or ExtentReports ) and opened with browser.
Is there a way to connect NUnitLite with Visual Studio 2019 Test Explorer or to NUnit GUI, so i can automatically see results there? Or some non automatic way?
Bonus question:
Is it possible to manually invoke NUnit tests (from assembly) in other way than using NUnitLite?
Thanks
This is one of the key use cases for which NUnitLite is designed. Unfortunately, since NUnitLite is nothing more than a console application, there is no way for it to transmit test results back to TestExplorer. It would have to be enhanced to function as some sort of agent using a communications channel - not impossible but definitely non-trivial, especially when used "inside" your AutoCad plugin.
After some research on the topic, i decided to share information i have found for future readers (and maybe future me :) )
Since NUnitLite outputs .xml file with test results, my idea was to somehow load and show these results to Test Explorer in Visual Studio. I have found some articles which explain how to do it, and basically you need to create Test Adapter - it loads data from .xml file and doesn't run any tests, but returns test results (looks like mocking tests - we can't just load results into Test Explorer).
I decided not to go further for now because: 1) i am not sure how much time will be needed, 2) i will get only readonly test results, without possibility to use features of Test Explorer - and this already exists as html preview.
Here are some blogs that could be helpful:
https://devblogs.microsoft.com/devops/writing-a-visual-studio-2012-unit-test-adapter/
https://blog.dantup.com/2014/02/some-things-i-learned-while-building-my-visual-studio-test-adapter/
http://matthewmanela.com/blog/anatomy-of-the-chutzpah-test-adapter-for-vs-2012-rc/
https://blogs.msdn.microsoft.com/aseemb/2012/03/03/how-to-make-your-extension-visible-to-the-test-explorer-in-visual-studio-11/
Of course check NUnit test adapter, it is adapter that works:
https://github.com/nunit/nunit3-vs-adapter
And some forum posts, with useful information:
How to create and install test adapter in Visual Studio
https://social.msdn.microsoft.com/Forums/vstudio/en-US/0ecdcbff-4c1b-43c0-8988-6071255a4f7e/plug-custom-unit-tests-into-visual-studios-test-explorer?forum=vcgeneral
https://social.msdn.microsoft.com/Forums/vstudio/en-US/0ecdcbff-4c1b-43c0-8988-6071255a4f7e/plug-custom-unit-tests-into-visual-studios-test-explorer?forum=vcgeneral
We use Jenkins to build C# project, to run unit tests (NUnit) and code coverage (NCover). As output, coverage.nccov and nunit-result.xml files.
Jenkins triggers SonarQube analysis (SonarQube 5.0.1 and up to date C# plugin). The SonarQube dashboard displays unit tests coverage and unit tests results, but list of failed tests cannot be displayed as drilldown.
When user clicks on the metrics, the page displayed is quite empty (no list of files, no drilldown, just the metric).
sonar-project.properties:
sonar.visualstudio.solution=MyProject.sln
sonar.cs.ncover3.reportsPaths=coverage.nccov
sonar.cs.nunit.reportsPaths=nunit-result.xml
Unit Tests Coverage metrics display drilldown as expected.
This indeed is a known limitation of the plugin, which depends on this ticket: https://jira.sonarsource.com/browse/SONARCS-657
For your information, the main difficulty to implement this feature is due to unit test reports not containing links back to the source code files, but only to assemblies/types/methods instead. SonarQube needs to know which files to show in the drilldown.
I am running tdd reports in Visual Studio 2015.
From what I have read I should be able to generate an html report with the specflow command line tool
specflow.exe mstestexecutionreport "Visual Studio 2015\Projects\blahSpec.csproj" /out:MyResult.html
This in turn complains that it cannot find a .trx type file.
NConsoler.NConsolerException: Could not find file 'C:\Visual Studio
2015\Projects\blahSpec\packages\SpecFlow.2.2.0\tools\TestResult.trx'.
This led me to believe that I need to set MsTest to generate a .trx file when I run the tests. The documentation I read was not specific to what I wanted to test (https://social.msdn.microsoft.com/Forums/azure/en-US/7577a33c-84bc-4144-b45f-76c8e03b0a8e/create-a-trx-file-whenever-any-tests-are-executed-from-test-explorer?forum=vsunittest)
It spoke of a settings file that I could not find, and I couldn't find any info on how to create one.
This foray in to the tall weeds made me feel that I was going about it the wrong way, and should get help here.
Meanwhile in Ruby...
SpecFlow's ruby equivalent: Cucumber, has a format flag that allows users to generate output into a nice HTML format that can be presented to higher ups.
cucumber --format html -o blah.html
What is the best way to do this in VS2015? Am I close or am I going about it the wrong way?
I am using
Specflow and Selenium Webdriver
So for those who employ SpecFlow and Selenium to test their web based application and want the report formatted in an HTML format
Navigate to ..\Projects[project name]..\bin\Debug
run MSTEST to generate a .trx file
mstest /testcontainer:[projectname].dll /ResultsFile:TestResult.trx
lastly from ..[project name]\packages\SpecFlow.[version number]\tools, run the SpecFlow command line tool to grab the .trx file and generate the report.
specflow.exe mstestexecutionreport "C:\Users\blah\Documents\Visual Studio 2015\Projects\MyProject\MyProject\MyProject.csproj" /out:MyResult.html /testresult:TestResult.trx
View file and celebrate
I have this strange problem. When I use the OpenCover from Visual Studio all tests pass and everything seems fine.
Image from Visual Studio here: http://i.stack.imgur.com/awmWy.png
But when I want to automate the tests using the OpenCover.Console.exe than some tests fail. I have noticed that the 2 failing tests read from config file.
Either the test or the code that is tested reads from config file using ConfigurationManager.AppSettings["someConfigValue"].
I have the same values in the main project and the test project appsettings.
The two failing tests when executing from console image here: http://i.stack.imgur.com/a77Tt.png
In addition I provide the command used for OpenCover.Console.exe.
"OpenCover.Console.exe"
-register
-target:"C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\MSTest.exe"
-targetargs:"/noisolation /testcontainer:\"E:\TEAM\TestProject.Tests\bin\Debug\TestProject.Tests.dll" /resultsfile:TestResults.trx"
-mergebyhash
-output:CodeCoverage.xml
UPDATE [04 September 2015]
Since I did not find any particular solution, I came up with a workaround.
Practically I have taken as granted that configuration file and all its data does not belong to unit tests, but to integration tests.
So in my code I do not use ConfigurationManager.AppSettings["someConfigValue"] any more. What I did is created a ConfigurationService and in the IConfigurationService I have all the methods I need for getting the values from the config file.
Having this injected in the BL of the solution, I use Mocks for unit testing purposes.
We are using TFS/VS 2013 Microsoft build stack to build our .NET ASP application (runs unit tests), then deploys the application based on the build to a web server which then has our API rest and Selenium UI tests ran against it.
I am getting the .coverage and .trx file, and I am able to merge the .coverage into a single .coveragexml. This is able to be processed by sonar and does display the correct results for coverage and test results, but places all the results under 'Unit Test Success or Coverage'.
Is there any way to separate out the results, i.e. Integration Tests or a widget that can display multiple test runs against a single project?
I can somewhat accomplish this by setting up multiple projects (using the same source) and attaching different .coverage and .trx to the project, but this doesn't give a clear picture of the results since the tests are all ran against the same source and bits.
I would like to display our unit tests (not all are .net), C# integration tests, API, and UI tests into separate "widgets"
Here is a copy of my sonar-project.properties file:
# Root project information
sonar.projectKey=XXX.XXX.Presentation
sonar.projectName=XXX.XXX.Presentation
sonar.projectVersion=1.0
# Enable the Visual Studio bootstrapper
sonar.visualstudio.enable=true
sonar.visualstudio.solution=XXX.XXX.sln
sonar.visualstudio.testProjectPattern=.*Tests
# Code Coverage
sonar.cs.vscoveragexml.reportsPaths=MergedResults.coveragexml
# Unit Test Results
sonar.cs.vstest.reportsPaths=TestResults/*.trx
# Some properties that will be inherited by the modules [P.S : Forward slashes]
sonar.sources=.
sonar.projectBaseDir=.
# Info required for SonarQube
sonar.language=cs
sonar.sourceEncoding=UTF-8
Integration code coverage is not yet supported by the C# plugin.
See http://jira.sonarsource.com/browse/SONARNTEST-5
Same story for integration test results:
http://jira.sonarsource.com/browse/SONARNTEST-22