NUnitLite is really useful for Autocad/Bricscad plugin testing, because i can load plugin, and then manually, from inside of an assembly, invoke tests new AutoRun().Execute(nunitArgs); Thanks to CADbloke github repository for tutorial on how to do this.
NUnitLite test results are saved to a xml file and can be converted to HTML (ReportUnit or ExtentReports ) and opened with browser.
Is there a way to connect NUnitLite with Visual Studio 2019 Test Explorer or to NUnit GUI, so i can automatically see results there? Or some non automatic way?
Bonus question:
Is it possible to manually invoke NUnit tests (from assembly) in other way than using NUnitLite?
Thanks
This is one of the key use cases for which NUnitLite is designed. Unfortunately, since NUnitLite is nothing more than a console application, there is no way for it to transmit test results back to TestExplorer. It would have to be enhanced to function as some sort of agent using a communications channel - not impossible but definitely non-trivial, especially when used "inside" your AutoCad plugin.
After some research on the topic, i decided to share information i have found for future readers (and maybe future me :) )
Since NUnitLite outputs .xml file with test results, my idea was to somehow load and show these results to Test Explorer in Visual Studio. I have found some articles which explain how to do it, and basically you need to create Test Adapter - it loads data from .xml file and doesn't run any tests, but returns test results (looks like mocking tests - we can't just load results into Test Explorer).
I decided not to go further for now because: 1) i am not sure how much time will be needed, 2) i will get only readonly test results, without possibility to use features of Test Explorer - and this already exists as html preview.
Here are some blogs that could be helpful:
https://devblogs.microsoft.com/devops/writing-a-visual-studio-2012-unit-test-adapter/
https://blog.dantup.com/2014/02/some-things-i-learned-while-building-my-visual-studio-test-adapter/
http://matthewmanela.com/blog/anatomy-of-the-chutzpah-test-adapter-for-vs-2012-rc/
https://blogs.msdn.microsoft.com/aseemb/2012/03/03/how-to-make-your-extension-visible-to-the-test-explorer-in-visual-studio-11/
Of course check NUnit test adapter, it is adapter that works:
https://github.com/nunit/nunit3-vs-adapter
And some forum posts, with useful information:
How to create and install test adapter in Visual Studio
https://social.msdn.microsoft.com/Forums/vstudio/en-US/0ecdcbff-4c1b-43c0-8988-6071255a4f7e/plug-custom-unit-tests-into-visual-studios-test-explorer?forum=vcgeneral
https://social.msdn.microsoft.com/Forums/vstudio/en-US/0ecdcbff-4c1b-43c0-8988-6071255a4f7e/plug-custom-unit-tests-into-visual-studios-test-explorer?forum=vcgeneral
Related
This is a common question, it's really quite astounding to me how difficult such a simple task is turning out to be ...
I have a Visual Studio (formerly Xamarin) C# cross-platform library. It has some code in the shared "abstract" library, and a few machine-specific implementation classes for Android, iOS, and UWP libraries. It also has a few automated (non-UI) unit tests that use the Microsoft.VisualStudio.TestTools.UnitTesting package. This is a great start, when Visual Studio 2017 is feeling generous it runs my unit tests and life is good.
Now I want to run the unit tests on simulators of the actual devices, from a command line (so I can run them from Jenkins and capture the output). So, for example, I want to run my unit tests on a simulator of an iPad. From various web sites I get the feeling that I can't use the Microsoft package for this, so I need to switch to either Xunit or NUnit. I gave up on Xunit earlier today, here's how NUnit went.
Go into the Unit Test project and remove the NuGet packages for Microsoft.NET.Test.Sdk, MSTest.TestFramework, and MSTest.TestAdapter.
Add NUnit instead (version 3.7.0).
Go into each CS file, change to "using NUnit.Framework", change [TestMethod] to [Test], and [TestClass] to [TestFixture].
Project:Manage NuGet Packages, add the NUnit3TestAdapter.
Test Explorer: Run All
At this point I get this weird error for the UWP machine-specific library:
The "XamlCTask" task could not be initialized with its input parameters
The "DebugType" parameter is not supported by the "XamlCTask" task. Verify the parameter exists on the task, and it is a settable public instance property.
That's a really strange message, and if I Rebuild the UWP library, it rebuilds without errors. Not sure why it's building the UWP library as part of the Unit Tests, the only thing listed under "Projects" is the abstract (non-machine-specific) library.
Also strange that building the Unit Test library works, it's only when I do the Test Explorer: Run All that I get these errors.
OK fine, it's a known error.
Close VS2017, delete bin and obj folders, reopen, no better. Close VS2017, hack the Xamarin.form.targets as listed. Same error. Realize error is in the global Xamarin.Forms.targets, not the one in my solution. Hack it. Close & reopen solution, rebuild. No error this time!
Test Explorer: Run all. Nothing. Output:Build shows no failures, Output:Tests is completely empty. Close & reopen solution. Get new errors:
[Failure] Could not find file 'C:\Workspace\PopComm.iOS\obj\Debug\TemporaryGeneratedFile_5937a670-0e60-4077-877b-f7221da3dda1.cs'.
[Failure] Could not find file 'C:\Workspace\PopComm.iOS\obj\Debug\TemporaryGeneratedFile_E7A71F73-0F8D-4B9B-B56E-8E70B10BC5D3.cs'.
[Failure] Could not find file 'C:\Workspace\PopComm.iOS\obj\Debug\TemporaryGeneratedFile_036C0B5B-1481-4323-8D20-8F5ADCB23D92.cs'.
Test Explorer: Run all. This time get more interesting errors:
------ Discover test started ------
NUnit Adapter 3.7.0.0: Test discovery starting
Assembly contains no NUnit 3.0 tests:
C:\Workspace\PopComm.iOS\bin\iPhone\Debug\PopComm.dll
NUnit Adapter 3.7.0.0: Test discovery complete
========== Discover test finished: 0 found (0:00:00.0530024) ==========
Of course that assembly doesn't contain any unit tests! It's not my unit test project!
Sigh.
OK, create a new project of type "NUnit 3 Unit Test Project". Move my test classes into this new project and add a dependency to my non-machine-specific library.
Test Explorer: Run all - Tests! There are tests! And they even run successfully.
Close VS2017, open again, Test Explorer: Run all. No tests. Output:Tests is empty.
Close project, delete all bin and obj folders, Rebuild Solution, Test Explorer: Run All. No tests.
Push code to git, pull on the Mac.
Open solution in Visual Studio for the Mac.
Use the mostly-hidden View:Test feature to bring up the Unit Tests window, run the tests. Tests found and run. Yippee!
Now I want to run my unit tests on an iPad simulator, not inside VSMac.
Add a new project to my solution, type iOS:Tests:Unified API:Unit Test App.
Wizard asks what is the minimum iOS version I want to support. I figure I should match the iOS version that my machine-specific library targets, so I go looking in the VSMac options for the project. The iOS target version isn't shown anywhere. Use the default (10.3) for the Unit Test App.
Do I have to copy all of my unit test CS files into this new app now? Or can I somehow reference my cross-platform unit test project in this Unit Test App?
Pushing everything and pulling on the Windows machine, I try adding a reference to my unit test library but get an error:
The project 'NUnit.Tests' cannot be referenced. The referenced project is targeted to a different framework family (.NETFramwork)
Hum. I wonder what framework the new NUnit.Tests.iOS application is targeting. Check its options - it doesn't say anywhere. Remove the reference from the Unit Test App back to the unit test files, and just copy the unit test source files over (non-optimal).
Run the application in the iPad simulator, it finds & runs the tests, but I don't know how to (1) run it from the command line, and (2) capture the unit test statuses in a file.
So after all this, I still don't know how to run my unit tests (1) from the command line, (2) on an iPad simulator, and (3) capturing the output in a text file.
Recently, I've been running into a perplexing issue regarding Sonar that I can't seem to figure out. I couldn't really find anything on this specific issue online, so I figured I'd reach out and see if anyone has experienced this as well.
I am using the following:
Sonar 6.0
TFS 2017 Update 1
SonarQube VSTS Extension 2.1.1
Build server with Visual Studio 2017
When I am performing Sonar analysis after a build and after tests complete, it is detecting both the resulting trx file and coverage file.
But once the analysis is done, it only reports code coverage. It does not report the unit test count.
Originally, I was using the following arguments in the "Additional Settings" in the Sonar Begin Analysis step to specifically point it to the trx:
/d:sonar.exclusions=**Scripts**,**Content** /d:sonar.cs.vscoveragexml.reportsPaths="$(Common.TestResultsDirectory)\*.coverage" /d:sonar.cs.vstest.reportsPaths="$(Common.TestResultsDirectory)\*.trx"
But I just tried removing all those arguments as I was told by a sonar rep in the past that you want to specify parameters from the site instead of by command line. When I removed my parameters, nothing changed (which I guess is to be expected). It still was able to find the trx file and still did not report unit test count.
I would try adding the vstest report path in the UI, but the issue is the path and file name are dynamically generated. The path depends on what build server/vsts agent the build is sent to, and the trx file name is generated based on date-time, and as far as I could find, there is no way to force the trx file to have a specific name.
I also checked the trx file to make sure I could open it and it reported test data.
Essentially, as far as I can tell, it should be reporting the unit test count. It did in the past but seems to have been not working as of recently. I'm not sure if this maybe was fixed in a newer version of Sonar? Unfortunately, I can't upgrade because dashboards were removed in newer versions of Sonar, and the organization I work for uses them. Has anyone experienced this issue before, and if so, were you able to fix it? Let me know if you need any additional information!
Try to use /d:sonar.cs.vstest.reportsPaths=..\**\TestResults\**\*.trx instead (Additional Settings).
On the other hand, there is SonarQube 2.1.2 aviable.
My result: https://1drv.ms/i/s!AresBGZVYryjhSpb9taJWFht7YB9
I have this strange problem. When I use the OpenCover from Visual Studio all tests pass and everything seems fine.
Image from Visual Studio here: http://i.stack.imgur.com/awmWy.png
But when I want to automate the tests using the OpenCover.Console.exe than some tests fail. I have noticed that the 2 failing tests read from config file.
Either the test or the code that is tested reads from config file using ConfigurationManager.AppSettings["someConfigValue"].
I have the same values in the main project and the test project appsettings.
The two failing tests when executing from console image here: http://i.stack.imgur.com/a77Tt.png
In addition I provide the command used for OpenCover.Console.exe.
"OpenCover.Console.exe"
-register
-target:"C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\MSTest.exe"
-targetargs:"/noisolation /testcontainer:\"E:\TEAM\TestProject.Tests\bin\Debug\TestProject.Tests.dll" /resultsfile:TestResults.trx"
-mergebyhash
-output:CodeCoverage.xml
UPDATE [04 September 2015]
Since I did not find any particular solution, I came up with a workaround.
Practically I have taken as granted that configuration file and all its data does not belong to unit tests, but to integration tests.
So in my code I do not use ConfigurationManager.AppSettings["someConfigValue"] any more. What I did is created a ConfigurationService and in the IConfigurationService I have all the methods I need for getting the values from the config file.
Having this injected in the BL of the solution, I use Mocks for unit testing purposes.
I have a test suite, comprised of both unit tests and integration tests, in a project using C# on .NET 4.0 with Visual Studio 2010. The test suite uses MSTest. When I run all tests in solution (either by hitting the button in the testing toolbar or using the Ctrl-R A shortcut chord) all of the tests, integration and unit, pass successfully.
When I either attempt to run the same tests from the command line with mstest (explicitly using the only .testsettings file present) or attempt to run them from the Test List Editor or using the .vsmdi file the integration tests fail.
The integration tests test the UI and so have dependencies on deployment items and such, whereas the unit tests do not. However, I cannot seem to pin down what is actually different between these two methods of running the tests.
When I inspect the appropriate Out directories from the test run, not all of the files are present.
What would cause some of the files that deploy correctly in one situation from Visual Studio to not deploy correctly in another?
The static content started being copied shortly after I wrote the comments above. The other major issue I ran into was that the integration test project referenced libraries that were dependencies of the system-under-test (with copy-local set to true) in order to ensure that the DLLs would be present when they were needed. For some reason, these stubbornly refused to copy when the tests were run through Test List or mstest.
What I eventually did to work around it was include [DeploymentItem] attributes for the DLLs that I needed. This got things working no matter how the tests were run. What I am still unclear on, that may have answered the underlying solution, or provided a better solution, is how Test View/mstest differ from the regular test runner (assuming that the correct .settings file was passed to mstest.). I'm putting these notes/workarounds in an answer, but I'll leave the question open in case anyone can address the underlying cause for how the different test execution paths differ.
I would like to run automated Silverlight unit tests from a Hudson build server. It seems there are two options:
Use Statlight, although it seems to be designed for TeamCity rather than Hudson, so it would involve a bit of hacking to get it to work.
Use NUnit Silverlight tests.
Can anyone recomend either of these options? Or is there a better alternative?
You can try using Lighthouse Silverlight Unit Test Runner, it works with every Build Server including Hudson, TeamCity and CCNet because it by default produces NUnit compatible xml results file:
http://lighthouse.codeplex.com/
In our company we are using NUnit with Hudson for automatized unit testing. It is simple to setup and execute.
Just download and unzip latest nunit somewhere on Hudson host.
Add Windows batch command as last buildstep with content like:
C:\NUnit\bin\net-2.0\nunit-console.exe "%WORKSPACE%\src\Test\AllTests.nunit" /config=Release /xml="%WORKSPACE%\src\Test\TestResults.xml"
This will execute tests as defined in "AllTests.nunit" file. It is possible tu point just to one assembly (.dll).
To populate test results within Hudson Job page, you would need to install Hudson NUnit plugin. Its possible directly from Hudson plugin management.
After instalation there will be new Post build action: Publish NUnit test result report.
If you check it, you've got line to enter path to test result report. Corresponding path for example above is:
src/Test/TestResults.xml
Hope it helps you to decide ;-)