I would like to run automated Silverlight unit tests from a Hudson build server. It seems there are two options:
Use Statlight, although it seems to be designed for TeamCity rather than Hudson, so it would involve a bit of hacking to get it to work.
Use NUnit Silverlight tests.
Can anyone recomend either of these options? Or is there a better alternative?
You can try using Lighthouse Silverlight Unit Test Runner, it works with every Build Server including Hudson, TeamCity and CCNet because it by default produces NUnit compatible xml results file:
http://lighthouse.codeplex.com/
In our company we are using NUnit with Hudson for automatized unit testing. It is simple to setup and execute.
Just download and unzip latest nunit somewhere on Hudson host.
Add Windows batch command as last buildstep with content like:
C:\NUnit\bin\net-2.0\nunit-console.exe "%WORKSPACE%\src\Test\AllTests.nunit" /config=Release /xml="%WORKSPACE%\src\Test\TestResults.xml"
This will execute tests as defined in "AllTests.nunit" file. It is possible tu point just to one assembly (.dll).
To populate test results within Hudson Job page, you would need to install Hudson NUnit plugin. Its possible directly from Hudson plugin management.
After instalation there will be new Post build action: Publish NUnit test result report.
If you check it, you've got line to enter path to test result report. Corresponding path for example above is:
src/Test/TestResults.xml
Hope it helps you to decide ;-)
Related
NUnitLite is really useful for Autocad/Bricscad plugin testing, because i can load plugin, and then manually, from inside of an assembly, invoke tests new AutoRun().Execute(nunitArgs); Thanks to CADbloke github repository for tutorial on how to do this.
NUnitLite test results are saved to a xml file and can be converted to HTML (ReportUnit or ExtentReports ) and opened with browser.
Is there a way to connect NUnitLite with Visual Studio 2019 Test Explorer or to NUnit GUI, so i can automatically see results there? Or some non automatic way?
Bonus question:
Is it possible to manually invoke NUnit tests (from assembly) in other way than using NUnitLite?
Thanks
This is one of the key use cases for which NUnitLite is designed. Unfortunately, since NUnitLite is nothing more than a console application, there is no way for it to transmit test results back to TestExplorer. It would have to be enhanced to function as some sort of agent using a communications channel - not impossible but definitely non-trivial, especially when used "inside" your AutoCad plugin.
After some research on the topic, i decided to share information i have found for future readers (and maybe future me :) )
Since NUnitLite outputs .xml file with test results, my idea was to somehow load and show these results to Test Explorer in Visual Studio. I have found some articles which explain how to do it, and basically you need to create Test Adapter - it loads data from .xml file and doesn't run any tests, but returns test results (looks like mocking tests - we can't just load results into Test Explorer).
I decided not to go further for now because: 1) i am not sure how much time will be needed, 2) i will get only readonly test results, without possibility to use features of Test Explorer - and this already exists as html preview.
Here are some blogs that could be helpful:
https://devblogs.microsoft.com/devops/writing-a-visual-studio-2012-unit-test-adapter/
https://blog.dantup.com/2014/02/some-things-i-learned-while-building-my-visual-studio-test-adapter/
http://matthewmanela.com/blog/anatomy-of-the-chutzpah-test-adapter-for-vs-2012-rc/
https://blogs.msdn.microsoft.com/aseemb/2012/03/03/how-to-make-your-extension-visible-to-the-test-explorer-in-visual-studio-11/
Of course check NUnit test adapter, it is adapter that works:
https://github.com/nunit/nunit3-vs-adapter
And some forum posts, with useful information:
How to create and install test adapter in Visual Studio
https://social.msdn.microsoft.com/Forums/vstudio/en-US/0ecdcbff-4c1b-43c0-8988-6071255a4f7e/plug-custom-unit-tests-into-visual-studios-test-explorer?forum=vcgeneral
https://social.msdn.microsoft.com/Forums/vstudio/en-US/0ecdcbff-4c1b-43c0-8988-6071255a4f7e/plug-custom-unit-tests-into-visual-studios-test-explorer?forum=vcgeneral
i'm looking to perform functional tests with a special app we build.
that app operates all sorts of embedded functionality and i need to be able to build test cases that perform these actions as a scenario.
we thought we could simply use the TFS API to get info and just write back test runs and their results but it proved to be difficult task to do.
so we researched the "associated automation" feature inside test cases, but it seems that i need a special framework for this. i was told only unit testing frameworks such as xunit nunit and mstest can be integrated.
i need functional testing, scenarios that are more complicated than a unit test.
do u have any ideas? on how can i simply run my own tests and update the TFS with runs that i created?
If your test tool can generate a JUnit, XUnit or TRX compatible file that contains the test results, then the data can be ingested using the "Publish Test Results" task in your build and/or release pipeline.
If you want a wrapper around any executable, then the "Generic Tests" feature of MsTest may also be an option. These configure how to run your executable and then point to a result file for reporting purposes. A sample result file is shown here in the docs.
<?xml version="1.0" encoding="utf-8" ?>
<SummaryResult>
<TestName>ParentTest</TestName>
<TestResult>Passed</TestResult>
<InnerTests>
<InnerTest>
<TestName>InnerTest1</TestName>
<TestResult>Passed</TestResult>
<ErrorMessage>Everything is fine.</ErrorMessage>
<DetailedResultsFile>D:\Documents and Settings\Results.txt</DetailedResultsFile>
</InnerTest>
<InnerTest>
<TestName>InnerTest2</TestName>
<TestResult>Failed</TestResult>
<ErrorMessage>Something went wrong.</ErrorMessage>
<DetailedResultsFile>D:\Documents and Settings\Results.txt</DetailedResultsFile>
</InnerTest>
</InnerTests>
</SummaryResult>
Alternatively, test results can be created directly through the REST API:
First create a test run
Then create test results
This is a common question, it's really quite astounding to me how difficult such a simple task is turning out to be ...
I have a Visual Studio (formerly Xamarin) C# cross-platform library. It has some code in the shared "abstract" library, and a few machine-specific implementation classes for Android, iOS, and UWP libraries. It also has a few automated (non-UI) unit tests that use the Microsoft.VisualStudio.TestTools.UnitTesting package. This is a great start, when Visual Studio 2017 is feeling generous it runs my unit tests and life is good.
Now I want to run the unit tests on simulators of the actual devices, from a command line (so I can run them from Jenkins and capture the output). So, for example, I want to run my unit tests on a simulator of an iPad. From various web sites I get the feeling that I can't use the Microsoft package for this, so I need to switch to either Xunit or NUnit. I gave up on Xunit earlier today, here's how NUnit went.
Go into the Unit Test project and remove the NuGet packages for Microsoft.NET.Test.Sdk, MSTest.TestFramework, and MSTest.TestAdapter.
Add NUnit instead (version 3.7.0).
Go into each CS file, change to "using NUnit.Framework", change [TestMethod] to [Test], and [TestClass] to [TestFixture].
Project:Manage NuGet Packages, add the NUnit3TestAdapter.
Test Explorer: Run All
At this point I get this weird error for the UWP machine-specific library:
The "XamlCTask" task could not be initialized with its input parameters
The "DebugType" parameter is not supported by the "XamlCTask" task. Verify the parameter exists on the task, and it is a settable public instance property.
That's a really strange message, and if I Rebuild the UWP library, it rebuilds without errors. Not sure why it's building the UWP library as part of the Unit Tests, the only thing listed under "Projects" is the abstract (non-machine-specific) library.
Also strange that building the Unit Test library works, it's only when I do the Test Explorer: Run All that I get these errors.
OK fine, it's a known error.
Close VS2017, delete bin and obj folders, reopen, no better. Close VS2017, hack the Xamarin.form.targets as listed. Same error. Realize error is in the global Xamarin.Forms.targets, not the one in my solution. Hack it. Close & reopen solution, rebuild. No error this time!
Test Explorer: Run all. Nothing. Output:Build shows no failures, Output:Tests is completely empty. Close & reopen solution. Get new errors:
[Failure] Could not find file 'C:\Workspace\PopComm.iOS\obj\Debug\TemporaryGeneratedFile_5937a670-0e60-4077-877b-f7221da3dda1.cs'.
[Failure] Could not find file 'C:\Workspace\PopComm.iOS\obj\Debug\TemporaryGeneratedFile_E7A71F73-0F8D-4B9B-B56E-8E70B10BC5D3.cs'.
[Failure] Could not find file 'C:\Workspace\PopComm.iOS\obj\Debug\TemporaryGeneratedFile_036C0B5B-1481-4323-8D20-8F5ADCB23D92.cs'.
Test Explorer: Run all. This time get more interesting errors:
------ Discover test started ------
NUnit Adapter 3.7.0.0: Test discovery starting
Assembly contains no NUnit 3.0 tests:
C:\Workspace\PopComm.iOS\bin\iPhone\Debug\PopComm.dll
NUnit Adapter 3.7.0.0: Test discovery complete
========== Discover test finished: 0 found (0:00:00.0530024) ==========
Of course that assembly doesn't contain any unit tests! It's not my unit test project!
Sigh.
OK, create a new project of type "NUnit 3 Unit Test Project". Move my test classes into this new project and add a dependency to my non-machine-specific library.
Test Explorer: Run all - Tests! There are tests! And they even run successfully.
Close VS2017, open again, Test Explorer: Run all. No tests. Output:Tests is empty.
Close project, delete all bin and obj folders, Rebuild Solution, Test Explorer: Run All. No tests.
Push code to git, pull on the Mac.
Open solution in Visual Studio for the Mac.
Use the mostly-hidden View:Test feature to bring up the Unit Tests window, run the tests. Tests found and run. Yippee!
Now I want to run my unit tests on an iPad simulator, not inside VSMac.
Add a new project to my solution, type iOS:Tests:Unified API:Unit Test App.
Wizard asks what is the minimum iOS version I want to support. I figure I should match the iOS version that my machine-specific library targets, so I go looking in the VSMac options for the project. The iOS target version isn't shown anywhere. Use the default (10.3) for the Unit Test App.
Do I have to copy all of my unit test CS files into this new app now? Or can I somehow reference my cross-platform unit test project in this Unit Test App?
Pushing everything and pulling on the Windows machine, I try adding a reference to my unit test library but get an error:
The project 'NUnit.Tests' cannot be referenced. The referenced project is targeted to a different framework family (.NETFramwork)
Hum. I wonder what framework the new NUnit.Tests.iOS application is targeting. Check its options - it doesn't say anywhere. Remove the reference from the Unit Test App back to the unit test files, and just copy the unit test source files over (non-optimal).
Run the application in the iPad simulator, it finds & runs the tests, but I don't know how to (1) run it from the command line, and (2) capture the unit test statuses in a file.
So after all this, I still don't know how to run my unit tests (1) from the command line, (2) on an iPad simulator, and (3) capturing the output in a text file.
I have NUnit test project in my solution. I'd like to be able to run its tests on a set of different environments using a network of virtual machines and then aggregate result XMLs from each machine to my server.
I guess I would need to install NUnit to all my VMs in order to do it that way. But maybe I can avoid it somehow? Hypotetically I can change my test assembly's output type from class library to console app and call all my test methods (those ones with right NUnit attributes) using reflection manually from Main method and write XML output myself.
So I'd like to know if there are any testing framework that can make it for me. Ideally I would like it to create standalone executable that I will be able to run on a bare installed system, with no need to install some additional testing software. This executable will run all its tests and generate some test result output (it better be NUnit-compatible, but it's fine if it's not, I can handle some post processing).
Any ideas how I can achieve this?
Thanks in advance.
I have a test suite, comprised of both unit tests and integration tests, in a project using C# on .NET 4.0 with Visual Studio 2010. The test suite uses MSTest. When I run all tests in solution (either by hitting the button in the testing toolbar or using the Ctrl-R A shortcut chord) all of the tests, integration and unit, pass successfully.
When I either attempt to run the same tests from the command line with mstest (explicitly using the only .testsettings file present) or attempt to run them from the Test List Editor or using the .vsmdi file the integration tests fail.
The integration tests test the UI and so have dependencies on deployment items and such, whereas the unit tests do not. However, I cannot seem to pin down what is actually different between these two methods of running the tests.
When I inspect the appropriate Out directories from the test run, not all of the files are present.
What would cause some of the files that deploy correctly in one situation from Visual Studio to not deploy correctly in another?
The static content started being copied shortly after I wrote the comments above. The other major issue I ran into was that the integration test project referenced libraries that were dependencies of the system-under-test (with copy-local set to true) in order to ensure that the DLLs would be present when they were needed. For some reason, these stubbornly refused to copy when the tests were run through Test List or mstest.
What I eventually did to work around it was include [DeploymentItem] attributes for the DLLs that I needed. This got things working no matter how the tests were run. What I am still unclear on, that may have answered the underlying solution, or provided a better solution, is how Test View/mstest differ from the regular test runner (assuming that the correct .settings file was passed to mstest.). I'm putting these notes/workarounds in an answer, but I'll leave the question open in case anyone can address the underlying cause for how the different test execution paths differ.