I would like to retrieve the list of tests from the test explorer in a visual studio 2012 extension.
Is this possible using the Package.GetGlobalService or via DTE?
[Edit]
Since this is getting no response I'll provide some more info.
Currently IDE extensions or plugins, that wish to run solution tests do so by providing their own runners. Gallio is one such example of a test runner that has plugin to run multiple types of tests. Resharper is an example of an IDE extension that includes a test runner
VS2012 is introducing a new Test Runner which 3rd party test libraries can develop against such as nunit, xunit, qunit. With this change, it is possible to run these different types of tests within VS itself, namely via the 'Test Explorer' window.
For 3rd parties to interact with the IDE tests, it is necessary to be able to get a list of the tests, run the tests and receive the output of the tests.
So with that in mind, is there any access to the test interfaces in the VS 2012 IDE?
This looks like a good starting point:
API Reference for Testing Tools for Visual Studio ALM
Related
I'm using Selenium with C# to create my automated tests. Now i have to use some sort of reporting tools to save the test cases whether they passed,failed...etc.
I've seen a lot of tools like Allure, and Jenkins. But they require an XML file. and i can't find it when i run my tests using Visual 2013. Why is that ?
how to obtain these XML files in details if i'm missing something ?
Is there any way i could achieve this with minimum effort ?
EDIT:
How can i run my tests using Nunit console runner ? where can i find it? i downloaded the .zip for nunit but i couldn't find the runner?
The NUnit Visual Studio Adapter does not currently produce XML results, although there is an issue on GitHub to add the ability. Your best bet is to run your tests using the NUnit Console runner when you want to create reports. It always produces an XML result file.
You can install the NUnit Console by adding the NUnit.Console NuGet package to your test project. The console will be in a directory like packages\NUnit.ConsoleRunner.3.4.0\tools in your solution root.
Another option is to install the MSI for the console runner. It will then be under C:\Program Files (x86)\NUnit.org\nunit-console
For creating reports, ReportUnit creates excellent HTML reports from your test results.
To run the tests, I would recommend setting up a simple command line build that builds your solution, runs the tests and then produces the report. For day to day development, just running your tests in Visual Studio or on the command line will likely be enough. Most developers find the Visual Studio Test Explorer Window to be a poor UI, but usable for seeing passed and failed tests and running/debugging them.
If you want to setup a command line build, one good tool is Cake Build. It will take a bit of time to setup, but it is an excellent way to run your build tasks as your project gets larger.
NUnit testing can be run with the nunit-console.exe application which is installed with nunit under {Project_root}/lib/nunit/nunit-console.exe. It downloads with NuGet when NUnit installs.
It can be passed a list of testing binaries, or testing project files, or an nunit project (listing multiple if needed).
{PathToProject}\lib\nunit\nunit-console.exe {PathToTestDll}\Project1.Tests.dll {PathToTestDll}\Project2.Tests.dll /xml=nunit-result.xml
or create an NUnit Project with the NUnit Project Editor if you want to group all your test projects into a single config file.
{PathToProject}\lib\nunit\nunit-console.exe {PathToNUnitProject}\Project.Tests.nunit /xml=nunit-result.xml
Allure doesn't require xml. At lease now :)
https://github.com/unickq/allure-nunit
You just use [AllureNUnit] attribure
When using nunit-console.exe to run tests from the command line you can have a result.xmlfile created, which can further be used for example by ReportUnit to create nice looking reports.
Question
Is there a way to produce this result.xml file when running the tests not via nunit-console but with the NUnit (3.0) Test Adapter for Visual Studio via the Test Explorer Window?
At least I wasn't able to find an xml file after the tests finished running, so I assume, that there is none created.
* I'm using Microsoft Visual Studio Community 2015
NUnit XML output could be produced by the NUnit console and GUI runners only. You won't get it with Visual Studio
It should be possible by specifying the ResultsDirectory property in your .runsettings file. If you don't have one, check the docs:
https://learn.microsoft.com/en-us/previous-versions/visualstudio/visual-studio-2015/test/configure-unit-tests-by-using-a-dot-runsettings-file?view=vs-2015
I have a suite of SpecFlow tests written in C# using MSTest as the framework. I understand its possible to upload the DLL to MTM and run them through there.
However I was wondering if it was possible for me to execute these tests outside of Visual Studio. For example via a dashboard. I wouldn't want the results uploaded to MTM or TFS as we have a few projects that are not connected to them.
My framework uploads results to a database so thats good enough for me in terms of logging.
I gather I would have to do something with the DLL thats generated from building the project that contains SpecFlow, but I want to avoid using reflection if at all possible.
Does anyone have any ideas on how I could go about this?
Tests written using Specflow are simply tests in selected testing framework (MSTest in your case) so you can run them using test runner for that framework.
In case of MSTest you can use MSTest.exe (located in Visual Studio directory, probably also part of TFS installation). Take a look here http://msdn.microsoft.com/en-US/library/ms182489.aspx on how to run tests (easiest way is to use /testcontainer option).
Please note that MSTest is not the best test framework and running outside of Visual Studio is not the same as running them inside of it. There are problems with config file and separate files needed by tests. If you hit these problems, you can switch to other test framework (I use NUnit).
I have been working on UI Automation to perform unit testing using NUnit in Visual Studio 2012. The NUnit is integrated in my VS2012. I need to store the results of the test such as pass/fail status, the error message and the other available details in the test explorer somewhere in a human readable form. I know that VS2010 we used to get the .trx file which could be used further. How do I get the result in VS2012? Is it even possible?
I can at least assure you that it's possible with TFS, as we did this in our company. But I don't remember every detail. The basic steps are:
Create a build definition to run NUnit tests (use the conrole runner for this)
Transform the result XML (TestResult.xml) to VS format (.trx) using a tool. There are several around, try to g00g1e it.
Publish the transformed results to TFS (done in the build script).
It seems that it is not possible to somehow log all test results, when run directly in VS. In this case the only chance you have, is to run the tests externally using the NUnit runners (GUI or Console) and store the TestResults.xml output file.
I have Gallio/MbUnit installed and am using VS 2010 RC and I want to be able to run a single unit test or just all unit tests inside of a TestFixture and not all the tests in the entire project everytime I debug. How do you do this in VS 2010?
Install testdriven.net, there is a free personal version
http://www.testdriven.net/
As Simon said, TD.Net is just working well.
If you work with VS2010 Premium or Ultimate, you may also use the built-in test run feature of VS2010. It should find and run your MbUnit tests as well (if the Gallio add-in was installed)
You may also run your tests by using Resharper 5, but it seems that the latest beta of Resharper 5 has broken the Gallio test driver. It should be fixed soon however.
Apart from the VS2010 IDE, you may also run your tests with Echo (a powerful console application located in the bin folder of the Gallio installation directory), Icarus (Winform application with a nice UI; same location), or from PowerShell (by using the Run-Gallio snap-in). You can also invoke the test runner from one of the numerous extensions of Gallio for MSBuild, NAnt, TeamCity, etc.
I didn't see it earlier but VS 2010 has a Test List Editor which allows me to add the specific test to a a list and run it from there so that only the one test is run. I know that TD.Net probably has a right click/run test option for a test method but I didn't want to buy it and install yet another tool.
Another alternative, how we do it is select the specification inside visual studio, press F5, and it runs the specification via a Rake task.
Example:
class DiffuserObserverSpecification
: AutoMockSpecification<DiffuserObserver, IDiffuserObserver>
class When_diffuser_observer_is_created : DiffuserObserverSpecification
class When_diffuser_observer_has_injected_diffuser_change
: DiffuserObserverSpecification
So to execute the behaviour expectations of all DiffuserObserverSpecification, we select the DiffuserObserverSpecification and press F5.
And to execute just the behaviour for When_diffuser_observer_is_created, select that and press F5.
After one month you forget what I debugger looks like.