Develop Reference

Reporting tool for Nunit 3 and Selenium C#

I'm using Selenium with C# to create my automated tests. Now i have to use some sort of reporting tools to save the test cases whether they passed,failed...etc.
I've seen a lot of tools like Allure, and Jenkins. But they require an XML file. and i can't find it when i run my tests using Visual 2013. Why is that ?
how to obtain these XML files in details if i'm missing something ?
Is there any way i could achieve this with minimum effort ?
EDIT:
How can i run my tests using Nunit console runner ? where can i find it? i downloaded the .zip for nunit but i couldn't find the runner?

The NUnit Visual Studio Adapter does not currently produce XML results, although there is an issue on GitHub to add the ability. Your best bet is to run your tests using the NUnit Console runner when you want to create reports. It always produces an XML result file.
You can install the NUnit Console by adding the NUnit.Console NuGet package to your test project. The console will be in a directory like packages\NUnit.ConsoleRunner.3.4.0\tools in your solution root.
Another option is to install the MSI for the console runner. It will then be under C:\Program Files (x86)\NUnit.org\nunit-console
For creating reports, ReportUnit creates excellent HTML reports from your test results.
To run the tests, I would recommend setting up a simple command line build that builds your solution, runs the tests and then produces the report. For day to day development, just running your tests in Visual Studio or on the command line will likely be enough. Most developers find the Visual Studio Test Explorer Window to be a poor UI, but usable for seeing passed and failed tests and running/debugging them.
If you want to setup a command line build, one good tool is Cake Build. It will take a bit of time to setup, but it is an excellent way to run your build tasks as your project gets larger.

NUnit testing can be run with the nunit-console.exe application which is installed with nunit under {Project_root}/lib/nunit/nunit-console.exe. It downloads with NuGet when NUnit installs.
It can be passed a list of testing binaries, or testing project files, or an nunit project (listing multiple if needed).
{PathToProject}\lib\nunit\nunit-console.exe {PathToTestDll}\Project1.Tests.dll {PathToTestDll}\Project2.Tests.dll /xml=nunit-result.xml
or create an NUnit Project with the NUnit Project Editor if you want to group all your test projects into a single config file.
{PathToProject}\lib\nunit\nunit-console.exe {PathToNUnitProject}\Project.Tests.nunit /xml=nunit-result.xml

Allure doesn't require xml. At lease now :)
https://github.com/unickq/allure-nunit
You just use [AllureNUnit] attribure

Generating Unit Test Reports in Visual Studio 2013

I have a set of unit tests in a C# project in Visual Studio 2013.
Is there a way to generate a report from the unit test results?
I need a report as evidence of time when the tests were run and for the results. I dont mean an online CI server report, I just mean a local file report. Are there any nice plugings for VS2013?
I looked into the NUnit plugin with its XML output but its not very accurate, a lot of the data displayed was duplicated and inaccurate (for example my OS type). I'm currently sticking with the Visual Studio MSTest testing framework for now.

I would suggest using vstest.console.exe from the command line to generate the results as follows:
vstest.console.exe YourTestProject.dll /Logger:trx
The results will be saved in the following format: “username_PCName YYYY-MM-DD HH_MM_SS.trx” Then navigate to the folder where your results have been output and use trx2html (this is the version that works with VS2012+) to convert the results into an html report!
trx2html.exe “username_PCName YYYY-MM-DD HH_MM_SS.trx”
I know you said you wanted a plug in, but maybe this will suit your purposes!

What your probably best doing is setting up Continous Integration and Continuous Delivery.
For example we have have setup the following process in our company:
Jenkins(To manage the process)
SVN TRIGGER -> MSBUILD -> UNIT TESTS(Nunit) -> SONAR -> DEPLOYMENT(MSDEPLOY) -> SMOKE TESTS(has deployment succeeded?) -> ROLLBACK(MSDEPLOY) -> SELENIUM TESTS
In the process we have code adherence and a build breaker to stop if any unit tests are not working, basically if any of that fails you cant deploy. The idea is that it's fool proof. Also the delivery pipeline visualisation tools are pretty cool.
You can pull XML reports from Nunit on the command line into Sonar, I had a few issues with it myself, but you need to rebuild before checking against the tests and run the tests against the dll file itself.
Hopefully that will be of some help.

How to schedule C# unit tests with Jenkins?

Over the last 6 months our test team have been using selenium webdriver to test our web based products. We have had great success with it and continue to use it on a daily basis. We use visual studio since we are a .net shop to write our c# unit tests. We don't use any other testing frameworks for .net.
We have up until recently been running our automation tests manually through the test explorer window in visual studio (2013), checking on the results and then logging them into a spread sheet. I know this isn't ideal but we don't have that many tests so this has suited us fine thus far. However, the number of automation tests we will be required to write and maintain is due to rapidly increase over the next few months.
I have recently been playing around with creating batch files for calling vstest.console.exe and its various commands and then adding those logs to a server. Not ideal. I know I can still do so much more, specifically integrate some sort of CI server.
We are already using team foundation server and have various virtual servers (all running windows 8.1) at our disposal so I thought about taking advantage of this so I began looking into Jenkins. Trouble is, I'm not finding much information regarding Jenkinks and c#. It looks primarily geared to a java setup. Am I missing something? What little information I have found is seriously outdated and didn't work for me.
I got as far as setting it up and installing the vstest.console.exe plugin but couldn't get a simple test to run. A current step by step guide that doesn’t pre-date 2012 would be great :)
Do you guys think Jenkins is the way to go for c# and the .net framework? Is there a "standard" used within the c# community? I have heard of cruise control and I’m going to check that out. Is it a viable alternative? Easier to use with .net?
Here is essentially what we need:
Continue writing our tests inside visual studio and creating c# unit tests
Schedule a run of our unit tests on a remote / local server
Write out a result / log file - nice reporting features on fails / passes would be great
Email said file to qa / dev teams
I'm hoping some of you guys have been down this road once and can share some insights

It is possible to use Jenkins to run tests via batch scripts, reporting back to Jenkins via the NUnit or MSTest plugins. To do this, simply call the test runner from a Jenkins-executed script (see links below). The primary reason for doing this in my shop is that Jenkins is used to automate the build process, and automated tests are run every time new code is promoted. If you don't use Jenkins for build automation and reporting - i.e. you just want scheduling - the most basic solution would be Task Scheduler (as John O indicated). Plus, if you are using MSTest rather than NUnit then, as others have suggested, it is better to have a look at TFS.
If you really want to use Jenkins with MSTest, the following links may be useful from a configuration perspective:
Error trying to run mstest on jenkins - 2012
Example of running MSTest from Jenkins from above link:
"C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\MSTest.exe" /testcontainer:Example.Tests\bin\Debug\Example.Tests.dll /resultsfile:TestResults.trx
Jenkins MSTestRunner plugin is unable to launch mstest.exe - 2014
If you can/want to use NUnit, check out the following:
How do you run NUnit tests from Jenkins?
Execute NUnit with Jenkins
Actually, for reporting purposes, logging to Excel or something similar isn't a horrible idea (particularly if your organization uses Sharepoint). Worst comes to the worst and you can't get anything to work, then one solution would be automating this reporting process by using Excel's COM Object Model to directly write results to the spreadsheet.
Would still suggest that TFS is your best bet, however.

I am going to give you some information on how the test can be handle from CI not exactly how to configure Jenkins or other CI server. This might be a partial answer to your question but at least something. Keep that in mind selenium tests always need a live site to point to and before you kick off the tests site deployment should happen(either manually or automatic). I prefer running the test suite once at night and let it go for whole night. You can simply accomplish this with the help of NUnit console, batch and windows task scheduler. See my answer here how to do that. Sure enough this can be accomplish from any other CI server. With NUnit Command switch you can export result automatically also.

How can I run SpecFlow tests outside of visual studio

I have a suite of SpecFlow tests written in C# using MSTest as the framework. I understand its possible to upload the DLL to MTM and run them through there.
However I was wondering if it was possible for me to execute these tests outside of Visual Studio. For example via a dashboard. I wouldn't want the results uploaded to MTM or TFS as we have a few projects that are not connected to them.
My framework uploads results to a database so thats good enough for me in terms of logging.
I gather I would have to do something with the DLL thats generated from building the project that contains SpecFlow, but I want to avoid using reflection if at all possible.
Does anyone have any ideas on how I could go about this?

Tests written using Specflow are simply tests in selected testing framework (MSTest in your case) so you can run them using test runner for that framework.
In case of MSTest you can use MSTest.exe (located in Visual Studio directory, probably also part of TFS installation). Take a look here http://msdn.microsoft.com/en-US/library/ms182489.aspx on how to run tests (easiest way is to use /testcontainer option).
Please note that MSTest is not the best test framework and running outside of Visual Studio is not the same as running them inside of it. There are problems with config file and separate files needed by tests. If you hit these problems, you can switch to other test framework (I use NUnit).

Test Explorer List in Visual Studio 2012 Extension VSIX

I would like to retrieve the list of tests from the test explorer in a visual studio 2012 extension.
Is this possible using the Package.GetGlobalService or via DTE?
[Edit]
Since this is getting no response I'll provide some more info.
Currently IDE extensions or plugins, that wish to run solution tests do so by providing their own runners. Gallio is one such example of a test runner that has plugin to run multiple types of tests. Resharper is an example of an IDE extension that includes a test runner
VS2012 is introducing a new Test Runner which 3rd party test libraries can develop against such as nunit, xunit, qunit. With this change, it is possible to run these different types of tests within VS itself, namely via the 'Test Explorer' window.
For 3rd parties to interact with the IDE tests, it is necessary to be able to get a list of the tests, run the tests and receive the output of the tests.
So with that in mind, is there any access to the test interfaces in the VS 2012 IDE?

This looks like a good starting point:
API Reference for Testing Tools for Visual Studio ALM