Develop Reference

Trying to run Specflow tests but error returned - Unit Test Provider already specified

I've been looking at similar questions on different websites in the hope I could find an answer for it but I've not come across something which has worked for me.
I have a Visual Studio 2019 solution (.Net Framework) which contains Specflow related tests. I can see all my tests in the Test Explorer but when I try to run any of them, I am returned with the error 'Unit test provider already specified'. My test runner was initially Specflow.SpecRun but I have since removed all references to this and am now using Specflow.MsTest - this is to allow me to associate my test cases to one in Azure DevOps. Only after making this change, I am getting these errors.
I have searched across my solution and there are no longer any references to SpecRun; there are no build errors either. The 'Enable SpecflowSingleFileGenerator' setting is set to False, I do have an app.config file but do not have a specflow.json file - I read that that is only needed for .NET Core projects.
Has anyone else come across the above or have any ideas to resolving it as I am a bit stuck.

I was having this problem when I moved from nUnit to MsTest. I want to add few more steps to comments mentioned
Check your feature.cs files and make sure it has onlyMicrosoft.VisualStudio.TestTools.UnitTesting attributes and not nUnit or specrun
Check whether you are using any local.testsettings file in visual studio
Remove nUnit from tools and extensions if you are not using it.
Restart VS, clear temp folder, clean and rebuild

So this may not be the most ideal solution but I created a new project within the same solution, installed the same nuget packages and copied all the files/folders across to the new project. I removed the previous project, cleaned solution and re-built. Tests appear in the Test Explorer and also run successfully too.

Is it possible to limit the projects which specflow searches for tests when generating Step bindings?

My new workplace has a very large C# solution in Visual Studio 2017 with over 100 projects, of which only one is directly relevant to my job (web acceptance tests via specflow & selenium).
We have intermittent problems with CPU usage & tests not being found in the test explorer, occurring whenever Specflow decides it needs to search for and regenerate test bindings - as it takes some hours to complete, during which Visual Studio is not useful.
From experimentation, I know that if I open only the one project relevant to my work, then specflow finds all the tests in seconds and is immediately useable.
But sadly we need the whole solution to be loaded at least when building new code to be tested.
Is there a way to limit which projects (or paths) Specflow and/or test explorer is searching in for tests, & subsequently generating step bindings from?
I’m hoping for a local settings file (references to .runsettings), but haven’t been able to find any instructions for how to configure this.

There isn't such an option available.
This looks to me we have to improve the VS Extension too handle that many projects.
Could you please open an issue here: https://github.com/techtalk/SpecFlow
There we can discuss, what we can do, to make SpecFlow for you usable.

How to schedule C# unit tests with Jenkins?

Over the last 6 months our test team have been using selenium webdriver to test our web based products. We have had great success with it and continue to use it on a daily basis. We use visual studio since we are a .net shop to write our c# unit tests. We don't use any other testing frameworks for .net.
We have up until recently been running our automation tests manually through the test explorer window in visual studio (2013), checking on the results and then logging them into a spread sheet. I know this isn't ideal but we don't have that many tests so this has suited us fine thus far. However, the number of automation tests we will be required to write and maintain is due to rapidly increase over the next few months.
I have recently been playing around with creating batch files for calling vstest.console.exe and its various commands and then adding those logs to a server. Not ideal. I know I can still do so much more, specifically integrate some sort of CI server.
We are already using team foundation server and have various virtual servers (all running windows 8.1) at our disposal so I thought about taking advantage of this so I began looking into Jenkins. Trouble is, I'm not finding much information regarding Jenkinks and c#. It looks primarily geared to a java setup. Am I missing something? What little information I have found is seriously outdated and didn't work for me.
I got as far as setting it up and installing the vstest.console.exe plugin but couldn't get a simple test to run. A current step by step guide that doesn’t pre-date 2012 would be great :)
Do you guys think Jenkins is the way to go for c# and the .net framework? Is there a "standard" used within the c# community? I have heard of cruise control and I’m going to check that out. Is it a viable alternative? Easier to use with .net?
Here is essentially what we need:
Continue writing our tests inside visual studio and creating c# unit tests
Schedule a run of our unit tests on a remote / local server
Write out a result / log file - nice reporting features on fails / passes would be great
Email said file to qa / dev teams
I'm hoping some of you guys have been down this road once and can share some insights

It is possible to use Jenkins to run tests via batch scripts, reporting back to Jenkins via the NUnit or MSTest plugins. To do this, simply call the test runner from a Jenkins-executed script (see links below). The primary reason for doing this in my shop is that Jenkins is used to automate the build process, and automated tests are run every time new code is promoted. If you don't use Jenkins for build automation and reporting - i.e. you just want scheduling - the most basic solution would be Task Scheduler (as John O indicated). Plus, if you are using MSTest rather than NUnit then, as others have suggested, it is better to have a look at TFS.
If you really want to use Jenkins with MSTest, the following links may be useful from a configuration perspective:
Error trying to run mstest on jenkins - 2012
Example of running MSTest from Jenkins from above link:
"C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\MSTest.exe" /testcontainer:Example.Tests\bin\Debug\Example.Tests.dll /resultsfile:TestResults.trx
Jenkins MSTestRunner plugin is unable to launch mstest.exe - 2014
If you can/want to use NUnit, check out the following:
How do you run NUnit tests from Jenkins?
Execute NUnit with Jenkins
Actually, for reporting purposes, logging to Excel or something similar isn't a horrible idea (particularly if your organization uses Sharepoint). Worst comes to the worst and you can't get anything to work, then one solution would be automating this reporting process by using Excel's COM Object Model to directly write results to the spreadsheet.
Would still suggest that TFS is your best bet, however.

I am going to give you some information on how the test can be handle from CI not exactly how to configure Jenkins or other CI server. This might be a partial answer to your question but at least something. Keep that in mind selenium tests always need a live site to point to and before you kick off the tests site deployment should happen(either manually or automatic). I prefer running the test suite once at night and let it go for whole night. You can simply accomplish this with the help of NUnit console, batch and windows task scheduler. See my answer here how to do that. Sure enough this can be accomplish from any other CI server. With NUnit Command switch you can export result automatically also.

How can I run SpecFlow tests outside of visual studio

I have a suite of SpecFlow tests written in C# using MSTest as the framework. I understand its possible to upload the DLL to MTM and run them through there.
However I was wondering if it was possible for me to execute these tests outside of Visual Studio. For example via a dashboard. I wouldn't want the results uploaded to MTM or TFS as we have a few projects that are not connected to them.
My framework uploads results to a database so thats good enough for me in terms of logging.
I gather I would have to do something with the DLL thats generated from building the project that contains SpecFlow, but I want to avoid using reflection if at all possible.
Does anyone have any ideas on how I could go about this?

Tests written using Specflow are simply tests in selected testing framework (MSTest in your case) so you can run them using test runner for that framework.
In case of MSTest you can use MSTest.exe (located in Visual Studio directory, probably also part of TFS installation). Take a look here http://msdn.microsoft.com/en-US/library/ms182489.aspx on how to run tests (easiest way is to use /testcontainer option).
Please note that MSTest is not the best test framework and running outside of Visual Studio is not the same as running them inside of it. There are problems with config file and separate files needed by tests. If you hit these problems, you can switch to other test framework (I use NUnit).

TFS Build Issues With Multiple Versions of the Same Assembly in Different Projects

I keep running into an issue with our TFS build server. I've got 2 projects (both in the same solution), 1 is a WebForms project, running .Net 4.0. The second is an ASP .Net MVC5 project running .Net 4.5. There is also a Silverlight project, but the problem is reproducible with just the first two.
Both of these projects use NuGet packages for various libraries. Sometimes there are different assemblies within a package for their respective environments. A .Net 4.0, 4.5, SL assembly, etc.
The build server seems to dump all of the libraries required into a single folder, then pulls from that to build the solution. This causes problems in many cases, with the wrong project getting the wrong assembly version. This does not occur locally, only on the build server. I can't figure out what I need to do to keep this from happening. Any ideas?

Yes, I hate this standard behavior, but TFS will output everything to the same folder by default, and then you will get various errors depending on which order msbuild compiles your projects if you have references with the same name or even project outputs with the same name.
The easiest workaround is to use the AsConfigured option on the Process tab, '2. Build' -> 'Output location' of the build definition window. This keeps your normal source structure intact, but I think you will lose support for automatically dropped outputs (i.e. you will have to provide a script to do that yourself). If you are only using TFS Build for validation, this is the cleanest approach.
You can also use the PerProject setting and split up your projects into two distinct solutions, perhaps suffixed by platform (we've done that numerous times in our company). Then, you specify both solutions to the build process and it will create two separate folders in the output, one for each solution.
This is all assuming you are using TFS 2013. In TFS2012, there is a similar option but it is in '3. Advanced' -> 'Solution Specific Build Outputs'. You will probably have to go this route if you are using TFS2012 or you will need to modify the default workflow yourself to add your own logic.
EDIT:
From your comment to the other poster I see you are using TFS 2010. Well... I think this was absolutely not supported at that time, I remember having similar problems, but we upgraded to TFS 2012 and all was well.
I think your only option is to either create two separate build definitions and build each solution that way, or you will need to checkout the xaml workflow and edit it with your own logic. Perhaps downloading the TFS2012 template and "porting" it to TFS2010 would be a better approach since at least you would not be reinventing the wheel that way.