Using Visual Studio and TFS & preferably Specflow or standard unit test.
I want devs to run ALL unit test as a policy before check in. If a unit test breaks, then vS should stop them from checking in, just like when running across a merge conflict.
I know there're post build scripts that will do this, but really if unit test breaks, I rather that it doesn't get into source control at all. Plus the turn around is rather slow to wait for the full build. And then there's the bickering on who breaks whose stuff.
So no, I want unit test to pass locally before a check in. How would I do that? Yes they can just hit the button, but I like to get them a bit more "incentive" than that.
It sounds like what you're after is a TFS Gated Check-in. This can ensure that the code builds, merges and that tests run successfully prior to committing the check-in. You can read more about it here:
An introduction to gated
check-in
It's worth noting that it's a much slower process than CI builds, so depending on how many check-ins your developers are doing you may be better off looking at a CI build with 'Create Work Item on Failure' enabled and a Project Alert set up to notify the developer that they broke the build.
The TeamCity Visual Studio plugin supports pre-tested commits. I can't speak for TFS, however.
Related
My new workplace has a very large C# solution in Visual Studio 2017 with over 100 projects, of which only one is directly relevant to my job (web acceptance tests via specflow & selenium).
We have intermittent problems with CPU usage & tests not being found in the test explorer, occurring whenever Specflow decides it needs to search for and regenerate test bindings - as it takes some hours to complete, during which Visual Studio is not useful.
From experimentation, I know that if I open only the one project relevant to my work, then specflow finds all the tests in seconds and is immediately useable.
But sadly we need the whole solution to be loaded at least when building new code to be tested.
Is there a way to limit which projects (or paths) Specflow and/or test explorer is searching in for tests, & subsequently generating step bindings from?
I’m hoping for a local settings file (references to .runsettings), but haven’t been able to find any instructions for how to configure this.
There isn't such an option available.
This looks to me we have to improve the VS Extension too handle that many projects.
Could you please open an issue here: https://github.com/techtalk/SpecFlow
There we can discuss, what we can do, to make SpecFlow for you usable.
We have 10K+ unit tests in C# solution which are passed when running in local and in TFS.
Now, we are setting up Jenkins for our solution stack and we are facing issue of around 250 Unitests failing consistently.
The same unit tests are passed when i tried running them in jenkins setup server by using Visual Studio and Commnad prompt(MSTest).
What do you think is the issue? Any leads to look at this issue will be helpful.
Edit 1:
I did the research and not able to find anything as the problem itself is strange one. If you are not clear please raise questions instead of Down voting.
Edit 2:
I am able to find out the issue. It is with the unittests dll config file. When i executed MSTest in server by removing the config file, i am seeing same set of tests failing which are failing with Jenkins setup.
I guess we need to modify the steps configure in Jenkins portal to load the unittests dll config file.
My guess would be you have tests that are not actually Unit tests, but instead are integration tests or worse, and they fail non-deterministically. Other than that, you're asking people to do the impossible here without source code posted.
Either post source, or hire a consultant who knows about Jenkins IMO.
There might be issues with conditional compilation symbols (e.g. DEBUG vs. RELEASE code): in VS you normally run the tests on a DEBUG build, on the CI server on a release build.
Also look at some global state not being cleaned up correctly. Some threads which may still be running after a test has seemingly finished may corrupt later tests even when those later tests are located in a different test dll. That can sometimes be detected if test failure depends on the order of tests run.
Another issue often faced is dependency on test data in files: the files may be missing in the virtualized environment where the test is actually run. Use the Deployment attribute.
I had a similar problem with NCrunch. Maybe you can check in VS your "Build"-tag the Platform target. It should be the same like Jenkins is configured. For example "Platform target" is x64, it should be used in Jenkins as well.
Over the last 6 months our test team have been using selenium webdriver to test our web based products. We have had great success with it and continue to use it on a daily basis. We use visual studio since we are a .net shop to write our c# unit tests. We don't use any other testing frameworks for .net.
We have up until recently been running our automation tests manually through the test explorer window in visual studio (2013), checking on the results and then logging them into a spread sheet. I know this isn't ideal but we don't have that many tests so this has suited us fine thus far. However, the number of automation tests we will be required to write and maintain is due to rapidly increase over the next few months.
I have recently been playing around with creating batch files for calling vstest.console.exe and its various commands and then adding those logs to a server. Not ideal. I know I can still do so much more, specifically integrate some sort of CI server.
We are already using team foundation server and have various virtual servers (all running windows 8.1) at our disposal so I thought about taking advantage of this so I began looking into Jenkins. Trouble is, I'm not finding much information regarding Jenkinks and c#. It looks primarily geared to a java setup. Am I missing something? What little information I have found is seriously outdated and didn't work for me.
I got as far as setting it up and installing the vstest.console.exe plugin but couldn't get a simple test to run. A current step by step guide that doesn’t pre-date 2012 would be great :)
Do you guys think Jenkins is the way to go for c# and the .net framework? Is there a "standard" used within the c# community? I have heard of cruise control and I’m going to check that out. Is it a viable alternative? Easier to use with .net?
Here is essentially what we need:
Continue writing our tests inside visual studio and creating c# unit tests
Schedule a run of our unit tests on a remote / local server
Write out a result / log file - nice reporting features on fails / passes would be great
Email said file to qa / dev teams
I'm hoping some of you guys have been down this road once and can share some insights
It is possible to use Jenkins to run tests via batch scripts, reporting back to Jenkins via the NUnit or MSTest plugins. To do this, simply call the test runner from a Jenkins-executed script (see links below). The primary reason for doing this in my shop is that Jenkins is used to automate the build process, and automated tests are run every time new code is promoted. If you don't use Jenkins for build automation and reporting - i.e. you just want scheduling - the most basic solution would be Task Scheduler (as John O indicated). Plus, if you are using MSTest rather than NUnit then, as others have suggested, it is better to have a look at TFS.
If you really want to use Jenkins with MSTest, the following links may be useful from a configuration perspective:
Error trying to run mstest on jenkins - 2012
Example of running MSTest from Jenkins from above link:
"C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\MSTest.exe" /testcontainer:Example.Tests\bin\Debug\Example.Tests.dll /resultsfile:TestResults.trx
Jenkins MSTestRunner plugin is unable to launch mstest.exe - 2014
If you can/want to use NUnit, check out the following:
How do you run NUnit tests from Jenkins?
Execute NUnit with Jenkins
Actually, for reporting purposes, logging to Excel or something similar isn't a horrible idea (particularly if your organization uses Sharepoint). Worst comes to the worst and you can't get anything to work, then one solution would be automating this reporting process by using Excel's COM Object Model to directly write results to the spreadsheet.
Would still suggest that TFS is your best bet, however.
I am going to give you some information on how the test can be handle from CI not exactly how to configure Jenkins or other CI server. This might be a partial answer to your question but at least something. Keep that in mind selenium tests always need a live site to point to and before you kick off the tests site deployment should happen(either manually or automatic). I prefer running the test suite once at night and let it go for whole night. You can simply accomplish this with the help of NUnit console, batch and windows task scheduler. See my answer here how to do that. Sure enough this can be accomplish from any other CI server. With NUnit Command switch you can export result automatically also.
Ok, so here's my current setup and my problem at the moment. I have a growing set of projects in a Visual Studio solution. The solution contains about 15 projects (give or take a few) and a quickly growing code-base. I realize that I should have had a continuous build system set-up before I got to this, but its never too late I suppose. So after doing a little research, I believe that my perfect setup would be:
NUnit 2.5.x (we are already tied to this... so a necessity)
Integration with CruiseControl.Net (open to other options, but only free ones with Git support)
Integration with a code-coverage tool (NCover, DotCover) would be nice
Integration to run shell commands (for JSLint and compression tools, etc.)
What I am missing is a tool to run the automated build. I looked at NAnt, but it's support for running MSBuild (to build the project) seemed fairly outdated (we're using VS2010) and utilizing the solution files in our build process would be a HUGE time saver. I also looked at MSBuild (for the obvious reasons) but the process that I found for running NUnit tests only supports 2.4.x (MSBuild extensions project).
I am curious how everyone else has organized their continuous build systems. NUnit if fairly popular, so I must not be the only one who is wondering about this.
my first question is how may build projects will you have?
Teamcity Professional is free for 20 build configurations per server and will make your like sooooo much easier, has dotcover build in, and is really easy to setup, run your tests, etc. and it's by far the most fully baked CI server out there.
Jenkins is the next runner up, it's a fork of Hudson and is very flexible with plugins
to do just about anything making it a little more flexible then Teamcity but it's not as easy to set up, code coverage is a pain to set up and has some annoying quirks, but is completely free.
unless you have some really strong reason to use CruiseControl.Net, don't bother, for it's time it was very powerful but it's now sadly out dated and painful to use.
As far as setting up builds, both Teamcity and Jenkins support MSBuild, NAnt, Rake, etc. they also support multiple build steps like would do in an msbuild or Nant file. What I have done in the past is just use the .sln file to do the build with one build step, used the build in task for unit tests, then used the built in task for code coverage then used a another build task for pushing the files.
I have used TeamCity, Jenkins, TFS, and I tried to used CruiseControl.Net but found it painfully clunky. By far Teamcity is the best, with Jenkins a close second, I would not willing use TFS even if I had it.
If you have any questions please feel free to contact me.
You can use NAnt to build your Visual Studio 2010 solutions. I do it all the time. I provided sample NAnt script in my answer here: <msbuild> task or msbuild.exe with NAnt?
If all you need is compile and run tests you can't go wrong with TeamCity it has great support for NUnit/VS and a bunch of reports built in.
If you need to run a more complex build script I suggest you use FinalBuilder for creating the build script and TeamCity command runner to execute that script.
By importing the test result into TeamCity you can still get the reports and there is a simple way to output build status from FinalBuilder to TeamCity:
How can I output messages from FinalBuilder that will be captured by TeamCity?
Outputting build status from FinalBuilder to TeamCity
What approach would you take while developing a custom MSBuild Task in a test driven way?
Are there any available test harnesses suitable for test drive development of a Microsoft.Build.Utilities.ToolTask extension?
I was considering attempting to use NUnit or MSUnit and check files generated and where they are placed, though this I forsee this as being a little clunky.
it's not really TDD way but look at the Tool MS Build Sidekicks
This tool really helps us to develop our nightly/daily builds (with database creation, structure compare, CodeAnalysis, test execution, clickonce deployment ...)
You can analyse and debug the build types on the build machine and on the local development machine.
Build scripts are not designed to be tested.. but
You can create some SmokeTests of your build to see if everything went ok. If you are deploying a website you can have some smoke tests to see:
Login page could be opened
Login page works (You can make a correct login and a failed one)
Core funcionality works (Once you accessed to your site you can perform some basic action like
opening product page or similar)
Those smoke test should be able to be called from command line, so you can call them from task AfterDropBuild to see the result of smoke tests just after build was created.