What approach would you take while developing a custom MSBuild Task in a test driven way?
Are there any available test harnesses suitable for test drive development of a Microsoft.Build.Utilities.ToolTask extension?
I was considering attempting to use NUnit or MSUnit and check files generated and where they are placed, though this I forsee this as being a little clunky.
it's not really TDD way but look at the Tool MS Build Sidekicks
This tool really helps us to develop our nightly/daily builds (with database creation, structure compare, CodeAnalysis, test execution, clickonce deployment ...)
You can analyse and debug the build types on the build machine and on the local development machine.
Build scripts are not designed to be tested.. but
You can create some SmokeTests of your build to see if everything went ok. If you are deploying a website you can have some smoke tests to see:
Login page could be opened
Login page works (You can make a correct login and a failed one)
Core funcionality works (Once you accessed to your site you can perform some basic action like
opening product page or similar)
Those smoke test should be able to be called from command line, so you can call them from task AfterDropBuild to see the result of smoke tests just after build was created.
Related
I have a Selenium/c#/specflow test project that is sitting locally and now on TFS.
The TFS version builds and picks up the tests fine, but I am currently using a hardcoded URL as the starting point for the tests.
I want to be able to create two different release definitions, one for the dev env and one for the test env.
I have found that there is a run settings file that can be added, and then overridden in the release definition (within the vstest assembly), but it seems this is geared towards unit tests.
My question is whether this is the correct place to specify a URL as the starting point for my tests (so that I can then create another release definition for dev and change this variable to my dev URL)? Is there a standard approach to doing this?
Ultimately I want to have a variable in my code for a URL, and to be able to over ride that from the release definition in TFS!
This is the first test project I've set up and built in TFS so I'm looking for a bit guidance on where this is best put.
Thanks in advance.
Not sure if totally got your point, instead of hardcode the url in tests, you could try to use this solution which only requires:
· Create runsettings file and add your parameters.
· Modify your code to call the parameters from the .runsettings file
· Changes to the “Test Steps” in VSTS release or build
More details take a look at this blog: Enabling Targeted Environment Testing during Continuous Delivery (Release Management) in VSTS
Another blog may be helpful to set up your selenium test CD through TFS: Continuous Delivery using VSO Release Manager with Selenium Automated Tests on Azure Web Apps (PaaS)
Over the last 6 months our test team have been using selenium webdriver to test our web based products. We have had great success with it and continue to use it on a daily basis. We use visual studio since we are a .net shop to write our c# unit tests. We don't use any other testing frameworks for .net.
We have up until recently been running our automation tests manually through the test explorer window in visual studio (2013), checking on the results and then logging them into a spread sheet. I know this isn't ideal but we don't have that many tests so this has suited us fine thus far. However, the number of automation tests we will be required to write and maintain is due to rapidly increase over the next few months.
I have recently been playing around with creating batch files for calling vstest.console.exe and its various commands and then adding those logs to a server. Not ideal. I know I can still do so much more, specifically integrate some sort of CI server.
We are already using team foundation server and have various virtual servers (all running windows 8.1) at our disposal so I thought about taking advantage of this so I began looking into Jenkins. Trouble is, I'm not finding much information regarding Jenkinks and c#. It looks primarily geared to a java setup. Am I missing something? What little information I have found is seriously outdated and didn't work for me.
I got as far as setting it up and installing the vstest.console.exe plugin but couldn't get a simple test to run. A current step by step guide that doesn’t pre-date 2012 would be great :)
Do you guys think Jenkins is the way to go for c# and the .net framework? Is there a "standard" used within the c# community? I have heard of cruise control and I’m going to check that out. Is it a viable alternative? Easier to use with .net?
Here is essentially what we need:
Continue writing our tests inside visual studio and creating c# unit tests
Schedule a run of our unit tests on a remote / local server
Write out a result / log file - nice reporting features on fails / passes would be great
Email said file to qa / dev teams
I'm hoping some of you guys have been down this road once and can share some insights
It is possible to use Jenkins to run tests via batch scripts, reporting back to Jenkins via the NUnit or MSTest plugins. To do this, simply call the test runner from a Jenkins-executed script (see links below). The primary reason for doing this in my shop is that Jenkins is used to automate the build process, and automated tests are run every time new code is promoted. If you don't use Jenkins for build automation and reporting - i.e. you just want scheduling - the most basic solution would be Task Scheduler (as John O indicated). Plus, if you are using MSTest rather than NUnit then, as others have suggested, it is better to have a look at TFS.
If you really want to use Jenkins with MSTest, the following links may be useful from a configuration perspective:
Error trying to run mstest on jenkins - 2012
Example of running MSTest from Jenkins from above link:
"C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\MSTest.exe" /testcontainer:Example.Tests\bin\Debug\Example.Tests.dll /resultsfile:TestResults.trx
Jenkins MSTestRunner plugin is unable to launch mstest.exe - 2014
If you can/want to use NUnit, check out the following:
How do you run NUnit tests from Jenkins?
Execute NUnit with Jenkins
Actually, for reporting purposes, logging to Excel or something similar isn't a horrible idea (particularly if your organization uses Sharepoint). Worst comes to the worst and you can't get anything to work, then one solution would be automating this reporting process by using Excel's COM Object Model to directly write results to the spreadsheet.
Would still suggest that TFS is your best bet, however.
I am going to give you some information on how the test can be handle from CI not exactly how to configure Jenkins or other CI server. This might be a partial answer to your question but at least something. Keep that in mind selenium tests always need a live site to point to and before you kick off the tests site deployment should happen(either manually or automatic). I prefer running the test suite once at night and let it go for whole night. You can simply accomplish this with the help of NUnit console, batch and windows task scheduler. See my answer here how to do that. Sure enough this can be accomplish from any other CI server. With NUnit Command switch you can export result automatically also.
I have a suite of SpecFlow tests written in C# using MSTest as the framework. I understand its possible to upload the DLL to MTM and run them through there.
However I was wondering if it was possible for me to execute these tests outside of Visual Studio. For example via a dashboard. I wouldn't want the results uploaded to MTM or TFS as we have a few projects that are not connected to them.
My framework uploads results to a database so thats good enough for me in terms of logging.
I gather I would have to do something with the DLL thats generated from building the project that contains SpecFlow, but I want to avoid using reflection if at all possible.
Does anyone have any ideas on how I could go about this?
Tests written using Specflow are simply tests in selected testing framework (MSTest in your case) so you can run them using test runner for that framework.
In case of MSTest you can use MSTest.exe (located in Visual Studio directory, probably also part of TFS installation). Take a look here http://msdn.microsoft.com/en-US/library/ms182489.aspx on how to run tests (easiest way is to use /testcontainer option).
Please note that MSTest is not the best test framework and running outside of Visual Studio is not the same as running them inside of it. There are problems with config file and separate files needed by tests. If you hit these problems, you can switch to other test framework (I use NUnit).
I need to organise the launch Unit-Tests in MS Test Manager. Can It be done? I create controller but I can't specify path for my solution or tests dll.
I'm not sure that launch Unit-Tests in Microsoft test manager are possible, but my boss requires it.
What you should probably do is setup your build server to run the tests. Then use test manager to trigger a build, which will in turn run the tests.
There's not much value in running the tests again on previously built code as it won't have changed. The tests should run on the code as it builds and then again when the code changes and is consequently rebuilt.
Using Visual Studio and TFS & preferably Specflow or standard unit test.
I want devs to run ALL unit test as a policy before check in. If a unit test breaks, then vS should stop them from checking in, just like when running across a merge conflict.
I know there're post build scripts that will do this, but really if unit test breaks, I rather that it doesn't get into source control at all. Plus the turn around is rather slow to wait for the full build. And then there's the bickering on who breaks whose stuff.
So no, I want unit test to pass locally before a check in. How would I do that? Yes they can just hit the button, but I like to get them a bit more "incentive" than that.
It sounds like what you're after is a TFS Gated Check-in. This can ensure that the code builds, merges and that tests run successfully prior to committing the check-in. You can read more about it here:
An introduction to gated
check-in
It's worth noting that it's a much slower process than CI builds, so depending on how many check-ins your developers are doing you may be better off looking at a CI build with 'Create Work Item on Failure' enabled and a Project Alert set up to notify the developer that they broke the build.
The TeamCity Visual Studio plugin supports pre-tested commits. I can't speak for TFS, however.