The problem I am facing is a unique one.. I have set up a testing suite for a windows app using LeanFT and NUnit. At this point, I have around 100 stable UI tests that I have automatically running nightly on a VM I have configured.
The problem is, I do not think there is an out of the box solution for running tests in parallel across multiple machines. When the tests are kicked off of TFS, they run on a single machine. What I want is for tests to be passed out 1 at a time to any available machine.
I know I can "fake" this in a sense by having two different build definitions run at the same time with a different set of tests, but this is not what I want to do. I want it to work like an actual grid and pass out tests like the selenium grid does. Has anyone had any luck with something like this by not faking it?
Note: we do not use Jenkins, we do everything out of TFS.
Have you looked at https://blogs.msdn.microsoft.com/devops/2016/10/10/parallel-test-execution/ ?
Team Foundation Server does have 'build agents' like Jenkins does. While I'm not familiar with this, it appears to be close to what you are looking for.
Related
I have a half dozen feature files with scenarios
I run the features one by one, all works fine
I click Run All on the Test Explorer
It opens windows for all scenarios simultaneously, skips the steps and then closes the windows, like its only processing my Before/After Scenario tags
(causing all my scenarios to fail)rather than working its way through the list of features alphabetically
Any ideas?
It could be that XUnit is executing your tests in parallel.
Try to disable it and run the tests again.
Docu is here: https://xunit.github.io/docs/running-tests-in-parallel.html
I have been playing around with testing a simple webforms ASP.NET website application and have come across the UIMap tool in Visual Studio 2010. However I do not know how to use it correctly and it never seems to work.
Do I need to be running my application in debug mode while recording?
Once the test is generated is there any way of visually seeing the steps as they are being performed?
Should I have the browser open before I begin recording, or should I do that once I start and browse to the page that way?
Can I use Chrome for the browser, or does it have to be IE?
Does using Dual Screens affect the running of the test?
Thanks for your help!
Do I need to be running my application in debug mode while recording?
No. You can not use debug/release mode of Visual Studio and do an record at same time. Start the application separately. This has the big advantage that you can test different versions of our application with one set of tests.
Once the test is generated is there any way of visually seeing the
steps as they are being performed?
No. You can get a log from test-run and of course you see the test-run in self, but there is no static graphic ... to be honest I never miss such feature.
Should I have the browser open before I begin recording, or should I
do that once I start and browse to the page that way?
That you can do as you please. Of course you can record the browser start. We start browser before, because browser start is not really a part of the test.
Can I use Chrome for the browser, or does it have to be IE?
At the moment IE is the forced browser. But there are ways to use other browsers.
Does using Dual Screens affect the running of the test?
No. All fine with dual screen. But don't forget when test is running you can't use your mouse and keyboard - you will fight the CUIT ;). We use a separate PC for long CUIT.
Here are some links which may be helpful for you:
should the coded ui test project share a solution or not?
Unit Testing Frameworks for Visual Studio 2012 Cons/Pros
Which Unit Test framework to use for the projects that requires User Input
I have a solution with a Web Application, Console Application (and Windows Service, but that's probably for another question).
Publishing and keeping the Web Application up-to-date is easy. I simply right-click Publish to publish initially or update the live version.
Currently to "publish" or update a console application I copy the release exe and files over to the live machine. In this way I can use the task manager to run the exe in the copied location.
This smacks of bad practice to me. The problem with doing a right-click, Publish on the console app is there's no easy way for the task scheduler to launch the exe. Additionally i'm not sure what would happen when it came to updating the application. Would the "An update is available" screen just sit there waiting for a user to click ok??
What is the best way of easily publishing and keeping a console application up to date?
Ideally something I only have to do from Visual Studio (2008). I'll need to be able to set up task scheduler and then forget about it (no need to do anything with task scheduler or click through anything when updating).
Automatic deploy of applications partly sounds like Continuous Integration.
You may not be interested in the TDD parts, so you can just focus on the tools that allow you to grab code from your repository (SVN for example), build it and deploy it automatically.
Personally, and based on personal experience, I would do this for backstage environments only, for production I would stick to copying exes manually (and keeping backups, of course!)
Personally for these server side executables I just copy the exes. If you're updating them a lot and you want to ensure that you don't put them in the wrong folder or miss a file you could write a .bat script to do the copying for you so that you can just double click on it.
To be honest if you're deploying a new program so often that this is a major issue for you then I'd say you have a different problem. You shouldn't be making changes to a production app all that often. There might be something that's currently code driven that ought to be data driven such that you can make modifications that you need to your program by modifying your database, config files, etc. and not the code of the program.
I have been researching on how to automate browser testing for a number of weeks now using all kind of different methods. Seleniums website is very vague on which is the best route to take.
I have installed
Selenium Webdriver
Remote Control
Selenium Grid
Apache Ant
Nunit
(and pretty much everything else you could need to do this)
I finally give up trying on my own and want to know the best way to do this.
I want to be able to
Load the same webpage on a number of different browsers
Load the same webpage on a number of virtual machines(which I have set up)
Be able to take snapshots comparing the different browser results.
I have knowledge of programming in C# and would prefer to run my tests through Nunit.
Can anyone give me directions or point me to a website that already has them?
Thank you.
I have built up a test framework using junit with Selenium WebDriver that satisfies every one of your points. While its not exactly what you're asking for, I feel it may be beneficial to you regardless.
Load the same webpage on a number of different browsers
Using Selenium's grid, this is very simple to set up. Set up some virtual machines with the environments you're looking to test in. In our environment, for example, we have a grid running with four nodes (as virtual machines) with a setup like the following
Windows with IE7 and FireFox
Windows with IE8 and FireFox
Windows with IE9 and Firefox
Linux with FireFox
Note that Selenium recommends that only one instance of IE be allowed to run on the Windows nodes. On each of the aforementioned nodes, there is one instance of the specified IE and five instances of the specified FF allowed to run at any given time. With the grid setup and the hub configured, firing off tests is a breeze. In WebDriver, use the DesiredCapabilities object to set up the desired environment and then just send the test off and wait for the result to return.
Platform desiredPlatform;
DesiredCapabilities desiredCapabilities;
desiredPlatform = Platform.LINUX;
desiredCapabilities = DesiredCapabilities.firefox();
desiredCapabilities.setPlatform(desiredPlatform);
desiredCapabilities.setVersion("11");
WebDriver driver = new RemoteWebDriver("http://hubURL", desiredCapabilities);
Load the same webpage on a number of virtual machines(which I have set up)
I solved this one by forcing the tests to run, albeit in an unconvential way, in a threaded manner. Each JUnit test uses a shared thread library I put together which creates all the necessary RemoteWebDrivers needed in separate threads. Each of these threads runs simultaneously on its node while the parent thread sits and waits for all to terminate. Then on to the next test which is run multithreaded as well.
There were a couple problems I encountered such as retrieving the Junit stack traces in all of the child threads. I solved this by redirecting Std.err to a bytestream on the parent thread. All errors get routed to that stream which I then convert to a string and print out to Std.out at the end of each test. The html pages generated at the end of the tests include Std.out which worked out perfectly.
Be able to take snapshots comparing the different browser results
While I have gotten this to work, there are some inherent problems with grabbing screenshots remotely. IE will return black screenshots if the process is running as a service. The workaround was to just run the jar from the command line and keep the user logged in, in which case the screenshots return correctly. This is a known issue in the browser and there really is no nice solution to the problem. Taking screenshots works roughly like this
WebDriver augmentedDriver = new Augmenter().augment(driver);
TakesScreenshot ss = (TakesScreenshot) augmentedDriver;
String base64Screenshot = ss.getScreenshotAs(OutputType.BASE64);
byte[] decodedScreenshot = Base64.decodeBase64(base64Screenshot.getBytes());
FileOutputStream fos = new FileOutputStream(new File(imageName));
fos.write(decodedScreenshot);
fos.close();
which saves the captured screenshot from the remote machine's running browser onto the local machine.
In reality, browser automation is still struggling to stabilize itself. There are a number of important features, such as the ones you're asking about, that just aren't implemented solidly that I know of in any framework. With time, though, I'm sure a lot of this will settle down and QA developers everywhere will rejoice.
As for the 2nd point: instead of using Grid you can let your continuous integration server do the job. In my company we use Jenkins and so called Configuration Matrix - it let's you run the same job on multiple Jenkins nodes.
As for the 1st one, I think Jenkins could be helpful here too. You can run multiple jobs on the same node. Although I've never tried that so I am not perfectly sure. And this is just an idea, I wouldn't really recommend such solution. You may also want to read this blog post describing how to run test in parallel using Selenium Grid. For people using Java I would recommend reading about parallel tests with TestNG.
Your third point is a little bit vague. What do you mean by snapshot? And what kind of result you want to compare?
Selenium RC is outdated and webdriver is more reliable way of creating selenium tests. I see the responses above cater more on java side. Below mentioned is more information on how to achieve the questions asked here using C# and selenium webdriver
On how to setup the IDE (VS express), nUnit and selenium refer
How to setup C#,nUnit and selenium client drivers on VSExpress for Automated tests
On Creating simple script that launches a browser does few steps refer
Creating Basic Selenium web driver test case using Nunit and C#
On how to Load the same webpage on a number of different browsers suggest referring
How to invoke locally different types of browser driver using selenium and c#
On Load the same webpage on a number of virtual machines(which I have set up) for this, you need to use Remote webdriver instead of normal webdriver. Also with remote webdriver, you can launch different types of browser. Refer this webpage
How to invoke/run different type of web driver browser using remote webdriver in C#
To take snapshot on different browser you can refer the link
Capturing screen shots using remote/local webdriver in C#/Selenium webdriver
You might also consider the free Telerik Testing Framework. This is the underpinning for Telerik's commercial Test Studio product. The Testing Framework provides cross-browser support, does a great job with dynamic content situations (AJAX), and also lets you handle OS-level dialogs like file upload/download dialogs. You can also take snapshots of the browser at any point.
You can wrap the framework inside whatever runner you prefer. I've used NUnit and MbUnit without trouble.
There's also an option for a support package if you need help with your automation.
(Disclosure: I work for Telerik as their Test Studio evangelist)
Can I run Coded-UI Tests from Windows Froms Applications? Looking for a sample.
Thanks.
Of course you can - you can use mstest.exe command line application for running Coded UI Tests. You just need to write simple algorithm ensuring that correct parameters will be passed while launching mstest. For reference go here
You should have a look at NUnitForms that is a extension to NUnit.
Your NUnit tests can open a window and interact with the controls.
Your tests will automatically manipulate and verify the properties of
the gui. NUnitForms takes care of cleaning up your forms between
tests, detecting and handling modal dialog boxes, and verifying that
your expectations for the test are fulfilled.
Reference here
see also
Unit testing Winforms UI
How to unit test winforms applications
Unit Test to verify that WinForms application doesn't load Assembly more than once
Basically you will better off separating your logic completely from the UI , making your UI as thin as possible, and testing the logic separately .
You can find solution here for console application. You have to add additional references in order to work test in windows forms application or other project(I had to add reference to Microsoft.VisualStudio.TestTools.UITest.WindowsStoreUtility other than the listed refrences in the article). But as explained in this so answer it would be nice if you can use it in a project that is meant to use these assemblies.