I have been researching on how to automate browser testing for a number of weeks now using all kind of different methods. Seleniums website is very vague on which is the best route to take.
I have installed
Selenium Webdriver
Remote Control
Selenium Grid
Apache Ant
Nunit
(and pretty much everything else you could need to do this)
I finally give up trying on my own and want to know the best way to do this.
I want to be able to
Load the same webpage on a number of different browsers
Load the same webpage on a number of virtual machines(which I have set up)
Be able to take snapshots comparing the different browser results.
I have knowledge of programming in C# and would prefer to run my tests through Nunit.
Can anyone give me directions or point me to a website that already has them?
Thank you.
I have built up a test framework using junit with Selenium WebDriver that satisfies every one of your points. While its not exactly what you're asking for, I feel it may be beneficial to you regardless.
Load the same webpage on a number of different browsers
Using Selenium's grid, this is very simple to set up. Set up some virtual machines with the environments you're looking to test in. In our environment, for example, we have a grid running with four nodes (as virtual machines) with a setup like the following
Windows with IE7 and FireFox
Windows with IE8 and FireFox
Windows with IE9 and Firefox
Linux with FireFox
Note that Selenium recommends that only one instance of IE be allowed to run on the Windows nodes. On each of the aforementioned nodes, there is one instance of the specified IE and five instances of the specified FF allowed to run at any given time. With the grid setup and the hub configured, firing off tests is a breeze. In WebDriver, use the DesiredCapabilities object to set up the desired environment and then just send the test off and wait for the result to return.
Platform desiredPlatform;
DesiredCapabilities desiredCapabilities;
desiredPlatform = Platform.LINUX;
desiredCapabilities = DesiredCapabilities.firefox();
desiredCapabilities.setPlatform(desiredPlatform);
desiredCapabilities.setVersion("11");
WebDriver driver = new RemoteWebDriver("http://hubURL", desiredCapabilities);
Load the same webpage on a number of virtual machines(which I have set up)
I solved this one by forcing the tests to run, albeit in an unconvential way, in a threaded manner. Each JUnit test uses a shared thread library I put together which creates all the necessary RemoteWebDrivers needed in separate threads. Each of these threads runs simultaneously on its node while the parent thread sits and waits for all to terminate. Then on to the next test which is run multithreaded as well.
There were a couple problems I encountered such as retrieving the Junit stack traces in all of the child threads. I solved this by redirecting Std.err to a bytestream on the parent thread. All errors get routed to that stream which I then convert to a string and print out to Std.out at the end of each test. The html pages generated at the end of the tests include Std.out which worked out perfectly.
Be able to take snapshots comparing the different browser results
While I have gotten this to work, there are some inherent problems with grabbing screenshots remotely. IE will return black screenshots if the process is running as a service. The workaround was to just run the jar from the command line and keep the user logged in, in which case the screenshots return correctly. This is a known issue in the browser and there really is no nice solution to the problem. Taking screenshots works roughly like this
WebDriver augmentedDriver = new Augmenter().augment(driver);
TakesScreenshot ss = (TakesScreenshot) augmentedDriver;
String base64Screenshot = ss.getScreenshotAs(OutputType.BASE64);
byte[] decodedScreenshot = Base64.decodeBase64(base64Screenshot.getBytes());
FileOutputStream fos = new FileOutputStream(new File(imageName));
fos.write(decodedScreenshot);
fos.close();
which saves the captured screenshot from the remote machine's running browser onto the local machine.
In reality, browser automation is still struggling to stabilize itself. There are a number of important features, such as the ones you're asking about, that just aren't implemented solidly that I know of in any framework. With time, though, I'm sure a lot of this will settle down and QA developers everywhere will rejoice.
As for the 2nd point: instead of using Grid you can let your continuous integration server do the job. In my company we use Jenkins and so called Configuration Matrix - it let's you run the same job on multiple Jenkins nodes.
As for the 1st one, I think Jenkins could be helpful here too. You can run multiple jobs on the same node. Although I've never tried that so I am not perfectly sure. And this is just an idea, I wouldn't really recommend such solution. You may also want to read this blog post describing how to run test in parallel using Selenium Grid. For people using Java I would recommend reading about parallel tests with TestNG.
Your third point is a little bit vague. What do you mean by snapshot? And what kind of result you want to compare?
Selenium RC is outdated and webdriver is more reliable way of creating selenium tests. I see the responses above cater more on java side. Below mentioned is more information on how to achieve the questions asked here using C# and selenium webdriver
On how to setup the IDE (VS express), nUnit and selenium refer
How to setup C#,nUnit and selenium client drivers on VSExpress for Automated tests
On Creating simple script that launches a browser does few steps refer
Creating Basic Selenium web driver test case using Nunit and C#
On how to Load the same webpage on a number of different browsers suggest referring
How to invoke locally different types of browser driver using selenium and c#
On Load the same webpage on a number of virtual machines(which I have set up) for this, you need to use Remote webdriver instead of normal webdriver. Also with remote webdriver, you can launch different types of browser. Refer this webpage
How to invoke/run different type of web driver browser using remote webdriver in C#
To take snapshot on different browser you can refer the link
Capturing screen shots using remote/local webdriver in C#/Selenium webdriver
You might also consider the free Telerik Testing Framework. This is the underpinning for Telerik's commercial Test Studio product. The Testing Framework provides cross-browser support, does a great job with dynamic content situations (AJAX), and also lets you handle OS-level dialogs like file upload/download dialogs. You can also take snapshots of the browser at any point.
You can wrap the framework inside whatever runner you prefer. I've used NUnit and MbUnit without trouble.
There's also an option for a support package if you need help with your automation.
(Disclosure: I work for Telerik as their Test Studio evangelist)
Related
So this has me quite mixed up.
We have a testing framework using Selenium, Specflow and C#. Currently running Selenium 3.13.1 (Just upgraded yesterday to see if the older version (3.11.2) was the issue)
The issue (only when Emulating a mobile device in Chrome):
When running tests locally all is well, buttons are clicked and the test passes.
When running tests remotely on the Jenkins slaves it works some of the times .i.e. sometimes I get a click and sometimes I do not.
When the Jobs are run as per schedule some of the tests fail consistently (these being tests that run perfectly well when run locally).
I have tried adding a Sleep of 5 seconds, however this did not help.
One thing that did work is changing Click() to SendKeys(Keys.Enter).
Executing javascript to click the button works well too however I do not wish to change from click for 2 reasons. 1. This is the closest action which simulates user input and 2. This works well locally.
Chrome versions on the Jenkins slaves are in line with my local version (67.0.3396.99) and Chrome driver is also inline (2.40.565498).
Chrome driver has been updated yesterday since I have tried with version 2.38 and 2.39 but to no avail.
Has anyone come across this issue and found a solution?
P.S. When running tests not in mobile emulation I have no issues with clicks both locally or on Jenkins.
i had the same problem.
After I allowed PopUps in my Browser the click and clickAndWait Event works fine.
Maybe you can change the browser too.
Best Regards
For anyone who has this issue, this is the current relevant answer at this point in time.
This was not an issue with the framework we are using but an issue with chromium and chrome driver.
The issue/bug link can be found here (this is the known issue quoted below).
Also if you check the documentation for chrome driver mobile emulation, it is mentioned on the first line as follows
Note that Mobile Emulation is subject to this known issue.
This answer does not help solve the issue per se but gives the reason why it is happening. The "best" workaround I have found is using JS to click and so on. In our case we stopped using mobile emulation for now since clicking with JS does not really emulate a real click. As new versions of Chrome and ChromeDriver are realeased I will keep updating until there is a fix for it.
Hopefully you will waste less time than I have searching around for an answer :)
The problem I am facing is a unique one.. I have set up a testing suite for a windows app using LeanFT and NUnit. At this point, I have around 100 stable UI tests that I have automatically running nightly on a VM I have configured.
The problem is, I do not think there is an out of the box solution for running tests in parallel across multiple machines. When the tests are kicked off of TFS, they run on a single machine. What I want is for tests to be passed out 1 at a time to any available machine.
I know I can "fake" this in a sense by having two different build definitions run at the same time with a different set of tests, but this is not what I want to do. I want it to work like an actual grid and pass out tests like the selenium grid does. Has anyone had any luck with something like this by not faking it?
Note: we do not use Jenkins, we do everything out of TFS.
Have you looked at https://blogs.msdn.microsoft.com/devops/2016/10/10/parallel-test-execution/ ?
Team Foundation Server does have 'build agents' like Jenkins does. While I'm not familiar with this, it appears to be close to what you are looking for.
Currently I am in process of building an automation tool for testing webpages. Already aware of selenium tool but will not be using that currently as our framework has already been built and requires minor changes to make it reliable. While testing this framework with test pages (html and javascript only) I encounter issues such as webpage takes a lot if time to load(happens like 1 out of 20 times). And when you try to find the co-ordinate and click the button or try to find the element in the webpage and click it sometimes it fails as button doesn't even exists at that point of time.
Currently using Thread.sleep or retry n number of times. Are there any better solutions to remove this flaky behavior?
look into WebDriverWait class. There is a respective binding for c# as well. Also, I have discussed the WebDriverWait here.
You can try to use Implicit waits
Read about it here http://www.seleniumhq.org/docs/04_webdriver_advanced.jsp
Basically you set it once per session. If you can't find an element, selenium waits the amount of time you set before throwing the exception.
I have been playing around with testing a simple webforms ASP.NET website application and have come across the UIMap tool in Visual Studio 2010. However I do not know how to use it correctly and it never seems to work.
Do I need to be running my application in debug mode while recording?
Once the test is generated is there any way of visually seeing the steps as they are being performed?
Should I have the browser open before I begin recording, or should I do that once I start and browse to the page that way?
Can I use Chrome for the browser, or does it have to be IE?
Does using Dual Screens affect the running of the test?
Thanks for your help!
Do I need to be running my application in debug mode while recording?
No. You can not use debug/release mode of Visual Studio and do an record at same time. Start the application separately. This has the big advantage that you can test different versions of our application with one set of tests.
Once the test is generated is there any way of visually seeing the
steps as they are being performed?
No. You can get a log from test-run and of course you see the test-run in self, but there is no static graphic ... to be honest I never miss such feature.
Should I have the browser open before I begin recording, or should I
do that once I start and browse to the page that way?
That you can do as you please. Of course you can record the browser start. We start browser before, because browser start is not really a part of the test.
Can I use Chrome for the browser, or does it have to be IE?
At the moment IE is the forced browser. But there are ways to use other browsers.
Does using Dual Screens affect the running of the test?
No. All fine with dual screen. But don't forget when test is running you can't use your mouse and keyboard - you will fight the CUIT ;). We use a separate PC for long CUIT.
Here are some links which may be helpful for you:
should the coded ui test project share a solution or not?
Unit Testing Frameworks for Visual Studio 2012 Cons/Pros
Which Unit Test framework to use for the projects that requires User Input
I have some YUI tests which I need them to run headless. Currently, these tests are run by launching their corresponding TestFileName.html. Upon launching, the browser shows passed or failed tests on the screen with green and red icons and their corresponding messages. During this process, the machine is unusable because the browser's UI keeps popping up and down.
I am trying to make the test run headless. For that I created a WebBrowser (from .NET) control in memory and launched the page in it. But, this way I can not see the UI and determine if the tests passed of failed. I need to log the success/failure and corresponding messages in a log file in file system.
I am not sure how to do so. Can someone please tell me what can I do to achieve headless execution of YUI and creating the logs?
Thanks
You might want to look at Phantom.js. It's a head-less version of WebKit (ie. Safari and Chrome), and people use it all the time for running head-less JS tests, so you can probably find a lot more information out there for it than for that .NET thing you mentioned.
YUI provides a Node.js command line tool named "grover" that does exactly this: https://github.com/yui/grover
Use npm to install grover and then run 'grover paths/to/test/files.html ...' You can also use grover to generate code coverage reports.