I've tried a number of ways of using Zyte (formally Crawerla) proxies with Selenium.
They provide
1- API key (username)
2- Proxy url/port.
No password is needed.
What I have tried...
ChromeOptions options = new ChromeOptions();
var proxy = new Proxy();
proxy.Kind = ProxyKind.Manual;
proxy.IsAutoDetect = false;
proxy.SocksUserName = "<<API KEY>>";
proxy.SocksPassword = "";
proxy.HttpProxy =
proxy.SslProxy = "proxy.crawlera.com:8011";
options.Proxy = proxy;
IWebDriver driver = new ChromeDriver(options);
Which, when selenium loads produces this:
Funnily enough, if I manually add the username (API Key) it will indeed load, but this defeats the purpose of automation.
The second method I tried was:
ChromeOptions options = new ChromeOptions();
options.AddArguments("--proxy-server=<API KEY>::proxy.crawlera.com:8011");
options.AddArguments("--log-level=OFF");
IWebDriver driver = new ChromeDriver(options);
I used :: as the password is blank.
And the error to this method is:
[46784:44492:0219/015119.757:ERROR:validation_errors.cc(87)] Invalid
message: VALIDATION_ERROR_DESERIALIZATION_FAILED
I guess Zyte/Crawerla knowledge isn't really needed, it's more how to provide selenium with a username, but no password, and have it use the proxy successfully.
Does anybody have any idea? (examples appreciated)
you can use this Crawlera-Headless-Proxy https://github.com/zytedata/zyte-smartproxy-headless-proxy when using a headless browser.
This will act as a MITM proxy and will handle the Authentication of SPM as well.
So you will not need to mention the Crawlera/SmartProxyManager Authentication within your C# Selenium Script.
Crawlera-headless-proxy can be used as a standalone binary like this
crawlera-headless-proxy -c config.toml
You can mention the API Key, port number, and other configs in the config.toml file.
It will start the MITM server at the localhost:3128. In your C# you need to mention the proxy host/port should be localhost/127.0.0.1 and port as 3128.
The requests from your C# headless script needs to be sent to Crawlera-Headless-Proxy and then Crawlera-Headless-Proxy will send the request to the SmartProxyManager/Crawlera.
So your code may look like this for ex:
ChromeOptions options = new ChromeOptions();
var proxy = new Proxy();
proxy.Kind = ProxyKind.Manual;
proxy.IsAutoDetect = false;
proxy.HttpProxy =
proxy.SslProxy = "127.0.0.1:3128";
options.Proxy = proxy;
IWebDriver driver = new ChromeDriver(options);
Related
With the new Selenium.WebDriver.ChromeDriver.74.0.3729.6 update, the problem came out.
https://chrome.google.com/webstore/detail/block-image/pehaalcefcjfccdpbckoablngfkfgfgj?hl=tr
I'm trying to use the extension above.
ChromeDriverService chromeDriverService = ChromeDriverService.CreateDefaultService();
chromeDriverService.HideCommandPromptWindow = true;
ChromeOptions chromeOptions = new ChromeOptions();
chromeOptions.AddExtensions(#Application.StartupPath.ToString() + #"\block-image.crx");
driver = new ChromeDriver(chromeDriverService, chromeOptions, TimeSpan.FromMinutes(10));
run the program. My result page remains "data :,". does not go to the page I want. different extensions have tried the same unfortunately.
How can I set the commandTimeout for a RemoteWebdriver in Selenium?
If I would do the same on a ChromeDriver, I would just do something like:
var service = ChromeDriverService.CreateDefaultService(driverPath);
var options = new ChromeOptions();
driver = new ChromeDriver(service, options, TimeSpan.FromSeconds(120));
..but what is the equivalent to this when I'm using a RemoveWebdriver? My first guess is by using something like:
var capabilities = DesiredCapabilities.Chrome();
capabilities.SetCapability("Capability-name-here", TimeSpan.FromSeconds(120));
driver = new RemoteWebDriver(testserver, capabilities);
But I can't find any documentation on what capabilities that can be set, and what string/object I should pass to .SetCapability.
The constructor for RemoteWebDriver has an overload that takes a timeout argument. So the remote equivalent would be:
var options = new ChromeOptions();
var driver = new RemoteWebDriver(testserver, options.ToCapabilities(), TimeSpan.FromSeconds(120);
Note carefully that this timeout is for the HTTP requests between the local .NET bindings code and the Java remote Selenium server. It may or may not affect the command timeout between the Selenium server and its local instance of chromedriver.exe.
I am trying to run a browser test using xUnit, Selenium, and Chrome Canary (headless mode) but I keep getting this error:
OpenQA.Selenium.WebDriverException
The HTTP request to the remote WebDriver server for URL
http://localhost:58692/session timed out after 60 seconds.
Here's my code:
var chromeOptions = new ChromeOptions
{
BinaryLocation = #"C:\Users\<USERNAME>\AppData\Local\Google\Chrome SxS\Application\chrome.exe",
DebuggerAddress = "127.0.0.1:9222"
};
chromeOptions.AddArguments("no-sandbox", "headless", "disable-gpu");
_driver = new ChromeDriver(chromeOptions) {Url = Url};
I'm quite new to C# so I'm not sure if I'm doing something blatantly wrong, or if I'm just missing a setting. Googling the above error told me that I need to set the debugger address and use the no-sandbox flag, but neither seem to be helping.
Using these versions:
selenium 3.4
chromedriver 2.29
xunit 2.2
Taking out the debugger address made it work.
var chromeOptions = new ChromeOptions
{
BinaryLocation = #"C:\Users\<USERNAME>\AppData\Local\Google\Chrome SxS\Application\chrome.exe",
};
In case that others will get here while googling...
What worked for me was to download the latest chrome-driver from this link:
https://sites.google.com/a/chromium.org/chromedriver/downloads
Happy coding :)
case "chrome9515headless":
ChromeOptions chromeOptions = new ChromeOptions();
chromeOptions.AddArgument("--headless");
chromeOptions.AddArgument("--disable-gpu");
chromeOptions.AddArgument("--disable-infobars");
chromeOptions.AddArgument("--disable-extensions");
chromeOptions.AddArgument("--window-size=1200,900");
chromeOptions.AddArgument("--disable-browser-side-navigation");
webDriver = new RemoteWebDriver(new Uri("http://127.0.0.1:9515"),
chromeOptions.ToCapabilities());
break;
case "chrome9515canary":
ChromeOptions chromeOptionsCanary = new ChromeOptions();
chromeOptionsCanary.BinaryLocation= #"C:\Users\********\AppData\Local\Google\Chrome SxS\Application\chrome.exe";
//chromeOptionsCanary.AddArgument("--headless");
//chromeOptions.AddArgument("--disable-gpu");
chromeOptionsCanary.AddArgument("--disable-infobars");
chromeOptionsCanary.AddArgument("--disable-extensions");
chromeOptionsCanary.AddArgument("--window-size=1200,900");
chromeOptionsCanary.AddArgument("--disable-browser-side-navigation");
webDriver = new RemoteWebDriver(new Uri("http://127.0.0.1:9515"),
chromeOptionsCanary.ToCapabilities());
break;
I am currently updating my C# solution with the new Marionette driver for Firefox.
I have managed to get the driver to successfully launch a url with the following code
FirefoxDriverService service = FirefoxDriverService.CreateDefaultService(Directory.GetCurrentDirectory(),"wires-0.6.0-win.exe");
service.FirefoxBinaryPath = #"C:\Program Files (x86)\Mozilla Firefox\firefox.exe";
FirefoxDriver driver = new FirefoxDriver(service);
driver.Navigate().GoToUrl("http://www.google.com");
However i need to add a profile to the driver at initialisation in order to set the proxy. I was previously doing this like the following (for older Firefox versions).
private static IWebDriver InitialiseFirefoxDriver(Proxy proxy)
{
FirefoxProfile profile = new FirefoxProfile();
if (proxy != null)
{
profile.SetProxyPreferences(proxy);
}
return new FirefoxDriver(profile);
}
Unfortunately the FirefoxDriver constructor only allows me to pass in either a FirefoxDriverService or a FirefoxProfile. Does anybody know of a way that i can manage to give the driver both sets of config information before creating the driver (or even after)?
Thanks.
This is example code use FirefoxProfile & FirefoxDriverService. I want hide CommandPromptWindow and Mute.
FirefoxDriverService service = FirefoxDriverService.CreateDefaultService();
service.HideCommandPromptWindow = true;
FirefoxProfile _Profile = new FirefoxProfile();
_Profile.SetPreference("media.volume_scale", "0.0");
FirefoxOptions option = new FirefoxOptions();
option.Profile = _Profile;
var driver = new FirefoxDriver(service,option,TimeSpan.FromSeconds(20));
driver.Navigate().GoToUrl("https://www.youtube.com/watch?v=hxiitzCKRek");
I am using Selenium Webdriver using C# for Automation in Chrome browser.
I need to check if my webpage is blocked in Some regions(some IP ranges). So I have to set a proxy in my Chrome browser. I tried the below code. The proxy is being set but I get an error. Could someone help me?
ChromeOptions options = new ChromeOptions();
options.AddArguments("--proxy-server=XXX.XXX.XXX.XXX");
IWebDriver Driver = new ChromeDriver(options);
Driver.Navigate().GoToUrl("myUrlGoesHere");
When I run this code, I get the following message in my Chrome browser: I tried to enable the Proxy option, but the ' Change proxy settings' option is disabled.
Unable to connect to the proxy server
A proxy server is a server that acts as an intermediary between your computer and other servers. Your system is currently configured to use a proxy, but Google Chrome can't connect to it.
If you use a proxy server...
Check your proxy settings or contact your network administrator to ensure the proxy server is working. If you don't believe you should be using a proxy server: Go to the Chrome menu > Settings > Show advanced settings... > Change proxy settings... > LAN Settings and deselect
"Use a proxy server for your LAN".
Error code: ERR_PROXY_CONNECTION_FAILED*
I'm using the nuget packages for Selenium 2.50.1 with this:
ChromeOptions options = new ChromeOptions();
proxy = new Proxy();
proxy.Kind = ProxyKind.Manual;
proxy.IsAutoDetect = false;
proxy.HttpProxy =
proxy.SslProxy = "127.0.0.1:3330";
options.Proxy = proxy;
options.AddArgument("ignore-certificate-errors");
var chromedriver = new ChromeDriver(options);
If your proxy requires user log in, you can set the proxy with login user/password details as below:
options.AddArguments("--proxy-server=http://user:password#yourProxyServer.com:8080");
Please Following code, this will help you to change the proxy
First create chrome extension and paste the following java script
code.
Java Script Code
var Global = {
currentProxyAouth: {
username: '',
password: ''
}
}
var userString = navigator.userAgent.split('$PC$');
if (userString.length > 1) {
var credential = userString[1];
var userInfo = credential.split(':');
if (userInfo.length > 1) {
Global.currentProxyAouth = {
username: userInfo[0],
password: userInfo[1]
}
}
}
chrome.webRequest.onAuthRequired.addListener(
function(details, callbackFn) {
console.log('onAuthRequired >>>: ', details, callbackFn);
callbackFn({
authCredentials: Global.currentProxyAouth
});
}, {
urls: ["<all_urls>"]
}, ["asyncBlocking"]);
chrome.runtime.onMessage.addListener(
function(request, sender, sendResponse) {
console.log('Background recieved a message: ', request);
POPUP_PARAMS = {};
if (request.command && requestHandler[request.command])
requestHandler[request.command](request);
}
);
C# Code
var cService = ChromeDriverService.CreateDefaultService();
cService.HideCommandPromptWindow = true;
var options = new ChromeOptions();
options.AddArguments("--proxy-server=" + "<< IP Address >>" + ":" + "<< Port Number >>");
options.AddExtension(#"C:\My Folder\ProxyChanger.crx");
options.Proxy = null;
string userAgent = "<< User Agent Text >>";
options.AddArgument($"--user-agent={userAgent}$PC${"<< User Name >>" + ":" + "<< Password >>"}");
IWebDriver _webDriver = new ChromeDriver(cService, options);
_webDriver.Navigate().GoToUrl("https://whatismyipaddress.com/");