Develop Reference

How to start driver with path, service and options

I work with driver´s for Chrome, Firefox and Edge in my Application and have always the same problem.
So for this post i reduce my request to the Chrome driver as an example.
What is needed:
Path to the chromedriver.exe
Hide the console
Change of the default Download directory of the browser
Here is a Page that show us the Chrome Driver Class and its options:
https://www.selenium.dev/selenium/docs/api/dotnet/html/T_OpenQA_Selenium_Chrome_ChromeDriver.htm
But my 3 needed Points are not listed together.
Here is a snipped of my Code with Commends:
string str_DriverPath = #"C:\_MT5_TOOLS\DRIVER\CHROME";
// hide Console
ChromeDriverService service = ChromeDriverService.CreateDefaultService();
service.HideCommandPromptWindow = true;
//// change Standard-Download-Path
ChromeOptions options = new ChromeOptions();
var downloadDirectory = GlobalVars.RootPath + #"Pool\" + GlobalVars.strSymbol + #"\" + GlobalVars.strSymbol + #"_" + GlobalVars.strPeriod;
options.AddUserProfilePreference("download.default_directory", downloadDirectory);
options.AddUserProfilePreference("download.prompt_for_download", false);
options.AddUserProfilePreference("disable-popup-blocking", "true");
// Start Driver:
//webdriver = new ChromeDriver(service, options); // works fine
//webdriver = new ChromeDriver(str_DriverPath, options); // works fine
webdriver = new ChromeDriver(str_DriverPath, service, options); // will not work
How to combine my 3 Points into one driver?

After trying every possible thing around the environment variable "PATH" to set the driver path, it seems to be impossible in 2022.
From manually until coded this still not work for me!
Maybe this was a trick around in the past...
Solution:
On another portal I found a really nice and easy working solution!
The second Line in the following Code does the Trick.
It is just the service, where the PATH to the driver can be placed in.
This works identical for Edge, Chrome, and Firefox.
// DriverService with Path to driver.exe
ChromeDriverService service = ChromeDriverService.CreateDefaultService(#"C:\_MT5_TOOLS\DRIVER\CHROME");
// hide driver Console? true/false
service.HideCommandPromptWindow = true;
// change Standard-Download-Path
ChromeOptions options = new ChromeOptions();
var downloadDirectory = GlobalVars.RootPath + #"Pool\" + GlobalVars.strSymbol + #"\" + GlobalVars.strSymbol + #"_" + GlobalVars.strPeriod;
options.AddUserProfilePreference("download.default_directory", downloadDirectory);
options.AddUserProfilePreference("download.prompt_for_download", false);
options.AddUserProfilePreference("disable-popup-blocking", "true");
// Selenium Driver starten:
webdriver = new ChromeDriver(service, options);

Crawlera/Zyte proxy authentication using C# and Selenium

I've tried a number of ways of using Zyte (formally Crawerla) proxies with Selenium.
They provide
1- API key (username)
2- Proxy url/port.
No password is needed.
What I have tried...
ChromeOptions options = new ChromeOptions();
var proxy = new Proxy();
proxy.Kind = ProxyKind.Manual;
proxy.IsAutoDetect = false;
proxy.SocksUserName = "<<API KEY>>";
proxy.SocksPassword = "";
proxy.HttpProxy =
proxy.SslProxy = "proxy.crawlera.com:8011";
options.Proxy = proxy;
IWebDriver driver = new ChromeDriver(options);
Which, when selenium loads produces this:
Funnily enough, if I manually add the username (API Key) it will indeed load, but this defeats the purpose of automation.
The second method I tried was:
ChromeOptions options = new ChromeOptions();
options.AddArguments("--proxy-server=<API KEY>::proxy.crawlera.com:8011");
options.AddArguments("--log-level=OFF");
IWebDriver driver = new ChromeDriver(options);
I used :: as the password is blank.
And the error to this method is:
[46784:44492:0219/015119.757:ERROR:validation_errors.cc(87)] Invalid
message: VALIDATION_ERROR_DESERIALIZATION_FAILED
I guess Zyte/Crawerla knowledge isn't really needed, it's more how to provide selenium with a username, but no password, and have it use the proxy successfully.
Does anybody have any idea? (examples appreciated)

you can use this Crawlera-Headless-Proxy https://github.com/zytedata/zyte-smartproxy-headless-proxy when using a headless browser.
This will act as a MITM proxy and will handle the Authentication of SPM as well.
So you will not need to mention the Crawlera/SmartProxyManager Authentication within your C# Selenium Script.
Crawlera-headless-proxy can be used as a standalone binary like this
crawlera-headless-proxy -c config.toml
You can mention the API Key, port number, and other configs in the config.toml file.
It will start the MITM server at the localhost:3128. In your C# you need to mention the proxy host/port should be localhost/127.0.0.1 and port as 3128.
The requests from your C# headless script needs to be sent to Crawlera-Headless-Proxy and then Crawlera-Headless-Proxy will send the request to the SmartProxyManager/Crawlera.
So your code may look like this for ex:
ChromeOptions options = new ChromeOptions();
var proxy = new Proxy();
proxy.Kind = ProxyKind.Manual;
proxy.IsAutoDetect = false;
proxy.HttpProxy =
proxy.SslProxy = "127.0.0.1:3128";
options.Proxy = proxy;
IWebDriver driver = new ChromeDriver(options);

C# Selenium Firefox Set socks proxy with Auth

In C# I'm trying to set socks proxy with authentification in firefox.
This doesn't work
Proxy proxy = new Proxy();
proxy.SocksProxy = sProxyIP + ":" + sProxyPort;
proxy.SocksUserName = sProxyUser;
proxy.SocksPassword = sProxyPass;
options.Proxy = proxy;
_driver = new FirefoxDriver(service, options);
This doesn't work too
profile.SetPreference("network.proxy.socks", sProxyUser + ":" + sProxyPass + "#" + sProxyIP + ":" + sProxyPort);
profile.SetPreference("network.proxy.socks_port", sProxyPort);
How can I solve this?

As far as I know you can't do it in that way with Firefox. You need to add a new Firefox profile and then to work with it with Selenium. On this profile you need to save the proxy information and to save the username and password.
You can set a Firefox profile following this steps link. Then it is easy to do the job. I use that code:
FirefoxProfile profile = new FirefoxProfile(pathToProfile);
FirefoxOptions options = new FirefoxOptions();
options.Profile = profile;
driver = new FirefoxDriver(options);
Then you would have to suppress the alert for username and password verification. You can use two ways of doing that. The first one is to do it programmatically. Something like that:
var alert = driver.SwitchTo().Alert();
alert.Accept();
The other way is to do that from Firefox Profile Settings.

After a lot of researching, here is the solution I decided to go with.
It's a dirty hack, but it works.
I used AutoitX in order to automate the proxy auth window but had to use System.Windows.Automation in order to get the right auth window since my app will be multithreaded.
sProxyIP = "154.5.5.5";
sProxyUser = "user here";
sProxyPass = "pass here";
sProxyPort = 4444;
//Set proxy
profile.SetPreference("network.proxy.socks", sProxyIP);
profile.SetPreference("network.proxy.socks_port", sProxyPort);
//deal with proxy auth
_driver.Manage().Timeouts().PageLoad = TimeSpan.FromMilliseconds(0);
WebsiteOpen(#"https://somewebsite.com/");
AuthInProxyWindow(sProxyUser, sProxyPass);
_driver.Manage().Timeouts().PageLoad = TimeSpan.FromSeconds(60);
void ProxyAuthWindow(string login, string pass)
{
try
{
//wait for the auth window
var sHwnd = AutoItX.WinWait("Authentication Required", "", 2);
AutoItX.WinSetOnTop("Authentication Required", "", 1);
//we are using Windows UIA so we make sure we got the right auth
//dialog(since there will be multiple threads we can easily hit the wrong one)
var proxyWindow = AutomationElement.RootElement.FindFirst(TreeScope.Subtree,
new PropertyCondition(AutomationElement.ClassNameProperty, "MozillaDialogClass"));
string hwnd = "[handle:" + proxyWindow.Current.NativeWindowHandle.ToString("X") + "]";
AutoItX.ControlSend(hwnd, "", "", login, 1);
AutoItX.ControlSend(hwnd, "", "", "{TAB}", 0);
AutoItX.ControlSend(hwnd, "", "", pass, 1);
AutoItX.ControlSend(hwnd, "", "", "{ENTER}", 0);
}
catch
{
}
}

Provide username and password for proxy - Selenium

I'm trying the below code. But its still giving dialog box for entering username and password when firefox browser starts. Where am I wrong?
FirefoxProfile profile = new FirefoxProfile();
Proxy firefox_proxy = new Proxy();
firefox_proxy.HttpProxy = proxy;
firefox_proxy.SslProxy = proxy;
profile.SetProxyPreferences(firefox_proxy);
Firefoxdriver driver = new FirefoxDriver(new FirefoxBinary(), profile, TimeSpan.FromMinutes(3));
driver.Navigate().GoToUrl("http://" + proxy_username + ":" + proxy_password + "#www.xyz.com/");

You should try with https:// instead of http:// as on some sites the Basic Authentication works on secure network only.
Syntax: driver.Navigate().GoToUrl("https://proxy_username:proxy_password#www.xyz.com/");

C# Selenium WebDriver FireFox Profile - using proxy with Authentication

When you set proxy server parameter in the code below if your proxy server requires authentication then FireFox will bring Authentication dialog and basically you can't fill it in automatically.
So is there is anyway to set USERNAME and PASSWORD ?
FirefoxProfile profile = new FirefoxProfile();
String PROXY = "192.168.1.100:8080";
OpenQA.Selenium.Proxy proxy = new OpenQA.Selenium.Proxy();
proxy.HttpProxy=PROXY;
proxy.FtpProxy=PROXY;
proxy.SslProxy=PROXY;
profile.SetProxyPreferences(proxy);
FirefoxDriver driver = new FirefoxDriver(profile);
If you try to format proxy string to something like that http://username:pass#192.168.1.1:8080
You get error that string is invalid. So I wonder there is must be a way of achieving this.
Any help would be appreciated.

String PROXY = "http://login:pass#proxy:port";
ChromeOptions options = new ChromeOptions();
options.AddArguments("user-data-dir=path/in/your/system");
Proxy proxy = new Proxy();
proxy.HttpProxy = PROXY;
proxy.SslProxy = PROXY;
proxy.FtpProxy = PROXY;
options.Proxy = proxy;
// Initialize the Chrome Driver
using (var driver = new ChromeDriver(options))

You can write own firefox extension for proxy, and launch from selenium. You need write 2 files and pack it.
background.js
var proxy_host = "YOUR_PROXY_HOST";
var proxy_port = YOUR_PROXY_PORT;
var config = {
mode: "fixed_servers",
rules: {
singleProxy: {
scheme: "http",
host: proxy_host,
port: proxy_port
},
bypassList: []
}
};
function proxyRequest(request_data) {
return {
type: "http",
host: proxy_host,
port: proxy_port
};
}
browser.proxy.settings.set({value: config, scope: "regular"}, function() {;});
function callbackFn(details) {
return {
authCredentials: {
username: "YOUR_USERNAME",
password: "YOUR_PASSWORD"
}
};
}
browser.webRequest.onAuthRequired.addListener(
callbackFn,
{urls: ["<all_urls>"]},
['blocking']
);
browser.proxy.onRequest.addListener(proxyRequest, {urls: ["<all_urls>"]});
manifest.json
{
"name": "My Firefox Proxy",
"version": "1.0.0b",
"manifest_version": 2,
"permissions": [
"browsingData",
"proxy",
"storage",
"tabs",
"webRequest",
"webRequestBlocking",
"downloads",
"notifications",
"<all_urls>"
],
"background": {
"scripts": ["background.js"]
},
"browser_specific_settings": {
"gecko": {
"id": "myproxy#example.org"
}
}
}
Next you need packed this files to zip archive in DEFLATED mode with .xpi at end like my_proxy_extension.xpi.
You have two choices:
Sign your extension Here you can read more about verify extension and extension's structure
OR
Run unsigned. For this step:
Open firefox flags at about:config and set options xpinstall.signatures.required to false
OR
Update firefox profile in:
Windows: C:\Program Files\Mozilla Firefox\defaults\pref\channel-prefs.js
Linux: /etc/firefox/syspref.js
Add next line to end of file:
pref("xpinstall.signatures.required",false);
After this steps run selenium and install this extension:
FirefoxProfile profile = new FirefoxProfile();
profile.addExtension(new File("path/to/my_proxy_extension.xpi"));
driver = new FirefoxDriver(profile);

What you can do is to create a profile and save the authentication data in it.
If your profile is called "webdriver" you can select it from your code in the initialization:
ProfilesIni allProfiles = new ProfilesIni();
FirefoxProfile profile = allProfiles.getProfile("WebDriver");
profile.setPreferences("foo.bar",23);
WebDriver driver = new FirefoxDriver(profile);

Did it with MS UI Automation without AutoIt:
public void AuthInProxyWindow (string login, string pass)
{
var proxyWindow = AutomationElement.RootElement
.FindFirst(TreeScope.Subtree,
new PropertyCondition(AutomationElement.ClassNameProperty, "MozillaDialogClass"));
var edits = proxyWindow.FindAll(TreeScope.Subtree,
new PropertyCondition(AutomationElement.ControlTypeProperty, ControlType.Edit));
var unamePoint = edits[1].GetClickablePoint();
Mouse.MoveTo(new Point((int) unamePoint.X, (int) unamePoint.Y));
Mouse.Click(MouseButton.Left);
SendKeys.SendWait(login);
var pwdPoint = edits[2].GetClickablePoint();
Mouse.MoveTo(new Point((int) pwdPoint.X, (int) pwdPoint.Y));
Mouse.Click(MouseButton.Left);
SendKeys.SendWait(pass);
Keyboard.Press(Key.Return);
Logger.Debug("Authefication in Firefox completed succesfully");
}
Mouse moves by Microsoft.TestApi

To stop firefox from giving you the auth pop up simple make sure you set your proxy URL to include the auth details in the setup stage as below:
var myProxy = user + ":" + pass + "#" + proxyIP + ":" + proxyPORT;
options.SetPreference("network.proxy.type", 1);
options.SetPreference("network.proxy.http", myProxy);
options.SetPreference("network.proxy.http_port", proxyPORT);
options.SetPreference("general.useragent.override", useragent);
driver = new FirefoxDriver(driverService, options);