I'm using .NET Core 6 with C# 10.
What I'm trying to achieve is to run Selenium in headless mode whilst remaining "undetectable". I followed the instructions from here: https://intoli.com/blog/not-possible-to-block-chrome-headless/ which provided a page to test your bot: https://intoli.com/blog/not-possible-to-block-chrome-headless/chrome-headless-test.html
Headless mode causes some JS vars (like window.chrome) to be unset or invalid which causes the bot to be detected.
IJavaScriptExecutor doesn't work since it runs after the page has loaded. The same author mentions that you have to capture the response and inject JS in this article: https://intoli.com/blog/making-chrome-headless-undetectable/ (Putting It All Together section)
Since the article uses python, I followed this: https://www.automatetheplanet.com/webdriver-capture-modify-http-traffic/ and this: Titanium Web Proxy - Can't modify request body which uses the Titanium Web Proxy library (found here: https://github.com/justcoding121/titanium-web-proxy)
For testing, I used this site http://www.example.com and tried to modify the response (change something in the HTML, set JS vars, etc)
Here is the proxy class:
public static class Proxy
{
static ProxyServer proxyServer = new ProxyServer(userTrustRootCertificate: true);
public static void StartProxy()
{
//Run on port 8080, decrypt ssl
ExplicitProxyEndPoint explicitEndPoint = new ExplicitProxyEndPoint(IPAddress.Any, 8080, true);
proxyServer.Start();
proxyServer.AddEndPoint(explicitEndPoint);
proxyServer.BeforeResponse += OnBeforeResponse;
}
static async Task OnBeforeResponse(object sender, SessionEventArgs ev)
{
var request = ev.HttpClient.Request;
var response = ev.HttpClient.Response;
//Modify title tag in example.com
if (String.Equals(ev.HttpClient.Request.RequestUri.Host, "www.example.com", StringComparison.OrdinalIgnoreCase))
{
var body = await ev.GetResponseBodyAsString();
body = body.Replace("<title>Example Domain</title>", "<title>Completely New Title</title>");
ev.SetResponseBodyString(body);
}
}
public static void StopProxy()
{
proxyServer.Stop();
}
}
And here is the selenium code:
Proxy.StartProxy();
string url = "localhost:8080";
var seleniumProxy = new OpenQA.Selenium.Proxy
{
HttpProxy = url,
SslProxy = url,
FtpProxy = url
};
ChromeOptions options = new ChromeOptions();
options.AddArgument("ignore-certificate-errors");
options.Proxy = seleniumProxy;
IWebDriver driver = new ChromeDriver(#"C:\ChromeDrivers\103\", options);
driver.Manage().Window.Maximize();
driver.Navigate().GoToUrl("http://www.example.com");
Console.ReadLine();
TornCityBot.Proxy.StopProxy();
When selenium loads http://www.example.com, the <title>Example Domain</title> should be changed to <title>Completely New Title</title>, but there was no change. I tried setting the proxy URL as http://localhost:8080, 127.0.0.1:8080, localhost:8080, etc but there was no change.
As a test, I ran the code and left the proxy on. I then ran curl --proxy http://localhost:8080 http://www.example.com in git bash and the output was:
<!doctype html>
<html>
<head>
<title>Completely New Title</title>
. . .
The proxy was working, it was modifying the response for the curl command. But for some reason, it wasn't working with selenium.
If you guys have a solution that can also work on HTTPS or a better method to execute JavaScript on page load, that would be great. If it's not possible, then I might need to forget about headless.
Thanks in advance for any help.
Selenium.WebDriver 4.3.0 and ChromeDriver 103
Try use the ExecuteCdpCommand method
var options = new ChromeOptions();
options.AddArgument("--headless");
options.AddArgument("--user-agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36'");
using var driver = new ChromeDriver(options);
Dictionary<string, object> cmdParams= new();
cmdParams.Add("source", "Object.defineProperty(navigator, 'webdriver', { get: () => false });");
driver.ExecuteCdpCommand("Page.addScriptToEvaluateOnNewDocument", cmdParams);
With this piece of code we bypass the first two but if you follow the guide you've already mentioned i think it's easy to bypass the rest.
UPDATE
var initialScript = #"Object.defineProperty(Notification, 'permission', {
get: function () { return ''; }
})
window.chrome = true
Object.defineProperty(navigator, 'webdriver', {
get: () => false})
Object.defineProperty(window, 'chrome', {
get: () => true})
Object.defineProperty(navigator, 'plugins', {
writeable: true,
configurable: true,
enumerable: true,
value: 'works'})
navigator.plugins.length = 1
Object.defineProperty(navigator, 'language', {
get: () => 'el - GR'});
Object.defineProperty(navigator, 'deviceMemory', {
get: () => 8});
Object.defineProperty(navigator, 'hardwareConcurrency', {
get: () => 8});";
cmdParams.Add("source", initialScript);
driver.ExecuteCdpCommand("Page.addScriptToEvaluateOnNewDocument", cmdParams);
Related
I am using OpenQA.Selenium.Chrome ChromeDriver for automating the browser changes.
As per the application, The URL will only send a response when is the user is login in into the browser, otherwise, it will return 400 Error
I need to identify post-login if the URL exists or not, I am unable to find any function to call a httpGet request from the IWebDriver driver object
IWebDriver driver = new ChromeDriver();
Thanks in advance.
Got a solution using class WebDriverWait that basically can run a javascript method from the current browser instance.
So what I did is calling a nonsynchronous i.e. async = false while raising XMLHttpRequest from javascript like below
return (function () {
{
var result = false;
try {
{
var xhttp = new XMLHttpRequest();
xhttp.open('GET', '<YOUR GET URL HERE>', false); // last param is async = false
xhttp.send();
console.log(xhttp.responseText);
result = !xhttp.responseText.includes('HTTP ERROR 404');
}
} catch (err) {
{}
}
return result;
}
})()
And calling this javascript method on loop till the timeout (TimeSpan is 5000 seconds) from the browser using the WebDriverWait class' config method and casting to IJavaScriptExecutor like below
IWebDriver driver = new ChromeDriver();
TimeSpan timeToWait = TimeSpan.FromSeconds(5000);
WebDriverWait wait1 = new WebDriverWait(driver, timeToWait);
wait1.Until(d =>
{
string url = "<Your GET request URL>";
bool isURLReachable = (bool)((IJavaScriptExecutor)d).ExecuteScript(String.Format(#"return (function() {{ var result = false; try {{ var xhttp = new XMLHttpRequest(); xhttp.open('GET', '{0}', false); xhttp.send(); console.log(xhttp.responseText); result = !xhttp.responseText.includes('HTTP ERROR 404'); }} catch (err) {{ }} return result;}})()", url));
return isURLReachable;
});
This will wait until isURLReachable has true value.
Hope this will help others as well.
I am scraping the web page and navigating to correct location, however as being a new to the whole c# world I am stuck with downloading pdf file.
Link is hiding behind this
var reportDownloadButton = driver.FindElementById("company_report_link");
It is something like: www.link.com/key/489498-654gjgh6-6g5h4jh/link.pdf
How to download the file to C:\temp\?
Here is my code:
using System.Linq;
using OpenQA.Selenium.Chrome;
namespace WebDriverTest
{
class Program
{
static void Main(string[] args)
{
var chromeOptions = new ChromeOptions();
chromeOptions.AddArguments("headless");
// Initialize the Chrome Driver // chromeOptions
using (var driver = new ChromeDriver(chromeOptions))
{
// Go to the home page
driver.Navigate().GoToUrl("www.link.com");
driver.Manage().Timeouts().ImplicitWait = System.TimeSpan.FromSeconds(15);
// Get the page elements
var userNameField = driver.FindElementById("loginForm:username");
var userPasswordField = driver.FindElementById("loginForm:password");
var loginButton = driver.FindElementById("loginForm:loginButton");
// Type user name and password
userNameField.SendKeys("username");
userPasswordField.SendKeys("password");
// and click the login button
loginButton.Click();
driver.Navigate().GoToUrl("www.link2.com");
driver.Manage().Timeouts().ImplicitWait = System.TimeSpan.FromSeconds(15);
var reportSearchField = driver.FindElementByClassName("form-control");
reportSearchField.SendKeys("Company");
var reportSearchButton = driver.FindElementById("search_filter_button");
reportSearchButton.Click();
var reportDownloadButton = driver.FindElementById("company_report_link");
reportDownloadButton.Click();
EDIT:
EDIT 2:
I am not the sharpest pen on Stackoverflow community yet. I don't understand how to do it with Selenium. I have done it with
var reportDownloadButton = driver.FindElementById("company_report_link");
var text = reportDownloadButton.GetAttribute("href");
// driver.Manage().Timeouts().ImplicitWait = System.TimeSpan.FromSeconds(15);
WebClient client = new WebClient();
// Save the file to desktop for debugging
var desktop = System.Environment.GetFolderPath(System.Environment.SpecialFolder.Desktop);
string fileName = desktop + "\\myfile.pdf";
client.DownloadFile(text, fileName);
However web page seems to be a little bit tricky. I am getting
System.Net.WebException: 'The remote server returned an error: (401)
Unauthorized.'
Debugger pointing at:
client.DownloadFile(text, fileName);
I think it should really simulate Right click and Save Link As, otherwise this download will not work. Also if I just click on button, it opens PDF in new Chrome tab.
EDIT3:
Should it be like this?
using System.Linq;
using OpenQA.Selenium.Chrome;
namespace WebDriverTest
{
class Program
{
static void Main(string[] args)
{
// declare chrome options with prefs
var options = new ChromeOptionsWithPrefs();
options.AddArguments("headless"); // we add headless here
// declare prefs
options.prefs = new Dictionary<string, object>
{
{ "download.default_directory", downloadFilePath }
};
// declare driver with these options
//driver = new ChromeDriver(options); we don't need this because we already declare driver below.
// Initialize the Chrome Driver // chromeOptions
using (var driver = new ChromeDriver(options))
{
// Go to the home page
driver.Navigate().GoToUrl("www.link.com");
driver.Manage().Timeouts().ImplicitWait = System.TimeSpan.FromSeconds(15);
// Get the page elements
var userNameField = driver.FindElementById("loginForm:username");
var userPasswordField = driver.FindElementById("loginForm:password");
var loginButton = driver.FindElementById("loginForm:loginButton");
// Type user name and password
userNameField.SendKeys("username");
userPasswordField.SendKeys("password");
// and click the login button
loginButton.Click();
driver.Navigate().GoToUrl("www.link.com");
driver.Manage().Timeouts().ImplicitWait = System.TimeSpan.FromSeconds(15);
var reportSearchField = driver.FindElementByClassName("form-control");
reportSearchField.SendKeys("company");
var reportSearchButton = driver.FindElementById("search_filter_button");
reportSearchButton.Click();
driver.Manage().Timeouts().ImplicitWait = System.TimeSpan.FromSeconds(15);
driver.Navigate().GoToUrl("www.link.com");
// click the link to download
var reportDownloadButton = driver.FindElementById("company_report_link");
reportDownloadButton.Click();
// if clicking does not work, get href attribute and call GoToUrl() -- this may trigger download
var href = reportDownloadButton.GetAttribute("href");
driver.Navigate().GoToUrl(href);
}
}
}
}
}
You could try setting the download.default_directory Chrome driver preference:
// declare chrome options with prefs
var options = new ChromeOptionsWithPrefs();
// declare prefs
options.prefs = new Dictionary<string, object>
{
{ "download.default_directory", downloadFilePath }
};
// declare driver with these options
driver = new ChromeDriver(options);
// ... run your code here ...
// click the link to download
var reportDownloadButton = driver.FindElementById("company_report_link");
reportDownloadButton.Click();
// if clicking does not work, get href attribute and call GoToUrl() -- this may trigger download
var href = reportDownloadButton.GetAttribute("href");
driver.Navigate().GoToUrl(href);
If reportDownloadButton is a link that triggers a download, then the file should download to the filePath you have set in download.default_directory.
Neither of these threads are in C#, but they speak of a similar issue:
How to control the download of files with Selenium + Python bindings in Chrome
How to use chrome webdriver in selenium to download files in python?
You can use WebClient.DownloadFile for that.
I am using Selenium Webdriver using C# for Automation in Chrome browser.
I need to check if my webpage is blocked in Some regions(some IP ranges). So I have to set a proxy in my Chrome browser. I tried the below code. The proxy is being set but I get an error. Could someone help me?
ChromeOptions options = new ChromeOptions();
options.AddArguments("--proxy-server=XXX.XXX.XXX.XXX");
IWebDriver Driver = new ChromeDriver(options);
Driver.Navigate().GoToUrl("myUrlGoesHere");
When I run this code, I get the following message in my Chrome browser: I tried to enable the Proxy option, but the ' Change proxy settings' option is disabled.
Unable to connect to the proxy server
A proxy server is a server that acts as an intermediary between your computer and other servers. Your system is currently configured to use a proxy, but Google Chrome can't connect to it.
If you use a proxy server...
Check your proxy settings or contact your network administrator to ensure the proxy server is working. If you don't believe you should be using a proxy server: Go to the Chrome menu > Settings > Show advanced settings... > Change proxy settings... > LAN Settings and deselect
"Use a proxy server for your LAN".
Error code: ERR_PROXY_CONNECTION_FAILED*
I'm using the nuget packages for Selenium 2.50.1 with this:
ChromeOptions options = new ChromeOptions();
proxy = new Proxy();
proxy.Kind = ProxyKind.Manual;
proxy.IsAutoDetect = false;
proxy.HttpProxy =
proxy.SslProxy = "127.0.0.1:3330";
options.Proxy = proxy;
options.AddArgument("ignore-certificate-errors");
var chromedriver = new ChromeDriver(options);
If your proxy requires user log in, you can set the proxy with login user/password details as below:
options.AddArguments("--proxy-server=http://user:password#yourProxyServer.com:8080");
Please Following code, this will help you to change the proxy
First create chrome extension and paste the following java script
code.
Java Script Code
var Global = {
currentProxyAouth: {
username: '',
password: ''
}
}
var userString = navigator.userAgent.split('$PC$');
if (userString.length > 1) {
var credential = userString[1];
var userInfo = credential.split(':');
if (userInfo.length > 1) {
Global.currentProxyAouth = {
username: userInfo[0],
password: userInfo[1]
}
}
}
chrome.webRequest.onAuthRequired.addListener(
function(details, callbackFn) {
console.log('onAuthRequired >>>: ', details, callbackFn);
callbackFn({
authCredentials: Global.currentProxyAouth
});
}, {
urls: ["<all_urls>"]
}, ["asyncBlocking"]);
chrome.runtime.onMessage.addListener(
function(request, sender, sendResponse) {
console.log('Background recieved a message: ', request);
POPUP_PARAMS = {};
if (request.command && requestHandler[request.command])
requestHandler[request.command](request);
}
);
C# Code
var cService = ChromeDriverService.CreateDefaultService();
cService.HideCommandPromptWindow = true;
var options = new ChromeOptions();
options.AddArguments("--proxy-server=" + "<< IP Address >>" + ":" + "<< Port Number >>");
options.AddExtension(#"C:\My Folder\ProxyChanger.crx");
options.Proxy = null;
string userAgent = "<< User Agent Text >>";
options.AddArgument($"--user-agent={userAgent}$PC${"<< User Name >>" + ":" + "<< Password >>"}");
IWebDriver _webDriver = new ChromeDriver(cService, options);
_webDriver.Navigate().GoToUrl("https://whatismyipaddress.com/");
run tests on Firefox after that on Chrome. Test which run on FF works correctly. Test with the same code but another driver settings(for Chrome) do not work correctly. I have following info from Saucelab's Chrome:
I create driver by that way:
[SetUp]
public void Init()
{
DesiredCapabilities capabillities = DesiredCapabilities.Chrome();
capabillities.SetCapability(CapabilityType.Platform, "Windows 8.1");
capabillities.SetCapability(CapabilityType.Version, "36");
capabillities.SetCapability("name", "R(...)");
capabillities.SetCapability("username", "My username");
capabillities.SetCapability("accessKey", "my acces key value");
driver = new RemoteWebDriver(
new Uri("http://ondemand.saucelabs.com:80/wd/hub"), capabillities);
baseURL = "http://starting address without www";
}
Test fails after one command. The page loaded and after that error that he can not find element. I have tried many posibilities to find elem(by id, css, xpath).
Any idea what am I doing wrong?
The "www" is necessary before web address to correctly runs test in Chrome on Saucelabs.
driver = new RemoteWebDriver(
new Uri("http://ondemand.saucelabs.com:80/wd/hub"), capabillities);
baseURL = "http://www.(...)";
When you set proxy server parameter in the code below if your proxy server requires authentication then FireFox will bring Authentication dialog and basically you can't fill it in automatically.
So is there is anyway to set USERNAME and PASSWORD ?
FirefoxProfile profile = new FirefoxProfile();
String PROXY = "192.168.1.100:8080";
OpenQA.Selenium.Proxy proxy = new OpenQA.Selenium.Proxy();
proxy.HttpProxy=PROXY;
proxy.FtpProxy=PROXY;
proxy.SslProxy=PROXY;
profile.SetProxyPreferences(proxy);
FirefoxDriver driver = new FirefoxDriver(profile);
If you try to format proxy string to something like that http://username:pass#192.168.1.1:8080
You get error that string is invalid. So I wonder there is must be a way of achieving this.
Any help would be appreciated.
String PROXY = "http://login:pass#proxy:port";
ChromeOptions options = new ChromeOptions();
options.AddArguments("user-data-dir=path/in/your/system");
Proxy proxy = new Proxy();
proxy.HttpProxy = PROXY;
proxy.SslProxy = PROXY;
proxy.FtpProxy = PROXY;
options.Proxy = proxy;
// Initialize the Chrome Driver
using (var driver = new ChromeDriver(options))
You can write own firefox extension for proxy, and launch from selenium. You need write 2 files and pack it.
background.js
var proxy_host = "YOUR_PROXY_HOST";
var proxy_port = YOUR_PROXY_PORT;
var config = {
mode: "fixed_servers",
rules: {
singleProxy: {
scheme: "http",
host: proxy_host,
port: proxy_port
},
bypassList: []
}
};
function proxyRequest(request_data) {
return {
type: "http",
host: proxy_host,
port: proxy_port
};
}
browser.proxy.settings.set({value: config, scope: "regular"}, function() {;});
function callbackFn(details) {
return {
authCredentials: {
username: "YOUR_USERNAME",
password: "YOUR_PASSWORD"
}
};
}
browser.webRequest.onAuthRequired.addListener(
callbackFn,
{urls: ["<all_urls>"]},
['blocking']
);
browser.proxy.onRequest.addListener(proxyRequest, {urls: ["<all_urls>"]});
manifest.json
{
"name": "My Firefox Proxy",
"version": "1.0.0b",
"manifest_version": 2,
"permissions": [
"browsingData",
"proxy",
"storage",
"tabs",
"webRequest",
"webRequestBlocking",
"downloads",
"notifications",
"<all_urls>"
],
"background": {
"scripts": ["background.js"]
},
"browser_specific_settings": {
"gecko": {
"id": "myproxy#example.org"
}
}
}
Next you need packed this files to zip archive in DEFLATED mode with .xpi at end like my_proxy_extension.xpi.
You have two choices:
Sign your extension Here you can read more about verify extension and extension's structure
OR
Run unsigned. For this step:
Open firefox flags at about:config and set options xpinstall.signatures.required to false
OR
Update firefox profile in:
Windows: C:\Program Files\Mozilla Firefox\defaults\pref\channel-prefs.js
Linux: /etc/firefox/syspref.js
Add next line to end of file:
pref("xpinstall.signatures.required",false);
After this steps run selenium and install this extension:
FirefoxProfile profile = new FirefoxProfile();
profile.addExtension(new File("path/to/my_proxy_extension.xpi"));
driver = new FirefoxDriver(profile);
What you can do is to create a profile and save the authentication data in it.
If your profile is called "webdriver" you can select it from your code in the initialization:
ProfilesIni allProfiles = new ProfilesIni();
FirefoxProfile profile = allProfiles.getProfile("WebDriver");
profile.setPreferences("foo.bar",23);
WebDriver driver = new FirefoxDriver(profile);
Did it with MS UI Automation without AutoIt:
public void AuthInProxyWindow (string login, string pass)
{
var proxyWindow = AutomationElement.RootElement
.FindFirst(TreeScope.Subtree,
new PropertyCondition(AutomationElement.ClassNameProperty, "MozillaDialogClass"));
var edits = proxyWindow.FindAll(TreeScope.Subtree,
new PropertyCondition(AutomationElement.ControlTypeProperty, ControlType.Edit));
var unamePoint = edits[1].GetClickablePoint();
Mouse.MoveTo(new Point((int) unamePoint.X, (int) unamePoint.Y));
Mouse.Click(MouseButton.Left);
SendKeys.SendWait(login);
var pwdPoint = edits[2].GetClickablePoint();
Mouse.MoveTo(new Point((int) pwdPoint.X, (int) pwdPoint.Y));
Mouse.Click(MouseButton.Left);
SendKeys.SendWait(pass);
Keyboard.Press(Key.Return);
Logger.Debug("Authefication in Firefox completed succesfully");
}
Mouse moves by Microsoft.TestApi
To stop firefox from giving you the auth pop up simple make sure you set your proxy URL to include the auth details in the setup stage as below:
var myProxy = user + ":" + pass + "#" + proxyIP + ":" + proxyPORT;
options.SetPreference("network.proxy.type", 1);
options.SetPreference("network.proxy.http", myProxy);
options.SetPreference("network.proxy.http_port", proxyPORT);
options.SetPreference("general.useragent.override", useragent);
driver = new FirefoxDriver(driverService, options);