I'm trying to automate a very basic task in a website using selenium and chrome but somehow the website detects when chrome is driven by selenium and blocks every request. I suspect that the website is relying on an exposed DOM variable like this one https://stackoverflow.com/a/41904453/648236 to detect selenium driven browser.
My question is, is there a way I can make the navigator.webdriver flag false? I am willing to go so far as to try and recompile the selenium source after making modifications, but I cannot seem to find the NavigatorAutomationInformation source anywhere in the repository https://github.com/SeleniumHQ/selenium
Any help is much appreciated
P.S: I also tried the following from https://w3c.github.io/webdriver/#interface
Object.defineProperty(navigator, 'webdriver', {
get: () => false,
});
But it only updates the property after the initial page load. I think the site detects the variable before my script is executed.
First the update 1
execute_cdp_cmd(): With the availability of execute_cdp_cmd(cmd, cmd_args) command now you can easily execute google-chrome-devtools commands using Selenium. Using this feature you can modify the navigator.webdriver easily to prevent Selenium from getting detected.
Preventing Detection 2
To prevent Selenium driven WebDriver getting detected a niche approach would include either / all of the below mentioned steps:
Adding the argument --disable-blink-features=AutomationControlled
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument('--disable-blink-features=AutomationControlled')
driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get("https://www.website.com")
You can find a relevant detailed discussion in Selenium can't open a second page
Rotating the user-agent through execute_cdp_cmd() command as follows:
#Setting up Chrome/83.0.4103.53 as useragent
driver.execute_cdp_cmd('Network.setUserAgentOverride', {"userAgent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.53 Safari/537.36'})
Change the property value of the navigator for webdriver to undefined
driver.execute_script("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})")
Exclude the collection of enable-automation switches
options.add_experimental_option("excludeSwitches", ["enable-automation"])
Turn-off useAutomationExtension
options.add_experimental_option('useAutomationExtension', False)
Sample Code 3
Clubbing up all the steps mentioned above and effective code block will be:
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.execute_script("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})")
driver.execute_cdp_cmd('Network.setUserAgentOverride', {"userAgent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.53 Safari/537.36'})
print(driver.execute_script("return navigator.userAgent;"))
driver.get('https://www.httpbin.org/headers')
History
As per the W3C Editor's Draft the current implementation strictly mentions:
The webdriver-active flag is set to true when the user agent is under remote control which is initially set to false.
Further,
Navigator includes NavigatorAutomationInformation;
It is to be noted that:
The NavigatorAutomationInformation interface should not be exposed on WorkerNavigator.
The NavigatorAutomationInformation interface is defined as:
interface mixin NavigatorAutomationInformation {
readonly attribute boolean webdriver;
};
which returns true if webdriver-active flag is set, false otherwise.
Finally, the navigator.webdriver defines a standard way for co-operating user agents to inform the document that it is controlled by WebDriver, so that alternate code paths can be triggered during automation.
Caution: Altering/tweaking the above mentioned parameters may block the navigation and get the WebDriver instance detected.
Update (6-Nov-2019)
As of the current implementation an ideal way to access a web page without getting detected would be to use the ChromeOptions() class to add a couple of arguments to:
Exclude the collection of enable-automation switches
Turn-off useAutomationExtension
through an instance of ChromeOptions as follows:
Java Example:
System.setProperty("webdriver.chrome.driver", "C:\\Utility\\BrowserDrivers\\chromedriver.exe");
ChromeOptions options = new ChromeOptions();
options.setExperimentalOption("excludeSwitches", Collections.singletonList("enable-automation"));
options.setExperimentalOption("useAutomationExtension", false);
WebDriver driver = new ChromeDriver(options);
driver.get("https://www.google.com/");
Python Example
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(options=options, executable_path=r'C:\path\to\chromedriver.exe')
driver.get("https://www.google.com/")
Ruby Example
options = Selenium::WebDriver::Chrome::Options.new
options.add_argument("--disable-blink-features=AutomationControlled")
driver = Selenium::WebDriver.for :chrome, options: options
Legends
1: Applies to Selenium's Python clients only.
2: Applies to Selenium's Python clients only.
3: Applies to Selenium's Python clients only.
ChromeDriver:
Finally discovered the simple solution for this with a simple flag! :)
--disable-blink-features=AutomationControlled
navigator.webdriver=true will no longer show up with that flag set.
For a list of things you can disable, check them out here
Do not use cdp command to change webdriver value as it will lead to inconsistency which later can be used to detect webdriver. Use the below code, this will remove any traces of webdriver.
options.add_argument("--disable-blink-features")
options.add_argument("--disable-blink-features=AutomationControlled")
Before (in browser console window):
> navigator.webdriver
true
Change (in selenium):
// C#
var options = new ChromeOptions();
options.AddExcludedArguments(new List<string>() { "enable-automation" });
// Python
options.add_experimental_option("excludeSwitches", ['enable-automation'])
After (in browser console window):
> navigator.webdriver
undefined
This will not work for version ChromeDriver 79.0.3945.16 and above. See the release notes here
To exclude the collection of enable-automation switches as mentioned in the 6-Nov-2019 update of the top voted answer doesn't work anymore as of April 2020. Instead I was getting the following error:
ERROR:broker_win.cc(55)] Error reading broker pipe: The pipe has been ended. (0x6D)
Here's what's working as of 6th April 2020 with Chrome 80.
Before (in the Chrome console window):
> navigator.webdriver
true
Python example:
options = webdriver.ChromeOptions()
options.add_argument("--disable-blink-features")
options.add_argument("--disable-blink-features=AutomationControlled")
After (in the Chrome console window):
> navigator.webdriver
undefined
Nowadays you can accomplish this with cdp command:
driver.execute_cdp_cmd("Page.addScriptToEvaluateOnNewDocument", {
"source": """
Object.defineProperty(navigator, 'webdriver', {
get: () => undefined
})
"""
})
driver.get(some_url)
by the way, you want to return undefined, false is a dead giveaway.
Finally this solved the problem for ChromeDriver, Chrome greater than v79.
ChromeOptions options = new ChromeOptions();
options.addArguments("--disable-blink-features");
options.addArguments("--disable-blink-features=AutomationControlled");
ChromeDriver driver = new ChromeDriver(options);
Map<String, Object> params = new HashMap<String, Object>();
params.put("source", "Object.defineProperty(navigator, 'webdriver', { get: () => undefined })");
driver.executeCdpCommand("Page.addScriptToEvaluateOnNewDocument", params);
Since this question is related to selenium a cross-browser solution to overriding navigator.webdriver is useful. This could be done by patching browser environment before any JS of target page runs, but unfortunately no other browsers except chromium allows one to evaluate arbitrary JavaScript code after document load and before any other JS runs (firefox is close with Remote Protocol).
Before patching we needed to check how the default browser environment looks like. Before changing a property we can see it's default definition with Object.getOwnPropertyDescriptor()
Object.getOwnPropertyDescriptor(navigator, 'webdriver');
// undefined
So with this quick test we can see webdriver property is not defined in navigator. It's actually defined in Navigator.prototype:
Object.getOwnPropertyDescriptor(Navigator.prototype, 'webdriver');
// {set: undefined, enumerable: true, configurable: true, get: ƒ}
It's highly important to change the property on the object that owns it, otherwise the following can happen:
navigator.webdriver; // true if webdriver controlled, false otherwise
// this lazy patch is commonly found on the internet, it does not even set the right value
Object.defineProperty(navigator, 'webdriver', {
get: () => undefined
});
navigator.webdriver; // undefined
Object.getOwnPropertyDescriptor(Navigator.prototype, 'webdriver').get.apply(navigator);
// true
A less naive patch would first target the right object and use right property definition, but digging deeper we can find more inconsistences:
const defaultGetter = Object.getOwnPropertyDescriptor(Navigator.prototype, 'webdriver').get;
defaultGetter.toString();
// "function get webdriver() { [native code] }"
Object.defineProperty(Navigator.prototype, 'webdriver', {
set: undefined,
enumerable: true,
configurable: true,
get: () => false
});
const patchedGetter = Object.getOwnPropertyDescriptor(Navigator.prototype, 'webdriver').get;
patchedGetter.toString();
// "() => false"
A perfect patch leaves no traces, instead of replacing getter function it would be good if we could just intercept the call to it and change the returned value. JavaScript has native support for that throught Proxy apply handler:
const defaultGetter = Object.getOwnPropertyDescriptor(Navigator.prototype, 'webdriver').get;
defaultGetter.apply(navigator); // true
defaultGetter.toString();
// "function get webdriver() { [native code] }"
Object.defineProperty(Navigator.prototype, 'webdriver', {
set: undefined,
enumerable: true,
configurable: true,
get: new Proxy(defaultGetter, { apply: (target, thisArg, args) => {
// emulate getter call validation
Reflect.apply(target, thisArg, args);
return false;
}})
});
const patchedGetter = Object.getOwnPropertyDescriptor(Navigator.prototype, 'webdriver').get;
patchedGetter.apply(navigator); // false
patchedGetter.toString();
// "function () { [native code] }"
The only inconsistence now is in the function name, unfortunately there is no way to override the function name shown in native toString() representation. But even so it can pass generic regular expressions that searches for spoofed browser native functions by looking for { [native code] } at the end of its string representation. To remove this inconsistence you can patch Function.prototype.toString and make it return valid native string representations for all native functions you patched.
To sum up, in selenium it could be applied with:
chrome.execute_cdp_cmd('Page.addScriptToEvaluateOnNewDocument', {'source': """
Object.defineProperty(Navigator.prototype, 'webdriver', {
set: undefined,
enumerable: true,
configurable: true,
get: new Proxy(
Object.getOwnPropertyDescriptor(Navigator.prototype, 'webdriver').get,
{ apply: (target, thisArg, args) => {
// emulate getter call validation
Reflect.apply(target, thisArg, args);
return false;
}}
)
});
"""})
The playwright project maintains a fork of Firefox and WebKit to add features for browser automation, one of them is equivalent to Page.addScriptToEvaluateOnNewDocument, but there is no implementation for Python of the communication protocol but it could be implemented from scratch.
Simple hack for python:
options = webdriver.ChromeOptions()
options.add_argument("--disable-blink-features=AutomationControlled")
As mentioned in the above comment - https://stackoverflow.com/a/60403652/2923098 the following option totally worked for me (in Java)-
ChromeOptions options = new ChromeOptions();
options.addArguments("--incognito", "--disable-blink-features=AutomationControlled");
I would like to add a Java alternative to the cdp command method mentioned by pguardiario
Map<String, Object> params = new HashMap<String, Object>();
params.put("source", "Object.defineProperty(navigator, 'webdriver', { get: () => undefined })");
driver.executeCdpCommand("Page.addScriptToEvaluateOnNewDocument", params);
In order for this to work you need to use the ChromiumDriver from the org.openqa.selenium.chromium.ChromiumDriver package. From what I can tell that package is not included in Selenium 3.141.59 so I used the Selenium 4 alpha.
Also, the excludeSwitches/useAutomationExtension experimental options do not seem to work for me anymore with ChromeDriver 79 and Chrome 79.
For those of you who've tried these tricks, please make sure to also check that the user-agent that you are using is the user agent that corresponds to the platform (mobile / desktop / tablet) your crawler is meant to emulate. It took me a while to realize that was my Achilles heel ;)
Python
I tried most of the stuff mentioned in this post and i was still facing issues.
What saved me for now is https://pypi.org/project/undetected-chromedriver
pip install undetected-chromedriver
import undetected_chromedriver.v2 as uc
from time import sleep
from random import randint
driver = uc.Chrome()
driver.get('www.your_url.here')
driver.maximize_window()
sleep(randint(3,9))
A bit slow but i will take slow over non working.
I guess if every interested could go over the source code and see what provides the win there.
If you use a Remote Webdriver , the code below will set navigator.webdriver to undefined.
work for ChromeDriver 81.0.4044.122
Python example:
options = webdriver.ChromeOptions()
# options.add_argument("--headless")
options.add_argument('--disable-gpu')
options.add_argument('--no-sandbox')
driver = webdriver.Remote(
'localhost:9515', desired_capabilities=options.to_capabilities())
script = '''
Object.defineProperty(navigator, 'webdriver', {
get: () => undefined
})
'''
driver.execute_script(script)
Use --disable-blink-features=AutomationControlled to disable navigator.webdriver
From what I understand, the default action of when the Webdriver finds an element is to scroll such that the element is as far up the top of the page as possible. This is an issue because the website I'm working on has a header so every time I try to click on a button, it will instead click on the header. Thus, I want to change the scroll setting so that the element will be at the bottom of the page.
From reading this I was able to find what I wanted to set, however, I'm unable to set the DesiredCapabilites or ChromeOptions when I initialise the ChromeDriver. Could some provide code/steps to do this please?
You can use something like this
var chromeOptions = new ChromeOptions();
chromeOptions.AddUserProfilePreference("intl.accept_languages", "en");
chromeOptions.AddUserProfilePreference("disable-popup-blocking", "true");
var driver = new ChromeDriver(chromeOptions);
Edit-2
If the option you want to set doesn't work for you then try using actions
var elem = driver.FindElements(By.Id("your element"));
Actions action = new Actions(driver);
action.MoveToElement(elem).Click(elem).Perform();//move to list element that needs to be hovered
Edit-3
If the above also doesn't work then your next option is to use Javascript
var elem = driver.FindElements(By.Id("your element"));
IJavaScriptExecutor js = (IJavaScriptExecutor)driver;
var success = js.ExecuteScript("arguments[0].click(); return true", elem);
As far as I know you can't change everything through addarguments.
There is a list of what you can do in the Github page. but I have a better solution. you can make your own default settings and save it as a chrome profile. for example I didn't find anything to change homepage with code but this works fine for almost evwerything.
you can use this code :
options.AddArguments( #"user-data -dir=C:\Users\kian\AppData\Local\Google\Chrome\User Data");
options.AddArgument("--profile-directory=Default");
make sure you write the right path and right profile name.
to check the profile name you can go to properties.
properties
you will see the profile name.
there is a good guide for what else u can do in link.
I am writing a script in Selenium WebDriver using C#. In the script, I am downloading some documents from the webpage and I want to download it in a dynamic path. I am using ChromeOptions class and its method to accomplish the task. Here is my sample code:
ChromeOptions options = new ChromeOptions();
options.AddUserProfilePreference("download.default_directory", "C:\Users\Desktop\MyDownloads");
IWebDriver driver = new ChromeDriver(#"C:\Users\chromedriver_win32\" , options);
If I am using the above code in the starting of the function then it works fine.
However, I want to set the properties of ChromeOptions class in the middle of the function because my path is dynamic. Hence I just change the hard coded path with the string variable and put the following code in the middle of the function
ChromeOptions options = new ChromeOptions();
options.AddUserProfilePreference("download.default_directory", strDownloadFinalPath);
IWebDriver driver = new ChromeDriver(#"C:\Users\chromedriver_win32\" , options);
Now, when I am updating the ChromeOptions in the middle of the function or at run time, then Its creating another instance of a ChromeDriver and its opening one more chrome window. It does not update the properties of ChromeOptions class. I did some experiments like removing the path of chromedriver.exe from IChromeDriver class but it started giving the following error:
The chromedriver.exe file does not exist in the current directory or
in a directory on the PATH environment variable.
What could be the way to set the ChromeOptions in the middle of the code without creating an another instance of a IWebDriver Class?
You can only set ChromeOptions, and thus the download path, via the class constructor(s). There is no property you can update once you've instantiated ChromeDriver. So the answer to your final question ("without creating another instance") is, you cannot.
What I have done to deal with this is to check the "Ask where to save each file before downloading" setting in Chrome and then I interact with the Save As dialog prompt in my test inputing the full dynamic save file path and clicking save. The problem is that this is a Windows dialog and Selenium cannot interact with it. I am using MS CodedUI to work with it. My dialog class for the Save As prompt:
using Microsoft.VisualStudio.TestTools.UITesting.WinControls;
public class WindowsDialogBoxView : WinWindow
{
public WindowsDialogBoxView()
{
this.SearchProperties[WinWindow.PropertyNames.ClassName] = "#32770";
}
public WinEdit FilenameEdit
{
get
{
this.filenameEdit = new WinEdit(this);
this.filenameEdit.SearchProperties[WinEdit.PropertyNames.Name] = "File name:";
return this.filenameEdit;
}
}
private WinEdit filenameEdit;
Usage:
WindowsDialogBoxView WindowsDialogBox = new WindowsDialogBoxView();
Keyboard.SendKeys(WindowsDialogBox.FilenameEdit, "C:\\myFileSavePath\\Blah\\FileToSave.abc");
I had difficulty interacting with the Save button of the dialog so I use Keyboard.SendKeys("{ENTER}"); You may have to add some {TAB}s in there.
I am using selenium to automate website filling in chrome. When I download exe or XML files I am getting a popup 'This type of file can harm your computer' with keep and discard options. How to disable this programmatically ?
I am implementing this in c#,
I have tried,
chromeOptions.AddUserProfilePreference("disable-popup-blocking", "true")
But this is not working for me.
How to disable this popup ?
If not possible how to accept the warning ?
Please help me.
To disable the message, you need to set the preference safebrowsing.enabled to true. Here is a working example with CSharp:
var options = new ChromeOptions();
options.AddUserProfilePreference("download.default_directory", "C:\\Downloads");
options.AddUserProfilePreference("download.prompt_for_download", false);
options.AddUserProfilePreference("download.directory_upgrade", true);
options.AddUserProfilePreference("safebrowsing.enabled", true);
var driver = new ChromeDriver(options);
driver.Navigate().GoToUrl("http://www.7-zip.org/a/7z1602.exe");
And for a description of the preferences:
https://chromium.googlesource.com/chromium/src/+/master/chrome/common/pref_names.cc
Probably what you need is change
chromeOptions.AddUserProfilePreference("disable-popup-blocking", "true") // disables blocking the popup
to
chromeOptions.AddUserProfilePreference("disable-popup-blocking", "false") // enables blocking the popup
instead to avoid the popup window.
Edit : In case you are getting stuck with the harmful file contents, you can try and set an experimental chrome options using :
Map<String, Object> prefs = new HashMap<String, Object>();
prefs.put("safebrowsing.enabled", "true");
chromeOptions.setExperimentalOption("prefs", prefs);
Quoting a sample from the doc here :
Set a Chrome preference
ChromeOptions options = new ChromeOptions();
Map<String, Object> prefs = new HashMap<String, Object>();
prefs.put("profile.default_content_settings.popups", 0);
options.setExperimentalOption("prefs", prefs);
Date - 4 June,2016 [C#]
Went through few links and could figure this out that .Net still doesn't have an setExperimentalOption preferences along with the ChromeOptions. So, one of the way to Add Argument to the ChromeOptions using C# with the flags listed here would be :
chromeOptions.AddArgument("--safebrowsing-disable-download-protection");
Disables safebrowsing feature that checks download url and downloads
content's hash to make sure the content are not malicious.
Date - 4 June,2016 [JAVA]
The document here quotes it as :
public void setExperimentalOption(java.lang.String name,
java.lang.Object value)
Sets an experimental option. Useful for new ChromeDriver options not yet
exposed through the ChromeOptions API.