Develop Reference

How to bypass the Walmart.com.mx block using Selenium Webdriver in C#

I'm trying to scrape at walmart.com.mx but after the first test it starts to block access and I get a kind of captcha.
I already tried adding proxy and it happens the same.
Could you help me?
string PROXY = "94.23.1.178:3128";
//string PROXY = "169.55.89.6:8123";
public void inicializarChrome()
{
var driverService = ChromeDriverService.CreateDefaultService();
options.AddExcludedArgument("--enable-automation");
options.AddArgument("--incognito");
options.AddArgument("--disable-blink-features=AutomationControlled");
//options.AddArguments($"--proxy-server={PROXY}");
options.AddArgument("--start-maximized");
driverService.HideCommandPromptWindow = true;
driver = new ChromeDriver(driverService, options);
try { driver.Manage().Window.Maximize(); } catch { }
//driver.Navigate().GoToUrl(dictionaryIni["Juguetes"]);
driver.Navigate().GoToUrl(dictionaryIni["Juguetes"]);
//driver.Quit();
}
I leave you my code

How to rename downloaded file using selenium c# webdriver

In my web application which has some automation process to download the files from the website. To achieve that I used selenium c# chrome driver.
Sample code snippets
public void Download(string name,string pass)
{
try
{
ChromeOptions options = new ChromeOptions();
options.AddArguments("--proxy-server=http://192.168.5.62:8095");
options.AddUserProfilePreference("safebrowsing.enabled", true);
options.AddUserProfilePreference("disable-popup-blocking", "true");
options.AddUserProfilePreference("download.default_directory",#"C:\Temp");
using (var driver = new ChromeDriver(HostingEnvironment.ApplicationPhysicalPath, options)){
//driver.findElement(By.xpath("//a/u[contains(text(),'Re-Submit')]")).click();
driver.FindElementById("save").Click();
}
}
catch (Exception ex)
{
Logger.LogWriter("LAS", ex, "CusDataLogic", "Download");
}
}
above code (not complete code) works fine and save file properly. But I need to rename that file downloading or after download. Have any possible way to rename that file?
Edited: Please don't mark this as a duplicate. I'm asking for C#, not python. I saw that question too. but it not helped to me

watching directory is not always good, because sometime saved filename is different than filename in URL.
go to chrome download page and loop until all download complete, you can see below how to select special element #shadow-root with CSS selector
using (var driver = new ChromeDriver(HostingEnvironment.ApplicationPhysicalPath, options)))
{
//driver.findElement(By.xpath("//a/u[contains(text(),'Re-Submit')]")).click();
driver.FindElementById("save").Click();
// wait 5 second until download started
Thread.Sleep(5000);
// Go to chrome download page
driver.Navigate().GoToUrl("chrome://downloads/");
string oldName = "";
bool downloadcomplete = false;
string cssNames = "downloads-manager /deep/ downloads-item /deep/ [id='name']";
string cssDlProgress = "downloads-manager /deep/ downloads-item /deep/ [class*='show-progress']";
while (!downloadcomplete)
{
var progressElements = driver.FindElements(By.CssSelector(cssDlProgress));
// check until no download progress bar
if (progressElements.Count() == 0)
{
oldName = driver.FindElement(By.CssSelector(cssNames)).Text;
downloadcomplete = true;
}
else
{
// download still in progress, wait.
Thread.Sleep(1000);
}
}
// download complete
// remove downloaded file
driver.FindElement(By.CssSelector("downloads-manager /deep/ downloads-item /deep/ [id='remove']")).Click();
// rename
File.Move(#"C:\Temp\" + oldName, #"C:\Temp\newname.ext");
}

The Snippet Below Will wait Until File downloaded Then return FilePath I Wrote this as an extension method :
public static string GetDonwloadedFileName(this IWebDriver driver)
{
IJavaScriptExecutor js = (IJavaScriptExecutor)driver;
js.ExecuteScript("window.open()");
var allWinowHandles = driver.WindowHandles;
foreach (var winHandle in allWinowHandles)
{
//Switch to second window
if (!winHandle.Equals(driver.CurrentWindowHandle))
{
driver.SwitchTo().Window(winHandle);
}
}
// navigate to chrome downloads
driver.Navigate().GoToUrl("chrome://downloads");
IJavaScriptExecutor downloadWindowExecutor = (IJavaScriptExecutor)driver;
// Wait for Download till 100% completion
double percentageProgress = (double)0;
while (percentageProgress != 100)
{
try
{
percentageProgress = (long)downloadWindowExecutor.ExecuteScript("return document.querySelector('downloads-manager').shadowRoot.querySelector('#downloadsList downloads-item').shadowRoot.querySelector('#progress').value");
Thread.Sleep(100);
}
catch (Exception)
{
break;
}
}
string fileTitle = (string)downloadWindowExecutor.ExecuteScript("return document.querySelector('downloads-manager').shadowRoot.querySelector('#downloadsList downloads-item').shadowRoot.querySelector('#show').getAttribute('title')");
downloadWindowExecutor.ExecuteScript("window.close()");
return fileTitle;
}
Then You can use file Path to rename it to whatever you need

Get WebDriver of already opened Chrome Browser

I would like to have a IWebDriver of an already opened browser like Chrome. Because then I need to automate a form authentication and/or a basic authentication.
I thought that this
IWebDriver driver = new RemoteWebDriver(new System.Uri("http://localhost:4445/wd/hub"), new ChromeOptions());
would do the trick but it only opens another chrome window. Instead I would like to "read" an already opened one.
Is it possible with selenium? O r should I use another library?

As per the Selenium Issues page:
https://github.com/seleniumhq/selenium-google-code-issue-archive/issues/18
The issue was closed and marked as not feasible
The process of connecting to an existing browser would be on a per-browser basis.
Doing it in IE might be easy, but doing it in Chrome or Firefox would be problematic.
Eg:
Chrome actually receives the commands from Selenium via network / tcp json requests to a specific port.
When Selenium driver stops running - it loses the port number for the Chrome debugging port.
The port may still be open, but it could be anything between 10000 and 30000 etc
Even if you solve it for Chrome, it would then require another bespoke solution for Firefox.
Unless your authentication has a 'Captcha' or bot check in place, I would suggest just automating the authentication stage.
Generally speaking - it is a good practice for Automated tests to be self-contained and not rely on outside interference or external tests.
A browser should start at the start of the test and be terminated at the end of the test.
Assuming you are using Selenium for testing and not for malicious purposes.
Selenium will not be helpful to you at this stage.
If however, you can live with your answer / solution being on Chrome but not the other browsers.
public static Chrome StartChromeDriver(int port)
{
try
{
string Path = Registry.Installation.GetChromeExecutable();
Process p = new Process();
ProcessStartInfo psi = new ProcessStartInfo(Path);
string args = "--remote-debugging-port="+ port.ToString()+" --user-data-dir=remote-profile";
psi.Arguments = args;
psi.Verb = "runas";
p.StartInfo = psi;
p.Start();
return new Chrome("http://localhost:" + port.ToString());
}
catch (Exception ee)
{
Console.WriteLine(ee.ToString());
return null;
}
}
This will start a chrome process with the debugging port opened to the number you provide.
(You can keep track of this, and reconnect and re-issue commands to the running chrome instance)
public dynamic EnablePage()
{
json = #"{""id"":12345,""method"":""Page.enable""}";
Thread.Sleep(1000);
return this.SendCommand(json);
}
public dynamic EnableRuntime()
{
json = #"{""id"":12345,""method"":""Runtime.enable""}";
Thread.Sleep(1000);
return this.SendCommand(json);
}
public dynamic EnableNetwork()
{
json = #"{""id"":12345,""method"":""Network.enable""}";
Thread.Sleep(1000);
return this.SendCommand(json);
}
This is some code I had lying around.
I was very bored one day and decided to reinvent the wheel with Chrome automation. Basically - this code is how you could automate Chrome without using Selenium at all.
It does have a dependency on WebSockets4Net
But that being said - it could probably be refactored to use TcpClient.
All the commands that are issued to Chrome, are done in the form of a json request.
Eg: the following json command would tell chrome to execute the following javascript - essentially navigating to the url provided.
{
"method": "Runtime.evaluate",
"params": {
"expression": "document.location='urlhere'",
"objectGroup": "console",
"includeCommandLineAPI": true,
"doNotPauseOnExceptions": false,
"returnByValue": false
},
"id": 1
}
public dynamic SendCommand(string cmd)
{
if (EventHandler == null)
{
EventHandler = new Events();
EventHandler.OnNavigateStart += new Events.OnPageNavigateStart(EventHandler_OnNavigateStart);
EventHandler.OnNavigateEnd += new Events.OnPageNavigateEnded(EventHandler_OnNavigateEnd);
}
WebSocket4Net.WebSocket j = new WebSocket4Net.WebSocket(this.sessionWSEndpoint);
ManualResetEvent waitEvent = new ManualResetEvent(false);
ManualResetEvent closedEvent = new ManualResetEvent(false);
dynamic message = null;
byte[] data;
Exception exc = null;
j.Opened += delegate(System.Object o, EventArgs e)
{
j.Send(cmd);
};
j.MessageReceived += delegate(System.Object o, WebSocket4Net.MessageReceivedEventArgs e)
{
message = e.Message;
EventHandler.ParseEvents(e);
waitEvent.Set();
};
j.Error += delegate(System.Object o, SuperSocket.ClientEngine.ErrorEventArgs e)
{
exc = e.Exception;
waitEvent.Set();
};
j.Closed += delegate(System.Object o, EventArgs e)
{
closedEvent.Set();
};
j.DataReceived += delegate(object sender, WebSocket4Net.DataReceivedEventArgs e)
{
data = e.Data;
waitEvent.Set();
};
j.Open();
waitEvent.WaitOne();
if (j.State == WebSocket4Net.WebSocketState.Open)
{
j.Close();
closedEvent.WaitOne();
j = null;
}
if (exc != null)
throw exc;
serializer = null;
serializer = new JavaScriptSerializer();
serializer.RegisterConverters(new[] { converter });
dynamic obj = serializer.Deserialize(message, typeof(object));
message = null;
data = null;
return obj;
}
To demonstrate how this could be used practically - you can implement page-object and create 'types' that encapsulate objects on screen.
For instance:
public class Link : Base.Element
{
public Link(string XPath)
{
this.XPath = String.Copy(XPath);
}
/// <summary>
/// Overriding it - just in case we need to handle clicks differently
/// </summary>
/// <returns></returns>
public virtual bool Click()
{
Sync();
Console.WriteLine(Chrome.Driver.Eval("document.evaluate('" + XPath.Replace("'", "\\\\'") + "', document.documentElement, null, XPathResult.ORDERED_NODE_SNAPSHOT_TYPE, null ).snapshotItem(0).click();"));
return true;
}
public virtual bool WaitForExistance(int iTimeout)
{
return base.WaitForExistance(iTimeout);
}
public virtual bool Exists()
{
return base.Exists();
}
public virtual string GetText()
{
Sync();
dynamic dval = Chrome.Driver.Eval("document.evaluate('" + XPath.Replace("'", "\\\\'") + "', document.documentElement, null, XPathResult.ORDERED_NODE_SNAPSHOT_TYPE, null ).snapshotItem(0).innerText");
return dval.result.result.value;
}
}
Be warned - there were memory leaks in WebSockets4Net when I was using this code - so the application eventually had to be restarted.
Perhaps if WebSockets4Net is removed and replaced - it will work better.

getting a list of installed browsers on the computer

I would like to know is there are any possibility to get a list of installed browsers on the computer using c#?
I'm using Selenium WebDriver in my task and I need to know which browsers are installed because in Selenium I can only run a specific browser, for example for Firefox it will be:
IWebDriver driver = new FirefoxDriver();
I will appreciate any help.

Look at localmachine registry...
Microsoft.Win32.RegistryKey key =
Registry.LocalMachine.OpenSubKey(#"SOFTWARE\Clients\StartMenuInternet");
var browsers = key.GetSubKeyNames();

You also need to take into account the machine architecture (x64 vs x86) and the fact that Microsoft Edge will not be under the specified key. Here is what I ended up using (based on multiple solutions found online):
private List<Browser> GetBrowsers()
{
RegistryKey browserKeys;
browserKeys = Registry.LocalMachine.OpenSubKey(#"SOFTWARE\WOW6432Node\Clients\StartMenuInternet");
if (browserKeys == null)
browserKeys = Registry.LocalMachine.OpenSubKey(#"SOFTWARE\Clients\StartMenuInternet");
string[] browserNames = browserKeys.GetSubKeyNames();
List<Browser> browsers = new List<Browser>();
for (int i = 0; i < browserNames.Length; i++)
{
Browser browser = new Browser();
RegistryKey browserKey = browserKeys.OpenSubKey(browserNames[i]);
browser.Name = (string)browserKey.GetValue(null);
RegistryKey browserKeyPath = browserKey.OpenSubKey(#"shell\open\command");
browser.Path = browserKeyPath.GetValue(null).ToString().StripQuotes();
browsers.Add(browser);
if (browser.Path != null)
browser.Version = FileVersionInfo.GetVersionInfo(browser.Path).FileVersion;
else
browser.Version = "unknown";
}
Browser edgeBrowser = GetEdgeVersion();
if (edgeBrowser != null)
{
browsers.Add(edgeBrowser);
}
return browsers;
}
private Browser GetEdgeVersion()
{
RegistryKey edgeKey =
Registry.CurrentUser.OpenSubKey(
#"SOFTWARE\Classes\Local Settings\Software\Microsoft\Windows\CurrentVersion\AppModel\SystemAppData\Microsoft.MicrosoftEdge_8wekyb3d8bbwe\Schemas");
if (edgeKey != null)
{
string version = edgeKey.GetValue("PackageFullName").ToString().StripQuotes();
Match result = Regex.Match(version, "(((([0-9.])\\d)+){1})");
if (result.Success)
{
return new Browser
{
Name = "MicrosoftEdge",
Version = result.Value
};
}
}
return null;
}
And the object returned is a simple DTO:
public class Browser{
public string Name { get; set; }
public string Path { get; set; }
public string Version { get; set; }
}

I've written a NuGet package for this:
https://www.nuget.org/packages/MintPlayer.PlatformBrowser/ targetting .net core.
You can get a list of all installed webbrowsers (including Edge) and the default webbrowser. I've also written a package with a dialog to let you pick a browser: https://www.nuget.org/packages/MintPlayer.BrowserDialog/

As far as I know there is no list of browsers in Windows.
However you could check for browser's existence by simply testing the *.exe file's existence:
if (File.Exists(#"C:\Program Files (x86)\Google\Chrome\Application\chrome.exe") ||
File.Exists(#"C:\Program Files\Google\Chrome\Application\chrome.exe")) {
// chrome is installed
}
if (File.Exists(#"C:\Program Files (x86)\Mozilla Firefox\firefox.exe") ||
File.Exists(#"C:\Program Files\Mozilla Firefox\firefox.exe") {
// firefox is installed
}

Selenium opens firefox and stops all processes

i am automating a process on firefox using selenium with c# .NET4.0:
static void Main(string[] args)
{
selenium = new DefaultSelenium("localhost", 4444, "*firefox", "https://www.microsoft.com/");
selenium.Start();
selenium.SetSpeed("900");
selenium.WindowMaximize();
selenium.WindowFocus();
Login();
}
public static void LogIn()
{
verificationErrors = new StringBuilder();
selenium.SetSpeed("900");
selenium.Open("/Login.aspx");
selenium.Type("id=ctl00_cphBody_objLogin_UserName", "user");
selenium.Type("id=ctl00_cphBody_objLogin_Password", "P#assword");
selenium.Click("id=ctl00_cphBody_objLogin_LoginImageButton");
selenium.WaitForPageToLoad("30000");
try
{
selenium.IsTextPresent("Orders - Drug Testing");
}
catch (Exception)
{
Console.WriteLine("Could't find text: Orders - Drug Testing.");
}
}
firefox opens, and then NOTHING happens at all. it does not navigate to the requested page.
here's what it looks like. what am i doing wrong? everything works fine with IE

To navigate to the requested page, need to use the following piece of code.
Selenium doesn't navigate to the page or base url automatically until & unless you call
selenium.open(" < your url goes here > ");
Ex : selenium.open("https://mail.google.com");