scraping website that is generated by javascript in C# - c#

I am new to coding. I am trying to make a simple console app that will run internet speed test. I've searched all over and I couldn't find the answer. I tried all the sample answers but I couldn't get the program to run. For now, my program returns 0 which is a value from HTML source document. I need the value from javascript. the website is https://fast.com/en/ I only need the speed test results. I need help. here is my code:
enter code here
class Program
{
[STAThread]
static void Main(string[] args)
{
HtmlWeb web = new HtmlWeb();
string url = "https://fast.com/en/";
HtmlDocument doc = web.LoadFromBrowser(url, html =>
{
return !html.Contains
("<div class=\"speed-results-container succeeded\"
id=\"speed_value\" ></div>");
});
var t1 = doc.DocumentNode.SelectSingleNode
("//div[#id='speed-value']").InnerText;
Console.WriteLine($"{t1}");
}
}

So the whole "magic" of the test is made in app-ea56f7.js file.
This file is sending request and receiving chunks of data from netflix. Unfortunately, as referenced in Running Scripts in HtmlAgilityPack there is no direct way of having this without using a headless browser.
Either use https://www.npmjs.com/package/speedtest-net

Related

Headless Browser C# and Alternatives

currently I have the following code using selenium and phantomjs in c#:
public class Driver
{
static void Main()
{
using (var driver = new PhantomJSDriver())
{
driver.Navigate().GoToUrl("https://www.website.com/");
driver.Navigate().GoToUrl("https://www.website.com/productpage/");
driver.ExecuteScript("document.getElementById('pdp_selectedSize').value = '10.0'"); //FindElementById("pdp_selectedSize").SendKeys("10.0");
driver.ExecuteScript("document.getElementById('product_form').submit()");
driver.Navigate().GoToUrl("http://www.website/cart/");
Screenshot sh = driver.GetScreenshot();
sh.SaveAsFile(#"C:\temp\test.jpg", ImageFormat.Png);
}
}
}
My objective is to be able to add a product to my cart and then checkout automatically. The screenshot is just included to test whether the code was successfully working. My first issue is that I often get an error that it cannot find the element with product id "pdp_selectedSize". I'm assuming this is because the the webdriver hasn't loaded the page yet, so I'm looking for a way to keep checking until it finds it without having to set a specific timeout.
I'm also looking for faster alternatives to use instead of a headless browser. I used a headless browser instead of http requests because I need certain cookies to be able to checkout on the page, and these cookies are set through javascript within the page. If anyone has a reccommendation for a faster method, it would be greatly appreciated, thanks!
For your first question, it would behoove you to look into using ExpectedConditions' which is part of theWebDriverWaitclass inSelenium`. The following code sample was taken from here and only serves as a reference point.
using (IWebDriver driver = new FirefoxDriver())
{
driver.Url = "http://somedomain/url_that_delays_loading";
WebDriverWait wait = new WebDriverWait(driver, TimeSpan.FromSeconds(10));
IWebElement myDynamicElement = wait.Until<IWebElement>(d =>
d.FindElement(By.Id("someDynamicElement")));
}
More on WebDriverWaits here.
As to your second question, that is a really subjective thing, in my opinion. Headless Browsers aren't necessarily any faster or slower than a real browser. See this article.

C# and selenium: Get current URL and store in a string

I Need to get the current URL. Split that URL and store it in a string.
What I have tried is -
String urlSecondpart = this.getDriver().getCurrentUrl().Split("/a/")[1];
This opens a new url not the current URL
You are doing something wrong in your code but you haven't posted enough for us to see the problem. Do some reading and create a 2 line script and make sure the basics are working.
IWebDriver driver = new FirefoxDriver();
String currentURL = driver.Url;
If they aren't, you have something installed incorrectly or version mismatching. Fix that and try again. If it is working, you will have to debug your script by placing a breakpoint when the script starts and walking through your code until you find the problem.
Read How to create a Minimal, Complete, and Verifiable example, especially the last link on how to debug small programs.
Here I'm using Firefox profile as an example
FirefoxProfile firefoxProfile = new FirefoxProfile();
// todo: initialize your profile using firefoxProfile.SetPreference
IWebDriver driver = new FirefoxDriver(firefoxProfile);
string url = driver.Url; // get the current full URL
Then make use of standard string operation to extract what you want.
For the details in SetPreference and setting up a profile, please refer to http://www.seleniumhq.org/docs/
Create a static object of the driver once outside of the main method and within a class.
public class Test
{
public static IWebDriver driver = new FirefoxDriver();
static void Main(string[] args) {
// Here your code
string url = driver.Url;
}
}
It will give you an URL of the current page.

Unable to find an element in Browser of the Android emulator using Appium and C#

I want to automate mobile web site testing on Android emulator using c# and Appium. There is a simple test scenario I want to automate for the start:
1. Start Browser
2. Find an element
3. Clear it
4. Send keys
I've got a problem with the second step. Every time MSTest tries to execute FindElementById line in the code below, I get the error:
"An element could not be located on the page using the given search parameters."
[TestClass]
public class UnitTest1
{
private DesiredCapabilities _capabilities;
private AndroidDriver _driver;
public void InitializeDriver()
{
Console.WriteLine("Connecting to Appium server");
_capabilities = new DesiredCapabilities();
_capabilities.SetCapability("deviceName", "test_02");
_capabilities.SetCapability(CapabilityType.BrowserName, "Chrome");
_capabilities.SetCapability(CapabilityType.Version, "5.0.1");
_capabilities.SetCapability(CapabilityType.Platform, "Android");
//Application path and configurations
_driver = new AndroidDriver(new Uri("http://127.0.0.1:4723/wd/hub"), _capabilities);
}
[TestMethod]
public void TestMethod1()
{
InitializeDriver();
var element = _driver.FindElementById("com.android.browser:id/url");
element.Clear();
element.SendKeys(#"http://stackoverflow.com/");
}
}
Input string for the method I've got from UIAutomator that is shown below.
I tried several combinations for the FindElementById input method:
"com.android.browser:id/url"
"id/url"
"url"
but no luck.
My environment:
Windows 8.1
Appium 1.3.4.1
ChromeDriver 2.14.313457
Android Device Monitor 24.0.2
Sorry for misleading !!!
In case of testing web apps in browser the elements should be located as usual elements on the web page ( not as some classes like android.widget.EditText and android.widget.Button). So try for example the following and you will see some result:
var element = _driver
.findElementByXPath("//input[#id='lst-ib']");
To get locators you should run the browser on your desktop, open the page and use some tools/extensions like Firebug in Firefox or Firebug Lite in Chrome browser.
Try these 2 statements:
var element = _driver.FindElement(By.Id("com.android.browser:id/url");
driver.findElementsByXPath("//*[#class='com.android.browser' and #index='1']");
Update ! The following approach is not for web testing:
Could you try to find the element using xpath?
#FindBy(xpath="//android.widget.EditText[contains(#resource-id, 'url')]")
So in your case you can try the following:
var element = _driver.findElementByXPath("//android.widget.EditText[contains(#resource-id, 'url')]");
Update: in case of testing web apps (not native) you should use web page locators instead of Android classes.

Using class WebBrowser to open IE and fetch HtmlElement

I'm using Microsoft Visual Studio 2013 Ultimate and am currently using a Coded UI Test Project. When using it for a Console application, I get the error "An unhandled exception of type 'System.Threading.ThreadStateException' occurred in System.Windows.Forms.dll" when executing the first line of code below, but I don't get it when using my coded UI test project.
The end intention of my code is to launch a web browser. Click some links, and read information from certain ID fields from within the browser. If someone knows of a better or simply working implementation of doing that, I am all for it.
In my current implementation, internet explorer is never launched.
WebBrowser web = new WebBrowser(); //create object web browser
web.CreateControl();
web.Visible = true;
web.ScriptErrorsSuppressed = true;
web.Navigate("www.gogle.com"); //Shouldn't this launch IE & go to google.com?
HtmlDocument doc = web.Document;
HtmlElement el = doc.GetElementById("hplogo");//get htmlelement on the google logo
//do something with info from el.
web.Dispose(); //de-allocate
From MS KB841295 you can add the STAThread() attribute to your main entry point of your program and this should resolve that specific exception.
[STAThread()]
static void Main(string[] args) {
WebBrowser web = new WebBrowser(); //create object web browser
//...
}

emulating a browser programmatically in C# / .Net

We would like to automate certain tasks in a website, like having a user 'login', perform some functionality, read their account history etc.
We have tried emulating this with normal POST/GETs, however the problem is that for example for 'login', the website uses javascript code to execute an AJAX call, and also generate some random tokens.
Is it possible to literally emulate a web-browser? For example:
Visit 'www.[test-website].com'
Fill in these DOM items
DOM item 'username' fill in with 'testuser'
DOM item 'password' fill in with 'testpass'
Click' button DOM item 'btnSubmit'
Visit account history
Read HTML (So we can parse information about each distinct history item)
...
The above could be translated into say the below sample code:
var browser = new Browser();
var pageHomepage = browser.Load("www.test-domain.com");
pageHomepage.DOM.GetField("username").SetValue("testUser");
pageHomepage.DOM.GetField("password").SetValue("testPass");
pageHomepage.DOM.GetField("btnSubmit").Click();
var pageAccountHistory = browser.Load("www.test-domain.com/account-history/");
var html = pageAccountHistory.GetHtml();
var historyItems = parseHistoryItems(html);
You could use for example Selenium in C#. There is a good tutorial: Data Driven Testing Using Selenium (webdriver) in C#.
I would suggest to instantiate a WebBrowser control in code and do all your work with this instance but never show it on any form. I've done this several times and it works pretty good. The only flaw is that it makes use of the Internet Explorer ;-)
Try JMeter, it is a nice too for automating web requests, also quite popularly used for performance testing of web sites
Or just try System.Windows.Forms.WebBrowser, for example:
this.webBrowser1.Navigate("http://games.powernet.com.ru/login");
while (webBrowser1.ReadyState != WebBrowserReadyState.Complete)
System.Windows.Forms.Application.DoEvents();
HtmlDocument doc = webBrowser1.Document;
HtmlElement elem1 = doc.GetElementById("login");
elem1.Focus();
elem1.InnerText = "login";
HtmlElement elem2 = doc.GetElementById("pass");
elem2.Focus();
elem2.InnerText = "pass";

Categories

Resources