Develop Reference

C# Wait for Web Page to Load Before Scraping

I am trying to make a Windows Forms app that logs in another web application, navigates for a few steps (clicks) until it reaches a specific page and then scrape some info (names and addresses).
The problem is that I am using the DocumentCompletedEventHandler in order to have a page loaded before I execute the code for navigating to the next page (in order to reach the final web page).
When it fires, DocumentCompletedEventHandler fires multiple times.
When I reach the loggin page, it enters the credentials and then the message "Page loaded!" appears multiple times.
I press enter, it appears again.
Then it navigates to the next page and with that new page I have the same problem.
how can I make DocumentCompletedEventHandler to fire only once and not multiple times?
private void loadEvent(object sender, WebBrowserDocumentCompletedEventArgs e)
{
MessageBox.Show("Page loaded!");
}
private void loadLogin(object sender, WebBrowserDocumentCompletedEventArgs e)
{
var inputElements = webBrowser1.Document.GetElementsByTagName("input");
foreach (HtmlElement i in inputElements)
{
if (i.GetAttribute("name").Equals("utilizator"))
{
i.InnerText = textBox1.Text;
}
if (i.GetAttribute("name").Equals("parola"))
{
i.Focus();
i.InnerText = textBox2.Text;
}
}
var buttonElements = webBrowser1.Document.GetElementsByTagName("input");
foreach (HtmlElement b in buttonElements)
{
if (b.GetAttribute("name").Equals("Intra"))
{
b.InvokeMember("Click");
}
}
webBrowser1.DocumentCompleted += new WebBrowserDocumentCompletedEventHandler(loadEvent);
var inputElements1 = webBrowser1.Document.GetElementsByTagName("input");
foreach (HtmlElement i1 in inputElements1)
{
if (i1.GetAttribute("id").Equals("headerqstext"))
{
i1.Focus();
i1.InnerText = textBox3.Text;
}
}
var buttonElements1 = webBrowser1.Document.GetElementsByTagName("button");
foreach (HtmlElement b1 in buttonElements1)
{
if (b1.GetAttribute("title").Equals("Caută"))
{
b1.InvokeMember("Click");
}
}
webBrowser1.DocumentCompleted += new WebBrowserDocumentCompletedEventHandler(loadEvent);
}
private void Button1_Click(object sender, EventArgs e)
{
webBrowser1.Navigate("http://10.1.104.23/ecris_cdms/");
webBrowser1.DocumentCompleted += new WebBrowserDocumentCompletedEventHandler(loadLogin);
}
}
}

try this :)
Uri last = null;
private void CompleteResponse(object sender, WebBrowserDocumentCompletedEventArgs e)
{
if (!(last != null && last != e.Url))
return;
//your code here
}

Converting a HtmlAgilityPack.HtmlNode to a string

private void button2_Click(object sender, EventArgs e)
{
MessageBox.Show("In devolopment","Error", MessageBoxButtons.OK);
HtmlAgilityPack.HtmlWeb hw = new HtmlAgilityPack.HtmlWeb();
HtmlAgilityPack.HtmlDocument doc = hw.Load("https://www.stackoverflow.com");
foreach (HtmlAgilityPack.HtmlNode link in doc.DocumentNode.SelectNodes("//a[#href]"))
{
usercon(link);
}
}
.
public void usercon(string toprint)
{
richTextBox1.Text += "\r\n";
richTextBox1.Text += toprint;
//richTextBox1.
}
I need to be able to convert link to a string so that in can be used in the function usercon
This is my first time using the HtmlAgilityPack.

According to the source code found here:
https://htmlagilitypack.codeplex.com/SourceControl/latest#Release/1_4_0/HtmlAgilityPack/HtmlNode.cs
See also (new) documentation: http://html-agility-pack.net/outer-html
HtmlNode has an OuterHtml property and its source on GitHub.
private void button2_Click(object sender, EventArgs e)
{
MessageBox.Show("In devolopment","Error", MessageBoxButtons.OK);
HtmlAgilityPack.HtmlWeb hw = new HtmlAgilityPack.HtmlWeb();
HtmlAgilityPack.HtmlDocument doc = hw.Load("https://www.stackoverflow.com");
foreach (HtmlAgilityPack.HtmlNode link in doc.DocumentNode.SelectNodes("//a[#href]"))
{
usercon(link.OuterHtml);
}
}

C# Web Browser control only loads one page, will not work on the second attempt

I have urls in a listbox. I am trying to navigate to a url when it is selected.
private void lstURL_SelectedIndexChanged(object sender, EventArgs e)
{
wbrBrowser.Navigate(lstURL.Text);
lblUrl.Text = lstURL.Text;
lblTitle.Text = "Loading...";
System.Windows.Forms.HtmlDocument document = wbrBrowser.Document;
document.MouseUp += new HtmlElementEventHandler(this.htmlDocument_Click);
}
private void wbrBrowser_DocumentCompleted(object sender, WebBrowserDocumentCompletedEventArgs e)
{
lblTitle.Text = wbrBrowser.Document.Title;
}
private void htmlDocument_Click(object sender, HtmlElementEventArgs e)
{
HtmlElement element = this.wbrBrowser.Document.GetElementFromPoint(e.ClientMousePosition);
var savedId = element.Id;
var uniqueId = Guid.NewGuid().ToString();
element.Id = uniqueId;
var doc = new HtmlAgilityPack.HtmlDocument();
doc.LoadHtml(element.Document.GetElementsByTagName("html")[0].OuterHtml);
element.Id = savedId;
var node = doc.GetElementbyId(uniqueId);
var xpath = node.XPath;
lblXpath.Text = xpath;
}
It works the first time I load a page, after that it just freezes and lblTitle.Text just stays at "Loading..."
I have been searching for a while but I can't figure out why this is happening.

Trying to log in to a website through a C# program

I'm new to C# so I looked for this topic in other questions but they weren't for me. What I am trying to do is I currently try to login to my school's servers using a c# program(Which I'm trying to implement). What I'm trying to do is I know the code of the page, so I am using web browser of c# to navigate then I just want to write name and password to the input boxes and this is where I stuck. Can you please give me any advices?
If you want to look at page: https://suis.sabanciuniv.edu/prod/twbkwbis.P_SabanciLogin
Thanks for your advices.
Here how I used the code(Edit: Added eventhandler but this is my first time using so it promts me "object reference not set to a instance of an object"):
private void buttonGo_Click(object sender, EventArgs e)
{
try
{
string input = "https://suis.sabanciuniv.edu/prod/twbkwbis.P_SabanciLogin";
webBrowser1.DocumentCompleted += new WebBrowserDocumentCompletedEventHandler(WebBrowser_DocumentCompleted);
webBrowser1.Navigate(input);
HtmlDocument doc = webBrowser1.Document;
HtmlElement userName = doc.GetElementById("UserID");
HtmlElement pass = doc.GetElementById("PIN");
HtmlElement submit = doc.GetElementById("Login");
userName.SetAttribute("value", textID.Text);
pass.SetAttribute("value", textPASS.Text);
submit.InvokeMember("Click");
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
}
}
public void WebBrowser_DocumentCompleted(object sender, WebBrowserDocumentCompletedEventArgs e)
{
var webBrowser = sender as WebBrowser;
webBrowser.DocumentCompleted -= WebBrowser_DocumentCompleted;
MessageBox.Show(webBrowser.Url.ToString());
}
}
}
Finally I solved problem I cheated a little but managed to solve. Here is the working code:
private void buttonGo_Click(object sender, EventArgs e)
{
try
{
string input = "https://suis.sabanciuniv.edu/prod/twbkwbis.P_SabanciLogin";
webBrowser1.DocumentCompleted += new WebBrowserDocumentCompletedEventHandler(WebBrowser_DocumentCompleted);
webBrowser1.Navigate(input);
HtmlDocument doc = webBrowser1.Document;
//HtmlElement userName = doc.GetElementById("UserID"); These not worked because ID of the elements were hidden so they are here to show which of these did not work.
//HtmlElement pass = doc.GetElementById("password");
HtmlElement submit = webBrowser1.Document.Forms[0].Document.All["PIN"].Parent.Parent.Parent.NextSibling.FirstChild;
//userName.SetAttribute("value", textID.Text);
//pass.SetAttribute("value", textPASS.Text);
webBrowser1.Document.Forms[0].All["UserID"].SetAttribute("value", textID.Text);
webBrowser1.Document.Forms[0].All["PIN"].FirstChild.SetAttribute("value", textPASS.Text);
submit.InvokeMember("Click");
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
}
}
public void WebBrowser_DocumentCompleted(object sender, WebBrowserDocumentCompletedEventArgs e)
{
var webBrowser = sender as WebBrowser;
webBrowser.DocumentCompleted -= WebBrowser_DocumentCompleted;
MessageBox.Show(webBrowser.Url.ToString());
}

You need to find the input boxes of the username and password fields as ID's or nodes first. Then assign them as such:
HtmlDocument doc = webBrowser1.Document;
HtmlElement email = doc.GetElementById("email");
HtmlElement pass = doc.GetElementById("pass");
HtmlElement submit = doc.GetElementById("LoginButton");
email.SetAttribute("value", "InsertYourEmailHere");
//Same for password
submit.InvokeMember("Click");

C# stopping an infinite foreach loop

This foreach loop checks a webpage and sees if there are any images then downloads them. How do i stop it? When i press the button it continues the loop forever.
private void button1_Click(object sender, EventArgs e)
{
WebBrowser browser = new WebBrowser();
browser.DocumentCompleted +=browser_DocumentCompleted;
browser.Navigate(textBox1.Text);
}
void browser_DocumentCompleted(object sender, WebBrowserDocumentCompletedEventArgs e)
{
WebBrowser browser = sender as WebBrowser;
HtmlElementCollection imgCollection = browser.Document.GetElementsByTagName("img");
WebClient webClient = new WebClient();
int count = 0; //if available
int maximumCount = imgCollection.Count;
try
{
foreach (HtmlElement img in imgCollection)
{
string url = img.GetAttribute("src");
webClient.DownloadFile(url, url.Substring(url.LastIndexOf('/')));
count++;
if(count >= maximumCount)
break;
}
}
catch { MessageBox.Show("errr"); }
}

use the break; keyword to break out of a loop

You do not have an infinite loop, you have an exception that is being thrown based on how you are writing the file to disk
private void button1_Click(object sender, EventArgs e)
{
WebBrowser browser = new WebBrowser();
browser.DocumentCompleted += browser_DocumentCompleted;
browser.Navigate("www.google.ca");
}
void browser_DocumentCompleted(object sender, WebBrowserDocumentCompletedEventArgs e)
{
WebBrowser browser = sender as WebBrowser;
HtmlElementCollection imgCollection = browser.Document.GetElementsByTagName("img");
WebClient webClient = new WebClient();
foreach (HtmlElement img in imgCollection)
{
string url = img.GetAttribute("src");
string name = System.IO.Path.GetFileName(url);
string path = System.IO.Path.Combine(Environment.CurrentDirectory, name);
webClient.DownloadFile(url, path);
}
}
That code works fine on my environment. The issue you seemed to be having was when you were setting the DownloadFile filepath, you were setting it to a value like `\myimage.png', and the webclient could not find the path so it threw and exception.
The above code drops it into the current directory with the extension name.

Maybe the Event browser.DocumentCompleted cause the error, if the page refreshes the event gets fired again. You could try to deregister the event.
void browser_DocumentCompleted(object sender, WebBrowserDocumentCompletedEventArgs e)
{
WebBrowser browser = sender as WebBrowser;
browser.DocumentCompleted -= browser_DocumentCompleted;
HtmlElementCollection imgCollection = browser.Document.GetElementsByTagName("img");
WebClient webClient = new WebClient();
foreach (HtmlElement img in imgCollection)
{
string url = img.GetAttribute("src");
string name = System.IO.Path.GetFileName(url);
string path = System.IO.Path.Combine(Environment.CurrentDirectory, name);
webClient.DownloadFile(url, path);
}
}