Develop Reference

How to have C# Webbrowser handle webpage login popup for webscraping

I'm trying to programmatically login to a site like espn.com. The way the site is setup is once I click on the Log In button located on the homepage, a Log In popup window is displayed in the middle of the screen with the background slightly tinted. My goal is to programmatically obtain that popup box, supply the username and password, and submit it -- hoping that a cookie is returned to me to use as authentication. However, because Javascript is used to display the form, I don't necessarily have easy access to the form's input tags via the main page's HTML.
I've tried researching various solutions such as HttpClient and HttpWebRequest, however it appears that a Webbrowser is best since the login form is displayed using Javascript. Since I don't necessarily have easy access to the form's input tags, a Webbrowser seems the best alternative to capturing the popup's input elements.
class ESPNLoginViewModel
{
private string Url;
private WebBrowser webBrowser1 = new WebBrowser();
private SHDocVw.WebBrowser_V1 Web_V1;
public ESPNLoginViewModel()
{
Initialize();
}
private void Initialize()
{
Url = "http://www.espn.com/";
Login();
}
private void Login()
{
webBrowser1.Navigate(Url);
webBrowser1.DocumentCompleted +=
new WebBrowserDocumentCompletedEventHandler(webpage_DocumentCompleted);
Web_V1 = (SHDocVw.WebBrowser_V1)this.webBrowser1.ActiveXInstance;
Web_V1.NewWindow += new SHDocVw.DWebBrowserEvents_NewWindowEventHandler(Web_V1_NewWindow);
}
//This never gets executed
private void Web_V1_NewWindow(string URL, int Flags, string TargetFrameName, ref object PostData, string Headers, ref bool Processed)
{
//I'll start determing how to code this once I'm able to get this invoked
}
private void webpage_DocumentCompleted(object sender, WebBrowserDocumentCompletedEventArgs e)
{
HtmlElement loginButton = webBrowser1.Document.GetElementsByTagName("button")[5];
loginButton.InvokeMember("click");
//I've also tried the below InvokeScript method to see if executing the javascript that
//is called when the Log In button is clicked, however Web_V1_NewWindow still wasn't called.
//webBrowser1.Document.InvokeScript("buildOverlay");
}
}
I'm expecting the Web_V1_NewWindow handler to be invoked when the InvokeMember("click") method is called. However, code execution only runs through the webpage_DocumentCompleted handler without any calls to Web_V1_NewWindow. It might be that I need to use a different method than InvokeMember("click") to invoke the Log In button's click event handler. Or I might need to try something completely different altogether. I'm not 100% sure the Web_V1.NewWindow is the correct approach for my needs, but I've seen NewWindow used often when dealing with popups so I figured I should give it a try.
Any help would be greatly appreciated as I've spent a significant amount of time on this.

I know it is the late answer. But it will help someone else.
You can extract the value from FRAME element by following
// Get frame using frame ID
HtmlWindow frameWindow = (from HtmlWindow win
in WbBrowser.Document.Window.Frames select win)
.Where(x => string.Compare(x.WindowFrameElement.Id, "frm1") == 0)
.FirstOrDefault();
// Get first frame textbox with ID
HtmlElement txtElement = (from HtmlElement element
in frameWindow.Document.GetElementsByTagName("input")
select element)
.Where(x => string.Compare(x.Id, "txt") == 0).FirstOrDefault();
// Check txtElement is nul or not
if(txtElement != null)
{
Label1.Text = txtElement.GetAttribute("value");
}
For more details check
this article

How to stop Windows Form program execution until text box input is given?

I'm building a windows form application that stores employee information records in a database, including SSN. Next to the textbox where the SSN is input is requested, I have a checkbox that when clicked, shows me the full SSN instead of the last four digits. To ensure that only an administrator is accessing this information, I created a prompt form connected to a MS SQL DB that stores an admin password and would like to ask the user for a password for security purposes. Also, i'd like to be able to call this form whenever needed. I successfully implemented it but would like to add a feature that allows for 3 tries. Is there a way to stop the program execution and keep prompting the user for input in a textBox?
output is a variable that stores the result of the 'SELECT' query that gets the password.
confirmation is the Accept Button.
The only option i could think of forcing input was calling a new form. Only problem is, this code is inside the form and my gut tells me that's not the answer to this problem. I must be missing something.
confirmation.Click += (sender, e) => {
//If Password is correct.
if (textBox.Text == output)
{
isCorrect = true;
Pprompt.Close();
}
else
{
isCorrect = false;
//While the password is incorrect.
while (isCorrect == false)
{
//textBox.Text = "";
if (textBox.Text == output)
{
isCorrect = true;
Pprompt.Close();
break;
}
tryCount++;
if (tryCount == 3)
{
MessageBox.Show("Access Denied.");
break;
}
}
}
}
What I'd like to happen is for the form to keep asking me for input until the try limit is exceeded.

You cannot have the loop inside the click handler, because the UI freezes while it is running and the user does not get the opportunity make any entries.
Process only one entry and process the next when the user clicks the button again.
confirmation.Click += (sender, e) => {
if (textBox.Text == output) // Password is correct.
{
isCorrect = true;
Pprompt.Close();
}
else
{
isCorrect = false;
textBox.Text = "";
tryCount++;
if (tryCount == 3)
{
MessageBox.Show("Access Denied.");
Pprompt.Close();
}
}
}
Unless you are using multi-threading or async/await, the following is true:
Winforms is event based. I.e., if no event handler is running, no code is running.
If code (i.e. an event handler) is running, the user interface (UI) is frozen and the user cannot make any input. He cannot enter text or click any buttons, cannot scroll lists and cannot resize or move the window.
While an event handler is running, no other event handler will ever be called. I.e., an event handler will never be interrupted by another one. This prevents you from having to deal with multi-threading issues unless you are using multi-threading explicitly.
If an event should be fired (e.g. a timer tick) while an event handler (e.g. a button click handler) is running, the execution of the event handler associated to this new event will be deferred until after the first event handler returns.

MS Band SDK - Button pressed event handler not being called

I've set up my tile and page layout with a button in my app but when I press the button the event handler does not get called. I tried with the tile open event handler but that doesn't work either. My code is as follows:
private async void OnConnectToBand()
{
IBandInfo[] pairedBands = await BandClientManager.Instance.GetBandsAsync();
try
{
using (IBandClient bandClient = await BandClientManager.Instance.ConnectAsync(pairedBands[0]))
{
//add tile, create page layout with button and add content with button
//subscribe to listeners
bandClient.TileManager.TileButtonPressed += EventHandler_TileButtonPressed;
// Start listening for events
bandClient.TileManager.StartReadingsAsync();
}
}
catch(BandException ex)
{
//handle a Band connection exception
}
}
void EventHandler_TileButtonPressed(object sender, BandTileEventArgs<IBandTileButtonPressedEvent> e)
{
// handle event
}
The tile and page get created fine but the button doesn't trigger the event handler. Any ideas why it's not being called?
UPDATE: I just went through my code and the SDK doco again and remembered I'm doing something different which is why it might not be working. The doco has the following for adding the button to the layout which doesn't compile:
// create the content to assign to the page
PageData pageContent = new PageData
(
pageGuid,
0, // index of our (only) layout
new Button(
TilePageElementId.Button_PushMe,
“Push Me!”)
);
The compiler says there isn't a constructor for Button that takes in 2 arguments.
I assumed there was an error in the sample code and changed it to TextButtonData which compiles fine but now I'm wondering if that is why the event handler isn't working? Code is:
PageData pageContent = new PageData(
pageGuid,
0, // index of our (only) layout
new TextButtonData(
(short)TilePageElementId.Button_PushMe, "Push"));
Any ideas?

This is great to see someone developing on the MS Band.... heres a few links that discuss the OnConnectToBand and its setup
void EventHandler_TileButtonPressed(object sender,
BandTileEventArgs<IBandTileButtonPressedEvent> e)
{
// This method is called when the user presses the
// button in our tile’s layout.
//
// e.TileEvent.TileId is the tile’s Guid.
// e.TileEvent.Timestamp is the DateTimeOffset of the event.
// e.TileEvent.PageId is the Guid of our page with the button.
// e.TileEvent.ElementId is the value assigned to the button
// in our layout (i.e.,
// TilePageElementId.Button_PushMe).
//
// handle the event
}
Section 9- Handling custom events
http://developer.microsoftband.com/Content/docs/Microsoft%20Band%20SDK.pdf
Talks about adding, clicking, removing tiles
http://www.jayway.com/2015/03/04/first-impression-of-microsoft-band-developing-2/

Try adding a dialog(below is windows code, for ios or android have a look at the above mentioned manual) to respond to the event (in your code above there is nothing in your event handler? this to see if it actually does something?
using Microsoft.Band.Notifications;
try
{
// send a dialog to the Band for one of our tiles
await bandClient.NotificationManager.ShowDialogAsync(tileGuid,
"Dialog title", "Dialog body");
}
catch (BandException ex)
{
// handle a Band connection exception
}

You can only receive events from the Band while you have an active IBandClient instance (i.e. an active connection to the Band). In your code above, the bandClient instance is disposed of immediately after StartReadingsAsync() is called, due to the use of the using() {} block. When an IBandClient instance is disposed, it causes the application to disconnect from the Band.
You need to hold onto the IBandClient instance for the length of time during which you wish to receive events, and dispose of the instance only after that time.

Button is Clicking Before File Upload is Complete in Webbrowser C#

On a Windows Form, I am using a Webbrowser control in C#. It's job is to upload a file and then press the submit button. My only problem is that my code tries to press the submit button before the file is finished uploading. I tried using:
System.Threading.Thread.Sleep(1000);
In between the two tasks (commented out below). This seems to pause the entire process so that didn't work. Can anyone tell me what the best way to do this is?
private void imageBrowser_DocumentCompleted(object sender, WebBrowserDocumentCompletedEventArgs e)
{
this.imageBrowser.DocumentCompleted -= imageBrowser_DocumentCompleted;
Populate().ContinueWith((_) =>
{
//MessageBox.Show("Form populated!");
}, TaskScheduler.FromCurrentSynchronizationContext());
//System.Threading.Thread.Sleep(10000);
try
{
var buttons = imageBrowser.Document.GetElementsByTagName("button");
foreach (HtmlElement button in buttons)
{
if (button.InnerText == "done")
{
button.InvokeMember("click");
}
}
}
catch
{
//debug
}
}
async Task Populate()
{
var elements = imageBrowser.Document.GetElementsByTagName("input");
foreach (HtmlElement file in elements)
{
if (file.GetAttribute("name") == "file")
{
file.Focus();
await PopulateInputFile(file);
}
}
}
async Task PopulateInputFile(HtmlElement file)
{
file.Focus();
// delay the execution of SendKey to let the Choose File dialog show up
var sendKeyTask = Task.Delay(500).ContinueWith((_) =>
{
// this gets executed when the dialog is visible
SendKeys.Send("C:\\Users\\00I0I_c0OlVXtE6FO_600x450.jpg" + "{ENTER}");
System.Threading.Thread.Sleep(1000);
SendKeys.Send("{ENTER}");
}, TaskScheduler.FromCurrentSynchronizationContext());
file.InvokeMember("Click"); // this shows up the dialog
await sendKeyTask;
// delay continuation to let the Choose File dialog hide
await Task.Delay(500);
//SendKeys.Send("{ENTER}");
}

Is the WebBrowser loading a local file? Can you post the html code as well?
I came across such a situation when I was working with google-maps-api-3. I was setting some markers on the form in the WebBrowser_DocumentCompleted but was getting a null object exception. So I moved the code for set marker to a .NET Button control. I noticed that the exception was not thrown when I set the marker after the map tiles completed loading. DocumentCompleted was firing before the tiles got loaded and I was getting a null object exception.
So what I did was to use a tilesLoaded event in my javascript. In this event, I set a property back in C# code which set the markers in the OnPropertyChanged event.
I know what I am posting here is not a solution. But if you post your html code, I can give you answer with some code.

I solved this. The code I was using to click the button was in the wrong spot. The code now looks like so:
private void imageBrowser_DocumentCompleted(object sender, WebBrowserDocumentCompletedEventArgs e)
{
this.imageBrowser.DocumentCompleted -= imageBrowser_DocumentCompleted;
try
{
Populate().ContinueWith((_) =>
{
var buttons = imageBrowser.Document.GetElementsByTagName("button");
foreach (HtmlElement button in buttons)
{
if (button.InnerText == "done")
{
button.InvokeMember("click");
}
}, TaskScheduler.FromCurrentSynchronizationContext());
}
catch
{
//debug
}
}
My mistake was thinking in terms of having a certain amount of seconds pass before executed the next line of code, when I should have been thinking in terms of having the next line of code execute when the previous task was complete.

How to make WebBrowser wait till it loads fully?

I have a C# form with a web browser control on it.
I am trying to visit different websites in a loop.
However, I can not control URL address to load into my form web browser element.
This is the function I am using for navigating through URL addresses:
public String WebNavigateBrowser(String urlString, WebBrowser wb)
{
string data = "";
wb.Navigate(urlString);
while (wb.ReadyState != WebBrowserReadyState.Complete)
{
Application.DoEvents();
}
data = wb.DocumentText;
return data;
}
How can I make my loop wait until it fully loads?
My loop is something like this:
foreach (string urlAddresses in urls)
{
WebNavigateBrowser(urlAddresses, webBrowser1);
// I need to add a code to make webbrowser in Form to wait till it loads
}

Add This to your code:
webBrowser1.DocumentCompleted += new WebBrowserDocumentCompletedEventHandler(webBrowser1_DocumentCompleted);
Fill in this function
private void webBrowser1_DocumentCompleted(object sender, WebBrowserDocumentCompletedEventArgs e) {
//This line is so you only do the event once
if (e.Url != webBrowser1.Url)
return;
//do you actual code
}

After some time of anger of the crappy IE functionality I've came across making something which is the most accurate way to judge page loaded complete.
Never use the WebBrowserDocumentCompletedEventHandler event
use WebBrowserProgressChangedEventHandler with some modifections seen below.
//"ie" is our web browser object
ie.ProgressChanged += new WebBrowserProgressChangedEventHandler(_ie);
private void _ie(object sender, WebBrowserProgressChangedEventArgs e)
{
int max = (int)Math.Max(e.MaximumProgress, e.CurrentProgress);
int min = (int)Math.Min(e.MaximumProgress, e.CurrentProgress);
if (min.Equals(max))
{
//Run your code here when page is actually 100% complete
}
}
Simple genius method of going about this, I found this question googling "How to sleep web browser or put to pause"

According to MSDN (contains sample source) you can use the DocumentCompleted event for that. Additional very helpful information and source that shows how to differentiate between event invocations can be found here.

what you experiencend happened to me . readyStete.complete doesnt work in some cases. here i used bool in document_completed to check state
button1_click(){
//go site1
wb.Navigate("site1.com");
//wait for documentCompleted before continue to execute any further
waitWebBrowserToComplete(wb);
// set some values in html page
wb.Document.GetElementById("input1").SetAttribute("Value", "hello");
// then click submit. (submit does navigation)
wb.Document.GetElementById("formid").InvokeMember("submit");
// then wait for doc complete
waitWebBrowserToComplete(wb);
var processedHtml = wb.Document.GetElementsByTagName("HTML")[0].OuterHtml;
var rawHtml = wb.DocumentText;
}
// helpers
//instead of checking readState . we get state from DocumentCompleted Event via bool value
bool webbrowserDocumentCompleted = false;
public static void waitWebBrowserToComplete(WebBrowser wb)
{
while (!webbrowserDocumentCompleted )
Application.DoEvents();
webbrowserDocumentCompleted = false;
}
form_load(){
wb.DocumentCompleted += (o, e) => {
webbrowserDocumentCompleted = true;
};
}