Develop Reference

How to have C# Webbrowser handle webpage login popup for webscraping

I'm trying to programmatically login to a site like espn.com. The way the site is setup is once I click on the Log In button located on the homepage, a Log In popup window is displayed in the middle of the screen with the background slightly tinted. My goal is to programmatically obtain that popup box, supply the username and password, and submit it -- hoping that a cookie is returned to me to use as authentication. However, because Javascript is used to display the form, I don't necessarily have easy access to the form's input tags via the main page's HTML.
I've tried researching various solutions such as HttpClient and HttpWebRequest, however it appears that a Webbrowser is best since the login form is displayed using Javascript. Since I don't necessarily have easy access to the form's input tags, a Webbrowser seems the best alternative to capturing the popup's input elements.
class ESPNLoginViewModel
{
private string Url;
private WebBrowser webBrowser1 = new WebBrowser();
private SHDocVw.WebBrowser_V1 Web_V1;
public ESPNLoginViewModel()
{
Initialize();
}
private void Initialize()
{
Url = "http://www.espn.com/";
Login();
}
private void Login()
{
webBrowser1.Navigate(Url);
webBrowser1.DocumentCompleted +=
new WebBrowserDocumentCompletedEventHandler(webpage_DocumentCompleted);
Web_V1 = (SHDocVw.WebBrowser_V1)this.webBrowser1.ActiveXInstance;
Web_V1.NewWindow += new SHDocVw.DWebBrowserEvents_NewWindowEventHandler(Web_V1_NewWindow);
}
//This never gets executed
private void Web_V1_NewWindow(string URL, int Flags, string TargetFrameName, ref object PostData, string Headers, ref bool Processed)
{
//I'll start determing how to code this once I'm able to get this invoked
}
private void webpage_DocumentCompleted(object sender, WebBrowserDocumentCompletedEventArgs e)
{
HtmlElement loginButton = webBrowser1.Document.GetElementsByTagName("button")[5];
loginButton.InvokeMember("click");
//I've also tried the below InvokeScript method to see if executing the javascript that
//is called when the Log In button is clicked, however Web_V1_NewWindow still wasn't called.
//webBrowser1.Document.InvokeScript("buildOverlay");
}
}
I'm expecting the Web_V1_NewWindow handler to be invoked when the InvokeMember("click") method is called. However, code execution only runs through the webpage_DocumentCompleted handler without any calls to Web_V1_NewWindow. It might be that I need to use a different method than InvokeMember("click") to invoke the Log In button's click event handler. Or I might need to try something completely different altogether. I'm not 100% sure the Web_V1.NewWindow is the correct approach for my needs, but I've seen NewWindow used often when dealing with popups so I figured I should give it a try.
Any help would be greatly appreciated as I've spent a significant amount of time on this.

I know it is the late answer. But it will help someone else.
You can extract the value from FRAME element by following
// Get frame using frame ID
HtmlWindow frameWindow = (from HtmlWindow win
in WbBrowser.Document.Window.Frames select win)
.Where(x => string.Compare(x.WindowFrameElement.Id, "frm1") == 0)
.FirstOrDefault();
// Get first frame textbox with ID
HtmlElement txtElement = (from HtmlElement element
in frameWindow.Document.GetElementsByTagName("input")
select element)
.Where(x => string.Compare(x.Id, "txt") == 0).FirstOrDefault();
// Check txtElement is nul or not
if(txtElement != null)
{
Label1.Text = txtElement.GetAttribute("value");
}
For more details check
this article

Using Window.Open instead of Response.Redirect to open new window?

My page code looks like this:
<asp:Button ID="btnSearch" runat="server" Text="Search" onclick="btnSearch_Click"/>
My method looks like this:
protected void btnSearch_Click(object sender, EventArgs e)
{
var value = lblGraphicNameValue.Text.ToString();
Response.Redirect("Search.aspx?txtGraphicName=" +
value);
}
Currently, when the user press the 'Search' button the page refreshes and loads the Search.aspx page. What I'd like to happen is have the Search.aspx open in a new window, instead. I've looked at using Window.Open, but I'm not sure if this is the correct route, or if I can use the same method of passing in my variable (querystring). Can someone point me in the right direction? What I have works, I just want it to open in a new page while leaving the prior page alone.
EDIT: I should mention that I cannot use javascript (secure environment, every browser has javascript disabled).
From what I'm reading, it seems to indicate that opening a new web page from within an asp.net page and having parms passed in is not do-able without javascript? Is this correct?

This code below ultimately does exactly what I needed it to:
<a href="<%= this.ResolveUrl("Search.aspx?id=" + lblGraphicNameValue.Text.Remove(lblGraphicNameValue.Text.Length -4)) %>"
target="_blank">Search Related</a>
This code does three things:
1) Opens Search in new page.
2) Truncates the search value by four
characters (I only needed part of the search string)
3) Passes in
parameter to new page.
This accomplished exactly what I needed without resorting to custom classes or javascript, although it did make me have to use a link instead of a button.

Use this class.
ResponseHelper .Redirect("popup.aspx", "_blank", "menubar=0,width=100,height=100");
public static class ResponseHelper {
public static void Redirect(string url, string target, string windowFeatures) {
HttpContext context = HttpContext.Current;
if ((String.IsNullOrEmpty(target) ||
target.Equals("_self", StringComparison.OrdinalIgnoreCase)) &&
String.IsNullOrEmpty(windowFeatures)) {
context.Response.Redirect(url);
}
else {
Page page = (Page)context.Handler;
if (page == null) {
throw new InvalidOperationException(
"Cannot redirect to new window outside Page context.");
}
url = page.ResolveClientUrl(url);
string script;
if (!String.IsNullOrEmpty(windowFeatures)) {
script = #"window.open(""{0}"", ""{1}"", ""{2}"");";
}
else {
script = #"window.open(""{0}"", ""{1}"");";
}
script = String.Format(script, url, target, windowFeatures);
ScriptManager.RegisterStartupScript(page,
typeof(Page),
"Redirect",
script,
true);
}
}
}

I think your on the right track, but you're confusing server side code, and client side code. window.open is a Javascript function which works on the client side. So you'll need to render some Javascript from C# to make the window popup. Try:
protected void btnSearch_Click(object sender, EventArgs e)
{
var value = lblGraphicNameValue.Text.ToString();
ClientScript.RegisterStartupScript(this.GetType(), "newWindow", String.Format("<script>window.open('Search.aspx?txtGraphicName={0}');</script>", value));
}
That will re-render the page, and then add a script on pageload that will popup the window. A little warning, this will probably be blocked by a browser popup blocker. If you want to get around that, you can probably achieve this without posting back to the server by using Javascript.

A better option would be to create a javascript function like:
function PreviewPOSTransaction(Id)
{
if (Id != null)
{
window.open('POSTransReport.aspx?TransID=' + Id);
return true;
}
}
</script>
and call this function on button "OnClientClick" event like:
OnClientClick="PreviewPOSTransaction(1);

Specified cast is not valid - accessing WebBrowser in different UI thread

I"m stumped on how my program is working. I'm using threading (was told to do so from another Stack Overflow answer) in order for the webBrowser2.Navigate(Url); in TestScenarios() to run asynchronously inside of the while loop in GetScenarios(). This all works fine.
Now, I added a chunk of code to inject and run some javascript inside of the WebBrowser control. However, every time I call the HtmlElement head = webBrowser2.Document.... line, I get the "Specified cast is not valid error."
I know this error has something to do with the WebBrowser control being accessed in a separate UI thread, and not being able to work that way, but I'm confused on exactly what that means and how I can fix it.
If you need more context just comment.
public void GetScenarios()
{
new Thread(() =>
{
while() {
...
TestScenarios();
}
}).Start();
}
TestScenarios() {
...
Action action = () =>
{
webBrowser2.Tag = signal;
webBrowser2.Navigate(Url);
webBrowser2.DocumentCompleted -= WebBrowserDocumentCompleted;
webBrowser2.DocumentCompleted += WebBrowserDocumentCompleted;
};
webBrowser2.Invoke(action);
signal.WaitOne();
...
//Run some javascript on the WebBrowser control
HtmlElement head = webBrowser2.Document.GetElementsByTagName("head")[0];
HtmlElement scriptEl = webBrowser2.Document.CreateElement("script");
IHTMLScriptElement element = (IHTMLScriptElement)scriptEl.DomElement;
element.text = "function sayHello() { alert('hello') }";
head.AppendChild(scriptEl);
webBrowser2.Document.InvokeScript("sayHello");
}

You are facing this problem because you are accessing the elements of webBrowser before the document is even loaded. You should move this code
HtmlElement head = webBrowser2.Document.GetElementsByTagName("head")[0];
HtmlElement scriptEl = webBrowser2.Document.CreateElement("script");
IHTMLScriptElement element = (IHTMLScriptElement)scriptEl.DomElement;
element.text = "function sayHello() { alert('hello') }";
head.AppendChild(scriptEl);
webBrowser2.Document.InvokeScript("sayHello");
To
WebBrowserDocumentCompleted
event.

First off... you should really step through it in the debugger and figure out what the object you are trying to cast is... this doesn't seem like a threading issue.
Based on your error webBrowser2.Document.GetElementsByTagName("head")[0] is not convertible to an HtmlElement.
You could also try something like this to see what the object is...
var head = webBrowser2.Document.GetElementsByTagName("head")[0] as HtmlElement;
if (head == null)
{
Console.WriteLine(typeof(head); // output the object type somehow
}

Managed to fix it by wrapping the JS scripting chunk in:
webBrowser2.Invoke(new Action(() =>
{
//......
}

Supress the "are you sure you want to leave this page" popup in the .NET webbrowser control

I have a web browser automation project written in WinForms C#.
During the navigation there is a point where the browser does the "are you sure you want to leave this page?" popup.
We need this popup, so I cannot remove it from the website code, which means I have to override it in my automation app.
Does anyone have an idea how to do this?

and here was the smooth solution..
add a reference to mshtml and add using mshtml;
Browser.Navigated +=
new WebBrowserNavigatedEventHandler(
(object sender, WebBrowserNavigatedEventArgs args) => {
Action<HtmlDocument> blockAlerts = (HtmlDocument d) => {
HtmlElement h = d.GetElementsByTagName("head")[0];
HtmlElement s = d.CreateElement("script");
IHTMLScriptElement e = (IHTMLScriptElement)s.DomElement;
e.text = "window.alert=function(){};";
h.AppendChild(s);
};
WebBrowser b = sender as WebBrowser;
blockAlerts(b.Document);
for (int i = 0; i < b.Document.Window.Frames.Count; i++)
try { blockAlerts(b.Document.Window.Frames[i].Document); }
catch (Exception) { };
}
);

Are you able to make any changes to the website code?
If so, you might look at exposing an object through ObjectForScripting, then having the website code check window.external (and possibly interrogating your object) before it decides to display the popup - so if it can't find your object, it assumes it's being used normally and shows it.

Don't need add anymore. Try it. Work like a charm. ^_^
private void webNavigated(object sender, WebBrowserNavigatedEventArgs e)
{
HtmlDocument doc = webBrowser.Document;
HtmlElement head = doc.GetElementsByTagName("head")[0];
HtmlElement s = doc.CreateElement("script");
s.SetAttribute("text", "function cancelOut() { window.onbeforeunload = null; window.alert = function () { }; window.confirm=function () { }}");
head.AppendChild(s);
webBrowser.Document.InvokeScript("cancelOut");
}

calling Javascript from c# using awesomium

I'm trying awesomium for create a basic app, I'm testing the js <----> c# communication but this doesn't seem work well...I create a local html and open it..so far so good..but when I try call js nothing happen, no error, no bug, nothing, simply this doesn't call js..
my basic js code is:
var base = {
newItem : function(item){
$("#botones").append('<div class="botonMenu">' + item + '</div>');
},
other : function(){
alert("hi!!");
}
}
if I test this inside firebug obviously I can call my functions well and the items are created or the alert box...
now..my c# code is this
//I've wrote this code inside the winForms sample..but change the code for load
//my local file and call js....
WebCore.BaseDirectory = #"C:\Documents and Settings\ME\dummytests\codes\views";
webView.LoadFile("base.html");
JSValue param1 = new JSValue("nameItem");
webView.CallJavascriptFunction("base", "other");
webView.CallJavascriptFunction("base","newItem", param1);
webView.Focus();
the file is load well but the js communication didn't work
thanks so much and I hope can help me...this awesomium really look awesome

The problem is that you're trying to call the Javascript on the page before it has finished loading. If you wait until after load has completed, you should see it execute correctly.
webView.LoadCompleted += ExecuteJavascript;
WebCore.BaseDirectory = #"C:\Documents and Settings\ME\dummytests\codes\views";
webView.LoadFile("base.html");
...
private void ExecuteJavascript(object sender, EventArgs eventArgs)
{
JSValue param1 = new JSValue("nameItem");
webView.CallJavascriptFunction("base", "other");
webView.CallJavascriptFunction("base", "newItem", param1);
webView.Focus();
}

This is a solution for Awesomium v1.7.0.5. It uses "JSObject" to get the javascript "window" object. From there it calls a javascript function that uses jQuery to dynamically set the text of a "div". This also uses jQuery to call the function when the document is ready.
One can use the JSObject.Bind method to call C# methods from javascript.
Head:
<script type="text/javascript">
function setDivText(s)
{
$("#msgDiv").text(s);
}
$(document).ready(function () {
setDivText("This is the start up text.");
});
</script>
Body:
<body>
<p>Test...</p>
<p></p>
<div id="msgDiv"></div>
</body>
C#:
This uses WPF WebControl with Name of "webView" inside a Button Click event handler.
using Awesomium.Core;
...
private void Button1_Click(object sender, RoutedEventArgs e)
{
JSObject window = webView.ExecuteJavascriptWithResult("window");
if (window == null)
return;
using (window)
{
window.InvokeAsync("setDivText", "You pressed button 1.");
}
}