Develop Reference

C# screen scraping an ASP.NET web forms page - POST request not completely working

Please bear with me for this slightly long winded description but I'm having a strange problem with C# screen scraping an ASP.NET web forms page. The steps I'm trying to do are as follows:-
1) The site is secured using basic authentication over HTTPS so I need to login appropriately.
2) I'm performing a GET request on the page to retrieve the __VIEWSTATE value (darn thing does nothing if I don't set this thing!)
3) Once logged in there are several form fields to complete then a submit button which POST's the form to the server
4) When the submit button is pressed the form is POST'd to the server and response is the same page and form but now with an extra little HTML table at the bottom of the form with some data I need to get at.
I've so far managed to sort the login and form post using the WebClient class. I've used fiddler (and firebug) to check the POST field values that are being sent when completing the form normally using a browser. I can successfully get a response from the POST request with the data table in question appearing below the form as expected. The problem however is that although the table is populated with data it is populated with data I don't expect. The data that appears is if I completed the form in a browser as normal but with one particular parameter (a drop down list) set to a different value than I'm passing in my POST request to the server. I've confirmed using fiddler and firebug that I'm passing exactly the same POST parameters that are sent as normal using a web browser human completed form. I'm now totally stuck as to why this one parameter is not being 'taken into consideration' by the server?
The one difference is that this particular control is a select list and it performs a page reload or 'postback' when changed. However this doesn't seem to do anything apart from change some other select lists content later in the form.
I guess I'm asking is there anything else I'm missing that would cause this? I'm totally tearing my hair out on this one. Can anyone help? I've posted the code below (with addresses and parameters blanked out for privacy).
// a place to store the html
string responseBody = "";
// create out web client to handle the request
using (WebClient webClient = new WebClient())
{
// space to store responses from the remote site
byte[] responseBytes;
// site uses basic authentication over HTTPS so we'll need to login
CredentialCache credentials = new CredentialCache();
credentials.Add(new Uri(Url), "Basic", new NetworkCredential(Username, Password));
// set the credentials in the web client
webClient.Credentials = credentials;
// a place for __VIEWSTATE
string viewState = "";
// try and get __VIEWSTATE from the web site
try
{
responseBytes = webClient.DownloadData(Url);
viewState = GetHtmlInputValue(Encoding.UTF8.GetString(responseBytes), "__VIEWSTATE");
}
catch (Exception e)
{
bool cancel = false;
ComponentMetaData.FireError(10, "Read web page data", "Error whilst trying to get __VIEWSTATE from web page: " + e.Message, "", 0, out cancel);
}
// add our POST parameters (don't forget the __VIEWSTATE or it won't work as its an ASP.NET web page)
NameValueCollection requestParameters = new NameValueCollection();
// add ASP.NET fields
requestParameters.Add("__EVENTTARGET", __EVENTTARGET);
requestParameters.Add("__EVENTARGUMENT", __EVENTARGUMENT);
requestParameters.Add("__LASTFOCUS", __LASTFOCUS);
// add __VIEWSTATE
requestParameters.Add("__VIEWSTATE", viewState);
// all other form parameters
requestParameters.Add("btnSubmit", btnSubmit);
/* I've hidden the rest of the parameters hidden for privacy just in case */
// see if we can connect and get data
try
{
// set content type
webClient.Headers.Clear();
webClient.Headers.Add("Content-Type", "application/x-www-form-urlencoded");
// 'POST' the form data using web client and hope we get a response
responseBytes = webClient.UploadValues(Url, "POST", requestParameters);
// transform the response to a string
responseBody = Encoding.UTF8.GetString(responseBytes);
}
catch (Exception e)
{
bool cancel = false;
ComponentMetaData.FireError(10, "Read web page data", "Error whilst trying to connect to web page: " + e.Message, "", 0, out cancel);
}
}
Please ignore the 'ComponentMetaData' references as this is part of SSIS script source.
Any ideas or help will be greatly appreciated - cheers!
RE: thanks for the quick responses, all I can say to those comments is...
There's the normal ASP session cookie but there's no values in the cookie (apart from the session ID of course), I figured as the site is using basic authentication not forms authentication I could just ignore the cookie - and as I'm getting into the site and getting data returned this was ok. I guess it's worth a try but I'll have to just alter the code to use the WebRequest class method instead...
As for the select list javascript, no there's no javascript changing the value of the select list after page load. The only javascript on the select list is an onchange event to do a 'postback' which only seems to change some other select lists on the form that are empty anyway in the final POST. Note I'm including all the POST parameters when generating the POST request even if they're empty and I'm also including all the 'web forms' special fields such as __VIEWSTATE, __EVENTTARGET etc...
I'm no expert in web forms (MVC man myself) but is there anything else that the web forms 'engine' is expecting? I've sent 1 header for the 'Content-Type' of 'application/x-www-form-urlencoded' but I've tried setting others such as copying the 'User-Agent' header from the original POST but this ends up with me getting a 500 error from the server, not sure why that would happen??
Here's the code for the 'GetHtmlInputValue' its a bit simple/basic and could be done better but:-
private string GetHtmlInputValue(string html, string inputID)
{
string valueDelimiter = "value=\"";
int namePosition = html.IndexOf(inputID);
int valuePosition = html.IndexOf(valueDelimiter, namePosition);
int startPosition = valuePosition + valueDelimiter.Length;
int endPosition = html.IndexOf("\"", startPosition);
return html.Substring(startPosition, endPosition - startPosition);
}

If I understand you correctly, then selecting an item in the dropdown will cause a POST to be performed, and the server alters the available options in another part of the form. The server will then include the current value of the dropdown in the __VIEWSTATE field value.
When you perform the scraping, you should make sure that the __VIEWSTATE contains the desired value for the dropdown. To investigate further, try to decode the viewstate from the server and see which values are sent back.

How to fill and submit a web form on a third party website using windows application? [closed]

Closed. This question needs debugging details. It is not currently accepting answers.
Edit the question to include desired behavior, a specific problem or error, and the shortest code necessary to reproduce the problem. This will help others answer the question.
Closed 8 years ago.
Improve this question
I am doing a project in which I have to make a windows application that can Take a URL in textbox from user. Now when the user press the Proceed button, the application should open that URl in a webbrowser control and fill the form on that page containing userID & password textboxes and submit it via the login button on that web page. Now my application should show the next page in that webbrowser control to the user.
I can open the url in the application's webbrowser control through my C# Code, but I can't figure it out that how to find the userID & pasword textboxes on that web page that is currently opened in the webbrowser control of my application, how to fill them, how to find the login button & how to click it through my C# Code.

For this you will have to look into the page source of the 3rd party site and find the id of the username, password textbox and submit button. (If you provide a link I would check it for you). Then use this code:
//add a reference to Microsoft.mshtml in solution explorer
using mshtml;
private SHDocVw.WebBrowser_V1 Web_V1;
Form1_Load()
{
Web_V1 = (SHDocVw.WebBrowser_V1)webBrowser1.ActiveXInstance;
}
webBrowser1_Document_Complete()
{
if (webBrowser1.ReadyState == WebBrowserReadyState.Complete)
{
if (webBrowser1.Url.ToString() == "YourLoginSite.Com")
{
try
{
HTMLDocument pass = new HTMLDocument();
pass = (HTMLDocument)Web_V1.Document;
HTMLInputElement passBox = (HTMLInputElement)pass.all.item("PassIDThatyoufoundinsource", 0);
passBox.value = "YourPassword";
HTMLDocument log = new HTMLDocument();
log = (HTMLDocument)Web_V1.Document;
HTMLInputElement logBox = (HTMLInputElement)log.all.item("loginidfrompagesource", 0);
logBox.value = "yourlogin";
HTMLInputElement submit = (HTMLInputElement)pass.all.item("SubmitButtonIDFromPageSource", 0);
submit.click();
}
catch { }
}
}
}

I would use Selenium as opposed to the WebBrowser control.
It has an excellent C# library, and this kind of thing is the main reason it was developed.

You don't have to simulate filling in the username/password fields nor clicking on the login button. You need to simulate the browser rather than the user.
Read the login page html and parse it to find the ids of the username and password fields. The username can be obtained by looking for tags with name set as "username", "user", "login", etc. The password will usually be an tag with type="password". Javascript based popup panels for login would involve parsing the js.
Then follow the example code shown here, How do you programmatically fill in a form and 'POST' a web page?

The important thing here is that you're simulating a browser POST event. Don't worry about text boxes and other visual form elements, your goal is to generate a HTTP POST request with the appropriate key-value pairs.
Your first step is to look through the HTML of the page you're pretend to be and figure out the names of the user id and password form elements. So, let's say for example that they're called "txtUsername" and "txtPassword" respectively, then the post arguments that the browser (or user-agent) will be sending up in its POST request will besomething like:
txtUsername=fflintstone&txtPassword=ilikerocks
As a background to this, you might like to do a little research on how HTTP works. But I'll leave that to you.
The other important thing is to figure out what URL it posts this login request to. Normally, this is whatever appears in the address bar of the browser when you log in, but it may be something else. You'll need to check the action attribute of the form element so see where it goes.
It may be useful to download a copy of Fiddler2. Yes, weird name, but it's a great web debugging tool that basically acts as a proxy and captures everything going between the browser and the remote host. Once you figure out how to use it, you can then pull apart each request-response to see what's happening. It'll give you the URL being called, the type of the request (usually GET or POST), the request arguments, and the full text of the response.
Now, you want to build your app. You need to build logic which make the correct HTTP requests, pass in the form arguments, and get back the results. Luckily, the System.Net.HttpWebRequest class will help you do just that.
Let's say the login page is at www.hello.org/login.aspx and it expects you to POST the login arguments. So your code might look something like this (obviously, this is very simplified):
Imports System.IO
Imports System.Net
Imports System.Web
Dim uri As String = "http://www.hello.org/login.aspx"
Dim request As HttpWebRequest = DirectCast(WebRequest.Create(uri), HttpWebRequest)
request.Timeout = 10000 ' 10 seconds
request.UserAgent = "FlintstoneFetcher/1.0" ' or whatever
request.Accept = "text/*"
request.Headers.Add("Accept-Language", "en")
request.Method = "POST"
Dim data As Byte() = New ASCIIEncoding().GetBytes("txtUsername=fflintstone&txtPassword=ilikerocks")
request.ContentType = "application/x-www-form-urlencoded"
request.ContentLength = data.Length
Dim postStream As Stream = request.GetRequestStream()
postStream.Write(data, 0, data.Length)
postStream.Close()
Dim webResponse As HttpWebResponse
webResponse = DirectCast(request.GetResponse(), HttpWebResponse)
Dim streamReader As StreamReader = New StreamReader(webResponse.GetResponseStream(), Encoding.GetEncoding(1252))
Dim response As String = streamReader.ReadToEnd()
streamReader.Close()
webResponse.Close()
The response string now contains the full response text from the remote host, and that host should consider you logged in. You may need to do a little extra work if the remote host is trying to set cookies (you'll need to return those cookies). Alternatively, if it expects you to pass integrated authentication on successive pages, you'll need to add credentials to your successive requests, something like:
request.Credentials = New NetworkCredential(theUsername, thePassword)
That should be enough information to get cracking. I would recommend that you modularise your logic for working with HTTP into a class of its own. I've implemented a complex solution that logs into a certain website, navigates to a pre-determined page, parses the html and looks for a daily file to be downloaded in the "invox" and if it exists then downloads it. I set this up as a batch process which runs each morning, saving someone having to do this manually. Hopefully, my experience will benefit you!

Set C# Property Value by Javascript

I have a C# Property CategoryID, I want to set it's value in Javascript.
I am trying to set the value CategoryID like below:
var sPath = window.location.pathname;
var catId = null;
var sPage = sPath.substring(sPath.lastIndexOf('/') + 1);
if (sPage == 'xyz.aspx')
{
<%=CommonUtility.CategoryID=4%>;
}
else if(sPage == 'zxy.aspx')
{
<%=CommonUtility.CategoryID=5%>;
}
But by this method I always get the value of CategoryID= 5(which is in else block) .
Please suggest me how can get the Property value based on condition.

You can't set a C# property from a client-side (js). You may use ajax to do some work, but you simply can't manipulate server-side code.
edit:
if you still wonder how it's possible you get a value, see Mike's explanation of that fact. But the truth remains. You can't. It's impossible. If you want to know the longer explanation, see how asp.net actually works, it's lifecycle etc. Simple way of putting it would be like this:
A user sends a request to the server using his browser. The server receives it, creates a requested page and instantiates needed classes etc. Then it's gets parsed and sent to the client as html (and other resources of course, like images, css...). The instantiated page class CAN'T be accessed and modified afterwards by the client, because it's already flushed by the server. Every request creates a new instance. There's no way of interacting js with c# anyway. Can you imagine what it would be like, if you could use some js to modify C# on a remote server? It doesn't make sense at all.

You cannot set properties in your code-behind using client-side script this way. The only way to do something like that would be to use AJAX to send data to your server, although I'm pretty sure that's not appropriate for your case.
When you call <%=CommonUtility.CategoryID = 4%>, the server actually executes that statement when it is parsing the page before it sends it to the client. The reason that the property value is 5 is that both of those statements get executed, regardless of the logic in your Javascript if block. Your client side code will not actually be executed by the browser until the server has already parsed both of those tags, which at that point it would be too late to accomplish what you want anyways.
Is there any reason that you simply can't do all of this in the code-behind on page load? Is there some reason you feel like this has to be handled in JS?
Edit:
If you are unable to access the code-behind file (.aspx.vb or .aspx.cs) then simply use a server script block in the top of your .aspx page
<%
If (Request.Path.ToLower().Contains("xyz.aspx")) Then
CommonUtility.CategoryId = 4
ElseIf (Request.Path.ToLower().Contains("zxy.aspx")) Then
CommonUtility.CategoryId = 5
End If
%>

You can't set the C# variable from client script, because all the server code runs first, then the page is sent to the browser.
The client code will end up looking like this:
var sPath = window.location.pathname;
var catId = null;
var sPage = sPath.substring(sPath.lastIndexOf('/') + 1);
if (sPage == 'xyz.aspx')
{
4;
}
else if(sPage == 'zxy.aspx')
{
5;
}
}

Get posted data in string form from another source in ASP.NET

I have a URL and data is being posted on that URL through ERP software from another vendor.....I want to collect the data posted on my URL in page load event from that vendor....What should be done for that in ASP.NET with c#?
He does not have a field name and he is auto-generating the string of data and then posting it automatically to my ASP.NET page.

First if you know what kind of data you're going to receive then you should add:
Page.Response.ContentType = "text/xml"; //For XML Data
Then read that data in stream reader:
StreamReader sr = new StreamReader(Page.Request.InputStream);
The data in streamreader is Url Encoded. So you've to decode it before you use that further:
string main = Server.UrlDecode(sr.ReadToEnd());
That's all. I hope it helps.

var parameter = Request.QueryString["parameterName"];
if (parameter != null)
{
//.. use it
}
See HttpRequest.QueryString

I understand you said that the Query doesn't have a field name; meaning you can't look for it like a normal QueryString, using the string indexer. If so, then you probably have to access it without knowing the query key.
assuming you know that the 'data' is the first parameter, you could access it like this:
string data = Request.QueryString.getKey(0);
If that won't work, you can access the url directly
string query = Request.Url.Query;

If they are doing a POST, you can use Request.Form. It will return a NameValueCollection of the elements posted to the Url, and you can loop through it if you don't know the name of what is being posted. If you know the name, then you can do Request.Form["NamedItem"].

foreach(var key in Request)
{
var data = Request[key];
}
will iterate through Request.Querystrung, Request.Form, and Request.Params.

Why Response.Write behavior varies in the given scenario?

When i POST the page using the following code, the Response.write("Hey") doesn't write the content ("Hey") to the parent page
<form method="post" name="upload" enctype="multipart/form-data"
action="http://localhost:2518/Web/CrossPage.aspx" >
<input type="file" name="filename" />
<input type="submit" value="Upload Data File" name="cmdSubmit" />
</form>
But When i use following code , and POST the data, the Response.write("Hey") can be obtained in the parent page
HttpWebRequest requestToSender = (HttpWebRequest)WebRequest.Create("http://localhost:2518/Web/CrossPage.aspx");
requestToSender.Method = "POST";
requestToSender.ContentType = "multipart/form-data";
HttpWebResponse responseFromSender = (HttpWebResponse)requestToSender.GetResponse();
string fromSender = string.Empty;
using (StreamReader responseReader = new StreamReader(responseFromSender.GetResponseStream()))
{
fromSender = responseReader.ReadToEnd();
}
In the CrossPage.aspx i have the following code
if (!Page.IsPostBack)
{
NameValueCollection postPageCollection = Request.Form;
foreach (string name in postPageCollection.AllKeys)
{
Response.Write(name + " " + postPageCollection[name]);
}
HttpFileCollection postCollection = Request.Files;
foreach (string name in postCollection.AllKeys)
{
HttpPostedFile aFile = postCollection[name];
aFile.SaveAs(Server.MapPath(".") + "/" + Path.GetFileName(aFile.FileName));
}
Response.Write("Hey");
}
I don't have any code in the Page_Load event of parent page.?
What could be the cause? I need to write the "hey" to the Parent page using the first scenario. Both the application are of different domain.
Edit: "Hey" would be from the CrossPage.aspx. I need to write this back to the Parent Page
when i post using the form action, after processing the Page_Load() event in CrossPage.aspx, the URL points to "http://localhost:2518/Web/CrossPage.aspx" which means the application is still in the CrossPage.aspx and didn't move to parent page.

You almost hit on the reason yourself:
when i post using the form action, after processing the Page_Load() event in CrossPage.aspx, the URL points to "http://localhost:2518/Web/CrossPage.aspx" which means the application is still in the CrossPage.aspx and didn't move to parent page.
Your form takes the user to CrossPage.aspx, so the parent page is gone, now the previous page in the user's history.
It sounds like you are trying to do some sort of asynchronous file upload. Try looking for AJAX file upload examples, something like this: How can I upload files asynchronously?

Probably it is because you have the code in an
if (!Page.IsPostBack) block? this code will be executed only the page is not loaded on a post-back.
(HttpWebResponse)requestToSender.GetResponse(); will trigger a GET request, that's why your code is working when you call Crosspage.aspx using that code.

You are treating the page like a service. E.G. Start at ParentPage.aspx > pass data to ServicePage.aspx for processing > write response back to ParentPage.aspx for display.
You got it to work with C# by passing the duty back to the server, where state can easily be maintained while crossing page boundries. It's not so simple when you try to solve the problem without C#. This isn't a Winform app. As NimsDotNet pointed out, changing the method to "get" will get you closer, but you will get redirected to CrossPage.aspx and lose the calling page.
You said you are on "different domains". By this I think you mean your using two different IIS servers. The C# solution should still work in this scenerio, just as you've shown. Just add a Response.Write() of the fromSender object. There's nothing you've told us that makes this not technically possible. Still, if you want a client side solution you could use javascript to make the get request without getting redirected.
This post shows you how to make a get request with JQuery.

I say your aFile.SaveAs(Server.MapPath(".") + "/".... is throwing an exception. try commenting it out and testing it.
UPDATE:
I'm guessing it works from a HttpWebRequest because there is no file being posted therefore the file loop is skipped. when posting from the HTML you have a file input so your file loop is getting used and resulting in the save logic being executed. so again, that leads me to think it's your save logic
Also,I think you have a try catch statement wrapped around all this that is catching exception so u have no idea what's going wrong. If that's the case, never do that. You rarely ever want to catch exception.
After quick glance at ur save logic, replace "/" with #"\". Server.MapPath(".") returns the path using back slashes not forward slashes.

I would suggest changing CrossPage.aspx into an .ashx HttpHandler. The Page.IsPostBack may not be working correctly because it is expecting ViewState and other hidden ASP.net form fields that tell it that it's a post back from an ASP form. You also don't need to go through the whole Page life cycle functionality that ASP.net Webforms goes through.