Develop Reference

C# HttpWebRequest session loss [duplicate]

I am creating an application for data retrieval from the web page. The page is password protected and when the user logs in the cookie is created.
In order to retrieve the data the application first has to log in: make web request with username and password and store the cookie. Then when the cookie is stored it has to be added into the headers of all requests.
Here is the method which makes requests and retrieves the content:
public void getAsyncDailyPDPContextActivationDeactivation()
{
HttpWebRequest httpWebRequest = (HttpWebRequest)WebRequest.Create(dailyPDPContextActivationDeactivation);
IAsyncResult asyncResult = httpWebRequest.BeginGetResponse(null, null);
asyncResult.AsyncWaitHandle.WaitOne();
using (HttpWebResponse httpWebResponse = (HttpWebResponse)httpWebRequest.EndGetResponse(asyncResult))
using (StreamReader responseStreamReader = new StreamReader(httpWebResponse.GetResponseStream()))
{
string responseText = responseStreamReader.ReadToEnd();
}
}
Does anyone know how to modify this method in order to add a cookie into the header?
I would be also thankful if anyone suggested a way to store cookie from the response (when the application makes a request http:xxx.xxx.xxx/login?username=xxx&password=xxx the cookie is created and has to be stored for future requests).

CookieContainer cookieContainer = new CookieContainer();
HttpWebRequest httpWebRequest = (HttpWebRequest)WebRequest.Create(...);
httpWebRequest.CookieContainer = cookieContainer;
Then you reuse this CookieContainer in subsequent requests:
HttpWebRequest httpWebRequest2 = (HttpWebRequest)WebRequest.Create(...);
httpWebRequest2.CookieContainer = cookieContainer;

Use the CookieContainer or you could use a CookieAwareWebClient

Getting php session using c#

I know how to get the source code but the website that I have been working on uses PHP session, this mean that you will need to login(which I have) and use the session ID that the server send back. How do I do this?

Assuming you meant Session, you'll need CookieContainer:
CookieContainer cookies = new CookieContainer();
HttpWebRequest getRequest = (HttpWebRequest)WebRequest.Create(someSite);
getRequest.CookieContainer = cookies;
getRequest.Method = "GET";
HttpWebResponse form = (HttpWebResponse)getRequest.GetResponse();
using (StreamReader response =
new StreamReader(form.GetResponseStream(), Encoding.UTF8))
{
formPage = response.ReadToEnd();
}
You first make a GET request to server and it'll return your SessionID in a cookie. If you need to make new requests to same server, you must pass it through and server will identify you as a returning user.

eventvalidation error for screen scraping

I'm trying to scrape the login page of a site I work on and submit a username/password via code to log into the site. I want to do this in a checking service for site health reasons. I'm running into a few issues, the first of which deals with getting this message:
Exception information:
Exception type: ArgumentException
Exception message: Invalid postback or callback argument. Event validation is enabled using <pages enableEventValidation="true"/> in configuration or <%# Page EnableEventValidation="true" %> in a page. For security purposes, this feature verifies that arguments to postback or callback events originate from the server control that originally rendered them. If the data is valid and expected, use the ClientScriptManager.RegisterForEventValidation method in order to register the postback or callback data for validation.
I've found sites that say I'd have to turn eventvalidation off, but I don't want to do that for security reasons. Is there a way to get around this?
Here is the code. I basically took it right off K. Scott Allen's article here: http://odetocode.com/Articles/162.aspx
StringBuilder sb = new StringBuilder();
var encryptedConnectionString = GlobalDataObject.EncryptSecure("conn string here", GlobalDataObject.Seed);
sb.AppendFormat("client_id={0};", "client");
sb.AppendFormat("client_directory={0};", "client");
sb.AppendFormat("user_id={0};", "12");
sb.AppendFormat("conn_string={0};", encryptedConnectionString);
StringBuilder cookiesString = sb;
HttpWebRequest webRequest = WebRequest.Create("http://localhost/site/login.aspx?c=client") as HttpWebRequest;
webRequest.Headers.Add("Cookie", cookiesString.ToString());
StreamReader responseReader = new StreamReader(
webRequest.GetResponse().GetResponseStream()
);
string responseData = responseReader.ReadToEnd();
responseReader.Close();
// extract the viewstate value and build out POST data
string viewState = ExtractViewState(responseData);
string postData = string.Format("__VIEWSTATE={0}&Login1$Password={1}&Login1$UserName={2}&Login1$LoginButton={3}",
viewState,
HttpUtility.UrlEncode(username),
HttpUtility.UrlEncode(password),
"Log In");
// have a cookie container ready to receive the forms auth cookie
CookieContainer cookies = new CookieContainer();
// now post to the login form
webRequest = WebRequest.Create("http://localhost/site/login.aspx") as HttpWebRequest;
webRequest.Method = "POST";
webRequest.ContentType = "application/x-www-form-urlencoded";
webRequest.CookieContainer = cookies;
// write the form values into the request message
StreamWriter requestWriter = new StreamWriter(webRequest.GetRequestStream());
requestWriter.Write(postData);
requestWriter.Close();
webRequest.AuthenticationLevel = AuthenticationLevel.None;
// we don't need the contents of the response, just the cookie it issues
webRequest.GetResponse().Close(); ///ERROR HAPPENS HERE
// now we can send out cookie along with a request for the protected page
webRequest = WebRequest.Create("http://localhost/site/user/home.aspx") as HttpWebRequest;
webRequest.CookieContainer = cookies;
responseReader = new StreamReader(webRequest.GetResponse().GetResponseStream());
// and read the response
responseData = responseReader.ReadToEnd();
responseReader.Close();
return responseData;
Thanks.

The idea of this error is that it's looking out for requests that are malformed in way that might compromise the app. Being as how it's a login page, I bet you're not trying to pass in unencoded HTML or something.
Edit: Capture the built-up event validation data and send that back with the login request, same as the viewstate.

Screen-scraping a site with a asp.net form login in C#?

Would it be possible to write a screen-scraper for a website protected by a form login. I have access to the site, of course, but I have no idea how to login to the site and save my credentials in C#.
Also, any good examples of screenscrapers in C# would be hugely appreciated.
Has this already been done?

It's pretty simple. You need your custom login (HttpPost) method.
You can come up with something like this (in this way you will get all needed cookies after login, and you need just to pass them to the next HttpWebRequest):
public static HttpWebResponse HttpPost(String url, String referer, String userAgent, ref CookieCollection cookies, String postData, out WebHeaderCollection headers, WebProxy proxy)
{
try
{
HttpWebRequest http = WebRequest.Create(url) as HttpWebRequest;
http.Proxy = proxy;
http.AllowAutoRedirect = true;
http.Method = "POST";
http.ContentType = "application/x-www-form-urlencoded";
http.UserAgent = userAgent;
http.CookieContainer = new CookieContainer();
http.CookieContainer.Add(cookies);
http.Referer = referer;
byte[] dataBytes = UTF8Encoding.UTF8.GetBytes(postData);
http.ContentLength = dataBytes.Length;
using (Stream postStream = http.GetRequestStream())
{
postStream.Write(dataBytes, 0, dataBytes.Length);
}
HttpWebResponse httpResponse = http.GetResponse() as HttpWebResponse;
headers = http.Headers;
cookies.Add(httpResponse.Cookies);
return httpResponse;
}
catch { }
headers = null;
return null;
}

Sure, this has been done. I have done it a couple of times. This is (generically) called Screen-scraping or Web Scraping.
You should take a look at this question (and also browse the questions under the tag "screen-scraping". Note that Scraping does not only relate to data extraction from a web resource. It also involves submission of data to online forms so as mimic the actions of a user when submitting input such as a Login form.

How to make my web scraper log in to this website via C#

I have an application that reads parts of the source code on a website. That all works; but the problem is that the page in question requires the user to be logged in to access this source code. What my program needs a way to initially log the user into the website- after that is done, I'll be able to access and read the source code.
The website that needs to be logged into is:
mmoinn.com/index.do?PageModule=UsersLogin

You can continue using WebClient to POST (instead of GET, which is the HTTP verb you're currently using with DownloadString), but I think you'll find it easier to work with the (slightly) lower-level classes WebRequest and WebResponse.
There are two parts to this - the first is to post the login form, the second is recovering the "Set-cookie" header and sending that back to the server as "Cookie" along with your GET request. The server will use this cookie to identify you from now on (assuming it's using cookie-based authentication which I'm fairly confident it is as that page returns a Set-cookie header which includes "PHPSESSID").
POSTing to the login form
Form posts are easy to simulate, it's just a case of formatting your post data as follows:
field1=value1&field2=value2
Using WebRequest and code I adapted from Scott Hanselman, here's how you'd POST form data to your login form:
string formUrl = "http://www.mmoinn.com/index.do?PageModule=UsersAction&Action=UsersLogin"; // NOTE: This is the URL the form POSTs to, not the URL of the form (you can find this in the "action" attribute of the HTML's form tag
string formParams = string.Format("email_address={0}&password={1}", "your email", "your password");
string cookieHeader;
WebRequest req = WebRequest.Create(formUrl);
req.ContentType = "application/x-www-form-urlencoded";
req.Method = "POST";
byte[] bytes = Encoding.ASCII.GetBytes(formParams);
req.ContentLength = bytes.Length;
using (Stream os = req.GetRequestStream())
{
os.Write(bytes, 0, bytes.Length);
}
WebResponse resp = req.GetResponse();
cookieHeader = resp.Headers["Set-cookie"];
Here's an example of what you should see in the Set-cookie header for your login form:
PHPSESSID=c4812cffcf2c45e0357a5a93c137642e; path=/; domain=.mmoinn.com,wowmine_referer=directenter; path=/; domain=.mmoinn.com,lang=en; path=/;domain=.mmoinn.com,adt_usertype=other,adt_host=-
GETting the page behind the login form
Now you can perform your GET request to a page that you need to be logged in for.
string pageSource;
string getUrl = "the url of the page behind the login";
WebRequest getRequest = WebRequest.Create(getUrl);
getRequest.Headers.Add("Cookie", cookieHeader);
WebResponse getResponse = getRequest.GetResponse();
using (StreamReader sr = new StreamReader(getResponse.GetResponseStream()))
{
pageSource = sr.ReadToEnd();
}
EDIT:
If you need to view the results of the first POST, you can recover the HTML it returned with:
using (StreamReader sr = new StreamReader(resp.GetResponseStream()))
{
pageSource = sr.ReadToEnd();
}
Place this directly below cookieHeader = resp.Headers["Set-cookie"]; and then inspect the string held in pageSource.

You can simplify things quite a bit by creating a class that derives from WebClient, overriding its GetWebRequest method and setting a CookieContainer object on it. If you always set the same CookieContainer instance, then cookie management will be handled automatically for you.
But the only way to get at the HttpWebRequest before it is sent is to inherit from WebClient and override that method.
public class CookieAwareWebClient : WebClient
{
private CookieContainer cookie = new CookieContainer();
protected override WebRequest GetWebRequest(Uri address)
{
WebRequest request = base.GetWebRequest(address);
if (request is HttpWebRequest)
{
(request as HttpWebRequest).CookieContainer = cookie;
}
return request;
}
}
var client = new CookieAwareWebClient();
client.BaseAddress = #"https://www.site.com/any/base/url/";
var loginData = new NameValueCollection();
loginData.Add("login", "YourLogin");
loginData.Add("password", "YourPassword");
client.UploadValues("login.php", "POST", loginData);
//Now you are logged in and can request pages
string htmlSource = client.DownloadString("index.php");

Matthew Brindley, your code worked very good for some website I needed (with login), but I needed to change to HttpWebRequest and HttpWebResponse otherwise I get a 404 Bad Request from the remote server. Also I would like to share my workaround using your code, and is that I tried it to login to a website based on moodle, but it didn't work at your step "GETting the page behind the login form" because when successfully POSTing the login, the Header 'Set-Cookie' didn't return anything despite other websites does.
So I think this where we need to store cookies for next Requests, so I added this.
To the "POSTing to the login form" code block :
var cookies = new CookieContainer();
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(formUrl);
req.CookieContainer = cookies;
And To the "GETting the page behind the login form" :
HttpWebRequest getRequest = (HttpWebRequest)WebRequest.Create(getUrl);
getRequest.CookieContainer = new CookieContainer();
getRequest.CookieContainer.Add(resp.Cookies);
getRequest.Headers.Add("Cookie", cookieHeader);
Doing this, lets me Log me in and get the source code of the "page behind login" (website based moodle) I know this is a vague use of the CookieContainer and HTTPCookies because we may ask first is there a previously set of cookies saved before sending the request to the server. This works without problem anyway, but here's a good info to read about WebRequest and WebResponse with sample projects and tutorial:
Retrieving HTTP content in .NET
How to use HttpWebRequest and HttpWebResponse in .NET

Sometimes, it may help switching off AllowAutoRedirect and setting both login POST and page GET requests the same user agent.
request.UserAgent = userAgent;
request.AllowAutoRedirect = false;