Develop Reference

WP8 C# Website Scraping with Login

I'm working on a project of a mobile version of an archaic Online Learning System in my campus. I've been trying for weeks to scrape something in this website, but I need to login first in order to get it. I have search anything including HttpWebRequest, CookiesAwareWebClient, etc
My method until now is:
Find the "action" URL in the login form of the site
Sent POST request to that URL
Receive response containing cookies in the Headers["Set-Cookie"]
Create new HttpWebRequest with the URL to the content(that need to be logged in first).
Copy the headers of set-cookie into that request.
Run it (but fails)
I also have tried using CookieCollection in CookieAwareWebClient but it didn't work too.
How to do it properly? Is the location of a Cookie in HttpWebRequest is only in Headers, or in HTTP Packets, where is the location of CookieCollection? Does CookieCollection included in the next request?
Thanks

You need to use a CookieContainer. That will process and hold the cookies for you between HttpWebRequest objects:
var cookieJar = new CookieContainer();
var loginWebRequest = WebRequest.Create(loginUrl) as HttpWebRequest;
loginWebRequest.CookieContainer = cookieJar;
// Execute the Web Request
var authRequiredWebRequest = WebRequest.Create(protectedUrl) as HttpWebRequest;
authRequiredWebRequest.CookieContainer = cookieJar;
// Execute the next request
// It will have the auth cookie set appropriately

Grab the contents of a Drupal website that is secured with a login form

I would like to grab some content from a website that is made with Drupal.
The challenge here is that i need to login on this site before i can access the page i want to scrape. Is there a way to automate this login process in my C# code, so i can grab the secure content?

To access the secured content, you'll need to store and send cookies with every request to your server, starting with the request that sends your log in info and then saving the session cookie that the server gives you (which is your proof that you are who you say you are).
You can use the System.Windows.Forms.WebBrowser for a less control but out-of-the-box solution that will handle cookies.
My preferred method is to use System.Net.HttpWebRequest to send and receive all web data and then use the HtmlAgilityPack to parse the returned data into a Document Object Model (DOM) which can be easily read from.
The trick to getting System.Net.HttpWebRequest to work is that you must create a long-lived System.Net.CookieContainer that will keep track of your log in info (and other things the server expects you to keep track of). The good news is that the HttpWebRequest will take care of all of this for you if you provide the container.
You need a new HttpWebRequest for each call you make, so you must sets their .CookieContainer to the same object every time. Here is an example:
UNTESTED
using System.Net;
public void TestConnect()
{
CookieContainer cookieJar = new CookieContainer();
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://www.mysite.com/login.htm");
request.CookieContainer = cookieJar;
HttpWebResponse response = (HttpWebResponse) request.GetResponse();
// do page parsing and request setting here
request = (HttpWebRequest)WebRequest.Create("http://www.mysite.com/submit_login.htm");
// add specific page parameters here
request.CookeContainer = cookieJar;
response = (HttpWebResponse) request.GetResponse();
request = (HttpWebRequest)WebRequest.Create("http://www.mysite.com/secured_page.htm");
request.CookeContainer = cookieJar;
// this will now work since you have saved your authentication cookies in 'cookieJar'
response = (HttpWebResponse) request.GetResponse();
}
http://msdn.microsoft.com/en-us/library/system.windows.forms.webbrowser.aspx
HttpWebRequest Class
http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.cookiecontainer.aspx

You'll have to use the Services module to do that. Also check out this link for a bit of explanation.

receiving details from POST

im trying to build a program that can "login" to site as user to get the html code,
then ill fix the code to make more options to the user:)
i googled it and as i understand i need to send a cookie to identify myself as the user,
i use firefox edit cookies to see which cookies saves at my cmputer and saw only one 'phpsessid' that saves a string represent the session ,
i use wireshark to see how its real going and as i saw when im getting the response page im getting also this line
PHPSESSID=xxxxxxxxxxxxxxxxxxxxxxx; path=/
how i can read it from the response string to be able surf another pages as the 'user'?
tyvm for your help:)
edit:
i got it:
Req.GetResponse().Headers.Get("Set-Cookie");

Before you do the login, create a CookieContainer and assign it to the request:
var request = (HttpWebRequest)WebRequest.Create(loginUrl);
var cookies = new CookieContainer();
request.CookieContainer = cookies;
After you make the request, cookies will contain the cookies that make you logged-in.
When you want to make another request, now as a logged-in user, use cookies again:
var request = (HttpWebRequest)WebRequest.Create(anotherUrl);
request.CookieContainer = cookies;

C# get or set cookies, to download content from the web using cookies

I need help with cookies. I'm planing use cookies to download web content. To get the content I need to log into a website because only authorized users can download web content or files. I'm using
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
and then I'm scraping source code, and I need to get link to file, but I can't get because I'm not authorized, so I need to set cookies. I have not worked with cookies before.
How do I to do this?

If you are talking about creating a asp.net web app that can create cookies, then doing this should work:
Not sure if this would work in asp.net mvc though, this works in asp.net web forms
Response.Cookies["nameOfCookie"].Value = "someValue";
Response.Cookies["nameOfCookies].Expires = DateTime.Now.AddDays(1);
then on a post back on say same page or different page, can get cookie by
string value = string.Empty;
if (Request.Cookies["nameOfCookie"] != null)
value = Request.Cookies["nameOfCookie"].Value;

I've created a quick little application that helps with generating web requests for me
public class HttpRequestHandler {
private CookieContainer cookies;
public HttpRequestHandler() {
cookies = new CookieContainer();
}
public HttpWebRequest GenerateWebRequest(string url) {
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(new System.Uri(url));
request.CookieContainer = cookies;
request.AllowAutoRedirect = true;
request.KeepAlive = true;
request.Referer = HttpUtility.UrlEncode(referer);
request.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.8) Gecko/2009021910 Firefox/3.0.7 (.NET CLR 3.5.30729)";
request.Headers.Add("Pragma", "no-cache");
request.Timeout = 40000;
return request;
}
}
Your problem is probably related to lack of a CookieContainer. If you create a cookie container you can save/use cookies in your web requests.

You should set the CookieContainer property of the HTTPWebRequest class to an instance of a CookieContainer class. From MSDN, it's stated that:
CookieContainer is null by default. You must assign a CookieContainer object to the property to have cookies returned in the Cookies property of the HttpWebResponse returned by the GetResponse method.
In other words, after you have set the CookieContainer property of the HTTPWebRequest object in your code, you can get the corresponding Cookies in the HTTPWebResponse object in your code. The sample code in the MSDN link shared above should get you started.

Could you be more speciffic about the project? is it a Desktop Application, ASP.NET, ASP.NET MVC?
in ASP.NET MVC I used SetPersistenCookie method every time when user is logging in, and for methods I use Authorize Attribute, every time when a user wants to acces somenthig and he is not logged in it redirects him to LogIn page...

how to get the source code as register user

i downloaded a sourcecode of a site,but i downloaded it i saw it identify my program as a guest,i search at google and figure out that i can send a cookie when i "ask" the source code.
that what i have managed to do and it still dont identify me as register user:
CookieContainer cj = new CookieContainer();
string all = "";
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(Url);
req.CookieContainer = cj;
HttpWebResponse res = (HttpWebResponse)req.GetResponse();
CookieCollection cs=cj.GetCookies(req.RequestUri);
CookieContainer cc = new CookieContainer();
cc.Add(cs);
req.CookieContainer = cc;
StreamReader read = new StreamReader(res.GetResponseStream());
all = read.ReadToEnd();
read.Close();
return all;
what is wrong here?
tyvm for help:)
(if that help,i can have a simple details of a register user of the site).

You would have to use the cookie that the server left behind in your cookie cache that identified you as a authenticated user in a previous session. You'll need to use the Cookie(name, value) constructor. Getting the value is the tricky part, look through your cookie cache to see if you can find one. It is still going to fail if the server expires the cookie.
Using a tool that lets you look at the HTTP headers and cookie values is important to debug this. Firebug is very nice.