Develop Reference

Cookies on Windows Phone 8.1 using HttpClient

I'm using the HttpClient in theSystem.Net.Http but I'm having problems dealing with cookies. I have come through other posts explaining how to setup up cookies with HttpClient. I create a CookieContainer for the HttpClientHandler.CookieHandler and use this container for the cookies. When initialized, its value is obviously 0 and after making the PostAsync, the CookieContainer contains one cookie. I run through a foreach loop to print the values and I get the value of the one cookie. This all seems like it's working as it should.
The problem is that even though it seems as I'm accepting cookies the website redirects me to a page informing me that I do not accept cookies. I have checked with a browser(making it not accept cookies) and it's the exact result i get with my HttpClient.
The overall function of my code is giving values for a login and try to post these. The login seems to be working as expected though it redirects me telling me I don't accept cookies - even though my CookieContainer does contain a cookie. I'm not sure where this goes wrong but it seems like the website can't see that I'm actually accepting cookies. I have tried setting httpClientHandler.UseCookies = true; but without any difference. The website I'm trying to access is my university's system for reading grades and I do not have access to any coding of it.
My code is shown below:
{
var httpClientHandler = new HttpClientHandler();
//Creating cookiecontainer and printing Count for debug purpose
var cookieContainer = new CookieContainer();
Debug.WriteLine("Cookies " + cookieContainer.Count);
httpClientHandler.CookieContainer = cookieContainer;
var httpClient = new HttpClient(httpClientHandler);
HttpResponseMessage response = await httpClient.PostAsync(url, new FormUrlEncodedContent(values));
Debug.WriteLine("Response code" + response.StatusCode);
Debug.WriteLine("Response: " + response.IsSuccessStatusCode);
response.EnsureSuccessStatusCode();
string content = await response.Content.ReadAsStringAsync();
//Print amount of cookies after PostAsync - Should contain cookie now
Debug.WriteLine("Cookies " + cookieContainer.Count);
//Printing the values of the added cookies
foreach (Cookie cookie in cookieContainer.GetCookies(new Uri(url)))
{
Debug.WriteLine("Cookie contains: " + cookie.Value.ToString());
}
}
This is my first Windows Phone App and it is my first time using the HttpClient but I have used a few days now reading about it so hope the answer is not to obvious. Thank you very much - your help is very much appreciated.

I solved the problem by tracking the the website with chrome. I realized that the login page was giving me 2 cookies when i first went to the page and 1 cookie when i logged in. The scenario can be described like this
Get loginPage - 2 cookies
Post LoginPage - 1 cookie
The problem was that I started out by doing a httpClient.PostAsync for the login. When doing the login i was missing the first 2 cookies.
I solved it by doing a httpClient.GetAsync before the post, I then got the totalt 3 cookies that was needed.
HttpResponseMessage responseGet = await httpClient.GetAsync(User.School.Url);
HttpResponseMessage responsePost = await httpClient.PostAsync(User.School.Url, new FormUrlEncodedContent(values));
The problem was probably caused by my lack of knowledge about cookies. If the website response is saying your are missing cookies then you probably are, be sure you tracked ALL the traffic correctly. Hope this can be helpful to someone else :)

WP8 C# Website Scraping with Login

I'm working on a project of a mobile version of an archaic Online Learning System in my campus. I've been trying for weeks to scrape something in this website, but I need to login first in order to get it. I have search anything including HttpWebRequest, CookiesAwareWebClient, etc
My method until now is:
Find the "action" URL in the login form of the site
Sent POST request to that URL
Receive response containing cookies in the Headers["Set-Cookie"]
Create new HttpWebRequest with the URL to the content(that need to be logged in first).
Copy the headers of set-cookie into that request.
Run it (but fails)
I also have tried using CookieCollection in CookieAwareWebClient but it didn't work too.
How to do it properly? Is the location of a Cookie in HttpWebRequest is only in Headers, or in HTTP Packets, where is the location of CookieCollection? Does CookieCollection included in the next request?
Thanks

You need to use a CookieContainer. That will process and hold the cookies for you between HttpWebRequest objects:
var cookieJar = new CookieContainer();
var loginWebRequest = WebRequest.Create(loginUrl) as HttpWebRequest;
loginWebRequest.CookieContainer = cookieJar;
// Execute the Web Request
var authRequiredWebRequest = WebRequest.Create(protectedUrl) as HttpWebRequest;
authRequiredWebRequest.CookieContainer = cookieJar;
// Execute the next request
// It will have the auth cookie set appropriately

Grab the contents of a Drupal website that is secured with a login form

I would like to grab some content from a website that is made with Drupal.
The challenge here is that i need to login on this site before i can access the page i want to scrape. Is there a way to automate this login process in my C# code, so i can grab the secure content?

To access the secured content, you'll need to store and send cookies with every request to your server, starting with the request that sends your log in info and then saving the session cookie that the server gives you (which is your proof that you are who you say you are).
You can use the System.Windows.Forms.WebBrowser for a less control but out-of-the-box solution that will handle cookies.
My preferred method is to use System.Net.HttpWebRequest to send and receive all web data and then use the HtmlAgilityPack to parse the returned data into a Document Object Model (DOM) which can be easily read from.
The trick to getting System.Net.HttpWebRequest to work is that you must create a long-lived System.Net.CookieContainer that will keep track of your log in info (and other things the server expects you to keep track of). The good news is that the HttpWebRequest will take care of all of this for you if you provide the container.
You need a new HttpWebRequest for each call you make, so you must sets their .CookieContainer to the same object every time. Here is an example:
UNTESTED
using System.Net;
public void TestConnect()
{
CookieContainer cookieJar = new CookieContainer();
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://www.mysite.com/login.htm");
request.CookieContainer = cookieJar;
HttpWebResponse response = (HttpWebResponse) request.GetResponse();
// do page parsing and request setting here
request = (HttpWebRequest)WebRequest.Create("http://www.mysite.com/submit_login.htm");
// add specific page parameters here
request.CookeContainer = cookieJar;
response = (HttpWebResponse) request.GetResponse();
request = (HttpWebRequest)WebRequest.Create("http://www.mysite.com/secured_page.htm");
request.CookeContainer = cookieJar;
// this will now work since you have saved your authentication cookies in 'cookieJar'
response = (HttpWebResponse) request.GetResponse();
}
http://msdn.microsoft.com/en-us/library/system.windows.forms.webbrowser.aspx
HttpWebRequest Class
http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.cookiecontainer.aspx

You'll have to use the Services module to do that. Also check out this link for a bit of explanation.

how to get the source code as register user

i downloaded a sourcecode of a site,but i downloaded it i saw it identify my program as a guest,i search at google and figure out that i can send a cookie when i "ask" the source code.
that what i have managed to do and it still dont identify me as register user:
CookieContainer cj = new CookieContainer();
string all = "";
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(Url);
req.CookieContainer = cj;
HttpWebResponse res = (HttpWebResponse)req.GetResponse();
CookieCollection cs=cj.GetCookies(req.RequestUri);
CookieContainer cc = new CookieContainer();
cc.Add(cs);
req.CookieContainer = cc;
StreamReader read = new StreamReader(res.GetResponseStream());
all = read.ReadToEnd();
read.Close();
return all;
what is wrong here?
tyvm for help:)
(if that help,i can have a simple details of a register user of the site).

You would have to use the cookie that the server left behind in your cookie cache that identified you as a authenticated user in a previous session. You'll need to use the Cookie(name, value) constructor. Getting the value is the tricky part, look through your cookie cache to see if you can find one. It is still going to fail if the server expires the cookie.
Using a tool that lets you look at the HTTP headers and cookie values is important to debug this. Firebug is very nice.

HttpRequest: pass through AuthLogin

I would need to make a simple program that logs with given credentials to certain website and then navigate to some element (link).
It is even possible (I mean this Authlogin thing)?
EDIT: SORRY - I am on my company machine and I cannot click on "Vote" or "Add comment" - the page says "Done, but with errors on page" (IE..). I do appreciate your answers and comments, you have helped me a lot!

Main things to do are:
Start using Fiddler to see what needs to be sent and in what way
Assuming we're talking a normal web form you'll probably need to use a CookieContainer with your WebRequests in order to accept the cookies that come from the login request and then re-supply them when sending subsequent requests (such context is not automagically maintained by HttpWebRequest) :-
CookieContainer _cookieContainer = new CookieContainer();
((HttpWebRequest)request).CookieContainer = _cookieContainer;

yes. it is possible.
see following code:
HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(uri);
req.AuthenticationLevel = System.Net.Security.AuthenticationLevel.MutualAuthRequested;
req.Credentials = new NetworkCredential("admin", "admin");
req.PreAuthenticate = true;

It will partly depend on how the login process is managed. Is this actually done via a web form? If so, you'll need to post the form, just as a normal browser would.
If it's done over HTTP authentication, you should be able to set the credentials in the web request, tell it to pre-authenticate, and all should be well.