Develop Reference

Authenticating requests to external REST service

I'm really new to web development, and I don't really have a good grip on the main concepts of web. However, I've been tasked with writing an asp.net application where users can search documents by querying an external RESTful web service. Requests to this REST service must be authenticated by HTTP Basic Authentication.
So far so good, I've been able to query the service using HttpWebRequest and HttpWebResponse, adding the encoded user:pass to the request's authorization header, deserialize the Json response and produce a list of strings with url's to the pdf documents resulting from the search.
So now I'm programmatically adding HyperLink elements to the page with these urls:
foreach (string url in urls) {
HyperLink link = new HyperLink();
link.Text = url;
link.NavigateUrl = url;
Page.Controls.Add(link);
}
The problem is that requests to these documents has to be authorized with the same basic http authentication and the same user:pass as when querying the REST service, and since I'm just creating links for the user to click, and not creating any HttpWebRequest objects, I don't know how to authenticate such a request resulting from a user clicking a link.
Any pointers to how I can accomplish this is very much appreciated. Thanks in advance!

You probably want to do the request server-side, as I think you're already doing, and then show the results embedded in your own pages, or just stream the result directly back to the users.
It's a bit unclear what it is you need (what are the links, what do you show the users, etc.), so this is the best suggesting I can do based on the info you give.
Update:
I would create a HttpHandler (an .ashx file in an ASP.NET project), and link to that, with arguments so you can make the request to the REST service and get the correct file, then stream the data directly back to the visitor. Here's a simple example:
public class DocumentHandler : IHttpHandler {
public Boolean IsReusable {
get { return true; }
}
public void ProcessRequest(HttpContext context) {
// TODO: Get URL of the document somehow for the REST request
// context.Request
// TODO: Make request to REST service
// Some pseudo-code for you:
context.Response.ContentType = "application/pdf";
Byte[] buffer = new WebClient().DownloadData(url);
context.Response.OutputStream.Write(buffer, 0, buffer.Length);
context.Response.End();
}
}
I hope you can fill in the blanks yourself.

Cookie with value separated by comma

I have a problem with getting and sending this cookie header. To be more specific - with that MenuData cookie.
I'm sending GET request and with response i get this one particular cookie which I need to send back in next POST request.
The problem is that i get it in two parts(in cookiecontainer) and i have no idea how to Encode/Decode it to send it properly.
Here's the cookie header of POST i'm trying to send.
(//// - parts i need to put together)
Cookie:
//// MenuData={'Type':null;
ASP.NET_SessionId=oe5qzthlb51ri5nzxddadzzo;
.LoginISerwis=2880E262ECC48BD7D12443EDC97D9641E85401A345B629C2002AC89F22CEBD201700417EB0D499C6E8F10816AC1F457FF7CBD671C83509CEF405236C91D6CDD81543BF1EC507319EDD587E6FFDEBA80DFAD30D769DF6F70C942ABBCB383A0C0A0BF127F40FB4C04F25A6F68469EFAF51503EF10DCFF2F51A9B31040575B14962;
CustomerLogin=ID=xxxx&Login=xxxxx&RememberLogin=True;
//// 'Id':null}=
I need an advices how I can handle this kind of cookies.

Unfortunately, CookieContainer is known to have problems with parsing cookies with commas in them. My advice is to manually read the cookie header at this particular step and pass the cookie along to your next request.

Grab the contents of a Drupal website that is secured with a login form

I would like to grab some content from a website that is made with Drupal.
The challenge here is that i need to login on this site before i can access the page i want to scrape. Is there a way to automate this login process in my C# code, so i can grab the secure content?

To access the secured content, you'll need to store and send cookies with every request to your server, starting with the request that sends your log in info and then saving the session cookie that the server gives you (which is your proof that you are who you say you are).
You can use the System.Windows.Forms.WebBrowser for a less control but out-of-the-box solution that will handle cookies.
My preferred method is to use System.Net.HttpWebRequest to send and receive all web data and then use the HtmlAgilityPack to parse the returned data into a Document Object Model (DOM) which can be easily read from.
The trick to getting System.Net.HttpWebRequest to work is that you must create a long-lived System.Net.CookieContainer that will keep track of your log in info (and other things the server expects you to keep track of). The good news is that the HttpWebRequest will take care of all of this for you if you provide the container.
You need a new HttpWebRequest for each call you make, so you must sets their .CookieContainer to the same object every time. Here is an example:
UNTESTED
using System.Net;
public void TestConnect()
{
CookieContainer cookieJar = new CookieContainer();
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://www.mysite.com/login.htm");
request.CookieContainer = cookieJar;
HttpWebResponse response = (HttpWebResponse) request.GetResponse();
// do page parsing and request setting here
request = (HttpWebRequest)WebRequest.Create("http://www.mysite.com/submit_login.htm");
// add specific page parameters here
request.CookeContainer = cookieJar;
response = (HttpWebResponse) request.GetResponse();
request = (HttpWebRequest)WebRequest.Create("http://www.mysite.com/secured_page.htm");
request.CookeContainer = cookieJar;
// this will now work since you have saved your authentication cookies in 'cookieJar'
response = (HttpWebResponse) request.GetResponse();
}
http://msdn.microsoft.com/en-us/library/system.windows.forms.webbrowser.aspx
HttpWebRequest Class
http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.cookiecontainer.aspx

You'll have to use the Services module to do that. Also check out this link for a bit of explanation.

Keeping session id when getting source code from web page

I'm making a C# windows form application that needs to parse data from an external site that requires me to log in.
First I send the POST data on the login page using a WebRequest. This works correctly as I can see the page source of a page that requires me to log in.
When I try to view the page source code of the data-page however, I'm logged out again as the session gets lost.
Is there any way of storing the PHPSESSID and re-using it in another request?
EDIT:
WebRequest _theRequest;
...
_theRequest = (HttpWebRequest)WebRequest.Create(url);
_theRequest.CookieContainer = new CookieContainer();
The property CookieContainer is not showing up in the intellisense list, anything I'm doing wrong here?

You could assign a CookieContainer and reuse the request object for subsequent requests.

HTTP Post in C# console app doesn't return the same thing as a browser request

I have a C# console app (.NET 2.0 framework) that does an HTTP post using the following code:
StringBuilder postData = new StringBuilder(100);
postData.Append("post.php?");
postData.Append("Key1=");
postData.Append(val1);
postData.Append("&Key2=");
postData.Append(val2);
byte[] dataArray = Encoding.UTF8.GetBytes(postData.ToString());
HttpWebRequest httpRequest = (HttpWebRequest)WebRequest.Create("http://example.com/");
httpRequest.Method = "POST";
httpRequest.ContentType = "application/x-www-form-urlencoded";
httpRequest.ContentLength = dataArray.Length;
Stream requestStream = httpRequest.GetRequestStream();
requestStream.Write(dataArray, 0, dataArray.Length);
requestStream.Flush();
requestStream.Close();
HttpWebResponse webResponse = (HttpWebResponse)httpRequest.GetResponse();
if (httpRequest.HaveResponse == true) {
Stream responseStream = webResponse.GetResponseStream();
StreamReader responseReader = new System.IO.StreamReader(responseStream, Encoding.UTF8);
String responseString = responseReader.ReadToEnd();
}
The outputs from this are:
webResponse.ContentLength = -1
webResponse.ContentType = text/html
webResponse.ContentEncoding is blank
The responseString is HTML with a title and body.
However, if I post the same URL into a browser (http://example.com/post.php?Key1=some_value&Key2=some_other_value), I get a small XML snippet like:
<?xml version="1.0" ?>
<RESPONSE RESULT="SUCCESS"/>
with none of the same HTML as in the application. Why are the responses so different? I need to parse the returned result which I am not getting in the HTML. Do I have to change how I do the post in the application? I don't have control over the server side code that accepts the post.

If you are indeed supposed to use the POST HTTP method, you have a couple things wrong. First, this line:
postData.Append("post.php?");
is incorrect. You want to post to post.php, you don't want post the value "post.php?" to the page. Just remove this line entirely.
This piece:
... WebRequest.Create("http://example.com/");
needs post.php added to it, so...
... WebRequest.Create("http://example.com/post.php");
Again this is assuming you are actually supposed to be POSTing to the specified page instead of GETing. If you are supposed to be using GET, then the other answers already supplied apply.

You'll want to get an HTTP sniffer tool like Fiddler and compare the headers that are being sent from your app to the ones being sent by the browser. There will be something different that is causing the server to return a different response. When you tweak your app to send the same thing browser is sending you should get the same response. (It could be user-agent, cookies, anything, but something is surely different.)

I've seen this in the past.
When you run from a browser, the "User-Agent" in the header is "Mozilla ...".
When you run from a program, it's different and generally specific to the language used.

I think you need to use a GET request, instead of POST. If the url you're using has querystring values (like ?Key1=some_value&Key2=some_other_value) then it's expecting a GET. Instead of adding post values to your webrequest, just put this data in the querystring.
HttpWebRequest httpRequest = (HttpWebRequest)WebRequest.Create("http://example.com/?val1=" + val1 + "&val2=" + val2);
httpRequest.Method = "GET";
httpRequest.ContentType = "application/x-www-form-urlencoded";
....
So, the result you're getting is different when you POST the data from your app because the server-side code has a different output when it can't read the data it's expecting in the querystring.

In your code you a specify the POST method which sends the data to the PHP file without putting the data in the web address. When you put the information in the address bar, that is not the POST method, that is the GET method. The name may be confusing, but GET just means that the data is being sent to the PHP file through the web address, instead of behind the scenes, not that it is supposed to get any information. When you put the address in the browser it is using a GET.
Create a simple html form and specify POST as the method and your url as the action. You will see that the information is sent without appearing in the address bar.
Then do the same thing but specify GET. You will see the information you sent in the address bar.

I believe the problem has something to do with the way your headers are set up for the WebRequest.
I have seen strange cases where attempting to simulate a browser by changing headers in the request makes a difference to the server.
The short answer is that your console application is not a web browser and the web server of example.com is expecting to interact with a browser.
You might also consider changing the ContentType to be "multipart/form-data".
What I find odd is that you are essentially posting nothing. The work is being done by the query string. Therefore, you probably should be using a GET instead of a POST.

Is the form expecting a cookie? That is another possible reason why it works in the browser and not from the console app.