Develop Reference

How to place HttpWebRequest Host header first?

I'm using standard HttpWebRequest to get html page:
using System;
using System.IO;
using System.Net;
namespace TestConsole.Classes {
class RequestHeadersOrder {
public void Test() {
var req = (HttpWebRequest)WebRequest.Create("https://www.google.com");
req.Proxy = new WebProxy("localhost", 8888); // for debug in Fiddler proxy (https://www.telerik.com/fiddler)
req.Host = "www.google.com";
req.UserAgent = "Robot-tester";
req.Accept = "*/*";
req.Headers.Add(HttpRequestHeader.AcceptLanguage, "en-US");
req.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip, deflate");
req.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;
string html;
using (var resp = req.GetResponse())
using (var respStream = resp.GetResponseStream())
using (var ms = new MemoryStream()) {
respStream.CopyTo(ms);
html = System.Text.Encoding.UTF8.GetString(ms.ToArray());
}
Console.WriteLine(html);
}
}
}
This produces request headers (raw format):
GET https://www.google.com/ HTTP/1.1
User-Agent: Robot-tester
Accept: */*
Accept-Language: en-US
Accept-Encoding: gzip, deflate
Host: www.google.com
Connection: Keep-Alive
But most browsers use different headers order: "Host" header first, then "User agent", then other headers. So, I need this:
GET https://www.google.com/ HTTP/1.1
Host: www.google.com
User-Agent: Robot-tester
Accept: */*
Accept-Language: en-US
Accept-Encoding: gzip, deflate
Connection: Keep-Alive
Please advice me how to place "Host" header first. This is important for some web sites.

Log in to site programmatically and redirect browser to signed in state

I want to sign in to a site when a link is clicked and then redirect the browser there with a signed in session. Im having some troubles and here is what Ive tried:
First I get the session cookies from the login site:
CookieContainer cookies= new CookieContainer();
HttpWebRequest myHttpWebRequest = (HttpWebRequest)WebRequest.Create("http://someuri.com");
myHttpWebRequest.CookieContainer = cookies;
HttpWebResponse myHttpWebResponse = (HttpWebResponse)myHttpWebRequest.GetResponse();
myHttpWebResponse.Close();
Then I post to the sign in page to get signed in:
HttpWebRequest getRequest = (HttpWebRequest)WebRequest.Create("http://signInURL.com");
getRequest.CookieContainer = cookies;
getRequest.Method = WebRequestMethods.Http.Post;
getRequest.UserAgent = "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.121 Safari/535.2";
getRequest.AllowWriteStreamBuffering = true;
getRequest.ProtocolVersion = HttpVersion.Version11;
getRequest.AllowAutoRedirect = true;
getRequest.ContentType = "application/x-www-form-urlencoded";
byte[] byteArray = Encoding.ASCII.GetBytes(PostParameterStringWithSignInInfo);
getRequest.ContentLength = byteArray.Length;
Stream newStream = getRequest.GetRequestStream();
newStream.Write(byteArray, 0, byteArray.Length);
newStream.Close();
HttpWebResponse getResponse = (HttpWebResponse)getRequest.GetResponse();
Then I figured I need to set the cookies to the client:
CookieCollection cooki = getRequest.CookieContainer.GetCookies(new Uri("http://someUri.com"));
for(int i = 0; i < cooki.Count; i++)
{
Cookie c = cooki[i];
Response.Cookies.Add(new HttpCookie(c.Name, c.Value));
}
And then redirect to where you end up being signed in:
Response.Redirect("http://URLwhenBeingSignedIn.com");
This doesnt work. When redirected Im still logged out.
Tried to do this with Fiddler and succeeded to sign in and get redirected:
Get the session cookies:
GET / HTTP/1.1
Content-type: application/x-www-form-urlencoded
Host: someuri.com
Post to the sign in page to get signed in:
POST /signIn HTTP/1.1
Accept: text/html, application/xhtml+xml, */*
Referer: http://someuri.com
Accept-Language: en-GB,en;q=0.7,tr;q=0.3
User-Agent: Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0)
Content-Type: application/x-www-form-urlencoded
Accept-Encoding: gzip, deflate
Connection: Keep-Alive
Content-Length: 90
DNT: 1
Host: signInURL.com
Pragma: no-cache
Cookie: JSESSIONID=fromBefore; Cookie2=fromBefore
PostParameterStringWithSignInInfo
Perhaps there's an easier way than the one I thought of now that you can see the fiddler requests that works, if so I'm happy to see it.

C# HTTP POST with Boundary

I need a little help setting up a HTTP Post in C#. I appreciate any assistance I receive in advance.
Using Fiddler here is my RAW POST:
POST http://www.domain.com/tester.aspx HTTP/1.1
User-Agent: Tegan
Content-Type: multipart/form-data; boundary=myboundary
Host: www.domain.com
Content-Length: 1538
Expect: 100-continue
<some-xml>
<customer>
<user-id>george</user-id>
<first-name>George</first-name>
<last-name>Jones</last-name>
</customer>
</some-xml>
My requirements are a little tricky. They require a multi-part post with a boundary. I'm not familiar with setting up a boundary. If any one can assist I would appreciate it.
Here are my requirements:
POST http://www.domain.com/tester.aspx HTTP/1.0(CRLF)
User-Agent: myprogramname(CRLF)
Content-Type: multipart/form-data; boundary=myboundary(CRLF)
Content-Length: nnn(CRLF)
(CRLF)
(CRLF)
--myboundary(CRLF)
Content-Disposition: form-data; name=”xmlrequest”(CRLF)
Content-Type: text/xml(CRLF)
(CRLF)
(XML request message)(CRLF)
(CRLF)
--myboundary--(CRLF)
So I think this is what the POST should look like but I need some help with my C#.
POST http://www.domain.com/tester.aspx HTTP/1.1
User-Agent: Tegan
Content-Type: multipart/form-data; boundary=myboundary
Content-Length: 1538
--myboundary
Content-Disposition: form-data; name="xmlrequest"
Content-Type: text/xml
<some-xml>
<customer>
<user-id>george</user-id>
<first-name>George</first-name>
<last-name>Jones</last-name>
</customer>
</some-xml>
(CRLF)
--myboundary--
Here is the C# code I'm using to create the WebRequest.
HttpWebRequest request = null;
Uri uri = new Uri("http://domain.com/tester.aspx");
request = (HttpWebRequest)WebRequest.Create(uri);
request.Method = "POST";
request.UserAgent = "NPPD";
request.ContentType = "multipart/form-data; boundary=myboundary";
request.ContentLength = postData.Length;
using (Stream writeStream = request.GetRequestStream())
{
writeStream.Write(postData, 0, postData.Length);
}
string result = string.Empty;
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
using (Stream responseStream = response.GetResponseStream())
{
using (StreamReader readStream = new StreamReader(responseStream, Encoding.UTF8))
{
result = readStream.ReadToEnd();
}
}
}
return result;

I blogged about this and presented a sample method that could be used to send multipart/form-data requests. Checkout here: http://www.bratched.com/en/home/dotnet/69-uploading-multiple-files-with-c.html

Problems with POST request parameters in C#

This website uses POST to send data whenever the user clicks on a calendar to change the date. I used Firebug to inspect it. The target URL is this. The post data (space-separated) for a particular example is LeagueID=9 GameDate=4-29-2011 Season=2010-2011 Refresh=false LastUpdateTime=01-01-1900 type=Matchups RefreshStartTime=15-5-2011-1308094688683 Week= conferenceID=.
And here are the headers:
Host scores.covers.com
User-Agent Mozilla/5.0 (Windows NT 6.1; rv:2.0.1) Gecko/20100101 Firefox/4.0.1
Accept text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language en-gb,en;q=0.5
Accept-Encoding gzip, deflate
Accept-Charset ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive 115
Connection keep-alive
Referer http://scores.covers.com/basketball-scores-matchups.aspx
Content-Length 169
Content-Type text/plain; charset=UTF-8
Cookie __loooooooooongCookieString
I'd like to make that POST request using WebRequest (or whetever else does the trick). Here's my attempt:
string parameters = "LeagueID=\"9\"&GameDate=\"4-29-2011\"&Season=\"2010-2011\"&Refresh=\"false\"&LastUpdateTime=\"01-01-1900\"&type=\"Matchups\"&RefreshStartTime=\"15-5-2011-1308094688683\"&Week=&conferenceID=";
byte[] bytes = Encoding.ASCII.GetBytes(parameters);
HttpWebRequest req = (HttpWebRequest)WebRequest.Create("http://scores.covers.com/ajax/SportsDirect.Controls.LiveScoresControls.Scoreboard,SportsDirect.Controls.LiveScoresControls.ashx?_method=UpdateScoreboard&_session=no");
req.Method = "POST";
req.ContentLength = bytes.Length;
req.ContentType = "text/plain; charset=UTF-8";
Console.WriteLine(req.ContentLength); // 175
Stream reqStream = req.GetRequestStream();
reqStream.Write(bytes, 0, bytes.Length);
reqStream.Close();
WebResponse resp = req.GetResponse();
Console.WriteLine(((HttpWebResponse)resp).StatusDescription); // OK
Stream respStream = resp.GetResponseStream();
StreamReader reader = new StreamReader(respStream);
Console.WriteLine(reader.ReadToEnd());
resp.Close();
But it doesn't work. The response code is OK, but the response itself is this:
new Object();r.error = new ajax_error('System.FormatException','Input string was
not in a correct format.\r\nCould not retreive parameters from HTTP request.',0
)new Object();r.error = new ajax_error('System.ArgumentException','Object of typ
e \'System.DBNull\' cannot be converted to type \'System.Int32\'.',0)
What's the deal? I can see that something's wrong with the params since the content length of the request is 175 (as opposed to the 169 from the request made by Firefox).

Why not use NameValueCollection to POST your parameters using a WebClient? It does the tricky stuff for you. The code at the bottom of the linked page is about as simple as it comes. Unlike the sample, you should probably deal thoughtfully with disposal of the WebClient.

Don't ASCII encode when you specify UTF-8 later. Make sure to url encode parameters. Try changing the content-type to 'x-www-form-urlencoded'.

HttpWebRequest POST gives 500 server error

I need to make from my app an authentificated httpwebrequest. the response to my request should be in json format. for this i'm using the code below:
// Create the web request
Uri address = new Uri("http://www.mysite.com/remote/user/login/format/json");
HttpWebRequest request = WebRequest.Create(address) as HttpWebRequest;
request.Method = "POST";
request.UseDefaultCredentials = false;
request.Credentials = new NetworkCredential(UserName, Password);
request.PreAuthenticate = true;
request.ContentType = "application/x-www-form-urlencoded";
request.Accept = "application/json";
string data = string.Format("username={0}&password={1}", otherusername, otherpassword);
// Create a byte array of the data we want to send
byte[] byteData = UTF8Encoding.UTF8.GetBytes(data);
// Set the content length in the request headers
request.ContentLength = byteData.Length;
//Write data
using (Stream postStream = request.GetRequestStream())
{
postStream.Write(byteData, 0, byteData.Length);
}
// Get response
try
{
using (HttpWebResponse response = request.GetResponse() as HttpWebResponse)
{
// Get the response stream
StreamReader reader = new StreamReader(response.GetResponseStream());
// Console application output
jsonResponse = reader.ReadToEnd();
reader.Close();
}
user = new User();
JObject o = JObject.Parse(jsonResponse);
user.Unguessable_id = (string)o["unguessable_id"];
user.Print_id = (string)o["print_id"];
user.Rrid = (string)o["rrid"];
user.Raid = (string)o["raid"];
}
catch (WebException ex) {
errorMessage = ex.Message;
}
the problem is that the very first call it always gives a 500 error on the server. and the request fails. if i redo the call(by making an refresh in my browser) the request is successful.
the request should look like this in normal conditions:
POST /remote/user/login/format/json HTTP/1.1
Host: <yourhost>
username=user&password=pass
but when the server sends out the 500 error he received something like this:
username=user&password=passwordPOST /remote/user/login/format/json HTTP/1.1
any idea why this is happening? in my test app if i refresh the page that makes the httpwebrequest the call is successful.
EDIT:
after installing Fiddler the requests made look like this:
=> the one that generates 500
POST http://www.mysite.com/remote/user/login/format/json HTTP/1.1
Content-Type: application/x-www-form-urlencoded
Accept: application/json
Host: www.mysite.com
Content-Length: 30
Expect: 100-continue
Connection: Keep-Alive
username=user&password=pass
=> the one made on refresh
POST http://www.mysite.com/remote/user/login/format/json HTTP/1.1
Content-Type: application/x-www-form-urlencoded
Accept: application/json
Authorization: Basic ZGNpOkFpR2g3YWVj
Host: www.mysite.com
Content-Length: 30
Expect: 100-continue
Connection: Keep-Alive
username=user&password=pass
it seems that Authorization: Basic ZGNpOkFpR2g3YWVj is not included in the first request...why is that happening?(i'm using the same code for both requests)

I would advice you to install Fiddler to see what's really happening

I needed to add:
request.Headers.Add("Authorization: Basic ZGNpOkFpR2g3YWVj");
weird though that for the second request it was added automatically..