In spite of the RFC stating that the order of uniquely-named headers shouldn't matter, the website I'm sending this request to does implement a check on the order of headers.
This works:
GET https://www.thewebsite.com HTTP/1.1
Host: www.thewebsite.com
Connection: keep-alive
Accept: */*
User-Agent: Mozilla/5.0 etc
This doesn't work:
GET https://www.thewebsite.com HTTP/1.1
Accept: */*
User-Agent: Mozilla/5.0 etc
Host: www.thewebsite.com
Connection: keep-alive
The default HttpWebRequest seems to put the Host and Connection headers at the end, before the blank line, rather than just after the url.
Is there any way (using a fork of HttpWebRequest or some other library in Nuget even) to specify the order of headers in a HttpWebRequest?
If possible, I'd rather not start going down the route of implementing a proxy to sort them or having to code the whole thing up using a TcpClient.
I'd appreciate any hints at all on this.
Update: With Fiddler running, header order in HttpWebrequest can be re-shuffled in CustomRules.cs. Still no closer to a solution without a proxy though.
Some server implement header ordering as a precaution for any attacks or spam, an article explaining Why ordering HTTP headers is important.
But the standard is, the order in which header fields with differing field names are received is not significant.
HttpWebRequest, there is no easy way to order the headers and the Connection and Host is added internally.
If ordering is really important, use the HttpClient instead, it can easily arrange the Headers based on the example of #Jason.
If you will be using HttpClient, you can create a custom HttpClientHandler and you can arrange your header from there. It can be something like this.
HANDLER
public class CustomHttpClientHandler : HttpClientHandler
{
protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
{
request.Headers.Clear();
request.Headers.Add("Host", $"{request.RequestUri.Authority}");
request.Headers.Add("Connection", "keep-alive");
request.Headers.Add("Accept", "*/*");
request.Headers.Add("User-Agent", "Mozilla/5.0 etc");
return await base.SendAsync(request, cancellationToken);
}
}
IMPLEMENTATION
HttpClient clientRequest = new HttpClient(new CustomHttpClientHandler());
await clientRequest.GetAsync(url);
.Net Core
If you set the headers yourself, you can specify the order. When the common headers are added it will find the existing headers instead of appending them:
using System.Net;
namespace ConsoleApp2
{
class Program
{
static void Main(string[] args)
{
var request = WebRequest.Create("http://www.google.com");
request.Headers.Add("Host", "www.google.com");
// this will be set within GetResponse.
request.Headers.Add("Connection", "");
request.Headers.Add("Accept", "*/*");
request.Headers.Add("User-Agent", "Mozilla/5.0 etc");
request.GetResponse();
}
}
}
Here is an example with HttpClient:
using System.Net.Http;
using System.Threading.Tasks;
namespace ConsoleApp3
{
class Program
{
static async Task Main(string[] args)
{
var client = new HttpClient();
client.DefaultRequestHeaders.Add("Host", "www.google.com");
client.DefaultRequestHeaders.Add("Connection", "keep-alive");
client.DefaultRequestHeaders.Add("Accept", "*/*");
client.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 etc");
await client.GetAsync("http://www.google.com");
await client.PostAsync("http://www.google.com", new StringContent(""));
}
}
}
Edit
The above code did not work on .Net Framework only .Net Core
.Net Framework
On .Net Framework the headers are reserved so they cannot be set like this, see Cannot set some HTTP headers when using System.Net.WebRequest.
One work around is to use reflection to modify the behavior of the framework class, but be warned this could break if the libraries are updated so it's not recommended!.
Essentially, HttpWebRequest calls ToString on WebHeaderCollection to serialize.
See https://referencesource.microsoft.com/#System/net/System/Net/HttpWebRequest.cs,5079
So a custom class can be made to override ToString. Unfortunately reflection is needed to set the headers as WebRequest copies the collection on assignment to Headers, instead of taking the new reference.
WARNING, THE FOLLOWING CODE CAN BREAK IF FRAMEWORK CHANGES
If you use this, write some unit tests that verify the behavior still stays consistent after updates to .NET Framework
using System.Collections.Generic;
using System.Linq;
using System.Net;
using System.Reflection;
namespace ConsoleApp2
{
class Program
{
static void Main(string[] args)
{
// WARNING, CODE CAN BREAK IF FRAMEWORK CHANGES
// If you use this, write some unit tests that verify the behavior still stays consistent after updates to .NET Framework
var request = (HttpWebRequest)WebRequest.Create("http://www.google.com");
var field = typeof(HttpWebRequest).GetField("_HttpRequestHeaders", BindingFlags.Instance | BindingFlags.NonPublic);
var headers = new CustomWebHeaderCollection(new Dictionary<string, string>
{
["Host"] = "www.google.com",
["Connection"] = "keep-alive",
["Accept"] = "*/*",
["User-Agent"] = "Mozilla/5.0 etc"
});
field.SetValue(request, headers);
request.GetResponse();
}
}
internal class CustomWebHeaderCollection : WebHeaderCollection
{
private readonly Dictionary<string, string> _customHeaders;
public CustomWebHeaderCollection(Dictionary<string, string> customHeaders)
{
_customHeaders = customHeaders;
}
public override string ToString()
{
// Could call base.ToString() split on Newline and sort as needed
var lines = _customHeaders
.Select(kvp => $"{kvp.Key}: {kvp.Value}")
// These two new lines are needed after the HTTP header
.Concat(new [] { string.Empty, string.Empty });
var headers = string.Join("\r\n", lines);
return headers;
}
}
}
Related
I have the following code to set content-md5 in my GET method request using HttpClient
httpClient.DefaultRequestHeaders.TryAddWithoutValidation("content-md5", "value");
I cannot use HttpRequestMessage content to set it because it's not a POST method. When using Postman it works like a charm but fails when using HttpClient.GetAsync.
Client request a hmac to the server as follows
{
"content_to_hash": "my content"
}
The server will give response like this
{
"content_md5": "88af7ceab9fdafb76xxxxx",
"date": "Sat, 02 May 2020 00:13:16 +0700",
"hmac_value": "WfHgFyT792IENmK8Mqz9LysmP8ftOP00qA="
}
Now I have to access a GET request using that hmac where it's the problem because I cannot set in httpClient GET request header.
Here's the image
From reading the HttpClient and related source code, there's no way you can get around this and add the header to the actual request object headers. There is an internal list of invalid headers, which includes any Content-* headers. It has to be on a content object.
Therefore, my suggest solution is to create your own content object:
public class NoContentMd5 : HttpContent
{
protected override Task SerializeToStreamAsync(Stream stream, TransportContext context)
{
return Task.CompletedTask;
}
protected override bool TryComputeLength(out long length)
{
length = 0;
return false;
}
public NoContentMd5(byte[] contentMd5)
{
this.Headers.ContentMD5 = contentMd5;
}
public NoContentMd5(string contentMd5)
{
this.Headers.TryAddWithoutValidation("Content-MD5", contentMd5);
}
}
This will add the Content-MD5 header with a value of your choosing, but the request won't contain a body.
The next problem you'll encounter is that you're trying to make a GET request with content, which isn't supported by the helper client.GetAsync(...) method. You'll have to make your own request object and use client.SendAsync(...) instead:
HttpClient client = new HttpClient();
var request = new HttpRequestMessage(HttpMethod.Get, "https://localhost/my/test/uri");
request.Content = new NoContentMd5("d41d8cd98f00b204e9800998ecf8427e ");
var result = await client.SendAsync(request);
Note that if you have your Content-MD5 hash as bytes, I've also added a constructor to NoContentMd5 for byte[] too.
The only potential issue with this is that it includes a Content-Length: 0 header. Hopefully that's OK with the API you're working with.
There's an alternative solution described in this answer to question with a similar issue. I'd argue against using it since is vulnerable to changes in the implementation details of HttpRequestHeaders (because it uses reflection, so if MS change the code, it might break) .
Aside from the fact that it's not considered good practice to send a body with GET request (see HTTP GET with request body), you can try this:
using (var content = new StringContent(string.Empty))
using (var request = new HttpRequestMessage
{
Method = HttpMethod.Get,
RequestUri = new Uri("http://localhost"),
Content = content
})
{
request.Headers.TryAddWithoutValidation("content-md5", "value");;
using (var response = await httpClient.SendAsync(request))
{
response.EnsureSuccessStatusCode();
}
}
UPDATE:
The proper way would be to set the ContentMD5 property on the HttpContentHeaders, for example:
content.Headers.ContentMD5 = Convert.FromBase64String(hashString);
But as you pointed out in the comments, trying to send content in a GET request causes an error.
I am developing an C# console application for testing whether a URL is valid or not. It works well for most of URLs. But we found that there are some cases the application always got 404 response from target site but the URLs actually work in the browser. And those URLs also works when I tried them in the tools such as DHC (Dev HTTP Client).
In the beginning, I though that this could be the reason of not adding right headers. But after tried using Fiddler to compose a http request with same headers, it works in Fiddler.
So what's wrong with my code? Is there any bug in .NET HttpClient?
Here are the simplified code of my test application:
class Program
{
static void Main(string[] args)
{
var urlTester = new UrlTester("http://www.hffa.it/short-master-programs/fashion-photography");
Console.WriteLine("Test is started");
Task.WhenAll(urlTester.RunTestAsync());
Console.WriteLine("Test is stoped");
Console.ReadKey();
}
public class UrlTester
{
private HttpClient _httpClient;
private string _url;
public UrlTester(string url)
{
_httpClient = new HttpClient
{
Timeout = TimeSpan.FromMinutes(1)
};
// Add headers
_httpClient.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36");
_httpClient.DefaultRequestHeaders.Add("Accept-Encoding", "gzip,deflate,sdch");
_httpClient.DefaultRequestHeaders.Add("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
_httpClient.DefaultRequestHeaders.Add("Accept-Language", "sv-SE,sv;q=0.8,en-US;q=0.6,en;q=0.4");
_url = url;
}
public async Task RunTestAsync()
{
var httpRequestMsg = new HttpRequestMessage(HttpMethod.Get, _url);
try
{
using (var response = await _httpClient.SendAsync(httpRequestMsg, HttpCompletionOption.ResponseHeadersRead))
{
Console.WriteLine("Response: {0}", response.StatusCode);
}
}
catch (HttpRequestException e)
{
Console.WriteLine(e.InnerException.Message);
}
}
}
}
This appears to be an issue with the accepted languages. I got a 200 response when using the following Accept-Language header value
_httpClient.DefaultRequestHeaders.Add("Accept-Language", "en-GB,en-US;q=0.8,en;q=0.6,ru;q=0.4");
p.s. I assume you know in your example _client should read _httpClient in the urlTester constructor or it wont build.
Another possible cause of this problem is if the url you are sending is over approx 2048 bytes long. At that point the content (almost certainly the query string) can become truncated and this in turn means that it may not be matched correctly with a server side route.
Although these urls were processed correctly in the browser, they also failed using the get command in power shell.
This issue was resolved by using a POST with key value pairs instead of using a GET with a long query string.
Can you tell me how to create a SOAP HTTP parameter in C#. I am not talking about the SOAP HEADER within the SOAP envelope, but the HTTP header. For example I have the Username and Password in the Header below:
POST /company/addThing HTTP/1.1
Host: webservices.foo
Connection: Keep-Alive
User-Agent: PHP-SOAP/5.6.2
Content-Type: text/xml; charset=utf-8
SOAPAction: ""
Content-Length: 295
Username: userx
Password: passwordy
I can add the parameters in PHP using the stream_context parameter of the SoapClient call, but I cannot find where to do it in .Net C#.
I have added the web Reference WSDL which has auto created the objects and methods for the web service and I'm comfortable with that part.
Any help would be appreciated.
Edit
I've ticked the answer below, but it only works for .Net 4.0 and above. One thing I failed to mention was that I am using Visual Studio 2008.
The final solution I used was to add the following code to the auto generated Reference.cs file:
protected override System.Net.WebRequest GetWebRequest(Uri uri)
{
System.Net.WebRequest request = base.GetWebRequest(uri);
request.Headers.Add("Username", "user");
request.Headers.Add("Password", "pass");
return request;
}
When you add a web reference from the WSDL it will generate a class for you that you use to make the calls to your service, let's call it "MyService". If you create a partial for that class, and include it in the same assembly, you can override the "GetWebRequest" method and directly add headers. Here's an example:
public partial class MyService
{
private ConcurrentDictionary<string, string> requestHeaders = new ConcurrentDictionary<string, string>();
public void SetRequestHeader(string headerName, string headerValue)
{
this.requestHeaders.AddOrUpdate(headerName, headerValue, (key, oldValue) => headerValue);
}
protected override WebRequest GetWebRequest(Uri uri)
{
var request = base.GetWebRequest(uri);
var httpRequest = request as HttpWebRequest;
if (httpRequest != null)
{
foreach (string headerName in this.requestHeaders.Keys)
{
httpRequest.Headers[headerName] = this.requestHeaders[headerName];
}
}
return request;
}
}
Scope:
I am developing a C# aplication to simulate queries into this site. I am quite familiar with simulating web requests for achieving the same human steps, but using code instead.
If you want to try yourself, just type this number into the CNPJ box:
08775724000119 and write the captcha and click on Confirmar
I've dealed with the captcha already, so it's not a problem anymore.
Problem:
As soon as i execute the POST request for a "CNPJ", a exception is thrown:
The remote server returned an error: (403) Forbidden.
Fiddler Debugger Output:
Link for Fiddler Download
This is the request generated by my browser, not by my code
POST https://www.sefaz.rr.gov.br/sintegra/servlet/hwsintco HTTP/1.1
Host: www.sefaz.rr.gov.br
Connection: keep-alive
Content-Length: 208
Cache-Control: max-age=0
Origin: https://www.sefaz.rr.gov.br
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.97 Safari/537.11
Content-Type: application/x-www-form-urlencoded
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Referer: https://www.sefaz.rr.gov.br/sintegra/servlet/hwsintco
Accept-Encoding: gzip,deflate,sdch
Accept-Language: pt-BR,pt;q=0.8,en-US;q=0.6,en;q=0.4
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
Cookie: GX_SESSION_ID=gGUYxyut5XRAijm0Fx9ou7WnXbVGuUYoYTIKtnDydVM%3D; JSESSIONID=OVuuMFCgQv9k2b3fGyHjSZ9a.undefined
// PostData :
_EventName=E%27CONFIRMAR%27.&_EventGridId=&_EventRowId=&_MSG=&_CONINSEST=&_CONINSESTG=08775724000119&cfield=rice&_VALIDATIONRESULT=1&BUTTON1=Confirmar&sCallerURL=http%3A%2F%2Fwww.sintegra.gov.br%2Fnew_bv.html
Code samples and References used:
I'm using a self developed library to handle/wrap the Post and Get requests.
The request object has the same parameters (Host,Origin, Referer, Cookies..) as the one issued by the browser (logged my fiddler up here).
I've also managed to set the ServicePointValidator of certificates by using:
ServicePointManager.ServerCertificateValidationCallback =
new RemoteCertificateValidationCallback (delegate { return true; });
After all that configuration, i stil getting the forbidden exception.
Here is how i simulate the request and the exception is thrown
try
{
this.Referer = Consts.REFERER;
// PARAMETERS: URL, POST DATA, ThrownException (bool)
response = Post (Consts.QUERYURL, postData, true);
}
catch (Exception ex)
{
string s = ex.Message;
}
Thanks in advance for any help / solution to my problem
Update 1:
I was missing the request for the homepage, which generates cookies (Thanks #W0lf for pointing me that out)
Now there's another weird thing. Fiddler is not showing my Cookies on the request, but here they are :
I made a successful request using the browser and recorded it in Fiddler.
The only things that differ from your request are:
my browser sent no value for the sCallerURL parameter (I have sCallerURL= instead of sCallerURL=http%3A%2F%2Fwww....)
the session ids are different (obviously)
I have other Accept-Language: values (I'm pretty sure this is not important)
the Content-Length is different (obviously)
Update
OK, I thought the Fiddler trace was from your application. In case you are not setting cookies on your request, do this:
before posting data, do a GET request to https://www.sefaz.rr.gov.br/sintegra/servlet/hwsintco. If you examine the response, you'll notice the website sends two session cookies.
when you do the POST request, make sure to attach the cookies you got at the previous step
If you don't know how to store the cookies and use them in the other request, take a look here.
Update 2
The problems
OK, I managed to reproduce the 403, figured out what caused it, and found a fix.
What happens in the POST request is that:
the server responds with status 302 (temporary redirect) and the redirect location
the browser redirects (basically does a GET request) to that location, also posting the two cookies.
.NET's HttpWebRequest attempts to do this redirect seamlessly, but in this case there are two issues (that I would consider bugs in the .NET implementation):
the GET request after the POST(redirect) has the same content-type as the POST request (application/x-www-form-urlencoded). For GET requests this shouldn't be specified
cookie handling issue (the most important issue) - The website sends two cookies: GX_SESSION_ID and JSESSIONID. The second has a path specified (/sintegra), while the first does not.
Here's the difference: the browser assigns by default a path of /(root) to the first cookie, while .NET assigns it the request url path (/sintegra/servlet/hwsintco).
Due to this, the last GET request (after redirect) to /sintegra/servlet/hwsintpe... does not get the first cookie passed in, as its path does not correspond.
The fixes
For the redirect problem (GET with content-type), the fix is to do the redirect manually, instead of relying on .NET for this.
To do this, tell it to not follow redirects:
postRequest.AllowAutoRedirect = false
and then read the redirect location from the POST response and manually do a GET request on it.
The cookie problem (that has happened to others as well)
For this, the fix I found was to take the misplaced cookie from the CookieContainer, set it's path correctly and add it back to the container in the correct location.
This is the code to do it:
private void FixMisplacedCookie(CookieContainer cookieContainer)
{
var misplacedCookie = cookieContainer.GetCookies(new Uri(Url))[0];
misplacedCookie.Path = "/"; // instead of "/sintegra/servlet/hwsintco"
//place the cookie in thee right place...
cookieContainer.SetCookies(
new Uri("https://www.sefaz.rr.gov.br/"),
misplacedCookie.ToString());
}
Here's all the code to make it work:
using System;
using System.IO;
using System.Net;
using System.Text;
namespace XYZ
{
public class Crawler
{
const string Url = "https://www.sefaz.rr.gov.br/sintegra/servlet/hwsintco";
public void Crawl()
{
var cookieContainer = new CookieContainer();
/* initial GET Request */
var getRequest = (HttpWebRequest)WebRequest.Create(Url);
getRequest.CookieContainer = cookieContainer;
ReadResponse(getRequest); // nothing to do with this, because captcha is f##%ing dumb :)
/* POST Request */
var postRequest = (HttpWebRequest)WebRequest.Create(Url);
postRequest.AllowAutoRedirect = false; // we'll do the redirect manually; .NET does it badly
postRequest.CookieContainer = cookieContainer;
postRequest.Method = "POST";
postRequest.ContentType = "application/x-www-form-urlencoded";
var postParameters =
"_EventName=E%27CONFIRMAR%27.&_EventGridId=&_EventRowId=&_MSG=&_CONINSEST=&" +
"_CONINSESTG=08775724000119&cfield=much&_VALIDATIONRESULT=1&BUTTON1=Confirmar&" +
"sCallerURL=";
var bytes = Encoding.UTF8.GetBytes(postParameters);
postRequest.ContentLength = bytes.Length;
using (var requestStream = postRequest.GetRequestStream())
requestStream.Write(bytes, 0, bytes.Length);
var webResponse = postRequest.GetResponse();
ReadResponse(postRequest); // not interested in this either
var redirectLocation = webResponse.Headers[HttpResponseHeader.Location];
var finalGetRequest = (HttpWebRequest)WebRequest.Create(redirectLocation);
/* Apply fix for the cookie */
FixMisplacedCookie(cookieContainer);
/* do the final request using the correct cookies. */
finalGetRequest.CookieContainer = cookieContainer;
var responseText = ReadResponse(finalGetRequest);
Console.WriteLine(responseText); // Hooray!
}
private static string ReadResponse(HttpWebRequest getRequest)
{
using (var responseStream = getRequest.GetResponse().GetResponseStream())
using (var sr = new StreamReader(responseStream, Encoding.UTF8))
{
return sr.ReadToEnd();
}
}
private void FixMisplacedCookie(CookieContainer cookieContainer)
{
var misplacedCookie = cookieContainer.GetCookies(new Uri(Url))[0];
misplacedCookie.Path = "/"; // instead of "/sintegra/servlet/hwsintco"
//place the cookie in thee right place...
cookieContainer.SetCookies(
new Uri("https://www.sefaz.rr.gov.br/"),
misplacedCookie.ToString());
}
}
}
Sometimes HttpWebRequest needs proxy initialization:
request.Proxy = new WebProxy();//in my case it doesn't need parameters, but you can set it to your proxy address
I need to change the order of headers, I'm using this:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(URL);
request.Method = context.Request.HttpMethod;
request.UserAgent = context.Request.UserAgent;
The output for that is:
GET /* HTTP/1.1
User-Agent: My Server
Host: 127.0.0.1:1080
But it should be
GET /* HTTP/1.1
Host: 127.0.0.1:1080
User-Agent: My Server
Any ideas?
Thanks for your time.
EDIT:
Maybe there's a way using other object ... it's also an option
There was an outstanding complaint that .NET doesn't let you modify the Host header a while back. It might not have been resolved. If it is really that important, you could always write socket-level code to send a prepared request (since it's just text).
I had this problem today but I created this hack:
/// <summary>
/// We aren't kids microsoft, we shouldn't need this
/// </summary>
public static void UnlockHeaders()
{
var tHashtable = typeof(WebHeaderCollection).Assembly.GetType("System.Net.HeaderInfoTable")
.GetFields(BindingFlags.NonPublic | BindingFlags.Static)
.Where(x => x.FieldType.Name == "Hashtable").Single();
var Table = (Hashtable)tHashtable.GetValue(null);
foreach (var Key in Table.Keys.Cast<string>().ToArray())
{
var HeaderInfo = Table[Key];
HeaderInfo.GetType().GetField("IsRequestRestricted", BindingFlags.NonPublic | BindingFlags.Instance).SetValue(HeaderInfo, false);
HeaderInfo.GetType().GetField("IsResponseRestricted", BindingFlags.NonPublic | BindingFlags.Instance).SetValue(HeaderInfo, false);
Table[Key] = HeaderInfo;
}
tHashtable.SetValue(null, Table);
}
Then You need call this UnlockHeaders function only one time in the program startup, after call the Header Collection in the HttpWebRequest class will accept any header to be manually added.
Then before add any header to the request, do this:
myHttpWebRequest.Headers["Host"] = "www.example.com";
After that first header will be the Host, since looks like in some .net versions the Headers field have more priority.
Note: This code don't works after .Net Core 3 because the reflection can't modify read-only values anymore, as a alternative, in my program I loaded a patched System.Net.WebHeaderCollection assembly early in my app initialization instead.