Develop Reference

How can I Bypass restricted HTTPWebRequest Headers

I've made a little helper function to deal with my HTTP requests that simply takes a string URL and a string array of headers formatted as Header-Name: Value
public static HttpWebResponse MakeHttpRequest(string URL, string[] Headers)
{
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(URL);
req.Headers.Clear();
foreach (string h in Headers)
{
var s = h.Split(new char[] { ':' }, 2);
req.Headers.Set(s[0], s[1]);
}
return (HttpWebResponse)req.GetResponse();
}
But if the Headers array contains a Restricted header (e.g User-Agent, or Content-Type, or Accept, etc..) I get an exception telling me to modify the header using its property or method, so I was thinking maybe there was some way to check if any of the Headers was restricted and automatically modify it using its property, unfortunately, I'm not too smart so I don't really know how to do that without having a lot of bloated code checking every single restricted header and having different code run for each one
Im guessing it can be done with Reflection but I'm not sure... Help?

There is static method to check if header is restricted:
bool restricted = System.Net.WebHeaderCollection.IsRestricted(key); // key = s[0]
You can use reflection to set related property like this:
public static HttpWebResponse MakeHttpRequest(string URL, string[] Headers) {
HttpWebRequest req = (HttpWebRequest) WebRequest.Create(URL);
req.Headers.Clear();
foreach (string h in Headers) {
var s = h.Split(new char[] {':'}, 2);
var key = s[0];
var value = s[1];
if (WebHeaderCollection.IsRestricted(key)) {
// remove "-" because some header names contain it, but .NET properties do not
key = key.Replace("-", "");
// get property with header name
var prop = typeof(HttpWebRequest).GetProperty(key, BindingFlags.Instance | BindingFlags.Public);
// set, changing type if necessary (some properties are long, or date, and your values are always strings)
prop.SetValue(req, Convert.ChangeType(value, prop.PropertyType, CultureInfo.InvariantCulture));
}
else {
req.Headers.Set(s[0], s[1]);
}
}
return (HttpWebResponse) req.GetResponse();
}
Ensure to test this code against all restricted headers (listed here) if you are going to use it, because I did not. Or just handle each header separately, because there are just 11 of them, not a hundred.

Register URL Prefix

I have this URL:
"data:image/png;base64,....."
If I open it on Chrome, I receive an image, but I don´t know how to configure the prefix "data" using WebRequest.
WebRequest.RegisterPrefix("data", new CustomWebRequestCreator());
I have to build a custom creator based on "IWebRequestCreate" but I don´t know what do I have to implement there.
public class CustomWebRequestCreator: IWebRequestCreate
{
public WebRequest Create(Uri uri)
{
HttpWebRequest webRequest = Activator.CreateInstance(typeof(HttpWebRequest),
BindingFlags.CreateInstance | BindingFlags.Public |
BindingFlags.NonPublic | BindingFlags.Instance,
null, new object[] { uri, null }, null) as HttpWebRequest;
webRequest.UserAgent = "OMG IT WORKED!";
return webRequest;
}
}
I always get "remote name could not be resolved" when trying to make a GET request:
WebRequest.RegisterPrefix("data", new CustomWebRequestCreator());
WebRequest request = WebRequest.Create("data:image/png;base64,.....");
request.Credentials = CredentialCache.DefaultCredentials;
WebResponse response = request.GetResponse(); <-- EXCEPTION
Can anyone help me with that?
Thanks!!

How to send lower-case Keep-Alive header through HttpWebRequest

I'm writing a bot, which should emulate firefox as closely as possible.
By examining the headers that it is sending, I've found one minor difference, that I do not know how to get rid off:
Firefox uses following keep-alive header:
Connection: keep-alive
While c# always sends out:
Connection: Keep-Alive
I know that it probably does not matter, but I'd still love to know if there is any way/hack to modify that header to be all lower case.
Any ideas how to do this?

In .net 4.0 this works:
request.Headers.GetType().InvokeMember(
"ChangeInternal",
BindingFlags.Instance | BindingFlags.NonPublic | BindingFlags.InvokeMethod,
Type.DefaultBinder,
request.Headers,
new object[] { "Connection", "keep-alive" }
);
Just be sure you don't actually set the KeepAlive property on the request

I also need to do this using C# and have been trying with HttpWebRequest. A non .NET SOAP web service I want to call expects Connection: keep-alive in lower case too.
I would rather not go down the route of doing socket programming for this, so if you have any suggestions on how you worked around this, if you did it would be very helpful.
My investigation so far :
When using http Protocol version 1.1, the header isn't even sent, even if you specify the property.
e.g.
var request = (HttpWebRequest)WebRequest.Create(Endpoint);
request.KeepAlive = true;
The solution for this is to use System.Reflection to modify the httpBehaviour which will mean Keep-Alive is sent. This will send an initial upper-case K and A 'Keep-Alive' on every request.
var sp = request.ServicePoint;
var prop = sp.GetType().GetProperty("HttpBehaviour", BindingFlags.Instance | BindingFlags.NonPublic);
prop.SetValue(sp, (byte)0, null);
I tried using System.Reflection to modify the header too. The below code will add the flag correctly in lower case :
request.Headers.GetType().InvokeMember("ChangeInternal", BindingFlags.Instance | BindingFlags.NonPublic | BindingFlags.InvokeMethod, Type.DefaultBinder, request.Headers, new object[] { "Connection", "keep-alive" });
However, at the point where I am calling GetResponse
var response = (HttpWebResponse)request.GetResponse()
The header is wiped out. Looking at the source code for HttpWebRequest.cs, this happens right before sending headers to the wire.
//
// The method is called right before sending headers to the wire*
// The result is updated internal _WriteBuffer
//
// See ClearRequestForResubmit() for the matching cleanup code path.
//
internal void SerializeHeaders() {
....
....
string connectionString = HttpKnownHeaderNames.Connection;
if (UsesProxySemantics || IsTunnelRequest ) {
_HttpRequestHeaders.RemoveInternal(HttpKnownHeaderNames.Connection);
connectionString = HttpKnownHeaderNames.ProxyConnection;
if (!ValidationHelper.IsBlankString(Connection)) {
_HttpRequestHeaders.AddInternal(HttpKnownHeaderNames.ProxyConnection, _HttpRequestHeaders[HttpKnownHeaderNames.Connection]);
}
}
else {
_HttpRequestHeaders.RemoveInternal(HttpKnownHeaderNames.ProxyConnection);
}
RemoveInternal will also remove the header we have hacked in using Reflection.
So this still leaves me stuck.
Are there other any way around this other than going at a socket level?
Are there other classes or 3rd party libraries which allow me to amend headers as I wish?
Sorry this is not an answer, but I can't yet comment on your question.

Connection: keep-alive is default header of Chrome and Firefox browser.
Connection: Keep-Alive is default header of Internet Explorer. Absolutely
Connection: Keep-Alive is default header of HttpWebRequest. I think you should writing a bot like IE is best choice if using HttpWebRequest.

Using reflection, you can replace a WebHeaderCollection's internal NameValueCollection with a custom implementation, such as the following:
// When injected into a WebHeaderCollection, ensures that
// there's always exactly one "Connection: keep-alive" header.
public class CustomNameValueCollection : NameValueCollection {
private const BindingFlags allInstance =
BindingFlags.Instance | BindingFlags.Public | BindingFlags.NonPublic;
private static readonly PropertyInfo innerCollProperty =
typeof(WebHeaderCollection).GetProperty("InnerCollection", allInstance);
private static readonly FieldInfo innerCollField =
typeof(WebHeaderCollection).GetField("m_InnerCollection", allInstance);
public static void InjectInto(WebHeaderCollection coll) {
// WebHeaderCollection uses a custom IEqualityComparer for its internal
// NameValueCollection. Here we get the InnerCollection property so that
// we can reuse its IEqualityComparer (via our constructor).
var innerColl = (NameValueCollection) innerCollProperty.GetValue(coll);
innerCollField.SetValue(coll, new CustomNameValueCollection(innerColl));
}
private CustomNameValueCollection(NameValueCollection coll) : base(coll) {
Remove("Connection");
base.Add("Connection", "keep-alive");
}
public override void Add(string name, string value) {
if (name == "Connection") return;
base.Add(name, value);
}
}
Use it like this:
var request = (HttpWebRequest) WebRequest.Create("https://www.google.com/");
CustomNameValueCollection.InjectInto(request.Headers);
using (var response = request.GetResponse()) {
...
}

Is it possible to change headers order using HttpWebRequest?

I need to change the order of headers, I'm using this:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(URL);
request.Method = context.Request.HttpMethod;
request.UserAgent = context.Request.UserAgent;
The output for that is:
GET /* HTTP/1.1
User-Agent: My Server
Host: 127.0.0.1:1080
But it should be
GET /* HTTP/1.1
Host: 127.0.0.1:1080
User-Agent: My Server
Any ideas?
Thanks for your time.
EDIT:
Maybe there's a way using other object ... it's also an option

There was an outstanding complaint that .NET doesn't let you modify the Host header a while back. It might not have been resolved. If it is really that important, you could always write socket-level code to send a prepared request (since it's just text).

I had this problem today but I created this hack:
/// <summary>
/// We aren't kids microsoft, we shouldn't need this
/// </summary>
public static void UnlockHeaders()
{
var tHashtable = typeof(WebHeaderCollection).Assembly.GetType("System.Net.HeaderInfoTable")
.GetFields(BindingFlags.NonPublic | BindingFlags.Static)
.Where(x => x.FieldType.Name == "Hashtable").Single();
var Table = (Hashtable)tHashtable.GetValue(null);
foreach (var Key in Table.Keys.Cast<string>().ToArray())
{
var HeaderInfo = Table[Key];
HeaderInfo.GetType().GetField("IsRequestRestricted", BindingFlags.NonPublic | BindingFlags.Instance).SetValue(HeaderInfo, false);
HeaderInfo.GetType().GetField("IsResponseRestricted", BindingFlags.NonPublic | BindingFlags.Instance).SetValue(HeaderInfo, false);
Table[Key] = HeaderInfo;
}
tHashtable.SetValue(null, Table);
}
Then You need call this UnlockHeaders function only one time in the program startup, after call the Header Collection in the HttpWebRequest class will accept any header to be manually added.
Then before add any header to the request, do this:
myHttpWebRequest.Headers["Host"] = "www.example.com";
After that first header will be the Host, since looks like in some .net versions the Headers field have more priority.
Note: This code don't works after .Net Core 3 because the reflection can't modify read-only values anymore, as a alternative, in my program I loaded a patched System.Net.WebHeaderCollection assembly early in my app initialization instead.

Request Web Page in c# spoofing the Host

I need to create a request for a web page delivered to our web sites, but I need to be able to set the host header information too. I have tried this using HttpWebRequest, but the Header information is read only (Or at least the Host part of it is). I need to do this because we want to perform the initial request for a page before the user can. We have 10 web server which are load balanced, so we need to request the file from each of the web servers.
I have tried the following:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://192.168.1.5/filename.htm");
request.Headers.Set("Host", "www.mywebsite.com");
WebResponse response = request.GetResponse();
Obviously this does not work, as I can't update the header, and I don't know if this is indeed the right way to do it.

Although this is a very late answer, maybe someone can get benefit of it
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(new Uri("http://192.168.1.1"));
request.Headers.GetType().InvokeMember("ChangeInternal", BindingFlags.NonPublic | BindingFlags.Instance | BindingFlags.InvokeMethod, null, request.Headers, new object[] {"Host","www.mysite.com"});
Reflection is your friend :)

I have managed to find out a more long winded route by using sockets. I found the answer in the MSDN page for IPEndPoint:
string getString = "GET /path/mypage.htm HTTP/1.1\r\nHost: www.mysite.mobi\r\nConnection: Close\r\n\r\n";
Encoding ASCII = Encoding.ASCII;
Byte[] byteGetString = ASCII.GetBytes(getString);
Byte[] receiveByte = new Byte[256];
Socket socket = null;
String strPage = null;
try
{
IPEndPoint ip = new IPEndPoint(IPAddress.Parse("10.23.1.93"), 80);
socket = new Socket(ip.AddressFamily, SocketType.Stream, ProtocolType.Tcp);
socket.Connect(ip);
}
catch (SocketException ex)
{
Console.WriteLine("Source:" + ex.Source);
Console.WriteLine("Message:" + ex.Message);
}
socket.Send(byteGetString, byteGetString.Length, 0);
Int32 bytes = socket.Receive(receiveByte, receiveByte.Length, 0);
strPage = strPage + ASCII.GetString(receiveByte, 0, bytes);
while (bytes > 0)
{
bytes = socket.Receive(receiveByte, receiveByte.Length, 0);
strPage = strPage + ASCII.GetString(receiveByte, 0, bytes);
}
socket.Close();

I had a problem where the URL dns I used had several different IP addresses, I wanted to call each address separately using the same dns name in the host - the solution is using a proxy:
string retVal = "";
// Can't change the 'Host' header property because .NET protects it
// HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
// request.Headers.Set(HttpRequestHeader.Host, DEPLOYER_HOST);
// so we must use a workaround
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Proxy = new WebProxy(ip);
using (WebResponse response = request.GetResponse())
{
using (TextReader reader = new StreamReader(response.GetResponseStream()))
{
string line;
while ((line = reader.ReadLine()) != null)
retVal += line;
}
}
return retVal;
Host header is set from 'url' automatically by .NET, and 'ip' contains the actual address of the web server you want to contact (you can use a dns name here too)

I know this is old, but I came across this same exact problem, and I found a better solution to this then using sockets or reflection...
What I did was create a new class that durives from WebHeaderCollection and bypasses validation of what you stick inside it:
public class MyHeaderCollection:WebHeaderCollection
{
public new void Set(string name, string value)
{
AddWithoutValidate(name, value);
}
//or
public new string this[string name]
{
get { return base[name]; }
set { AddWithoutValidate(name, value); }
}
}
and here is how you use it:
var http = WebRequest.Create("http://example.com/");
var headers = new MyHeaderCollection();
http.Headers = headers;
//Now you can add/override anything you like without validation:
headers.Set("Host", http.RequestUri.Host);
//or
headers["Host"] = http.RequestUri.Host;
Hope this helps anyone looking for this!

I know this is an old question, but these days, you can do.
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://192.168.1.5/filename.htm");
request.Host = "www.mywebstite.com";
WebResponse response = request.GetResponse();

The "Host" header is protected and cannot be modified programmatically. I suppose to work around this, you could try and bind via reflection to the private "InnerCollection" property of the WebRequest object and calling the "Set" ar "Add" method on it to modify the Host header. I haven't tried this, but from a quick look at the source code in Reflector, I think it's easily accomplished. But yeah, binding to private properties of framework objects is not the best way to accomplish things. :) Use only if you MUST.
edit: Or like the other guy mentions in the linked question, just open up a socket and do a quick "GET" manually. Should be a no brainer, if you don't need to tinker with other stuff, like cookies or whatever else niceties the HttpWebRequest provides.

Alright, little bit of research turns up this:
https://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=384456
Seems MS may do something about this at some point.

You can use my solution for this problem, it posted here :
How to set custom "Host" header in HttpWebRequest?
This can help you to edit host header, and avoid to using proxy and direct socket requests.

Necromancing.
For those still on .NET 2.0
It is in fact quite easy, if you know how.
Problem is, you can't set the host header, because the framework won't let you change the value at runtime. (.net framework 4.0+ will let you override host in a httpwebrequest).
Next attempt will be setting the header with reflection - as demonstrated in the top upvoted answer here - to get around it, which will let you change the header value. But at runtime, it will overwrite this value with the host part of the url, which means reflection will bring you nothing, which is why I don't understand why people keep upvoting this.
If the dns-name doesn't exist, which is quite frankly the only case in which you want to do this in the first place, you can't set it, because .NET can't resolve it, and you can't override the .NET DNS resolver.
But what you can do, is setting a webproxy with the exact same IP as the destination server.
So, if your server IP is 28.14.88.71:
public class myweb : System.Net.WebClient
{
protected override System.Net.WebRequest GetWebRequest(System.Uri address)
{
System.Net.WebRequest request = (System.Net.WebRequest)base.GetWebRequest(address);
//string host = "redmine.nonexistantdomain.com";
//request.Headers.GetType().InvokeMember("ChangeInternal",
// System.Reflection.BindingFlags.NonPublic |
// System.Reflection.BindingFlags.Instance |
// System.Reflection.BindingFlags.InvokeMethod, null,
// request.Headers, new object[] { "Host", host }
//);
//server IP and port
request.Proxy = new System.Net.WebProxy("http://28.14.88.71:80");
// .NET 4.0 only
System.Net.HttpWebRequest foo = (System.Net.HttpWebRequest)request;
//foo.Host = host;
// The below reflection-based operation is not necessary,
// if the server speaks HTTP 1.1 correctly
// and the firewall doesn't interfere
// https://yoursunny.com/t/2009/HttpWebRequest-IP/
System.Reflection.FieldInfo horribleProxyServicePoint = (typeof(System.Net.ServicePoint))
.GetField("m_ProxyServicePoint", System.Reflection.BindingFlags.NonPublic |
System.Reflection.BindingFlags.Instance);
horribleProxyServicePoint.SetValue(foo.ServicePoint, false);
return foo; // or return request; if you don't neet this
}
}
and voila, now
myweb wc = new myweb();
string str = wc.DownloadString("http://redmine.netexistantdomain.com");
and you get the correct page back, if 28.14.88.71 is a webserver with virtual name-based hosting (based on http-host-header).
Now you have the correct answer to the original question, for both WebRequest and WebClient. I think using custom sockets to do this would be the wrong approach, particularly when SSL should be used, and when an actual solution is that simple...