I need to change the order of headers, I'm using this:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(URL);
request.Method = context.Request.HttpMethod;
request.UserAgent = context.Request.UserAgent;
The output for that is:
GET /* HTTP/1.1
User-Agent: My Server
Host: 127.0.0.1:1080
But it should be
GET /* HTTP/1.1
Host: 127.0.0.1:1080
User-Agent: My Server
Any ideas?
Thanks for your time.
EDIT:
Maybe there's a way using other object ... it's also an option
There was an outstanding complaint that .NET doesn't let you modify the Host header a while back. It might not have been resolved. If it is really that important, you could always write socket-level code to send a prepared request (since it's just text).
I had this problem today but I created this hack:
/// <summary>
/// We aren't kids microsoft, we shouldn't need this
/// </summary>
public static void UnlockHeaders()
{
var tHashtable = typeof(WebHeaderCollection).Assembly.GetType("System.Net.HeaderInfoTable")
.GetFields(BindingFlags.NonPublic | BindingFlags.Static)
.Where(x => x.FieldType.Name == "Hashtable").Single();
var Table = (Hashtable)tHashtable.GetValue(null);
foreach (var Key in Table.Keys.Cast<string>().ToArray())
{
var HeaderInfo = Table[Key];
HeaderInfo.GetType().GetField("IsRequestRestricted", BindingFlags.NonPublic | BindingFlags.Instance).SetValue(HeaderInfo, false);
HeaderInfo.GetType().GetField("IsResponseRestricted", BindingFlags.NonPublic | BindingFlags.Instance).SetValue(HeaderInfo, false);
Table[Key] = HeaderInfo;
}
tHashtable.SetValue(null, Table);
}
Then You need call this UnlockHeaders function only one time in the program startup, after call the Header Collection in the HttpWebRequest class will accept any header to be manually added.
Then before add any header to the request, do this:
myHttpWebRequest.Headers["Host"] = "www.example.com";
After that first header will be the Host, since looks like in some .net versions the Headers field have more priority.
Note: This code don't works after .Net Core 3 because the reflection can't modify read-only values anymore, as a alternative, in my program I loaded a patched System.Net.WebHeaderCollection assembly early in my app initialization instead.
Related
In spite of the RFC stating that the order of uniquely-named headers shouldn't matter, the website I'm sending this request to does implement a check on the order of headers.
This works:
GET https://www.thewebsite.com HTTP/1.1
Host: www.thewebsite.com
Connection: keep-alive
Accept: */*
User-Agent: Mozilla/5.0 etc
This doesn't work:
GET https://www.thewebsite.com HTTP/1.1
Accept: */*
User-Agent: Mozilla/5.0 etc
Host: www.thewebsite.com
Connection: keep-alive
The default HttpWebRequest seems to put the Host and Connection headers at the end, before the blank line, rather than just after the url.
Is there any way (using a fork of HttpWebRequest or some other library in Nuget even) to specify the order of headers in a HttpWebRequest?
If possible, I'd rather not start going down the route of implementing a proxy to sort them or having to code the whole thing up using a TcpClient.
I'd appreciate any hints at all on this.
Update: With Fiddler running, header order in HttpWebrequest can be re-shuffled in CustomRules.cs. Still no closer to a solution without a proxy though.
Some server implement header ordering as a precaution for any attacks or spam, an article explaining Why ordering HTTP headers is important.
But the standard is, the order in which header fields with differing field names are received is not significant.
HttpWebRequest, there is no easy way to order the headers and the Connection and Host is added internally.
If ordering is really important, use the HttpClient instead, it can easily arrange the Headers based on the example of #Jason.
If you will be using HttpClient, you can create a custom HttpClientHandler and you can arrange your header from there. It can be something like this.
HANDLER
public class CustomHttpClientHandler : HttpClientHandler
{
protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
{
request.Headers.Clear();
request.Headers.Add("Host", $"{request.RequestUri.Authority}");
request.Headers.Add("Connection", "keep-alive");
request.Headers.Add("Accept", "*/*");
request.Headers.Add("User-Agent", "Mozilla/5.0 etc");
return await base.SendAsync(request, cancellationToken);
}
}
IMPLEMENTATION
HttpClient clientRequest = new HttpClient(new CustomHttpClientHandler());
await clientRequest.GetAsync(url);
.Net Core
If you set the headers yourself, you can specify the order. When the common headers are added it will find the existing headers instead of appending them:
using System.Net;
namespace ConsoleApp2
{
class Program
{
static void Main(string[] args)
{
var request = WebRequest.Create("http://www.google.com");
request.Headers.Add("Host", "www.google.com");
// this will be set within GetResponse.
request.Headers.Add("Connection", "");
request.Headers.Add("Accept", "*/*");
request.Headers.Add("User-Agent", "Mozilla/5.0 etc");
request.GetResponse();
}
}
}
Here is an example with HttpClient:
using System.Net.Http;
using System.Threading.Tasks;
namespace ConsoleApp3
{
class Program
{
static async Task Main(string[] args)
{
var client = new HttpClient();
client.DefaultRequestHeaders.Add("Host", "www.google.com");
client.DefaultRequestHeaders.Add("Connection", "keep-alive");
client.DefaultRequestHeaders.Add("Accept", "*/*");
client.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 etc");
await client.GetAsync("http://www.google.com");
await client.PostAsync("http://www.google.com", new StringContent(""));
}
}
}
Edit
The above code did not work on .Net Framework only .Net Core
.Net Framework
On .Net Framework the headers are reserved so they cannot be set like this, see Cannot set some HTTP headers when using System.Net.WebRequest.
One work around is to use reflection to modify the behavior of the framework class, but be warned this could break if the libraries are updated so it's not recommended!.
Essentially, HttpWebRequest calls ToString on WebHeaderCollection to serialize.
See https://referencesource.microsoft.com/#System/net/System/Net/HttpWebRequest.cs,5079
So a custom class can be made to override ToString. Unfortunately reflection is needed to set the headers as WebRequest copies the collection on assignment to Headers, instead of taking the new reference.
WARNING, THE FOLLOWING CODE CAN BREAK IF FRAMEWORK CHANGES
If you use this, write some unit tests that verify the behavior still stays consistent after updates to .NET Framework
using System.Collections.Generic;
using System.Linq;
using System.Net;
using System.Reflection;
namespace ConsoleApp2
{
class Program
{
static void Main(string[] args)
{
// WARNING, CODE CAN BREAK IF FRAMEWORK CHANGES
// If you use this, write some unit tests that verify the behavior still stays consistent after updates to .NET Framework
var request = (HttpWebRequest)WebRequest.Create("http://www.google.com");
var field = typeof(HttpWebRequest).GetField("_HttpRequestHeaders", BindingFlags.Instance | BindingFlags.NonPublic);
var headers = new CustomWebHeaderCollection(new Dictionary<string, string>
{
["Host"] = "www.google.com",
["Connection"] = "keep-alive",
["Accept"] = "*/*",
["User-Agent"] = "Mozilla/5.0 etc"
});
field.SetValue(request, headers);
request.GetResponse();
}
}
internal class CustomWebHeaderCollection : WebHeaderCollection
{
private readonly Dictionary<string, string> _customHeaders;
public CustomWebHeaderCollection(Dictionary<string, string> customHeaders)
{
_customHeaders = customHeaders;
}
public override string ToString()
{
// Could call base.ToString() split on Newline and sort as needed
var lines = _customHeaders
.Select(kvp => $"{kvp.Key}: {kvp.Value}")
// These two new lines are needed after the HTTP header
.Concat(new [] { string.Empty, string.Empty });
var headers = string.Join("\r\n", lines);
return headers;
}
}
}
Scope:
I am developing a C# aplication to simulate queries into this site. I am quite familiar with simulating web requests for achieving the same human steps, but using code instead.
If you want to try yourself, just type this number into the CNPJ box:
08775724000119 and write the captcha and click on Confirmar
I've dealed with the captcha already, so it's not a problem anymore.
Problem:
As soon as i execute the POST request for a "CNPJ", a exception is thrown:
The remote server returned an error: (403) Forbidden.
Fiddler Debugger Output:
Link for Fiddler Download
This is the request generated by my browser, not by my code
POST https://www.sefaz.rr.gov.br/sintegra/servlet/hwsintco HTTP/1.1
Host: www.sefaz.rr.gov.br
Connection: keep-alive
Content-Length: 208
Cache-Control: max-age=0
Origin: https://www.sefaz.rr.gov.br
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.97 Safari/537.11
Content-Type: application/x-www-form-urlencoded
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Referer: https://www.sefaz.rr.gov.br/sintegra/servlet/hwsintco
Accept-Encoding: gzip,deflate,sdch
Accept-Language: pt-BR,pt;q=0.8,en-US;q=0.6,en;q=0.4
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
Cookie: GX_SESSION_ID=gGUYxyut5XRAijm0Fx9ou7WnXbVGuUYoYTIKtnDydVM%3D; JSESSIONID=OVuuMFCgQv9k2b3fGyHjSZ9a.undefined
// PostData :
_EventName=E%27CONFIRMAR%27.&_EventGridId=&_EventRowId=&_MSG=&_CONINSEST=&_CONINSESTG=08775724000119&cfield=rice&_VALIDATIONRESULT=1&BUTTON1=Confirmar&sCallerURL=http%3A%2F%2Fwww.sintegra.gov.br%2Fnew_bv.html
Code samples and References used:
I'm using a self developed library to handle/wrap the Post and Get requests.
The request object has the same parameters (Host,Origin, Referer, Cookies..) as the one issued by the browser (logged my fiddler up here).
I've also managed to set the ServicePointValidator of certificates by using:
ServicePointManager.ServerCertificateValidationCallback =
new RemoteCertificateValidationCallback (delegate { return true; });
After all that configuration, i stil getting the forbidden exception.
Here is how i simulate the request and the exception is thrown
try
{
this.Referer = Consts.REFERER;
// PARAMETERS: URL, POST DATA, ThrownException (bool)
response = Post (Consts.QUERYURL, postData, true);
}
catch (Exception ex)
{
string s = ex.Message;
}
Thanks in advance for any help / solution to my problem
Update 1:
I was missing the request for the homepage, which generates cookies (Thanks #W0lf for pointing me that out)
Now there's another weird thing. Fiddler is not showing my Cookies on the request, but here they are :
I made a successful request using the browser and recorded it in Fiddler.
The only things that differ from your request are:
my browser sent no value for the sCallerURL parameter (I have sCallerURL= instead of sCallerURL=http%3A%2F%2Fwww....)
the session ids are different (obviously)
I have other Accept-Language: values (I'm pretty sure this is not important)
the Content-Length is different (obviously)
Update
OK, I thought the Fiddler trace was from your application. In case you are not setting cookies on your request, do this:
before posting data, do a GET request to https://www.sefaz.rr.gov.br/sintegra/servlet/hwsintco. If you examine the response, you'll notice the website sends two session cookies.
when you do the POST request, make sure to attach the cookies you got at the previous step
If you don't know how to store the cookies and use them in the other request, take a look here.
Update 2
The problems
OK, I managed to reproduce the 403, figured out what caused it, and found a fix.
What happens in the POST request is that:
the server responds with status 302 (temporary redirect) and the redirect location
the browser redirects (basically does a GET request) to that location, also posting the two cookies.
.NET's HttpWebRequest attempts to do this redirect seamlessly, but in this case there are two issues (that I would consider bugs in the .NET implementation):
the GET request after the POST(redirect) has the same content-type as the POST request (application/x-www-form-urlencoded). For GET requests this shouldn't be specified
cookie handling issue (the most important issue) - The website sends two cookies: GX_SESSION_ID and JSESSIONID. The second has a path specified (/sintegra), while the first does not.
Here's the difference: the browser assigns by default a path of /(root) to the first cookie, while .NET assigns it the request url path (/sintegra/servlet/hwsintco).
Due to this, the last GET request (after redirect) to /sintegra/servlet/hwsintpe... does not get the first cookie passed in, as its path does not correspond.
The fixes
For the redirect problem (GET with content-type), the fix is to do the redirect manually, instead of relying on .NET for this.
To do this, tell it to not follow redirects:
postRequest.AllowAutoRedirect = false
and then read the redirect location from the POST response and manually do a GET request on it.
The cookie problem (that has happened to others as well)
For this, the fix I found was to take the misplaced cookie from the CookieContainer, set it's path correctly and add it back to the container in the correct location.
This is the code to do it:
private void FixMisplacedCookie(CookieContainer cookieContainer)
{
var misplacedCookie = cookieContainer.GetCookies(new Uri(Url))[0];
misplacedCookie.Path = "/"; // instead of "/sintegra/servlet/hwsintco"
//place the cookie in thee right place...
cookieContainer.SetCookies(
new Uri("https://www.sefaz.rr.gov.br/"),
misplacedCookie.ToString());
}
Here's all the code to make it work:
using System;
using System.IO;
using System.Net;
using System.Text;
namespace XYZ
{
public class Crawler
{
const string Url = "https://www.sefaz.rr.gov.br/sintegra/servlet/hwsintco";
public void Crawl()
{
var cookieContainer = new CookieContainer();
/* initial GET Request */
var getRequest = (HttpWebRequest)WebRequest.Create(Url);
getRequest.CookieContainer = cookieContainer;
ReadResponse(getRequest); // nothing to do with this, because captcha is f##%ing dumb :)
/* POST Request */
var postRequest = (HttpWebRequest)WebRequest.Create(Url);
postRequest.AllowAutoRedirect = false; // we'll do the redirect manually; .NET does it badly
postRequest.CookieContainer = cookieContainer;
postRequest.Method = "POST";
postRequest.ContentType = "application/x-www-form-urlencoded";
var postParameters =
"_EventName=E%27CONFIRMAR%27.&_EventGridId=&_EventRowId=&_MSG=&_CONINSEST=&" +
"_CONINSESTG=08775724000119&cfield=much&_VALIDATIONRESULT=1&BUTTON1=Confirmar&" +
"sCallerURL=";
var bytes = Encoding.UTF8.GetBytes(postParameters);
postRequest.ContentLength = bytes.Length;
using (var requestStream = postRequest.GetRequestStream())
requestStream.Write(bytes, 0, bytes.Length);
var webResponse = postRequest.GetResponse();
ReadResponse(postRequest); // not interested in this either
var redirectLocation = webResponse.Headers[HttpResponseHeader.Location];
var finalGetRequest = (HttpWebRequest)WebRequest.Create(redirectLocation);
/* Apply fix for the cookie */
FixMisplacedCookie(cookieContainer);
/* do the final request using the correct cookies. */
finalGetRequest.CookieContainer = cookieContainer;
var responseText = ReadResponse(finalGetRequest);
Console.WriteLine(responseText); // Hooray!
}
private static string ReadResponse(HttpWebRequest getRequest)
{
using (var responseStream = getRequest.GetResponse().GetResponseStream())
using (var sr = new StreamReader(responseStream, Encoding.UTF8))
{
return sr.ReadToEnd();
}
}
private void FixMisplacedCookie(CookieContainer cookieContainer)
{
var misplacedCookie = cookieContainer.GetCookies(new Uri(Url))[0];
misplacedCookie.Path = "/"; // instead of "/sintegra/servlet/hwsintco"
//place the cookie in thee right place...
cookieContainer.SetCookies(
new Uri("https://www.sefaz.rr.gov.br/"),
misplacedCookie.ToString());
}
}
}
Sometimes HttpWebRequest needs proxy initialization:
request.Proxy = new WebProxy();//in my case it doesn't need parameters, but you can set it to your proxy address
I have a WebService that is being called from an Iphone app (that I am also building)
In my webservice is it being self hosted inside a Service and it is all working well, except I would like to move a security token into the Headers of the Request so that the class objects remain neat. (If I can't get it in the header, i'll resort to putting it in the class but that's a bit ugly imo).
I have looked at the code in this http://msdn.microsoft.com/en-us/library/system.servicemodel.operationcontext.incomingmessageheaders.aspx#Y342 and I can't seem to enumerate the header value.
Looking in Fiddler, I can see the header is being passed through
POST http://192.168.1.221:11001/StockControl/json/SubmitResults HTTP/1.1
Device-Token: bwI2YiAHR4q3Ba5JVj99Cw==
Content-Type: application/json
Content-Length: 1663
User-Agent: StockManage/1.0 CFNetwork/609 Darwin/12.1.0
I'm not sure if I haven't set up my SelfHosted configuration correctly or if I haven't implemented a necessary interface .
WCF IClientMessageInspector and the incoming SOAP headers but this is using SOAP and I'm using JSON.
My Endpoint is setup using the following
WebHttpBinding jsonBind = new WebHttpBinding();
ServiceEndpoint jsonServer = host.AddServiceEndpoint(typeof(POSServer.StockControl.IStockService), jsonBind, "json");
jsonServer.Behaviors.Add(new WebHttpBehavior
{
DefaultBodyStyle = System.ServiceModel.Web.WebMessageBodyStyle.Bare,
HelpEnabled = true,
DefaultOutgoingResponseFormat = WebMessageFormat.Json
});
Finally in my SubmitResults function in my Service implementation
public bool SubmitResults(Business.StockResultData theData)
{
DateTime uploadTime = DateTime.Now;
int index = OperationContext.Current.IncomingMessageHeaders.FindHeader("Device-Token", "");
this.WriteHeaders(OperationContext.Current.IncomingMessageHeaders);
this.WriteHeaders(OperationContext.Current.RequestContext.RequestMessage.Headers);
but index is always -1 (not found) and the WriteHeaders cannot see the header.
After a lot of searching I believe I found the answer here . (http://social.msdn.microsoft.com/Forums/pl-PL/wcf/thread/72ee44cc-58bb-45b2-aff7-49d9bbc8176e)
HttpRequestMessageProperty reqMsg =
OperationContext.Current.IncomingMessageProperties["httpRequest"] as
HttpRequestMessageProperty;
This works for me...where apiKey is Header name
> var headers =OperationContext.Current.IncomingMessageProperties["httpRequest"];
var apiToken = ((HttpRequestMessageProperty)headers).Headers["apiKey"];
I'm writing a bot, which should emulate firefox as closely as possible.
By examining the headers that it is sending, I've found one minor difference, that I do not know how to get rid off:
Firefox uses following keep-alive header:
Connection: keep-alive
While c# always sends out:
Connection: Keep-Alive
I know that it probably does not matter, but I'd still love to know if there is any way/hack to modify that header to be all lower case.
Any ideas how to do this?
In .net 4.0 this works:
request.Headers.GetType().InvokeMember(
"ChangeInternal",
BindingFlags.Instance | BindingFlags.NonPublic | BindingFlags.InvokeMethod,
Type.DefaultBinder,
request.Headers,
new object[] { "Connection", "keep-alive" }
);
Just be sure you don't actually set the KeepAlive property on the request
I also need to do this using C# and have been trying with HttpWebRequest. A non .NET SOAP web service I want to call expects Connection: keep-alive in lower case too.
I would rather not go down the route of doing socket programming for this, so if you have any suggestions on how you worked around this, if you did it would be very helpful.
My investigation so far :
When using http Protocol version 1.1, the header isn't even sent, even if you specify the property.
e.g.
var request = (HttpWebRequest)WebRequest.Create(Endpoint);
request.KeepAlive = true;
The solution for this is to use System.Reflection to modify the httpBehaviour which will mean Keep-Alive is sent. This will send an initial upper-case K and A 'Keep-Alive' on every request.
var sp = request.ServicePoint;
var prop = sp.GetType().GetProperty("HttpBehaviour", BindingFlags.Instance | BindingFlags.NonPublic);
prop.SetValue(sp, (byte)0, null);
I tried using System.Reflection to modify the header too. The below code will add the flag correctly in lower case :
request.Headers.GetType().InvokeMember("ChangeInternal", BindingFlags.Instance | BindingFlags.NonPublic | BindingFlags.InvokeMethod, Type.DefaultBinder, request.Headers, new object[] { "Connection", "keep-alive" });
However, at the point where I am calling GetResponse
var response = (HttpWebResponse)request.GetResponse()
The header is wiped out. Looking at the source code for HttpWebRequest.cs, this happens right before sending headers to the wire.
//
// The method is called right before sending headers to the wire*
// The result is updated internal _WriteBuffer
//
// See ClearRequestForResubmit() for the matching cleanup code path.
//
internal void SerializeHeaders() {
....
....
string connectionString = HttpKnownHeaderNames.Connection;
if (UsesProxySemantics || IsTunnelRequest ) {
_HttpRequestHeaders.RemoveInternal(HttpKnownHeaderNames.Connection);
connectionString = HttpKnownHeaderNames.ProxyConnection;
if (!ValidationHelper.IsBlankString(Connection)) {
_HttpRequestHeaders.AddInternal(HttpKnownHeaderNames.ProxyConnection, _HttpRequestHeaders[HttpKnownHeaderNames.Connection]);
}
}
else {
_HttpRequestHeaders.RemoveInternal(HttpKnownHeaderNames.ProxyConnection);
}
RemoveInternal will also remove the header we have hacked in using Reflection.
So this still leaves me stuck.
Are there other any way around this other than going at a socket level?
Are there other classes or 3rd party libraries which allow me to amend headers as I wish?
Sorry this is not an answer, but I can't yet comment on your question.
Connection: keep-alive is default header of Chrome and Firefox browser.
Connection: Keep-Alive is default header of Internet Explorer. Absolutely
Connection: Keep-Alive is default header of HttpWebRequest. I think you should writing a bot like IE is best choice if using HttpWebRequest.
Using reflection, you can replace a WebHeaderCollection's internal NameValueCollection with a custom implementation, such as the following:
// When injected into a WebHeaderCollection, ensures that
// there's always exactly one "Connection: keep-alive" header.
public class CustomNameValueCollection : NameValueCollection {
private const BindingFlags allInstance =
BindingFlags.Instance | BindingFlags.Public | BindingFlags.NonPublic;
private static readonly PropertyInfo innerCollProperty =
typeof(WebHeaderCollection).GetProperty("InnerCollection", allInstance);
private static readonly FieldInfo innerCollField =
typeof(WebHeaderCollection).GetField("m_InnerCollection", allInstance);
public static void InjectInto(WebHeaderCollection coll) {
// WebHeaderCollection uses a custom IEqualityComparer for its internal
// NameValueCollection. Here we get the InnerCollection property so that
// we can reuse its IEqualityComparer (via our constructor).
var innerColl = (NameValueCollection) innerCollProperty.GetValue(coll);
innerCollField.SetValue(coll, new CustomNameValueCollection(innerColl));
}
private CustomNameValueCollection(NameValueCollection coll) : base(coll) {
Remove("Connection");
base.Add("Connection", "keep-alive");
}
public override void Add(string name, string value) {
if (name == "Connection") return;
base.Add(name, value);
}
}
Use it like this:
var request = (HttpWebRequest) WebRequest.Create("https://www.google.com/");
CustomNameValueCollection.InjectInto(request.Headers);
using (var response = request.GetResponse()) {
...
}
I need to create a request for a web page delivered to our web sites, but I need to be able to set the host header information too. I have tried this using HttpWebRequest, but the Header information is read only (Or at least the Host part of it is). I need to do this because we want to perform the initial request for a page before the user can. We have 10 web server which are load balanced, so we need to request the file from each of the web servers.
I have tried the following:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://192.168.1.5/filename.htm");
request.Headers.Set("Host", "www.mywebsite.com");
WebResponse response = request.GetResponse();
Obviously this does not work, as I can't update the header, and I don't know if this is indeed the right way to do it.
Although this is a very late answer, maybe someone can get benefit of it
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(new Uri("http://192.168.1.1"));
request.Headers.GetType().InvokeMember("ChangeInternal", BindingFlags.NonPublic | BindingFlags.Instance | BindingFlags.InvokeMethod, null, request.Headers, new object[] {"Host","www.mysite.com"});
Reflection is your friend :)
I have managed to find out a more long winded route by using sockets. I found the answer in the MSDN page for IPEndPoint:
string getString = "GET /path/mypage.htm HTTP/1.1\r\nHost: www.mysite.mobi\r\nConnection: Close\r\n\r\n";
Encoding ASCII = Encoding.ASCII;
Byte[] byteGetString = ASCII.GetBytes(getString);
Byte[] receiveByte = new Byte[256];
Socket socket = null;
String strPage = null;
try
{
IPEndPoint ip = new IPEndPoint(IPAddress.Parse("10.23.1.93"), 80);
socket = new Socket(ip.AddressFamily, SocketType.Stream, ProtocolType.Tcp);
socket.Connect(ip);
}
catch (SocketException ex)
{
Console.WriteLine("Source:" + ex.Source);
Console.WriteLine("Message:" + ex.Message);
}
socket.Send(byteGetString, byteGetString.Length, 0);
Int32 bytes = socket.Receive(receiveByte, receiveByte.Length, 0);
strPage = strPage + ASCII.GetString(receiveByte, 0, bytes);
while (bytes > 0)
{
bytes = socket.Receive(receiveByte, receiveByte.Length, 0);
strPage = strPage + ASCII.GetString(receiveByte, 0, bytes);
}
socket.Close();
I had a problem where the URL dns I used had several different IP addresses, I wanted to call each address separately using the same dns name in the host - the solution is using a proxy:
string retVal = "";
// Can't change the 'Host' header property because .NET protects it
// HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
// request.Headers.Set(HttpRequestHeader.Host, DEPLOYER_HOST);
// so we must use a workaround
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Proxy = new WebProxy(ip);
using (WebResponse response = request.GetResponse())
{
using (TextReader reader = new StreamReader(response.GetResponseStream()))
{
string line;
while ((line = reader.ReadLine()) != null)
retVal += line;
}
}
return retVal;
Host header is set from 'url' automatically by .NET, and 'ip' contains the actual address of the web server you want to contact (you can use a dns name here too)
I know this is old, but I came across this same exact problem, and I found a better solution to this then using sockets or reflection...
What I did was create a new class that durives from WebHeaderCollection and bypasses validation of what you stick inside it:
public class MyHeaderCollection:WebHeaderCollection
{
public new void Set(string name, string value)
{
AddWithoutValidate(name, value);
}
//or
public new string this[string name]
{
get { return base[name]; }
set { AddWithoutValidate(name, value); }
}
}
and here is how you use it:
var http = WebRequest.Create("http://example.com/");
var headers = new MyHeaderCollection();
http.Headers = headers;
//Now you can add/override anything you like without validation:
headers.Set("Host", http.RequestUri.Host);
//or
headers["Host"] = http.RequestUri.Host;
Hope this helps anyone looking for this!
I know this is an old question, but these days, you can do.
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://192.168.1.5/filename.htm");
request.Host = "www.mywebstite.com";
WebResponse response = request.GetResponse();
The "Host" header is protected and cannot be modified programmatically. I suppose to work around this, you could try and bind via reflection to the private "InnerCollection" property of the WebRequest object and calling the "Set" ar "Add" method on it to modify the Host header. I haven't tried this, but from a quick look at the source code in Reflector, I think it's easily accomplished. But yeah, binding to private properties of framework objects is not the best way to accomplish things. :) Use only if you MUST.
edit: Or like the other guy mentions in the linked question, just open up a socket and do a quick "GET" manually. Should be a no brainer, if you don't need to tinker with other stuff, like cookies or whatever else niceties the HttpWebRequest provides.
Alright, little bit of research turns up this:
https://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=384456
Seems MS may do something about this at some point.
You can use my solution for this problem, it posted here :
How to set custom "Host" header in HttpWebRequest?
This can help you to edit host header, and avoid to using proxy and direct socket requests.
Necromancing.
For those still on .NET 2.0
It is in fact quite easy, if you know how.
Problem is, you can't set the host header, because the framework won't let you change the value at runtime. (.net framework 4.0+ will let you override host in a httpwebrequest).
Next attempt will be setting the header with reflection - as demonstrated in the top upvoted answer here - to get around it, which will let you change the header value. But at runtime, it will overwrite this value with the host part of the url, which means reflection will bring you nothing, which is why I don't understand why people keep upvoting this.
If the dns-name doesn't exist, which is quite frankly the only case in which you want to do this in the first place, you can't set it, because .NET can't resolve it, and you can't override the .NET DNS resolver.
But what you can do, is setting a webproxy with the exact same IP as the destination server.
So, if your server IP is 28.14.88.71:
public class myweb : System.Net.WebClient
{
protected override System.Net.WebRequest GetWebRequest(System.Uri address)
{
System.Net.WebRequest request = (System.Net.WebRequest)base.GetWebRequest(address);
//string host = "redmine.nonexistantdomain.com";
//request.Headers.GetType().InvokeMember("ChangeInternal",
// System.Reflection.BindingFlags.NonPublic |
// System.Reflection.BindingFlags.Instance |
// System.Reflection.BindingFlags.InvokeMethod, null,
// request.Headers, new object[] { "Host", host }
//);
//server IP and port
request.Proxy = new System.Net.WebProxy("http://28.14.88.71:80");
// .NET 4.0 only
System.Net.HttpWebRequest foo = (System.Net.HttpWebRequest)request;
//foo.Host = host;
// The below reflection-based operation is not necessary,
// if the server speaks HTTP 1.1 correctly
// and the firewall doesn't interfere
// https://yoursunny.com/t/2009/HttpWebRequest-IP/
System.Reflection.FieldInfo horribleProxyServicePoint = (typeof(System.Net.ServicePoint))
.GetField("m_ProxyServicePoint", System.Reflection.BindingFlags.NonPublic |
System.Reflection.BindingFlags.Instance);
horribleProxyServicePoint.SetValue(foo.ServicePoint, false);
return foo; // or return request; if you don't neet this
}
}
and voila, now
myweb wc = new myweb();
string str = wc.DownloadString("http://redmine.netexistantdomain.com");
and you get the correct page back, if 28.14.88.71 is a webserver with virtual name-based hosting (based on http-host-header).
Now you have the correct answer to the original question, for both WebRequest and WebClient. I think using custom sockets to do this would be the wrong approach, particularly when SSL should be used, and when an actual solution is that simple...