When inspecting a third party api using chrome browser's F12 tool I found there are several interesting headers listed:
:authority:m.somedomain.com
:method:GET
:path:/api/somevalues
:scheme:https
Along with some headers I'm familiar with, such as accept, accept-encoding, etc.
I'm using .Net 4.0 to make http/https requests. When trying to add these headers starting with a colon, an error is thrown on the first item:
httpRequest.Headers.Add(":authority", "m.somedomain.com");
httpRequest.Headers.Add(":method", "get");
httpRequest.Headers.Add(":path", sPath);
httpRequest.Headers.Add(":scheme", "https");
Error message:
Specified value has invalid HTTP Header characters.
After some searching I found article talking about http/2. However in .NET 4.0 there are only http/1.0 and http/1.1 available.
Does that mean I need to upgrade to newer .NET version?
Thanks in advance.
Possibly. http/2 is supported from .NET version 4.6.2.
Click on the Project tab and select properties at the bottom. And then change your .NET version to 4.6.2.
Also I am 99.9% sure that you shouldn't include semi colon in your headers.
Further:
httpRequest.Headers.Add(":method", "get");
The request-method should not be defined in the header. Do it like so.
httpRequest.Method = "GET";
The scheme is usually indicated by the prefix of your url. ie:
string webAddr = "https://www.google.com/";
HttpWebRequest httpWebRequest = (HttpWebRequest)WebRequest.Create(webAddr);
Related
I'm trying to get a HttpClient to pull favicons from websites, and for 99% of cases this code is working as expected:
var httpClient = new HttpClient();
httpClient.Timeout = TimeSpan.FromSeconds(10);
var response = await httpClient.GetAsync("https://www.tesco.com/favicon.ico");
response.EnsureSuccessStatusCode();
I've been finding with a couple of websites that my GetAsync method is just timing out, and I believe it is to do with it's redirect. If you run the above code in a console app, the following exception is thrown inside a TaskCanceledException and an IOException after 10 seconds:
SocketException: The I/O operation has been aborted because of either a thread exit or an application request.
This request works completely fine on Postman, and I've tried https and http without success. When visiting the site and using Chrome's dev tools, it looks like if you use http rather than https, it returns a 307 and redirects you to the https site, which then returns a 304. I don't understand why the HttpClient is just timing out rather than giving a useful response.
Is there any way to get this to work? Am I missing something simple?
Update - 2021-01-29
I've tried this on multiple different .NET versions, and found that this could be a bug with HttpClient, as this code works for .NET Core 2.0 but not for .NET Core 2.1.
Versions Tested
.NET Framework 4.8: Works
.NET Core 1.0: Works
.NET Core 1.1: Works
.NET Core 2.0: Works
.NET Core 2.1: Does not work
.NET Core 2.2: Does not work
.NET Core 3.0: Does not work
.NET Core 3.1: Does not work
.NET 5: Does not work
It turns out this is not a .NET issue at all. I raised this issue on GitHub as all evidence was pointing towards the HttpClientHandler breaking after .NET Core 2.1. Stephen was able to point out to me that before .NET Core 2.1, a Connection: Keep-Alive header is set by default in the HttpClient.
The culprit URLs that I have been testing against seem to require one of the following things to work:
A Connection: Keep-Alive header
An Accept: */* header, or something more specific to what is being requested
The request to be sent using the HTTP/2 protocol.
For reference, to apply one of the Header fixes, use httpClient.DefaultRequestHeaders.Add(string, string), and to set the HTTP protocol, use httpClient.DefaultRequestVersion = HttpVersion.Version20;
This issue turns out to not be a direct issue with C#, but that the default headers that are sent changed after .NET Core 2.1. I found the issue to be reproducible on Postman if you disable all headers (including the Postman token header).
I have to be able to send a HTTP request for a response like this:
HTTP/1.0 200 OK
/bin/sh: check_updatelock: not found
Cache-Control: no-cache, must-revalidate
Expires: ...
Content-Type: application/json
{"response": "..."}
Now here /bin/sh causes exception: Received an invalid header name: '/bin/sh' (at System.Net.Http.HttpConnection.ParseHeaderNameValue).
I've tried with HttpClient, WebClient, HttpWebRequest, and even with a 3rd party lib named Flurl, but that uses HttpClient under the hood.
The result is always the same expection.
Even in PowerShell, with Invoke-WebRequest, there is an execption: The server committed a protocol violation. Section=ResponseHeader Detail=Header name is invalid
How could I read such a response from a C# .NET Core console app?
(The server is out of my control)
Any idea highly appreciated!
Separator chars, like '/', are not part of the field name in the header. So HttpClient (and other libraries built on top of it) will throw an error.
You can try other third party libraries, which are not based on HttpClient like RestSharp (I'm not sure whether it supports non-standard header, but its not based on HttpClient)
HttpRequest request = new HttpRequest();
request.AddHeader("Host", "api.dwebsir.com");
request.Proxy = ProxyClient.Parse(ProxyType.Http, proxy);
this way of adding custom host header doesnt seems to work.
When i ignore the the proxy and while posting without headers also seems to work.
But when i postdata with proxy and request.Header it is not taking and i am not getting any response
But when i postdata with proxy and i am receiving this The requested URL /as/token.oauth2 was not found on this server.
i am pretty sure the error is with Host as headers i am not getting the proper fix
There is a way to do this, as Explanation given here:
http://blogs.msdn.com/feroze_daud/archive/2005/03/31/404328.aspx
next version of the framework (.NET Framework 4.0) will make it easier.
http://blogs.msdn.com/ncl/archive/2009/07/20/new-ncl-features-in-net-4-0-beta-2.aspx
Hope this helps you.
I'm using an embedded Nancy webserver and I've used Nu-Get to install the proper plugins for the built in protocol buffer serialization (Nancy.Serialization.ProtoBuf and protobuf-net).
However, the mere presence of these packages doesn't seem to be enabling protocol buffers for me. I'm sending a request with an Accepts header of application/x-protobuf. Using the Nancy diagnostic page (built-in) I can trace my request and see:
response Headers
status Code
406
log
New Request Started
[DefaultResponseNegotiator] Processing as negotiation
[DefaultResponseNegotiator] Original accept header: application/x-protobuf;q=1
[DefaultResponseNegotiator] Coerced accept header: application/x-protobuf
[DefaultResponseNegotiator] Acceptable media ranges: */*
[DefaultResponseNegotiator] Unable to negotiate response - no headers compatible
So it appears as though it can't handle protocol buffers. When I check the main page of the Diagnostics site:
response Processors
Nancy.Responses.Negotiation.JsonProcessor
Nancy.Responses.Negotiation.ResponseProcessor
Nancy.Responses.Negotiation.ViewProcessor
Nancy.Responses.Negotiation.XmlProcessor
It appears as though only the default response processors are there. Protobuf isn't listed.
Any pointers on how I can get the plugin to register with Nancy?
When i'm trying to edit the Host key on Request.Headers under console application i'm getting exception that says:
The 'Host' header cannot be modified directly.
Parameter name: name
So how can i change it?
As you've seen the .Net Fx does not allow to edit the host header, but since .Net Fx 4.0 there is a seperate 'Host' definition on the HttpWebRequest object. You can use it like this:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://127.0.0.1/");
request.Host = "yourdomain.com";
Hope that helps you out.