HttpWebRequest is used to download files from ASP.NET site with basic authentication. Everything works fine in many cases, but some proxies make answer chunked and HttpWebRequest.GetResponse() throws an exception if answer with 401 status code is chunked. The exception is:
System.Net.WebException: The server committed a protocol violation.
Section=ResponseStatusLine
at System.Net.HttpWebRequest.GetResponse()
Trace of answer is:<pre>
"HTTP/1.1 401 Authorization Required\r\nDate: Fri, 26 Jun 2009 04:45:18 GMT\r\nServer: Microsoft-IIS/6.0\r\nX-Powered-By: ASP.NET\r\nX-AspNet-Version: 2.0.50727\r\nWWW-Authenticate: Basic realm=\"iis-server\"\r\nCache-Control: private\r\nContent-Type: text/html; charset=iso-8859-1\r\nVia: 1.1 server\r\nKeep-Alive: timeout=15, max=100\r\nConnection: Keep-Alive\r\nTransfer-Encoding: chunked\r\nContent-Language: en\r\n\r\n0\r\n\r\n0\r\n\r\n
I made test and found out that Transfer-Encoding: chunked is the only one reason of exception. Is this bug of .NET Framework 2.0 or there any RFC says that 401 answer shouldn't be chunked?
I actually think the error isn't that Chunked Transfer Encoding (CTE) isn't allowed, I think it's that the server is doing the CTE incorrectly. If you look at the response body, it's looking like this:
Language: en \r\n
\r\n
0 \r\n
\r\n
0 \r\n
\r\n
If you notice, there are two 0 length chunks in the response body. The CTE RFC requires that the last chunk of a message be a 0 length chunk, and, by extension, no previous chunk can be of zero length. In this message, you have 0 zero length chunks which is a protocol violation of CTE.
This is akin to setting a content length header for a message and then transmitting more data than is allowed by that content length header.
Related
TL;DR
If my Content-Type "application/x-ourformat" bound custom TextInputFormatter cannot parse the input, I want to return HTTP 400 (possibly 415) back to the client.
What is the correct way to return from the implementation of ReadRequestBodyAsync to get the framework to return a 4xx HTTP response?
We have a custom formatter for application/x-ourformat built upon the TextInputFormatter as explained in the link.
The example from the MS docs above does error handling as follows:
ReadRequestBodyAsync:
public override async Task<InputFormatterResult> ReadRequestBodyAsync(
InputFormatterContext context, Encoding effectiveEncoding)
{
...
return await InputFormatterResult.SuccessAsync(contact);
}
catch
{
logger.LogError("Read failed: nameLine = {nameLine}", nameLine);
return await InputFormatterResult.FailureAsync();
}
}
That is, if the processing of the input fails with an exception, it will return FailureAsync().
However, this will NOT result in a HTTP 400 response, instead the input object bound to the Api will simply be null.
What does result in a 400 response is:
Throwing an exception out of ReadRequestBodyAsync
Setting context.ModelState.AddModelError("My Key", "Couldn't really understand you."); prior to returning FailureAsync.
But I do neither understand which one is "correct" nor do I really understand what the ModelError is supposed to represent.
I have now dug into the call chain for ReadRequestBodyAsync:
Eventually the result will be processed in BindModelAsync, which:
On HasError, Logs, and returns nothing, stating // Formatter encountered an error. Do not use the model it returned.
On exception, does ModelState.AddModelError(modelBindingKey, exception
So, since the framework itself will set a ModelError on an exception, it sure seems at least that setting the ModelError oneself is more useful.
Like ckuri noted above :
A model error represents that the model ... doesn’t satisfy a
constraint or contains malformed data. Therefore, it kinda applies in
your case.
Below is a sample response. Notice the 200 OK is before the http headers. The HTTP headers have a key and a value separated by a colon. The body is after the headers. the content length is the number of bytes in the body.
HTTP/1.1 200 OK
Date: Mon, 27 Jul 2009 12:28:53 GMT
Server: Apache
Last-Modified: Wed, 22 Jul 2009 19:15:56 GMT
ETag: "34aa387-d-1568eb00"
Accept-Ranges: bytes
Content-Length: 51
Vary: Accept-Encoding
Content-Type: text/plain
Hello World! My content includes a trailing CRLF.
Request:
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
String responseString = new StreamReader(response.GetResponseStream()).ReadToEnd();
Console.WriteLine(responseString);
Response:
{"code":"SUCCESS","details":
{"created_time":"","id":"xxxx"},
"message":"uploaded",
"status":"success"}
HTTP/1.1 200 OK
Date: Wed, 18 Dec 2019 11:42:26 IST
Last-Modified: Wed, 18 Dec 2019 11:42:25 IST
Content-Type: application/json
Connection: Keep-Alive
Server: AWServer
Pragma: no-cache
Cache-Control: no-cache
Expires: 1
Whenever the above-mentioned C# request is executed, the response occasionally contains headers(HTTP/1.1 200 OK...), When I'm only trying to get the body part({"code"....} alone(response.GetResponseStream()). Is this the intended behavior?
Take a look at the basic article on http headers
HTTP headers let the client and the server pass additional information with an HTTP request or response. An HTTP header consists of its case-insensitive name followed by a colon (:), then by its value. Whitespace before the value is ignored.
Headers are additional information. I guess that since you left out the url and the whole creation of the Request and the url, this means that some responses have Headers and some not. That depends on the additional non-body information the api or web server wants to respond with.
It's in the control of the responder and not the receiver.
Don't ignore them: Some times interesting metadata come from Headers. It should not be data but information about it, like encoding, CORS info etc.
last modified header link
date header link
I'am quiet new in Azure, and I'm trying to implement the Put Block List operation according to the documentation.
Here are the steps, which I do to upload my file
Read a file from a local folder
Upload it using Put block operation. it gets succes.
Then I use
Get Block List to see if my blocks are uploaded and in the response I can see
<BlockList><CommittedBlocks /><UncommittedBlocks><Block><Name>MDAwMDAwMDAwMA==</Name><Size>17</Size></Block></UncommittedBlocks></BlockList>
that I have one uncommited block with ID MDAwMDAwMDAwMA==.
And finally I use Put Block List to commit the blob.
Here I get an error code 403 "Forbidden".
My Signature is the following
"PUT\n
\n
\n
110\n
\n
\n
\n
\n
\n
\n
\n
\n
x-ms-date:Tue, 15 May 2018 10:54:08 GMT\nx-ms-version:2017-07-29\n
/storagekaren/dbstore/ddd.txt\ncomp:blocklist"
Here is the uri
https://storagekaren.blob.core.windows.net/dbstore/ddd.txt?comp=blocklist
request content
"<?xml version=\"1.0\" encoding=\"utf-8\"?
>\r\n<BlockList>\r\n<Uncommitted>MDAwMDAwMDAwMA==</Uncommitted>
</BlockList>\r\n"
authorizationHeader is computed using this method
public static String CreateAuthorizationHeader(String canonicalizedString)
{
string signature;
using (var hmacSha256 = new
HMACSHA256(Convert.FromBase64String(STORAGE_ACCOUNT_KEY)))
{
var dataToHmac = Encoding.UTF8.GetBytes(canonicalizedString);
signature =
Convert.ToBase64String(hmacSha256.ComputeHash(dataToHmac));
}
var authorizationHeader = String.Format(
CultureInfo.InvariantCulture,
"{0} {1}:{2}",
"SharedKey",
ACCOUNT_NAME,
signature
);
return authorizationHeader;
}
"SharedKey storagekaren:eIHacFz/PWypTWg6SN/4BOuqlCLVLctABhi6Ay7TYiA="
And this is my HttpClient object to make a request
{System.Net.Http.HttpClient}
BaseAddress: null
DefaultRequestHeaders: {x-ms-date: Tue, 15 May 2018 11:07:51 GMT
x-ms-version: 2017-07-29
Authorization: SharedKey storagekaren:eIHacFz/PWypTWg6SN/4BOuqlCLVLctABhi6Ay7TYiA=
}
MaxResponseContentBufferSize: 2147483647
Timeout: {01:00:00}
Here is the response with error
HTTP/1.1 403 Server failed to authenticate the request. Make sure the value
of Authorization header is formed correctly including the signature.
Content-Length: 686
Content-Type: application/xml
Server: Microsoft-HTTPAPI/2.0
x-ms-request-id: 7a96deee-201e-00fc-78de-ec0ffc000000
x-ms-error-code: AuthenticationFailed
Date: Wed, 16 May 2018 06:22:35 GMT
<?xml version="1.0" encoding="utf-8"?><Error>
<Code>AuthenticationFailed</Code><Message>Server failed to authenticate the
request. Make sure the value of Authorization header is formed correctly
including the signature.
RequestId:7a96deee-201e-00fc-78de-ec0ffc000000
Time:2018-05-16T06:22:36.0842958Z</Message><AuthenticationErrorDetail>The
MAC signature found in the HTTP request
'E9M4w8nHaBbAsgW3Qhf+u5nHipvmxMvLp09AFdaxYZg=' is not the same as any
computed signature. Server used following string to sign: 'PUT
110
text/plain
x-ms-date:Wed, 16 May 2018 06:22:34 GMT
x-ms-version:2017-07-29
/storagekaren/dbstore/ddd.txt
comp:blocklist'.</AuthenticationErrorDetail></Error>
Please help me understand what am I doing wrong?
Essentially the problem is coming because HttpClient is adding text/plain for Content-Type header whereas you're passing an empty string for that. If you change your signature string to include content-type header, I believe you should not get this error. Essentially, your signature string should look like the following:
"PUT\n
\n
\n
110\n
\n
text/plain\n
\n
\n
\n
\n
\n
\n
x-ms-date:Tue, 15 May 2018 10:54:08 GMT\nx-ms-version:2017-07-29\n
/storagekaren/dbstore/ddd.txt\ncomp:blocklist"
I am trying to get a few cURL commands translated to a C# implementation, but I am running into an Exception which I can't solve.
I tried to gather up as much information as I can in the hopes that somebody can help me further. Here it goes...
The cURL statement:
curl -i -X PUT "http://[ipaddress]:[port]/webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=[username]&op=CREATE"
The C# version:
var response = await client.PutAsync(
"http://[ipaddress]:[port]/webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=[username]&op=CREATE",
null);
The C# version results in a WebException:
HResult = -2146233088
Message = The remote name could not be resolved: 'ip-172-31-9-79.eu-central-1.compute.internal'
The server I am connecting to, is a Hadoop server. There are multiple instances running on that server. So when I store a file, the server replies with information on which instance this file is stored (so that I can reference that instance when I want to write to this file).
Based on the error message, it seems to be that it receives some kind of reference to an IP address it can't access (which makes sense, as that ip address is an internal IP address in that Hadoop server.
I used Wireshark to find the difference in the requests which are sent.
Using cURL:
Frame 57: 204 bytes on wire (1632 bits), 204 bytes captured (1632 bits) on interface 0
Ethernet II, Src: IntelCor_da:f4:44 (fc:f8:ae:da:f4:44), Dst: AsustekC_32:7d:b0 (ac:22:0b:32:7d:b0)
Internet Protocol Version 4, Src: 192.168.1.107, Dst: [ipaddress]
Transmission Control Protocol, Src Port: 60454, Dst Port: 50070, Seq: 1, Ack: 1, Len: 150
Hypertext Transfer Protocol
PUT /webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE HTTP/1.1\r\n
[Expert Info (Chat/Sequence): PUT /webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE HTTP/1.1\r\n]
[PUT /webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE HTTP/1.1\r\n]
[Severity level: Chat]
[Group: Sequence]
Request Method: PUT
Request URI: /webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE
Request URI Path: /webhdfs/v1/[appname]/staging/sensors/aap.txt
Request URI Query: user.name=hdfs&op=CREATE
Request URI Query Parameter: user.name=hdfs
Request URI Query Parameter: op=CREATE
Request Version: HTTP/1.1
Host: [ipaddress]:50070\r\n
User-Agent: curl/7.50.0\r\n
Accept: */*\r\n
\r\n
[Full request URI: http://[ipaddress]:50070/webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE]
[HTTP request 1/1]
[Response in frame: 59]
Using the HttpClient:
Frame 381: 209 bytes on wire (1672 bits), 209 bytes captured (1672 bits) on interface 0
Ethernet II, Src: IntelCor_da:f4:44 (fc:f8:ae:da:f4:44), Dst: AsustekC_32:7d:b0 (ac:22:0b:32:7d:b0)
Internet Protocol Version 4, Src: 192.168.1.107, Dst: [ipaddress]
Transmission Control Protocol, Src Port: 60541, Dst Port: 50070, Seq: 1, Ack: 1, Len: 155
Hypertext Transfer Protocol
PUT /webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE HTTP/1.1\r\n
[Expert Info (Chat/Sequence): PUT /webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE HTTP/1.1\r\n]
[PUT /webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE HTTP/1.1\r\n]
[Severity level: Chat]
[Group: Sequence]
Request Method: PUT
Request URI: /webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE
Request URI Path: /webhdfs/v1/[appname]/staging/sensors/aap.txt
Request URI Query: user.name=hdfs&op=CREATE
Request URI Query Parameter: user.name=hdfs
Request URI Query Parameter: op=CREATE
Request Version: HTTP/1.1
Host: [ipaddress]:50070\r\n
Content-Length: 0\r\n
Connection: Keep-Alive\r\n
\r\n
[Full request URI: http://[ipaddress]:50070/webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE]
[HTTP request 1/1]
[Response in frame: 383]
To me, the only notable (but for me meaningless) differences are:
User-Agent: curl/7.50.0\r\n
Accept: /\r\n
vs
Content-Length: 0\r\n
Connection: Keep-Alive\r\n
I have very little experience with REST programming, and clearly not enough knowledge on analyzing web requests. Some help / guidance / explanation would be greatly appreciated.
Just for completeness:
Yes I am aware that there are Windows executables for cURL, but I run on Raspberry PI / Windows IoT. As far as I know there is nothing for that specific platform (yet)
In the above statements I left out ipaddress/appname/etc intentionally just to be safe side wrt security issues
I found out that the only thing which goes wrong is reading the response in C#. The HTTP responses monitored over Wireshark are identical (using cURL.exe and using PutAsync in C#).
I was able to ignore the redirect and be given the opportunity to handle the response myself.
So disabling auto redirect (and handle the response myself) solved my problem:
var httpClientHandler = new HttpClientHandler {AllowAutoRedirect = false};
var client = new HttpClient(httpClientHandler);
In my OwinMiddleware I just write out to the body:
await context.Response.Body.WriteStringAsync("Hey There");
The raw response I get in Fiddler is:
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Server: Microsoft-HTTPAPI/2.0
Date: Fri, 13 May 2016 16:00:07 GMT
A
/hey/there
5
there
9
Hey There
26
{ec86d5c7-4ea4-437f-be2a-4e294b655227}
0
And Fiddler is saying this encoded (with a message saying please click here decode)
The single letters and numbers are not mine. I wrote /hey/there and there and Hey There and the Guid with braces.
How do I just get the plain text response?