I'm currently trying to program my own HttpWebRequest class. I've already written the code that sends the header and the body of the request to the server and awaits a response. However, I am unsure which charset I should use for the header.
I've also been wondering what would be a good way of processing the response (header + body). Should I try to decode all received data into a string, or should I do it differently? I was thinking of splitting the header from the body using the two line feeds/carriage returns that separate those two parts. Then I would be able to decode the header and leave the body for later, when I know its charset.
So my questions in short:
What charset does HTTP use for its headers?
What's a good way of processing the response?
First I would recommend that you become intimately familiar with RFC-2616 which is the RFC for the HTTP 1.1 protocol.
From the above RFC you will find the following statement
The TEXT rule is only used for descriptive field contents and values
that are not intended to be interpreted by the message parser. Words
of *TEXT MAY contain characters from character sets other than ISO-
8859-1 [22] only when encoded according to the rules of RFC 2047
[14].
The headers should use ISO-8859-1 encoding unless encoded using the MIME encoding outlined in RFC-2047.
As for the parsing of the response, that really depends on the message. Personally, I would processes the response based on the BNF defined for HTTP as I identify tokens that I recognize I would update the state of the parser to process the rest of the response accordingly. For example as the response data is processed you might find that the response is a JPG image and the the content length is X, so you can setup the appropriate memory stream to read the content into and then create an Image etc.
Related
The server on which I'm trying to send POST is probably written in PHP, and I'm using VB.Net - HttpWebRequest as client.
Trying to encode message to send post but it doesn't work.
I decided to caught the exact encoding characters which the server uses through Fiddler application, by trying to send data via a regular browser, and I realize that it's slightly different with .Net WebUtility.UrlEncode & EscapeUriString & "Uri.EscapeDataString" characters.
I also used "HtmlEncode" and other .net encoding methods without success.
Now, I'd like to know what url encoding to use in order to get below characters.
1. Message to Send POST
MR CHEAP TRANSPORT - 50 YEARS EXCELLENCE
WE ARE SERIOUS ON SERVICE AND AFFORDABILITY
HOME AND OFFICE MOVES
2. POST Characters from Regular Browser
%3Cp%3EMR+CHEAP+TRANSPORT+-+20+YEARS+EXCELLENCE%3C%2Fp%3E%3Cp%3EWE+ARE+SERIOUS+ABOUT+SERVICE+AND+AFFORDABILITY%26nbsp%3B%3C%2Fp%3E%3Cp%3E%3Cbr%3E%3C%2Fp%3E%3Cp%3EHOME+AND+OFFICE+MOVES%3C%2Fp%3E%3Cp%3E
3. POST Characters from Regular Browser
MR+CHEAP+TRANSPORT+-+30+YEARS+EXCELLENCE%0AWE+ARE+SERIOUS+ABOUT+SERVICE+AND+AFFORDABILITY+%0A%0AHOME+AND+OFFICE+MOVES%0A
I'm rather new to sending/receiving over networks/sockets/network streams and so on.
I'm making an IRC program that can communicate with Twitch.tv. They have an API, and they have examples of all sorts of requests you would use to get different kinds of information.
https://github.com/justintv/Twitch-API/tree/master/v3_resources
One example of their requests is this:
curl -H 'Accept: application/vnd.twitchtv.v3+json' \
-X GET https://api.twitch.tv/kraken/chat/kraken_test_user
I have tried to do some research on requests, and I sort of understand some, but for the most part I could not find any resources that help make it click for me.
In the above example, what are the important parts of that request? curl? -H? Is that one big command, or is it two commands separated by the \ at the end of the first line?
Then, the biggest question, how to send requests like the one above using C#?
EDIT 1:
I also know that I will be getting responses in JSON. Is there anything built in that assists with receiving/parsing JSON?
And also using PUT to change some JSON? (some things in the API allow PUT).
For the first bit of the question, you asked what are the important parts
It has an accept header of application/vnd.twitchtv.v3+json
It is a GET request
The api url: https://api.twitch.tv/kraken/chat/kraken_test_user
This request in c# could look like the following (could because there is more than one way to do it)
private async Task<object> GetRequest(string url)
{
var httpClient = new HttpClient();
httpClient.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/vnd.twitchtv.v3+json"));
var response = await httpClient.GetAsync(url);
var contents = await response.Content.ReadAsStringAsync();
return contents;
}
Note that the files in the link you posted are to Mark Down files that Google describes as:
MD, or markdown document is a text file created using one of several possible dialects of the Markdown language. MD files use plain text formatting but includes inline text symbols that define how to format the text, and is designed for authoring plain text documentation that can be easily converted to HTML.
curl -H 'Accept: application/vnd.twitchtv.v3+json' \
-X GET https://api.twitch.tv/kraken/chat/kraken_test_user
http://curl.haxx.se/docs/manpage.html explains what the curl command is that then has 2 switches, H and X. where quoting the link:
-H, --header
(HTTP) Extra header to include in the request when sending HTTP to a
server. You may specify any number of extra headers. Note that if you
should add a custom header that has the same name as one of the
internal ones curl would use, your externally set header will be used
instead of the internal one. This allows you to make even trickier
stuff than curl would normally do. You should not replace internally
set headers without knowing perfectly well what you're doing. Remove
an internal header by giving a replacement without content on the
right side of the colon, as in: -H "Host:". If you send the custom
header with no-value then its header must be terminated with a
semicolon, such as -H "X-Custom-Header;" to send "X-Custom-Header:".
curl will make sure that each header you add/replace is sent with the
proper end-of-line marker, you should thus not add that as a part of
the header content: do not add newlines or carriage returns, they will
only mess things up for you.
See also the -A, --user-agent and -e, --referer options.
Starting in 7.37.0, you need --proxy-header to send custom headers
intended for a proxy.
Example:
# curl -H "X-First-Name: Joe" http://192.168.0.1/
WARNING: headers set with this option will be set in all requests -
even after redirects are followed, like when told with -L, --location.
This can lead to the header being sent to other hosts than the
original host, so sensitive headers should be used with caution
combined with following redirects.
This option can be used multiple times to add/replace/remove multiple
headers.
The "\" makes the next line be added to the first line.
-X, --request
(HTTP) Specifies a custom request method to use when communicating
with the HTTP server. The specified request method will be used
instead of the method otherwise used (which defaults to GET). Read the
HTTP 1.1 specification for details and explanations. Common additional
HTTP requests include PUT and DELETE, but related technologies like
WebDAV offers PROPFIND, COPY, MOVE and more.
Normally you don't need this option. All sorts of GET, HEAD, POST and
PUT requests are rather invoked by using dedicated command line
options.
This option only changes the actual word used in the HTTP request, it
does not alter the way curl behaves. So for example if you want to
make a proper HEAD request, using -X HEAD will not suffice. You need
to use the -I, --head option.
The method string you set with -X will be used for all requests, which
if you for example use -L, --location may cause unintended
side-effects when curl doesn't change request method according to the
HTTP 30x response codes - and similar.
(FTP) Specifies a custom FTP command to use instead of LIST when doing
file lists with FTP.
(POP3) Specifies a custom POP3 command to use instead of LIST or RETR.
(Added in 7.26.0)
(IMAP) Specifies a custom IMAP command to use instead of LIST. (Added
in 7.30.0)
(SMTP) Specifies a custom SMTP command to use instead of HELP or VRFY.
(Added in 7.34.0)
If this option is used several times, the last one will be used.
In C#, there is a WebRequest class that https://msdn.microsoft.com/en-CA/library/456dfw4f(v=vs.110).aspx has a good example of how to use to get data from a given URL.
As for handling JSON, please look into http://www.newtonsoft.com/json which is a rather common library used for parsing JSON responses. PUT would be the HTTP verb like GET or POST used to tell the server how to process a request. I'd suggest in the future be careful about posting a rather broad set of questions here as I could see this being something that a class could spend an hour covering somewhere that I doubt your intention is getting someone else to do your homework, right?
I'm currently working with a number of third parties, sending them data via an HttpPost with an encrypted url string. Each of the third parties sends back an xml string which I then read and use. All is good!
The question I have is this. If I send a request to a third party, and they do not provide a response, what happens? To clarify, I mean that a response string isn't returned in all cases - not just errors and timeouts.
Few things can happen:
A response is sent by the third party: It consists of the desired output (an XML string in your case.
A response is sent, but the content is 0 length: Take it into mind that HTTP response consists of two parts: HTTP headers, and content. In this case third party sends only the headers. You can consider this as String.Empty output.
A response is not sent: Third part should have sent a response according to the HTTP protocol, but they don't. Third party may close the socket or just timeout. That can happen because of an internal problem, or they are just choosing not to respond to you. This will normally result in an exception. What the exception is, though, depends on the methods/library you use to send HTTP requests.
The HTTP protocol does not include a way to not send a response. A response is always sent. It might be zero length, though.
I mean that a response string isn't returned in all cases
Is this a question or an assertion? This statement is false.
and they do not provide a response, what happens?
So what happens in case of a zero-length response? That depends entirely on your code. Neither protocol not framework care about this case.
Couldn't the socket actually be closed before any headers are sent back? Or, if the server is down?
That's a protocol violation. That's not HTTP, then. In such a situation the app is generally notified through an exception.
I am trying to extract the multipart attachment portion of a SOAP response.
I have a project that uses a Web Reference to talk to a 3rd party web service. I'm able to successfully make requests and get valid responses back from the service, but I don't understand where if anywhere the attachment data is going. Looking through the Web Reference autogenerated code there aren't any objects that match up to the actual attachment data.
Are there any kind options that I need to set when originally consuming the 3rd party WSDL to make this work correctly?
I had hoped there would be a .NET multipart boundary parser already, but wasn't able to find one so I wrote my own.
I have done this by getting the raw HTTP response as a string, splitting on the part boundaries ------=_Part_*. Then for each part looking for Content-Transfer-Encoding: base64, and then extracting the base 64 region and using Convert.FromBase64String to convert that to a byte array, and finally saving that as a file.
I want to stream data from an IHttpHandler class. I'm loading a large number of rows from the DB, serializing, and compressing them, then sending them down the wire. On the other end, I want my client to be able decompress, and deserialize the data before the server is even done serializing all the objects.
I'm using context.Response.OutputSteam.Write to write my data, but it still seems like the output data is being put into a buffer before being sent to the client. Is there a way to avoid this buffering?
The Response.Flush method should send it down the wire; however, there are some exceptions. If IIS is using Dynamic Compression, that is it's configured to compress dynamic content, then IIS will not flush the stream. Then there is the whole 'chunked' transfer encoding. If you have not specified Content-Length then the recieving end does not know how large the response body will be. This is accomplished with the chunked transfer encoding. Some HTTP servers require that the client uses an Accept-Encoding request header containing the chunked keyword. Others just default to chunked when you begin writing bytes before the full length is specified; however, they do not do this if you have specified your own Transfer-Encoding response header.
With IIS 7 and compression disabled, Response.Flush should then always do the trick, right? Not really. IIS 7 can have many modules that intercept and interact with the request and response. I don't know if any that are installed/enabled by default, but you should still be aware that they can effect your desired result.
... I'm loading a large number of rows from the DB, serializing, and compressing them, then sending them down the wire...
Curious that you are compressing this content. If you are using GZIP then you will not be in control of when and how much data is sent by calling flush. Additionally using GZIP content means that the receiving end may also be unable to start reading data right away.
You may want to break the records into smaller, digestible chucks of 10, 50, or 100 rows. Compress that and send it, then work on the next set of rows. Of course now you will need to write something to the client so they know how big each compressed set of rows is, and when they have reached the end. see http://en.wikipedia.org/wiki/Chunked_transfer_encoding for an example of how the chunked transfer works.
You can use context.Response.Flush() or context.Response.OutputSteam.Flush() to force buffered content to be written immediately.