I am creating a Http request parser. I am getting data in chunks (byte array) and parsing it simultaneously. I want to know the condition of detecting end of http request. The request may or may not contain message body.
Thanks
Three different ways:
content-length header (number of bytes following the headers)
chunked encoding (content length unknown at start of request, chunked encoding will indicate when the end is reached)
connection closed by server (http "0.9")
Related
I am using HttpWebRequest with the AddRange function like:
HttpWebRequest myHttpWebRequest = (HttpWebRequest)WebRequest.Create(URL);
myHttpWebRequest.AddRange(20, 30);
HttpWebResponse myHttpWebResponse = (HttpWebResponse)myHttpWebRequest.GetResponse();
Stream streamResponse = myHttpWebResponse.GetResponseStream();
SaveFileStream(name, streamResponse); //save file function
...but the entire file downloaded.
AddRange() in the above code expects the bytes between 20 and 30 to be downloaded (in other words, to download those 10 bytes from the file).
But my code is not working, since the download is not segmented. This link provides an example: http://stackoverflow.com/robots.txt That file was downloaded in its entirety. Why?
HTTP server are not required to support Range header requests. You can verify server's range support by issuing HEAD request and checking value of Accept-Ranges header in response (see HTTP range requests). But this still doesn't guarantee that server will not ignore Range header, in particular for very small ranges (it would be very inefficient for HTTP server to serve content in such small segments).
From RFC7233:
Because servers are free to ignore Range, many implementations will
simply respond with the entire selected representation in a 200 (OK)
response. That is partly because most clients are prepared to receive
a 200 (OK) to complete the task (albeit less efficiently) and partly
because clients might not stop making an invalid partial request until
they have received a complete representation. Thus, clients cannot
depend on receiving a 416 (Range Not Satisfiable) response even when
it is most appropriate.
To determine if server accepted or ignored Range header, you must check response status code. 200 (OK) indicates that server ignored Range header and returned whole response body, 206 (Partial Content) indicates that range specified in header was returned by server and 416 (Range Not Satisfiable) indicates that the set of ranges requested has been rejected due to invalid ranges or an excessive request of small or overlapping ranges.
In case of http://stackoverflow.com/robots.txt, server indicates support of Range header by returning Accept-Ranges: bytes header in response on HEAD request, but on GET request with AddRange(20, 30) specified, response is 200 (OK), so server just ignored such small range a returned whole reponse body. You have to cut requested range from response yourself, if you need to.
I have a small IHttpModule that's reading a POST request from another server and relaying it on. The response from the remote server has the header
Content-Encoding: gzip
How do i specify this in the HttpResponse i'm returning to the caller? Content-Encoding is defined as a text encoding type, so it's expecting a text encoding such as UTF8.
context.Response.ContentEncoding = ???;
Should i be ignoring this and manually setting the header?
If you modifying response, then you should, decode and read the content, gzip retrieved value and add header in response.
//Code for gzip the content and add header
context.Response.Filter = new System.IO.Compression.GZipStream(
context.Response.Filter,
System.IO.Compression.CompressionMode.Compress);
context.Response.AppendHeader("Content-Encoding", "gzip");
If relaying the response without any change, then no need to do any thing.
I'm writing a Windows UWP program using Windows.Web.Http namespace. I stumbled upon a MSDN posting that mentions the below code will force chunked encoding. It does indeed do so. I've made some attempts at changing chunk size (http request) but the size is stuck at 0x10000.
Is there a way to change the chunk size?
What is the rationale for making GetInputStreamAt(0) trigger chunking? Seems like a weird, not-documented-anywhere side-effect. Is there some logic to it?
Here's the MSDN posting:
HttpStreamContent won't save stream
Windows.Storage.Streams.InMemoryRandomAccessStream contentStream = new Windows.Storage.Streams.InMemoryRandomAccessStream();
Windows.Storage.Streams.DataWriter dw = new Windows.Storage.Streams.DataWriter(contentStream);
dw.WriteBytes(System.Text.Encoding.UTF8.GetBytes(body));
dw.StoreAsync().AsTask();
content = new Windows.Web.Http.HttpStreamContent(contentStream.GetInputStreamAt(0));
content.Headers.Add("Content-Type", "application/json; charset=utf-8");
MSFT Guy Says:
... the difference between the contentStream.Seek(0) and contentStream.GetInputStreamAt(0) is: using the first approach actually sends the Content-Length HTTP header followed by the entity body, whereas the second approach uses the Transfer-Encoding: chunked HTTP header, followed by the HTTP chunks. This is helpful in scenarios where the target server only accepts a "specific" format of the HTTP request.
I'm trying to download first bytes of a webpage.
I add Range to the HTTP request header. it's my code in C#:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://example.com");
request.AddRange(0,1000);
//request.Proxy = null;
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
Stream st = response.GetResponseStream();
StreamReader sr = new StreamReader(st);
string str = sr.ReadToEnd();
sr.Close();
st.Close();
}
for some webpages it works fine, but some servers will ignore the HTTP range header, so the server send all the page in the response.
I changed my code to this:
string str = sr.Read(buffer,0,999);
but it doesn't work! because the problem is not in this line. actually the response will send to my program when I call request.GetResponse()! in this line all the bytes received by the program and write the all bytes to the RAM. I want to stop receiving data from the server when I received first 1000 bytes.
but there is no control on HttpWebRequest class and .GetResponse() method to stop receiving data after 1000 bytes received.
How can I do that?
I think there would be another HTTP Request custom class that allow us to stop receiving data when we want.
please tell me any Idea about this problem. I'm thinking to override the HttpWebRequest or write a MFC Library (C++ language) and import it into my project, but I don't know how to do this.
EDIT: I know it's optional for server to allow or ignore the Range header! but my question is how can I stop receiving data from the server! for example the server is sending 10,000 bytes to my computer, I want to stop receiving bytes after I see the 1000th byte! (I don't want the server to send just first 1000 bytes, I want to close connection and stop receiving the bytes after first 1000 bytes! even if the server send all 10,000 bytes)
i am trying to get the content of http request...my program is using threads and sockets to comunicate to a access terminal..
i ne
this is the request sent from the terminal:
POST /iclock/devicecmd?SN=2182682370001 HTTP/1.1 and the content is
ID1&Return=0&CMD=INFO..
and this is my function to get he full content of http request:
private string GetPedido(NetworkStream stream)
{
string sPedido = "" ;
Byte[] bytesFromStream = new Byte[GlobalFunctionAndVariables.iStreamBufferSize];
while (_tcpClient.Available>0)
{
stream.Read(bytesFromStream, 0, bytesFromStream.Length);
//Console.Write("available: {0}\n", _tcpClient.Available);
sPedido += System.Text.Encoding.ASCII.GetString(bytesFromStream, 0, bytesFromStream.Length);
}
Console.WriteLine("Terminou, a enviar resultado \n");
}
the buffer for now is 32 bytes.
for the moment i am only getting the http header and not the content.
if i want to return the content, do i need to parse the string byte by byte..then find the value of content-length and ask to fecth x more bytes.
is this my only option?
Thanks in advance
The Available property does not what you think it does. Remove its use.
Also, you are not using the return value of Read. TCP offers you a stream of bytes. A Read operation can return any amount starting with one byte. You code must handle that case.
You should probably continue reading until the remote side is done sending. You can find out about that by checking the amount of bytes read against zero.