I need to get content length in order to tell my app where the buffer ends. The problem is that httpwebresponse.ContentLength returns -1 even though Content-Length header is presented in response.
Then I though I'm going to read the actual header to find out the length. The Content-Length returned by the page I'm testing on is 1646. An HTTP sniffer claims that I received 1900 bytes, so I assume the difference are the header length. Then I copied the whole body from response and pasted it into online strlen site and the body size is actually 1850!!
How is this possible? Why does response return invalid content-length and why does httpwebrequest.ContentLength returns -1? How can I calculate the actual response length before receiving the response itself?
EDIT:
This is the code I'm using to get the response:
using (System.IO.Stream responseStream = hwresponse.GetResponseStream())
{
using (MemoryStream memoryStream = new MemoryStream())
{
int count = 0;
do
{
count = responseStream.Read(buffer, 0, buffer.Length);
TCP_R.SendBytes(buffer);
} while (count != 0);
}
}
byte[] PACKET_END_IDENTIFIER = { 0x8, 0x01, 0x8, 0x1, 0x8 };
TCP_R.SendBytes(PACKET_END_IDENTIFIER);
TCP_R.Close();
I have a proxy server application that takes a request, sends it to another application (my client) client executes the request and using TCP_R class returns the result. When server gets response from client, it returns response back to browser.
Each time I do a request, I get all the data + extra garbage, here's an example:
<tag1><tag2><tag3> ag3>
ag3> is the garbage data, it's like the ending of buffer is cut off and added again. It apprears that the client responds with a valid response, the garbage data is added onDataRecieve event.. any tips? thanks!
-1 isn't an invalid value of the ContentLength property. I assume you mean the ContentLength property of the response is -1... asking the request what the length is would be non-sensical. Even so, it's perfectly valid:
The ContentLength property contains the value of the Content-Length header returned with the response. If the Content-Length header is not set in the response, ContentLength is set to the value -1.
If the body length is 1850, that suggests it's using chunked transfer encoding. But that should be transparent to you - just keep reading from the response stream until the end. If you're using .NET 4, it's dead easy - just create a MemoryStream and use Stream.CopyTo to copy the data to that MemoryStream.
Related
I am using HttpWebRequest with the AddRange function like:
HttpWebRequest myHttpWebRequest = (HttpWebRequest)WebRequest.Create(URL);
myHttpWebRequest.AddRange(20, 30);
HttpWebResponse myHttpWebResponse = (HttpWebResponse)myHttpWebRequest.GetResponse();
Stream streamResponse = myHttpWebResponse.GetResponseStream();
SaveFileStream(name, streamResponse); //save file function
...but the entire file downloaded.
AddRange() in the above code expects the bytes between 20 and 30 to be downloaded (in other words, to download those 10 bytes from the file).
But my code is not working, since the download is not segmented. This link provides an example: http://stackoverflow.com/robots.txt That file was downloaded in its entirety. Why?
HTTP server are not required to support Range header requests. You can verify server's range support by issuing HEAD request and checking value of Accept-Ranges header in response (see HTTP range requests). But this still doesn't guarantee that server will not ignore Range header, in particular for very small ranges (it would be very inefficient for HTTP server to serve content in such small segments).
From RFC7233:
Because servers are free to ignore Range, many implementations will
simply respond with the entire selected representation in a 200 (OK)
response. That is partly because most clients are prepared to receive
a 200 (OK) to complete the task (albeit less efficiently) and partly
because clients might not stop making an invalid partial request until
they have received a complete representation. Thus, clients cannot
depend on receiving a 416 (Range Not Satisfiable) response even when
it is most appropriate.
To determine if server accepted or ignored Range header, you must check response status code. 200 (OK) indicates that server ignored Range header and returned whole response body, 206 (Partial Content) indicates that range specified in header was returned by server and 416 (Range Not Satisfiable) indicates that the set of ranges requested has been rejected due to invalid ranges or an excessive request of small or overlapping ranges.
In case of http://stackoverflow.com/robots.txt, server indicates support of Range header by returning Accept-Ranges: bytes header in response on HEAD request, but on GET request with AddRange(20, 30) specified, response is 200 (OK), so server just ignored such small range a returned whole reponse body. You have to cut requested range from response yourself, if you need to.
I'm trying to download first bytes of a webpage.
I add Range to the HTTP request header. it's my code in C#:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://example.com");
request.AddRange(0,1000);
//request.Proxy = null;
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
Stream st = response.GetResponseStream();
StreamReader sr = new StreamReader(st);
string str = sr.ReadToEnd();
sr.Close();
st.Close();
}
for some webpages it works fine, but some servers will ignore the HTTP range header, so the server send all the page in the response.
I changed my code to this:
string str = sr.Read(buffer,0,999);
but it doesn't work! because the problem is not in this line. actually the response will send to my program when I call request.GetResponse()! in this line all the bytes received by the program and write the all bytes to the RAM. I want to stop receiving data from the server when I received first 1000 bytes.
but there is no control on HttpWebRequest class and .GetResponse() method to stop receiving data after 1000 bytes received.
How can I do that?
I think there would be another HTTP Request custom class that allow us to stop receiving data when we want.
please tell me any Idea about this problem. I'm thinking to override the HttpWebRequest or write a MFC Library (C++ language) and import it into my project, but I don't know how to do this.
EDIT: I know it's optional for server to allow or ignore the Range header! but my question is how can I stop receiving data from the server! for example the server is sending 10,000 bytes to my computer, I want to stop receiving bytes after I see the 1000th byte! (I don't want the server to send just first 1000 bytes, I want to close connection and stop receiving the bytes after first 1000 bytes! even if the server send all 10,000 bytes)
i am trying to get the content of http request...my program is using threads and sockets to comunicate to a access terminal..
i ne
this is the request sent from the terminal:
POST /iclock/devicecmd?SN=2182682370001 HTTP/1.1 and the content is
ID1&Return=0&CMD=INFO..
and this is my function to get he full content of http request:
private string GetPedido(NetworkStream stream)
{
string sPedido = "" ;
Byte[] bytesFromStream = new Byte[GlobalFunctionAndVariables.iStreamBufferSize];
while (_tcpClient.Available>0)
{
stream.Read(bytesFromStream, 0, bytesFromStream.Length);
//Console.Write("available: {0}\n", _tcpClient.Available);
sPedido += System.Text.Encoding.ASCII.GetString(bytesFromStream, 0, bytesFromStream.Length);
}
Console.WriteLine("Terminou, a enviar resultado \n");
}
the buffer for now is 32 bytes.
for the moment i am only getting the http header and not the content.
if i want to return the content, do i need to parse the string byte by byte..then find the value of content-length and ask to fecth x more bytes.
is this my only option?
Thanks in advance
The Available property does not what you think it does. Remove its use.
Also, you are not using the return value of Read. TCP offers you a stream of bytes. A Read operation can return any amount starting with one byte. You code must handle that case.
You should probably continue reading until the remote side is done sending. You can find out about that by checking the amount of bytes read against zero.
First, check this message which I was trying to do:
Login
To log on to Windows8 service is through the URL: http://app.proceso.com.mx/win8/login
This URL HTTP Request Method receives POST variables user and pass. The variable user is the user's email and pass the variable is the same password. In the event that the user or password are invalid return plain text number zero 0, in the opposite case, that the username and password are valid return plain text an alphanumeric string of 32 characters, as this b17f27a16589fee247c666da6ed15569, this string is the hash of the valid user valid and will run from 00:00 hours to 23:59 hours the day it was generated.
To test the URL was created: http://app.proceso.com.mx/win8/login_test
Note: It should be clear that the hash generated will only be valid for Windows8 service to the user that gender and the effect from 00:00 hours to 23:59 on the day it was generated.
Note: All services generate text in UTF-8
Here is a test account:
User: javier.lopez.contreras10#gmail.com
Pass: policarpio20
So, if you set the data in this page: http://app.proceso.com.mx/win8/login_test you will receive a hash code.
And that's what I'm trying to accomplish in a metro application, but I feel lost in the situation. I have no idea to send those data to receive the hash code. I was using HttpClient and HttpContent but I'm not sure.
Thanks in advance.
UPDATE: Thanks to dharnitski for the code, right now I'm modifying this code for Win8 CP:
// this is what we are sending
string post_data = "user=javier.lopez.contreras10#gmail.com&pass=policarpio20";
// this is where we will send it
string uri = "http://app.proceso.com.mx/win8/login";
// create a request
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);
request.Method = "POST";
// turn our request string into a byte stream
byte[] postBytes = Encoding.UTF8.GetBytes(post_data);
// this is important - make sure you specify type this way
request.ContentType = "application/x-www-form-urlencoded";
Stream requestStream = await request.GetRequestStreamAsync();
// now send it
requestStream.Write(postBytes, 0, postBytes.Length);
// grab te response and print it out to the console along with the status code
WebResponse response = await request.GetResponseAsync();
//var a = new StreamReader(response.GetResponseStream()).ReadToEnd();
StreamReader requestReader = new StreamReader(response.GetResponseStream());
String webResponse = requestReader.ReadToEnd();
And I realized, HttpWebRequest does not contain ProtocolVersion and is throwing me this error in this line:
WebResponse response = await request.GetResponseAsync();
// ERROR: The remote server returned an error: (417) Expectation Failed.
How can I solve this problem if I can modify protocol version?
This is an sample code to implement HTTP POST in C#
http://www.terminally-incoherent.com/blog/2008/05/05/send-a-https-post-request-with-c/
IMPORTANT: You must switch your web page to HTTPS (SSL). It is very bad practice to send not encrypted passwords.
I have the following code that uploads the file to the server by making the POST request:
string fileName = #"C:\Console_sk.txt";
HttpWebRequest request = (HttpWebRequest) HttpWebRequest.Create("http://" + Environment.MachineName + ":8000/Upload");
request.Method = "POST";
request.ContentType = "text/plain";
request.AllowWriteStreamBuffering = false;
Stream fileStream = new FileStream(fileName, FileMode.Open);
request.ContentLength = fileStream.Length;
Stream serverStream = request.GetRequestStream();
byte[] buffer = new byte[4096];
while (true)
{
int bytesRead = fileStream.Read(buffer, 0, buffer.Length);
if (bytesRead > 0)
{
serverStream.Write(buffer, 0, bytesRead);
}
else
{
break;
}
}
serverStream.Close();
fileStream.Close();
request.GetResponse();
This will be used to upload potentially large files, therefore it is essential for us to track the progress of this upload.
Based on various sources, I've thought that my code will upload file in 4096 byte chunks i.e. multiple POST requests will be made. But when tracking requests by using Fiddler, it shows that only one POST request is made. On other hand, when I enabled .NET network tracing, the resulting log showed that 4096 byte chunks are indeed written to request stream and the socket is always being opened for each write operation.
My understanding of networking in general is quite poor, so I don't quite understand how this really works. When calling serverStream.Write, is the provided chunk really sent over the network, or is it just buffered somewhere? And when serverStream.Close is called, can I be sure that the whole file has been uploaded?
The HttpWebRequest class sends a single HTTP request. What you are doing is that you are writing to the request in chunks to avoid loading the whole file in memory. You are reading from the file and directly writing to the request stream in chunks of 4KB.
When calling serverStream.Write, is the provided chunk really sent over the network
Yes, it is written to the network socket.