TL;DR version
When a transfer error occurs while writing to the request stream, I can't access the response, even though the server sends it.
Full version
I have a .NET application that uploads files to a Tomcat server, using HttpWebRequest. In some cases, the server closes the request stream prematurely (because it refuses the file for one reason or another, e.g. an invalid filename), and sends a 400 response with a custom header to indicate the cause of the error.
The problem is that if the uploaded file is large, the request stream is closed before I finish writing the request body, and I get an IOException:
Message: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host.
InnerException: SocketException: An existing connection was forcibly closed by the remote host
I can catch this exception, but then, when I call GetResponse, I get a WebException with the previous IOException as its inner exception, and a null Response property. So I can never get the response, even though the server sends it (checked with WireShark).
Since I can't get the response, I don't know what the actual problem is. From my application point of view, it looks like the connection was interrupted, so I treat it as a network-related error and retry the upload... which, of course, fails again.
How can I work around this issue and retrieve the actual response from the server? Is it even possible? To me, the current behavior looks like a bug in HttpWebRequest, or at least a severe design issue...
Here's the code I used to reproduce the problem:
var request = HttpWebRequest.CreateHttp(uri);
request.Method = "POST";
string filename = "foo\u00A0bar.dat"; // Invalid characters in filename, the server will refuse it
request.Headers["Content-Disposition"] = string.Format("attachment; filename*=utf-8''{0}", Uri.EscapeDataString(filename));
request.AllowWriteStreamBuffering = false;
request.ContentType = "application/octet-stream";
request.ContentLength = 100 * 1024 * 1024;
// Upload the "file" (just random data in this case)
try
{
using (var stream = request.GetRequestStream())
{
byte[] buffer = new byte[1024 * 1024];
new Random().NextBytes(buffer);
for (int i = 0; i < 100; i++)
{
stream.Write(buffer, 0, buffer.Length);
}
}
}
catch(Exception ex)
{
// here I get an IOException; InnerException is a SocketException
Console.WriteLine("Error writing to stream: {0}", ex);
}
// Now try to read the response
try
{
using (var response = (HttpWebResponse)request.GetResponse())
{
Console.WriteLine("{0} - {1}", (int)response.StatusCode, response.StatusDescription);
}
}
catch(Exception ex)
{
// here I get a WebException; InnerException is the IOException from the previous catch
Console.WriteLine("Error getting the response: {0}", ex);
var webEx = ex as WebException;
if (webEx != null)
{
Console.WriteLine(webEx.Status); // SendFailure
var response = (HttpWebResponse)webEx.Response;
if (response != null)
{
Console.WriteLine("{0} - {1}", (int)response.StatusCode, response.StatusDescription);
}
else
{
Console.WriteLine("No response");
}
}
}
Additional notes:
If I correctly understand the role of the 100 Continue status, the server shouldn't send it to me if it's going to refuse the file. However, it seems that this status is controlled directly by Tomcat, and can't be controlled by the application. Ideally, I'd like the server not to send me 100 Continue in this case, but according to my colleagues in charge of the back-end, there is no easy way to do it. So I'm looking for a client-side solution for now; but if you happen to know how to solve the problem on the server side, it would also be appreciated.
The app in which I encounter the issue targets .NET 4.0, but I also reproduced it with 4.5.
I'm not timing out. The exception is thrown long before the timeout.
I tried an async request. It doesn't change anything.
I tried setting the request protocol version to HTTP 1.0, with the same result.
Someone else has already filed a bug on Connect for this issue: https://connect.microsoft.com/VisualStudio/feedback/details/779622/unable-to-get-servers-error-response-when-uploading-file-with-httpwebrequest
I am out of ideas as to what can be a client side solution to your problem. But I still think the server side solution of using a custom tomcat valve can help here. I currently doesn`t have a tomcat setup where I can test this but I think a server side solution here would be along the following lines :
RFC section 8.2.3 clearly states :
Requirements for HTTP/1.1 origin servers:
- Upon receiving a request which includes an Expect request-header
field with the "100-continue" expectation, an origin server MUST
either respond with 100 (Continue) status and continue to read
from the input stream, or respond with a final status code. The
origin server MUST NOT wait for the request body before sending
the 100 (Continue) response. If it responds with a final status
code, it MAY close the transport connection or it MAY continue
to read and discard the rest of the request. It MUST NOT
perform the requested method if it returns a final status code.
So assuming tomcat confirms to the RFC, while in the custom valve you would have recieved the HTTP request header, but the request body would not be sent since the control is not yet in the servlet that reads the body.
So you can probably implement a custom valve, something similar to :
import org.apache.catalina.connector.Request;
import org.apache.catalina.connector.Response;
import org.apache.catalina.valves.ErrorReportValve;
public class CustomUploadHandlerValve extends ValveBase {
#Override
public void invoke(Request request, Response response) throws IOException, ServletException {
HttpServletRequest httpRequest = (HttpServletRequest) request;
String fileName = httpRequest.getHeader("Filename"); // get the filename or whatever other parameters required as per your code
bool validationSuccess = Validate(); // perform filename check or anyother validation here
if(!validationSuccess)
{
response = CreateResponse(); //create your custom 400 response here
request.SetResponse(response);
// return the response here
}
else
{
getNext().invoke(request, response); // to pass to the next valve/ servlet in the chain
}
}
...
}
DISCLAIMER : Again I haven`t tried this to success, need sometime and a tomcat setup to try it out ;).
Thought it might be a starting point for you.
I had the same problem. The server sends a response before the client end of the transmission of the request body, when I try to do async request. After a series of experiments, I found a workaround.
After the request stream has been received, I use reflection to check the private field _CoreResponse of the HttpWebRequest. If it is an object of class CoreResponseData, I take his private fields (using reflection): m_StatusCode, m_StatusDescription, m_ResponseHeaders, m_ContentLength. They contain information about the server's response!
In most cases, this hack works!
What are you getting in the status code and response of the second exception not the internal exception?
If a WebException is thrown, use the Response and Status properties of the exception to determine the response from the server.
http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.getresponse(v=vs.110).aspx
You are not saying what exactly version of Tomcat 7 you are using...
checked with WireShark
What do you actually see with WireShark?
Do you see the status line of response?
Do you see the complete status line, up to CR-LF characters at its end?
Is Tomcat asking for authentication credentials (401), or it is refusing file upload for some other reason (first acknowledging it with 100 but then aborting it mid-flight)?
The problem is that if the uploaded file is large, the request stream
is closed before I finish writing the request body, and I get an IOException:
If you do not want the connection to be closed but all the data transferred over the wire and swallowed at the server side, on Tomcat 7.0.55 and later it is possible to configure maxSwallowSize attribute on HTTP connector, e.g. maxSwallowSize="-1".
http://tomcat.apache.org/tomcat-7.0-doc/config/http.html
If you want to discuss Tomcat side of connection handling, you would better ask on the Tomcat users' mailing list,
http://tomcat.apache.org/lists.html#tomcat-users
At .Net side:
Is it possible to perform stream.Write() and request.GetResponse() simultaneously, from different threads?
Is it possible to performs some checks at the client side before actually uploading the file?
hmmm... i don't get it - that is EXACTLY why in many real-life scenarios large files are uploaded in chunks (and not as a single large file)
by the way: many internet servers have size limitations. for instance in tomcat that is representad by maxPostSize (as seen in this link: http://tomcat.apache.org/tomcat-5.5-doc/config/http.html)
so tweaking the server configurations seems like the easy way, but i do think that the right way is to split the file to several requests
EDIT: replace Uri.EscapeDataString with HttpServerUtility.UrlEncode
Uri.EscapeDataString(filename) // a problematic .net implementation
HttpServerUtility.UrlEncode(filename) // the proper way to do it
I am experience a pretty similar problem currently also with Tomcat and a Java client. The Tomcat REST service sends a HTTP returncode with response body before reading the whole request body. The client however fails with IOException. I inserted a HTTP Proxy on the client to sniff the protocol and actually the HTTP response is sent to the client eventually. Most likly the Tomcat closed the request input stream before sending the response.
One solution is to use a different HTTP server like Jetty which does not have this problem. The other solution is a add a Apache HTTP server with AJP in front of Tomcat. Apache HTTP server has a different handling of streams and with that the problem goes away.
i am trying to get the content of http request...my program is using threads and sockets to comunicate to a access terminal..
i ne
this is the request sent from the terminal:
POST /iclock/devicecmd?SN=2182682370001 HTTP/1.1 and the content is
ID1&Return=0&CMD=INFO..
and this is my function to get he full content of http request:
private string GetPedido(NetworkStream stream)
{
string sPedido = "" ;
Byte[] bytesFromStream = new Byte[GlobalFunctionAndVariables.iStreamBufferSize];
while (_tcpClient.Available>0)
{
stream.Read(bytesFromStream, 0, bytesFromStream.Length);
//Console.Write("available: {0}\n", _tcpClient.Available);
sPedido += System.Text.Encoding.ASCII.GetString(bytesFromStream, 0, bytesFromStream.Length);
}
Console.WriteLine("Terminou, a enviar resultado \n");
}
the buffer for now is 32 bytes.
for the moment i am only getting the http header and not the content.
if i want to return the content, do i need to parse the string byte by byte..then find the value of content-length and ask to fecth x more bytes.
is this my only option?
Thanks in advance
The Available property does not what you think it does. Remove its use.
Also, you are not using the return value of Read. TCP offers you a stream of bytes. A Read operation can return any amount starting with one byte. You code must handle that case.
You should probably continue reading until the remote side is done sending. You can find out about that by checking the amount of bytes read against zero.
First, check this message which I was trying to do:
Login
To log on to Windows8 service is through the URL: http://app.proceso.com.mx/win8/login
This URL HTTP Request Method receives POST variables user and pass. The variable user is the user's email and pass the variable is the same password. In the event that the user or password are invalid return plain text number zero 0, in the opposite case, that the username and password are valid return plain text an alphanumeric string of 32 characters, as this b17f27a16589fee247c666da6ed15569, this string is the hash of the valid user valid and will run from 00:00 hours to 23:59 hours the day it was generated.
To test the URL was created: http://app.proceso.com.mx/win8/login_test
Note: It should be clear that the hash generated will only be valid for Windows8 service to the user that gender and the effect from 00:00 hours to 23:59 on the day it was generated.
Note: All services generate text in UTF-8
Here is a test account:
User: javier.lopez.contreras10#gmail.com
Pass: policarpio20
So, if you set the data in this page: http://app.proceso.com.mx/win8/login_test you will receive a hash code.
And that's what I'm trying to accomplish in a metro application, but I feel lost in the situation. I have no idea to send those data to receive the hash code. I was using HttpClient and HttpContent but I'm not sure.
Thanks in advance.
UPDATE: Thanks to dharnitski for the code, right now I'm modifying this code for Win8 CP:
// this is what we are sending
string post_data = "user=javier.lopez.contreras10#gmail.com&pass=policarpio20";
// this is where we will send it
string uri = "http://app.proceso.com.mx/win8/login";
// create a request
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);
request.Method = "POST";
// turn our request string into a byte stream
byte[] postBytes = Encoding.UTF8.GetBytes(post_data);
// this is important - make sure you specify type this way
request.ContentType = "application/x-www-form-urlencoded";
Stream requestStream = await request.GetRequestStreamAsync();
// now send it
requestStream.Write(postBytes, 0, postBytes.Length);
// grab te response and print it out to the console along with the status code
WebResponse response = await request.GetResponseAsync();
//var a = new StreamReader(response.GetResponseStream()).ReadToEnd();
StreamReader requestReader = new StreamReader(response.GetResponseStream());
String webResponse = requestReader.ReadToEnd();
And I realized, HttpWebRequest does not contain ProtocolVersion and is throwing me this error in this line:
WebResponse response = await request.GetResponseAsync();
// ERROR: The remote server returned an error: (417) Expectation Failed.
How can I solve this problem if I can modify protocol version?
This is an sample code to implement HTTP POST in C#
http://www.terminally-incoherent.com/blog/2008/05/05/send-a-https-post-request-with-c/
IMPORTANT: You must switch your web page to HTTPS (SSL). It is very bad practice to send not encrypted passwords.
I need to get content length in order to tell my app where the buffer ends. The problem is that httpwebresponse.ContentLength returns -1 even though Content-Length header is presented in response.
Then I though I'm going to read the actual header to find out the length. The Content-Length returned by the page I'm testing on is 1646. An HTTP sniffer claims that I received 1900 bytes, so I assume the difference are the header length. Then I copied the whole body from response and pasted it into online strlen site and the body size is actually 1850!!
How is this possible? Why does response return invalid content-length and why does httpwebrequest.ContentLength returns -1? How can I calculate the actual response length before receiving the response itself?
EDIT:
This is the code I'm using to get the response:
using (System.IO.Stream responseStream = hwresponse.GetResponseStream())
{
using (MemoryStream memoryStream = new MemoryStream())
{
int count = 0;
do
{
count = responseStream.Read(buffer, 0, buffer.Length);
TCP_R.SendBytes(buffer);
} while (count != 0);
}
}
byte[] PACKET_END_IDENTIFIER = { 0x8, 0x01, 0x8, 0x1, 0x8 };
TCP_R.SendBytes(PACKET_END_IDENTIFIER);
TCP_R.Close();
I have a proxy server application that takes a request, sends it to another application (my client) client executes the request and using TCP_R class returns the result. When server gets response from client, it returns response back to browser.
Each time I do a request, I get all the data + extra garbage, here's an example:
<tag1><tag2><tag3> ag3>
ag3> is the garbage data, it's like the ending of buffer is cut off and added again. It apprears that the client responds with a valid response, the garbage data is added onDataRecieve event.. any tips? thanks!
-1 isn't an invalid value of the ContentLength property. I assume you mean the ContentLength property of the response is -1... asking the request what the length is would be non-sensical. Even so, it's perfectly valid:
The ContentLength property contains the value of the Content-Length header returned with the response. If the Content-Length header is not set in the response, ContentLength is set to the value -1.
If the body length is 1850, that suggests it's using chunked transfer encoding. But that should be transparent to you - just keep reading from the response stream until the end. If you're using .NET 4, it's dead easy - just create a MemoryStream and use Stream.CopyTo to copy the data to that MemoryStream.
Does any one know, how to communicate to Apache server from client side using Win32 or C#.
and can i get the control to execute the process at server? if so how to do?.
how to receive file sent by apache?.
Please help me...
Thanks
You can use HTTPWebRequest and HTTPWebResponse to make a request to a web server and receive a response.
For example
static void Main(string[] args)
{
// used to build entire input
StringBuilder sb = new StringBuilder();
// used on each read operation
byte[] buf = new byte[8192];
// prepare the web page we will be asking for
HttpWebRequest request = (HttpWebRequest)
WebRequest.Create("http://www.mayosoftware.com");
// execute the request
HttpWebResponse response = (HttpWebResponse)
request.GetResponse();
// we will read data via the response stream
Stream resStream = response.GetResponseStream();
string tempString = null;
int count = 0;
do
{
// fill the buffer with data
count = resStream.Read(buf, 0, buf.Length);
// make sure we read some data
if (count != 0)
{
// translate from bytes to ASCII text.
// Not needed if you'll get binary content.
tempString = Encoding.ASCII.GetString(buf, 0, count);
// continue building the string
sb.Append(tempString);
}
}
while (count > 0); // any more data to read?
// print out page source
Console.WriteLine(sb.ToString());
}
Okay. Assuming you are talking about "Apache httpd", then we are talking about a server which talks HTTP. Pretty much any programming language has some "HTTP Client" classes. In the case of C#, you can use HttpWebRequest. Vinko gave a great "copy/paste" example.
To run a process on the server-side, you need some "server side scripting". If there is no module available for the httpd server, then you can always resort to CGI. This is potentially dangerous, if not careful though. But you can say this about all server-side scripting (the "if not careful" is important here).
As it sounds like, you are new to dynamic web development. And, as much as I dislike it, I have to say that the PHP community is a great starting point.
To give a rough overview of how things work:
The client (You C# app) make a "request" to the server (by default a TCP connection to port 80)
As the server (httpd) is listening on this port, it accepts this connection and starts "talking HTTP"
The server retrieves the URL from the HTTP GET command and decides what to do with it.
If this URL maps to a static resource, the server sends a "response" with the resource's content and is done.
If this URL maps to a dynamic resource, the server delegates the work to another process. This process executes, and the http-server receives the standard output of the process as result (In the case of PHP, CGI, and other similar technologies). The server then sends this result back to the client.
What I understood the least when I got started were the "cryptic" httpd-configurations. They are simple to understand though. They are there to tell the server how (and what) to map to the external process (as noted in step 4a and 4b).
So, in the case of PHP:
LoadModule php5_module modules/libphp5.so
This makes the "external process" and configuarion directives available to the server (loads the extensions).
<FilesMatch \.php$>
SetHandler application/x-httpd-php
</FilesMatch>
The default Apache-httpd handler assumes that a URL maps to a File on the file system. This configuration directive tells httpd to process (handle) each filename ending with ".php" using the PHP module before returning it to the client.
Final note
These steps are very similar for other server-side applications. I only gave PHP as a practical example. And it's got very good documentation.