I have a method which is intended to download a file from an HTTP URL to a byte array:
private static byte[] DownloadFileToByteArrayWorker(HttpWebRequest Request, int bufferLength)
{
byte[] responseByes = null;
//Round up to the nearest multiple of 1024
bufferLength = AdjustBufferLength(bufferLength);
Request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
Request.ServicePoint.Expect100Continue = false;
Request.Headers.Add(HttpRequestHeader.CacheControl, "no-cache");
using (var Response = (HttpWebResponse)Request.GetResponse())
{
using (Stream ResponseStream = Response.GetResponseStream())
{
using (MemoryStream ms = new MemoryStream())
{
int count = 0;
byte[] buf = new byte[bufferLength];
while ((count = ResponseStream.Read(buf, 0, buf.Length)) > 0)
{
ms.Write(buf, 0, count);
}
responseByes = ms.ToArray();
}
}
}
return responseByes;
}
Request.GetResponse() is throwing a time out exception no matter how long I make the Timeout property of the HttpWebRequest. I can verify via my logs that the program is waiting the full Timeout period before erroring out, however, correlating my logs with the web server logs indicates that the web server is sending back a response almost immediately.
An interesting note is that when I access the same web server via the load balancer rather than directly, it downloads the file practically instantly. Also, if I access the URL via the web server directly in a web browser (no proxy needed, btw) I can download the file from individual web servers instantly that way too.
Some additional details:
I am using .NET Framework 4.7 on Windows 2012 R2.
The web server I'm trying to connect to is Apache on RHEL7. I'm not sure about the specific Apache version
I am connecting to the web server on a specific port which is reserved for HTTP traffic (a separate website is hosted on a different port number for HTTPS)
There's no web proxy
Any suggestions?
As you said your code has problem only when you call the load balancer,
I think the problem is the your client send a 100 continue request but your load balancer don't know how to handle it.
That is the reason you client doesn't send all the data right after the beginning of connection.
You can find more information about 100 continue in HTTP rfc section 8.2.3.
To fix the behavior from client side in c# you have to add this code:
ServicePointManager.Expect100Continue = false;
You can see the full documentation about this feature here.
I'm trying to download first bytes of a webpage.
I add Range to the HTTP request header. it's my code in C#:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://example.com");
request.AddRange(0,1000);
//request.Proxy = null;
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
Stream st = response.GetResponseStream();
StreamReader sr = new StreamReader(st);
string str = sr.ReadToEnd();
sr.Close();
st.Close();
}
for some webpages it works fine, but some servers will ignore the HTTP range header, so the server send all the page in the response.
I changed my code to this:
string str = sr.Read(buffer,0,999);
but it doesn't work! because the problem is not in this line. actually the response will send to my program when I call request.GetResponse()! in this line all the bytes received by the program and write the all bytes to the RAM. I want to stop receiving data from the server when I received first 1000 bytes.
but there is no control on HttpWebRequest class and .GetResponse() method to stop receiving data after 1000 bytes received.
How can I do that?
I think there would be another HTTP Request custom class that allow us to stop receiving data when we want.
please tell me any Idea about this problem. I'm thinking to override the HttpWebRequest or write a MFC Library (C++ language) and import it into my project, but I don't know how to do this.
EDIT: I know it's optional for server to allow or ignore the Range header! but my question is how can I stop receiving data from the server! for example the server is sending 10,000 bytes to my computer, I want to stop receiving bytes after I see the 1000th byte! (I don't want the server to send just first 1000 bytes, I want to close connection and stop receiving the bytes after first 1000 bytes! even if the server send all 10,000 bytes)
TL;DR version
When a transfer error occurs while writing to the request stream, I can't access the response, even though the server sends it.
Full version
I have a .NET application that uploads files to a Tomcat server, using HttpWebRequest. In some cases, the server closes the request stream prematurely (because it refuses the file for one reason or another, e.g. an invalid filename), and sends a 400 response with a custom header to indicate the cause of the error.
The problem is that if the uploaded file is large, the request stream is closed before I finish writing the request body, and I get an IOException:
Message: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host.
InnerException: SocketException: An existing connection was forcibly closed by the remote host
I can catch this exception, but then, when I call GetResponse, I get a WebException with the previous IOException as its inner exception, and a null Response property. So I can never get the response, even though the server sends it (checked with WireShark).
Since I can't get the response, I don't know what the actual problem is. From my application point of view, it looks like the connection was interrupted, so I treat it as a network-related error and retry the upload... which, of course, fails again.
How can I work around this issue and retrieve the actual response from the server? Is it even possible? To me, the current behavior looks like a bug in HttpWebRequest, or at least a severe design issue...
Here's the code I used to reproduce the problem:
var request = HttpWebRequest.CreateHttp(uri);
request.Method = "POST";
string filename = "foo\u00A0bar.dat"; // Invalid characters in filename, the server will refuse it
request.Headers["Content-Disposition"] = string.Format("attachment; filename*=utf-8''{0}", Uri.EscapeDataString(filename));
request.AllowWriteStreamBuffering = false;
request.ContentType = "application/octet-stream";
request.ContentLength = 100 * 1024 * 1024;
// Upload the "file" (just random data in this case)
try
{
using (var stream = request.GetRequestStream())
{
byte[] buffer = new byte[1024 * 1024];
new Random().NextBytes(buffer);
for (int i = 0; i < 100; i++)
{
stream.Write(buffer, 0, buffer.Length);
}
}
}
catch(Exception ex)
{
// here I get an IOException; InnerException is a SocketException
Console.WriteLine("Error writing to stream: {0}", ex);
}
// Now try to read the response
try
{
using (var response = (HttpWebResponse)request.GetResponse())
{
Console.WriteLine("{0} - {1}", (int)response.StatusCode, response.StatusDescription);
}
}
catch(Exception ex)
{
// here I get a WebException; InnerException is the IOException from the previous catch
Console.WriteLine("Error getting the response: {0}", ex);
var webEx = ex as WebException;
if (webEx != null)
{
Console.WriteLine(webEx.Status); // SendFailure
var response = (HttpWebResponse)webEx.Response;
if (response != null)
{
Console.WriteLine("{0} - {1}", (int)response.StatusCode, response.StatusDescription);
}
else
{
Console.WriteLine("No response");
}
}
}
Additional notes:
If I correctly understand the role of the 100 Continue status, the server shouldn't send it to me if it's going to refuse the file. However, it seems that this status is controlled directly by Tomcat, and can't be controlled by the application. Ideally, I'd like the server not to send me 100 Continue in this case, but according to my colleagues in charge of the back-end, there is no easy way to do it. So I'm looking for a client-side solution for now; but if you happen to know how to solve the problem on the server side, it would also be appreciated.
The app in which I encounter the issue targets .NET 4.0, but I also reproduced it with 4.5.
I'm not timing out. The exception is thrown long before the timeout.
I tried an async request. It doesn't change anything.
I tried setting the request protocol version to HTTP 1.0, with the same result.
Someone else has already filed a bug on Connect for this issue: https://connect.microsoft.com/VisualStudio/feedback/details/779622/unable-to-get-servers-error-response-when-uploading-file-with-httpwebrequest
I am out of ideas as to what can be a client side solution to your problem. But I still think the server side solution of using a custom tomcat valve can help here. I currently doesn`t have a tomcat setup where I can test this but I think a server side solution here would be along the following lines :
RFC section 8.2.3 clearly states :
Requirements for HTTP/1.1 origin servers:
- Upon receiving a request which includes an Expect request-header
field with the "100-continue" expectation, an origin server MUST
either respond with 100 (Continue) status and continue to read
from the input stream, or respond with a final status code. The
origin server MUST NOT wait for the request body before sending
the 100 (Continue) response. If it responds with a final status
code, it MAY close the transport connection or it MAY continue
to read and discard the rest of the request. It MUST NOT
perform the requested method if it returns a final status code.
So assuming tomcat confirms to the RFC, while in the custom valve you would have recieved the HTTP request header, but the request body would not be sent since the control is not yet in the servlet that reads the body.
So you can probably implement a custom valve, something similar to :
import org.apache.catalina.connector.Request;
import org.apache.catalina.connector.Response;
import org.apache.catalina.valves.ErrorReportValve;
public class CustomUploadHandlerValve extends ValveBase {
#Override
public void invoke(Request request, Response response) throws IOException, ServletException {
HttpServletRequest httpRequest = (HttpServletRequest) request;
String fileName = httpRequest.getHeader("Filename"); // get the filename or whatever other parameters required as per your code
bool validationSuccess = Validate(); // perform filename check or anyother validation here
if(!validationSuccess)
{
response = CreateResponse(); //create your custom 400 response here
request.SetResponse(response);
// return the response here
}
else
{
getNext().invoke(request, response); // to pass to the next valve/ servlet in the chain
}
}
...
}
DISCLAIMER : Again I haven`t tried this to success, need sometime and a tomcat setup to try it out ;).
Thought it might be a starting point for you.
I had the same problem. The server sends a response before the client end of the transmission of the request body, when I try to do async request. After a series of experiments, I found a workaround.
After the request stream has been received, I use reflection to check the private field _CoreResponse of the HttpWebRequest. If it is an object of class CoreResponseData, I take his private fields (using reflection): m_StatusCode, m_StatusDescription, m_ResponseHeaders, m_ContentLength. They contain information about the server's response!
In most cases, this hack works!
What are you getting in the status code and response of the second exception not the internal exception?
If a WebException is thrown, use the Response and Status properties of the exception to determine the response from the server.
http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.getresponse(v=vs.110).aspx
You are not saying what exactly version of Tomcat 7 you are using...
checked with WireShark
What do you actually see with WireShark?
Do you see the status line of response?
Do you see the complete status line, up to CR-LF characters at its end?
Is Tomcat asking for authentication credentials (401), or it is refusing file upload for some other reason (first acknowledging it with 100 but then aborting it mid-flight)?
The problem is that if the uploaded file is large, the request stream
is closed before I finish writing the request body, and I get an IOException:
If you do not want the connection to be closed but all the data transferred over the wire and swallowed at the server side, on Tomcat 7.0.55 and later it is possible to configure maxSwallowSize attribute on HTTP connector, e.g. maxSwallowSize="-1".
http://tomcat.apache.org/tomcat-7.0-doc/config/http.html
If you want to discuss Tomcat side of connection handling, you would better ask on the Tomcat users' mailing list,
http://tomcat.apache.org/lists.html#tomcat-users
At .Net side:
Is it possible to perform stream.Write() and request.GetResponse() simultaneously, from different threads?
Is it possible to performs some checks at the client side before actually uploading the file?
hmmm... i don't get it - that is EXACTLY why in many real-life scenarios large files are uploaded in chunks (and not as a single large file)
by the way: many internet servers have size limitations. for instance in tomcat that is representad by maxPostSize (as seen in this link: http://tomcat.apache.org/tomcat-5.5-doc/config/http.html)
so tweaking the server configurations seems like the easy way, but i do think that the right way is to split the file to several requests
EDIT: replace Uri.EscapeDataString with HttpServerUtility.UrlEncode
Uri.EscapeDataString(filename) // a problematic .net implementation
HttpServerUtility.UrlEncode(filename) // the proper way to do it
I am experience a pretty similar problem currently also with Tomcat and a Java client. The Tomcat REST service sends a HTTP returncode with response body before reading the whole request body. The client however fails with IOException. I inserted a HTTP Proxy on the client to sniff the protocol and actually the HTTP response is sent to the client eventually. Most likly the Tomcat closed the request input stream before sending the response.
One solution is to use a different HTTP server like Jetty which does not have this problem. The other solution is a add a Apache HTTP server with AJP in front of Tomcat. Apache HTTP server has a different handling of streams and with that the problem goes away.
I'm having a lot of problems calling a Java based Google App Engine server from a C# client
This is how my client code looks like:
// C# Client
static void Main(string[] args)
{
const string URL = "http://localhost:8888/googlewebapptest7/greet";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(URL);
request.Method = "POST";
request.ContentType = "text/x-gwt-rpc; charset=utf-8";
string content = "<?xml version='1.0'?><methodCall><methodName>greetServlet.GetName</methodName><params></params></methodCall>";
byte[] contentBytes = UTF8Encoding.UTF8.GetBytes(content);
request.ContentLength = contentBytes.Length;
using (Stream stream = request.GetRequestStream())
{
stream.Write(contentBytes, 0, contentBytes.Length);
}
// get response
WebResponse response = request.GetResponse();
using (Stream responseStream = response.GetResponseStream())
{
string res = new StreamReader(responseStream).ReadToEnd();
Console.WriteLine("response from server:");
Console.WriteLine(res);
Console.ReadKey();
}
The server is basically the Google default web project with an additional method:
public String GetName()
{
return "HI!";
}
added to GreetingServiceImpl.
Everytime I run my client, I get the following exception from the server:
An IncompatibleRemoteServiceException was thrown while processing this call.
com.google.gwt.user.client.rpc.IncompatibleRemoteServiceException: This application is out of date, please click the refresh button on your browser. ( Malformed or old RPC message received - expecting version 5 )
I would like to keep it in plain HTTP requests.
Any idea what's going on?
As Nick pointed out you're trying to emulate GWT's RPC format. I tried that too :)
Then I took different approach. I used Google Protocol Buffers as encoder/decoder over HTTP(S).
One of my projects is desktop application written in C#. Server-side is also C#.Net. Naturally we use WCF as transport.
You can plug-in Protocol Buffers into WCF transport. You'll have same interface configuration both for C# client and Java server. It's very convenient.
I'll update this answer with code-samples when I'm less busy with work.
I never found a good way to use XML based RPC on the Google App Engine. Instead i found this excellent JSON library with tutorial:
http://werxltd.com/wp/portfolio/json-rpc/simple-java-json-rpc/
it works very well!
I'm currently measuring the time spent to load a web page from a C# program.
The Visual Studio solution has a console program and an ASP.NET website with just one page. The website is hosted on ASP.NET Development Server. The console application queries the web page like this:
bool isSuccess;
Stopwatch timeSpentToDownloadPage = Stopwatch.StartNew();
WebRequest request = HttpWebRequest.Create(new Uri("http://localhost:12345/Test.aspx", UriKind.Absolute));
request.Timeout = 200;
using (WebResponse response = request.GetResponse())
{
Stream responseStream = response.GetResponseStream();
StreamReader sr = new StreamReader(responseStream);
string responseText = sr.ReadToEnd().Trim();
isSuccess = (responseText == "Hello World");
}
timeSpentToDownloadPage.Stop();
The web page has nothing special, just a response on load and no ASP.NET code:
protected void Page_Load(object sender, EventArgs e)
{
Response.Write("Hello World");
}
Now, the stopwatch shows every time that the code spent 0 ms. to do all the work (querying the server, getting the response, etc.).
How is it possible? Isn't there something wrong with what I'm doing? Really, I expected at least 10-20 ms. to execute all client-side code, then 100 ms. to:
[client side] Find the page to query from uri (which does not require DNS query, so it's quite fast),
[client side] Do the request,
[server side] Initialize ASP.NET engine,
[server side] Process the request,
[server side] Find and read .aspx file and execute compiled code (at least reading file may cost several ms.),
[server side] Build response, including headers, then send it,
[client side] Receive the response and process it (trim).
So why is it so extremely fast? Is there a tricky cache which just skips all/most of the steps and return "Hello World" to the client?
If there is a "hidden" cache, where is it and how can I disable it to measure the "real" time spent?
Yes this can and must be 0 ms! Because your server is on a localhost. Try to get ASP.NET server somewhere else in a world wide network.
And you don't resolving localhost because this is a constant address to 127.0.0.1.
And communication to this address is operated only by the your OS kernel.