Develop Reference

How do I download a file in segments in C#?

I am using HttpWebRequest with the AddRange function like:
HttpWebRequest myHttpWebRequest = (HttpWebRequest)WebRequest.Create(URL);
myHttpWebRequest.AddRange(20, 30);
HttpWebResponse myHttpWebResponse = (HttpWebResponse)myHttpWebRequest.GetResponse();
Stream streamResponse = myHttpWebResponse.GetResponseStream();
SaveFileStream(name, streamResponse); //save file function
...but the entire file downloaded.
AddRange() in the above code expects the bytes between 20 and 30 to be downloaded (in other words, to download those 10 bytes from the file).
But my code is not working, since the download is not segmented. This link provides an example: http://stackoverflow.com/robots.txt That file was downloaded in its entirety. Why?

HTTP server are not required to support Range header requests. You can verify server's range support by issuing HEAD request and checking value of Accept-Ranges header in response (see HTTP range requests). But this still doesn't guarantee that server will not ignore Range header, in particular for very small ranges (it would be very inefficient for HTTP server to serve content in such small segments).
From RFC7233:
Because servers are free to ignore Range, many implementations will
simply respond with the entire selected representation in a 200 (OK)
response. That is partly because most clients are prepared to receive
a 200 (OK) to complete the task (albeit less efficiently) and partly
because clients might not stop making an invalid partial request until
they have received a complete representation. Thus, clients cannot
depend on receiving a 416 (Range Not Satisfiable) response even when
it is most appropriate.
To determine if server accepted or ignored Range header, you must check response status code. 200 (OK) indicates that server ignored Range header and returned whole response body, 206 (Partial Content) indicates that range specified in header was returned by server and 416 (Range Not Satisfiable) indicates that the set of ranges requested has been rejected due to invalid ranges or an excessive request of small or overlapping ranges.
In case of http://stackoverflow.com/robots.txt, server indicates support of Range header by returning Accept-Ranges: bytes header in response on HEAD request, but on GET request with AddRange(20, 30) specified, response is 200 (OK), so server just ignored such small range a returned whole reponse body. You have to cut requested range from response yourself, if you need to.

HttpWebRequest using GET on redirect instead of POST like idHttp

Sorry for my english.
In Delphi I have an idHttp component with the hoWaitForUnexpectedData option activated.
When I send a POST request to a URL, it redirects the client to a second URL with the same POST request and headers. Also, the server response contains "Connection: keep-alive" in its header.
However, when I try to do the same request in C# with a HttpWebRequest component, it redirects to the second URL using the method GET.
I need the C# HttpWebRequest component to work like the Delphi idHTTP one does. I don't understand why it uses a GET instead of a POST when following the redirection.
Here's my code in Delphi, using hoWaitForUnexpectedData:
// The server is supposed to send a 'Content-Length' header without sending
// the actual data. 1xx, 204, and 304 replies are not supposed to contain
// entity bodies, either...
if TextIsSame(ARequest.Method, Id_HTTPMethodHead) or
TextIsSame(ARequest.MethodOverride, Id_HTTPMethodHead) or
((AResponse.ResponseCode div 100) = 1) or
(AResponse.ResponseCode = 204) or
(AResponse.ResponseCode = 304) then
begin
// Have noticed one case where a non-conforming server did send an
// entity body in response to a HEAD request. If requested, ignore
// anything the server may send by accident
if not (hoWaitForUnexpectedData in FOptions) then begin
Exit;
end;
Result := CheckForPendingData(100);
end
else if (AResponse.ResponseCode div 100) = 3 then
begin
// This is a workaround for buggy HTTP 1.1 servers which
// does not return any body with 302 response code
Result := CheckForPendingData(5000);
end else begin
Result := True;
end;

An HTTP redirect, by definition from the standard, should be handled using a GET. Therefore, if you send a POST and get a redirect as an answer, the expected behavior is to perform a GET to the redirect address. I suspect the old Delphi component is following old practices and replicates the call including with the POST verb.
I would try to disable AllowAutoRedirect in the HttpWebRequest object and handle this manually, as your case seems to differ from the standard.

The hoWaitForUnexpectedData option has no effect on how TIdHTTP handles redirects, and neither does the section of code you quoted.
However, the hoTreat302Like303 option does affect redirect handling. If TIdHTTP receives a 303 redirect, or receives a 302 redirect with hoTreat302Like303 enabled, TIdHTTP sends the new request as a GET. Otherwise, it sends the new request using the same verb as the redirected request. This is by design, and there is a series of comments in the implementation code of the TIdHTTPProtocol.ProcessResponse() method explaining the rational behind this behavior:
// GDG 21/11/2003. If it's a 303, we should do a get this time
// RLebeau 7/15/2004 - do a GET on 302 as well, as mentioned in RFC 2616
// RLebeau 1/11/2008 - turns out both situations are WRONG! RFCs 2068 and
// 2616 specifically state that changing the method to GET in response
// to 302 and 303 is errorneous. Indy 9 did it right by reusing the
// original method and source again and only changing the URL, so lets
// revert back to that same behavior!
// RLebeau 12/28/2012 - one more time. RFCs 2068 and 2616 actually say that
// changing the method in response to 302 is erroneous, but changing the
// method to GET in response to 303 is intentional and why 303 was introduced
// in the first place. Erroneous clients treat 302 as 303, though. Now
// encountering servers that actually expect this 303 behavior, so we have
// to enable it again! Adding an optional HTTPOption flag so clients can
// enable the erroneous 302 behavior if they really need it.
The jist of it is that the HTTP spec says to send a GET for a 303 redirect, whereas it is ambiguous about whether to send a GET for 302. Some browsers do, some do not. That is why the hoTreat302Like303 option was added, though it is disabled by default for backwards compatibility with earlier Indy versions.
So, the behavior you describe means you must be encountering a 302 redirect, with hoTreat302Like303 disabled (which it is default). If you enable that option, TIdHTTP will behave more like HttpWebRequest, not the other way around.

How to get the HTTP response when the request stream was closed during transfer

TL;DR version
When a transfer error occurs while writing to the request stream, I can't access the response, even though the server sends it.
Full version
I have a .NET application that uploads files to a Tomcat server, using HttpWebRequest. In some cases, the server closes the request stream prematurely (because it refuses the file for one reason or another, e.g. an invalid filename), and sends a 400 response with a custom header to indicate the cause of the error.
The problem is that if the uploaded file is large, the request stream is closed before I finish writing the request body, and I get an IOException:
Message: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host.
InnerException: SocketException: An existing connection was forcibly closed by the remote host
I can catch this exception, but then, when I call GetResponse, I get a WebException with the previous IOException as its inner exception, and a null Response property. So I can never get the response, even though the server sends it (checked with WireShark).
Since I can't get the response, I don't know what the actual problem is. From my application point of view, it looks like the connection was interrupted, so I treat it as a network-related error and retry the upload... which, of course, fails again.
How can I work around this issue and retrieve the actual response from the server? Is it even possible? To me, the current behavior looks like a bug in HttpWebRequest, or at least a severe design issue...
Here's the code I used to reproduce the problem:
var request = HttpWebRequest.CreateHttp(uri);
request.Method = "POST";
string filename = "foo\u00A0bar.dat"; // Invalid characters in filename, the server will refuse it
request.Headers["Content-Disposition"] = string.Format("attachment; filename*=utf-8''{0}", Uri.EscapeDataString(filename));
request.AllowWriteStreamBuffering = false;
request.ContentType = "application/octet-stream";
request.ContentLength = 100 * 1024 * 1024;
// Upload the "file" (just random data in this case)
try
{
using (var stream = request.GetRequestStream())
{
byte[] buffer = new byte[1024 * 1024];
new Random().NextBytes(buffer);
for (int i = 0; i < 100; i++)
{
stream.Write(buffer, 0, buffer.Length);
}
}
}
catch(Exception ex)
{
// here I get an IOException; InnerException is a SocketException
Console.WriteLine("Error writing to stream: {0}", ex);
}
// Now try to read the response
try
{
using (var response = (HttpWebResponse)request.GetResponse())
{
Console.WriteLine("{0} - {1}", (int)response.StatusCode, response.StatusDescription);
}
}
catch(Exception ex)
{
// here I get a WebException; InnerException is the IOException from the previous catch
Console.WriteLine("Error getting the response: {0}", ex);
var webEx = ex as WebException;
if (webEx != null)
{
Console.WriteLine(webEx.Status); // SendFailure
var response = (HttpWebResponse)webEx.Response;
if (response != null)
{
Console.WriteLine("{0} - {1}", (int)response.StatusCode, response.StatusDescription);
}
else
{
Console.WriteLine("No response");
}
}
}
Additional notes:
If I correctly understand the role of the 100 Continue status, the server shouldn't send it to me if it's going to refuse the file. However, it seems that this status is controlled directly by Tomcat, and can't be controlled by the application. Ideally, I'd like the server not to send me 100 Continue in this case, but according to my colleagues in charge of the back-end, there is no easy way to do it. So I'm looking for a client-side solution for now; but if you happen to know how to solve the problem on the server side, it would also be appreciated.
The app in which I encounter the issue targets .NET 4.0, but I also reproduced it with 4.5.
I'm not timing out. The exception is thrown long before the timeout.
I tried an async request. It doesn't change anything.
I tried setting the request protocol version to HTTP 1.0, with the same result.
Someone else has already filed a bug on Connect for this issue: https://connect.microsoft.com/VisualStudio/feedback/details/779622/unable-to-get-servers-error-response-when-uploading-file-with-httpwebrequest

I am out of ideas as to what can be a client side solution to your problem. But I still think the server side solution of using a custom tomcat valve can help here. I currently doesn`t have a tomcat setup where I can test this but I think a server side solution here would be along the following lines :
RFC section 8.2.3 clearly states :
Requirements for HTTP/1.1 origin servers:
- Upon receiving a request which includes an Expect request-header
field with the "100-continue" expectation, an origin server MUST
either respond with 100 (Continue) status and continue to read
from the input stream, or respond with a final status code. The
origin server MUST NOT wait for the request body before sending
the 100 (Continue) response. If it responds with a final status
code, it MAY close the transport connection or it MAY continue
to read and discard the rest of the request. It MUST NOT
perform the requested method if it returns a final status code.
So assuming tomcat confirms to the RFC, while in the custom valve you would have recieved the HTTP request header, but the request body would not be sent since the control is not yet in the servlet that reads the body.
So you can probably implement a custom valve, something similar to :
import org.apache.catalina.connector.Request;
import org.apache.catalina.connector.Response;
import org.apache.catalina.valves.ErrorReportValve;
public class CustomUploadHandlerValve extends ValveBase {
#Override
public void invoke(Request request, Response response) throws IOException, ServletException {
HttpServletRequest httpRequest = (HttpServletRequest) request;
String fileName = httpRequest.getHeader("Filename"); // get the filename or whatever other parameters required as per your code
bool validationSuccess = Validate(); // perform filename check or anyother validation here
if(!validationSuccess)
{
response = CreateResponse(); //create your custom 400 response here
request.SetResponse(response);
// return the response here
}
else
{
getNext().invoke(request, response); // to pass to the next valve/ servlet in the chain
}
}
...
}
DISCLAIMER : Again I haven`t tried this to success, need sometime and a tomcat setup to try it out ;).
Thought it might be a starting point for you.

I had the same problem. The server sends a response before the client end of the transmission of the request body, when I try to do async request. After a series of experiments, I found a workaround.
After the request stream has been received, I use reflection to check the private field _CoreResponse of the HttpWebRequest. If it is an object of class CoreResponseData, I take his private fields (using reflection): m_StatusCode, m_StatusDescription, m_ResponseHeaders, m_ContentLength. They contain information about the server's response!
In most cases, this hack works!

What are you getting in the status code and response of the second exception not the internal exception?
If a WebException is thrown, use the Response and Status properties of the exception to determine the response from the server.
http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.getresponse(v=vs.110).aspx

You are not saying what exactly version of Tomcat 7 you are using...
checked with WireShark
What do you actually see with WireShark?
Do you see the status line of response?
Do you see the complete status line, up to CR-LF characters at its end?
Is Tomcat asking for authentication credentials (401), or it is refusing file upload for some other reason (first acknowledging it with 100 but then aborting it mid-flight)?
The problem is that if the uploaded file is large, the request stream
is closed before I finish writing the request body, and I get an IOException:
If you do not want the connection to be closed but all the data transferred over the wire and swallowed at the server side, on Tomcat 7.0.55 and later it is possible to configure maxSwallowSize attribute on HTTP connector, e.g. maxSwallowSize="-1".
http://tomcat.apache.org/tomcat-7.0-doc/config/http.html
If you want to discuss Tomcat side of connection handling, you would better ask on the Tomcat users' mailing list,
http://tomcat.apache.org/lists.html#tomcat-users
At .Net side:
Is it possible to perform stream.Write() and request.GetResponse() simultaneously, from different threads?
Is it possible to performs some checks at the client side before actually uploading the file?

hmmm... i don't get it - that is EXACTLY why in many real-life scenarios large files are uploaded in chunks (and not as a single large file)
by the way: many internet servers have size limitations. for instance in tomcat that is representad by maxPostSize (as seen in this link: http://tomcat.apache.org/tomcat-5.5-doc/config/http.html)
so tweaking the server configurations seems like the easy way, but i do think that the right way is to split the file to several requests
EDIT: replace Uri.EscapeDataString with HttpServerUtility.UrlEncode
Uri.EscapeDataString(filename) // a problematic .net implementation
HttpServerUtility.UrlEncode(filename) // the proper way to do it

I am experience a pretty similar problem currently also with Tomcat and a Java client. The Tomcat REST service sends a HTTP returncode with response body before reading the whole request body. The client however fails with IOException. I inserted a HTTP Proxy on the client to sniff the protocol and actually the HTTP response is sent to the client eventually. Most likly the Tomcat closed the request input stream before sending the response.
One solution is to use a different HTTP server like Jetty which does not have this problem. The other solution is a add a Apache HTTP server with AJP in front of Tomcat. Apache HTTP server has a different handling of streams and with that the problem goes away.

HTTPS Redirect Causing Error "Server cannot append header after HTTP headers have been sent"

I need to check that our visitors are using HTTPS. In BasePage I check if the request is coming via HTTPS. If it's not, I redirect back with HTTPS. However, when someone comes to the site and this function is used, I get the error:
System.Web.HttpException: Server
cannot append header after HTTP
headers have been sent. at
System.Web.HttpResponse.AppendHeader(String
name, String value) at
System.Web.HttpResponse.AddHeader(String
name, String value) at
Premier.Payment.Website.Generic.BasePage..ctor()
Here is the code I started with:
// If page not currently SSL
if (HttpContext.Current.Request.ServerVariables["HTTPS"].Equals("off"))
{
// If SSL is required
if (GetConfigSetting("SSLRequired").ToUpper().Equals("TRUE"))
{
string redi = "https://" +
HttpContext.Current.Request.ServerVariables["SERVER_NAME"].ToString() +
HttpContext.Current.Request.ServerVariables["SCRIPT_NAME"].ToString() +
"?" + HttpContext.Current.Request.ServerVariables["QUERY_STRING"].ToString();
HttpContext.Current.Response.Redirect(redi.ToString());
}
}
I also tried adding this above it (a bit I used in another site for a similar problem):
// Wait until page is copletely loaded before sending anything since we re-build
HttpContext.Current.Response.BufferOutput = true;
I am using c# in .NET 3.5 on IIS 6.

Chad,
Did you try ending the output when you redirect? There is a second parameter that you'd set to true to tell the output to stop when the redirect header is issued. Or, if you are buffering the output then maybe you need to clear the buffer before doing the redirect so the headers are not sent out along with the redirect header.
Brian

This error usually means that something has bee written to the response stream before a redirection is initiated. So you should make sure that the test for https is done fairly high up in the page load function.

Override IP in HTTP Request

Looking for a way to issue an HTTPwebrequest, or use the browser control, or winhttp to make a request to a URL, but override the IP address it connects to from the DNS lookup to a specific one.
Trying to do something similar to the HOSTS file, but programatically without having to modify this file. It can be C# or C+
Why I need it, the host i am sending the request has multiple IPs, and their Domain servers are doing load balancing accross the different IPs. Trying to force the request to a particular IP, but I need the host in the http request to be still the original host. I need this programatically because changing the host file every time i need to run this test is too time consuming.

All you had to do was this:
var request = (HttpWebRequest) WebRequest.Create("http://192.168.1.1");
request.Host = "news.bbc.co.uk";

If I understand correctly you have to make an http request to a web server using virtualhosts but the DNS isn't setup yet so you have to specify the ip address in the url but send something else in the Host: header.
If that's the case you may be able to do so..
In C# using WebProxy:
See Kayode Leonard's answer for .NET 4 and up.
Here's the code I would use if I have my server running on 67.223.227.171:8888 but I need to have www.example.com in the Host: header.
System.Net.WebRequest r = System.Net.WebRequest.Create("http://www.example.com");
r.Proxy = new WebProxy("http://67.223.227.171:8888");
See this link
In C++ using WinHttp:
Using WinHttp you can simply set the Host: header with WinHttpAddRequestHeaders.
So once again if I have my server running on 67.223.227.171:8888 but I need to have www.example.com in the Host: header:
#include <windows.h>
#include <winhttp.h>
#include <assert.h>
int main() {
HINTERNET hSession = WinHttpOpen(L"A WinHTTP Example Program/1.0",
WINHTTP_ACCESS_TYPE_DEFAULT_PROXY,
WINHTTP_NO_PROXY_NAME,
WINHTTP_NO_PROXY_BYPASS, 0);
assert(hSession != NULL);
// Use WinHttpConnect to specify an HTTP server.
HINTERNET hConnect = WinHttpConnect( hSession,
L"67.223.227.171",
8888,
0 );
assert(hConnect != NULL);
// Open and Send a Request Header.
HINTERNET hRequest = WinHttpOpenRequest( hConnect,
L"GET",
L"/downloads/samples/internet/winhttp/retoptions/redirect.asp",
NULL,
WINHTTP_NO_REFERER,
WINHTTP_DEFAULT_ACCEPT_TYPES,
0 );
assert(hRequest != NULL);
BOOL httpResult = WinHttpAddRequestHeaders(
hRequest,
L"Host: www.example.com",
-1L,
0);
assert(httpResult);
httpResult = WinHttpSendRequest( hRequest,
WINHTTP_NO_ADDITIONAL_HEADERS,
0,
WINHTTP_NO_REQUEST_DATA,
0,
0,
0 );
assert(httpResult);
httpResult = WinHttpReceiveResponse( hRequest, NULL );
assert(httpResult);
}
Edited: The class name is WebProxy. Added C# sample code. Added C++ sample code.

[Note, further to Kayode Leonard's answer: A Host property was added to the request in .Net 4.0, making this answer obsolete]
I think you are saying that you want to be able to override the ip address for a given host, without changing the host header.
For example, news.bbc.co.uk maps to IP address 212.58.226.139, but you want to be able to map this to another ip address, while still presenting the same news.bbc.co.uk "Host" http header to the overriden address. This is what you'd acheive by overriding the HOSTS file as you say, which is slightly different to Jason's answer as his won't present the original "Host" http header.
I don't believe you can do this easily (although I'm about to experiment to find out!). Certainly you can't do the following:
var request = (HttpWebRequest) WebRequest.Create("http://192.168.1.1");
request.Headers["Host"] = "news.bbc.co.uk";
as this will fail with an error saying you can't modify the "Host" header.
You probably can do it if your are willing to go down a level below the HttpWebRequest and deal at a more TCP level, but I'm not sure how you'd approach it without going down to that level.
[Edit]: Having played around with various approaches of overriding HttpWebRequest and WebHeaderCollection, I'm pretty sure it can't be done this way. However, Alexandre Jasmin's answer seems to be the solution.

I'm adding an answer that relies on curl and the --resolve command line option.
https://curl.haxx.se/docs/manpage.html#--resolve
From higher level languages, it's usually possible to invoke curl.