We are using one load balance Server which redirect us to another four different server and other two servers which are not on the farm of load balance server. The configuration looks like completed perfectly from the administrator of the system.
The problem I face is that when I would like to have an HTTP web response and take the response headers I faced an issue. The problem I faced is that the Load balanced "link" which I use from load balance server does not give me the response I expect. I use the below code as example.
try
{
var myHttpWebRequest = WebRequest.Create(url);
myHttpWebRequest.Credentials = credentials;
var myHttpWebResponse = myHttpWebRequest.GetResponse();
myHttpWebResponse.Close();
}
catch (WebException e)
{
Console.WriteLine("WebException: " + url);
Console.WriteLine("Web Exception Status: " + e.Status);
Console.WriteLine("\n");
if (e.Status == WebExceptionStatus.ProtocolError)
{
Console.WriteLine(((HttpWebResponse)e.Response).Headers);
}
}
Now, the WebExceptionStatus I take when execute the code on the load balanced server is "SendFailure" which does not give me any "response.headers" result.
If I install and use "curl"
curl [load balance server url] -k -I
on the command line I will take headers when I use load balance server URL.
In conclusion, my target is I would like to take and use theWebExceptionStatus, "X-FEServer","www-Authenticate" and "Date" for some purposes but it is not possible with the c# approach as above to be retrieved.
Please, make sure that you understand the load balanced server approach before any comment or answer. Any suggestion will be helpful
Related
I'm good with the code, it works great for other solutions of mine. I have a knowledge gap as I do not understand what constitutes a URI. This should work, but does not:
https://www.nasdaq.com/screening/companies-by-name.aspx?letter=0&exchange=nasdaq&render=download
Now I'm thinking that this is not a file right? Throwing the above at a browser provides a file though. The exception message is "The underlying connection was closed: An unexpected error occurred on a receive."
String address = "https://www.nasdaq.com/screening/companies-by-name.aspx?letter=0&exchange=nasdaq&render=download";
.....
using (WebClient Client = new WebClient())
{
try
{
Client.DownloadFile(address, destPath + filename);
}
catch (Exception ex)
{
Log.Line("Error: " + ex.Message);
return 1;
}
}
The URI:
this link
You've got a perfectly valid URI. The target server may respond to requests in a different way than you expect though. For example depending on your web client. To debug issues like this use curl.
curl -v https://www.nasdaq.com/screening/companies-by-name.aspx?letter=0&exchange=nasdaq&render=download
The above command shows you that the server does not reply with the expected csv file. That's not a problem in your code. You can try to pretend a different user agent using the curl -H flag or set some redirection options until you get there.
In your specific case it seems to be the header Accept-Encoding: gzip that solves the issue.
TL;DR version
When a transfer error occurs while writing to the request stream, I can't access the response, even though the server sends it.
Full version
I have a .NET application that uploads files to a Tomcat server, using HttpWebRequest. In some cases, the server closes the request stream prematurely (because it refuses the file for one reason or another, e.g. an invalid filename), and sends a 400 response with a custom header to indicate the cause of the error.
The problem is that if the uploaded file is large, the request stream is closed before I finish writing the request body, and I get an IOException:
Message: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host.
InnerException: SocketException: An existing connection was forcibly closed by the remote host
I can catch this exception, but then, when I call GetResponse, I get a WebException with the previous IOException as its inner exception, and a null Response property. So I can never get the response, even though the server sends it (checked with WireShark).
Since I can't get the response, I don't know what the actual problem is. From my application point of view, it looks like the connection was interrupted, so I treat it as a network-related error and retry the upload... which, of course, fails again.
How can I work around this issue and retrieve the actual response from the server? Is it even possible? To me, the current behavior looks like a bug in HttpWebRequest, or at least a severe design issue...
Here's the code I used to reproduce the problem:
var request = HttpWebRequest.CreateHttp(uri);
request.Method = "POST";
string filename = "foo\u00A0bar.dat"; // Invalid characters in filename, the server will refuse it
request.Headers["Content-Disposition"] = string.Format("attachment; filename*=utf-8''{0}", Uri.EscapeDataString(filename));
request.AllowWriteStreamBuffering = false;
request.ContentType = "application/octet-stream";
request.ContentLength = 100 * 1024 * 1024;
// Upload the "file" (just random data in this case)
try
{
using (var stream = request.GetRequestStream())
{
byte[] buffer = new byte[1024 * 1024];
new Random().NextBytes(buffer);
for (int i = 0; i < 100; i++)
{
stream.Write(buffer, 0, buffer.Length);
}
}
}
catch(Exception ex)
{
// here I get an IOException; InnerException is a SocketException
Console.WriteLine("Error writing to stream: {0}", ex);
}
// Now try to read the response
try
{
using (var response = (HttpWebResponse)request.GetResponse())
{
Console.WriteLine("{0} - {1}", (int)response.StatusCode, response.StatusDescription);
}
}
catch(Exception ex)
{
// here I get a WebException; InnerException is the IOException from the previous catch
Console.WriteLine("Error getting the response: {0}", ex);
var webEx = ex as WebException;
if (webEx != null)
{
Console.WriteLine(webEx.Status); // SendFailure
var response = (HttpWebResponse)webEx.Response;
if (response != null)
{
Console.WriteLine("{0} - {1}", (int)response.StatusCode, response.StatusDescription);
}
else
{
Console.WriteLine("No response");
}
}
}
Additional notes:
If I correctly understand the role of the 100 Continue status, the server shouldn't send it to me if it's going to refuse the file. However, it seems that this status is controlled directly by Tomcat, and can't be controlled by the application. Ideally, I'd like the server not to send me 100 Continue in this case, but according to my colleagues in charge of the back-end, there is no easy way to do it. So I'm looking for a client-side solution for now; but if you happen to know how to solve the problem on the server side, it would also be appreciated.
The app in which I encounter the issue targets .NET 4.0, but I also reproduced it with 4.5.
I'm not timing out. The exception is thrown long before the timeout.
I tried an async request. It doesn't change anything.
I tried setting the request protocol version to HTTP 1.0, with the same result.
Someone else has already filed a bug on Connect for this issue: https://connect.microsoft.com/VisualStudio/feedback/details/779622/unable-to-get-servers-error-response-when-uploading-file-with-httpwebrequest
I am out of ideas as to what can be a client side solution to your problem. But I still think the server side solution of using a custom tomcat valve can help here. I currently doesn`t have a tomcat setup where I can test this but I think a server side solution here would be along the following lines :
RFC section 8.2.3 clearly states :
Requirements for HTTP/1.1 origin servers:
- Upon receiving a request which includes an Expect request-header
field with the "100-continue" expectation, an origin server MUST
either respond with 100 (Continue) status and continue to read
from the input stream, or respond with a final status code. The
origin server MUST NOT wait for the request body before sending
the 100 (Continue) response. If it responds with a final status
code, it MAY close the transport connection or it MAY continue
to read and discard the rest of the request. It MUST NOT
perform the requested method if it returns a final status code.
So assuming tomcat confirms to the RFC, while in the custom valve you would have recieved the HTTP request header, but the request body would not be sent since the control is not yet in the servlet that reads the body.
So you can probably implement a custom valve, something similar to :
import org.apache.catalina.connector.Request;
import org.apache.catalina.connector.Response;
import org.apache.catalina.valves.ErrorReportValve;
public class CustomUploadHandlerValve extends ValveBase {
#Override
public void invoke(Request request, Response response) throws IOException, ServletException {
HttpServletRequest httpRequest = (HttpServletRequest) request;
String fileName = httpRequest.getHeader("Filename"); // get the filename or whatever other parameters required as per your code
bool validationSuccess = Validate(); // perform filename check or anyother validation here
if(!validationSuccess)
{
response = CreateResponse(); //create your custom 400 response here
request.SetResponse(response);
// return the response here
}
else
{
getNext().invoke(request, response); // to pass to the next valve/ servlet in the chain
}
}
...
}
DISCLAIMER : Again I haven`t tried this to success, need sometime and a tomcat setup to try it out ;).
Thought it might be a starting point for you.
I had the same problem. The server sends a response before the client end of the transmission of the request body, when I try to do async request. After a series of experiments, I found a workaround.
After the request stream has been received, I use reflection to check the private field _CoreResponse of the HttpWebRequest. If it is an object of class CoreResponseData, I take his private fields (using reflection): m_StatusCode, m_StatusDescription, m_ResponseHeaders, m_ContentLength. They contain information about the server's response!
In most cases, this hack works!
What are you getting in the status code and response of the second exception not the internal exception?
If a WebException is thrown, use the Response and Status properties of the exception to determine the response from the server.
http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.getresponse(v=vs.110).aspx
You are not saying what exactly version of Tomcat 7 you are using...
checked with WireShark
What do you actually see with WireShark?
Do you see the status line of response?
Do you see the complete status line, up to CR-LF characters at its end?
Is Tomcat asking for authentication credentials (401), or it is refusing file upload for some other reason (first acknowledging it with 100 but then aborting it mid-flight)?
The problem is that if the uploaded file is large, the request stream
is closed before I finish writing the request body, and I get an IOException:
If you do not want the connection to be closed but all the data transferred over the wire and swallowed at the server side, on Tomcat 7.0.55 and later it is possible to configure maxSwallowSize attribute on HTTP connector, e.g. maxSwallowSize="-1".
http://tomcat.apache.org/tomcat-7.0-doc/config/http.html
If you want to discuss Tomcat side of connection handling, you would better ask on the Tomcat users' mailing list,
http://tomcat.apache.org/lists.html#tomcat-users
At .Net side:
Is it possible to perform stream.Write() and request.GetResponse() simultaneously, from different threads?
Is it possible to performs some checks at the client side before actually uploading the file?
hmmm... i don't get it - that is EXACTLY why in many real-life scenarios large files are uploaded in chunks (and not as a single large file)
by the way: many internet servers have size limitations. for instance in tomcat that is representad by maxPostSize (as seen in this link: http://tomcat.apache.org/tomcat-5.5-doc/config/http.html)
so tweaking the server configurations seems like the easy way, but i do think that the right way is to split the file to several requests
EDIT: replace Uri.EscapeDataString with HttpServerUtility.UrlEncode
Uri.EscapeDataString(filename) // a problematic .net implementation
HttpServerUtility.UrlEncode(filename) // the proper way to do it
I am experience a pretty similar problem currently also with Tomcat and a Java client. The Tomcat REST service sends a HTTP returncode with response body before reading the whole request body. The client however fails with IOException. I inserted a HTTP Proxy on the client to sniff the protocol and actually the HTTP response is sent to the client eventually. Most likly the Tomcat closed the request input stream before sending the response.
One solution is to use a different HTTP server like Jetty which does not have this problem. The other solution is a add a Apache HTTP server with AJP in front of Tomcat. Apache HTTP server has a different handling of streams and with that the problem goes away.
As you all know there is many file host websites, is there a way to process the http link of a of file on one of those sites and retrieve a result if the file exists or if the http link even exists or not. I know that maybe some of those file host websites uses their own APIs but i want a more generic way.
Edit:
So as i understand there is no file on a server, it's just that i have to read the response and read it properly. I want to ask another thing, what about redirection, does that mean if i got the response of a link that redirects to other link, i will get the final target from the response ?
You can find out if a file exist using the exists method:
bool System.IO.File.Exists(string path)
///
in order to find out if a file exist on a remove server you can try this:
WebRequest request;
WebResponse response;
String strMSG = string.Empty;
request = WebRequest.Create(new Uri(“http://www.yoururl.com/yourfile.jpg”));
request.Method = “HEAD”;
try
{
response = request.GetResponse();
strMSG = string.Format(“{0} {1}”, response.ContentLength, response.ContentType);
}
catch (Exception ex)
{
//In case of File not Exist Server return the (404) Error
strMSG = ex.Message;
}
see this:
If I understand you correctly, you're trying to tell if a given URL has content.
Use the
WebClient
class.
Call the url, if you receive a 200, you're good to go. A 404 exception or similar probably means the link is no good.
Or, even better way to do this is to do a HEAD http request. See here for more info on that.
I need to check that our visitors are using HTTPS. In BasePage I check if the request is coming via HTTPS. If it's not, I redirect back with HTTPS. However, when someone comes to the site and this function is used, I get the error:
System.Web.HttpException: Server
cannot append header after HTTP
headers have been sent. at
System.Web.HttpResponse.AppendHeader(String
name, String value) at
System.Web.HttpResponse.AddHeader(String
name, String value) at
Premier.Payment.Website.Generic.BasePage..ctor()
Here is the code I started with:
// If page not currently SSL
if (HttpContext.Current.Request.ServerVariables["HTTPS"].Equals("off"))
{
// If SSL is required
if (GetConfigSetting("SSLRequired").ToUpper().Equals("TRUE"))
{
string redi = "https://" +
HttpContext.Current.Request.ServerVariables["SERVER_NAME"].ToString() +
HttpContext.Current.Request.ServerVariables["SCRIPT_NAME"].ToString() +
"?" + HttpContext.Current.Request.ServerVariables["QUERY_STRING"].ToString();
HttpContext.Current.Response.Redirect(redi.ToString());
}
}
I also tried adding this above it (a bit I used in another site for a similar problem):
// Wait until page is copletely loaded before sending anything since we re-build
HttpContext.Current.Response.BufferOutput = true;
I am using c# in .NET 3.5 on IIS 6.
Chad,
Did you try ending the output when you redirect? There is a second parameter that you'd set to true to tell the output to stop when the redirect header is issued. Or, if you are buffering the output then maybe you need to clear the buffer before doing the redirect so the headers are not sent out along with the redirect header.
Brian
This error usually means that something has bee written to the response stream before a redirection is initiated. So you should make sure that the test for https is done fairly high up in the page load function.
I'm busy writing a class that monitors the status of RAS connections. I need to test to make sure that the connection is not only connected, but also that it can communicate with my web service. Since this class will be used in many future projects, I'd like a way to test the connection to the webservice without knowing anything about it.
I was thinking of passing the URL to the class so that it at least knows where to find it. Pinging the server is not a sufficient test. It is possible for the server to be available, but the service to be offline.
How can I effectively test that I'm able to get a response from the web service?
You could try the following which tests the web site's existence:
public static bool ServiceExists(
string url,
bool throwExceptions,
out string errorMessage)
{
try
{
errorMessage = string.Empty;
// try accessing the web service directly via it's URL
HttpWebRequest request =
WebRequest.Create(url) as HttpWebRequest;
request.Timeout = 30000;
using (HttpWebResponse response =
request.GetResponse() as HttpWebResponse)
{
if (response.StatusCode != HttpStatusCode.OK)
throw new Exception("Error locating web service");
}
// try getting the WSDL?
// asmx lets you put "?wsdl" to make sure the URL is a web service
// could parse and validate WSDL here
}
catch (WebException ex)
{
// decompose 400- codes here if you like
errorMessage =
string.Format("Error testing connection to web service at" +
" \"{0}\":\r\n{1}", url, ex);
Trace.TraceError(errorMessage);
if (throwExceptions)
throw new Exception(errorMessage, ex);
}
catch (Exception ex)
{
errorMessage =
string.Format("Error testing connection to web service at " +
"\"{0}\":\r\n{1}", url, ex);
Trace.TraceError(errorMessage);
if (throwExceptions)
throw new Exception(errorMessage, ex);
return false;
}
return true;
}
You are right that pinging the server isn't sufficient. The server can be up, but the web service unavailable due to a number of reasons.
To monitor our web service connections, I created a IMonitoredService interface that has a method CheckService(). The wrapper class for each web service implements this method to call an innocuous method on the web service and reports if the service is up or not. This allows any number of services to be monitored with out the code responsible for the monitoring knowing the details of the service.
If you know a bit about what the web service will return from accessing the URL directly, you could try just using the URL. For example, Microsoft's asmx file returns a summary of the web service. Other implementations may behave differently though.
The tip: create a interface/baseclass with method "InvokeWithSomeParameters". The meaning of "SomeParameters" should be a "parameters which 100% does not affect any important state".
I think, there are 2 cases:
Simple webservice, which does not affect any data on server. For example: GetCurrentTime(). This web service can be called without parameters.
Complex webservice, which can affect some daty on server. For example: Enlist pending tasks. You fill-up parameter with values which 100% throws a exception (resp. does not change affect pending tasks), if you got some exception like "ArgumentException", you know the service is alive.
I don't think, this is most clear solution, but it should works.
How about opening a TCP/IP connection to the port used by the webservice? If the connection works, the RAS connection, the rest of the network and the host are all working. The webservice is almost certainly running too.
If it is a Microsoft SOAP or WCF service and service discovery is allowed, you can request the web page serviceurl + "?disco" for discovery. If what is returned is a valid XML document , you know the service is alive and well. A non-Microsoft SOAP service that does not allow ?disco will probably return valid XML too.
Sample code:
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(URL + "?disco");
request.ClientCertificates.Add(
new X509Certificate2(#"c:\mycertpath\mycert.pfx", "<privatekeypassword>")); // If server requires client certificate
request.Timeout = 300000; // 5 minutes
using (WebResponse response = request.GetResponse())
using (Stream stream = response.GetResponseStream())
using (StreamReader sr = new StreamReader(stream, Encoding.UTF8))
{
XmlDocument xd = new XmlDocument();
xd.LoadXml(sr.ReadToEnd());
return xd.DocumentElement.ChildNodes.Count > 0;
}
If the web server is present, but the service does not exist, an exception will be raised quickly for the 404 error. The fairly long time-out in the example is to allow a slow WCF service to restart after a long period of inactivity or after iisreset. If the client needs to be responsive, you could poll with a shorter timeout until the service is available.