I have a method which is intended to download a file from an HTTP URL to a byte array:
private static byte[] DownloadFileToByteArrayWorker(HttpWebRequest Request, int bufferLength)
{
byte[] responseByes = null;
//Round up to the nearest multiple of 1024
bufferLength = AdjustBufferLength(bufferLength);
Request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
Request.ServicePoint.Expect100Continue = false;
Request.Headers.Add(HttpRequestHeader.CacheControl, "no-cache");
using (var Response = (HttpWebResponse)Request.GetResponse())
{
using (Stream ResponseStream = Response.GetResponseStream())
{
using (MemoryStream ms = new MemoryStream())
{
int count = 0;
byte[] buf = new byte[bufferLength];
while ((count = ResponseStream.Read(buf, 0, buf.Length)) > 0)
{
ms.Write(buf, 0, count);
}
responseByes = ms.ToArray();
}
}
}
return responseByes;
}
Request.GetResponse() is throwing a time out exception no matter how long I make the Timeout property of the HttpWebRequest. I can verify via my logs that the program is waiting the full Timeout period before erroring out, however, correlating my logs with the web server logs indicates that the web server is sending back a response almost immediately.
An interesting note is that when I access the same web server via the load balancer rather than directly, it downloads the file practically instantly. Also, if I access the URL via the web server directly in a web browser (no proxy needed, btw) I can download the file from individual web servers instantly that way too.
Some additional details:
I am using .NET Framework 4.7 on Windows 2012 R2.
The web server I'm trying to connect to is Apache on RHEL7. I'm not sure about the specific Apache version
I am connecting to the web server on a specific port which is reserved for HTTP traffic (a separate website is hosted on a different port number for HTTPS)
There's no web proxy
Any suggestions?
As you said your code has problem only when you call the load balancer,
I think the problem is the your client send a 100 continue request but your load balancer don't know how to handle it.
That is the reason you client doesn't send all the data right after the beginning of connection.
You can find more information about 100 continue in HTTP rfc section 8.2.3.
To fix the behavior from client side in c# you have to add this code:
ServicePointManager.Expect100Continue = false;
You can see the full documentation about this feature here.
I need help. I am creating an MVC 6 application and stuck on the part where I should consume JSON from a REST service. I could not find the way I should connect my project to the service and then consume it.
There is no way to add service reference as in previous versions and I could not find it ASP.NET 5 documentation where the policy for using 3rd party services in MVC 6 is regulated. Did someone had the same issue?
To get the JSON from RESTful service in MVC you just make a http call to the service API and parse the response with model that contains the properties of the json. You can read more about that here:
http://bitoftech.net/2014/11/18/getting-started-asp-net-5-mvc-6-web-api-entity-framework-7/
An example would look like something like this:
public YourModel MakeRequest(string requestUrl)
{
try
{
HttpWebRequest request = WebRequest.Create(requestUrl) as HttpWebRequest;
using (HttpWebResponse response = request.GetResponse() as HttpWebResponse)
{
if (response.StatusCode != HttpStatusCode.OK)
{
throw new Exception(String.Format("Server error (HTTP {0}: {1}).", response.StatusCode, response.StatusDescription));
}
JavaScriptSerializer serializer = new JavaScriptSerializer();
var responseObject = serializer.Deserialize<YourModel>(response);
return responseObject;
}
}
catch (Exception e)
{
// catch exception and log it
return null;
}
}
There is no "add a service reference" feature for REST services in ASP.NET (like it is for WSDL described ones).
There never was. You consume the service as you would consume it directly from your browser using javascript.
The difference is that you need to write similar code in .NET using any http client (HttpClient or RestSharp are the most popular).
There are some efforts to make the REST services easier to consume. Swagger is the tool I use to describe my API. It also allows to generate client code for various languages.
TL;DR version
When a transfer error occurs while writing to the request stream, I can't access the response, even though the server sends it.
Full version
I have a .NET application that uploads files to a Tomcat server, using HttpWebRequest. In some cases, the server closes the request stream prematurely (because it refuses the file for one reason or another, e.g. an invalid filename), and sends a 400 response with a custom header to indicate the cause of the error.
The problem is that if the uploaded file is large, the request stream is closed before I finish writing the request body, and I get an IOException:
Message: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host.
InnerException: SocketException: An existing connection was forcibly closed by the remote host
I can catch this exception, but then, when I call GetResponse, I get a WebException with the previous IOException as its inner exception, and a null Response property. So I can never get the response, even though the server sends it (checked with WireShark).
Since I can't get the response, I don't know what the actual problem is. From my application point of view, it looks like the connection was interrupted, so I treat it as a network-related error and retry the upload... which, of course, fails again.
How can I work around this issue and retrieve the actual response from the server? Is it even possible? To me, the current behavior looks like a bug in HttpWebRequest, or at least a severe design issue...
Here's the code I used to reproduce the problem:
var request = HttpWebRequest.CreateHttp(uri);
request.Method = "POST";
string filename = "foo\u00A0bar.dat"; // Invalid characters in filename, the server will refuse it
request.Headers["Content-Disposition"] = string.Format("attachment; filename*=utf-8''{0}", Uri.EscapeDataString(filename));
request.AllowWriteStreamBuffering = false;
request.ContentType = "application/octet-stream";
request.ContentLength = 100 * 1024 * 1024;
// Upload the "file" (just random data in this case)
try
{
using (var stream = request.GetRequestStream())
{
byte[] buffer = new byte[1024 * 1024];
new Random().NextBytes(buffer);
for (int i = 0; i < 100; i++)
{
stream.Write(buffer, 0, buffer.Length);
}
}
}
catch(Exception ex)
{
// here I get an IOException; InnerException is a SocketException
Console.WriteLine("Error writing to stream: {0}", ex);
}
// Now try to read the response
try
{
using (var response = (HttpWebResponse)request.GetResponse())
{
Console.WriteLine("{0} - {1}", (int)response.StatusCode, response.StatusDescription);
}
}
catch(Exception ex)
{
// here I get a WebException; InnerException is the IOException from the previous catch
Console.WriteLine("Error getting the response: {0}", ex);
var webEx = ex as WebException;
if (webEx != null)
{
Console.WriteLine(webEx.Status); // SendFailure
var response = (HttpWebResponse)webEx.Response;
if (response != null)
{
Console.WriteLine("{0} - {1}", (int)response.StatusCode, response.StatusDescription);
}
else
{
Console.WriteLine("No response");
}
}
}
Additional notes:
If I correctly understand the role of the 100 Continue status, the server shouldn't send it to me if it's going to refuse the file. However, it seems that this status is controlled directly by Tomcat, and can't be controlled by the application. Ideally, I'd like the server not to send me 100 Continue in this case, but according to my colleagues in charge of the back-end, there is no easy way to do it. So I'm looking for a client-side solution for now; but if you happen to know how to solve the problem on the server side, it would also be appreciated.
The app in which I encounter the issue targets .NET 4.0, but I also reproduced it with 4.5.
I'm not timing out. The exception is thrown long before the timeout.
I tried an async request. It doesn't change anything.
I tried setting the request protocol version to HTTP 1.0, with the same result.
Someone else has already filed a bug on Connect for this issue: https://connect.microsoft.com/VisualStudio/feedback/details/779622/unable-to-get-servers-error-response-when-uploading-file-with-httpwebrequest
I am out of ideas as to what can be a client side solution to your problem. But I still think the server side solution of using a custom tomcat valve can help here. I currently doesn`t have a tomcat setup where I can test this but I think a server side solution here would be along the following lines :
RFC section 8.2.3 clearly states :
Requirements for HTTP/1.1 origin servers:
- Upon receiving a request which includes an Expect request-header
field with the "100-continue" expectation, an origin server MUST
either respond with 100 (Continue) status and continue to read
from the input stream, or respond with a final status code. The
origin server MUST NOT wait for the request body before sending
the 100 (Continue) response. If it responds with a final status
code, it MAY close the transport connection or it MAY continue
to read and discard the rest of the request. It MUST NOT
perform the requested method if it returns a final status code.
So assuming tomcat confirms to the RFC, while in the custom valve you would have recieved the HTTP request header, but the request body would not be sent since the control is not yet in the servlet that reads the body.
So you can probably implement a custom valve, something similar to :
import org.apache.catalina.connector.Request;
import org.apache.catalina.connector.Response;
import org.apache.catalina.valves.ErrorReportValve;
public class CustomUploadHandlerValve extends ValveBase {
#Override
public void invoke(Request request, Response response) throws IOException, ServletException {
HttpServletRequest httpRequest = (HttpServletRequest) request;
String fileName = httpRequest.getHeader("Filename"); // get the filename or whatever other parameters required as per your code
bool validationSuccess = Validate(); // perform filename check or anyother validation here
if(!validationSuccess)
{
response = CreateResponse(); //create your custom 400 response here
request.SetResponse(response);
// return the response here
}
else
{
getNext().invoke(request, response); // to pass to the next valve/ servlet in the chain
}
}
...
}
DISCLAIMER : Again I haven`t tried this to success, need sometime and a tomcat setup to try it out ;).
Thought it might be a starting point for you.
I had the same problem. The server sends a response before the client end of the transmission of the request body, when I try to do async request. After a series of experiments, I found a workaround.
After the request stream has been received, I use reflection to check the private field _CoreResponse of the HttpWebRequest. If it is an object of class CoreResponseData, I take his private fields (using reflection): m_StatusCode, m_StatusDescription, m_ResponseHeaders, m_ContentLength. They contain information about the server's response!
In most cases, this hack works!
What are you getting in the status code and response of the second exception not the internal exception?
If a WebException is thrown, use the Response and Status properties of the exception to determine the response from the server.
http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.getresponse(v=vs.110).aspx
You are not saying what exactly version of Tomcat 7 you are using...
checked with WireShark
What do you actually see with WireShark?
Do you see the status line of response?
Do you see the complete status line, up to CR-LF characters at its end?
Is Tomcat asking for authentication credentials (401), or it is refusing file upload for some other reason (first acknowledging it with 100 but then aborting it mid-flight)?
The problem is that if the uploaded file is large, the request stream
is closed before I finish writing the request body, and I get an IOException:
If you do not want the connection to be closed but all the data transferred over the wire and swallowed at the server side, on Tomcat 7.0.55 and later it is possible to configure maxSwallowSize attribute on HTTP connector, e.g. maxSwallowSize="-1".
http://tomcat.apache.org/tomcat-7.0-doc/config/http.html
If you want to discuss Tomcat side of connection handling, you would better ask on the Tomcat users' mailing list,
http://tomcat.apache.org/lists.html#tomcat-users
At .Net side:
Is it possible to perform stream.Write() and request.GetResponse() simultaneously, from different threads?
Is it possible to performs some checks at the client side before actually uploading the file?
hmmm... i don't get it - that is EXACTLY why in many real-life scenarios large files are uploaded in chunks (and not as a single large file)
by the way: many internet servers have size limitations. for instance in tomcat that is representad by maxPostSize (as seen in this link: http://tomcat.apache.org/tomcat-5.5-doc/config/http.html)
so tweaking the server configurations seems like the easy way, but i do think that the right way is to split the file to several requests
EDIT: replace Uri.EscapeDataString with HttpServerUtility.UrlEncode
Uri.EscapeDataString(filename) // a problematic .net implementation
HttpServerUtility.UrlEncode(filename) // the proper way to do it
I am experience a pretty similar problem currently also with Tomcat and a Java client. The Tomcat REST service sends a HTTP returncode with response body before reading the whole request body. The client however fails with IOException. I inserted a HTTP Proxy on the client to sniff the protocol and actually the HTTP response is sent to the client eventually. Most likly the Tomcat closed the request input stream before sending the response.
One solution is to use a different HTTP server like Jetty which does not have this problem. The other solution is a add a Apache HTTP server with AJP in front of Tomcat. Apache HTTP server has a different handling of streams and with that the problem goes away.
I'm building an api that uses OAuth2. I've successfully unit tested all of the individual pieces but now I need to do some integration testing. Basically I want to be able to inspect the http responses from the server to make sure everything is in working order. Ideally I'd like to be able to spin up the web developement server in Visual Studio to host the site in and then make a bunch of requests to it and inspect the results.
What's the best approach to doing this and what tools should I be using?
I would recommend you deploying your application on a staging server (maybe even as a part of the build process so that this would be a single button click action) and then firing the HTTP client requests against this server, the way a real .NET client is supposed to use your API.
Create a Continuous Integration Server
install TeamCity
install Ruby
install Albacore
Have a rake script do the following
1. check out files from source control
2. build locally
3. deploy api to local iis
4. run integration tests against localhost api
This started to sound familiar. See here
Here is an example from my API integration tests. Let me know if you want more details.
I am using mspec.
I run it aganst localhost, our staging server, and our production server (a limited set of tests) to make sure all http connections are working.
public class _GET_no_criteria : specs_for_endpoint_test
{
Establish context = () =>
{
Uri = C.Endpoint;
Querystring = "";
ExecuteJsonGetRequest();
SetValidId();
};
It should_have_status_code_200_ok =()=>
IsHttp_200OK();
It should_have_categories = () =>
{
responseText.ShouldNotBeEmpty();
PutsAll(responseText);
};
}
From the base class
public static void ExecuteGetRequest(string contentType)
{
httpcontext = HttpContext.Current;
request = (HttpWebRequest)WebRequest.Create(BaseUri + Uri + Querystring);
request.Method = C.HTTP_GET;
request.ContentType = contentType;
request.Headers[C.AUTHORIZATION] = token;
// GetResponse reaises an exception on http status code 400
// We can pull response out of the exception and continue on our way
try
{
response = (HttpWebResponse)request.GetResponse();
}
catch (WebException ex)
{
response = (HttpWebResponse) ex.Response;
}
using (StreamReader reader = new StreamReader(response.GetResponseStream()))
{
responseText = reader.ReadToEnd();
reader.Close();
}
}
public static void ExecuteJsonGetRequest()
{
ExecuteGetRequest(C.CONTENT_JSON);
}
My question is similar to this one: need to call soap ws without wsdl except that my application does not use Spring so the answer was not helpful.
Here's what I have:
A web service that only accepts SOAP requests
A current endpoint URL for the web service
An outdated wsdl and xsd file
An outdated sample SOAP request file
What I need to do is:
Successfully make a SOAP request using some combination of the above.
I've tried to approach this from two different angles, with no luck so far. My background is familiarity with web services with POST and GETs, but not SOAP. After googling 'C# SOAP', I wrote the following code:
void soap(String xmlfile)
{
Stream outputStream = null;
StreamReader reader = null;
WebResponse response = null;
try
{
string text = System.IO.File.ReadAllText(xmlfile);
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("https://misapi.ercot.com/2007-08/Nodal/eEDS/EWS");
request.PreAuthenticate = true;
X509Certificate ercotCert = X509Certificate.CreateFromCertFile("D:\\Amigo\\WebSite1\\Nodal_Test_Cert.cer");
request.ClientCertificates.Clear();
request.ClientCertificates.Add(ercotCert);
ServicePointManager.ServerCertificateValidationCallback +=
new System.Net.Security.RemoteCertificateValidationCallback(customValidation);
request.Credentials = CredentialCache.DefaultNetworkCredentials;
// I don't actually have a SOAPAction, but have tried adding a fake one just to see
//request.Headers.Add("SOAPAction", "http://www.ercot.com/Nodal/Payload");
request.Method = "POST";
request.ContentType = "text/xml;charset=\"utf-8\"";
request.ContentLength = text.Length;
StreamWriter writer = new StreamWriter(request.GetRequestStream(), System.Text.Encoding.ASCII);
writer.Write(text);
writer.Close();
response = request.GetResponse();
outputStream = response.GetResponseStream();
reader = new StreamReader(outputStream);
Response.Write(reader.ReadToEnd());
}
catch (WebException e)
{
// This is where it ends up, with Status="ProtocolError"
}
catch (System.Web.Services.Protocols.SoapException soapE)
{
// Never gets in here
}
catch (Exception e)
{
// Never gets in here
}
finally
{
if (outputStream != null)
outputStream.Close();
if(reader != null)
reader.Close();
if(response != null)
response.Close();
}
}
private static bool customValidation(object sender, X509Certificate cert, X509Chain chain, System.Net.Security.SslPolicyErrors error)
{
return true;
}
This yields a 500 Internal Server Error. It throws a WebException which contains no other message or inner exception, but the Status is 'ProtocolError'. I have tried other permutations including using XmlDocument and other content types, but none have worked.
The other thing I've tried is using "Add Web Reference" in Visual Studio. Putting in the endpoint URL doesn't work and gives a soap fault. If, instead, I point to the local copy of my outdated wsdl, then it will add the WebReference but won't let me use it due to numerous errors that I cannot correct. My guess is that these errors are due to the wsdl being outdated, things like the namespace not matching or being unable to find things at the URLs included. If I replace those URLs with the current web service endpoint URL, it still does not work.
If anyone could pinpoint a problem in my code, or direct me on how to get the "Add Web Reference" working, I would be greatly greatly appreciated!
Thanks in advance!
SOAP is just the format for the payload that you POST to the service, that's all. There's a defined specification for fields and namespaces and such, and there are different SOAP versions, but there's really not that much to it. (Other than it being remarkably verbose for it's usage, but that's a different topic.)
You need to start with a current WSDL. If you know you're working WSDL is outdated, that means that something the webservice is expecting (required) or could work with (optional) is different from your definition. And the WSDL is your contractual gateway into the webservice. You need to get a current WSDL.
As luck would have it, when I navigate to https://misapi.ercot.com/2007-08/Nodal/eEDS/EWS/?WSDL (which I derived by appending "?WSDL" to the end of the url), after skipping past the certificate error, bingo -- there's the WSDL used by the service. You can either (a) save it locally, or (b) reference it directly from Visual Studio in building a web service client. Because of the cert error, I would recommend saving it locally and building from there.
This should get you started.