Develop Reference

Simple WebClient works locally, but not when deployed to Azure

I have a very simple test app that creates a WebClient and gets a response from a third-party API (sensitive info removed):
using (WebClient client = new WebClient())
{
var baseAddress = new Uri("https://test-url");
client.Headers["Content-Type"] = "application/json";
var parms = Encoding.Default.GetBytes(
"{ \"username\":\"<USERNAME>\"," +
"\"password\":\<PASSWORD>\"," +
"\"client_id\":\"<CLIENT_ID>\"," +
"\"client_secret\":\"<CLIENT_SECRET>\"" +
"}"
);
// Get access token
var responseObject = new ResponseObject();
var responseBytes = new byte[] { };
string responseBody;
try
{
responseBytes = client.UploadData(baseAddress, "POST", parms);
responseBody = Encoding.UTF8.GetString(responseBytes);
responseObject = JsonConvert.DeserializeObject<ResponseObject>(responseBody);
}
catch (WebException ex)
{
var resp = (HttpWebResponse)ex.Response;
Console.WriteLine(resp.StatusDescription);
}
}
This code works perfectly when I run it locally, multiple times. I get a response back and I'm able to extract the information I need from it.
However, once I deploy this code to Azure, I try to call the API with POSTMAN and I get an error telling me that there's an "Object reference not set to an instance of an object." error on this line in the try/catch:
var resp = (HttpWebResponse)ex.Response;
So, I attached a remote debugger to the API and stepped through this small bit of code. What's weird to me is that I can see right when the program jumps into the catch portion of the code, but there is no ex Exception object. What I mean by that is that I can hover over the ex variable in debug and nothing appears. Even hovering over ex.Response shows nothing. So, since there is nothing there (i.e. ex doesn't appear to exist, not even a null value) the application bombs with the null reference error.
My thought is that maybe something in Azure is blocking the URL, since I can run this locally without issue. I asked our network guys here about it and they said there shouldn't be any kind of constraints set up that they are aware of to prevent the call from going out. The weird thing is that I turned on diagnostic logging in Azure for this app and nothing is showing up. Even though I can hit the app with a browser or PostMan, nothing is appearing in the log. So, that's another weird thing. I'd expect to be able to see something.
Anyway, I'm hoping that perhaps someone has ran into something like this before and can point me to an article/solution/blog that may shed light on this and get me to a working solution.
Thanks!
EDIT: Added a screenshot of the error message as it appears in the browser:
EDIT: Added the line on which the app is throwing the exception below:
responseBytes = client.UploadData(baseAddress, "POST", parms);
Immediately after trying to execute this line, it jumps to the catch, but there is no WebException, it appears.
EDIT: I've found the solution and added it as an answer below.

I found a solution that worked and it took one line of code:
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
So, basically, I just had to set the SecurityProtocol. I don't know how this was working locally without it, but running it on Azure requires it to be set, apparently.

c# Rest client response error

I am trying to develop a rest client using RestSharp in C#.
Code:
var client = new RestClient("url goes here");
var response = client.Execute(new RestRequest()) as RestResponse;
Console.WriteLine(response.ResponseStatus);//Coming as Error
Console.WriteLine(response.StatusCode);//Coming as 0
I am not getting any compilation or runtime exceptions but the ResponseStatus is coming as "Error" and Status Code as "0" in the console.
can you anyone help me understand the reason for this?
Your inputs on this would be really helpful.
Thank you.

HTTP response 0 indicates that client could not connect with server and hence forth time out happened.
which means that you have a problem sending your request to the given URL, either by wrong URL or any other reason and hitting a timeout, this is why you receive status code of 0, which is not a standard HTTP status code.
for standard status code table see:
Status code table.

How to get the HTTP response when the request stream was closed during transfer

TL;DR version
When a transfer error occurs while writing to the request stream, I can't access the response, even though the server sends it.
Full version
I have a .NET application that uploads files to a Tomcat server, using HttpWebRequest. In some cases, the server closes the request stream prematurely (because it refuses the file for one reason or another, e.g. an invalid filename), and sends a 400 response with a custom header to indicate the cause of the error.
The problem is that if the uploaded file is large, the request stream is closed before I finish writing the request body, and I get an IOException:
Message: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host.
InnerException: SocketException: An existing connection was forcibly closed by the remote host
I can catch this exception, but then, when I call GetResponse, I get a WebException with the previous IOException as its inner exception, and a null Response property. So I can never get the response, even though the server sends it (checked with WireShark).
Since I can't get the response, I don't know what the actual problem is. From my application point of view, it looks like the connection was interrupted, so I treat it as a network-related error and retry the upload... which, of course, fails again.
How can I work around this issue and retrieve the actual response from the server? Is it even possible? To me, the current behavior looks like a bug in HttpWebRequest, or at least a severe design issue...
Here's the code I used to reproduce the problem:
var request = HttpWebRequest.CreateHttp(uri);
request.Method = "POST";
string filename = "foo\u00A0bar.dat"; // Invalid characters in filename, the server will refuse it
request.Headers["Content-Disposition"] = string.Format("attachment; filename*=utf-8''{0}", Uri.EscapeDataString(filename));
request.AllowWriteStreamBuffering = false;
request.ContentType = "application/octet-stream";
request.ContentLength = 100 * 1024 * 1024;
// Upload the "file" (just random data in this case)
try
{
using (var stream = request.GetRequestStream())
{
byte[] buffer = new byte[1024 * 1024];
new Random().NextBytes(buffer);
for (int i = 0; i < 100; i++)
{
stream.Write(buffer, 0, buffer.Length);
}
}
}
catch(Exception ex)
{
// here I get an IOException; InnerException is a SocketException
Console.WriteLine("Error writing to stream: {0}", ex);
}
// Now try to read the response
try
{
using (var response = (HttpWebResponse)request.GetResponse())
{
Console.WriteLine("{0} - {1}", (int)response.StatusCode, response.StatusDescription);
}
}
catch(Exception ex)
{
// here I get a WebException; InnerException is the IOException from the previous catch
Console.WriteLine("Error getting the response: {0}", ex);
var webEx = ex as WebException;
if (webEx != null)
{
Console.WriteLine(webEx.Status); // SendFailure
var response = (HttpWebResponse)webEx.Response;
if (response != null)
{
Console.WriteLine("{0} - {1}", (int)response.StatusCode, response.StatusDescription);
}
else
{
Console.WriteLine("No response");
}
}
}
Additional notes:
If I correctly understand the role of the 100 Continue status, the server shouldn't send it to me if it's going to refuse the file. However, it seems that this status is controlled directly by Tomcat, and can't be controlled by the application. Ideally, I'd like the server not to send me 100 Continue in this case, but according to my colleagues in charge of the back-end, there is no easy way to do it. So I'm looking for a client-side solution for now; but if you happen to know how to solve the problem on the server side, it would also be appreciated.
The app in which I encounter the issue targets .NET 4.0, but I also reproduced it with 4.5.
I'm not timing out. The exception is thrown long before the timeout.
I tried an async request. It doesn't change anything.
I tried setting the request protocol version to HTTP 1.0, with the same result.
Someone else has already filed a bug on Connect for this issue: https://connect.microsoft.com/VisualStudio/feedback/details/779622/unable-to-get-servers-error-response-when-uploading-file-with-httpwebrequest

I am out of ideas as to what can be a client side solution to your problem. But I still think the server side solution of using a custom tomcat valve can help here. I currently doesn`t have a tomcat setup where I can test this but I think a server side solution here would be along the following lines :
RFC section 8.2.3 clearly states :
Requirements for HTTP/1.1 origin servers:
- Upon receiving a request which includes an Expect request-header
field with the "100-continue" expectation, an origin server MUST
either respond with 100 (Continue) status and continue to read
from the input stream, or respond with a final status code. The
origin server MUST NOT wait for the request body before sending
the 100 (Continue) response. If it responds with a final status
code, it MAY close the transport connection or it MAY continue
to read and discard the rest of the request. It MUST NOT
perform the requested method if it returns a final status code.
So assuming tomcat confirms to the RFC, while in the custom valve you would have recieved the HTTP request header, but the request body would not be sent since the control is not yet in the servlet that reads the body.
So you can probably implement a custom valve, something similar to :
import org.apache.catalina.connector.Request;
import org.apache.catalina.connector.Response;
import org.apache.catalina.valves.ErrorReportValve;
public class CustomUploadHandlerValve extends ValveBase {
#Override
public void invoke(Request request, Response response) throws IOException, ServletException {
HttpServletRequest httpRequest = (HttpServletRequest) request;
String fileName = httpRequest.getHeader("Filename"); // get the filename or whatever other parameters required as per your code
bool validationSuccess = Validate(); // perform filename check or anyother validation here
if(!validationSuccess)
{
response = CreateResponse(); //create your custom 400 response here
request.SetResponse(response);
// return the response here
}
else
{
getNext().invoke(request, response); // to pass to the next valve/ servlet in the chain
}
}
...
}
DISCLAIMER : Again I haven`t tried this to success, need sometime and a tomcat setup to try it out ;).
Thought it might be a starting point for you.

I had the same problem. The server sends a response before the client end of the transmission of the request body, when I try to do async request. After a series of experiments, I found a workaround.
After the request stream has been received, I use reflection to check the private field _CoreResponse of the HttpWebRequest. If it is an object of class CoreResponseData, I take his private fields (using reflection): m_StatusCode, m_StatusDescription, m_ResponseHeaders, m_ContentLength. They contain information about the server's response!
In most cases, this hack works!

What are you getting in the status code and response of the second exception not the internal exception?
If a WebException is thrown, use the Response and Status properties of the exception to determine the response from the server.
http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.getresponse(v=vs.110).aspx

You are not saying what exactly version of Tomcat 7 you are using...
checked with WireShark
What do you actually see with WireShark?
Do you see the status line of response?
Do you see the complete status line, up to CR-LF characters at its end?
Is Tomcat asking for authentication credentials (401), or it is refusing file upload for some other reason (first acknowledging it with 100 but then aborting it mid-flight)?
The problem is that if the uploaded file is large, the request stream
is closed before I finish writing the request body, and I get an IOException:
If you do not want the connection to be closed but all the data transferred over the wire and swallowed at the server side, on Tomcat 7.0.55 and later it is possible to configure maxSwallowSize attribute on HTTP connector, e.g. maxSwallowSize="-1".
http://tomcat.apache.org/tomcat-7.0-doc/config/http.html
If you want to discuss Tomcat side of connection handling, you would better ask on the Tomcat users' mailing list,
http://tomcat.apache.org/lists.html#tomcat-users
At .Net side:
Is it possible to perform stream.Write() and request.GetResponse() simultaneously, from different threads?
Is it possible to performs some checks at the client side before actually uploading the file?

hmmm... i don't get it - that is EXACTLY why in many real-life scenarios large files are uploaded in chunks (and not as a single large file)
by the way: many internet servers have size limitations. for instance in tomcat that is representad by maxPostSize (as seen in this link: http://tomcat.apache.org/tomcat-5.5-doc/config/http.html)
so tweaking the server configurations seems like the easy way, but i do think that the right way is to split the file to several requests
EDIT: replace Uri.EscapeDataString with HttpServerUtility.UrlEncode
Uri.EscapeDataString(filename) // a problematic .net implementation
HttpServerUtility.UrlEncode(filename) // the proper way to do it

I am experience a pretty similar problem currently also with Tomcat and a Java client. The Tomcat REST service sends a HTTP returncode with response body before reading the whole request body. The client however fails with IOException. I inserted a HTTP Proxy on the client to sniff the protocol and actually the HTTP response is sent to the client eventually. Most likly the Tomcat closed the request input stream before sending the response.
One solution is to use a different HTTP server like Jetty which does not have this problem. The other solution is a add a Apache HTTP server with AJP in front of Tomcat. Apache HTTP server has a different handling of streams and with that the problem goes away.

Why does setting WebOperationContext.Current.OutgoingResponse.StatusCode = HttpStatusCode.Redirect cause SecurityException?

I'm working on a Rest Service in .Net 4, and I need to perform a redirect.
A little description of what I'm doing: I have a silverlight application that contains a button. When I click the button, I send a POST request to my REST service with some information. From that information, I create a URL and (try to) redirect the browser. Here's the code for the method in the service:
[WebInvoke(Method = "POST", UriTemplate = "OpenBinder")]
public void OpenBinder(int someId)
{
string url = getUrl(someId);
if (WebOperationContext.Current != null)
{
WebOperationContext.Current.OutgoingResponse.Location = url;
WebOperationContext.Current.OutgoingResponse.StatusCode = HttpStatusCode.Redirect;
}
}
This seems to execute correctly, but when I call EndGetResponse on the client, I see that a "Security Error" occurred. It's System.Security.SecurityException, but I don't get any more details than that. Any ideas on what could be going on here?

Without more info, I am not sure what your specific security error is, but generally with this type of situation, your redirect should happen on the client side in the response handler. Can you restructure your code to have the client redirect?

OK, so my code was actually working correctly. When I looked through Fiddler, I noticed it was it was making the correct request to the Url. The problem was clientacesspolicy.xml was stopping it.

HttpWebRequest type "GET" returning error 400

It seems to be occurring only one machine and none of the other machines.
HttpWebRequest myRequest =(HttpWebRequest)WebRequest.Create("https://connect.zystemsgo.com/auto/");
myRequest.Method = "GET";
SetCertificatePolicy();
Application.DoEvents();
WebResponse myResponse = myRequest.GetResponse();
StreamReader sr = new StreamReader(myResponse.GetResponseStream(),System.Text.Encoding.UTF8);
string result = sr.ReadToEnd();
I tried searching other 400 request errors, but it is not clear. How do I go about debugging this?

HTTP Error 400 means Bad Request. This is being returned by the server.
Usually, when I'm debugging HTTP requests, I use Fiddler to monitor the requests and responses and find out what's going on. It never fails.

(Not really an answer, but too big for comment)
For what it's worth, I ran the following Python code (too lazy to spin up C# :), and it worked fine:
import httplib
conn = httplib.HTTPSConnection('connect.zystemsgo.com')
conn.request('GET', '/auto/')
resp = conn.getresponse()
data = resp.read()
print data # expected ouput, just like visiting in a browser
print resp.status # 200
Are you sure you are showing us the URL that is actually failing, or is your code a more general example?
Perhaps the server certificate is not installed on that machine? I wouldn't expect a HTTP 400 in that case, but it's the only thing I can think of so far...

it is a bad request error .Are there no parameters in the request?
Can you post the response message,it will give some idea of what is going wrong.

The code that i supplied in the comment above works.
WebClient webClient = new WebClient();
webClient.DownloadFile("Your complete url for the file", #"c:\myfile.txt");
you need to have permission to write in the directory of your choice.
You could also try and use the async download if you want.I am not getting why it would not work on a certain machine.