Develop Reference

HttpClient post method throws "System.IO.IOException: Unable to read data from the transport connection: The connection was closed." exception

I have a problem with calling a external service with post method via HttpClient. When I call the external service to get data the HttpClient throws following exception:
System.IO.IOException: Unable to read data from the transport connection: The connection was closed.
This error only occurs when the response message contains larger amount of data. When I prepare requests that return a JSON with an empty array in it the post call proceeds correctly. But when I try a request, which result should be populated with larger amount of data I get this exception.
I have searched for the solution of this problem on Internet, and I found a solution that implies, that I should set properly System.Net.ServicePointManager.SecurityProtocol correctly with following line:
System.Net.ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12;
But this didn’t help in my case. I noticed that when I send my request through Postman, the service responds correctly, no matter how big the response payload will be. On Postman I checked the headers in requests and after a while I noticed, that when I do not include Accept-Encoding: gzip header in my request, Postman will have too problems with returning response.
I tried to use this in my code and force HttpClient to use Accept-Encoding header but with no success. Below is my current HttpClient configuration:
using (HttpClient client = new HttpClient(new HttpClientHandler
{
AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate,
}))
{
client.DefaultRequestHeaders.AcceptEncoding.Add(new StringWithQualityHeaderValue("gzip"));
client.DefaultRequestHeaders.AcceptEncoding.Add(new StringWithQualityHeaderValue("deflate"));
client.BaseAddress = _uri;
client.DefaultRequestHeaders.Accept.Clear();
client.Timeout = TimeSpan.FromSeconds(_timeout);
client.DefaultRequestHeaders.Add("ApiKey", _apiKey);
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
client.DefaultRequestHeaders.ConnectionClose = false;
System.Net.ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12;
HttpResponseMessage result = await client.PostAsync(uri, content)
if (result.IsSuccessStatusCode)
{
string resultString = await result.Content.ReadAsStringAsync();
model = JsonConvert.DeserializeObject<T>(resultString);
}
else
{
await ResponseErrorThrow(result);
}
return model;
}
I have no more ideas at the moment. Can you help me with this problem?

I recently faced the same issue and through logging around our retry logic we found that the request had gone through.
Through some research I found that the issue might be because of client side connection being open to more than the default timeout of 100s.
Unable to read data from the transport connection: Operation canceled
The client code tries to read the response even when the remote connection is closed and thats when it throws this exception.
C# Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. Reading networkstream
As suggested in the answer above, the best way is to handle the exception thrown and check (if you can) that the request went through.
There are ways to check tcp connection before reading from it but one must remember that any check is at that point in time. A check can return a connection-alive but that still doesn't mean the connection wouldn't die out while reading.

HttpClient timeouts without reason

In my .net Application I am using the Microsoft's HttpClient to send Requests to a server. In my team we have a problem that several people get a timeout on each single request without any knowledgable reason.
Following code is used to send the request:
var _httpClient = new HttpClient();
// Here are some values that are identical on each system.
var values = new Dictionary<string, string>();
var content = new FormUrlEncodedContent(values);
var response = await _httpClient.PostAsync("http://myservice.com/endpoint", content);
Is there any known issue or reason that the httpClient runs into a timeout on specific systems?
Important fact: Calling the same endpoint with the same data via Postman works on every collegues system.
EDIT:
The error message is:
"The request was canceled due to the configured HttpClient.Timeout of
100 seconds elapsing."
When navigating through the inner Exceptions the native error code is 995 and SocketErrorCode is OperationAborted
What I expect:
After max. 5 Secs there should a 200 Response with Json Data.

Ok the problem was that c# couldn't resolve the DNS name on particular systems and Postman could...

Asana API 403 Response in C#

I am trying to implement a Xamarin app that works with the Asana API.
I have successfully implemented the OAuth as documented in the Asana documentation here... at least I assume it is successful. I get an access token from the token endpoint in an HTTPResponse with HTTP Status "OK".
But then when I turn around and try to make an API call with that same access token, I get a 403 Forbidden error. I tried the same API call in my browser (after logging in to Asana), and it works fine, which leads me to believe that I do have access to the resource, I must have an issue with authorizing the request on my end.
The API call in question is (documented here): https://app.asana.com/api/1.0/workspaces.
My C# code is as follows (abbreviated to relevant parts, and assume that ACCESS_TOKEN contains the access token I got from the token exchange endpoint):
HttpClient client = new HttpClient();
client.BaseAddress = "https://app.asana.com/api/1.0";
client.DefaultRequestHeaders.Authorization = new System.Net.Http.Headers.AuthenticationHeaderValue("Bearer", ACCESS_TOKEN);
client.DefaultRequestHeaders.Add("Accept", "application/json");
And then I use this HttpClient (named client) in the following function:
// Returns a list of the Asana workspace names for the logged in user.
private async Task<List<string>> GetWorkspacesAsync()
{
List<string> namesList = new List<string>();
// Send the HTTP Request and get a response.
this.UpdateToken(); // Refreshes the token if needed using the refresh token.
using (HttpResponseMessage response = await client.GetAsync("/workspaces"))
{
// Handle a bad (not ok) response.
if (response.StatusCode != HttpStatusCode.OK)
{
// !!!THIS KEEPS TRIGGERING WITH response.StatusCode AS 403 Forbidden!!!
// Set up a stream reader to read the response.
// This is for TESTING ONLY
using (StreamReader reader = new StreamReader(await response.Content.ReadAsStreamAsync()))
{
// Extract the json object from the response.
string content = reader.ReadToEnd();
Debug.WriteLine(content);
}
throw new HttpRequestException("Bad HTTP Response was returned.");
}
// If execution reaches this point, the Http Response returned with code OK.
// Set up a stream reader to read the response.
using (StreamReader reader = new StreamReader(await response.Content.ReadAsStreamAsync()))
{
// Extract the json object from the response.
string content = reader.ReadToEnd();
JsonValue responseJson = JsonValue.Parse(content);
foreach (JsonValue workspaceJson in responseJson["data"])
{
string workspaceName = workspaceJson["name"];
Debug.WriteLine("Workspace Name: " + workspaceName);
namesList.Add(workspaceName);
}
}
}
// I have other awaited interactions with app storage in here, hence the need for the function to be async.
return namesList;
}

Finally found the answer. It looks like I was using HttpClient incorrectly; a subtle thing that should be equivalent, but is not due to the way it is implemented.
The answer
I needed to place the final slash at the end of the BaseAddress property of HttpClient, and NOT at the start of the relative address for the specific request. This answered question explains this.
To fix my code
I needed to change the setting up of the BaseAddress:
HttpClient client = new HttpClient();
client.BaseAddress = "https://app.asana.com/api/1.0/"; // FINAL SLASH NEEDED HERE
And remove the slash from the request's relative address:
// DO NOT put slash before relative address "workspaces" here
using (HttpResponseMessage response = await client.GetAsync("workspaces"))
Why I got the original error
When HttpClient combined the BaseAddress with the relative URI I specified in GetAsync(), it dropped off some of the base address, since the final slash was not included. The resulting address from combining the BaseAddress with the relative URI was a valid URL, but not a valid page/API call in Asana. Asana thus did an automatic redirect to a login page, which, of course, the rest of the API call would be forbidden from there.
How I discovered this
In debugging, I grabbed the access token returned during my app's authorization with Asana. I then recreated the request to the "/workspaces" API myself in Postman, and the request worked as expected. This confirmed that my authorization worked fine, and the issue must be with the specific request rather than the authorization. In debugging I then looked into the HttpResponseMessage, which has a property called RequestMessage, that includes the actual URL the GetAsync() made the request against. I observed the Login URL from Asana, rather than the BaseAddress I specified... which led me to the question/
answer linked above.
Hope this explanation helps anyone who comes across a similar error!

"The underlying connection was closed: An unexpected error occurred on a send.". Postman goes OK with same headers

Scenario
Win10 x64
VS2013
I'm trying to make a WebRequest, but I'm getting the following error:
The underlying connection was closed: An unexpected error occurred on a send.
Digging into the inner exception, I got:
"Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host."
The code which does the request is the following:
private static Hashtable exec (String method, String uri, Object data, String contentType) {
Hashtable response;
HttpWebRequest request = (HttpWebRequest)WebRequest.Create (API_BASE_URL + uri);
request.UserAgent = "MercadoPago .NET SDK v"+MP.version; //version resolves to 0.3.4
request.Accept = MIME_JSON; // application/json
request.Method = method; //GET
request.ContentType = contentType; //application/json
setData (request, data, contentType); //setData in this case does nothing.
String responseBody = null;
try {
HttpWebResponse apiResult = (HttpWebResponse)request.GetResponse (); //Error throws here
responseBody = new StreamReader (apiResult.GetResponseStream ()).ReadToEnd ();
response = new Hashtable();
response["status"] = (int) apiResult.StatusCode;
response["response"] = JSON.JsonDecode(responseBody);
} catch (WebException e) {
Console.WriteLine (e.Message);
}
}
What i've already done:
Made the request via Console Application and MVC Application controller. Both throws the same exception
Called the API via Postman with the exact same headers, which brings me the content correctly.
Those requests were working okay via c# about 4 days ago and I suddenly started having issues, but considering the fact that it responds okay for Postman, I can't figure out where's the problem.
Here's Postman's response
EDIT: Did both requests with Fiddler listening. The result for Postman shows a direct request to the API with HTTPS. When trying with my ConsoleApplication, it shows a HTTP request, which makes a tunnel to the API endpoint, port 443.
The TextView from Fiddler for the tunnel request says the following:
I noticed the "time" field which refers to a very old date, but i don't know what does it mean.

It is kind of bad practice to enable Tls12 like this-
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
In future, if you'd need to use higher version of TLS, you'll have to update your code.
If you are using an older version of .NET, you can simply switch it higher version in which Tls12 is enabled by default.
For example, this simple change in your web.config will enable Tls12 automatically-
<httpRuntime targetFramework="4.6.1"/>

You can try the code below:
string url = ""; // url of the endpoint
WebClient client = new WebClient();
ServicePointManager.Expect100Continue = true;
ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3;
client.Encoding = Encoding.UTF8;
client.Headers.Add("content-type", "application/json"); // same as other parameters in the header
var data = client.DownloadString(url);

Figured it out. I needed to include the use of TLS1.2.
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;

(As a reference for others who have the same issue)this also might be the result of a Double Hop issue , where you should pass the credited user along(in the pool) to the passing server or from one Environment to the other , otherwise the user is set to "ANONYMOUS/USER" and you will get a "An existing connection was forcibly closed by the remote host." Error

i found same error, just mention
request.UserAgent = "anything u want";

How to get the HTTP response when the request stream was closed during transfer

TL;DR version
When a transfer error occurs while writing to the request stream, I can't access the response, even though the server sends it.
Full version
I have a .NET application that uploads files to a Tomcat server, using HttpWebRequest. In some cases, the server closes the request stream prematurely (because it refuses the file for one reason or another, e.g. an invalid filename), and sends a 400 response with a custom header to indicate the cause of the error.
The problem is that if the uploaded file is large, the request stream is closed before I finish writing the request body, and I get an IOException:
Message: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host.
InnerException: SocketException: An existing connection was forcibly closed by the remote host
I can catch this exception, but then, when I call GetResponse, I get a WebException with the previous IOException as its inner exception, and a null Response property. So I can never get the response, even though the server sends it (checked with WireShark).
Since I can't get the response, I don't know what the actual problem is. From my application point of view, it looks like the connection was interrupted, so I treat it as a network-related error and retry the upload... which, of course, fails again.
How can I work around this issue and retrieve the actual response from the server? Is it even possible? To me, the current behavior looks like a bug in HttpWebRequest, or at least a severe design issue...
Here's the code I used to reproduce the problem:
var request = HttpWebRequest.CreateHttp(uri);
request.Method = "POST";
string filename = "foo\u00A0bar.dat"; // Invalid characters in filename, the server will refuse it
request.Headers["Content-Disposition"] = string.Format("attachment; filename*=utf-8''{0}", Uri.EscapeDataString(filename));
request.AllowWriteStreamBuffering = false;
request.ContentType = "application/octet-stream";
request.ContentLength = 100 * 1024 * 1024;
// Upload the "file" (just random data in this case)
try
{
using (var stream = request.GetRequestStream())
{
byte[] buffer = new byte[1024 * 1024];
new Random().NextBytes(buffer);
for (int i = 0; i < 100; i++)
{
stream.Write(buffer, 0, buffer.Length);
}
}
}
catch(Exception ex)
{
// here I get an IOException; InnerException is a SocketException
Console.WriteLine("Error writing to stream: {0}", ex);
}
// Now try to read the response
try
{
using (var response = (HttpWebResponse)request.GetResponse())
{
Console.WriteLine("{0} - {1}", (int)response.StatusCode, response.StatusDescription);
}
}
catch(Exception ex)
{
// here I get a WebException; InnerException is the IOException from the previous catch
Console.WriteLine("Error getting the response: {0}", ex);
var webEx = ex as WebException;
if (webEx != null)
{
Console.WriteLine(webEx.Status); // SendFailure
var response = (HttpWebResponse)webEx.Response;
if (response != null)
{
Console.WriteLine("{0} - {1}", (int)response.StatusCode, response.StatusDescription);
}
else
{
Console.WriteLine("No response");
}
}
}
Additional notes:
If I correctly understand the role of the 100 Continue status, the server shouldn't send it to me if it's going to refuse the file. However, it seems that this status is controlled directly by Tomcat, and can't be controlled by the application. Ideally, I'd like the server not to send me 100 Continue in this case, but according to my colleagues in charge of the back-end, there is no easy way to do it. So I'm looking for a client-side solution for now; but if you happen to know how to solve the problem on the server side, it would also be appreciated.
The app in which I encounter the issue targets .NET 4.0, but I also reproduced it with 4.5.
I'm not timing out. The exception is thrown long before the timeout.
I tried an async request. It doesn't change anything.
I tried setting the request protocol version to HTTP 1.0, with the same result.
Someone else has already filed a bug on Connect for this issue: https://connect.microsoft.com/VisualStudio/feedback/details/779622/unable-to-get-servers-error-response-when-uploading-file-with-httpwebrequest

I am out of ideas as to what can be a client side solution to your problem. But I still think the server side solution of using a custom tomcat valve can help here. I currently doesn`t have a tomcat setup where I can test this but I think a server side solution here would be along the following lines :
RFC section 8.2.3 clearly states :
Requirements for HTTP/1.1 origin servers:
- Upon receiving a request which includes an Expect request-header
field with the "100-continue" expectation, an origin server MUST
either respond with 100 (Continue) status and continue to read
from the input stream, or respond with a final status code. The
origin server MUST NOT wait for the request body before sending
the 100 (Continue) response. If it responds with a final status
code, it MAY close the transport connection or it MAY continue
to read and discard the rest of the request. It MUST NOT
perform the requested method if it returns a final status code.
So assuming tomcat confirms to the RFC, while in the custom valve you would have recieved the HTTP request header, but the request body would not be sent since the control is not yet in the servlet that reads the body.
So you can probably implement a custom valve, something similar to :
import org.apache.catalina.connector.Request;
import org.apache.catalina.connector.Response;
import org.apache.catalina.valves.ErrorReportValve;
public class CustomUploadHandlerValve extends ValveBase {
#Override
public void invoke(Request request, Response response) throws IOException, ServletException {
HttpServletRequest httpRequest = (HttpServletRequest) request;
String fileName = httpRequest.getHeader("Filename"); // get the filename or whatever other parameters required as per your code
bool validationSuccess = Validate(); // perform filename check or anyother validation here
if(!validationSuccess)
{
response = CreateResponse(); //create your custom 400 response here
request.SetResponse(response);
// return the response here
}
else
{
getNext().invoke(request, response); // to pass to the next valve/ servlet in the chain
}
}
...
}
DISCLAIMER : Again I haven`t tried this to success, need sometime and a tomcat setup to try it out ;).
Thought it might be a starting point for you.

I had the same problem. The server sends a response before the client end of the transmission of the request body, when I try to do async request. After a series of experiments, I found a workaround.
After the request stream has been received, I use reflection to check the private field _CoreResponse of the HttpWebRequest. If it is an object of class CoreResponseData, I take his private fields (using reflection): m_StatusCode, m_StatusDescription, m_ResponseHeaders, m_ContentLength. They contain information about the server's response!
In most cases, this hack works!

What are you getting in the status code and response of the second exception not the internal exception?
If a WebException is thrown, use the Response and Status properties of the exception to determine the response from the server.
http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.getresponse(v=vs.110).aspx

You are not saying what exactly version of Tomcat 7 you are using...
checked with WireShark
What do you actually see with WireShark?
Do you see the status line of response?
Do you see the complete status line, up to CR-LF characters at its end?
Is Tomcat asking for authentication credentials (401), or it is refusing file upload for some other reason (first acknowledging it with 100 but then aborting it mid-flight)?
The problem is that if the uploaded file is large, the request stream
is closed before I finish writing the request body, and I get an IOException:
If you do not want the connection to be closed but all the data transferred over the wire and swallowed at the server side, on Tomcat 7.0.55 and later it is possible to configure maxSwallowSize attribute on HTTP connector, e.g. maxSwallowSize="-1".
http://tomcat.apache.org/tomcat-7.0-doc/config/http.html
If you want to discuss Tomcat side of connection handling, you would better ask on the Tomcat users' mailing list,
http://tomcat.apache.org/lists.html#tomcat-users
At .Net side:
Is it possible to perform stream.Write() and request.GetResponse() simultaneously, from different threads?
Is it possible to performs some checks at the client side before actually uploading the file?

hmmm... i don't get it - that is EXACTLY why in many real-life scenarios large files are uploaded in chunks (and not as a single large file)
by the way: many internet servers have size limitations. for instance in tomcat that is representad by maxPostSize (as seen in this link: http://tomcat.apache.org/tomcat-5.5-doc/config/http.html)
so tweaking the server configurations seems like the easy way, but i do think that the right way is to split the file to several requests
EDIT: replace Uri.EscapeDataString with HttpServerUtility.UrlEncode
Uri.EscapeDataString(filename) // a problematic .net implementation
HttpServerUtility.UrlEncode(filename) // the proper way to do it

I am experience a pretty similar problem currently also with Tomcat and a Java client. The Tomcat REST service sends a HTTP returncode with response body before reading the whole request body. The client however fails with IOException. I inserted a HTTP Proxy on the client to sniff the protocol and actually the HTTP response is sent to the client eventually. Most likly the Tomcat closed the request input stream before sending the response.
One solution is to use a different HTTP server like Jetty which does not have this problem. The other solution is a add a Apache HTTP server with AJP in front of Tomcat. Apache HTTP server has a different handling of streams and with that the problem goes away.