.NET WebRequest to CloudFlare causes SSL exception after 1 hour idle period - c#

I've been having issues with a tool I use to upload images to a website behind CloudFlare. This tool works fine at first, and continues functioning unless there is a > 1 hour pause between requests. After this pause, an exception occurs on the next connection attempt.
A first chance exception of type 'System.Net.WebException' occurred in System.dll
System.Net.WebException: The request was aborted: Could not create SSL/TLS secure channel.
at System.Net.HttpWebRequest.GetResponse()
I attempted to use a debugger to poke into this deeper, but there was no InnerException and it seems the actual issue originated from SChannel before any connection was established. This is readily replicable using the following small program:
class Program
{
static void Main(string[] args)
{
Console.WriteLine("Initial connection attempt, this should succeed:");
RunCFRequest();
Console.WriteLine("Wait 70 minutes for next connection attempt...");
Thread.Sleep(70*60*1000);
Console.WriteLine("Second connection attempt, this one should reproduce the failure:");
try
{
RunCFRequest();
}
catch (Exception exc)
{
Console.WriteLine(exc.ToString());
}
Console.WriteLine("Performing another connection attempt after failure to verify we continue working:");
RunCFRequest();
Console.WriteLine("Demo complete. Press any key to exit.");
Console.ReadKey();
}
private static void RunCFRequest()
{
Console.WriteLine("Attempting connection at " + DateTime.Now);
var request = (HttpWebRequest) WebRequest.Create("https://up1.ca");
using (var response = request.GetResponse())
{
using (var responseStream = response.GetResponseStream())
{
using (var streamReader = new StreamReader(responseStream))
{
string recvd = streamReader.ReadToEnd();
Console.WriteLine("Successfully read stream, got " + recvd.Length + " bytes of data");
}
}
}
}
}
Is there something wrong with this simple code? I attempted to do packet capture in order to resolve why this is occurring.
Capture is available at https://up1.ca/#4MMkdD_u8v5pLAsSvrCtHw in pcapng format.
The capture contains 3 TCP streams, they can be accessed using the following wireshark filters:
tcp.stream eq 0 = inital connection, succeeds
tcp.stream eq 1 = second connection after 70 minutes, this fails with the above exception
tcp.stream eq 2 = another attempt after handling and ignoring that exception, this succeeds
Based on the capture my best guess is that it has something to do with how CloudFlare does SSL session resumption. Are there any known problems with HttpWebRequest or Microsoft SChannel itself and SSL resumption or is this issue specific to CloudFlare? I've successfully replicated this problem on multiple sites behind CloudFlare, but I have not experienced it when using my own server directly. I don't have SSL resumption though.
Any help or even wild theories are appreciated. I'm not sure where to go from here, I'd appreciate it if someone could have a look at the capture, I'll report this to CF if needed.

Related

Why HttpClient throw an exception after some minutes?

i made a program, to retrieve some price from some website, it works pretty good, but if i let my application rest or idle for 1 or 2 minutes then when i want to run a price check againm it would throw an exception "an error occurred while sending the request" , i have tried everything i found in google, like adding SecurityProtocol, running it on a Task or using WebClient, but nothing works.
here is the function that retrieve my http code, i then extract the price from it.
HttpClient client = new HttpClient();
public async Task ListViewScanner(string URL, int SelectedItem)
{
string webData;
try
{
System.Net.ServicePointManager.SecurityProtocol = System.Net.SecurityProtocolType.Tls12 |
System.Net.SecurityProtocolType.Tls11 |
System.Net.SecurityProtocolType.Tls;
using (HttpResponseMessage response = await client.GetAsync(URL))
{
response.EnsureSuccessStatusCode();
webData = await response.Content.ReadAsStringAsync(); //HERE MY CATCH GETS TRIGGERED
}
}
catch (HttpRequestException e)
{
MessageBox.Show($"Message :{e.Message} ");
webData = "ERROR";
}
}
I commented where i get the exception.
exceptions
SocketException: An existing connection was forcibly closed by the remote host
IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.
WebException: The underlying connection was closed: An unexpected error occurred on a send.
HttpRequestException: An error occurred while sending the request.
Use a new instance of HttpClient every time, it's maybe related to the old instance that close the connection.
don't forge to use with a using statement to make sure that is destroyed when the call end.

Azure Event Hub timeouts

I occasionally see timeout errors in event hub client when trying to send messages to Azure event hub. It looks like resource limit is reached for client but I'm not sure...
Here the code:
MessagingFactory messagingFactory = null;
EventHubClient hubClient = null;
try
{
messagingFactory = MessagingFactory.CreateFromConnectionString(this.connectionString);
hubClient = messagingFactory.CreateEventHubClient(this.configurationManager.EventHubName);
var batchIterator = new EventBatchIterator<T>(events);
foreach (var batch in batchIterator)
{
await hubClient.SendBatchAsync(batch);
}
}
catch (Exception e)
{
this.logger.Error("An error occurred during sent messages to event hub", e);
}
finally
{
if (hubClient != null)
{
await hubClient.CloseAsync();
}
if (messagingFactory != null)
{
await messagingFactory.CloseAsync();
}
}
Exception is:
An error occurred during communication with 'N/A'. Check the
connection information, then retry.
Microsoft.ServiceBus.Messaging.MessagingCommunicationException
A connection attempt failed because the connected party did not
properly respond after a period of time, or established connection
failed because connected host has failed to respond
System.Net.Sockets.SocketException
According to you mentioned execption indicates that a user-initiated operation is taking longer than the operation timeout. My workaround is that you could increase OperationTimeout or retry count.
Demo code for increasing timeout
var builder = new ServiceBusConnectionStringBuilder("connection string")
{
TransportType = TransportType.Amqp,
OperationTimeout = TimeSpan.FromSeconds(90)
};
messagingFactory = MessagingFactory.CreateFromConnectionString(builder.ToString());
More info about Timeout exception you refer to this document.
Common causes
There are two common causes for this error: incorrect configuration, or a transient service error.
Incorrect configuration The operation timeout might be too small for the operational condition. The default value for the operation timeout in the client SDK is 60 seconds. Check to see if your code has the value set to something too small. Note that the condition of the network and CPU usage can affect the time it takes for a particular operation to complete, so the operation timeout should not be set to a very small value.
Transient service error Sometimes the Event Hubs service can experience delays in processing requests; for example, during periods of high traffic. In such cases, you can retry your operation after a delay, until the operation is successful. If the same operation still fails after multiple attempts, please visit the Azure service status site to see if there are any known service outages.

prevent application crashes when sending data over a closed websocket connection

The ASP.NET Core application uses websocket connection on the client side and Microsoft.AspNetCore.WebSockets.Server 0.1.0 (latest stable version on nuget as I know) on the server side. The simple sending code is
await _ws.SendAsync(new ArraySegment<byte>(arrbr), WebSocketMessageType.Text, true, ctk);
the problem is this line throws error when it is a closed connection. I would like that method to return a Boolean if process was successful. I already check if the connection is open like this:
_ws.State == WebSocketState.Open
But this does not work if user has
unplugged the network cable or disconnected his device(almost all situations except closing the browsers).
As an extra, I do not know how to simulate network connection loss for one of two clients and I suppose WebSocketState is readonly, please warn me if I am wrong and I do not think shorter pings will solve this problem.
I have two ideas:
I may use the sender code in a try catch block. (I am not comfortable with using try catch in production code)
I may set interval on the client side, and ask the server like "what is new for me". I feel bad with this because it is away from being a websocket(closer to http).
Is there any way to fix this without using try catch? I know that this is not a qualified question but a qualified problem for me. I can share full code if needed.
Update 1
after using server-side logging:
While messaging is working well in production environment, I disconnect the client by unplugging the network cable and send data to the client. I use try catch and no catch. then i get this error.
This means I cant detect lost connection by using try catch. and i think i can solve this by handling this throw.
How can I handle this error?
update2
I have noticed that "Exceptions from an Async Void Method Can’t Be Caught with Catch" and "it's possible to use await in catch" since c# 6 link however I can not catch the throw. I may try running synchronously await _ws.SendAsync(new ArraySegment<byte>(arrbr), WebSocketMessageType.Text, true, ctk).runsynchronously(); if I cant fix in this way
update3
running synchronously does not help. using try catch does not differ. as a result question, asp.net core websocket how to ping
update4
my log file when using Microsoft.aspnetcore.websockets.server 0.1.0
fail: Microsoft.AspNetCore.Server.Kestrel[13]
Connection id "0HL1940BSEV8O": An unhandled exception was thrown by the application.
System.IO.IOException: Unexpected end of stream
at Microsoft.AspNetCore.WebSockets.Protocol.CommonWebSocket.<EnsureDataAvailableOrReadAsync>d__38.MoveNext()
my log file when using Microsoft.aspnetcore.websockets 1.0.0
fail: Microsoft.AspNetCore.Server.Kestrel[13]
Connection id "0HL19H5R4CS21": An unhandled exception was thrown by the application.
System.Net.WebSockets.WebSocketException (0x80004005): The remote party closed the WebSocket connection without completing the close handshake. ---> System.IO.IOException: Error -4077 ECONNRESET connection reset by peer ---> Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.UvException: Error -4077 ECONNRESET connection reset by peer
I might be missing something, but why can't wrap the sending operation in a method that returns bool in the following manner:
private async Task<bool> DoSend()
{
bool success = true;
try
{
await _ws.SendAsync(new ArraySegment<byte>(arrbr), WebSocketMessageType.Text, true, ctk);
}
catch (Exception ex)
{
// Do some logging with ex
success = false;
}
return success;
}
I also suggest reading about Async All The Way, it should clear some of the confusion with async void, async Task and async Task<T>
Until C# 6.0 to capture an exceptions from async methods you should use the ExceptionDispatchInfo type. Then the code will look like this:
private async Task<bool> DoSend()
{
bool success = true;
ExceptionDispatchInfo capturedException = null;
try
{
await _ws.SendAsync(new ArraySegment<byte>(arrbr), WebSocketMessageType.Text, true, ctk);
}
catch (Exception ex)
{
capturedException = ExceptionDispatchInfo.Capture(ex);
}
if (capturedException != null)
{
await ExceptionHandler();
if (needsThrow)
{
capturedException.Throw();
}
}
success = capturedException == null;
return success;
}

HttpClient File Download Error - Unable to read data from the transport connection

I've written an application that in part can download files from a specific web service. The code utilizes an HttpClient to make the calls. The problem is that occasionally I will get a failed request with the following exception message:
Unable to read data from the transport connection: The connection was closed.
I did run across these blog posts, in which the author had to revert the protocol version to 1.0, disable keep alive, and limit the number of service point connections:
http://briancaos.wordpress.com/2012/07/06/unable-to-read-data-from-the-transport-connection-the-connection-was-closed/
http://briancaos.wordpress.com/2012/06/15/an-existing-connection-was-forcibly-closed-by-the-remote-host/
I followed those instructions, as best I knew how and still got the error. I also made sure to keep a single instance of the HttpClient around (following the Singleton principle).
What is interesting is that when running Fiddler I've yet to get the error, which makes me think that there is something that can be done on the client side since Fiddler appears to be doing something to keep the connection alive (though the issue is so sporadic this may be a red herring).
A couple more notes:
The error invariably occurs in the middle of a download (never when initiating the request).
The file continues to download up to the point of failure (there are no extended pauses or delays first).
--UPDATE--
The error occurs on the following line:
responseTask.Wait(cancellationTokenSource.Token);
The following is the full exception:
System.AggregateException occurred HResult=-2146233088 Message=One
or more errors occurred. Source=mscorlib StackTrace:
at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at Form1.StartDownload() in c:\Projects\Visual Studio 2012\Demo\Demo\Form1.cs:line 88 InnerException:
System.Net.Http.HttpRequestException
HResult=-2146233088
Message=Error while copying content to a stream.
InnerException: System.IO.IOException
HResult=-2146232800
Message=Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.
Source=System
StackTrace:
at System.Net.ConnectStream.EndRead(IAsyncResult asyncResult)
at System.Net.Http.HttpClientHandler.WebExceptionWrapperStream.EndRead(IAsyncResult
asyncResult)
at System.Net.Http.Handlers.ProgressStream.EndRead(IAsyncResult
asyncResult)
at System.Net.Http.StreamToStreamCopy.BufferReadCallback(IAsyncResult ar)
InnerException: System.Net.Sockets.SocketException
HResult=-2147467259
Message=An existing connection was forcibly closed by the remote host
Source=System
ErrorCode=10054
NativeErrorCode=10054
StackTrace:
at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
InnerException:
--UPDATE #2--
I thought I would try changing the completion option from 'content read' to 'headers read'. This also failed with the same exception, albeit in a different location (where the TODO comment is, reading the content stream).
--UPDATE #3--
I can confirm that the web service (which is hosted in IIS) is aborting the connections (the IIS logs show a win32 status code of 1236 - ERROR_CONNECTION_ABORTED). To try and narrow things down, the MinFileBytesPerSec metabase property was set to zero (on the off chance the client stopped pulling down data momentarily) and the connection is still being aborted. I've double checked all the timeouts and buffer sizes I can think of to no avail. Clawing at thin air at the moment. Any ideas would be appreciated.
Client Setup:
private void SetupClient()
{
// In case we're taxing the web server, limit the number
// connections we're allowed to make to one.
ServicePointManager.DefaultConnectionLimit = 1;
// Set up the progress handler so that we can keep track of the download progress.
_progressHandler = new ProgressMessageHandler();
_progressHandler.HttpReceiveProgress += ProgressHandler_HttpReceiveProgress;
// Create our HttpClient.
_client = HttpClientFactory.Create(_progressHandler);
_client.BaseAddress = new Uri("http://localhost");
_client.Timeout = TimeSpan.FromMinutes(30);
_client.DefaultRequestHeaders.TransferEncodingChunked = true;
}
Download Logic:
private void StartDownload()
{
// Create the request.
using (HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Post, "http://localhost/Download"))
{
// Revert the protocol version and turn off keep alive in accordance with:
// http://briancaos.wordpress.com/2012/07/06/unable-to-read-data-from-the-transport-connection-the-connection-was-closed/
// http://briancaos.wordpress.com/2012/06/15/an-existing-connection-was-forcibly-closed-by-the-remote-host/
request.Version = new Version("1.0");
request.Headers.Add("Keep-Alive", "false");
// Set the cancellation token's timeout to 30 minutes.
int timeoutInMilliseconds = 30 * 60 * 1000;
using (CancellationTokenSource cancellationTokenSource = new CancellationTokenSource(timeoutInMilliseconds))
{
// Making sure that the message isn't "complete" until everything is read in so we can cancel it at anytime.
Task<HttpResponseMessage> responseTask = _client.SendAsync(request, HttpCompletionOption.ResponseContentRead);
responseTask.Wait(cancellationTokenSource.Token);
using (HttpResponseMessage response = responseTask.Result)
{
if (!response.IsSuccessStatusCode)
{
throw new Exception("Request failed!");
}
Task<Stream> streamTask = response.Content.ReadAsStreamAsync();
using (Stream contentStream = streamTask.Result)
{
// TODO: Save to disk.
}
}
}
}
}

Getting a web response timesout

I've been asked to write a small program which checks that all the pages we have online are not erroring.
To do this, I use the following code (where pathsToCheck is List, each string is a URL like http://www.domain.com/webpage)
foreach (string path in pathsToCheck)
{
HttpWebResponse response = null;
try
{
HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(path);
webRequest.AllowAutoRedirect = true;
response = (HttpWebResponse)webRequest.GetResponse();
System.Diagnostics.Debug.Assert(response.StatusDescription == "OK", "Look into this, it doesn't like the response code");
System.Threading.Thread.Sleep(1000);
}
catch (Exception ex)
{
Console.WriteLine("Failed : " + path);
}
finally
{
Write(--totalPathsToCheck);
}
}
The problem I am having is it always fails (timesout) from the third item in the list (everything fails from the third). Naturally, I guessed there must be a fault with the third item, but there isn't.
Since the first item doesn't time out, I created a new list, with 5 items, all of the same URL (one I know doesn't time out). The same issue occurs, on the third iteration, it times out and continues to time out for the remainder of the list.
I then decided to test a different URL (on a different domain) and the same issue persists.
I added the sleep to the code, and increased it in-case there were too many requests within a given period but that made no differences.
What should I be doing?
You need to close your connections. Add
response.Close();
From http://msdn.microsoft.com/en-us/library/system.net.httpwebresponse.close.aspx:
The Close method closes the response stream and releases the connection to the resource for reuse by other requests.
You must call either the Stream.Close or the HttpWebResponse.Close method to close the stream and release the connection for reuse. It is not necessary to call both Stream.Close and HttpWebResponse.Close, but doing so does not cause an error. Failure to close the stream can cause your application to run out of connections.

Categories

Resources