In our Winforms client app that is disconnected from the business layer and communicates via MVC web API, we get a nasty error. Are there any experts out there who can please explain!
What does all this mean?
---> (Inner Exception #0) System.Net.Http.Httprequestexception: An Error Occurred While Sending The Request.
---> System.Net.Webexception: The Underlying Connection Was Closed: An Unexpected Error Occurred On A Receive.
---> System.Io.Ioexception: Unable To Read Data From The Transport Connection: An Existing Connection Was Forcibly Closed By The Remote Host.
---> System.Net.Sockets.Socketexception: An Existing Connection Was Forcibly Closed By The Remote Host
At System.Net.Sockets.Socket.Endreceive(Iasyncresult Asyncresult)
At System.Net.Sockets.Networkstream.Endread(Iasyncresult Asyncresult)
--- End Of Inner Exception Stack Trace ---
At System.Net.Sockets.Networkstream.Endread(Iasyncresult Asyncresult)
At System.Net.Pooledstream.Endread(Iasyncresult Asyncresult)
At System.Net.Connection.Readcallback(Iasyncresult Asyncresult)
--- End Of Inner Exception Stack Trace ---
At System.Net.Httpwebrequest.Endgetresponse(Iasyncresult Asyncresult)
At System.Net.Http.Httpclienthandler.Getresponsecallback(Iasyncresult Ar)
--- End Of Inner Exception Stack Trace ---<---
Well, in general, this exception is very vexing, indeed. But what you have expected? You are working with the network. It lies down and gets up frequently. There are two main reasons of such a behaviour:
The quality of the network in which the code is executing.
Errors made by programmers if they implemented their own protocol of communication.
If these two reasons are not in place, then you should implement something like auto connection using timeouts. Also, the TCP-protocol implemetation internals will be very helpful!
Related
We have developed WCF service and hosted as windows service. We have written .net core client application.
We have observed that the service is not responding after a few calls. It gets stopped somehow.
I tried to create logging and trace in WCF service but it is not creating any .svclog file.
In exception handling, it gives the following message
Exception: One or more errors occurred. (An error occurred while
sending the request.) System.ServiceModel.CommunicationException: An
error occurred while sending the request. --->
System.Net.Http.HttpRequestException: An error occurred while sending
the request. ---> System.IO.IOException: Unable to read data from the
transport connection: An existing connection was forcibly closed by
the remote host.. ---> System.Net.Sockets.SocketException (10054): An
existing connection was forcibly closed by the remote host. --- End
of inner exception stack trace --- at
System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError
error, CancellationToken cancellationToken) at
System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16
token) at
System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage
request, CancellationToken cancellationToken)
I want to know what forced the service to be stopped? How can I find this?
In my application, there is a functionality to extract reports based on a date range. In the background, it is calling my Web API with authentication(A), which consists of api caller functionality using httpclient, which is calling another non-secured DMZ server API (B) and a DMZ API-caliing WCF service (C) to collect data from the database.
My problem is that whenever I select a large date range, I get the following error in my WebAPI (A).
System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: The underlying connection was closed: The connection was closed unexpectedly.
at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
--- End of inner exception stack trace ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at LRAS.OrderServices.Core.Helpers.ApiCaller.d__4.MoveNext()
What I have tried: I have increased the httpclient timeout from 300 to 1000 seconds, but it is breaking nearly every minute.
You can try several methods:
1.Configure the service point security protocol and select the SecurityProtocolType that suits you (Tls12, Ssl3, Tls, Tls11).
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
C# System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send
2.Set the ConnectionClose property of HttpClient.
_client.DefaultRequestHeaders.ConnectionClose = true;
HttpClient throwing "An error occurred while sending the request."
3.Try to hold the request long enough for it to complete normally and receive a valid response.
_client.DefaultRequestHeaders.Add("Connection", "Keep-Alive");
_client.DefaultRequestHeaders.Add("Keep-Alive", "3600");
.NET HttpClient - An existing connection was forcibly closed by the remote host
I'm encountering a situation on production that is difficult to debug.
Sometimes the connection to an external service can't be established with the following error:
System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 93.39.196.220:443
at System.Net.Sockets.Socket.InternalEndConnect(IAsyncResult asyncResult)
at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.EndGetRequestStream(IAsyncResult asyncResult, TransportContext& context)
at System.Net.Http.HttpClientHandler.GetRequestStreamCallback(IAsyncResult ar)
--- End of inner exception stack trace ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
Sometimes it works without any problem. This is what I've tried:
Replacing the way I perform the connection (from Restsharp to HttpClient)
HttpClient is long lived as suggested by this post
Adjusting the timeout
Using async and sync code
Analyzing SNAT Port exhaustion
More info:
The IPs of my server are authorized by the external service (and if they weren't I would be receiving a different error)
While on production I'm having the problem I'm able to connect to the service from development or staging (so it shouldn't be a problem with the external service)
While I'm having trouble contacting this service I don't have trouble contacting a different (but similar) service
I'm on a Web App Service on Azure. I'm using the .NET framework 4.7.2
The same code sometimes works and sometimes it doesn't
This is (more or less) the current iteration
private async Task<ResultSet> SendRequestAsync(HttpClient client, RateRequestBody document){
ServicePointManager.SecurityProtocol |= SecurityProtocolType.Tls12;
var response = await client.PostAsXmlAsync(baseUrl + "/XMLServices", document);
if (response.IsSuccessStatusCode){
...
}else{
...
}
}
Any ideas on what I might have missed?
Thanks
The answer may lay here: Default SecurityProtocol in .NET 4.5
Assuming both servers are operating correctly, one works fine with your client code and the other gives you problems based on your 'More Info' section: "While I'm having trouble contacting this service I don't have trouble contacting a different (but similar) service"
I recommend removing or commenting out the following line and giving it a try:
//ServicePointManager.SecurityProtocol |= SecurityProtocolType.Tls12;
...or replace it with:
ServicePointManager.SecurityProtocol |= SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12;
We already had a server with signalr hub where clients connect using signalr. We recently moved it to two nodes with a load balancer for scalability. But after moving it to load balancer now clients are unable to connect to the server using signalr. Clients first try using web sockets and it gives the following error on signalr trace.
fae26beb-a806-4957-b52a-39b80856e492 - Auto: Failed to connect to using transport webSockets. System.Net.WebSockets.WebSocketException (0x80004005): Unable to connect to the remote server ---> System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a receive. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)
at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
--- End of inner exception stack trace ---
at System.Net.Security._SslStream.EndRead(IAsyncResult asyncResult)
at System.Net.TlsStream.EndRead(IAsyncResult asyncResult)
at System.Net.PooledStream.EndRead(IAsyncResult asyncResult)
at System.Net.Connection.ReadCallback(IAsyncResult asyncResult)
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Net.WebSockets.ClientWebSocket.<ConnectAsyncCore>d__21.MoveNext()
at System.Net.WebSockets.ClientWebSocket.<ConnectAsyncCore>d__21.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.AspNet.SignalR.Client.Transports.WebSocketTransport.<PerformConnect>d__1.MoveNext()
But after couple of times trying to connect it manages to connect using SSE (Server Sent Events).
fae26beb-a806-4957-b52a-39b80856e492 - SSE: OnMessage(Data: {"C":"s-0,BA9F","S":1,"M":[]})
fae26beb-a806-4957-b52a-39b80856e492 - ChangeState(Connecting, Connected)
fae26beb-a806-4957-b52a-39b80856e492 - SSE: OnMessage(Data: {"C":"s-0,BAA0","M":[{"H":"PrintConnectorHub","M":"SignalRConnectedSuccessfully","A":[true]}]})
fae26beb-a806-4957-b52a-39b80856e492 - OnMessage({"R":false,"I":"3"})
But when the server tries to send a message back to the client, client does not receive the message. We have implemented SignalR scale out with SQL Server and following are the log traces in the server.
TRACE SignalR.SqlMessageBus Stream 0 : SqlReceiver last payload ID=47781, new payload ID=47782 (Microsoft.AspNet.SignalR.SqlServer.SqlReceiver.ProcessRecord)
TRACE SignalR.SqlMessageBus Stream 0 : Updated receive reader initial payload ID parameter=47782 (Microsoft.AspNet.SignalR.SqlServer.SqlReceiver.ProcessRecord)
TRACE SignalR.SqlMessageBus Stream 0 : Payload 47782 containing 1 message(s) received (Microsoft.AspNet.SignalR.SqlServer.SqlReceiver.ProcessRecord)
TRACE SignalR.SqlMessageBus Stream 0 : 1 records received (Microsoft.AspNet.SignalR.SqlServer.ObservableDbOperation.ExecuteReaderWithUpdates)
TRACE SignalR.SqlMessageBus Created DbCommand: CommandType=Text, CommandText=SELECT [PayloadId], [Payload], [InsertedOn] FROM [SignalR].[Messages_0] WHERE [PayloadId] > #PayloadId, Parameters= [Name=PayloadId, Value=47782] (Microsoft.AspNet.SignalR.SqlServer.DbOperation.TraceCommand)
Signalr initialization in clients
signalrHostUrl = "SignalR_Host";
try
{
hubConnection = new HubConnection(signalrHostUrl);
hubConnection.Error += ex =>
{
signalRlogger.Error(ex, "Exception from hubConnection");
};
hubConnection.ConnectionSlow += () => ConnectionSlow();
hubConnection.Closed += () => SignalrConnectionConnectionClosed();
hubConnection.StateChanged += signalRConnectionStateController.StateChanged;
hubConnection.Start().Wait();
}
catch (Exception ex)
{
signalRlogger.Error(ex, "initialzing hub connection");
}
Before moving to the load balancer the clients were perfectly able to connect via web sockets and communicate. But after moving to load balancer it completely fails to connect via web sockets. We even tried to open a web socket connection by this tool (https://chrome.google.com/webstore/detail/simple-websocket-client/pfdhoblngboilpfeibdedpjgfnlcodoo?hl=en) which also fails to open a connection. Although it manages to connect via SSE it still unable to communicate to clients. What are the complications when using web sockets in load balancing and in this situation why signalr fails to make a connection to the clients?
When the other machine tries to decrypt the connection token you got when negotiating the connection it can't and the request fails. You need to ensure that all machines you are using have the same machine key. This way all the machines will be able to decrypt connectionTokens created by other machines.
Are you able to enable stickyness on the load balancers? For example session based stickyness. This will make sure the clients connect to the same machine. The downside is that if a machine fails the client won't be able to reconnect.
I am getting the following error in WCF+Silverlight+RIA Service application. It's happening only on production DB.
I copied the production DB to my development machine, I am getting that error when I point the web.config to my production DB from development environment. If I point to test DB then everything works fine.
So I thought that some mismatch problem in DB so I tried to compare test and production DB and all the DB structures are same.
This error occurs only on one function/query 'GetEventViewAll', all other operations are working on that DB. So I thought that the function 'GetEventViewAll' returns more rows, so I deleted all rows on that table, but still getting that error.
This error happens only on production DB, not production environment. I am using same machine, same code, same VS2010. I am just making changes only on web.config to point other DB(production) then getting error.
Could anyone please tell me what I missed or what needs to be done?
the error from output window of VS2010 is:
{System.ServiceModel.DomainServices.Client.DomainOperationException: Load operation failed for query 'GetEventViewAll'. The remote server returned an error: NotFound. ---> System.ServiceModel.CommunicationException: The remote server returned an error: NotFound. ---> System.Net.WebException: The remote server returned an error: NotFound. ---> System.Net.WebException: The remote server returned an error: NotFound.
at System.Net.Browser.BrowserHttpWebRequest.InternalEndGetResponse(IAsyncResult asyncResult)
at System.Net.Browser.BrowserHttpWebRequest.<>c__DisplayClassa.b__9(Object sendState)
at System.Net.Browser.AsyncHelper.<>c__DisplayClass4.b__1(Object sendState)
--- End of inner exception stack trace ---
at System.Net.Browser.AsyncHelper.BeginOnUI(SendOrPostCallback beginMethod, Object state)
at System.Net.Browser.BrowserHttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelAsyncRequest.CompleteGetResponse(IAsyncResult result)
--- End of inner exception stack trace ---
at System.ServiceModel.DomainServices.Client.WebDomainClient1.EndQueryCore(IAsyncResult asyncResult)
at System.ServiceModel.DomainServices.Client.DomainClient.EndQuery(IAsyncResult asyncResult)
at System.ServiceModel.DomainServices.Client.DomainContext.CompleteLoad(IAsyncResult asyncResult)
--- End of inner exception stack trace ---}
[System.ServiceModel.DomainServices.Client.DomainOperationException]: {System.ServiceModel.DomainServices.Client.DomainOperationException: Load operation failed for query 'GetEventViewAll'. The remote server returned an error: NotFound. ---> System.ServiceModel.CommunicationException: The remote server returned an error: NotFound. ---> System.Net.WebException: The remote server returned an error: NotFound. ---> System.Net.WebException: The remote server returned an error: NotFound.
at System.Net.Browser.BrowserHttpWebRequest.InternalEndGetResponse(IAsyncResult asyncResult)
at System.Net.Browser.BrowserHttpWebRequest.<>c__DisplayClassa.<EndGetResponse>b__9(Object sendState)
at System.Net.Browser.AsyncHelper.<>c__DisplayClass4.<BeginOnUI>b__1(Object sendState)
--- End of inner exception stack trace ---
at System.Net.Browser.AsyncHelper.BeginOnUI(SendOrPostCallback beginMethod, Object state)
at System.Net.Browser.BrowserHttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelAsyncRequest.CompleteGetResponse(IAsyncResult result)
--- End of inner exception stack trace ---
at System.ServiceModel.DomainServices.Client.WebDomainClient1.EndQueryCore(IAsyncResult asyncResult)
at System.ServiceModel.DomainServices.Client.DomainClient.EndQuery(IAsyncResult asyncResult)
at System.ServiceModel.DomainServices.Client.DomainContext.CompleteLoad(IAsyncResult asyncResult)
--- End of inner exception stack trace ---}
Data: {System.Collections.ListDictionaryInternal}
InnerException: {System.ServiceModel.CommunicationException: The remote server returned an error: NotFound. ---> System.Net.WebException: The remote server returned an error: NotFound. ---> System.Net.WebException: The remote server returned an error: NotFound.
at System.Net.Browser.BrowserHttpWebRequest.InternalEndGetResponse(IAsyncResult asyncResult)
at System.Net.Browser.BrowserHttpWebRequest.<>c__DisplayClassa.b__9(Object sendState)
at System.Net.Browser.AsyncHelper.<>c__DisplayClass4.b__1(Object sendState)
--- End of inner exception stack trace ---
at System.Net.Browser.AsyncHelper.BeginOnUI(SendOrPostCallback beginMethod, Object state)
at System.Net.Browser.BrowserHttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelAsyncRequest.CompleteGetResponse(IAsyncResult result)
--- End of inner exception stack trace ---
at System.ServiceModel.DomainServices.Client.WebDomainClient`1.EndQueryCore(IAsyncResult asyncResult)
at System.ServiceModel.DomainServices.Client.DomainClient.EndQuery(IAsyncResult asyncResult)
at System.ServiceModel.DomainServices.Client.DomainContext.CompleteLoad(IAsyncResult asyncResult)}
Message: "Load operation failed for query 'GetEventViewAll'. The remote server returned an error: NotFound."
StackTrace: null
It's resolved by adding Query-HasSideEffects attribute to the function.
[Query(HasSideEffects = true)] //<--- this line solved the problem
public IQueryable<EventView> GetEventView(List<string> siteName)
{
.....
}
I passed a list to that function. That list is filled from a table. Test DB has less values, production DB has more values, the list count was more in production db. So I got the following error
The Maximum URI length of 2083 was exceeded
so I put Query attribute to that function to resolve!