I'm encountering an app pool timing out on one request and it always times out around 5 minutes. I've enabled tracing, but no Warning, Error, Critical events are being logged. There are also no events in the EventViewer. I enabled FailedReqLogging and set it to log any requests over 4 minutes 30 sec. I've found the following in the C:\inetpub\logs\FailedReqLogFiles\W3SVC2 directory:
What IIS setting do I change to accomidate this request? By the Failure Reason it's listed as TIME_TAKEN so I'm guessing the AppPool exceeded some time limit. What is that limit?
I also notice that the Final Status is listed at 200. That would mean I wouldn't see anything in the logs for Warning, Critical, or Fatal because it was logged as OK. Am I right on that assumption?
Below is a stack trace of the exception found in the client trace logs. Note there is no error shown in the servers trace log. What's odd is that there is multiple nested exceptions within, but none hold real value.
...
...
<Exception>
<ExceptionType>System.ServiceModel.CommunicationException, System.ServiceModel, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</ExceptionType>
<Message>The underlying connection was closed: A connection that was expected to be kept alive was closed by the server.</Message>
<StackTrace>
at System.ServiceModel.Channels.HttpChannelUtilities.ProcessGetResponseWebException(WebException webException, HttpWebRequest request, HttpAbortReason abortReason)
at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)
at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)
at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at OSI.Framework.ServiceContracts.IWCFObjectDirectorWebService.Fetch(Byte[] serializedObjectRequest)
at OSI.Framework.WCF.Director.WCFWebObjectDirectorProxy.Fetch(Byte[] serializedObjectRequest)
at OSI.Framework.WCF.Director.WCFWebObjectDirector.Fetch(Object criteria)
at ActvMonitorSystemBusiness.Views.ActvView.GetActvDetails(DateTime fromDate, DateTime thruDate, String userGrpCd, String userCatCd, String userMajorCd, String userStatusCd, Nullable`1 userMemberNbr, Nullable`1 subjNbr, String searchBy, String dispActivity, Boolean showUncategorizedActv, Object[] fetchOptions)
at ActvMonitorSystemScreen.frmActvMonitor.Populate(Object sender, EventArgs e)
at ActvMonitorSystemScreen.frmActvMonitor.InitializeExtension(Object sender, EventArgs e)
at OpenSolutions.Sys.DnaExtensions.Client.SecureCoreBaseForm.Initialize(Object sender, EventArgs e)
at OSI.Base.Windows.Forms.BaseForm.OnShown(EventArgs e)
at OSI.Core.Windows.Forms.CoreBaseForm.OnShown(EventArgs e)
at ActvMonitorSystemScreen.SecureCoreBaseFormShim.OnShown(EventArgs e)
at System.Windows.Forms.Form.CallShownEvent()
at System.Windows.Forms.Control.InvokeMarshaledCallbackDo(ThreadMethodEntry tme)
at System.Windows.Forms.Control.InvokeMarshaledCallbackHelper(Object obj)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Windows.Forms.Control.InvokeMarshaledCallback(ThreadMethodEntry tme)
at System.Windows.Forms.Control.InvokeMarshaledCallbacks()
at System.Windows.Forms.Control.WndProc(Message& m)
at System.Windows.Forms.ScrollableControl.WndProc(Message& m)
at System.Windows.Forms.Form.WndProc(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.OnMessage(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m)
at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)
at System.Windows.Forms.UnsafeNativeMethods.DispatchMessageW(MSG& msg)
at System.Windows.Forms.Application.ComponentManager.System.Windows.Forms.UnsafeNativeMethods.IMsoComponentManager.FPushMessageLoop(IntPtr dwComponentID, Int32 reason, Int32 pvLoopData)
at System.Windows.Forms.Application.ThreadContext.RunMessageLoopInner(Int32 reason, ApplicationContext context)
at System.Windows.Forms.Application.ThreadContext.RunMessageLoop(Int32 reason, ApplicationContext context)
at System.Windows.Forms.Application.Run(Form mainForm)
at OSI.Runtime.Program.Main(String[] args)
</StackTrace>
<ExceptionString>System.ServiceModel.CommunicationException: The underlying connection was closed: A connection that was expected to be kept alive was closed by the server. ---> System.Net.WebException: The underlying connection was closed: A connection that was expected to be kept alive was closed by the server. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
--- End of inner exception stack trace ---
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.FixedSizeReader.ReadPacket(Byte[] buffer, Int32 offset, Int32 count)
at System.Net.Security._SslStream.StartFrameHeader(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security._SslStream.StartReading(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security._SslStream.ProcessRead(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.TlsStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.PooledStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.Connection.SyncRead(HttpWebRequest request, Boolean userRetrievedStream, Boolean probeRead)
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.GetResponse()
at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)
--- End of inner exception stack trace ---</ExceptionString>
<InnerException>
<ExceptionType>System.Net.WebException, System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</ExceptionType>
<Message>The underlying connection was closed: A connection that was expected to be kept alive was closed by the server.</Message>
<StackTrace>
at System.Net.HttpWebRequest.GetResponse()
at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)
</StackTrace>
<ExceptionString>System.Net.WebException: The underlying connection was closed: A connection that was expected to be kept alive was closed by the server. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
--- End of inner exception stack trace ---
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.FixedSizeReader.ReadPacket(Byte[] buffer, Int32 offset, Int32 count)
at System.Net.Security._SslStream.StartFrameHeader(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security._SslStream.StartReading(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security._SslStream.ProcessRead(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.TlsStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.PooledStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.Connection.SyncRead(HttpWebRequest request, Boolean userRetrievedStream, Boolean probeRead)
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.GetResponse()
at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)</ExceptionString>
<InnerException>
<ExceptionType>System.IO.IOException, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</ExceptionType>
<Message>Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.</Message>
<StackTrace>
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.FixedSizeReader.ReadPacket(Byte[] buffer, Int32 offset, Int32 count)
at System.Net.Security._SslStream.StartFrameHeader(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security._SslStream.StartReading(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security._SslStream.ProcessRead(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.TlsStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.PooledStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.Connection.SyncRead(HttpWebRequest request, Boolean userRetrievedStream, Boolean probeRead)
</StackTrace>
<ExceptionString>System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
--- End of inner exception stack trace ---
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.FixedSizeReader.ReadPacket(Byte[] buffer, Int32 offset, Int32 count)
at System.Net.Security._SslStream.StartFrameHeader(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security._SslStream.StartReading(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security._SslStream.ProcessRead(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.TlsStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.PooledStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.Connection.SyncRead(HttpWebRequest request, Boolean userRetrievedStream, Boolean probeRead)</ExceptionString>
<InnerException>
<ExceptionType>System.Net.Sockets.SocketException, System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</ExceptionType>
<Message>An existing connection was forcibly closed by the remote host</Message>
<StackTrace>
at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
</StackTrace>
<ExceptionString>System.Net.Sockets.SocketException (0x80004005): An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)</ExceptionString>
<NativeErrorCode>2746</NativeErrorCode>
This service works for other clients just fine. It's just this client that's having an issue with requests over 5 minutes. Other clients on different web servers complete the request fine.
The five minute limit doesn't sound like any of the default timeouts
so I took a look at the Advanced settings for app pools and looked for settings that default to 5 minutes and look possibly problematic. I found a couple you might try. You might want to take this approach:
In IIS Manager, go to App Pools
Find your app pool, right click, select Advanced Settings
Scan the list of settings and look for the 5 minute settings and try selectively tuning by increasing the values (strongly suggest reading and learning what each does first)
A couple of 5 minute settings that jump out at me are Limit Interval (minutes) under CPU and Failure Interval (minutes) under Rapid-Fail Protection.
Can't guarantee this will be it but unless you are manually setting a default value to 5 minutes somewhere, this is a reasonable place to start looking for a 5 minute setting. Change one setting at a time, restart the app pool, and test. Note that if you have a problem like an infinite loop in your code, obviously no timeout value is going to help :-(
Good luck.
Related
I have a problem with exception appearing while closing my WPF application.
HResult=0x80131515
Message=The Read method cannot be called when another read operation is pending.
Source=System
StackTrace:
at System.Net.Security._SslStream.ProcessRead(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.Read(Byte[] buffer, Int32 offset, Int32 count)
at MySql.Data.MySqlClient.TimedStream.Read(Byte[] buffer, Int32 offset, Int32 count)
at MySql.Data.MySqlClient.MySqlStream.ReadFully(Stream stream, Byte[] buffer, Int32 offset, Int32 count)
at MySql.Data.MySqlClient.MySqlStream.LoadPacket()
at MySql.Data.MySqlClient.MySqlStream.ReadPacket()
at MySql.Data.MySqlClient.NativeDriver.GetResult(Int32& affectedRow, Int64& insertedId)
at MySql.Data.MySqlClient.Driver.GetResult(Int32 statementId, Int32& affectedRows, Int64& insertedId)
at MySql.Data.MySqlClient.Driver.NextResult(Int32 statementId, Boolean force)
at MySql.Data.MySqlClient.MySqlDataReader.NextResult()
at MySql.Data.MySqlClient.MySqlDataReader.Close()
at MySql.Data.MySqlClient.MySqlConnection.Close()
at MySql.Data.MySqlClient.MySqlConnection.Dispose(Boolean disposing)
at MySql.Data.MySqlClient.MySqlConnection.Finalize()
This exception was originally thrown at this call stack:
[External Code]
I am using MySQL.Data version 8.0.20
It appears only, when some MySQL queries working in background (like Select's from DB)
I am using the 'using' statements when the program opening MySQL Connections to close MySQL Connections after operations are done.
has anyone encountered this problem?
I am facing a critical problem while sending push notifications using C# in ASP.NET MVC. I put the code in the Global.asax file and execute it every 5 minutes.
My code works well on localhost, but it's not working when I upload to the production server.
I get this error:
System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send.
System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.
System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
--- End of inner exception stack trace ---
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.FixedSizeReader.ReadPacket(Byte[] buffer, Int32 offset, Int32 count)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Net.TlsStream.ProcessAuthentication(LazyAsyncResult result)
at System.Net.TlsStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.ConnectStreaenter code herem.WriteHeaders(Boolean async)
Please help me, because I am not understanding what the actual issue is, and my code runs fine on localhost.
We are developing a .Net application (compiled for .net 4.0) that communicates to a soap web service.
The application has been working fine on several machines when retreiving the http web response content. However on one of the machines (Windows Server 2008 R2) the application is not able to retreive the response. This is a code example:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("https://200.57.3.82:3443/AdministradorQr/WebServiceDodaPort?wsdl");
request.Method = "GET";
WebResponse response;
Stream responseStream = response.GetResponseStream();
string result = string.Empty;
using (StreamReader streamReader = new StreamReader(responseStream))
{
result = streamReader.ReadToEnd();
}
The exception we get is:
The underlying connection was closed: An unexpected error occurred on a send. System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. ---> System.IO.IOException: Unable to read data from the transport connection: An established connection was aborted by the software in your host machine. ---> System.Net.Sockets.SocketException: An established connection was aborted by the software in your host machine
at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
--- End of inner exception stack trace ---
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.FixedSizeReader.ReadPacket(Byte[] buffer, Int32 offset, Int32 count)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
at System.Net.TlsStream.CallProcessAuthentication(Object state)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean ignoreSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Net.TlsStream.ProcessAuthentication(LazyAsyncResult result)
at System.Net.TlsStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.PooledStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.ConnectStream.WriteHeaders(Boolean async)
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.GetResponse()
The thing is that using the same code with a different URL we get a sucessfull response, and the problem is only happening in one machine even though there is another machinge also using Windows 2008 that works just fine.
Currently we cannot upgrade .net framework to try using solutions like use of TLS 1.2 found in some places like this one: "The underlying connection was closed: An unexpected error occurred on a send." With SSL Certificate
As I mentioned, it is very strange that the same code works just fine in other machines using the same Windows/.net version and it is also strange that for other URLs like http://www.yahoo.com works fine
Not sure if it can help you but you can try adding below line in "Global.asax -> Application_Start" function. See it it works. Basically this code will force your application to use TLS1.2 only and you do not need to upgrade your framework.
"System.Net.ServicePointManager.SecurityProtocol = System.Net.SecurityProtocolType.Tls12;"
I have been working through a head-scratcher with WCF. I have a WCF Service (hosted as a Windows Service) that is called by a console application. It works great, but recently we have run into a timeout issue at 10 minutes when we had to run a query against a legacy system.
I use log4net in both the service and the client .exe, and what is strange is that the service actually completes the job and throws no exception. It runs the query (which takes about 12 minutes, and creates a file, and logs success).
The console application, however, logs an exception, claiming that the "host" returned a timeout or an error. I presume it's a timeout and not a buffer issue, because the exception always happens exactly 10 minutes to the second after the call is initially made.
Here is the exception/stack info from my log4net:
2015-07-28 17:35:47,364 [1]
System.ServiceModel.CommunicationException: The socket connection was aborted. This could be caused by an error processing your message or a receive timeout being exceeded by the remote host, or an underlying network resource issue. Local socket timeout was '00:29:59.9843999'. ---> System.IO.IOException: The read operation failed, see inner exception. ---> System.ServiceModel.CommunicationException: The socket connection was aborted. This could be caused by an error processing your message or a receive timeout being exceeded by the remote host, or an underlying network resource issue. Local socket timeout was '00:29:59.9843999'. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
at System.ServiceModel.Channels.SocketConnection.ReadCore(Byte[] buffer, Int32 offset, Int32 size, TimeSpan timeout, Boolean closing)
--- End of inner exception stack trace ---
at System.ServiceModel.Channels.SocketConnection.ReadCore(Byte[] buffer, Int32 offset, Int32 size, TimeSpan timeout, Boolean closing)
at System.ServiceModel.Channels.SocketConnection.Read(Byte[] buffer, Int32 offset, Int32 size, TimeSpan timeout)
at System.ServiceModel.Channels.DelegatingConnection.Read(Byte[] buffer, Int32 offset, Int32 size, TimeSpan timeout)
at System.ServiceModel.Channels.ConnectionStream.Read(Byte[] buffer, Int32 offset, Int32 count)
at System.Net.FixedSizeReader.ReadPacket(Byte[] buffer, Int32 offset, Int32 count)
at System.Net.Security.NegotiateStream.StartFrameHeader(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.NegotiateStream.ProcessRead(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
--- End of inner exception stack trace ---
at System.Net.Security.NegotiateStream.ProcessRead(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.NegotiateStream.Read(Byte[] buffer, Int32 offset, Int32 count)
at System.ServiceModel.Channels.StreamConnection.Read(Byte[] buffer, Int32 offset, Int32 size, TimeSpan timeout)
--- End of inner exception stack trace ---
Server stack trace:
at System.ServiceModel.Channels.StreamConnection.Read(Byte[] buffer, Int32 offset, Int32 size, TimeSpan timeout)
at System.ServiceModel.Channels.SessionConnectionReader.Receive(TimeSpan timeout)
at System.ServiceModel.Channels.SynchronizedMessageSource.Receive(TimeSpan timeout)
at System.ServiceModel.Channels.TransportDuplexSessionChannel.Receive(TimeSpan timeout)
at System.ServiceModel.Channels.TransportDuplexSessionChannel.TryReceive(TimeSpan timeout, Message& message)
at System.ServiceModel.Dispatcher.DuplexChannelBinder.Request(Message message, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)
Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at FileGeneratorConsole.FileGeneratorServiceReference.IFileGeneratorService.GenerateXmlFileFromSql(String sqlServer, String sqlDatabase, String sqlQuery, Int32 commandTimeout, Boolean allowEmptyResult, String outputFileName, String xslFileName, Boolean archiveExistingFile)
at FileGeneratorConsole.Program.Main(String[] args)
Now, I thought I had done my research and figured out what the issue could be. I assumed it was the sendTimeout and receiveTimeout that are part of the binding definition, so I've added that:
<bindings>
<netTcpBinding>
<binding name="NetTcpBinding_IFileGeneratorService"
maxBufferPoolSize="2147483647"
maxBufferSize="2147483647"
maxReceivedMessageSize="2147483647"
receiveTimeout="00:30:00"
closeTimeout="00:30:00"
openTimeout="00:30:00"
sendTimeout="00:30:00">
</binding>
</netTcpBinding>
</bindings>
And, if you notice in the exception above, you'll see that the error says:
Local socket timeout was '00:29:59.9843999'
It used to say 00:09:59.... so it looks like that took.
I also added this to the service config file, just in case the binding needed to be on both sides, but that didn't help.
Lastly, out of desperation, I added a few other things. I tried adding timeouts to the service section:
and I tried adding
to the behavior itself.
I'm at a loss now, it always dies at 10 minutes.
Thanks in advance
It's always simpler than you think.
Sadly, what was missing all this time was the bindingConfiguration reference in the service.endpoint. All this time, the service was running assumedly using the default binding definition. I had a binding defined in my config, but had missed the part where you specify:
I had:
<endpoint address="" binding="netTcpBinding" bindingConfiguration="">
I needed:
<endpoint address="" binding="netTcpBinding" bindingConfiguration="NetTcpBinding_IFileGeneratorService">
Once that was in place, my actual service endpoint was looking at the right binding properties.
Ok, for our site, which I will say is "https://www.foo-bar.com", and is a .NET MVC 4.5.1 application, we have the following code:
var http = (HttpWebRequest)WebRequest.Create("https://www.foo-bar.com/category.rss");
var response = http.GetResponse();
var stream = response.GetResponseStream();
var sr = new StreamReader(stream);
s = sr.ReadToEnd();
This works locally, AND it works in production under "http", but as soon as we go to https, we get the following error:
{
"ClassName":"System.Net.WebException",
"Message":"The underlying connection was closed: An unexpected error occurred on a send.",
"Data":null,
"InnerException":{
"ClassName":"System.IO.IOException",
"Message":"Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.",
"Data":null,
"InnerException":{
"NativeErrorCode":10054,
"ClassName":"System.Net.Sockets.SocketException",
"Message":"An existing connection was forcibly closed by the remote host",
"Data":null,
"InnerException":null,
"HelpURL":null,
"StackTraceString":" at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)\r\n at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)",
"RemoteStackTraceString":null,
"RemoteStackIndex":0,
"ExceptionMethod":"8\nReceive\nSystem, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089\nSystem.Net.Sockets.Socket\nInt32 Receive(Byte[], Int32, Int32, System.Net.Sockets.SocketFlags)",
"HResult":-2147467259,
"Source":"System",
"WatsonBuckets":null
},
"HelpURL":null,
"StackTraceString":" at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)\r\n at System.Net.FixedSizeReader.ReadPacket(Byte[] buffer, Int32 offset, Int32 count)\r\n at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)\r\n at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)\r\n at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)\r\n at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)\r\n at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)\r\n at System.Net.TlsStream.CallProcessAuthentication(Object state)\r\n at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)\r\n at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)\r\n at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)\r\n at System.Net.TlsStream.ProcessAuthentication(LazyAsyncResult result)\r\n at System.Net.TlsStream.Write(Byte[] buffer, Int32 offset, Int32 size)\r\n at System.Net.PooledStream.Write(Byte[] buffer, Int32 offset, Int32 size)\r\n at System.Net.ConnectStream.WriteHeaders(Boolean async)",
"RemoteStackTraceString":null,
"RemoteStackIndex":0,
"ExceptionMethod":"8\nRead\nSystem, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089\nSystem.Net.Sockets.NetworkStream\nInt32 Read(Byte[], Int32, Int32)",
"HResult":-2146232800,
"Source":"System",
"WatsonBuckets":null
},
"HelpURL":null,
"StackTraceString":" at System.Net.HttpWebRequest.GetResponse()\r\n at ASP._Page_Views_home_Test_cshtml.Execute() in c:\\inetpub\\wwwroot\\foobarweb_dev\\Views\\Home\\Test.cshtml:line 14",
"RemoteStackTraceString":null,
"RemoteStackIndex":0,
"ExceptionMethod":"8\nGetResponse\nSystem, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089\nSystem.Net.HttpWebRequest\nSystem.Net.WebResponse GetResponse()",
"HResult":-2146233079,
"Source":"System",
"WatsonBuckets":null
}
I've read suggestion to edit the following settings, and I have tinkered with them all, but no matter what combination I use, I still get the above error:
http.KeepAlive
http.ProtocolVersion
http.ServicePoint.ConnectionLimit
http.Timeout
If I try this in another project on my local computer, connecting to the "https" version of the site works. It is only when connecting from within the site over SSL.
I am stumped, and I'm not even sure if this is a code issue, a server issue, or maybe even a proxy/firewall issue.
Check the TLS version of the host "https://www.foo-bar.com" and the .NET TLS version of your project. They should be equal.
Check by "Immediate window":
System.Net.ServicePointManager.SecurityProtocol
The issue is something with the SSL -- probably a certificate issue of some sort. If it is public facing I would start by running SSL Labs Analyizer at it to see if that helps sniff out the cert problem.
I had this same problem and resolved it by adding the following to my Powershell script:
$AllProtocols = [System.Net.SecurityProtocolType]'Ssl3,Tls,Tls11,Tls12'
[System.Net.ServicePointManager]::SecurityProtocol = $AllProtocols
[System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy
Had this on a .NET Framework 4.0 app and the solution for us was as found here. Create the following four DWORD keys and set them all to a value of 1 to enable strong cryptography and have the OS manage which version to use:
HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\.NETFramework\v4.0.30319\SystemDefaultTlsVersions
HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\.NETFramework\v4.0.30319\SchUseStrongCrypto
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework\v4.0.30319\SystemDefaultTlsVersions
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework\v4.0.30319\SchUseStrongCrypto