Develop Reference

ObjectDisposedException SslStream and TcpClient

I am using a TcpClient with an SslStream. In general everything works fine but we are getting occasional logs of crashes. It seems to be coming from unstable network connections, and I'm unfortunately not able to reproduce it locally despite trying to create "unexpected circumstances":
using Fiddler to replicate a very slow network connection
killing the server application while its sending data
disconnecting network cables
The situation is this:
I call BeginRead like this:
_tcpClientStream.BeginRead(receiveBuffer, 0, receiveBuffer.Length, pfnCallBack, null);
And then EndRead is being called like this:
iRx = _tcpClientStream.EndRead(asyn);
The only place that the iRx is used is to copy the received data to a new array:
var bytes = new byte[iRx];
Array.Copy(receiveBuffer, bytes, iRx);
In the OnDataReceived callback function, I call EndRead on the SslStream, which throws an exception that is handled:
Error in OnDataReceived System.IO.IOException: The read operation failed, see inner exception. ---> System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values.
Parameter name: size
at System.Net.Sockets.NetworkStream.BeginRead(Byte[] buffer, Int32 offset, Int32 size, AsyncCallback callback, Object state)
at System.Net.FixedSizeReader.StartReading()
at System.Net.FixedSizeReader.ReadCallback(IAsyncResult transportResult)
--- End of inner exception stack trace ---
at System.Net.Security._SslStream.EndRead(IAsyncResult asyncResult)
at System.Net.Security.SslStream.EndRead(IAsyncResult asyncResult)
at XXXXXXX.OnDataRecevied(IAsyncResult asyn)
Normally this is not a problem - this exception is caught and the TcpClient is closed and re-initiated to attempt to reconnect.
However, in this case there is a second Exception that is thrown and comes from one of the UnhandledException error handlers:
AppDomain.CurrentDomain.UnhandledException
Current.DispatcherUnhandledException
Current.Dispatcher.UnhandledException
TaskScheduler.UnobservedTaskException
This stack trace is like this:
System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'System.Net.Sockets.NetworkStream'.
at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
at System.Net.FixedSizeReader.ReadCallback(IAsyncResult transportResult)
at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
at System.Net.ContextAwareResult.CompleteCallback(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Net.ContextAwareResult.Complete(IntPtr userToken)
at System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object result, IntPtr userToken)
at System.Net.Sockets.BaseOverlappedAsyncResult.CompletionPortCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)
Our handling for this type of unhandled error is to display a message to the user that there is a fatal error and the software has to close.
My questions are:
Has anyone ever seen this error before and been able to reproduce the exact out of range exception I'm getting? Despite my attempts to recreate the issue (using the methods described above), I'm not able to ever recreate the exception, it is always the more standard Socket IO exception:
Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)
at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
--- End of inner exception stack trace ---
at System.Net.Security._SslStream.EndRead(IAsyncResult asyncResult)
at System.Net.Security.SslStream.EndRead(IAsyncResult asyncResult)
at XXXXXXXXX.OnDataRecevied(IAsyncResult asyn)
Is there any way I can handle this exception in my Socket code or is it always going to appear in one of these UnhandledException events? If so, what is the best practice way of handling this? Basically I need some logic in the event handler to identify that the software should not shut down if this specific error is received?

Avoid recursive event in c#

Hi I 've a little question,
I'm using socket and I find many situation of infinite recursion see source: http://pastebin.com/Cbd2Z2uE
the problem is this:
private static void ReceiveCallback(IAsyncResult ar)
{
....
// receive again
socket.BeginReceive(state.Buffer,
0,
StateObject.BufferSize,
0,
ReceiveCallback,
state);
....
}
So we have an async function that is called again in a recursive way. Here We have a problem with the stack?
To be more clear, this situation is ok and recursion is fine, the problem and question is:
in this case can I have a problem of stackoverflow?
thanks

You are not recursing (or in danger or recursing). You are simply scheduling a callback to be executed when an IO competion port receives data for you to process (checked source code to see this). This will not happen while your function is running on the current thread because the very same thread has to check for message on IO port via polling or another worker thread will run your function all together. In either case you are not recursing or in danger of recusing so the stack will not blow up (socket.BeginReceive should return immediatly & will not call your function in it).
I believe you are looking at scenario 14 shown on this page but it may be the previous scenario since your console app has a message pump. But the bottom line is that there is no actual functions recursing in your application (or supporitng OS framework functionality used by it).

Your ReceiveCallback is called by framework code unless you call it directly.
BeginReceive stores specified delegate as callback function, and it doesn't calls the delegate at that time.
For example, I added some codes to get stack trace:
private static void ReceiveCallback(IAsyncResult ar)
{
Console.WriteLine(Environment.StackTrace);
// retrieve the state and socket
and gets:
at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo)
at System.Environment.get_StackTrace()
at Program.ReceiveCallback(IAsyncResult ar) in r:\Temp\LINQPad\aqwfvqfb\query_ettlka.cs:line 102
at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
at System.Net.ContextAwareResult.CompleteCallback(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Net.ContextAwareResult.Complete(IntPtr userToken)
at System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object result, IntPtr userToken)
at System.Net.Sockets.BaseOverlappedAsyncResult.CompletionPortCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)
And It's shown even you call BeginReceive in ReceiveCallback.

System.TimeoutException when calling wcf service from Biztalk

I am getting the exception shown below. BizTalk is able to call the service when there is very light load. But once I increase the load and the number of messsages to transmit increases after a while I keep getting this message for all the calls to the service.
Event Type: Error
Event Source: BizTalk Server 2006
Event Category: (1)
Event ID: 5799
Date: 11/21/2011
Time: 12:55:38 PM
User: N/A
Computer: CLBIZBTS07S
Description:
The description for Event ID ( 5799 ) in Source ( BizTalk Server 2006 ) cannot be found. The local computer may not have the necessary registry information or message DLL files to display messages from a
remote computer. You may be able to use the /AUXSOURCE= flag to retrieve this description; see Help and Support for details. The following information is part of the event: sndManageServiceEndPoint_WCF,
http://service.com:8100/ManageService.svc/ws, The adapter failed to transmit message going to send port "sndManageServiceEndPoint_WCF" with URL "http://service.com:8100/ManageService.svc/ws". It will be
retransmitted after the retry interval specified for this Send Port. Details:"System.TimeoutException: Client is unable to finish the security negotiation within the configured timeout (00:01:00). The
current negotiation leg is 3 (00:00:20.6239920). ---> System.TimeoutException: The request channel timed out attempting to send after 00:00:20.6239920. Increase the timeout value passed to the call to
Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout. ---> System.TimeoutException: The HTTP request to
'http://service.com:8100/ManageService.svc/ws' has exceeded the allotted timeout of 00:00:20.6230000. The time allotted to this operation may have been a portion of a longer
timeout. ---> System.Net.WebException: The operation has timed out
at System.Net.HttpWebRequest.GetRequestStream(TransportContext& context)
at System.Net.HttpWebRequest.GetRequestStream()
at System.ServiceModel.Channels.HttpOutput.WebRequestHttpOutput.GetOutputStream()
--- End of inner exception stack trace ---
at System.ServiceModel.Channels.HttpOutput.WebRequestHttpOutput.GetOutputStream()
at System.ServiceModel.Channels.HttpOutput.Send(TimeSpan timeout)
at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelRequest.SendRequest(Message message, TimeSpan timeout)
at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)
--- End of inner exception stack trace ---
at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)
at System.ServiceModel.Security.IssuanceTokenProviderBase`1.DoNegotiation(TimeSpan timeout)
--- End of inner exception stack trace ---
Server stack trace:
at System.ServiceModel.Security.IssuanceTokenProviderBase`1.DoNegotiation(TimeSpan timeout)
at System.ServiceModel.Security.SspiNegotiationTokenProvider.OnOpen(TimeSpan timeout)
at System.ServiceModel.Security.WrapperSecurityCommunicationObject.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Security.SecurityUtils.OpenCommunicationObject(ICommunicationObject obj, TimeSpan timeout)
at System.ServiceModel.Security.SymmetricSecurityProtocol.OnOpen(TimeSpan timeout)
at System.ServiceModel.Security.WrapperSecurityCommunicationObject.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Channels.SecurityChannelFactory`1.ClientSecurityChannel`1.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Security.SecuritySessionSecurityTokenProvider.DoOperation(SecuritySessionOperation operation, EndpointAddress target, Uri via, SecurityToken currentToken, TimeSpan timeout)
at System.ServiceModel.Security.SecuritySessionSecurityTokenProvider.GetTokenCore(TimeSpan timeout)
at System.IdentityModel.Selectors.SecurityTokenProvider.GetToken(TimeSpan timeout)
at System.ServiceModel.Security.SecuritySessionClientSettings`1.ClientSecuritySessionChannel.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at System.ServiceModel.ICommunicationObject.Open()
at Microsoft.BizTalk.Adapter.Wcf.Runtime.WcfClient`2.GetChannel[TChannel](IBaseMessage bizTalkMessage, ChannelFactory`1& cachedFactory)
at Microsoft.BizTalk.Adapter.Wcf.Runtime.WcfClient`2.SendMessage(IBaseMessage bizTalkMessage)". .
On the service side I have the set maxReceivedMessageSize="2147483647" in the endpoint
and in the behavior section I have set
<serviceThrottling
maxConcurrentCalls="16"
maxConcurrentInstances="2147483647"
maxConcurrentSessions="10"/>
These two settings did not help.
When I turned on the tracing on wcf service I see these type of errors
Activity Name Process action 'http://schemas.xmlsoap.org/ws/2005/02/trust/RST/SCT/Cancel'.
Time 2011-11-22 15:03:12.5301
Level Error
Source System.ServiceModel
Process w3wp
Thread 32
Computer FRA1BSHR01S
Trace Identifier/Code http://msdn.microsoft.com/en-US/library/System.ServiceModel.Diagnostics.ThrowingException.aspx
System.ServiceModel.EndpointNotFoundException, System.ServiceModel, Version=3.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089
There was no channel that could accept the message with action 'http://schemas.xmlsoap.org/ws/2005/02/trust/RST/SCT/Cancel'.
System.ServiceModel.Dispatcher.ErrorBehavior.ThrowAndCatch(Exception e, Message message)
System.ServiceModel.Channels.DatagramChannelDemuxer`2.ProcessItem(TInnerItem item)
System.ServiceModel.Channels.DatagramChannelDemuxer`2.HandleReceiveResult(IAsyncResult result)
System.ServiceModel.Channels.DatagramChannelDemuxer`2.OnReceiveComplete(IAsyncResult result)
System.ServiceModel.Channels.DatagramChannelDemuxer`2.OnReceiveCompleteStatic(IAsyncResult result)
System.ServiceModel.Diagnostics.Utility.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
System.ServiceModel.AsyncResult.Complete(Boolean completedSynchronously)
System.ServiceModel.Channels.ReplyChannel.HelpReceiveRequestAsyncResult.OnReceiveRequest(IAsyncResult result)
System.ServiceModel.Diagnostics.Utility.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
System.ServiceModel.AsyncResult.Complete(Boolean completedSynchronously)
System.ServiceModel.Channels.InputQueue`1.AsyncQueueReader.Set(Item item)
System.ServiceModel.Channels.InputQueue`1.EnqueueAndDispatch(Item item, Boolean canDispatchOnThisThread)
System.ServiceModel.Channels.InputQueue`1.EnqueueAndDispatch(T item, ItemDequeuedCallback dequeuedCallback, Boolean canDispatchOnThisThread)
System.ServiceModel.Channels.InputQueueChannel`1.EnqueueAndDispatch(TDisposable item, ItemDequeuedCallback dequeuedCallback, Boolean canDispatchOnThisThread)
System.ServiceModel.Channels.SingletonChannelAcceptor`3.Enqueue(QueueItemType item, ItemDequeuedCallback dequeuedCallback, Boolean canDispatchOnThisThread)
System.ServiceModel.Channels.SingletonChannelAcceptor`3.Enqueue(QueueItemType item, ItemDequeuedCallback dequeuedCallback)
System.ServiceModel.Channels.HttpChannelListener.HttpContextReceived(HttpRequestContext context, ItemDequeuedCallback callback)
System.ServiceModel.Activation.HostedHttpTransportManager.HttpContextReceived(HostedHttpRequestAsyncResult result)
System.ServiceModel.Activation.HostedHttpRequestAsyncResult.BeginRequest()
System.ServiceModel.Activation.HostedHttpRequestAsyncResult.OnBeginRequest(Object state)
System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.WorkItem.Invoke2()
System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.WorkItem.Invoke()
System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.ProcessCallbacks()
System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.CompletionCallback(Object state)
System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.ScheduledOverlapped.IOCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
System.ServiceModel.Diagnostics.Utility.IOCompletionThunk.UnhandledExceptionFrame(UInt32 error, UInt32 bytesRead, NativeOverlapped* nativeOverlapped)
System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)
[TraceRecord] Severity Error
TraceIdentifier http://msdn.microsoft.com/en-US/library/System.ServiceModel.Diagnostics.ThrowingException.aspx
Description Throwing an exception.
AppDomain /LM/W3SVC/1373337114/Root-4-129664654928269938

Although increasing the timeout on both ends of the wire (BTS and WCF Service) may help avoid the error in some instances, you might just be kicking the can further down the road and you probably need to get to the root cause (does this specific web service call NORMALLY take over 20 seconds to complete?).
It is likely that under high load, or low capacity conditions, that your WCF Web Service cannot keep up with the rate of messages that BizTalk is able to send to it.
Depending on what your WCF Service does, you might look at optimising:
Database
SQL blocking while waiting for locks
DB performance may reduced during admin tasks such as Reindexing Jobs, Backups etc
IIS / ASP.NET
See if requests are being queued
Look at any throttling on the WCF Service side (e.g. MaxConcurrentCalls)
If there is no chance of further optimisation (or scale up / out) on the WCF Service side, you may need to look at throttling the number of concurrent orchestrations (using a variant of a Singleton Orch) or limiting the number of threads on your WCF Send Port Adapter (unfortunately, this will affect all messages, so you might need to split out separate SendHost
Be wary when simply 'maxing out' all of the WCF knobs - settings like maxReceivedMessageSize (or maxBufferSize) ="2147483647" will often result in OutOfMemoryExceptions and / or increase the threat surface area.

You need to change the timeout of the WCF service.
I am not sure if that solution is correct, but it is definitely in the web.config file.

I had similar problem when using SOAP adapter(I'm still in biztalk 2006)
You should try the following :
YourMessage(SOAP.ClientConnectionTimeout) = 900000;
900000 is the default value, so increase it to fit your need, the time is in milliseconds.
If this property does not apply to WCF web services, I assume there is a WCF equivalent for that property.

System.IO.IOException: The handshake failed due to an unexpected > packet format?

Does anyone know what this means?
System.Net.WebException: The
underlying connection was closed: An
unexpected error occurred on a send.
---> System.IO.IOException: The handshake failed due to an unexpected
packet format. at
System.Net.Security.SslState.StartReadFrame(Byte[]
buffer, Int32 readBytes,
AsyncProtocolRequest asyncRequest) at
System.Net.Security.SslState.StartReceiveBlob(Byte[]
buffer, AsyncProtocolRequest
asyncRequest) at
System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken
message, AsyncProtocolRequest
asyncRequest) at
System.Net.Security.SslState.StartSendBlob(Byte[]
incoming, Int32 count,
AsyncProtocolRequest asyncRequest) at
System.Net.Security.SslState.ForceAuthentication(Boolean
receiveFirst, Byte[] buffer,
AsyncProtocolRequest asyncRequest) at
System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult
lazyResult) at
System.Net.TlsStream.CallProcessAuthentication(Object
state) at
System.Threading.ExecutionContext.runTryCode(Object
userData) at
System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode
code, CleanupCode backoutCode, Object
userData) at
System.Threading.ExecutionContext.RunInternal(ExecutionContext
executionContext, ContextCallback
callback, Object state) at
System.Threading.ExecutionContext.Run(ExecutionContext
executionContext, ContextCallback
callback, Object state) at
System.Net.TlsStream.ProcessAuthentication(LazyAsyncResult
result) at
System.Net.TlsStream.Write(Byte[]
buffer, Int32 offset, Int32 size) at
System.Net.PooledStream.Write(Byte[]
buffer, Int32 offset, Int32 size) at
System.Net.ConnectStream.WriteHeaders(Boolean
async) --- End of inner exception
stack trace --- at
System.Net.HttpWebRequest.GetResponse()
at
System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan
timeout) at
EDIT:
This is the method I was calling:
_productsService = new ProductsPortTypeClient();
_productsService.GetResortProducts(GetProductsCredentials(),
GetResortProductParams());

It looks like a problem with SSL authentication, it fails at handshake phase, maybe two implementations are not compatible, you may check the WSDL format of the response, figure out what format should be used for communication and check if the one sent by you complies with it (you may use any kind of a network monitor software, eg. Microsoft Network Monitor)

I just wanted to make an answer based on one of the comments because it can cause this error to happen.
In some cases, if you have a BaseURL while calling a API, then the problem could happen if you mixed up http and https. This happened in my case while calling a Google API. This is a simple fix in some cases and can happen to anyone.
Hope this help's someone out.

In my case it was Skype blocking 443 port. Go to Skype tools-advanced-connections and disable "Use port 80 & 443 for connections..."

summary: wrong port was used.
My service was running in development on port :58328 but next time I debugged it was running on port :44315, and this caused the issue.
So changing the port on the client side to 44315 helped to resolve the issue!

Can an async I/O throw an uncatchable exception?

I have a multi threaded .NET app that uses async I/O and AsyncCallbacks to handle the I/O completions. Rarely, the app will fail with an exception like:
Arithmetic operation resulted in an overflow.
at MyApp.ReadComplete(IAsyncResult ar) in c:\MyApp.cs:line 123
at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
at System.Net.ContextAwareResult.CompleteCallback(Object state)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Net.ContextAwareResult.Complete(IntPtr userToken)
at System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object result, IntPtr userToken)
at System.Net.Sockets.BaseOverlappedAsyncResult.CompletionPortCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)
Line 123 in MyApp.cs is the first executable line of the AsyncCallback and it is inside of a try/catch (Exception ex) but, the catch is NOT being executed.
Is the .NET Framework lying to me about where the exception occurred? Did the exception actually occur out in the async netherworld where I can't catch it? Why can't I catch this exception?

Try to use catch instead of catch(Exception ex). This catches everything incl. COM exceptions.

The actual answer is yes, the .NET Framework was wrong about where the exception occurred but, not because the exception occurred off in the async netherworld. The exception actually occurred in the callback method, many lines after the line indicated in the stack trace. The line where the exception actually occurred was NOT inside of a try/catch block so, the unhandled exception was expected.
I reported the incorrect stack trace to Microsoft but have no idea if it is considered a bug.
I think that it's safe to say that if your stack trace points to the first executable line of an async callback method, you should suspect that it could be wrong.