I am using a TcpClient with an SslStream. In general everything works fine but we are getting occasional logs of crashes. It seems to be coming from unstable network connections, and I'm unfortunately not able to reproduce it locally despite trying to create "unexpected circumstances":
using Fiddler to replicate a very slow network connection
killing the server application while its sending data
disconnecting network cables
The situation is this:
I call BeginRead like this:
_tcpClientStream.BeginRead(receiveBuffer, 0, receiveBuffer.Length, pfnCallBack, null);
And then EndRead is being called like this:
iRx = _tcpClientStream.EndRead(asyn);
The only place that the iRx is used is to copy the received data to a new array:
var bytes = new byte[iRx];
Array.Copy(receiveBuffer, bytes, iRx);
In the OnDataReceived callback function, I call EndRead on the SslStream, which throws an exception that is handled:
Error in OnDataReceived System.IO.IOException: The read operation failed, see inner exception. ---> System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values.
Parameter name: size
at System.Net.Sockets.NetworkStream.BeginRead(Byte[] buffer, Int32 offset, Int32 size, AsyncCallback callback, Object state)
at System.Net.FixedSizeReader.StartReading()
at System.Net.FixedSizeReader.ReadCallback(IAsyncResult transportResult)
--- End of inner exception stack trace ---
at System.Net.Security._SslStream.EndRead(IAsyncResult asyncResult)
at System.Net.Security.SslStream.EndRead(IAsyncResult asyncResult)
at XXXXXXX.OnDataRecevied(IAsyncResult asyn)
Normally this is not a problem - this exception is caught and the TcpClient is closed and re-initiated to attempt to reconnect.
However, in this case there is a second Exception that is thrown and comes from one of the UnhandledException error handlers:
AppDomain.CurrentDomain.UnhandledException
Current.DispatcherUnhandledException
Current.Dispatcher.UnhandledException
TaskScheduler.UnobservedTaskException
This stack trace is like this:
System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'System.Net.Sockets.NetworkStream'.
at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
at System.Net.FixedSizeReader.ReadCallback(IAsyncResult transportResult)
at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
at System.Net.ContextAwareResult.CompleteCallback(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Net.ContextAwareResult.Complete(IntPtr userToken)
at System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object result, IntPtr userToken)
at System.Net.Sockets.BaseOverlappedAsyncResult.CompletionPortCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)
Our handling for this type of unhandled error is to display a message to the user that there is a fatal error and the software has to close.
My questions are:
Has anyone ever seen this error before and been able to reproduce the exact out of range exception I'm getting? Despite my attempts to recreate the issue (using the methods described above), I'm not able to ever recreate the exception, it is always the more standard Socket IO exception:
Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)
at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
--- End of inner exception stack trace ---
at System.Net.Security._SslStream.EndRead(IAsyncResult asyncResult)
at System.Net.Security.SslStream.EndRead(IAsyncResult asyncResult)
at XXXXXXXXX.OnDataRecevied(IAsyncResult asyn)
Is there any way I can handle this exception in my Socket code or is it always going to appear in one of these UnhandledException events? If so, what is the best practice way of handling this? Basically I need some logic in the event handler to identify that the software should not shut down if this specific error is received?
Related
The error is as following:
The transport process failed during message processing with the following call stack: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.InvalidOperationException: Cannot find the X.509 certificate using the following search criteria: StoreName 'My', StoreLocation 'CurrentUser', FindType 'FindByThumbprint', FindValue 'c454f0a61e4cb0ed77ae87ede892ff09e584eca2'.
at System.ServiceModel.Security.SecurityUtils.GetCertificateFromStoreCore(StoreName storeName, StoreLocation storeLocation, X509FindType findType, Object findValue, EndpointAddress target, Boolean throwIfMultipleOrNoMatch)
at System.ServiceModel.Security.SecurityUtils.GetCertificateFromStore(StoreName storeName, StoreLocation storeLocation, X509FindType findType, Object findValue, EndpointAddress target)
at System.ServiceModel.Security.X509CertificateInitiatorClientCredential.SetCertificate(StoreLocation storeLocation, StoreName storeName, X509FindType findType, Object findValue)
at Zeva.WCFHelper.ClientBaseWrapper`1..ctor(BindingSettings bindingSettings, String address, String spnIdentity, Maybe`1 certificateAuthenticationSettings)
--- End of inner exception stack trace ---
at SampleCompany.TransportAgent.ModificationAgent.OnEndOfDataHandler(ReceiveMessageEventSource source, EndOfDataEventArgs eodArgs) in C:\Users\rani\Source\Workspaces\Mobility\Dev\SampleCompany\SampleCompany.TransportAgent\ModificationAgent.cs:line 48
at Microsoft.Exchange.Data.Transport.Smtp.SmtpReceiveAgent.Invoke(String eventTopic, Object source, Object e)
at Microsoft.Exchange.Data.Transport.Internal.MExRuntime.Dispatcher.Invoke(MExSession session)
at Microsoft.Exchange.Data.Transport.Internal.MExRuntime.MExSession.AsyncInvoke(Object state)
at Microsoft.Exchange.Data.Transport.Internal.MExRuntime.MExSession.BeginInvoke(String topic, Object source, Object e, AsyncCallback callback, Object callbackState)
at Microsoft.Exchange.Protocols.Smtp.BaseDataSmtpCommand.RaiseEODEvent(Object state)
at Microsoft.Exchange.Protocols.Smtp.BaseDataSmtpCommand.OnEod(Boolean isAsync)
at Microsoft.Exchange.Protocols.Smtp.BaseDataSmtpCommand.RawDataReceived(Byte[] data, Int32 offset, Int32 numBytes)
at Microsoft.Exchange.Protocols.Smtp.SmtpInSession.ReadComplete(IAsyncResult asyncResult)
at Microsoft.Exchange.Net.LazyAsyncResult.WorkerThreadComplete(Object state)
at Microsoft.Exchange.Net.LazyAsyncResult.Complete(IntPtr userToken)
at Microsoft.Exchange.Net.LazyAsyncResult.ProtectedInvokeCallback(Object value, IntPtr userToken)
at Microsoft.Exchange.Net.NetworkConnection.InvokeRecvCallback(LazyAsyncResultWithTimeout asyncResult, Object result)
at Microsoft.Exchange.Net.NetworkConnection.ReadDataAvailable(IAsyncResult asyncResult)
at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Net.ContextAwareResult.Complete(IntPtr userToken)
at System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object result, IntPtr userToken)
at System.Net.Sockets.BaseOverlappedAsyncResult.CompletionPortCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)
Description:
I am implementing a transport agent that do some modification emails, it is required that I call a WCF service that returns some data and based on that data I do the modification.
the issue arises when calling the WCF service (which is very well tested and in use for production). apparently, the transport agent cannot find the certificates in the current user store.
What I have already tried:
1. install the certificate (pfx file) into current user account (logically this should not work as the current user for transport agent is Network service).
2. install for the local machine, also did not work.
3. install for the service account (Microsoft Exchange Transport), also did not find the certificate.
So my question is how to install certificates to be used by code within transport agents?
I have followed the tutorial in ( http://www.c-sharpcorner.com/UploadFile/a6fd36/understand-self-host-of-a-web-apiC-Sharp/ ) and ( http://www.c-sharpcorner.com/uploadfile/a6fd36/understanding-how-to-call-the-web-api-from-a-client-applica/ ) but seem to still be getting System.AggregateException on
//Call HttpClient.GetAsync to send a GET request to the appropriate URI
HttpResponseMessage resp = client.GetAsync("api/books").Result;
and
var resp = client.GetAsync(string.Format("api/books/{0}", id)).Result;
and
var resp = client.GetAsync(query).Result;
The error states: An unhandled exception of type 'System.AggregateException' occurred in mscorlib.dll
Exception Detail -
System.AggregateException was unhandled
HResult=-2146233088
Message=One or more errors occurred.
Source=mscorlib
StackTrace:
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
at System.Threading.Tasks.Task`1.get_Result()
at TestClient.Program.ListAllBooks() in c:\Users\wa\Documents\Visual Studio 2013\Projects\SelfHost\TestClient\Program.cs:line 35
at TestClient.Program.Main(String[] args) in c:\Users\wa\Documents\Visual Studio 2013\Projects\SelfHost\TestClient\Program.cs:line 20
at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args)
at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)
at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()
InnerException: System.Net.Http.HttpRequestException
HResult=-2146233088
Message=An error occurred while sending the request.
InnerException: System.Net.WebException
HResult=-2146233079
Message=Unable to connect to the remote server
Source=System
StackTrace:
at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
InnerException: System.Net.Sockets.SocketException
HResult=-2147467259
Message=No connection could be made because the target machine actively refused it [::1]:8080
Source=System
ErrorCode=10061
NativeErrorCode=10061
StackTrace:
at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
InnerException:
Any Ideas?
Thank You
Please read the last line :)
InnerException: System.Net.WebException
HResult=-2146233079
Message=Unable to connect to the remote server
The application I've been working on has recently started experiencing intermittent hard crashes which I'm unable to explain. To make this more weird we have both AppDomain.CurrentDomain.UnhandledException and TaskScheduler.UnobservedTaskException set up to catch problems like these.
Usually when something goes wrong the above two work correctly, but we have certain circumstances where they are not catching unhandled exceptions. To make matters even worse the exception that's causing this is a NRE which means we have nothing to go on in terms of tracking down what's going on.
My first thought was that maybe the code we have in the unhandled exception handlers might be throwing exceptions, so we doubled down on exception handling here, but so far that hasn't made any difference. The logs we have also haven't been able to shine any light on this.
This may be a bug with webapi and/or the owin bits we are using to host the application, and I'm going to try bumping the dependencies to see if that helps, but I don't think it will make any difference.
Is there a way to trap unhandled exceptions occurring in the thread pool? Is there some kind of diagnostics we can enable to get more info as to where this is happening?
I'm at my wits end, any help would be greatly appreciated.
System.NullReferenceException: Object reference not set to an instance of an object.
at System.Net.HttpListener.EndGetContext(IAsyncResult asyncResult)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at Microsoft.Owin.Host.HttpListener.OwinHttpListener.<ProcessRequestsAsync>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.<ThrowAsync>b__5(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
When using tasks we do not do async void, but we have one case of Task.Run where we do not await or hang on the task. This is:
Task.Run(() => CleanUpAllExpiredErrors());
This method cleans up expired client errors (we capture client side stack traces on unhandled client errors).
By adding extensive tracing and logging we saw that this crash was consistently happening when serving static assets, specifically stylesheets. We white-listed our static assets directories to not require authentication and the crash went away.
We configure certain endpoints to not require authentication by doing the following:
var listener = (HttpListener)app.Properties["System.Net.HttpListener"];
listener.AuthenticationSchemeSelectorDelegate = request =>
{
// Options requests should always be anonymous otherwise preflight cors requests
// will fail in Firefox.
if (request.HttpMethod == "OPTIONS")
{
return AuthenticationSchemes.Anonymous;
}
// here we add additional whitelisted paths
var pathsWithoutAuthentication = KnownPaths.AnonymousWhiteList.Concat(KnownPaths.Assets);
return pathsWithoutAuthentication.Any(a => request.RawUrl.ToLower().StartsWith(a))
? AuthenticationSchemes.Anonymous
: AuthenticationSchemes.IntegratedWindowsAuthentication;
};
Hi I 've a little question,
I'm using socket and I find many situation of infinite recursion see source: http://pastebin.com/Cbd2Z2uE
the problem is this:
private static void ReceiveCallback(IAsyncResult ar)
{
....
// receive again
socket.BeginReceive(state.Buffer,
0,
StateObject.BufferSize,
0,
ReceiveCallback,
state);
....
}
So we have an async function that is called again in a recursive way. Here We have a problem with the stack?
To be more clear, this situation is ok and recursion is fine, the problem and question is:
in this case can I have a problem of stackoverflow?
thanks
You are not recursing (or in danger or recursing). You are simply scheduling a callback to be executed when an IO competion port receives data for you to process (checked source code to see this). This will not happen while your function is running on the current thread because the very same thread has to check for message on IO port via polling or another worker thread will run your function all together. In either case you are not recursing or in danger of recusing so the stack will not blow up (socket.BeginReceive should return immediatly & will not call your function in it).
I believe you are looking at scenario 14 shown on this page but it may be the previous scenario since your console app has a message pump. But the bottom line is that there is no actual functions recursing in your application (or supporitng OS framework functionality used by it).
Your ReceiveCallback is called by framework code unless you call it directly.
BeginReceive stores specified delegate as callback function, and it doesn't calls the delegate at that time.
For example, I added some codes to get stack trace:
private static void ReceiveCallback(IAsyncResult ar)
{
Console.WriteLine(Environment.StackTrace);
// retrieve the state and socket
and gets:
at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo)
at System.Environment.get_StackTrace()
at Program.ReceiveCallback(IAsyncResult ar) in r:\Temp\LINQPad\aqwfvqfb\query_ettlka.cs:line 102
at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
at System.Net.ContextAwareResult.CompleteCallback(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Net.ContextAwareResult.Complete(IntPtr userToken)
at System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object result, IntPtr userToken)
at System.Net.Sockets.BaseOverlappedAsyncResult.CompletionPortCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)
And It's shown even you call BeginReceive in ReceiveCallback.
I have a multi threaded .NET app that uses async I/O and AsyncCallbacks to handle the I/O completions. Rarely, the app will fail with an exception like:
Arithmetic operation resulted in an overflow.
at MyApp.ReadComplete(IAsyncResult ar) in c:\MyApp.cs:line 123
at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
at System.Net.ContextAwareResult.CompleteCallback(Object state)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Net.ContextAwareResult.Complete(IntPtr userToken)
at System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object result, IntPtr userToken)
at System.Net.Sockets.BaseOverlappedAsyncResult.CompletionPortCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)
Line 123 in MyApp.cs is the first executable line of the AsyncCallback and it is inside of a try/catch (Exception ex) but, the catch is NOT being executed.
Is the .NET Framework lying to me about where the exception occurred? Did the exception actually occur out in the async netherworld where I can't catch it? Why can't I catch this exception?
Try to use catch instead of catch(Exception ex). This catches everything incl. COM exceptions.
The actual answer is yes, the .NET Framework was wrong about where the exception occurred but, not because the exception occurred off in the async netherworld. The exception actually occurred in the callback method, many lines after the line indicated in the stack trace. The line where the exception actually occurred was NOT inside of a try/catch block so, the unhandled exception was expected.
I reported the incorrect stack trace to Microsoft but have no idea if it is considered a bug.
I think that it's safe to say that if your stack trace points to the first executable line of an async callback method, you should suspect that it could be wrong.