I have a multi threaded .NET app that uses async I/O and AsyncCallbacks to handle the I/O completions. Rarely, the app will fail with an exception like:
Arithmetic operation resulted in an overflow.
at MyApp.ReadComplete(IAsyncResult ar) in c:\MyApp.cs:line 123
at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
at System.Net.ContextAwareResult.CompleteCallback(Object state)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Net.ContextAwareResult.Complete(IntPtr userToken)
at System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object result, IntPtr userToken)
at System.Net.Sockets.BaseOverlappedAsyncResult.CompletionPortCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)
Line 123 in MyApp.cs is the first executable line of the AsyncCallback and it is inside of a try/catch (Exception ex) but, the catch is NOT being executed.
Is the .NET Framework lying to me about where the exception occurred? Did the exception actually occur out in the async netherworld where I can't catch it? Why can't I catch this exception?
Try to use catch instead of catch(Exception ex). This catches everything incl. COM exceptions.
The actual answer is yes, the .NET Framework was wrong about where the exception occurred but, not because the exception occurred off in the async netherworld. The exception actually occurred in the callback method, many lines after the line indicated in the stack trace. The line where the exception actually occurred was NOT inside of a try/catch block so, the unhandled exception was expected.
I reported the incorrect stack trace to Microsoft but have no idea if it is considered a bug.
I think that it's safe to say that if your stack trace points to the first executable line of an async callback method, you should suspect that it could be wrong.
Related
I am using a TcpClient with an SslStream. In general everything works fine but we are getting occasional logs of crashes. It seems to be coming from unstable network connections, and I'm unfortunately not able to reproduce it locally despite trying to create "unexpected circumstances":
using Fiddler to replicate a very slow network connection
killing the server application while its sending data
disconnecting network cables
The situation is this:
I call BeginRead like this:
_tcpClientStream.BeginRead(receiveBuffer, 0, receiveBuffer.Length, pfnCallBack, null);
And then EndRead is being called like this:
iRx = _tcpClientStream.EndRead(asyn);
The only place that the iRx is used is to copy the received data to a new array:
var bytes = new byte[iRx];
Array.Copy(receiveBuffer, bytes, iRx);
In the OnDataReceived callback function, I call EndRead on the SslStream, which throws an exception that is handled:
Error in OnDataReceived System.IO.IOException: The read operation failed, see inner exception. ---> System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values.
Parameter name: size
at System.Net.Sockets.NetworkStream.BeginRead(Byte[] buffer, Int32 offset, Int32 size, AsyncCallback callback, Object state)
at System.Net.FixedSizeReader.StartReading()
at System.Net.FixedSizeReader.ReadCallback(IAsyncResult transportResult)
--- End of inner exception stack trace ---
at System.Net.Security._SslStream.EndRead(IAsyncResult asyncResult)
at System.Net.Security.SslStream.EndRead(IAsyncResult asyncResult)
at XXXXXXX.OnDataRecevied(IAsyncResult asyn)
Normally this is not a problem - this exception is caught and the TcpClient is closed and re-initiated to attempt to reconnect.
However, in this case there is a second Exception that is thrown and comes from one of the UnhandledException error handlers:
AppDomain.CurrentDomain.UnhandledException
Current.DispatcherUnhandledException
Current.Dispatcher.UnhandledException
TaskScheduler.UnobservedTaskException
This stack trace is like this:
System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'System.Net.Sockets.NetworkStream'.
at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
at System.Net.FixedSizeReader.ReadCallback(IAsyncResult transportResult)
at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
at System.Net.ContextAwareResult.CompleteCallback(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Net.ContextAwareResult.Complete(IntPtr userToken)
at System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object result, IntPtr userToken)
at System.Net.Sockets.BaseOverlappedAsyncResult.CompletionPortCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)
Our handling for this type of unhandled error is to display a message to the user that there is a fatal error and the software has to close.
My questions are:
Has anyone ever seen this error before and been able to reproduce the exact out of range exception I'm getting? Despite my attempts to recreate the issue (using the methods described above), I'm not able to ever recreate the exception, it is always the more standard Socket IO exception:
Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)
at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
--- End of inner exception stack trace ---
at System.Net.Security._SslStream.EndRead(IAsyncResult asyncResult)
at System.Net.Security.SslStream.EndRead(IAsyncResult asyncResult)
at XXXXXXXXX.OnDataRecevied(IAsyncResult asyn)
Is there any way I can handle this exception in my Socket code or is it always going to appear in one of these UnhandledException events? If so, what is the best practice way of handling this? Basically I need some logic in the event handler to identify that the software should not shut down if this specific error is received?
I have a WCF service
[ServiceContract(CallbackContract = typeof(ICallback))]
public interface IMonitorService
{
[OperationContract]
[FaultContract(typeof(MonitorFaultException))]
void StartMonitoring(MonitorRequest monitorRequest);
}
In the service implementation, I have following service behavior
[ServiceBehavior(ConcurrencyMode = ConcurrencyMode.Reentrant)]
I have two clients for this service
Console application (for testing).
Web application REST API.
In case of console application client, everything works fine. But when I use Web application client and there is a callback from service side I get following error:
An asynchronous operation cannot be started at this time. Asynchronous operations may only be started within an asynchronous handler or module or during certain events in the Page lifecycle. If this exception occurred while executing a Page, ensure that the Page is marked <%# Page Async="true" %>.
My web application controller looks like
// I am using same types as WCF for input just for simplicity
public ApiResponse StartMonitor(MonitorRequest request)
{
IMonitorService monitorService = new MonitorServiceClient(new InstanceContext(callbackEventHandler));
monitorService.StartMonitoring(request);
//create ApiResponse and return.
}
Console application also has the same code as shown above. The only difference here is I have a Console.Read() at the end of the main.
Following is the stack trace from windows event viewer
StackTrace: at
System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage4(MessageRpc&
rpc) at
System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage11(MessageRpc&
rpc) at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean
isOperationContextSet) at
System.ServiceModel.Dispatcher.ChannelHandler.DispatchAndReleasePump(RequestContext
request, Boolean cleanThread, OperationContext
currentOperationContext) at
System.ServiceModel.Dispatcher.ChannelHandler.HandleRequest(RequestContext
request, OperationContext currentOperationContext) at
System.ServiceModel.Dispatcher.ChannelHandler.AsyncMessagePump(IAsyncResult
result) at
System.ServiceModel.Dispatcher.ChannelHandler.OnAsyncReceiveComplete(IAsyncResult
result) at
System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(IAsyncResult
result) at System.Runtime.AsyncResult.Complete(Boolean
completedSynchronously) at
System.ServiceModel.Channels.TransportDuplexSessionChannel.TryReceiveAsyncResult.OnReceive(IAsyncResult
result) at
System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(IAsyncResult
result) at System.Runtime.AsyncResult.Complete(Boolean
completedSynchronously) at
System.ServiceModel.Channels.SynchronizedMessageSource.ReceiveAsyncResult.OnReceiveComplete(Object
state) at
System.ServiceModel.Channels.SessionConnectionReader.OnAsyncReadComplete(Object
state) at
System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(IAsyncResult
result) at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
at System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object result,
IntPtr userToken) at
System.Net.Security.NegotiateStream.ProcessFrameBody(Int32 readBytes,
Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest
asyncRequest) at
System.Net.Security.NegotiateStream.ReadCallback(AsyncProtocolRequest
asyncRequest) at
System.Net.AsyncProtocolRequest.CompleteRequest(Int32 result) at
System.Net.FixedSizeReader.CheckCompletionBeforeNextRead(Int32 bytes)
at System.Net.FixedSizeReader.ReadCallback(IAsyncResult
transportResult) at System.Runtime.AsyncResult.Complete(Boolean
completedSynchronously) at
System.ServiceModel.Channels.ConnectionStream.IOAsyncResult.OnAsyncIOComplete(Object
state) at
System.Net.Sockets.SocketAsyncEventArgs.OnCompleted(SocketAsyncEventArgs
e) at
System.Net.Sockets.SocketAsyncEventArgs.FinishOperationSuccess(SocketError
socketError, Int32 bytesTransferred, SocketFlags flags) at
System.Net.Sockets.SocketAsyncEventArgs.CompletionPortCallback(UInt32
errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped) at
System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32
errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)
InnerException: System.InvalidOperationException
But When I call the service on a separate thread everything starts working fine:-
Changing
monitorService.StartMonitoring(request)
to
Task.Run(() => monitorService.StartMonitoring(request)).Wait();
I also tried calling the async version (by changing the return types of controller accordingly)
await monitorService.StartMonitoringAsync(request);
but then the error messages changes to:
An asynchronous module or handler completed while an asynchronous operation was still pending.
EDIT The above error is not at the callback but it occurs as ApiResponse.
I have following questions:-
Q-1 Why console client callback is happening and web application callback is not happening? I guess Console.Read() (which I am doing in the console application) is making the thread to wait to receive the callback. Please clarify.
Q-2 Why callback is happening on web application when I shoot the call on a separate thread?
Q-3 Edit: New question In general on which thread callback happens from WCF service? Will it be a new thread at client side to handle callback every time?
Q-4 Why async version of the method is not working?
The application I've been working on has recently started experiencing intermittent hard crashes which I'm unable to explain. To make this more weird we have both AppDomain.CurrentDomain.UnhandledException and TaskScheduler.UnobservedTaskException set up to catch problems like these.
Usually when something goes wrong the above two work correctly, but we have certain circumstances where they are not catching unhandled exceptions. To make matters even worse the exception that's causing this is a NRE which means we have nothing to go on in terms of tracking down what's going on.
My first thought was that maybe the code we have in the unhandled exception handlers might be throwing exceptions, so we doubled down on exception handling here, but so far that hasn't made any difference. The logs we have also haven't been able to shine any light on this.
This may be a bug with webapi and/or the owin bits we are using to host the application, and I'm going to try bumping the dependencies to see if that helps, but I don't think it will make any difference.
Is there a way to trap unhandled exceptions occurring in the thread pool? Is there some kind of diagnostics we can enable to get more info as to where this is happening?
I'm at my wits end, any help would be greatly appreciated.
System.NullReferenceException: Object reference not set to an instance of an object.
at System.Net.HttpListener.EndGetContext(IAsyncResult asyncResult)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at Microsoft.Owin.Host.HttpListener.OwinHttpListener.<ProcessRequestsAsync>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.<ThrowAsync>b__5(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
When using tasks we do not do async void, but we have one case of Task.Run where we do not await or hang on the task. This is:
Task.Run(() => CleanUpAllExpiredErrors());
This method cleans up expired client errors (we capture client side stack traces on unhandled client errors).
By adding extensive tracing and logging we saw that this crash was consistently happening when serving static assets, specifically stylesheets. We white-listed our static assets directories to not require authentication and the crash went away.
We configure certain endpoints to not require authentication by doing the following:
var listener = (HttpListener)app.Properties["System.Net.HttpListener"];
listener.AuthenticationSchemeSelectorDelegate = request =>
{
// Options requests should always be anonymous otherwise preflight cors requests
// will fail in Firefox.
if (request.HttpMethod == "OPTIONS")
{
return AuthenticationSchemes.Anonymous;
}
// here we add additional whitelisted paths
var pathsWithoutAuthentication = KnownPaths.AnonymousWhiteList.Concat(KnownPaths.Assets);
return pathsWithoutAuthentication.Any(a => request.RawUrl.ToLower().StartsWith(a))
? AuthenticationSchemes.Anonymous
: AuthenticationSchemes.IntegratedWindowsAuthentication;
};
Hi I 've a little question,
I'm using socket and I find many situation of infinite recursion see source: http://pastebin.com/Cbd2Z2uE
the problem is this:
private static void ReceiveCallback(IAsyncResult ar)
{
....
// receive again
socket.BeginReceive(state.Buffer,
0,
StateObject.BufferSize,
0,
ReceiveCallback,
state);
....
}
So we have an async function that is called again in a recursive way. Here We have a problem with the stack?
To be more clear, this situation is ok and recursion is fine, the problem and question is:
in this case can I have a problem of stackoverflow?
thanks
You are not recursing (or in danger or recursing). You are simply scheduling a callback to be executed when an IO competion port receives data for you to process (checked source code to see this). This will not happen while your function is running on the current thread because the very same thread has to check for message on IO port via polling or another worker thread will run your function all together. In either case you are not recursing or in danger of recusing so the stack will not blow up (socket.BeginReceive should return immediatly & will not call your function in it).
I believe you are looking at scenario 14 shown on this page but it may be the previous scenario since your console app has a message pump. But the bottom line is that there is no actual functions recursing in your application (or supporitng OS framework functionality used by it).
Your ReceiveCallback is called by framework code unless you call it directly.
BeginReceive stores specified delegate as callback function, and it doesn't calls the delegate at that time.
For example, I added some codes to get stack trace:
private static void ReceiveCallback(IAsyncResult ar)
{
Console.WriteLine(Environment.StackTrace);
// retrieve the state and socket
and gets:
at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo)
at System.Environment.get_StackTrace()
at Program.ReceiveCallback(IAsyncResult ar) in r:\Temp\LINQPad\aqwfvqfb\query_ettlka.cs:line 102
at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
at System.Net.ContextAwareResult.CompleteCallback(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Net.ContextAwareResult.Complete(IntPtr userToken)
at System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object result, IntPtr userToken)
at System.Net.Sockets.BaseOverlappedAsyncResult.CompletionPortCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)
And It's shown even you call BeginReceive in ReceiveCallback.
I'm getting the following error in our QA environment but only through the IIS7 reverse proxy. If I hit the application server directly it works fine. In our DEV environment it works fine through the proxy or directly.
I'm not really sure where to go with this, any ideas?
Server Error in '/' Application.
Padding is invalid and cannot be removed.
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.
Exception Details: System.Security.Cryptography.CryptographicException: Padding is invalid and cannot be removed.
Source Error:
An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.
Stack Trace:
[CryptographicException: Padding is invalid and cannot be removed.]
System.Security.Cryptography.RijndaelManagedTransform.DecryptData(Byte[] inputBuffer, Int32 inputOffset, Int32 inputCount, Byte[]& outputBuffer, Int32 outputOffset, PaddingMode paddingMode, Boolean fLast) +2910
System.Security.Cryptography.RijndaelManagedTransform.TransformFinalBlock(Byte[] inputBuffer, Int32 inputOffset, Int32 inputCount) +286
System.Security.Cryptography.CryptoStream.FlushFinalBlock() +51
System.Web.Configuration.MachineKeySection.EncryptOrDecryptData(Boolean fEncrypt, Byte[] buf, Byte[] modifier, Int32 start, Int32 length, IVType ivType, Boolean useValidationSymAlgo) +318
System.Web.UI.Page.DecryptStringWithIV(String s, IVType ivType) +115
System.Web.Handlers.AssemblyResourceLoader.System.Web.IHttpHandler.ProcessRequest(HttpContext context) +140
System.Web.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +599
System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +171
Version Information: Microsoft .NET Framework Version:2.0.50727.4952; ASP.NET Version:2.0.50727.4927
If this is happens on page with out post back then probably your proxy is convertting the url in lowercase chars and break the code.
I have one more bigger answer here with a trick to log the error and see whats really get on your page:
CryptographicException: Padding is invalid and cannot be removed and Validation of viewstate MAC failed