We are running an NServiceBus-based service, using the NServiceBus.Host.exe host process.
Twice in production during the last few months the Windows Service has suddenly stopped, leaving the following event in the Application Event Log:
Application: NServiceBus.Host.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.InvalidOperationException
Stack:
at System.Transactions.TransactionState.ChangeStatePromotedPhase0(System.Transactions.InternalTransaction)
at System.Transactions.Phase0VolatileDemultiplexer.InternalPrepare()
at System.Transactions.VolatileDemultiplexer.PoolablePrepare(System.Object)
at System.Transactions.Oletx.OletxVolatileEnlistment.Prepare(System.Transactions.Oletx.OletxVolatileEnlistmentContainer)
at System.Transactions.Oletx.OletxPhase0VolatileEnlistmentContainer.Phase0Request(Boolean)
at System.Transactions.Oletx.OletxTransactionManager.ShimNotificationCallback(System.Object, Boolean)
at System.Threading._ThreadPoolWaitOrTimerCallback.PerformWaitOrTimerCallback(System.Object, Boolean)
We got this error during a period with some minutes of network instability (e.g. lots of timeouts against the database, which are visible in our log4net log files)
Any ideas as to what is failing here?
We see no fatal errors in our log4net logfiles.
Versions:
Windows Server 2008 R2
.NET Framework 4.5.2
NServiceBus 4.7.5
NHibernate 3.3.3.4001 (used for saga, subscription and timeout persister)
SQL Server 2012
It seems the behavior you see is related to how NHibernate handles the TransactionCompleted event. This questions is also somehow related to this question.
The AdoNetWithDistributedTransactionFactory registers an anonymous delegate on the TransactionCompleted event. This event gets fired on a background thread by using ThreadPool.QueueUserWorkItem. If that operation throws an exception due to connectivity issues with your database server (i.ex. due to a network partition) this exception gets raised as an unobserved exception on the AppDomain. UnhandledExceptions tear down the AppDomain and therefore also the NServiceBus.Host.
The best possible workaround for that would be to register an UnhandledException handler on the current AppDomain like the following
AppDomain.CurrentDomain.UnhandledException += OnUnhandledException
private static void OnUnhandledException(object sender, UnhandledExceptionEventArgs e)
{
LogManager.GetLogger(typeof(AppDomain)).Fatal(“Unhandled exception”, e.ExceptionObject as Exception);
}
More information see
When this fixed the intermediate problem it would make sense to find the root cause of the connection issues with your database server in combination with NHibernate
Related
I am seeing a number of these errors in the logs for a .NET Core 2.2.x web app:
Connection ID ""13546832108852449106"", Request ID ""80018f57-0407-bc00-b63f-84710c7967bb"": An unhandled exception was thrown by the application.
System.ObjectDisposedException: The CancellationTokenSource has been disposed.
at System.Threading.CancellationTokenSource.ThrowObjectDisposedException()
at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext.<>c__DisplayClass314_0.<AbortIO>b__0(Object t)
As you can see, the error occurs in the .net core framework rather than my code.
I looked up the specific Request Id mentioned in the error and it seems to have processed just fine. The error appears in the log about a second after the request has processed and returned data. I also checked the Event Viewer and there wasn't anything suspicious around the time frame of the error.
How can I troubleshoot this issue?
The Windows 2016 server has .NET Core 2.2.6 runtime installed.
This is the error encountered in Event log of BizTalk Admin
xlang/s engine event log entry: Uncaught exception (see the 'inner
exception' below) has suspended an instance of service
'orchestration.AsyncOrchestration(ebb0442e-bff4-cd1c-f5dc-aff7c5086e57)'.
The service instance will remain suspended until administratively
resumed or terminated. If resumed the instance will continue from its
last persisted state and may re-throw the same unexpected exception.
InstanceId: 50fc5f8b-c9b9-44e1-9781-2a1391c145ec Shape name:
Expression_1 ShapeId: ddfccde4-3444-47c4-977d-228e66ffaae6 Exception
thrown from: segment 1, progress 17 Inner exception: Resolution of the
dependency failed, type =
"BizTalkComponents.Interface.ISsoSettingsReader", name = "(none)".
Exception occurred while: while resolving. Exception is:
InvalidOperationException - The current type,
BizTalkComponents.Interface.ISsoSettingsReader, is an interface and
cannot be constructed.
Are you missing a type mapping?
First, this has noting to do with SOAPUI or any specific client.
It seems that your deployment process has failed to include the SSO lookup components that are called in the failing Orchestration.
So, whichever set of component the app is using must be installed on the server.
I have these lines in my App.xaml.cs, to globally catch the exceptions:
this.Dispatcher.UnhandledException += new
DispatcherUnhandledExceptionEventHandler(
Current_DispatcherUnhandledException);
AppDomain.CurrentDomain.UnhandledException += new
UnhandledExceptionEventHandler(
CurrentDomain_UnhandledException);
Application.Current.DispatcherUnhandledException += new
DispatcherUnhandledExceptionEventHandler(
Current_DispatcherUnhandledException);
Most exceptions can be caught and a log file will be generated so that I know what is causing the error. But on my customer's PC (only windows 8 has
this problem), the WPF application sometimes crashed but I was not able to get the exception log, means the exception handlers above was not called.
Is there any handler I can set to catch exception? Or is there any log collector/reader like android DDMS for Windows/WPF app?
Thank you!
Since .NET 4.0 you could try and HandleProcessCorruptedStateExceptionsAttribute This attribute allows you to handle exceptions that since .Net 4 terminate the process. This is rather dangerous as you are allowing a process to continue (handle the exception) even when its memory has become corrupted.
In general: do not trust the application after catching such an exception. Log the exception and stop the application. Only when you know what is causing the exception and you are sure the application can recover you could apply this attribute to a method and make it resume normal execution.
My asp.net application gives an exception once in a while as a result application pool stopped. and it needs to be started manually.
Does anyone know how to trace it, how to find what cause the problem?
I don't see any errors in Event Viewer.
In IIS 7.0, events (configurable) and exceptions (always) are logged to the Windows event log when an application pool recycles.
Sccot Guthrie's round-up of Tess's related posts is great:
Outstanding Blog for Learning how to Debug ASP.NET App Issues (Memory Leaks, Crashes, Deadlocks, etc)
ASP.NET 2.0 Crash case study: Unhandled exceptions
Try putting an Application_Error method on Global.asax.cs like this:
protected void Application_Error(object sender, EventArgs e)
{
// Get the exception object.
Exception exception = Server.GetLastError();
_Log.Error(exception.Message, exception);
// Clear the error from the server
Server.ClearError();
}
The _Log line assumes you're using log4net or some other logging framework but you could do whatever you want with the error.
PS - There's some misinformation in some of the other comments. But I don't have enough rep to comment on them. All IIS / WCF web service apps that I've written will terminate if any thread has an unhandled exception. I can't speak to whether that's true of ASP.Net apps as well.
Are you absolutely certain your application pool is stopping due to an error? There are many reasons the application pool can stop. Unless configured otherwise by default it will stop if inactive after a set ammount of time. I think about 20 minutes.
I am using VSTS 2008 + C# + .Net 3.5 to develop a Windows Service application. The code is very simple below, and I find when File.Copy throws exception (e.g. path not valid, no permission, etc.), the service will crash and pop-up a dialog to let me debug. My confusion is, I think unhandled exception of a thread created by a Windows service will never make the service crash. Why my service crashes and the debug dialog pop-up?
public partial class Service1 : ServiceBase
{
public Service1()
{
InitializeComponent();
}
protected override void OnStart(string[] args)
{
Thread t = new Thread (WriteF);
t.Start();
}
static void WriteF() {
File.Copy("dummy.txt", #"\\somelocation\dummy.txt");
}
protected override void OnStop()
{
}
}
I think unhandled exception of a thread created by a Windows service will never make the service crash
Your assumption is simply wrong. Unhandled exceptions in a Windows service will cause it to crash. The debug dialog appears because there is a debugger configured on your system. From .NET 2.0 onwards, an unhandled exception on any thread shuts down the whole application. Just because you're using a thread doesn't mean exceptions in the thread will magically go away.
Your file copy operation is throwing an exception which is not being caught. Simple as that. The service will fail unless you wrap the File.Copy operation in a try catch block. It does not matter that the operation is being performed on a background thread. I would also suggest adding some logging otherwise you won't discover why it's failing.
You start a new thread for running the code in WriteF. If this code fails with an exception, the runtime will try to find an exception handler by unwinding the stack for that particular thread. If it fails in locating a handler, the runtime has no option but the report an unhandled exception and thus the process is terminated. Remember exception handling is per thread. This is not a Windows Service issue.