We are having a problem with our virtual application shutting down. Site is running ASP.NET 4.5
Exception summary:
System.InvalidOperationException: Request to the service at '~/Services/ServiceExceptionLogger.svc' cannot be dispatched because the virtual application at '/Site' is shutting down.
Stack trace:
System.ServiceModel.ServiceActivationException: Request to the service at
'~/Services/ServiceExceptionLogger.svc' cannot be dispatched because the virtual application
at '/Site' is shutting down. ---> System.InvalidOperationException: Request to the service at '~/Services/ServiceExceptionLogger.svc' cannot be dispatched because the virtual
application at '/Site' is shutting down.
--- End of inner exception stack trace ---
at System.Runtime.AsyncResult.End[TAsyncResult](IAsyncResult result)
at System.ServiceModel.Activation.HostedHttpRequestAsyncResult.End(IAsyncResult result)
at System.Web.HttpApplication.CallHandlerExecutionStep.OnAsyncHandlerCompletion(IAsyncResult ar)
Or here is another version of it that shows up in the event logs
Stack Trace:
System.ServiceModel.ServiceActivationException: Request to the service at '~/Services/Service.svc' cannot be dispatched because the virtual application at '/Site' is shutting down. ---> System.InvalidOperationException: Request to the service at '~/Services/Service.svc' cannot be dispatched because the virtual application at '/Site' is shutting down.
--- End of inner exception stack trace ---
at System.Runtime.AsyncResult.End[TAsyncResult](IAsyncResult result)
at System.ServiceModel.Activation.HostedHttpRequestAsyncResult.End(IAsyncResult result)
at System.Web.HttpApplication.CallHandlerExecutionStep.OnAsyncHandlerCompletion(IAsyncResult ar)
Anyone have any ideas or seen this before? Seems to happen quite frequently on one of our environments, like 10+ times a day. On the QA environment its not happening at all. Idle timeout is set to 20minutes on both environments. Reason for 20 minute timeout is to avoid wasted memory
Update 1: Found some supplementary information on logging app pool recycles here https://webmasters.stackexchange.com/questions/17630/which-event-log-file-does-iis-7-app-pool-log-to/17633#17633?newreg=d562bf378cc545b49a7ea8f2a3c1b48d
Update 2: Apparently the app pool is not recycling when the above exception occurs. We are watching the process in task manager and it is not shutting down or reducing the memory when this exception occurs. Also, no log entry occurs (see update 1) when this exception happens
After escalating this to Microsoft support, we think this issue is a bug .NET 4.5.1. It happens most often when running in integrated mode, but also occurs in classic mode, although much less frequently. We haven't been able to pinpoint the actual case or problem but it doesn't seem to be occuring on older .NET runtimes.
Assuming this is what you are experiencing:
In IIS, Application Pools recycle themselves every ~20 minutes (I don't remember the exact default time) after inactivity.
This is changeable though in the IIS Manager under the "Advanced Settings" for your App Pool.
Related
We are running an NServiceBus-based service, using the NServiceBus.Host.exe host process.
Twice in production during the last few months the Windows Service has suddenly stopped, leaving the following event in the Application Event Log:
Application: NServiceBus.Host.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.InvalidOperationException
Stack:
at System.Transactions.TransactionState.ChangeStatePromotedPhase0(System.Transactions.InternalTransaction)
at System.Transactions.Phase0VolatileDemultiplexer.InternalPrepare()
at System.Transactions.VolatileDemultiplexer.PoolablePrepare(System.Object)
at System.Transactions.Oletx.OletxVolatileEnlistment.Prepare(System.Transactions.Oletx.OletxVolatileEnlistmentContainer)
at System.Transactions.Oletx.OletxPhase0VolatileEnlistmentContainer.Phase0Request(Boolean)
at System.Transactions.Oletx.OletxTransactionManager.ShimNotificationCallback(System.Object, Boolean)
at System.Threading._ThreadPoolWaitOrTimerCallback.PerformWaitOrTimerCallback(System.Object, Boolean)
We got this error during a period with some minutes of network instability (e.g. lots of timeouts against the database, which are visible in our log4net log files)
Any ideas as to what is failing here?
We see no fatal errors in our log4net logfiles.
Versions:
Windows Server 2008 R2
.NET Framework 4.5.2
NServiceBus 4.7.5
NHibernate 3.3.3.4001 (used for saga, subscription and timeout persister)
SQL Server 2012
It seems the behavior you see is related to how NHibernate handles the TransactionCompleted event. This questions is also somehow related to this question.
The AdoNetWithDistributedTransactionFactory registers an anonymous delegate on the TransactionCompleted event. This event gets fired on a background thread by using ThreadPool.QueueUserWorkItem. If that operation throws an exception due to connectivity issues with your database server (i.ex. due to a network partition) this exception gets raised as an unobserved exception on the AppDomain. UnhandledExceptions tear down the AppDomain and therefore also the NServiceBus.Host.
The best possible workaround for that would be to register an UnhandledException handler on the current AppDomain like the following
AppDomain.CurrentDomain.UnhandledException += OnUnhandledException
private static void OnUnhandledException(object sender, UnhandledExceptionEventArgs e)
{
LogManager.GetLogger(typeof(AppDomain)).Fatal(“Unhandled exception”, e.ExceptionObject as Exception);
}
More information see
When this fixed the intermediate problem it would make sense to find the root cause of the connection issues with your database server in combination with NHibernate
We have an application that runs on the Microsoft Azure cloud platform. The communication between some components is made using Service Bus. Everything was working fine, until recently we started to get the following type of timeout exceptions:
When calling QueueClient x.Send(...)
Exception rethrown at [0]: at
Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult
result) at
Microsoft.ServiceBus.Messaging.Sbmp.DuplexRequestBindingElement.DuplexRequestSessionChannel.DuplexCorrelationAsyncResult.End(IAsyncResult
result) at
Microsoft.ServiceBus.Messaging.Channels.ReconnectBindingElement.ReconnectChannelFactory `1.RequestSessionChannel.RequestAsyncResult.b__4(RequestAsyncResult
thisPtr, IAsyncResult r) at
Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.StepCallback(IAsyncResult
result)
When calling NamespaceManager x.GetQueue(...)
PROGRESS queue processing failed. System.TimeoutException: The request
has timed out after 60000 milliseconds. The successful completion of
the request cannot be determined. Additional queries should be made to
determine whether or not the operation has succeeded.
TrackingId:bdffb6bd-5367-4573-aaa3-8ea9a03f5a2b,TimeStamp:5/28/2015
8:39:46 AM ---> System.Net.WebException: The request was aborted: The
request was canceled. at
System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at
Microsoft.ServiceBus.Messaging.ServiceBusResourceOperations.GetAsyncResult`1.b__49(GetAsyncResult`1
thisPtr, IAsyncResult r) at
Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.StepCallback(IAsyncResult
result)
When calling NamespaceManager x.SubscriptionExists(...)
Exception doing periodic work: System.TimeoutException: The request
has timed out after 00:10:00 milliseconds. The successful completion
of the request cannot be determined. Additional queries should be made
to determine whether or not the operation has succeeded. Server stack
trace: Exception rethrown at [0]: at
Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult
result) at
Microsoft.ServiceBus.NamespaceManager.OnEndSubscriptionExists(IAsyncResult
result) at
Microsoft.ServiceBus.NamespaceManager.SubscriptionExists(String
topicPath, String name) ...
When calling QueueClient x.Receive(...)
PROGRESS queue processing failed.
Microsoft.ServiceBus.Messaging.MessagingCommunicationException: Error
during communication with Service Bus. Check the connection
information, then retry. --->
System.ServiceModel.CommunicationObjectFaultedException: Internal
Server Error: The server did not provide a meaningful reply; this
might be caused by a premature session shutdown.
TrackingId:04ba0220-0350-4806-9c65-c2bba9671054, Timestamp:28.05.2015
13:00:55 Server stack trace: Exception rethrown at [0]: at
Microsoft.ServiceBus.Common.ExceptionDispatcher.Throw(Exception
exception) at
Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult
result) at
Microsoft.ServiceBus.Messaging.Sbmp.DuplexRequestBindingElement.DuplexRequestSessionChannel.DuplexCorrelationAsyncResult.End(IAsyncResult
result) at
Microsoft.ServiceBus.Messaging.Sbmp.DuplexRequestBindingElement.DuplexRequestSessionChannel.EndRequest(IAsyncResult
result) at
Microsoft.ServiceBus.Messaging.Channels.ReconnectBindingElement.ReconnectChannelFactory`1.RequestSessionChannel.RequestAsyncResult.b__4(RequestAsyncResult
thisPtr, IAsyncResult r) at
Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.StepCallback(IAsyncResult
result) ….
The exceptions are clearly related to ServiceBus and they are non-deterministic. The functions throwing them, e.g., Send, GetQueue, SubscriptionExists, are called no more than 100-120 times per minute. We changed nothing in the code and increasing the timeout values (even to ridiculously high values, like 10min) did not help. Also, we do not believe it is some network related problem (on our side) since the same error occur when the application is run from different places.
Has anyone else encountered recently these kind of exceptions? Is there a problem on the side of Microsoft or we are missing something?
A few weeks back we had sudden and unexplainable timing issues crop up with our Service Bus app that has been in production for many months. Ours continued to work but every few calls would take 10+ seconds when they are normally 100-200 millisconds. This went on for a couple weeks and I spent most of that trying to figure out what was going on and never did as the problem suddenly vanished.
We did learn that new Service Bus namespaces we created in the same and other data centers for testing while the problem was occurring did not exhibit the same issue. The Service Bus group offered no help and would only say response times are not guaranteed only the SLA is.
I had similar problem when my running code started to generate Timeout exception. Upon research found that firewall blocked the port used for communication. However, port 80 and 443 were still open. So adding following line of code worked for me:
ServiceBusEnvironment.SystemConnectivity.Mode = ConnectivityMode.Https;
I maintain a very large Silverlight application that makes heavy use of WCF RIA Services.
When a user is in the SL application, and the app pool running the RIA services is restarted for any reason, subsequent calls from the client all fail until the user closes the app, and comes in from the main entry point again.
The error in the logs does not correlate to the problem it describes. These services work just fine if a user comes in "clean" essentially.
That said, there should be no reason for this, services do restart from time to time.
The error I get is below. Does anyone have any ideas at what to look at?
[ServiceActivationException]: The service '/ade/Services/DomainServices-Web-
UserService.svc'
cannot be activated due to an exception during compilation.
The exception message is: Entity 'DomainServices.Web.Role_Permission'
has a property 'PermissionReference' with an unsupported type
As stated above, this same call, 10 seconds before the restart, works swimmingly.
Any help would be most appreciated.
I'm sorry I cant answer your question but I can confirm that I get a very similar problem with our large Silverlight application that also makes heavy use of WCF RIA Services. Every so often when the app pool recycles due it's scheduled recycle I see the following type of exceptions in the event log on the IIS server.
WebHost failed to process a request. Sender Information:
System.ServiceModel.ServiceHostingEnvironment+HostingManager/12036987
Exception: System.ServiceModel.ServiceActivationException: The service
'/Services/DomainServices-Web-ServiceLayer-ReportsDomainService.svc' cannot
be activated due to an exception during compilation. The exception
message is: Invalid Include specification for member
'Activity.ActivitySourceType'. Non-projection includes can only be
specified on members with the AssociationAttribute applied.. --->
System.InvalidOperationException: Invalid Include specification for
member 'Activity.ActivitySourceType'. Non-projection includes can only
be specified on members with the AssociationAttribute applied. at
System.ServiceModel.DomainServices.Server.DomainServiceDescription.ValidateEntityTypes()
at
System.ServiceModel.DomainServices.Server.DomainServiceDescription.Initialize()
at
System.ServiceModel.DomainServices.Server.DomainServiceDescription.CreateDescription(Type
domainServiceType) at
System.Collections.Concurrent.ConcurrentDictionary2.GetOrAdd(TKey
key, Func2 valueFactory) at
System.ServiceModel.DomainServices.Server.DomainServiceDescription.GetDescription(Type
domainServiceType) at
System.ServiceModel.DomainServices.Hosting.DomainServiceHost..ctor(Type
domainServiceType, Uri[] baseAddresses) at
System.ServiceModel.DomainServices.Hosting.DomainServiceHostFactory.CreateServiceHost(Type
serviceType, Uri[] baseAddresses) at
System.ServiceModel.Activation.ServiceHostFactory.CreateServiceHost(String
constructorString, Uri[] baseAddresses) at
System.ServiceModel.ServiceHostingEnvironment.HostingManager.CreateService(String
normalizedVirtualPath) at
System.ServiceModel.ServiceHostingEnvironment.HostingManager.ActivateService(String
normalizedVirtualPath) at
System.ServiceModel.ServiceHostingEnvironment.HostingManager.EnsureServiceAvailable(String
normalizedVirtualPath) --- End of inner exception stack trace ---
at
System.ServiceModel.ServiceHostingEnvironment.HostingManager.EnsureServiceAvailable(String
normalizedVirtualPath) at
System.ServiceModel.ServiceHostingEnvironment.EnsureServiceAvailableFast(String
relativeVirtualPath) Process Name: w3wp Process ID: 15172
These exceptions are logged immediately after the application pool recycles and our users cannot use the application until I manually recycle the application pool again.
Have you made any progress on this?
Thanks
Kevin
What exactly does all this mean below. I am running an asynchronous web request that calls a page that sends code to my database. Then my database fires off a stored procedure that runs into one of my SQL assemblies. Most times it works, but sometimes I get this:
Thread was being aborted.
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.
Exception Details: System.Threading.ThreadAbortException: Thread was being aborted.
Source Error:
An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.
Stack Trace:
[ThreadAbortException: Thread was being aborted.]
System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +486
System.Web.ApplicationStepManager.ResumeSteps(Exception error) +501
System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProcessRequest(HttpContext context, AsyncCallback cb, Object extraData) +123
System.Web.HttpRuntime.ProcessRequestInternal(HttpWorkerRequest wr) +379
The reason why you are getting this error most likely is that you are getting timed out. Increase the ExecutionTimeout since at default its 110 seconds.
<compilation debug="false"></compilation>
<httpRuntime executionTimeout="1800"/> //1800 seconds / 30 minutes.
Here's an article that goes more into detail as to how to increase the ExecutionTimeout.
The app domain could be getting recycled which can happen for a number of reasons. That can cause a thread abort exception.
Here is an article which explains some of the reasons for a recycle
If a process is long running, IIS/ASP.Net is not usually a good place to run it.
We have an application that can create e-books.
This application has an export module that creates an AIR file but this can take a while (some books have 2500 pages).
If we export we get the following error:
Thread was being aborted.
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.
Exception Details: System.Threading.ThreadAbortException: Thread was being aborted.
Source Error:
An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.
Stack Trace:
[ThreadAbortException: Thread was being aborted.]
System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +501
System.Web.ApplicationStepManager.ResumeSteps(Exception error) +564
System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProcessRequest(HttpContext context, AsyncCallback cb, Object extraData) +141
System.Web.HttpRuntime.ProcessRequestInternal(HttpWorkerRequest wr) +436
I've changed my runtime executiontimeout to 3600 secs but it keeps crashing arround 3 minutes.
so it is time related ... everytime we approach the 3 minutes it crashes, I hope someone can help me out.
I think Paul is right about the cause of the exception. Both IIS and ASP.NET have settings that limit the maximum amount of time a request can take. For ASP.NET it's in the Machine.Config file (look for the httpRuntime element, executionTimeout attribute). It's set to 90 seconds on my development machine.
I would however not advise you to change that setting as it's there to make sure your application doesn't hang on a bad request.
Long running tasks should use asynchronous execution. With async execution, the actual work is handled on a separate thread. This frees the thread that handles the request to handle other requests which is good for the overall performance of your application.
There are some good articles on this available. For example : http://msdn.microsoft.com/en-us/magazine/cc163725.aspx
Quite often, this error actually occurs from an OutOfMemory exception.
Is there an InnerException available?
IIS has a 'run-away' thread protection that will kill a thread/appdomain if it runs for too long.