Microsoft Azure Service Bus Timeout Exceptions - c#

We have an application that runs on the Microsoft Azure cloud platform. The communication between some components is made using Service Bus. Everything was working fine, until recently we started to get the following type of timeout exceptions:
When calling QueueClient x.Send(...)
Exception rethrown at [0]: at
Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult
result) at
Microsoft.ServiceBus.Messaging.Sbmp.DuplexRequestBindingElement.DuplexRequestSessionChannel.DuplexCorrelationAsyncResult.End(IAsyncResult
result) at
Microsoft.ServiceBus.Messaging.Channels.ReconnectBindingElement.ReconnectChannelFactory `1.RequestSessionChannel.RequestAsyncResult.b__4(RequestAsyncResult
thisPtr, IAsyncResult r) at
Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.StepCallback(IAsyncResult
result)
When calling NamespaceManager x.GetQueue(...)
PROGRESS queue processing failed. System.TimeoutException: The request
has timed out after 60000 milliseconds. The successful completion of
the request cannot be determined. Additional queries should be made to
determine whether or not the operation has succeeded.
TrackingId:bdffb6bd-5367-4573-aaa3-8ea9a03f5a2b,TimeStamp:5/28/2015
8:39:46 AM ---> System.Net.WebException: The request was aborted: The
request was canceled. at
System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at
Microsoft.ServiceBus.Messaging.ServiceBusResourceOperations.GetAsyncResult`1.b__49(GetAsyncResult`1
thisPtr, IAsyncResult r) at
Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.StepCallback(IAsyncResult
result)
When calling NamespaceManager x.SubscriptionExists(...)
Exception doing periodic work: System.TimeoutException: The request
has timed out after 00:10:00 milliseconds. The successful completion
of the request cannot be determined. Additional queries should be made
to determine whether or not the operation has succeeded. Server stack
trace: Exception rethrown at [0]: at
Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult
result) at
Microsoft.ServiceBus.NamespaceManager.OnEndSubscriptionExists(IAsyncResult
result) at
Microsoft.ServiceBus.NamespaceManager.SubscriptionExists(String
topicPath, String name) ...
When calling QueueClient x.Receive(...)
PROGRESS queue processing failed.
Microsoft.ServiceBus.Messaging.MessagingCommunicationException: Error
during communication with Service Bus. Check the connection
information, then retry. --->
System.ServiceModel.CommunicationObjectFaultedException: Internal
Server Error: The server did not provide a meaningful reply; this
might be caused by a premature session shutdown.
TrackingId:04ba0220-0350-4806-9c65-c2bba9671054, Timestamp:28.05.2015
13:00:55 Server stack trace: Exception rethrown at [0]: at
Microsoft.ServiceBus.Common.ExceptionDispatcher.Throw(Exception
exception) at
Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult
result) at
Microsoft.ServiceBus.Messaging.Sbmp.DuplexRequestBindingElement.DuplexRequestSessionChannel.DuplexCorrelationAsyncResult.End(IAsyncResult
result) at
Microsoft.ServiceBus.Messaging.Sbmp.DuplexRequestBindingElement.DuplexRequestSessionChannel.EndRequest(IAsyncResult
result) at
Microsoft.ServiceBus.Messaging.Channels.ReconnectBindingElement.ReconnectChannelFactory`1.RequestSessionChannel.RequestAsyncResult.b__4(RequestAsyncResult
thisPtr, IAsyncResult r) at
Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.StepCallback(IAsyncResult
result) ….
The exceptions are clearly related to ServiceBus and they are non-deterministic. The functions throwing them, e.g., Send, GetQueue, SubscriptionExists, are called no more than 100-120 times per minute. We changed nothing in the code and increasing the timeout values (even to ridiculously high values, like 10min) did not help. Also, we do not believe it is some network related problem (on our side) since the same error occur when the application is run from different places.
Has anyone else encountered recently these kind of exceptions? Is there a problem on the side of Microsoft or we are missing something?

A few weeks back we had sudden and unexplainable timing issues crop up with our Service Bus app that has been in production for many months. Ours continued to work but every few calls would take 10+ seconds when they are normally 100-200 millisconds. This went on for a couple weeks and I spent most of that trying to figure out what was going on and never did as the problem suddenly vanished.
We did learn that new Service Bus namespaces we created in the same and other data centers for testing while the problem was occurring did not exhibit the same issue. The Service Bus group offered no help and would only say response times are not guaranteed only the SLA is.

I had similar problem when my running code started to generate Timeout exception. Upon research found that firewall blocked the port used for communication. However, port 80 and 443 were still open. So adding following line of code worked for me:
ServiceBusEnvironment.SystemConnectivity.Mode = ConnectivityMode.Https;

Related

How can a slow internet on client side be a reason for server time out exception in asp.net?

I have a .Net application up and running.
We have had a fluctuating connection yesterday. While testing in such scenarios we had received multiple server time out exception emails like below.
Server Time Out
Type : System.Web.HttpException, System.Web, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a
Message : Request timed out.
Source :
Help link :
WebEventCode : 3001
ErrorCode : -2147467259
Data : System.Collections.ListDictionaryInternal
TargetSite :
HResult : -2147467259
Stack Trace : The stack trace is unavailable.
Additional Info:
IMPORTANT: Above exception occurred while doing a ajax post by a button placed with in update panel.
My question here is why would a slow internet on client side raise such server time out exception?
Isn't server timeout exception is related to such cases where server cannot execute the request in underlying time mention in HttpRuntime setting? May be due to some lengthy operation or some long database execution which takes longer than the time mentioned in setting under HttpRuntime.
If server is not able to connect to the client due to clients fluctuating internet, then Client Disconnected exception would be raised which we did yesterday. But I am not able to conclude the reason for this server timeout exception.
I already know that increasing the execution timeout will fix the issue, but I have to provide technical explanation for the reason as to why such exception of Server Timeout raised.
My best guess here is that the ajax request would be doing some continuous communication with server for executing of single request server and would raise timeout exception if it does not receives some required further communication messages due to client's bad internet. I have search over internet for the same to support my guess but in vain.
Also to provide environmental details, there is a load balancer serving the request.
Any help would be highly appreciated.
It is because (as you write) the connection of client to server is slow, so if the server (or client) sending data to this server, connection can´t handle it, so you get timeout error, because the data can´t been transfered in defined time.
You also write, that this is caused by sending Ajax request, so maybe try to increase execusion timeout in web configuration file (web.config):
<httpRuntime executionTimeout = "number(in seconds)"/>
More about executionTimeout here and here about Ajax requests.
Firstly, I think the cause of this error is because of execution time required by your application request connecting with the remote server is exceeding the currently set ASP.NET request execution timeout value. As per the MSDN Exception Document, default value is set to 110 seconds, in that it is remarked like:
The ExecutionTimeout property indicates the maximum number of seconds
a request is allowed to execute before being automatically shut down
by ASP.NET.
So based on error detail with event code 3001 occurs because no response was received during the time-out period for a request. You can use IIS troubleshooting failed request mechanism to figure it out exact issue like any poor performance/deadlocks when making calls from your ASP.NET application.
Secondly, it is not related to user's internet connectivity issue otherwise you get exception with status like connection-closed or keep alive failure. See this article for detail. The browser is going to wait for a 60 minutes(which is very long period of time that server isn't going to answer any request)for server to response.
And at any case when the browser abandons any request, it is going to close the socket and you'll get an error page from the browser. You don't get anything related to sever-end.

Could awaiting network cause client timeouts?

I have a server that is doing work instructed by an Azure queue. It is almost always on very high CPU doing multiple tasks in parallel and some of the tasks use Parallel.ForEach.
During the running of the tasks I write analytic events to another Azure queue by calling CloudQueue.AddMessageAsync with await.
I noticed thousands of these analytic writings that fail with the following error:
WebException: The remote server returned an error: (500) Internal Server Error.
I checked Azure's storage event logs, and I have a nice bunch of PutMessage commands that take 80.000ms end to end, but they only take 1ms for Azure itself. The HTTP status code I get is 500 and Azure describes the reason as client timeout.
What I think is happening is that my code calls the AddMessageAsync and from that point my thread is released and the network driver is sending the request and waiting for a response. When getting a response, the network driver needs a thread to get the response and a task is scheduled to do that and calls my continuation. Because my server is constantly on high load, the task takes a long time to get a thread and by then the Azure server decides this is a client timeout.
The code calling azure:
await cloudQueue.AddMessageAsync(new CloudQueueMessage(aMessageContent));
The exception:
StorageException: The remote server returned an error: (500) Internal Server Error.
Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndExecuteAsync[T](IAsyncResult result):11
Microsoft.WindowsAzure.Storage.Core.Util.AsyncExtensions+<>c__DisplayClass4.<CreateCallbackVoid>b__3(IAsyncResult ar):45
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task):82
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task):41
AzureCommon.Data.AsyncQueueDataContext+<AddMessage>d__d.MoveNext() in c:\BuildAgent\work\14078ab89161833\Azure\AzureCommon\Data\Async\AsyncQueueDataContext.cs:60
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task):82
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task):41
AzureCommon.Storage.AzureEvent+<DispatchAsync>d__1.MoveNext() in c:\BuildAgent\work\14078ab89161833\Azure\AzureCommon\Events\AzureEvent.cs:354
WebException: The remote server returned an error: (500) Internal Server Error.
System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult):41
Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndGetResponse[T](IAsyncResult getResponseResult):44
Am I right about why this is happening? If so, would using a single-threaded synchronization context for this call be better for me?
A row from Azure storage log. You can find details about what each property means here.
<request-start-time> <operation-type> <request-status> <http-status-code> <end-to-end-latency-in-ms> <server-latency-in-ms>
2014-07-29T14:55:20.0794198Z PutMessage ClientTimeoutError 500 86929 1
Thanks.
The error 500 means that the server has received a bad request or it has crashed for various other reasons. I don't believe that it has to do with the high load of your threads. Please consider taking the following actions:
Check the name of the queue you are using. The name needs to be lowercase, starting with a character. This is a common issue that causes error 500 with no enlighting error message from the server.
Set up the retry policy of the Azure Storage SDK client, preferably with an Exponential retry policy.
Make sure you are using the latest Azure Storage SDK, as the underlying protocol has recently changed to a more efficient one.
'Bad Request' is a 400 error, not a 500 error. A 500 Error indicates any kind of server error, so it's perfectly reasonable to get that response, and many client-side libraries will use a 500 error code for similar types of unexpected issues.
Normally a 'client timeout' response would never make it to the client (because it timed out!). The only situation I can think of where a client timeout response could make it to the client would be if the request was more than a single network packet and the client was too slow in sending packets after the first one. This could easily be caused by CPU contention on the client device. I would recommend using a higher priority thread for listening to network responses but then immediately pass off the processing of the response to a normal priority thread. Overloaded CPU will cause all sorts of timeout issues because the code can't tell the difference between a network response not coming in soon enough and the CPU not scheduling the listener in time to receive the response (or even to send the request). Even local disk I/O and locking can timeout in these situations, depending on the underlying implementation.

ServiceActivationException: Request cannot be dispatched because the virtual application is shutting down

We are having a problem with our virtual application shutting down. Site is running ASP.NET 4.5
Exception summary:
System.InvalidOperationException: Request to the service at '~/Services/ServiceExceptionLogger.svc' cannot be dispatched because the virtual application at '/Site' is shutting down.
Stack trace:
System.ServiceModel.ServiceActivationException: Request to the service at
'~/Services/ServiceExceptionLogger.svc' cannot be dispatched because the virtual application
at '/Site' is shutting down. ---> System.InvalidOperationException: Request to the service at '~/Services/ServiceExceptionLogger.svc' cannot be dispatched because the virtual
application at '/Site' is shutting down.
--- End of inner exception stack trace ---
at System.Runtime.AsyncResult.End[TAsyncResult](IAsyncResult result)
at System.ServiceModel.Activation.HostedHttpRequestAsyncResult.End(IAsyncResult result)
at System.Web.HttpApplication.CallHandlerExecutionStep.OnAsyncHandlerCompletion(IAsyncResult ar)
Or here is another version of it that shows up in the event logs
Stack Trace:
System.ServiceModel.ServiceActivationException: Request to the service at '~/Services/Service.svc' cannot be dispatched because the virtual application at '/Site' is shutting down. ---> System.InvalidOperationException: Request to the service at '~/Services/Service.svc' cannot be dispatched because the virtual application at '/Site' is shutting down.
--- End of inner exception stack trace ---
at System.Runtime.AsyncResult.End[TAsyncResult](IAsyncResult result)
at System.ServiceModel.Activation.HostedHttpRequestAsyncResult.End(IAsyncResult result)
at System.Web.HttpApplication.CallHandlerExecutionStep.OnAsyncHandlerCompletion(IAsyncResult ar)
Anyone have any ideas or seen this before? Seems to happen quite frequently on one of our environments, like 10+ times a day. On the QA environment its not happening at all. Idle timeout is set to 20minutes on both environments. Reason for 20 minute timeout is to avoid wasted memory
Update 1: Found some supplementary information on logging app pool recycles here https://webmasters.stackexchange.com/questions/17630/which-event-log-file-does-iis-7-app-pool-log-to/17633#17633?newreg=d562bf378cc545b49a7ea8f2a3c1b48d
Update 2: Apparently the app pool is not recycling when the above exception occurs. We are watching the process in task manager and it is not shutting down or reducing the memory when this exception occurs. Also, no log entry occurs (see update 1) when this exception happens
After escalating this to Microsoft support, we think this issue is a bug .NET 4.5.1. It happens most often when running in integrated mode, but also occurs in classic mode, although much less frequently. We haven't been able to pinpoint the actual case or problem but it doesn't seem to be occuring on older .NET runtimes.
Assuming this is what you are experiencing:
In IIS, Application Pools recycle themselves every ~20 minutes (I don't remember the exact default time) after inactivity.
This is changeable though in the IIS Manager under the "Advanced Settings" for your App Pool.

Thread is being aborted

What exactly does all this mean below. I am running an asynchronous web request that calls a page that sends code to my database. Then my database fires off a stored procedure that runs into one of my SQL assemblies. Most times it works, but sometimes I get this:
Thread was being aborted.
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.
Exception Details: System.Threading.ThreadAbortException: Thread was being aborted.
Source Error:
An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.
Stack Trace:
[ThreadAbortException: Thread was being aborted.]
System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +486
System.Web.ApplicationStepManager.ResumeSteps(Exception error) +501
System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProcessRequest(HttpContext context, AsyncCallback cb, Object extraData) +123
System.Web.HttpRuntime.ProcessRequestInternal(HttpWorkerRequest wr) +379
The reason why you are getting this error most likely is that you are getting timed out. Increase the ExecutionTimeout since at default its 110 seconds.
<compilation debug="false"></compilation>
<httpRuntime executionTimeout="1800"/> //1800 seconds / 30 minutes.
Here's an article that goes more into detail as to how to increase the ExecutionTimeout.
The app domain could be getting recycled which can happen for a number of reasons. That can cause a thread abort exception.
Here is an article which explains some of the reasons for a recycle
If a process is long running, IIS/ASP.Net is not usually a good place to run it.

Thread being aborded error

We have an application that can create e-books.
This application has an export module that creates an AIR file but this can take a while (some books have 2500 pages).
If we export we get the following error:
Thread was being aborted.
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.
Exception Details: System.Threading.ThreadAbortException: Thread was being aborted.
Source Error:
An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.
Stack Trace:
[ThreadAbortException: Thread was being aborted.]
System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +501
System.Web.ApplicationStepManager.ResumeSteps(Exception error) +564
System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProcessRequest(HttpContext context, AsyncCallback cb, Object extraData) +141
System.Web.HttpRuntime.ProcessRequestInternal(HttpWorkerRequest wr) +436
I've changed my runtime executiontimeout to 3600 secs but it keeps crashing arround 3 minutes.
so it is time related ... everytime we approach the 3 minutes it crashes, I hope someone can help me out.
I think Paul is right about the cause of the exception. Both IIS and ASP.NET have settings that limit the maximum amount of time a request can take. For ASP.NET it's in the Machine.Config file (look for the httpRuntime element, executionTimeout attribute). It's set to 90 seconds on my development machine.
I would however not advise you to change that setting as it's there to make sure your application doesn't hang on a bad request.
Long running tasks should use asynchronous execution. With async execution, the actual work is handled on a separate thread. This frees the thread that handles the request to handle other requests which is good for the overall performance of your application.
There are some good articles on this available. For example : http://msdn.microsoft.com/en-us/magazine/cc163725.aspx
Quite often, this error actually occurs from an OutOfMemory exception.
Is there an InnerException available?
IIS has a 'run-away' thread protection that will kill a thread/appdomain if it runs for too long.

Categories

Resources