We have a web application which uses Microsoft.Web.Redis.RedisSessionStateProvider as the session state provider. We've had no problems with this until recently when suddenly the application is reporting a large number of exceptions as detailed below.
The message suggests some kind of timeout. But I've no idea how to resolve this. The application is an instance of Kentico CMS and we haven't customised its session caching mechanism in any way.
Main exception:
Message:
Exception of type 'System.Web.HttpException' was thrown.
Stack Trace:
at System.Web.HttpAsyncResult.End()
at System.Web.HttpApplication.AsyncEventExecutionStep.OnAsyncEventCompletion(IAsyncResult ar)
Inner exception:
Message:
Timeout performing EVAL, inst: 1, mgr: Inactive, err: never, queue: 0, qu: 0, qs: 0, qc: 0, wr: 0, wq: 0, in: 0, ar: 0, IOCP: (Busy=0,Free=1000,Min=2,Max=1000), WORKER: (Busy=3,Free=32764,Min=2,Max=32767), clientName: KSCOMAZUATWEB2
Stack Trace:
at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor`1 processor, ServerEndPoint server) in c:\TeamCity\buildAgent\work\3ae0647004edff78\StackExchange.Redis\StackExchange\Redis\ConnectionMultiplexer.cs:line 1927
at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor`1 processor, ServerEndPoint server) in c:\TeamCity\buildAgent\work\3ae0647004edff78\StackExchange.Redis\StackExchange\Redis\RedisBase.cs:line 80
at StackExchange.Redis.RedisDatabase.ScriptEvaluate(String script, RedisKey[] keys, RedisValue[] values, CommandFlags flags) in c:\TeamCity\buildAgent\work\3ae0647004edff78\StackExchange.Redis\StackExchange\Redis\RedisDatabase.cs:line 866
at Microsoft.Web.Redis.StackExchangeClientConnection.<>c__DisplayClass4.<Eval>b__3() in d:\TeamCityBuildAgent\work\f55792526e6d9089\src\Shared\StackExchangeClientConnection.cs:line 113
at Microsoft.Web.Redis.StackExchangeClientConnection.RetryForScriptNotFound(Func`1 redisOperation) in d:\TeamCityBuildAgent\work\f55792526e6d9089\src\Shared\StackExchangeClientConnection.cs:line 129
at Microsoft.Web.Redis.StackExchangeClientConnection.RetryLogic(Func`1 redisOperation) in d:\TeamCityBuildAgent\work\f55792526e6d9089\src\Shared\StackExchangeClientConnection.cs:line 155
at Microsoft.Web.Redis.StackExchangeClientConnection.Eval(String script, String[] keyArgs, Object[] valueArgs) in d:\TeamCityBuildAgent\work\f55792526e6d9089\src\Shared\StackExchangeClientConnection.cs:line 113
at Microsoft.Web.Redis.RedisConnectionWrapper.TryTakeWriteLockAndGetData(DateTime lockTime, Int32 lockTimeout, Object& lockId, ISessionStateItemCollection& data, Int32& sessionTimeout) in d:\TeamCityBuildAgent\work\f55792526e6d9089\src\RedisSessionStateProvider\RedisConnectionWrapper.cs:line 182
at Microsoft.Web.Redis.RedisSessionStateProvider.GetItemFromSessionStore(Boolean isWriteLockRequired, HttpContext context, String id, Boolean& locked, TimeSpan& lockAge, Object& lockId, SessionStateActions& actions) in d:\TeamCityBuildAgent\work\f55792526e6d9089\src\RedisSessionStateProvider\RedisSessionStateProvider.cs:line 272
at Microsoft.Web.Redis.RedisSessionStateProvider.GetItemExclusive(HttpContext context, String id, Boolean& locked, TimeSpan& lockAge, Object& lockId, SessionStateActions& actions) in d:\TeamCityBuildAgent\work\f55792526e6d9089\src\RedisSessionStateProvider\RedisSessionStateProvider.cs:line 190
at System.Web.SessionState.SessionStateModule.GetSessionStateItem()
at System.Web.SessionState.SessionStateModule.PollLockedSessionCallback(Object state)
After experiencing the same issue and having read various posts on setting min & max thread values etc, it transpired that the issue was with the subscription tier.
C1 basic (£35/GBP mth) boasts a 1gb cache, 1000 concurrent connections which are exactly the same headline figures as the C1 Standard tier (£95GBP/Mth).
It's a natural assumption that when you only require a few MB of cache and have only 50 concurrent sessions active at any point, that the C! basic would be adequate, if not overkill.
The difference seems to be that C1 Basic works, if you have 10 or less concurrent users on your website.
In fairness, it is mentioned in the fine print that the basic is only suitable for development and testing environments.
I was just working through this on a client site and what helped me was to add this to the connection string of your provider:
throwOnError="true" retryTimeoutInMilliseconds="500"
What I learned is sometimes there is a latency issue and if it doesn't resolve fast enough the default is to just "quit". So adding this allows it to continue to retry.
Your string might look like this
<add name="MySessionStateStore" throwOnError="true" retryTimeoutInMilliseconds="500" type="Microsoft.Web.Redis.RedisSessionStateProvider" host="mysite-prod.redis.cache.windows.net" accessKey="mykey" ssl="true" />
Related
we are using Elastic cache redis that aws provides in our .Net Core application and when we perform the below operation sometimes we are getting the below TimeoutException after sometime can someone please help me to solve this attaching exception for reference
_redis.KeyExists("cusID"+customerID) -> operation performed
Exception has occurred: CLR/StackExchange.Redis.RedisTimeoutException
Exception thrown: 'StackExchange.Redis.RedisTimeoutException' in StackExchange.Redis.dll: 'Timeout performing EXISTS (5000ms), next: EXISTS cusID1, inst: 0, qu: 0, qs: 0, aw: False, bw: SpinningDown, rs: ReadAsync, ws: Idle, in: 0, serverEndpoint: master.test-stage-test-new-jayaram3.mvg9d6.aps1.cache.amazonaws.com:6379, mc: 1/1/0, mgr: 10 of 10 available, clientName: JAYA-397(SE.Redis-v2.5.61.22961), IOCP: (Busy=0,Free=10000,Min=10000,Max=10000), WORKER: (Busy=2,Free=32765,Min=10000,Max=32767), POOL: (Threads=13,QueuedItems=0,CompletedItems=790), v: 2.5.61.22961 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)'
at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 1857 at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in //src/StackExchange.Redis/RedisBase.cs:line 53
at StackExchange.Redis.RedisDatabase.KeyExists(RedisKey key, CommandFlags flags) in //src/StackExchange.Redis/RedisDatabase.cs:line 667
An exception of type 'StackExchange.Redis.RedisTimeoutException' occurred in StackExchange.Redis.dll but was not handled in user code: 'Timeout performing GET (5000ms), active: GET cusID1, next: GET cusID1 ** possible thread-theft indicated; see https://stackexchange.github.io/StackExchange.Redis/ThreadTheft ** , inst: 1, qu: 0, qs: 0, aw: False, bw: SpinningDown, rs: CompletePendingMessageSync, ws: Idle, in: 0, serverEndpoint: master.suremdm-stage-test-new-jayaram3.mvg9d5.aps1.cache.amazonaws.com:6379, mc: 1/1/0, mgr: 8 of 10 available, clientName: JAYA-397(SE.Redis-v2.5.61.22961), IOCP: (Busy=3,Free=9997,Min=10000,Max=10000), WORKER: (Busy=5,Free=32762,Min=10000,Max=32767), POOL: (Threads=12,QueuedItems=0,CompletedItems=2634), v: 2.5.61.22961 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)'
If need some more information, just ask me that I'll try to provide. Thanks in advance.
Are you setting "AbortOnConnectFail" to false so it continues to try to reconnect? When a connection breaks (could be due to a failover etc), commands already sent to Redis on the now broken connection will timeout on the client.
Event Viewer on my workstation have the following error log:
ERROR NServiceBus.Transports.Msmq.MsmqDequeueStrategy [(null)] - Error in receiving messages.
System.Transactions.TransactionAbortedException: The transaction has aborted. ---> System.Transactions.TransactionManagerCommunicationException: Communication with the underlying transaction manager has failed. ---> System.Runtime.InteropServices.COMException: The Transaction Manager is not available. (Exception from HRESULT: 0x8004D01B)
at System.Transactions.Oletx.IDtcProxyShimFactory.ConnectToProxy(String nodeName, Guid resourceManagerIdentifier, IntPtr managedIdentifier, Boolean& nodeNameMatches, UInt32& whereaboutsSize, CoTaskMemHandle& whereaboutsBuffer, IResourceManagerShim& resourceManagerShim)
at System.Transactions.Oletx.DtcTransactionManager.Initialize()
--- End of inner exception stack trace ---
at System.Transactions.Oletx.OletxTransactionManager.ProxyException(COMException comException)
at System.Transactions.Oletx.DtcTransactionManager.Initialize()
at System.Transactions.Oletx.DtcTransactionManager.get_ProxyShimFactory()
at System.Transactions.Oletx.OletxTransactionManager.CreateTransaction(TransactionOptions properties)
at System.Transactions.TransactionStatePromoted.EnterState(InternalTransaction tx)
--- End of inner exception stack trace ---
at System.Transactions.TransactionStateAborted.CheckForFinishedTransaction(InternalTransaction tx)
at System.Transactions.Transaction.Promote()
at System.Transactions.TransactionInterop.ConvertToOletxTransaction(Transaction transaction)
at System.Transactions.TransactionInterop.GetDtcTransaction(Transaction transaction)
at System.Messaging.MessageQueue.StaleSafeReceiveMessage(UInt32 timeout, Int32 action, MQPROPS properties, NativeOverlapped* overlapped, ReceiveCallback receiveCallback, CursorHandle cursorHandle, IntPtr transaction)
at System.Messaging.MessageQueue.ReceiveCurrent(TimeSpan timeout, Int32 action, CursorHandle cursor, MessagePropertyFilter filter, MessageQueueTransaction internalTransaction, MessageQueueTransactionType transactionType)
at System.Messaging.MessageQueue.Receive(TimeSpan timeout, MessageQueueTransactionType transactionType)
at NServiceBus.Transports.Msmq.MsmqDequeueStrategy.TryReceiveMessage(Func`1 receive, Message& message) in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Transports\Msmq\MsmqDequeueStrategy.cs:line 332
Facts:
Distributed Transaction Coordinator (DTC) Component Service ==> DTC Enabled
Distributed Transaction Coordinator (DTC) Service ==> Running (referenced to Answered Stack Oveflow Question)
Realtek Audio Universal Service ==> Disabled (as per reference in Fix for 0x8004d01b)
Can anyone recommend a direction on how to check/troubleshoot the cause of the issue?
After long ardous hours, the issue I encountered boils down to permission issue. The services involved in DTC which uses credentials of LocalSystem do not have enough permission and blocked by company-imposed policy implemented on the server.
Unfortunately, I only managed to find an alternative or workaround by using account that have enough permissions to operate and manage the services involved (including custom services).
#gnud - Thanks a lot for your inputs giving me a direction for investigation and that help me continue move forward on my investigation. I appreciate the help. Thanks.
I am using StackExchange.Redis in my application to store key/values.
The issue I am getting is that while fetching/checking the cache for a key I am getting TimeOut Exception:
Here is the detailed exception:
System.TimeoutException was unhandled HResult=-2146233083
Message=Timeout performing EXISTS GlobalSettings, inst: 0, mgr:
ExecuteSelect, err: never, queue: 56, qu: 56, qs: 0, qc: 0, wr: 0, wq:
1, in: 0, ar: 0, IOCP: (Busy=0,Free=1000,Min=2,Max=1000), WORKER:
(Busy=20,Free=32747,Min=2,Max=32767), clientName: WIN-VDIGHSLJUBV
Source=StackExchange.Redis.StrongName StackTrace:
at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message
message, ResultProcessor1 processor, ServerEndPoint server)
at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor1 processor, ServerEndPoint server)
at StackExchange.Redis.RedisDatabase.KeyExists(RedisKey key, CommandFlags flags)
I have the following code:
IDatabase db = GetDatabase();
return db.KeyExists(cacheKey);
and the exception I am getting is at line "db.KeyExists(cacheKey)".
Based on the issue I thought increasing timeout might help and therefore I did this in initialization:
redis = ConnectionMultiplexer.Connect(string.Format("{0},allowAdmin=true,syncTimeout=30000,connectTimeout=30000", redisConfig));
i.e I added the "syncTimeout=30000,connectTimeout=30000"
but that didn't help either.
Do I need to add timeouts to some other locations?
Do I need to look the fix for this issue at some other point of Redis logic?
I just read through the following stack overflow post, and I think it might provide another piece of the puzzle: StackExchange.Redis.RedisTimeoutException: Timeout awaiting response
MÇT points out that one might need to increase the minimum thread count in the application's ThreadPool when, as I saw in your exception message,
WORKER: (Busy=20,Free=32747,Min=2,Max=32767)
The number of Busy worker threads is greater than the Min value, which represents the number of worker threads initially created.
Please take a look and his answer for the specific details. This posting is a simple attempt to direct an unanswered question to actionable information.
I am running Workflow 4 activities, and using SQL Durable Instancing to persist the Workflow when it's suspended. I am intermittently getting an InstanceOwnerException, which is remedied by recreating the instance store. Is there any way of avoiding this exception in the first place?
System.OperationCanceledException: An error processing the current work item has caused the workflow to abort. See the inner exception for details. ---> System.Runtime.DurableInstancing.InstanceOwnerException: The execution of an InstancePersistenceCommand was interrupted because the instance owner registration for owner ID '55413a1f-0e3f-4a0f-b2b0-54c121aa0c52' has become invalid. This error indicates that the in-memory copy of all instances locked by this owner have become stale and should be discarded, along with the InstanceHandles. Typically, this error is best handled by restarting the host.
at System.Runtime.AsyncResult.End[TAsyncResult](IAsyncResult result)
at System.Runtime.DurableInstancing.InstanceStore.EndExecute(IAsyncResult result)
at System.Activities.WorkflowApplication.UnloadOrPersistAsyncResult.OnPersisted(IAsyncResult result)
at System.Runtime.AsyncResult.SyncContinue(IAsyncResult result)
at System.Activities.WorkflowApplication.UnloadOrPersistAsyncResult.Persist()
at System.Activities.WorkflowApplication.UnloadOrPersistAsyncResult.CollectAndMap()
at System.Activities.WorkflowApplication.UnloadOrPersistAsyncResult.Track()
at System.Activities.WorkflowApplication.UnloadOrPersistAsyncResult.InitializeProvider()
at System.Activities.WorkflowApplication.UnloadOrPersistAsyncResult..ctor(WorkflowApplication instance, TimeSpan timeout, PersistenceOperation operation, Boolean isWorkflowThread, Boolean isInternalPersist, AsyncCallback callback, Object state)
at System.Activities.WorkflowApplication.BeginInternalPersist(PersistenceOperation operation, TimeSpan timeout, Boolean isInternalPersist, AsyncCallback callback, Object state)
at System.Activities.WorkflowApplication.OnBeginPersist(AsyncCallback callback, Object state)
at System.Activities.Runtime.ActivityExecutor.PersistenceWaiter.PersistWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)
--- End of inner exception stack trace ---
The code is running on Azure, using SQL Azure to persist the Workflows. There are roles that are accessing the same instance store.
This actually turned out to be down to attempting to resume a workflow after it had already finished.
I've just spend 4 hours (it's 3am in the UK) trying to debug ASP.NET application which caused an exception in a thread managed by Framework (i.e. not my thread). I've just found out that a result from the static method ChannelFactory.CreateChannel can be cast to IClientChannel and explicitly Disposed. I mean that's all fine and nice but why:
1) ChannelFactory.CreateChannel does not return IClientChannel as an out parameter?
2) .Net documentation for CreateChannel does not mention it?
3) .Net documentation does not show a proper usage pattern in examples (no dispose code)?
Don't get me wrong - I love .Net framework. Microsoft (and Krzysztof Cwalina: see Designing Framework Guidelines) has done a really great job. That's why I didn't expect a such disaster. I mean how the hell I should know that my IMyService variable also supports IClientChannel and I should explicitly dispose it?
Here is an ASP.NET log if someone is interested.
Event Type: Error
Event Source: ASP.NET 2.0.50727.0
Event Category: None
Event ID: 1334
Date: 12/08/2009
Time: 01:55:47
User: N/A
Computer: WLGH3GIS
Description:
An unhandled exception occurred and the process was terminated.
Application ID: /LM/W3SVC/1/Root/Maps
Process ID: 3044
Exception: System.NullReferenceException
Message: Object reference not set to an instance of an object.
StackTrace: at System.Threading.Overlapped.Free(NativeOverlapped* nativeOverlappedPtr)
at System.ServiceModel.Channels.OverlappedContext.Free()
at System.ServiceModel.Channels.PipeConnection.CloseHandle(Boolean abort, String timeoutErrorString, TransferOperation transferOperation)
at System.ServiceModel.Channels.PipeConnection.Close(TimeSpan timeout)
at System.ServiceModel.Channels.BufferedConnection.Close(TimeSpan timeout)
at System.ServiceModel.Channels.ConnectionPool.CloseItem(IConnection item, TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationPool`2.EndpointConnectionPool.CloseItem(TItem item, TimeSpan timeout)
at System.ServiceModel.Channels.IdlingCommunicationPool`2.IdleTimeoutEndpointConnectionPool.CloseItem(TItem item, TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationPool`2.EndpointConnectionPool.CloseIdleConnection(TItem connection, TimeSpan timeout)
at System.ServiceModel.Channels.IdlingCommunicationPool`2.IdleTimeoutEndpointConnectionPool.IdleTimeoutIdleConnectionPool.OnIdle()
at System.ServiceModel.Channels.IdlingCommunicationPool`2.IdleTimeoutEndpointConnectionPool.IdleTimeoutIdleConnectionPool.OnIdle(Object state)
at System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.WorkItem.Invoke2()
at System.Security.SecurityContext.Run(SecurityContext securityContext, ContextCallback callback, Object state)
at System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.WorkItem.Invoke()
at System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.ProcessCallbacks()
at System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.CompletionCallback(Object state)
at System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.ScheduledOverlapped.IOCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
at System.ServiceModel.Diagnostics.Utility.IOCompletionThunk.UnhandledExceptionFrame(UInt32 error, UInt32 bytesRead, NativeOverlapped* nativeOverlapped)
at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)
Karol, I know this is a bit old, but I wanted to say thank you. Your post really helped me out.
The symptom I had was that when I tried to Close a ChannelFactory on the Server, it would always give a Timeout, regardless of the length of timespan I set for OpenTimeout, ReceiveTimeout, SendTimeout, InactivityTimeout, or CloseTimeout.
The solution was actually on the client, to cast the IMyServiceInterface that was returned by ChannelFactory.CreateChannel to ICommunicationObject. Then, it can be passed nicely to the Util.SafeCloseAndDispose(ICommunicationObject) method that you see copy and pasted all over the web.
After I did that on my client, then the Server's ChannelFactory can be closed in only a second or two, with no more timeout.
As far as I know, this insight from Karol's post is one of the only places online where this problem is spelled out.
Thanks again, Karol! :)