we are using Elastic cache redis that aws provides in our .Net Core application and when we perform the below operation sometimes we are getting the below TimeoutException after sometime can someone please help me to solve this attaching exception for reference
_redis.KeyExists("cusID"+customerID) -> operation performed
Exception has occurred: CLR/StackExchange.Redis.RedisTimeoutException
Exception thrown: 'StackExchange.Redis.RedisTimeoutException' in StackExchange.Redis.dll: 'Timeout performing EXISTS (5000ms), next: EXISTS cusID1, inst: 0, qu: 0, qs: 0, aw: False, bw: SpinningDown, rs: ReadAsync, ws: Idle, in: 0, serverEndpoint: master.test-stage-test-new-jayaram3.mvg9d6.aps1.cache.amazonaws.com:6379, mc: 1/1/0, mgr: 10 of 10 available, clientName: JAYA-397(SE.Redis-v2.5.61.22961), IOCP: (Busy=0,Free=10000,Min=10000,Max=10000), WORKER: (Busy=2,Free=32765,Min=10000,Max=32767), POOL: (Threads=13,QueuedItems=0,CompletedItems=790), v: 2.5.61.22961 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)'
at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 1857 at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in //src/StackExchange.Redis/RedisBase.cs:line 53
at StackExchange.Redis.RedisDatabase.KeyExists(RedisKey key, CommandFlags flags) in //src/StackExchange.Redis/RedisDatabase.cs:line 667
An exception of type 'StackExchange.Redis.RedisTimeoutException' occurred in StackExchange.Redis.dll but was not handled in user code: 'Timeout performing GET (5000ms), active: GET cusID1, next: GET cusID1 ** possible thread-theft indicated; see https://stackexchange.github.io/StackExchange.Redis/ThreadTheft ** , inst: 1, qu: 0, qs: 0, aw: False, bw: SpinningDown, rs: CompletePendingMessageSync, ws: Idle, in: 0, serverEndpoint: master.suremdm-stage-test-new-jayaram3.mvg9d5.aps1.cache.amazonaws.com:6379, mc: 1/1/0, mgr: 8 of 10 available, clientName: JAYA-397(SE.Redis-v2.5.61.22961), IOCP: (Busy=3,Free=9997,Min=10000,Max=10000), WORKER: (Busy=5,Free=32762,Min=10000,Max=32767), POOL: (Threads=12,QueuedItems=0,CompletedItems=2634), v: 2.5.61.22961 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)'
If need some more information, just ask me that I'll try to provide. Thanks in advance.
Are you setting "AbortOnConnectFail" to false so it continues to try to reconnect? When a connection breaks (could be due to a failover etc), commands already sent to Redis on the now broken connection will timeout on the client.
Related
I'm using Microsoft.Web.Redis.RedisSessionStateProvider
our network team provide me
1- Hostname host_ip
2- Ports
a. Read 6379
b. Write 6380
and my connection string
connectionString="host_ip:6379,host_ip:6380,user=*****,password=****,ConnectTimeout=50000,connectRetry=5,syncTimeout=50000,abortConnect=false"
Timeout performing EVAL (30000ms), inst: 0, qu: 0, qs: 0, aw: False, bw: Inactive, rs: ReadAsync, ws: Idle, in: 0, in-pipe: 0, out-pipe: 0, last-in: 4, cur-in: 0, sync-ops: 1, async-ops: 3, serverEndpoint: 192.168.33.107:6380, conn-sec: 52.08, mc: 1/1/0, mgr: 10 of 10 available, clientName: *******(SE.Redis-v2.6.90.64945), PerfCounterHelperkeyHashSlot: 9255, IOCP: (Busy=0,Free=1000,Min=8,Max=1000), WORKER: (Busy=1,Free=32766,Min=8,Max=32767), v: 2.6.90.64945 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)
serverEndpoint: 192.168.33.107:6380 this port for write
rs: ReadAsync operation is reading
I have a problem with StackExchange.Redis. In our process we have two redis databases. These databases are used to store similar information which the user can execute lookup onto, so we are switching between them on imports, so the user can continuosly access the information. Everyday around 50000000 records are inserted to one database and old records are deleted from the other, to prepare it for next day's import. The records are imported in batches of 100000 records. The problem we are facing is that after the data import has completed, Redis throws an exception, while trying to execute flushdb command.
StackExchange.Redis.RedisTimeoutException Timeout awaiting response (outbound=0KiB, inbound=0KiB, 5976ms elapsed, timeout is 5000ms), command=UNKNOWN, next: SELECT, inst: 0, qu: 0, qs: 2, aw: False, rs: ReadAsync, ws: Idle, in: 0, in-pipe: 0, out-pipe: 0, serverEndpoint: 127.0.0.1:6379, mgr: 10 of 10 available, clientName: MX-ENUM-VM, IOCP: (Busy=0,Free=1000,Min=2,Max=1000), WORKER: (Busy=3,Free=32764,Min=2,Max=32767), v: 2.0.601.3402 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts) at Project.Synchronizer.Service.RedisHelper.FlushDb(Int32 dbIndex)
at Project.Synchronizer.Service.RedisHelper.SelectAndClearRedisDbForNumberUpload()
at Project.Synchronizer.Service.RedisHelper.UploadNumbers()
at Project.Synchronizer.Service.DataUploadService.UploadNumbersToRedis() StackExchange.Redis.RedisTimeoutException: Timeout awaiting response (outbound=0KiB, inbound=0KiB, 5976ms elapsed, timeout is 5000ms), command=UNKNOWN, next: SELECT, inst: 0, qu: 0, qs: 2, aw: False, rs: ReadAsync, ws: Idle, in: 0, in-pipe: 0, out-pipe: 0, serverEndpoint: 127.0.0.1:6379, mgr: 10 of 10 available, clientName: CLIENT-ENUM-VM, IOCP: (Busy=0,Free=1000,Min=2,Max=1000), WORKER: (Busy=3,Free=32764,Min=2,Max=32767), v: 2.0.601.3402 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)
at Project.Synchronizer.Service.RedisHelper.FlushDb(Int32 dbIndex)
at Project.Synchronizer.Service.RedisHelper.SelectAndClearRedisDbForNumberUpload()
at Project.Synchronizer.Service.RedisHelper.UploadNumbers()
at Project.Synchronizer.Service.DataUploadService.UploadNumbersToRedis()
The way I try to execute this command is
private async Task FlushDb(int dbIndex)
{
await getDB(dbIndex).ExecuteAsync("flushdb");
}
Any kind of help would be greatly appreciated, because I am stuck on this for quite a while now. For the record, I was trying to get some answers here, but couldn't figure anything out https://stackexchange.github.io/StackExchange.Redis/Timeouts.html
Well, It seems that I somehow managed to solve the problem, but haven't delved too deep to what causes it. Long story short increasing syncTimeout seemed to help and the exception is no longer thrown. This thread was helpful in my case stackexchange.redis throws timeout even after increasing timeout?
I'm using version 1.2.6 of the package and often see the following exceptions in logs while connecting to Azure RedisCaches. Any ideas what may cause such errors?
StackExchange.Redis.RedisTimeoutException: Timeout performing EXISTS controllers.usercontroller-getusersscroll-userIds=&userIds=&skip=0&take=2&status=Active, Inactive&orderBy=0&sortDirection=Desc:application/json; charset=utf-8:Bearer , inst: 0, mgr: Inactive, err: never, queue: 1769, qu: 1769, qs: 0, qc: 0, wr: 0, wq: 0, in: 0, ar: 0, clientName: <...>, serverEndpoint: Unspecified/<...>.redis.cache.windows.net:6380, keyHashSlot: 2692, IOCP: (Busy=0,Free=1000,Min=250,Max=1000), WORKER: (Busy=1,Free=32766,Min=250,Max=32767)
I'm implementing a cache using the tagging mechanism in CachingFramework.Redis.
My app can read and write to Redis successfully, and I can create tags in Redis (which I can see in Desktop Manager). However I get a timeout exception only when I attempt to read from the tag store in Redis:
System.TimeoutException: 'Timeout performing TYPE :$tag$:_HB, inst: 63, mgr: ExecuteSelect, err: never, queue: 1, qu: 1, qs: 0, qc: 0, wr: 0, wq: 1, in: 0, ar: 0, clientName: LAPTOP-GLFEVSO4, serverEndpoint: 127.0.0.1:6379, keyHashSlot: 14024, IOCP: (Busy=0,Free=1000,Min=200,Max=1000), WORKER: (Busy=3,Free=997,Min=200,Max=1000), Local-CPU: 100% (Please take a look at this article for some common client-side issues that can cause timeouts: https://github.com/StackExchange/StackExchange.Redis/tree/master/Docs/Timeouts.md)'
The calling code is like this:
var tags = new List<string>();
if (conditionA == true) { tags.Add("_HB"); }
if (conditionB == false) { tags.Add("_NHB");}
var foundKeys = Redis.Cache.GetKeysByTag(tags.ToArray()); //the timeout happens here
I've checked that memory usage and CPU are not an issue, and I've increased MinIOThreads in machine.config. I'm out of ideas... is this a bug in CachingFramework.Redis (or the underlying StackExchange.Redis), or am I doing something wrong?
[* I'm using VS2017/IIS Express, in debug mode, against a local Redis standalone instance on my development machine]
UPDATE
The issue only occurs in debug mode in Visual Studio. At runtime, it seems fine. However, it would be nice to be able to debug.
I am using StackExchange.Redis in my application to store key/values.
The issue I am getting is that while fetching/checking the cache for a key I am getting TimeOut Exception:
Here is the detailed exception:
System.TimeoutException was unhandled HResult=-2146233083
Message=Timeout performing EXISTS GlobalSettings, inst: 0, mgr:
ExecuteSelect, err: never, queue: 56, qu: 56, qs: 0, qc: 0, wr: 0, wq:
1, in: 0, ar: 0, IOCP: (Busy=0,Free=1000,Min=2,Max=1000), WORKER:
(Busy=20,Free=32747,Min=2,Max=32767), clientName: WIN-VDIGHSLJUBV
Source=StackExchange.Redis.StrongName StackTrace:
at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message
message, ResultProcessor1 processor, ServerEndPoint server)
at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor1 processor, ServerEndPoint server)
at StackExchange.Redis.RedisDatabase.KeyExists(RedisKey key, CommandFlags flags)
I have the following code:
IDatabase db = GetDatabase();
return db.KeyExists(cacheKey);
and the exception I am getting is at line "db.KeyExists(cacheKey)".
Based on the issue I thought increasing timeout might help and therefore I did this in initialization:
redis = ConnectionMultiplexer.Connect(string.Format("{0},allowAdmin=true,syncTimeout=30000,connectTimeout=30000", redisConfig));
i.e I added the "syncTimeout=30000,connectTimeout=30000"
but that didn't help either.
Do I need to add timeouts to some other locations?
Do I need to look the fix for this issue at some other point of Redis logic?
I just read through the following stack overflow post, and I think it might provide another piece of the puzzle: StackExchange.Redis.RedisTimeoutException: Timeout awaiting response
MÇT points out that one might need to increase the minimum thread count in the application's ThreadPool when, as I saw in your exception message,
WORKER: (Busy=20,Free=32747,Min=2,Max=32767)
The number of Busy worker threads is greater than the Min value, which represents the number of worker threads initially created.
Please take a look and his answer for the specific details. This posting is a simple attempt to direct an unanswered question to actionable information.