I have a problem with StackExchange.Redis. In our process we have two redis databases. These databases are used to store similar information which the user can execute lookup onto, so we are switching between them on imports, so the user can continuosly access the information. Everyday around 50000000 records are inserted to one database and old records are deleted from the other, to prepare it for next day's import. The records are imported in batches of 100000 records. The problem we are facing is that after the data import has completed, Redis throws an exception, while trying to execute flushdb command.
StackExchange.Redis.RedisTimeoutException Timeout awaiting response (outbound=0KiB, inbound=0KiB, 5976ms elapsed, timeout is 5000ms), command=UNKNOWN, next: SELECT, inst: 0, qu: 0, qs: 2, aw: False, rs: ReadAsync, ws: Idle, in: 0, in-pipe: 0, out-pipe: 0, serverEndpoint: 127.0.0.1:6379, mgr: 10 of 10 available, clientName: MX-ENUM-VM, IOCP: (Busy=0,Free=1000,Min=2,Max=1000), WORKER: (Busy=3,Free=32764,Min=2,Max=32767), v: 2.0.601.3402 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts) at Project.Synchronizer.Service.RedisHelper.FlushDb(Int32 dbIndex)
at Project.Synchronizer.Service.RedisHelper.SelectAndClearRedisDbForNumberUpload()
at Project.Synchronizer.Service.RedisHelper.UploadNumbers()
at Project.Synchronizer.Service.DataUploadService.UploadNumbersToRedis() StackExchange.Redis.RedisTimeoutException: Timeout awaiting response (outbound=0KiB, inbound=0KiB, 5976ms elapsed, timeout is 5000ms), command=UNKNOWN, next: SELECT, inst: 0, qu: 0, qs: 2, aw: False, rs: ReadAsync, ws: Idle, in: 0, in-pipe: 0, out-pipe: 0, serverEndpoint: 127.0.0.1:6379, mgr: 10 of 10 available, clientName: CLIENT-ENUM-VM, IOCP: (Busy=0,Free=1000,Min=2,Max=1000), WORKER: (Busy=3,Free=32764,Min=2,Max=32767), v: 2.0.601.3402 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)
at Project.Synchronizer.Service.RedisHelper.FlushDb(Int32 dbIndex)
at Project.Synchronizer.Service.RedisHelper.SelectAndClearRedisDbForNumberUpload()
at Project.Synchronizer.Service.RedisHelper.UploadNumbers()
at Project.Synchronizer.Service.DataUploadService.UploadNumbersToRedis()
The way I try to execute this command is
private async Task FlushDb(int dbIndex)
{
await getDB(dbIndex).ExecuteAsync("flushdb");
}
Any kind of help would be greatly appreciated, because I am stuck on this for quite a while now. For the record, I was trying to get some answers here, but couldn't figure anything out https://stackexchange.github.io/StackExchange.Redis/Timeouts.html
Well, It seems that I somehow managed to solve the problem, but haven't delved too deep to what causes it. Long story short increasing syncTimeout seemed to help and the exception is no longer thrown. This thread was helpful in my case stackexchange.redis throws timeout even after increasing timeout?
Related
I'm using Microsoft.Web.Redis.RedisSessionStateProvider
our network team provide me
1- Hostname host_ip
2- Ports
a. Read 6379
b. Write 6380
and my connection string
connectionString="host_ip:6379,host_ip:6380,user=*****,password=****,ConnectTimeout=50000,connectRetry=5,syncTimeout=50000,abortConnect=false"
Timeout performing EVAL (30000ms), inst: 0, qu: 0, qs: 0, aw: False, bw: Inactive, rs: ReadAsync, ws: Idle, in: 0, in-pipe: 0, out-pipe: 0, last-in: 4, cur-in: 0, sync-ops: 1, async-ops: 3, serverEndpoint: 192.168.33.107:6380, conn-sec: 52.08, mc: 1/1/0, mgr: 10 of 10 available, clientName: *******(SE.Redis-v2.6.90.64945), PerfCounterHelperkeyHashSlot: 9255, IOCP: (Busy=0,Free=1000,Min=8,Max=1000), WORKER: (Busy=1,Free=32766,Min=8,Max=32767), v: 2.6.90.64945 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)
serverEndpoint: 192.168.33.107:6380 this port for write
rs: ReadAsync operation is reading
we are using Elastic cache redis that aws provides in our .Net Core application and when we perform the below operation sometimes we are getting the below TimeoutException after sometime can someone please help me to solve this attaching exception for reference
_redis.KeyExists("cusID"+customerID) -> operation performed
Exception has occurred: CLR/StackExchange.Redis.RedisTimeoutException
Exception thrown: 'StackExchange.Redis.RedisTimeoutException' in StackExchange.Redis.dll: 'Timeout performing EXISTS (5000ms), next: EXISTS cusID1, inst: 0, qu: 0, qs: 0, aw: False, bw: SpinningDown, rs: ReadAsync, ws: Idle, in: 0, serverEndpoint: master.test-stage-test-new-jayaram3.mvg9d6.aps1.cache.amazonaws.com:6379, mc: 1/1/0, mgr: 10 of 10 available, clientName: JAYA-397(SE.Redis-v2.5.61.22961), IOCP: (Busy=0,Free=10000,Min=10000,Max=10000), WORKER: (Busy=2,Free=32765,Min=10000,Max=32767), POOL: (Threads=13,QueuedItems=0,CompletedItems=790), v: 2.5.61.22961 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)'
at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 1857 at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in //src/StackExchange.Redis/RedisBase.cs:line 53
at StackExchange.Redis.RedisDatabase.KeyExists(RedisKey key, CommandFlags flags) in //src/StackExchange.Redis/RedisDatabase.cs:line 667
An exception of type 'StackExchange.Redis.RedisTimeoutException' occurred in StackExchange.Redis.dll but was not handled in user code: 'Timeout performing GET (5000ms), active: GET cusID1, next: GET cusID1 ** possible thread-theft indicated; see https://stackexchange.github.io/StackExchange.Redis/ThreadTheft ** , inst: 1, qu: 0, qs: 0, aw: False, bw: SpinningDown, rs: CompletePendingMessageSync, ws: Idle, in: 0, serverEndpoint: master.suremdm-stage-test-new-jayaram3.mvg9d5.aps1.cache.amazonaws.com:6379, mc: 1/1/0, mgr: 8 of 10 available, clientName: JAYA-397(SE.Redis-v2.5.61.22961), IOCP: (Busy=3,Free=9997,Min=10000,Max=10000), WORKER: (Busy=5,Free=32762,Min=10000,Max=32767), POOL: (Threads=12,QueuedItems=0,CompletedItems=2634), v: 2.5.61.22961 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)'
If need some more information, just ask me that I'll try to provide. Thanks in advance.
Are you setting "AbortOnConnectFail" to false so it continues to try to reconnect? When a connection breaks (could be due to a failover etc), commands already sent to Redis on the now broken connection will timeout on the client.
I'm pushing some data to a Redis instance from a console app every few seconds. This is roughly how I'm doing it:
int foo = GetFoo();
BigObject bar = GetBigObject();
_cache.StringSet("Foo", JsonConvert.SerializeObject(foo));
_cache.StringSet("Bar", JsonConvert.SerializeObject(bar));
but after a while I get an exception:
StackExchange.Redis.RedisTimeoutException: 'Timeout performing SET
RtSignal-op (5000ms), inst: 0, qu: 0, qs: 0, aw: False, rs:
CompletePendingMessage, ws: Idle, in: 0, in-pipe: 5, out-pipe: 0,
serverEndpoint: Unspecified/localhost:5002, mgr: 9 of 10 available,
clientName: SVGD0083, IOCP: (Busy=0,Free=1000,Min=16,Max=1000),
WORKER: (Busy=3,Free=32764,Min=16,Max=32767), v: 2.0.593.37019 (Please
take a look at this article for some common client-side issues that
can cause timeouts:
https://stackexchange.github.io/StackExchange.Redis/Timeouts)'
In the linked to page there is a suggestion that the issue might be a result of Thread Theft and the solution is to include the following line:
ConnectionMultiplexer.SetFeatureFlag("preventthreadtheft", true);
The problem is that there doesn't seem to exist a SetFeatureFlag method in .NET framework.
Any ideas?
What is the size of your objects? If JsonConvert.SerializeObject(foo) is returning a 500MB string, then yes: you're going to have a bad day.
Also, what else is the server doing? This needs a little digging at the server, but: the redis SLOWLOG command gives you information into long-running operations. If anything is taking serious time (I'd start to get twitchy at anything that takes more than say 10 milliseconds), then your server is essentially stalling, which needs to be investigated - but that's not something the library can fix.
If you're talking to a far away server, you may want a larger timeout.
Finally, I very much doubt that this particular scenario is related to thread-theft; if anything, that sounds like you're clutching at straws (metaphorically). But if ConnectionMultiplexer.SetFeatureFlag doesn't exist, you're probably using an old library version. So: what library version are you using?
StackExchange.Redis is not part of the .net framework.
_cache probably is the StackExchange.Redis.IDatabase object type.
So, just add the line where you initialize redis support at your application - probably at startup.cs.
I'm implementing a cache using the tagging mechanism in CachingFramework.Redis.
My app can read and write to Redis successfully, and I can create tags in Redis (which I can see in Desktop Manager). However I get a timeout exception only when I attempt to read from the tag store in Redis:
System.TimeoutException: 'Timeout performing TYPE :$tag$:_HB, inst: 63, mgr: ExecuteSelect, err: never, queue: 1, qu: 1, qs: 0, qc: 0, wr: 0, wq: 1, in: 0, ar: 0, clientName: LAPTOP-GLFEVSO4, serverEndpoint: 127.0.0.1:6379, keyHashSlot: 14024, IOCP: (Busy=0,Free=1000,Min=200,Max=1000), WORKER: (Busy=3,Free=997,Min=200,Max=1000), Local-CPU: 100% (Please take a look at this article for some common client-side issues that can cause timeouts: https://github.com/StackExchange/StackExchange.Redis/tree/master/Docs/Timeouts.md)'
The calling code is like this:
var tags = new List<string>();
if (conditionA == true) { tags.Add("_HB"); }
if (conditionB == false) { tags.Add("_NHB");}
var foundKeys = Redis.Cache.GetKeysByTag(tags.ToArray()); //the timeout happens here
I've checked that memory usage and CPU are not an issue, and I've increased MinIOThreads in machine.config. I'm out of ideas... is this a bug in CachingFramework.Redis (or the underlying StackExchange.Redis), or am I doing something wrong?
[* I'm using VS2017/IIS Express, in debug mode, against a local Redis standalone instance on my development machine]
UPDATE
The issue only occurs in debug mode in Visual Studio. At runtime, it seems fine. However, it would be nice to be able to debug.
I am using StackExchange.Redis in my application to store key/values.
The issue I am getting is that while fetching/checking the cache for a key I am getting TimeOut Exception:
Here is the detailed exception:
System.TimeoutException was unhandled HResult=-2146233083
Message=Timeout performing EXISTS GlobalSettings, inst: 0, mgr:
ExecuteSelect, err: never, queue: 56, qu: 56, qs: 0, qc: 0, wr: 0, wq:
1, in: 0, ar: 0, IOCP: (Busy=0,Free=1000,Min=2,Max=1000), WORKER:
(Busy=20,Free=32747,Min=2,Max=32767), clientName: WIN-VDIGHSLJUBV
Source=StackExchange.Redis.StrongName StackTrace:
at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message
message, ResultProcessor1 processor, ServerEndPoint server)
at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor1 processor, ServerEndPoint server)
at StackExchange.Redis.RedisDatabase.KeyExists(RedisKey key, CommandFlags flags)
I have the following code:
IDatabase db = GetDatabase();
return db.KeyExists(cacheKey);
and the exception I am getting is at line "db.KeyExists(cacheKey)".
Based on the issue I thought increasing timeout might help and therefore I did this in initialization:
redis = ConnectionMultiplexer.Connect(string.Format("{0},allowAdmin=true,syncTimeout=30000,connectTimeout=30000", redisConfig));
i.e I added the "syncTimeout=30000,connectTimeout=30000"
but that didn't help either.
Do I need to add timeouts to some other locations?
Do I need to look the fix for this issue at some other point of Redis logic?
I just read through the following stack overflow post, and I think it might provide another piece of the puzzle: StackExchange.Redis.RedisTimeoutException: Timeout awaiting response
MÇT points out that one might need to increase the minimum thread count in the application's ThreadPool when, as I saw in your exception message,
WORKER: (Busy=20,Free=32747,Min=2,Max=32767)
The number of Busy worker threads is greater than the Min value, which represents the number of worker threads initially created.
Please take a look and his answer for the specific details. This posting is a simple attempt to direct an unanswered question to actionable information.