Develop Reference

RedisDB throws timeout exception when trying to execute flushdb

I have a problem with StackExchange.Redis. In our process we have two redis databases. These databases are used to store similar information which the user can execute lookup onto, so we are switching between them on imports, so the user can continuosly access the information. Everyday around 50000000 records are inserted to one database and old records are deleted from the other, to prepare it for next day's import. The records are imported in batches of 100000 records. The problem we are facing is that after the data import has completed, Redis throws an exception, while trying to execute flushdb command.
StackExchange.Redis.RedisTimeoutException Timeout awaiting response (outbound=0KiB, inbound=0KiB, 5976ms elapsed, timeout is 5000ms), command=UNKNOWN, next: SELECT, inst: 0, qu: 0, qs: 2, aw: False, rs: ReadAsync, ws: Idle, in: 0, in-pipe: 0, out-pipe: 0, serverEndpoint: 127.0.0.1:6379, mgr: 10 of 10 available, clientName: MX-ENUM-VM, IOCP: (Busy=0,Free=1000,Min=2,Max=1000), WORKER: (Busy=3,Free=32764,Min=2,Max=32767), v: 2.0.601.3402 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts) at Project.Synchronizer.Service.RedisHelper.FlushDb(Int32 dbIndex)
at Project.Synchronizer.Service.RedisHelper.SelectAndClearRedisDbForNumberUpload()
at Project.Synchronizer.Service.RedisHelper.UploadNumbers()
at Project.Synchronizer.Service.DataUploadService.UploadNumbersToRedis() StackExchange.Redis.RedisTimeoutException: Timeout awaiting response (outbound=0KiB, inbound=0KiB, 5976ms elapsed, timeout is 5000ms), command=UNKNOWN, next: SELECT, inst: 0, qu: 0, qs: 2, aw: False, rs: ReadAsync, ws: Idle, in: 0, in-pipe: 0, out-pipe: 0, serverEndpoint: 127.0.0.1:6379, mgr: 10 of 10 available, clientName: CLIENT-ENUM-VM, IOCP: (Busy=0,Free=1000,Min=2,Max=1000), WORKER: (Busy=3,Free=32764,Min=2,Max=32767), v: 2.0.601.3402 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)
at Project.Synchronizer.Service.RedisHelper.FlushDb(Int32 dbIndex)
at Project.Synchronizer.Service.RedisHelper.SelectAndClearRedisDbForNumberUpload()
at Project.Synchronizer.Service.RedisHelper.UploadNumbers()
at Project.Synchronizer.Service.DataUploadService.UploadNumbersToRedis()
The way I try to execute this command is
private async Task FlushDb(int dbIndex)
{
await getDB(dbIndex).ExecuteAsync("flushdb");
}
Any kind of help would be greatly appreciated, because I am stuck on this for quite a while now. For the record, I was trying to get some answers here, but couldn't figure anything out https://stackexchange.github.io/StackExchange.Redis/Timeouts.html

Well, It seems that I somehow managed to solve the problem, but haven't delved too deep to what causes it. Long story short increasing syncTimeout seemed to help and the exception is no longer thrown. This thread was helpful in my case stackexchange.redis throws timeout even after increasing timeout?

CachingFramework.Redis Timeout when using Tags

I'm implementing a cache using the tagging mechanism in CachingFramework.Redis.
My app can read and write to Redis successfully, and I can create tags in Redis (which I can see in Desktop Manager). However I get a timeout exception only when I attempt to read from the tag store in Redis:
System.TimeoutException: 'Timeout performing TYPE :$tag$:_HB, inst: 63, mgr: ExecuteSelect, err: never, queue: 1, qu: 1, qs: 0, qc: 0, wr: 0, wq: 1, in: 0, ar: 0, clientName: LAPTOP-GLFEVSO4, serverEndpoint: 127.0.0.1:6379, keyHashSlot: 14024, IOCP: (Busy=0,Free=1000,Min=200,Max=1000), WORKER: (Busy=3,Free=997,Min=200,Max=1000), Local-CPU: 100% (Please take a look at this article for some common client-side issues that can cause timeouts: https://github.com/StackExchange/StackExchange.Redis/tree/master/Docs/Timeouts.md)'
The calling code is like this:
var tags = new List<string>();
if (conditionA == true) { tags.Add("_HB"); }
if (conditionB == false) { tags.Add("_NHB");}
var foundKeys = Redis.Cache.GetKeysByTag(tags.ToArray()); //the timeout happens here
I've checked that memory usage and CPU are not an issue, and I've increased MinIOThreads in machine.config. I'm out of ideas... is this a bug in CachingFramework.Redis (or the underlying StackExchange.Redis), or am I doing something wrong?
[* I'm using VS2017/IIS Express, in debug mode, against a local Redis standalone instance on my development machine]
UPDATE
The issue only occurs in debug mode in Visual Studio. At runtime, it seems fine. However, it would be nice to be able to debug.

Randomly strange behaviour when using select() with timeout on Windows

I'm using a tool to communicate with a gameserver. To establish the connection with the gameserver I'm sending a login packet and then go on from there. I also used a tool, that does the same, but which is written by someone else in C# with a pre-made library. This app has some issues with stackoverflow exceptions after using it for hours and porting it linux isn't much fun aswell, therefore I decided to write my own application from scratch in C++.
My script pretty much looks like this:
while (!connected) {
if (connectCounter == 0)
std::cout << "Trying to connect..." << std::flush;
else
std::cout << "." << std::flush; // add point
connectCounter++;
int selectSize = 0;
struct timeval timeout;
timeout.tv_sec = 5;
timeout.tv_usec = 0;
fd_set fds;
FD_ZERO(&fds);
FD_SET(mysocket, &fds);
selectSize = select(mysocket + 1, &fds, 0, 0, &timeout);
if (selectSize == 1) {
// we might now be logged in, check routines
connected = true;
}
}
Now there's a "bug" randomly happening to me in both applications, the one written by someone else in C# and in my own one. I should probably mention that I've never had this behaviour before, but sinced I formatted my computer I saw this issue happpening for the first time.
Issue: Gameserver was offline for some hours, computer was probably freshly booted. Gameserver is still down and I start the application. Now it tries to login but won't have success as the gameserver is still offline. Now it writes "Trying to connect". Because of the timeout settings it should wait 5 seconds and then add 1 point after every unsuccessful try. Instead it fires point after point without waiting for the timeout. This happens in both application, the C# app written by someone else and in my own application. In both applications it only happens randomly and not every time I'm starting the application. As I mentioned I've never experienced this issues before formatting my computer. I also ported this application to my linux server and didn't not experience that behaviour on linux. A friend of mine also uses both applications and never reported that kind of issue to me.
This is so strange to me and I can't figure out the reason for it. From what I get this can't really be code related because it happens in two totally different applications and from what I can tell only since I reinstalled Windows.
EDIT 1: Now I found something interesting, I added the following code on windows and linux:
selectSize = select(mysocket + 1, &fds, 0, 0, &timeout);
std::cout << selectSize << std::cout;
Interesting thing is that on Windows my console will now output: Trying to connect...0.1.0.1.0.1.0.1
Restarted the application and it outputs Trying to connect...0.0.0.0.0.1
On linux it always returns Trying to connect...0.0.0.0.0, never ever with a false positive.
Still only happening on windows. Don't even know what approach the guy from the C# application used but there it's the same problem happening randomly after reinstalling windows.
EDIT 2: I guess I found the problem.
Before the timeout settings and select() I'm doing a sendto() with my login packet. I guess for whatever reason there is something incoming in return, so that selectSize might change to 1 in some cases. Is it possible that this is causing the issue on Windows, while its working on linux?

Quoting from "the" POSIX specification (a copy of it online):
A descriptor shall be considered ready for reading when a call to an input function with O_NONBLOCK clear would not block, whether or not the function would transfer data successfully. (The function might return data, an end-of-file indication, or an error other than one indicating that it is blocked, and in each of these cases the descriptor shall be considered ready for reading.)
So I'd say in order to fix your code you must additionally check whether file descriptors that are "ready for reading" don't have any error or eof indication.

To check if the socket is connected, you should check it for writability, not readability. Change
selectSize = select(mysocket + 1, &fds, 0, 0, &timeout);
to
selectSize = select(mysocket + 1, 0, &fds, 0, &timeout);

Okay, so it seems that I finally found at least a partial answer to my initial question, why linux gives me a working result while windows breaks my application. From what I have read on windows platforms select() returns WSAECONNECTRESET instead of blocking or timeout, see: WinSock Recvfrom() now returns WSAECONNRESET instead of blocking or timing out
So this seems to be the reason why the application is working perfectly fine (for my purposes) on linux, where select() still seems to return a timeout while Windows returns that error and breaks my application to a certain extent.
Solution:
So I finally found a fix. Special thanks to the guy who reminded me to use Wireshark. At first I tought the select() giving back 1 when it should be 0 after sending the login packet to gameserver while it's offline is totally random but in fact I found out that from time to time I get a "ICMP port unreachable", this caused select() to return 1 instead of 0 (see the link above) Obviously I only want select() to return 1 when an actual login reponse is coming from the sever. On linux this works out of the box and doesn't cause any problems.
For Windows I found a simple fix by adding this code before the select() function:
#define SIO_UDP_CONNRESET _WSAIOW(IOC_VENDOR, 12)
DWORD lpcbBytesReturned = 0;
BOOL lpvInBuffer = FALSE;
WSAIoctl(mysocket, SIO_UDP_CONNRESET, &lpvInBuffer, sizeof(lpvInBuffer), NULL, 0, &lpcbBytesReturned, NULL, NULL);

Timeout exception while using Redis

I am using StackExchange.Redis in my application to store key/values.
The issue I am getting is that while fetching/checking the cache for a key I am getting TimeOut Exception:
Here is the detailed exception:
System.TimeoutException was unhandled HResult=-2146233083
Message=Timeout performing EXISTS GlobalSettings, inst: 0, mgr:
ExecuteSelect, err: never, queue: 56, qu: 56, qs: 0, qc: 0, wr: 0, wq:
1, in: 0, ar: 0, IOCP: (Busy=0,Free=1000,Min=2,Max=1000), WORKER:
(Busy=20,Free=32747,Min=2,Max=32767), clientName: WIN-VDIGHSLJUBV
Source=StackExchange.Redis.StrongName StackTrace:
at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message
message, ResultProcessor1 processor, ServerEndPoint server)
at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor1 processor, ServerEndPoint server)
at StackExchange.Redis.RedisDatabase.KeyExists(RedisKey key, CommandFlags flags)
I have the following code:
IDatabase db = GetDatabase();
return db.KeyExists(cacheKey);
and the exception I am getting is at line "db.KeyExists(cacheKey)".
Based on the issue I thought increasing timeout might help and therefore I did this in initialization:
redis = ConnectionMultiplexer.Connect(string.Format("{0},allowAdmin=true,syncTimeout=30000,connectTimeout=30000", redisConfig));
i.e I added the "syncTimeout=30000,connectTimeout=30000"
but that didn't help either.
Do I need to add timeouts to some other locations?
Do I need to look the fix for this issue at some other point of Redis logic?

I just read through the following stack overflow post, and I think it might provide another piece of the puzzle: StackExchange.Redis.RedisTimeoutException: Timeout awaiting response
MÇT points out that one might need to increase the minimum thread count in the application's ThreadPool when, as I saw in your exception message,
WORKER: (Busy=20,Free=32747,Min=2,Max=32767)
The number of Busy worker threads is greater than the Min value, which represents the number of worker threads initially created.
Please take a look and his answer for the specific details. This posting is a simple attempt to direct an unanswered question to actionable information.

Socket.SendAsync is not sending in-order on Mono/Linux

There is a a single-threaded server using .NET Socket with TCP protocol, and Socket.Pool(), Socket.Select(), Socket.Receive().
To send, I used:
public void SendPacket(int clientid, byte[] packet)
{
clients[clientid].socket.Send(packet);
}
But it was very slow when sending a lot of data to one client (halting the whole main thread), so I replaced it with this:
public void SendPacket(int clientid, byte[] packet)
{
using (SocketAsyncEventArgs e = new SocketAsyncEventArgs())
{
e.SetBuffer(packet, 0, packet.Length);
clients[clientid].socket.SendAsync(e);
}
}
It works fine on Windows with .NET (I don't know if it's perfect), but on Linux with Mono, packets are either dropped or reordered (I don't know). Reverting to slow version with Socket.Send() works on Linux. Source for whole server.
How to write non-blocking SendPacket() function that works on Linux?

I'm going to take a guess that it has to do with your using statement and your SendAsync call. Perhaps e falls out of scope and is being disposed while SendAsync is still processing the buffer. But then this might throw an exception. I am really just taking a guess. Try removing the using statement and see what happens.

I would say by not abusing the async method. YOu will find nowhere a documentation stating that this acutally is forced to maintain order. it queues iem for a scheuler which get distributed to threads, and by ignoring that the oder is not maintained per documentation you open yourself up to implementation details.
The best possibly is to:
Have a queue per socket.
When you write dasta into this queue, and there is no worker thread, start a work item (ThreadPool) to process the thread.
This way you have separate distinct queues that maintain order. Only one thread will ever process one queue / socket.

I got the same problem; Linux and windows react not in the same way with SendAsync. Sometimes linux truncate the data, but there is a workaround. First of all you need to use a queue. Each time you use SendAsync you have to check the callback.
If e.Offset + e.BytesTransferred < e.Buffer.Length, you just have to e.SetBuffer(e.Offset + e.BytesTransferred, e.Buffer.Length - e.BytesTransferred - e.Offset); and call SendAsync again.
I dont know why mono-linux believe it's completed before sending all the data and it's strange but i'm sure he does.

just like #mathieu, 10y later, I can confirm on Unity Mono+Linux complete callback is called without all bytes being sent in some cases. For me it was large packets only.