RedisResponseException from BlockingDequeue - c#

I am getting what looks like a timeout exception when using a BlockingDequeue on a RedisTypedClient.
The calling code looks like
using (var client = ClientPool.GetClient())
return client.As<TMessage>().Lists[_channel].BlockingDequeue(timeout);
Where the timeout is set to 0 and the ClientPool is a PooledRedisClientManager.
The stack trace looks like
ServiceStack.Redis.RedisResponseException: No more data, sPort: 51100, LastCommand:
at ServiceStack.Redis.RedisNativeClient.CreateResponseError(String error)
at ServiceStack.Redis.RedisNativeClient.ReadMultiData()
at ServiceStack.Redis.RedisNativeClient.SendExpectMultiData(Byte[][] cmdWithBinaryArgs)
at ServiceStack.Redis.RedisNativeClient.BRPop(String listId, Int32 timeOutSecs)
at ServiceStack.Redis.Generic.RedisTypedClient`1.BlockingDequeueItemFromList(IRedisList`1 fromList, Nullable`1 timeOut)
at ServiceStack.Redis.Generic.RedisClientList`1.BlockingDequeue(Nullable`1 timeOut)
From what I can find, this is an issue with the client holding an open connection. I thought this was supposed to be fixed by using the PooledRedisClientManager, but it seems to still happen. The issue is easy to reproduce. Simply call the BlockingDequeue method and wait about 2-3 minutes and the exception throws.

I had this once on Windows Azure and on redis i did config set timeout 30 and in ServiceStack.Redis i did
var redisFactory = new PooledRedisClientManager(redisConn);
redisFactory.ConnectTimeout = 5;
redisFactory.IdleTimeOutSecs = 30;
And now for some reason it works

It turns out we were sending our Redis requests through a dns entry that was pointing towards an F5 Big IP traffic controller that was setup to drop idle connections after 300 seconds. Once we increased the timeout on the Big IP the error stopped occurring.

Related

Biometrics device ping becoming failed after sometime

I have ZKTeco Biometrics device which is connected with a C# windows application using This tutorial (C# ZKTeco Biometric Device Getting Started).
It is working fine but after sometime, my application becoming failed to ping the device. As below code suggested, I am trying to ping the device after every 25 seconds.
private void TimerCheckPingAndCloseAttendanceForm() {
timerCheckPingAndCloseAttendanceForm = new Timer();
timerCheckPingAndCloseAttendanceForm.Tick += new EventHandler(CheckPingAndCloseAttendanceForm);
timerCheckPingAndCloseAttendanceForm.Interval = 25000;//25 seconds.
timerCheckPingAndCloseAttendanceForm.Start();
}
private void CheckPingAndCloseAttendanceForm(object sender, EventArgs e) {
string ipAddress = tbxDeviceIP.Text.Trim();
if (UniversalStatic.PingTheDevice(ipAddress) == false) {
//CloseAttendaceListForm();
IsDeviceConnected = false;
string infoString = "Application started on " + applicationStartDateTime.ToString() + " and ping failed on " + DateTime.Now.ToString() + " then, app closed while device ip is "+ ipAddress;
File.AppendAllText("ConnectionLog.txt", infoString + Environment.NewLine);
Application.Exit();
//timerCheckPingAndCloseAttendanceForm.Tick -= new EventHandler(CheckPingAndCloseAttendanceForm);
}
}
And when I am trying to ping the command from cmd the device show destination host is unreachable. But whenever I restart the device, the ping working fine. I don't know where is the problem? Either the network problem or its coding issue?
Note: I am doing a ping on regular time interval, because on Disconnected Event is not working. I am assuming ping failed meaning is the device has disconnected with the application.
First of all : Thank you for going through my article
You are doing it the wrong way.
Trying to ping the device after every 25 seconds is unnecessary.
The only job of the UniversalStatic.PingTheDevice method is to check if the device is presumably active, the first time you connect with the device.
If you want to check the status of the device i.e IsDeviceConnected, All you need to do is register to the device OnDisConnected event provided by the SDK.
It seems the code here at line number 57 has already done the OnDisConnected event registration for you.
All you need to do now is set your IsDeviceConnected to false when the objCZKEM_OnDisConnected method in the ZkemClient.cs class is called upon by the device itself.
Sample snippet :
In the ZkemClient.cs class file, between line number 81-84
void objCZKEM_OnDisConnected()
{
IsDeviceConnected = false; // <-- Add this line
}
Now, Every time you try to make a call to the device, All you need to do is check for the value of your IsDeviceConnected.
Not having the actual code and the hardware setup, this answer is a bit of a shot in the dark, but here goes …
Since it works initially, this is not a hardware configuration or network configuration issue. Yet it says that after a while the destination (reader) becomes unavailable. This is probably not a network keepalive issue because you are pinging every 25 sec. Looking at the code that you referenced, it shows opening a connection and hooking up callbacks, and making a call to a hardware feature.
My guess would be maybe you are opening the connection each ping and not closing the connection, then after a number of attempts the hardware jams because there are too many open connections. Just a guess. If this is the problem then to fix it, either close the connection or, better, keep the connection open and re-use it.
Alternative possibility would be that the router(s) between your code and the device are detecting too many pings and blocking the connection as a possible DOS attack. If this is the problem then to fix it, configure the router to allow the traffic.
This sounds like the device misbehaving. The error "destination host is unreachable" corresponds to an ICMP packet, same type of packet as ping but different job, being sent by your router saying "I have no idea which device has that IP". This normally happens when the device stop responding to ARP, which basically asks "who has this IP?" and expects a machine to respond "I have it" with its MAC address. The router constantly refreshes its ARP table, forgetting old values.
So when you boot the device it is 'happy', responding to ARP and responding to pings; however, something happens and it at least stops responding to ARP (probably something more wrong with it). Depending on its architecture it could be loaded down doing other stuff and unable to respond, or it could just be locked up.
Try slowing down other actions to the device (if your polling it for information other than ping, do it slower) and also see if you can get status from the device via another output (does it have a uart?).
OPTION 1
Since that restarting the device fixes your problem for a period of time, check that the IP that you are using is not in use on another device/computer/element_of_the_network.
ZKTeco devices come with the IP 192.168.1.201 configured by default. Configure a different static IP and avoid using DHCP (it´s well known that using DHCP in ZKTeco devices isn´t a good choice since they don´t refresh automatically the IP after rebooting the system or any network change).
Make sure that the IP is not in use and that nobody else will use it.
OPTION 2
Another thing that It may be the cause of your problem, is that you are using zkemkeeper in a different part of your application (or into a different application) and you are not closing the oppened connections properly... That may be blocking all network activity from the device. To close the connection just make sure that you call this sdk method after performing all the necessary actions:
sdk.Disconnect();
It looks like a code issue. While investigating UniversalStatic.PingTheDevice(ipAddress), its found that its calling System.Net.NetworkInformation.Ping.Send setting DontFragment = true. Reference: https://github.com/zaagan/BioMetrix/blob/master/BioMetrixCore/Utilities/UniversalStatic.cs#LC51. The timeout for the ping is set to 120 milli seconds. This tries to send 32 bytes of data to the given IP.
Following is the snippet taken from https://learn.microsoft.com/en-us/dotnet/api/system.net.networkinformation.ping.send?view=netframework-4.7.2 would answer the root-cause of your issue
If the DontFragment property is true and the total packet size exceeds the maximum packet size that can be transmitted by one of the routing nodes between the local and remote computers, the ICMP echo request fails. When this happens, the Status is set to PacketTooBig.
So when you restart your device, possibly, the data travelling on the network gets lost. Hence it started working till the packets reaching its limit.
Few suggestions:
Try calling System.Net.NetworkInformation.Ping.Dispose in PingTheDevice before returns
Increase the timeout from 120 milliseconds to seconds.
Increase the timerCheckPingAndCloseAttendanceForm.Interval to 1 min.
Check the return code of the System.Net.NetworkInformation.Ping.Send and find the associated failure meaning
Please share your findings if the above suggestions do not help you finding the root-cause.
you try this code for ping the device,
try
{
IPAddress ipAddress = IPAddress.Parse(ipAdd);
Ping pingSender = new Ping();
PingOptions options = new PingOptions();
options.DontFragment = true;
// Create a buffer of 32 bytes of data to be transmitted.
string data = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa";
byte[] buffer = Encoding.ASCII.GetBytes(data);
int timeout = 120;
PingReply reply = pingSender.Send(ipAddress, timeout, buffer, options);
if (reply.Status == IPStatus.Success)
return true;
else
{
return false;
}
}
catch (Exception)
{
return false;
}
Thanks.

What would cause a click operation to timeout after 60 seconds?

What would cause a click operation to timeout after 60 seconds? I get an exception thrown, even though the element is clearly displayed in the page and is clickable (I have a ExpectedConditions.ElementToBeClickable check before the click). OpenQA.Selenium.WebDriverException : The HTTP request to the remote WebDriver server for URL http://grid:4444/wd/hub/session/9e9693f0-0288-47a1-97f9-8f61c300bc41/element/29/click timed out after 60 seconds. ----> System.Net.WebException : The operation has timed out
WebDriverWait wait = new WebDriverWait(this.driverController.driver, TimeSpan.FromSeconds(30.00));
IWebElement clickThis;
wait.Until(ExpectedConditions.ElementToBeClickable(By.Id("elementId")));
clickThis = this.driverController.driver.FindElement(By.Id("elementId"));
clickThis.Click();
The exception occurs at clickThis.Click()
Thanks,
Your exception indicates that it is the selenium grid hub that causes the timeout. The hub itself has a timeout, which works as described in the selenium documentation and below:
-timeout 30 (300 is default) The timeout in seconds before the hub automatically releases a node that hasn't received any requests for
more than the specified number of seconds. After this time, the node
will be released for another test in the queue. This helps to clear
client crashes without manual intervention. To remove the timeout
completely, specify -timeout 0 and the hub will never release the
node.
If the issue would have been related to finding the element you wish to interact with, you should have receive a NoSuchElementException or similiar.

After using the InternetConnect() API in wininet how can I tell if I'm still connected?

I use the InternetConnect() method from the WinINet APIs. I connect to my FTP server just fine with no issues. After I connect, I wait about 1 min and the server disconnects me because of no activity as expected. I then try to send a file but I'm not connected.
Is there a way to "check" the FTP connection to see if I'm still connected? Or is there some type of way for me to attach an event to tell me when I get disconnected?
I haven't used wininet for FTP, and I use the classes, not the global functions directly. But I suspect that CFtpConnection behaves the same way as CHttpConnection in this respect. Anyway, you might learn something from what I have discovered about the latter.
CHttpConnection seems to be a high and abstract level of connection. When I started out I expected its member functions to throw exceptions once the server closed the underlying socket (for timeout). I now know better or at least believe otherwise. NORMAL closing of the socket does NOT cause exceptions to be thrown at this high level of classy wininet. You might suspect as much from inspecting the wininet error codes: There is no code corresponding to the server having closed the connection.
I experimented with this and found: The server closing the socket (for timeout) is considered normal and does not cause an exception to be thrown. You can go ahead and use CHttpConnection without worrying about this. It will simply reconnect if needed without alerting you. So once you have called GetHttpConnection and got your CHttpConnection object, it will normally last forever!
The exceptions that might be thrown, ERROR_INTERNET_CONNECTION_ABORTED and ERROR_INTERNET_CONNECTION_RESET, are caused by abnormal conditions, f.ex. a proxy server crashing or somebody accidentally pulling the power plug to your modem. The server closing the socket for timeout is considered NORMAL and is transparent to the user of wininet classes.
So the tentative conclusion is that you don't have to worry about the connection being closed by the server. If that happens, CHttpConnection will reconnect backstage and you won't be bothered. You can pretend that the connection always stays open - it seems so to the user of wininet classes.
Consider the following experiment. A connection is opened and then a request is sent once a minute. The function returns once an exception is thrown. But if not, then it loops forever. I tried it on two different web sites: An exception is NEVER thrown! Despite a whole minute of inactivity between requests.
int httpclient::test(string host)
{
int flags = INTERNET_FLAG_RELOAD;
int port = INTERNET_DEFAULT_HTTP_PORT;
CHttpConnection *connection = session.GetHttpConnection(host.cstring(),flags,port);
int secs = 0;
while (true)
{
CHttpFile *fil;
try
{
fil = connection->OpenRequest(CHttpConnection::HTTP_VERB_HEAD, "index.htm",
0,1,0, "HTTP/1.1", flags);
}
catch (CInternetException *exc)
{
connection->Close();
int feil = exc->m_dwError;
exc->Delete();
return -feil;
}
fil->AddRequestHeaders("Connection: Keep-Alive");
try
{
fil->SendRequest();
}
catch (CInternetException *exc)
{
connection->Close();
int feil = exc->m_dwError;
exc->Delete();
return feil;
}
fil->Close();
Sleep(60 * 1000);
secs += 60;
printf("%u seconds passed\n", secs);
}
return 0;
}
Take all this with a grain of salt. wininet is poorly documented and all I know is what experiments have taught me.

Quick check if WCF endpoint is available (and how to handle if not)

I have a very simple WCF service running.
An ASP.NET web application is consuming this service from code behind during pageload like this:
try
{
using (var myWCFClient = new MyWCFClient())
{
int myInt = myWCFClient.GetValue();
}
}
catch (Exception ex)
{
}
Where "MyWCFClient" is the proxy object which is autogenerated when adding a service reference to my project.
Everything works fine, but the problem is when the Service Endpoint is down for some reason, it takes more than 30 seconds for the EndpointNotFoundException is catched. Of course this is unacceptable because it delays pageload since everything is synchronous. And because i have no simple mechanism to push the data to the page async after pageload, an async call to the WCF service is not preferred.
I also tried to set the sentTimeout in web.config to 5 seconds, but this doen't solve the problem.
When is set the OperationTimeout of the InnerChannel to 5 seconds like this...
((IContextChannel)myWCFClient.InnerChannel).OperationTimeout = TimeSpan.FromSeconds(5);
...i do get a TimeOutException, but this exception is not throwed within 5 seconds, it also takes more than 30 seconds...
Does anybody knows how to handle this situation, or is there a way to quick check if the service is running before doing the actual call???
We use a ping function on the host that simply returns True in the response payload, and if you response takes longer than 5 seconds, we throw our own exception and exit the process. All the logic is on the client side, minus the Ping() function. We don't wait for the timeout exception.

Sending SMS for a bulk of users

I want to send SMS to bulk of users(4000 user) so i put the following method on loop :
protected int SendSMS(string url)
{
// Now to Send Data.
StreamWriter writer = null;
StringBuilder postData = new StringBuilder();
Uri myUri = new Uri(url);
postData.Append(HttpUtility.ParseQueryString(myUri.Query).Get("Username"));
postData.Append(HttpUtility.ParseQueryString(myUri.Query).Get("Password"));
postData.Append(HttpUtility.ParseQueryString(myUri.Query).Get("Sender"));
postData.Append(HttpUtility.ParseQueryString(myUri.Query).Get("Recipients"));
postData.Append(HttpUtility.ParseQueryString(myUri.Query).Get("MessageData"));
string webpageContent = string.Empty;
byte[] byteArray = Encoding.UTF8.GetBytes(postData.ToString());
HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(url);
webRequest.Method = "POST";
webRequest.ContentType = "application/x-www-form-urlencoded";
webRequest.ContentLength = webRequest.ContentLength = byteArray.Length;
writer = new StreamWriter(webRequest.GetRequestStream());
try
{
using (Stream webpageStream = webRequest.GetRequestStream())
{
webpageStream.Write(byteArray, 0, byteArray.Length);
}
using (HttpWebResponse webResponse = (HttpWebResponse)webRequest.GetResponse())
{
using (StreamReader reader = new StreamReader(webResponse.GetResponseStream()))
{
webpageContent = reader.ReadToEnd();
//TODO:parse webpagecontent: iF response contains "OK"
if (webpageContent.Contains("OK")) return 1;
else return 0;
}
}
//return 1;
}
catch (Exception ee)
{
ErrMapping.WriteLog(url);
string error = ee.Message + "<br><br>Stack Trace : " + ee.StackTrace;
ErrMapping.WriteLog(error);
return -1;
}
}
After a specific number of users like 65 user, no sms had been sent for the rest of users and
I get the following exception :
Error Message:Thread was being aborted.<br><br>Stack Trace : at System.Net.UnsafeNclNativeMethods.OSSOCK.recv(IntPtr socketHandle, Byte* pinnedBuffer, Int32 len, SocketFlags socketFlags)
at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags, SocketError& errorCode)
at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.PooledStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.Connection.SyncRead(HttpWebRequest request, Boolean userRetrievedStream, Boolean probeRead)
at System.Net.ConnectStream.ProcessWriteCallDone(ConnectionReturnResult returnResult)
at System.Net.HttpWebRequest.CheckDeferredCallDone(ConnectStream stream)
at System.Net.HttpWebRequest.GetResponse()
at SendSMS_EmailUI.Frm_SMS_send.SendSMS(String url)
IMHO, the bulk operations that will be performed on behalf of the application can be done easily by the following procedure
When a bulk SMS is triggered, the details will be entered in a database table
A windows service will be constantly monitoring this table for any updates
When the windows service finds the new entries, it will take few records like few hundreds and then send them. Batch Processing.
There can be a delay between consequent requests.
This will ensure you to track which line items have failed and also does not clog the server with the bulk data.
This is a most widely suggested approach.
Please provide your comments on this implementation.
I have built an SMS portal. What you describe was also experienced in v1.0 of my application. The solution was to have my SMS gateway provide me with Bulk SMS access via HTTP. I could put up to 1000 destinations into an XML (or comma delimited) package and send to the Bulk SMS Gateway. Because I run on a shared host, I limited this to 500 destinations.
I have a cache/temporary storage/table where I batch large destination (up to 1,000,000) in some cases and a scheduler (timer based) sends each batch of 500 every few seconds (by calling a script repeatedly) until the messages are sent. Works like charm!
For personalized messages, I encourage the client to use my desktop application for personalization before forwarding to my SMS portal. Good luck
PROCESS:
You'll need three items
The Script that receives the SendSMS request
The script that sends the SMS
The Scheduler/Timer (Script/Host Service)
A. The Send SMS request arrives with
a. The Message and Sender ID/ Sender GSM Number.
b. The Destinations (as a comma delimited list). We'll assume 10,000 destinations
B. Split the destinations into 500 (any size you wish) and log the 500 destinations along with the message and SenderID in each INBOX row/record.
Note: if you count 500 out by looping (10,000 loops), the script could Time out. GSM Numbers in My country are 13 digits.
So I do a loop Sub String of length 500(13+1) to get 500 destinations per batch (20 loops).*
C. Call the Script that Sends the SMS. Sends the first 500 and tag the message as Sent. You can add Time Sent. Start the Scheduler
D. The Scheduler checks every 1.5 minutes if any unsent messages exist in the INBOX and sends it. If nothing, Scheduler Stops.
So, 10,000 messages are sent within 30 mintues
We do something similar to what saravan suggested for email messages, and suspect it will work SMS.
Basically the service that runs on our web server only sends x at a time, and there is a custom delay y between each send.
It can send two thousand in less than ten minutes with neither a CPU nor a bandwidth spike.
Some tips that weren't in our original design to keep in mind:
Have a way to manually stop all sending (see next tip)
Have a user friendly way to terminate a particular message. If your code (or the user) accidentally sends the same thing five times, you want a way to abort it, quickly.
Use a config file for both numbers (x & y above) so you can adjust without redeploying. I think the y delay is 50 ms.
Before you decide to bundle the same message to multiple recipients, make sure that smartphones can't reply to everyone else in their bundle.
HTH,
-Chris C.
I suspect the problem is that because you are on an asp.net website and this is a long-running task that it is timing out the response.
this timeout is controlled in the web.config and it's default is set to 110 seconds. Increase this to a much longer number and see if it starts working.
<system.web>
<!-- 600 seconds = 10 minute timeout --->
<httpRuntime executionTimeout="600"/>
</system.web>
A better approach would be to using a separate thread and returning updates of the progress
to the user, although this would be more complex, but ultimately more reliable.
See Parallel.ForEach for a simple way to thread this process.
Msdn - httpRuntime documention
This is not answering your original question, but is related useful info. Once you solve this problem, you may hit another -- telcos routinely block bulk-sent SMS messages as a way to suppress SMS spam. If you will be doing this on a commercial scale, you will need to file a brief with the Mobile Marketing Association, and get their approval, which can take a considerable amount of time.
check if your StreamWriter is getting disposed
using (StreamWriter writer = new StreamWriter(webRequest.GetRequestStream()))
{
}
May be the buffer value is full please check it and put it to maximum value in web-config file
Maybe your are making web request too fast: try slowing them down to rule out timing problems, for example adding a sleep:
System.Threading.Thread.Sleep(1000);
1000ms = 1 sec is only a hint, you should try different values to see if something changes

Categories

Resources