Develop Reference

FTP error: 530 Sorry, Only 5 connections per host are allowed

I have a FTP solution that works great. However after a while, perheps 20-30 minutes, I can get this error:
530 Sorry, Only 5 connections per host are allowed
I have googled on this problem but not really finding any solution to it.
I have different threads calling the function but only allow the function to be called maximum each 2 seconds.
I am make use of: using statements but I think I don't want to use:
request.KeepAlive = false;
to be sure that the connection will be closed. As I before was told to have it open in "ServicePoint" as I use "ConnectionGroupName" because open up a new connection is resource expensive and it is smoother to keep it open.
I think but are not sure. Please tell me if that is a bad approach. But when this error occurs. I like to close all connections, in this case 5 connections so it starts over from Zero connections. Is this possible to do and how would I go about to do that in code since I am not sure how to call those connections in the code?
public async Task<List<String>> FtpGetFiles(String host, String folderpath, String user_name, String password)
{
List<String> serverfileLIST = new List<String>(); String error = "";
try
{
var request = (FtpWebRequest)WebRequest.Create(host + "/" + folderpath + "/");
request.Credentials = new NetworkCredential(user_name, password);
request.Method = WebRequestMethods.Ftp.ListDirectory;
request.ConnectionGroupName = host.Replace(".", "").Replace(":", "").Replace("/", "").Replace("-", "").Replace("_", "") + user_name;
request.ServicePoint.ConnectionLimit = 4;
request.Timeout = 20000;
using (var responseWeb = await request.GetResponseAsync())
{
var response = (FtpWebResponse)responseWeb;
if (response.StatusDescription.Contains("150") || //150 opening binary mode data connection for file list
response.StatusDescription.Contains("226"))
{
using (StreamReader streamReader = new StreamReader(response.GetResponseStream()))
{
while (streamReader.Peek() >= 0)
{
serverfileLIST.Add(streamReader.ReadLine());
}
}
return serverfileLIST;
}
else { error = response.StatusDescription; }
}
}
catch (WebException ex) { String status = ((FtpWebResponse)ex.Response).StatusDescription; error = status; }
return serverfileLIST;
}
EDIT:
I have attached a network.txtfile. I can see in my own logfile that at this time GMT, I got the below error:
09 Dec 2019 16:09:40 530 Sorry, Only 5 connections per host are allowed
09 Dec 2019 16:09:42 530 Sorry, Only 5 connections per host are allowed
The network.txtfile is attached which is GMT also in the link below and I wonder if it is possible to trace what the problem is and how it could be solved?
network.txt
I could find those lines in the txt file where it occurs:
System.Net Information: 0 : [14868] FtpControlStream#13919402 - Sending command [PASS ********]
System.Net Information: 0 : [14868] FtpControlStream#13919402 - Received response [530 Sorry, Only 5 connections per host are allowed]
System.Net Information: 0 : [14868] FtpWebRequest#45677341::(Releasing FTP connection#13919402.)
System.Net Error: 0 : [14868] Exception in FtpWebRequest#45677341::EndGetResponse - The remote server returned an error: (530) Not logged in..
at System.Net.FtpWebRequest.EndGetResponse(IAsyncResult asyncResult)

Just the maximum throughput of one call per every 2 seconds does not prevent the external threads from invoking more than 5 connections. Just imagine you will temporarily have a very slow connection to the server or the list you request will be very long. It could take more time to download than usual (let's say it will take 15 seconds for some reason). Because it will take more time to finish, you will eventually reach the cap of 5 concurrent connections and get the error, because you will be initiating new calls before the previous downloads finished.
You could use SemaphoreSlim to "queue" all the calls to your method FtpGetFiles. This would make sure your threads are not executing more than 5 concurrent calls. When one call finishes, semaphore would release a spot and another call would start.
It would look something like this:
public class TaskQueue
{
private readonly SemaphoreSlim semaphore;
private int pendingTasks;
private int runningTasks;
public TaskQueue(int maxConcurrencyLevel = 5)
{
semaphore = new SemaphoreSlim(maxConcurrencyLevel);
}
private async Task AddTask(Func<Task> task)
{
Interlocked.Increment(ref pendingTasks);
await semaphore.WaitAsync();
Interlocked.Decrement(ref pendingTasks);
Interlocked.Increment(ref runningTasks);
try
{
await task();
}
finally
{
Interlocked.Decrement(ref runningTasks);
semaphore.Release();
}
}
}
Every thread calling your method would call this:
TaskQueue.AddTask(async () => await FtpGetFiles(host, folderpath, username, password));

zeromq receive times out with EGAIN exactly the 255th time

library: clrzmq4 (https://github.com/zeromq/clrzmq4) in a c# project.
I am using zmq router-dealer configuration. The server is written in python and runs on linux. My dealer client written in c# runs on a windows machine. It sends messages and waits from the response
public Boolean sendMessage(Dictionary<String, String> msgDict)
{
ZError err;
String errStr;
var reqFrame = new ZFrame(JsonConvert.SerializeObject(msgDict));
retval = socket.Send(reqFrame, out err);
if (err != null)
{
errStr = String.Format("Error while sending command {3} {0} {1}", err.Text, err.Number, err.Name);
return false;
}
err = null;
respFrame = socket.ReceiveFrame(out err);
if (err != null)
{
errStr = String.Format("Error while receiving response data {0} {1} {2} {3}", err.Text, err.Number, err.Name, num_messages);
return false;
}
return true;
}
I set the sendTimeout and receiveTimeout on the socket to 2 min each.
When I keep calling sendMessage, exactly at the 255th time, receiveFrame timesout . On the server I see the message being processed and response being sent like everytime. And after this point, my send also timesout with the same error "EAGAIN" Resource temporarily unavailable.
There are the things I tried
Data with different lengths from 2 KB to 20 MB
set the sendhighwatermark and receivehighwatermark to different values: 10, 1000, 10000
Tried polling on the socket instead of ReceiveFrame
Tried making the sockets completely blocking.
In each of the above cases the failure occured at exactly the 255th time. In case of blocking sockets, it got blocked at the 255th time too.
I can't use netmq as much as I would like to because it doesn't have curvezmq and the server needs it.
I also tried a dealer client from another linux machine and it had no issues 255th time or even later.

What is the prefered or accepted method for testing proxy settings?

I have a lot of trouble with the internet connectivity in the program I am working on and it all seems to spawn from some issue with the proxy settings. Most of the issues at this point are fixed, but the issue I am having now is that my method of testing the proxy settings makes some users wait for long periods of time.
Here is what I do:
System.Net.WebClient webClnt = new System.Net.WebClient();
webClnt.Proxy = proxy;
webClnt.Credentials = proxy.Credentials;
byte[] tempBytes;
try
{
tempBytes = webClnt.DownloadData(url.Address);
}
catch
{
//Invalid proxy settings
//Code to handle the exception goes here
}
This is the only way that I've found to test if the proxy settings are correct. I tried making a web service call to our web service, but no proxy settings are needed when making the call. It will work even if I have bogus proxy settings. The above method, though, has no timeout member that I can set that I can find and I use the DownloadData as opposed to the DownloadDataAsync because I need to wait til the method is done so that I can know if the settings are correct before continuing on in the program.
Any suggestions on a better method or a work around for this method is appreciated.
Mike
EDIT: I tried something else, but no luck. I used the DownloadDataAsync method to download the data in a separate thread which raises the DownloadDataCompleted event of the WebClient when finished. While I wait for the event to get called I have a loop: while(DateTime.Now < downloadStart.AddMinutes(timeout) && !TestIsDone) {} The DownloadDataCompleted event sets the TestIsDone member to true when the event is called. The problem here is if the proxy settings are bad the Event never gets called, no exception is thrown, and the program waits for the entire timeout period before continuing. Here is the code for this approach:
public static bool TestProxy(System.Net.WebProxy proxy)
{
ProxySettingsTestDone = false; //public static var
string address = //url to some arbitrary data on our server
System.Net.WebClient webClnt = new System.Net.WebClient();
webClnt.Proxy = proxy;
webClnt.Credentials = proxy.Credentials;
try
{
webClnt.DownloadDataCompleted += new System.Net.DownloadDataCompletedEventHandler(DownloadDataCallback);
webClnt.DownloadDataAsync(new Uri(address));
//Timeout period
DateTime dnldStartTime = DateTime.Now;
while (DateTime.Now < dnldStartTime.AddMinutes(1.0) && !ProxySettingsTestDone)
{ }
if (!ProxySettingsTestDone) //Exceded timeout
{
throw new System.Net.WebException("Invalid Proxy Settings");
}
}
catch (System.Net.WebException e)
{
if (e.Status == System.Net.WebExceptionStatus.ProxyNameResolutionFailure)
{
//Proxy failed, server may or may not be there
Util.ConnectivityErrorMsg = e.Message;
return false;
}
else if (e.Status == System.Net.WebExceptionStatus.ProtocolError)
{
//File not found, server is down, but proxy settings succeded
ServerUp = false;
Util.ConnectivityErrorMsg = e.Message;
return true;
}
return false;
}
Util.ConnectivityErrorMsg = "";
return true;
}
private static void DownloadDataCallback(object sender, System.Net.DownloadDataCompletedEventArgs e)
{
if (!e.Cancelled && e.Error == null)
ProxySettingsTestDone = true;
else
throw new System.Net.WebException("Invalid Proxy Settings");
}
Sorry about the long post. I wanted to update this question with the information that I found after testing this new approach.
Thanks,
Mike

You can run the proxycheck in a seperate thread. And consider the check to be failed if the thread takes too long.
Or you could use WebRequest, it allows you set a timeout:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://url");
request.Proxy = proxy;
request.Timeout = 2000;
If the request has not finished within the given timeout a WebException with the Status property set to WebExceptionStatus.Timeout will be thrown.

Every method mentioned here are valid. But the most important one is to test the Proxy connection using the same Windows user account for the process that you want to test. Many proxies has specific privileges for each Windows user.

A recent post Testing IP:Port proxies explains proxy check using a simple Python script. The script checks for availability and correctness of proxy servers.

My IsConnected always returns true

Test-cases:
Before connection starts it should return false
Connection is closed by other end return false
Connection is closed by the client return false
Connection exists even if no data is avaliable return true
class MyConnection
{
//Assume I have all initialization for _socket
public bool IsConnected()
{
return !(_socket.Poll(1, SelectMode.SelectRead)
&& _socket.Available == 0);
}
private Socket _socket;
}
class Test
{
static void Main(string[] args)
{
MyConnection my = new MyConnection()
if(my.IsConnected())
/*always return true even when I am not connected*/;
}
}
Any ideas how to prevent that?
So far, none of the answers were satisfactory....
The following can be done:
public bool IsConnected()
{
bool bConnected = false;
bool bState = _socket.Poll(1, SelectMode.SelectRead);
try
{
if (bState && (_socket.Available == 0))
bConnected = false;
else
bConnected = true;
}
catch (SocketException)
{
//_socket.Available can throw an exception
bConnected = false;
}
return bConnected;
}

I think your _socket.Poll() call is backwards. If the poll fails, that will help the method evaluate as true rather than false. Try this:
public bool IsConnected()
{
return !(!_socket.Poll(1, SelectMode.SelectRead)
&& _socket.Available == 0);
}
I'm also not sure it's a good idea to make _socket.Available part of this check. According to the documentation, a return value of 0 just means that "no data is queued in the network buffer." That could very easily be true even of a connected socket.

I'm not sure exactly what you're trying to achieve, but assuming you're trying to tell if the connection has been broken, this post may be helpful:
How can I tell if the connection has been broken in my sockets based client?
Edit: A troubleshooting step would be to determine which of the two boolean expressions are returning false, or if they are both returning false.

It looks to me like you may not get the functionality you are expecting from the Socket class. My understanding is that the Socket class is only aware of the connection state as of the last socket operation.
Note that the Poll method has some limitations:
This method cannot detect certain kinds of connection problems, such as a broken network cable, or that the remote host was shut down ungracefully. You must attempt to send or receive data to detect these kinds of errors.
This would imply that in the event of an ungraceful disconnect, it would be normal for a socket to continue to report true until a subsequent socket operation times out (which may explain the several minute delay you experienced in your previous post on this).
This means that if you want to detect ungraceful disconnects, you will likely need to implement an application level heartbeat/ping, as suggested in a previous answer. You may need to play with the interval between the pings, otherwise you lose a degree of fault tolerance and a lag spike may cause unwanted reports of a disconnect.

HttpWebRequest fails after third call

here's my method:
private static void UpdatePref(List<EmailPrefer> prefList)
{
if(prefList.Count > 0)
{
foreach (EmailPref pref in prefList)
{
UpdateEmailRequest updateRequest = new UpdateEmailRequest(pref.ID.ToString(), pref.Email, pref.ListID.ToString());
UpdateEmailResponse updateResponse =(UpdateEmailResponse) updateRequest.SendRequest();
if (updateResponse.Success)
{
Console.WriteLine(String.Format("Update Succsesful. ListID:{0} Email:{2} ID:{1}", pref.ListID, pref.Email, pref.ID));
continue;
}
Console.WriteLine( String.Format("Update Unsuccessful. ListID:{0} Email:{2} ID:{1}\n", pref.ListID, pref.Email, pref.ID));
Console.WriteLine(String.Format("Error:{0}", updateResponse.ErrorMessage));
}
Console.WriteLine("Updates Complete.");
}
Console.WriteLine("Procses ended. No records found to update");
}
the list has around 84 valid records that it's looping through and sending an API request for. But it stops on the 3rd API call and only processes 2 out of the 84 records. When I debug to see what's happening, I only see that it stops here in my SendRequest method without spitting out any error. It's stops at the GetRequestStream and when I step to that and try to keep stepping, it just stops and my application stops running without any error!
HttpWebRequest request = CreateWebRequest(requestURI, data.Length);
request.ContentLength = data.Length;
request.KeepAlive = false;
request.Timeout = 30000;
// Send the Request
requestStream = request.GetRequestStream();
wtf? Eventually if I let it keep running I do get the error "The Operation Has Timed Out". But then why did the first 2 calls go through and this one timed out? I don't get it.
Also, a second question. Is it inefficient to have it create a new object inside my foreach for sending and receiving? But that's how I stubbed out those classes and required that an email, ListID and so forth be a requirement to send that type of API call. I just didn't know if it's fine or not efficient to create a new instance through each iteration in the foreach. Might be common but just felt weird and inefficient to me.

EDIT: It seems you answered your own question already in the comments.
I don't have personal experience with this, but it seems you need to call close on the HTTP web request after you've fetched the response. There's a limit of 2 on the number of open connections and the connection isn't freed until you Close(). See http://blogs.msdn.com/feroze_daud/archive/2004/01/21/61400.aspx, which gives the following code to demonstrate the symptoms you're seeing.
for(int i=0; i < 3; i++) {
HttpWebRequest r = WebRequest.Create(“http://www.microsoft.com“) as HttpWebRequest;
HttpWebResponse w = r.GetResponse() as HttpWebResponse;
}

One possibility for it timing out is that the server you're talking to is throttling you. You might try inserting a delay (a second, maybe?) after each update.
Assuming that UpdateEmailRequest and UpdateEmailResponse are somehow derived from WebRequest and WebResponse respectively, it's not particularly inefficient to create the requests the way you're doing it. That's pretty standard. However, note that WebResponse is IDisposable, meaning that it probably allocates unmanaged resources, and you should dispose of it--either by calling the Dispose method. Something like this:
UpdateEmailResponse updateResponse =(UpdateEmailResponse) updateRequest.SendRequest();
try
{
if (updateResponse.Success)
{
Console.WriteLine(String.Format("Update Succsesful. ListID:{0} Email:{2} ID:{1}", pref.ListID, pref.Email, pref.ID));
continue;
}
Console.WriteLine( String.Format("Update Unsuccessful. ListID:{0} Email:{2} ID:{1}\n", pref.ListID, pref.Email, pref.ID));
Console.WriteLine(String.Format("Error:{0}", updateResponse.ErrorMessage));
}
finally
{
updateResponse.Dispose();
}
I guess it's possible that not disposing of the response objects keeps an open connection to the server, and the server is timing out because you have too many open connections.