Please help me to understand grpc client connection error handling.
I've used Micrsoft WCF before. It was easy to try to connect several times and to give up if there is no service accepting incoming connections like below:
// this is pseudo code
int attemptCount = 0;
while (true)
try
{
client.Connect(); // exception is raised if there is not service listening for incoming connection
break;
}
catch (Exception)
{
client.Abort(); // to clear connection faulted state
if (++attempCount == 5)
throw;
Thread.Wait(500); // waiting for service to start
}
5 connection attempts is taken and then client application is terminated if there is no service listening on certain IP:port
This was used when I have client and service started from VisualStudio debug at the same time so sometime client starts first and it has to wait for service to start.
I've tried to do the same using gRPC client but there is no method to reset channel.State from ChannelState.TransientFailure back to working. I know that gRPC makes pauses between connection if something is wrong:
For many non-fatal failures (e.g., TCP connection attempts timing out
because the server is not yet available), the channel may spend
increasingly large amounts of time in this state.
https://grpc.io/grpc-java/javadoc/io/grpc/ConnectivityState.html
I can try to use WaitForStateChangedAsync but how can I configure gRPC client to wait for certain amount of time between reconnection attempts ?
Is there any other way to connect several times and terminate client for gRPC ?
Thank you
gRPC has exponential backoff control on channel reconnection internally.
We do have some backoff parameters, and some of them are configurable by channel arguments.
https://github.com/grpc/grpc/blob/2bd7ad0112f56d2bdbc37d01a431c1e375039f2e/src/core/ext/filters/client_channel/subchannel.cc#L61
But we don't have any parameter to control the max attempt times as far as I know. Please file a feature request on https://github.com/grpc/grpc so that we can follow up.
Related
Existing scenario is explained below.
Our application is running on Client Server architecture; Client is developed with VC++ and Server is developed with C#.
On the Server side there are two exe's running (myServer1.exe -Windows service based, and myServer2.exe -Windows application). myServer2.exe is communicating to myServer1.exe through TCP socket connection.On the Client side, an exe (myApp1.exe -Windows Service based) runs another exe based on user sessions present in the machine (myUser.exe for all user sessions). Every myUser.exe instances are communicating to myApp1.exe through PIPE communication. And myApp1.exe is also communicating to myServer1.exe through another TCP communication.
New scenario.
We are now creating a TCP socket in listening mode in myServer2.exe (Server application -C#). myUser.exe (Client application -VC++) is trying to connect to myServer2.exe through a TCP connection by using CAsyncSocket. But the framework calls (OnConnect, OnReceive and OnClose) are not happening.
Socket creation- Create(0,SOCK_STREAM); // CAyncSocket
Socket connection- Connect("ServerIP", "ServerPort"); // CAsyncSocket
Note: when we move the socket creation and connection functionalities into Windows service based exe (myApp1.exe), the connection works fine, OnConnect OnReceive and OnClose are happening.
Why framework call to OnConnect is not happening in myUser.exe while in myApp1.exe is?
Your OnConnect method is not called because probably you don't have the message loop in myUser.exe while you have it in myApp.exe.
Error code 10035 is WSAEWOULDBLOCK and it' normal for your case, from MSDN:
It is normal for WSAEWOULDBLOCK to be reported as the result from
calling connect on a nonblocking SOCK_STREAM socket, since some time
must elapse for the connection to be established.
So don't worry about it. If you have a message loop, after your Connect call, the OnConnect method will finally be called at a certain time with a successful result or with an error code.
See also codeproject and SO
just searched for a posibble solution to indetify when the client disconnecets.
i found this:
public bool IsConnected( Socket s)
{
try
{
return !(s.Poll(1, SelectMode.SelectRead) &&s.Available == 0);
}
catch (SocketException) { return false; }
}
im using a while loop in my main with thread.sleep(500) and running the Isconnectedmthod it works allright when i run it through the visual studio and when i click stop debugging it actually notify me in the server side program but when i just go to the exe in the bin directory and launch it-it's Indeed notify me for a connection but when i close the program (manually from the 'x' button) or through the task manager theIsConnected method apparently return still true.....
im using a simple tcp connection
client = new TcpClient();
client.Connect("10.0.0.2", 10);
server:
Socket s = tcpClient.Client;
while(true)
{
if (!IsConnected(s))
MessageBox.Show("disconnected");
}
(it's running on a thread btw).
any suggestion guys?
i even tried to close the connection when the client closes:
private void Form1_FormClosing(object sender, FormClosingEventArgs e)
{
client.Close();
s.Close();
Environment.Exit(0);
}
dont know what to do
What you are asking for is not possible. TCP will not report an error on the connection unless an attempt is made to send on the connection. If all your program ever does is receive, it will never notice that the connection no longer exists.
There are some platform-dependent exceptions to this rule, but none involving the simple disappearance of the remote endpoint.
The correct way for a client to disconnect is for it to gracefully close the connection with a "shutdown" operation. In .NET, this means the client code calls Socket.Shutdown(SocketShutdown.Send). The client must then continue to receive until the server calls Socket.Shutdown(SocketShutdown.Both). Note that the shutdown "reason" is generally "send" for the endpoint initiating the closure, and "both" for the endpoint acknowledging and completing the closure.
Each endpoint will detect that the other endpoint has shutdown its end by the completion of a receive operation with 0 as the byte count return value for that operation. Neither endpoint should actually close the socket (i.e. call Socket.Close()) until this two-way graceful closure has completed. I.e. each endpoint has both called Socket.Shutdown() and seen a zero-byte receive operation completion.
The above is how graceful closure works, and it should be the norm for server/client interactions. Of course, things do break. A client could crash, the network might be disconnected, etc. Typically, the right thing to do is to delay recognition of such problems as long as possible; for example, as long as the server and client have no need to actually communicate, then a temporary network outage should not cause an error. Forcing one is pointless in that case.
In other words, don't add code to try to detect a connection failure. For maximum reliability, let the network try to recover on its own.
In some less-common cases, it is desirable to detect connection failures earlier. In these cases, you can enable "keep alive" on the socket (to force data to be sent over the connection, thus detecting interruptions in the connection…see SocketOptionName.KeepAlive) or implement some timeout mechanism (to force the connection to fail if no data is sent after some period of time). I would generally recommend against the use of this kind of technique, but it's a valid approach in some cases.
In short
How to prevent a duplex callback channel to be closed after an idle period?
In detail
I have a mostly working duplex WCF setup over NetTcpBinding i.e. the client can talk to the server and the server can call back to the client.
Furthermore, I have a reliable session such that the client does not lose the connection to the server after the default period of inactivity, achieved with the following configuration on both client and server:
var binding = new NetTcpBinding(SecurityMode.None);
// Need to prevent channel being closed after inactivity
// i.e. need to prevent the exception: This channel can no longer be used to send messages as the output session was auto-closed due to a server-initiated shutdown. Either disable auto-close by setting the DispatchRuntime.AutomaticInputSessionShutdown to false, or consider modifying the shutdown protocol with the remote server.
binding.ReceiveTimeout = TimeSpan.MaxValue;
binding.ReliableSession.Enabled = true;
binding.ReliableSession.InactivityTimeout = TimeSpan.MaxValue;
However, after a period of inactivity of less than half an hour (haven't measured the minimum time exactly), the server is unable to use the callback again - the server just blocks for a minute or so and I do not see any exceptions, while nothing happens on the client side (no evidence of callback).
Leads and root causes?
Note that I can use the callback fine twice in a row consecutively, as long as I do not wait long in between the callback calls.
Are the callbacks configured somewhere else? Do callbacks have their own timeouts etc?
Might it be a blocking/threading issue - need to either set UseSynchronizationContext=false on your client, or avoid blocking while waiting for the message to be received
Should DispatchRuntime.AutomaticInputSessionShutdown be set to false, and if so, how? I'm not really sure how it relates to reliable sessions and I do not know where to access this property
Anything else?
I achieved this by extending the BaseClient class with an automatic keep alive message to be invoked on the target interface when no other calls are made.
These errors are getting more and more frequent on my Game Server. They are causing the server to keep closing and restarting...
System.Net.Sockets.SocketException (0x80004005): An established connection was aborted by the software in your host machine
at System.Net.Sockets.Socket.BeginSend(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags, AsyncCallback callback, Object state)
at iRP.Game.Sessions.Session.SendData(Byte[] Data)
This is the code from which these errors are generated:
public void SendData(byte[] Data)
{
try
{
if (mSocket == null)
{
//Output.WriteLine("[SND] Socket has a null exception, which means it is now invalid. Remove this socket!", OutputLevel.CriticalError);
}
else
{
mSocket.BeginSend(Data, 0, Data.Length, SocketFlags.None, sendCallback, mSocket);
}
}
catch (Exception e)
{
string WhatToWrite = "Error handled (SESSION): " + e.ToString() + "\n\n" + e.Message + "\n\nStack: " + e.StackTrace + Environment.NewLine + "\n\n";
File.AppendAllText(Environment.CurrentDirectory + "\\data\\fatal.txt", WhatToWrite);
Program.Stop();
}
}
The buffer sizes are correctly set, we are using KeepAlive on the socket and were using Send and Receive Timeouts.
People suggested that disabling the firewall would help, but whenever I do this our Game Server (Dedicated Server) restarts itself as if it's under attack, so the firewall must remain enabled.
Anyone else got any other solutions for this?
PS: We are behind DDoS Mitigation Services which may be limiting the number of connections...
An established connection was aborted by the software in your host machine
That is a boiler-plate error message, it comes out of Windows. The underlying error code is WSAECONNABORTED. Which really doesn't mean more than "connection was aborted". You have to be a bit careful about the "your host machine" part of the phrase. In the vast majority of Windows application programs, it is indeed the host that the desktop app is connected to that aborted the connection. Usually a server somewhere else.
The roles are reversed however when you implement your own server. Now you need to read the error message as "aborted by the application at the other end of the wire". Which is of course not uncommon when you implement a server, client programs that use your server are not unlikely to abort a connection for whatever reason. It can mean that a fire-wall or a proxy terminated the connection but that's not very likely since they typically would not allow the connection to be established in the first place.
You don't really know why a connection was aborted unless you have insight what is going on at the other end of the wire. That's of course hard to come by. If your server is reachable through the Internet then don't discount the possibility that you are being probed by a port scanner. Or your customers, looking for a game cheat.
This problem appear if two software use same port for connecting to the server
try to close the port by cmd according to your operating system
then reboot your Android studio or your Eclipse or your Software.
Could be related to the maximum number of concurrent requests. I was able to fix it with two solutions:
Increase the default limit of the max concurrent connections (by default it is set to 2):
ServicePointManager.DefaultConnectionLimit = 25
Wrap sending requests around a buffer: could use ConcurrentQueue to limit the rate, or implement a simple wait as the following:
while (activeRequests >= maxConcurrentRequests)
Thread.Sleep(1000);
Interlocked.Increment(ref activeRequests);
var response = await _client.GetStreamAsync(endpoint);
Interlocked.Decrement(ref activeRequests);
While the answer from Hans is an excellent high level summary of the error that worked great for getting me started, I ended up finding a page that explained the root cause well enough that I was able to recreate it with a python script.
The page presents a couple different descriptions of this error that are more detailed than the "connection was aborted" paraphrasing from Hans's answer but still pretty cryptic and not very informative.
The post then explains this scenario that would produce the error:
An HTTP POST is to be sent to an HTTP server.
The server begins reading the POST and notices that the HTTP request header is invalid.
It immediately sends an HTTP response (with an error status, perhaps status=400) and closes the connection without trying to continue reading the remainder of the HTTP request that is forthcoming.
Meanwhile, the client is still happily writing the remainder of the HTTP request to the socket. (Remember a TCP/IP socket connection needs to be closed from both sides. In this case, the server has closed its side, but the client is still pumping data into the half-open connection.)
The client finishes writing the HTTP POST to the socket — meaning that data has been buffered to Winsock. The client application then tries to read the HTTP response, but it cannot because the outgoing retransmission (of the buffered data by WinSock) failed and the socket connection was shutdown on the client side (by Winsock). Even though the HTTP server sent the response, it is lost and cannot be retrieved. The error your application will receive when trying to read the HTTP response on the socket is WSAECONNABORTED
Wikipedia also has a page to explain what winsock is, but all you really need to know for this scenario is that its the Windows socket API.
I had "ManageEngine Agent" installed on my computer, which was blocking my connection to the databases, so uninstalling it was the solution.
Check which application could block the connection to your DB.
I run my application on a network and in some cases the client lost connection to the server. After this time, when I wanted to send a message to the server I receive the following error: Operation not allowed on non-connected sockets (something like this).
I thought to create an event for object type TcpClient and when tcp_obj.Connected = false to call a function to discontinue execution of the current code. How could I do this?
Or giving me other suggestios.
Thanks.
I know at least from socket programming in Java that when a client loses connection to the server, the server does not and can not know about it. You need a heartbeat of some sort to detect the early disconnection.
We often use a heartbeat in our client/server applications to detect early disconnections and log them on the server. This way the server can close the associated socket and release the connection back to the pool.
Simply send a command to the client periodically and wait for a response. If no response is garnered within a timeout assume disconnect and close streams.
I would simply first check your connection object to ensure you are connected, prior to attempting to send the message. Also make sure that you are putting your send-logic inside of a try-catch, so that if you do happen to get disconnected mid transmission, you'll be able to resume without blowing your application apart.
Psuedo-Code:
private void SendMessage(string message, Socket socket)
{
if(socket.connectionState = States.Connected)
{
try{
// Attempt to Send
}
catch(SocketException Ex)
{
// Disconenct, Additional Cleanup Etc.
}
}
}
If you are in C#, prior to your connection state changing, you will have a socket disconnected event fire, prior to your connection state changing. Make sure you tie this event up as soon as your socket connects.
Can we know why you use TCP sockets? Is for calling a tcp device o server code?
I recommend you if is for calling a .net server app use Windows Communication Foudation. It is simple to expose services by net.tcp, http, etc.
Regards,
Actually this is a very old problem,
If I understand your question correctly you need a way to know whether you're application is still connected to the server or vice versa.
If so then a workaround is to have a UDP connection just to check the connectivity (overhead I know, but its much better then polling on Connected state), you could check just before you send you're data.
Since UDP is not Connection oriented you don't need to be connected when you send the data