My system is composed of
SignalR server
Multiple C# clients connecting to said SignalR server.
As I understand, once connected each of these clients would have its own associated ConnectionId.
Now, I want to implement a resiliency strategy where after the SignalR server is restarted, it should still retain the Groups and Connections it used to have in the Hub.
I was thinking of achieving this by storing the Groups and ConnectionIds in an external storage (e.g. database), and restore it when the application starts up.
When the server goes down, the clients' connection might have dropped. But this can be mitigated somewhat by making the client always attempt to reconnect on disconnection. Once the server is up, the client would reconnect.
However, this solution feels rather flaky. In particular, I'm not sure whether once the client reconnects it will retain the same ConnectionId.
Does this approach make sense? Is there a better way to do it?
Yes, client-reconnects ALWAYS happen with the same connectionID. The connectionID is renewed ONLY in case:
the connection.stop() method is called from the client,
the server has detected client disconnection and does a disconnect on the server + sends a disconnect message to clients that happen to reconnect too late. Then those clients will close connection. So client-server are synced again.
If the client is connected to a server that is about to reboot, the clients will notice upon disconnection and try to reconnect to the server with the same connectionID, all within a given timeframe, defined by the connection-timeout. If the server, then, reboots within the connection timeout frame, the client reconnects to the server, with the existing ID.
In this case the Reconnect() event is fired on the server, without the OnConnected() event is happened. This is an exceptional signalr case.
Code your Reconnect() events very defensively.
link to official documentation explaining this issue
chapter: Server disconnection scenarios
Related
I am writing a C# client to connect with an embedded system (server). Initially i am able to connect to the server and send data. Then i reboot the server (with the client not being shutdown) and on the server coming to ready state, I first try to disconnect (shutdown) client and reconnect the same. Now during client shutdown i am getting the socket exception 10053 - An established connection was aborted by the software in your host machine.
Can you help me in understanding what could be the issue?
Note: if i try to reconnect without trying to shutdown (after the server reboot) then the connect is working fine and i am able to transmit data.
The socket probably uses a TCP protocol. TCP is designed to keep a constant line of communication between the two. This means that in order to close the connection, both client and server say to each other that the connection will be ended. But as you only get the error after a restart I believe that when you shutdown the server, the server did not get a chance to properly shut down the connection too. So, when trying to restart the connection on the client, it already 'lost' its connection without knowing it as it didn't hear it from the server. Thus, it cannot officially close the socket as the server does not communicate to the socket anymore.
You have two options: accept the exception and use a try catch, which might be less neat. The other option is to try and force the server to officially close the socket before or during shutdown, so that the client is informed. Then, the client will retry starting a connection every few minutes.
I can't go into specifics as I haven't worked much with the code yet, but I hope this seems clear to you.
I've created a server-client communicate program in .NET (c# or vb.net) using TCPListener - Socket on port 8080. In simple words, the program work like a chat software, client connect to server, and both wait for message from each other and then process it.
To retrieve packet from client, i using are using a "While" method like this :
While true
Dim Buffer(4096) As Byte
s.Receive(Buffer)
Dim strDataReceived As String = System.Text.Encoding.ASCII.GetString(Buffer)
ProcessData(strDataReceived) 'Process data received...........
End while
When testing both server.exe-client.exe in local, the software work fine for several hours without any problem.
But when i start to run the server.exe in my real server, the connection between server-client usually become lost each other when client connected after a few dozen minutes. The symptom is client send packet to server but server does not receive the packet from client when server is still standing in 'sck.receive(Buffer)' command. I have tested many times but i still have no lucky to keep the connection run over 1 hour.
I have investigated about this problem but it still very strange :
The server did not installed any firewall software.
The client did not using any proxy and antivirus, firewall software
I using a bandwidth logging software on server to make sure the internet in my server is stable.
I make a 'ping -t' from my client computer to the server and keep looking on it to sure there are no connection lost between client and server . The ping command indicate that the ping time is usually range from 5ms to 50ms and no connection time out occur.
Even I try to unplug the network cable in the client computer for a few seconds, and then replug again to simulation the disconnect event. I've awesome that my connection between server-client is still maintain and it's not the problem that cause my symptom.
I was thinking to write a code for auto reconnect if received timeout. But it could make my software usually delay when reconnecting if the above symptom still there. I really want to know what wrong with my code and which is the solution for me to fix the above symptom?
Likely the server is behind some sort of firewall (Cisco ASA, etc.) which has idle connection timeouts. When you "punch" through a firewall / NAT device, a "session" is created within the firewall kernel. There is an associated resource that has to be reclaimed, so firewalls do not usually allow unlimited connection timeout, but firewalls do support things like dead connection detection.
Adding a keepalive packet / activity every 5 minutes, or disconnecting / reconnecting is the only way around that. Few network admins are going to change their configs to accomodate this. It is pretty simple to implement a "ping" or "keepalive" command in custom TCP application protocols. Just send the string and consume it, you don't even have to respond to the packet to accomplish resetting the idle timer within the firewall, though that would probably be best practice.
When I say keepalive, I don't mean the TCP keepalive socket option. That is a zero-length packet, and is detectable by a good firewall, like Cisco. Cisco admins can setup rules to quietly deny your keepalive packet, so the solution is to implement it above the TCP layer, in the Application layer, by sending a small string of data like "KEEPALIVE\r\n".
I'm using SignalR (0.5.3) Hubs for a chat app where each keystroke is sent to the server (saved in the DB), relayed to all clients and a return value (a string token of sorts) is sent back from the server.
It works fine, until the app pool recycles, then it stops relaying the keystrokes to all the clients (because the in-memory server state is lost I suppose) and the server doesn't return back any values as well. At this point, I suppose all requests via SignalR are queued by IIS and then processed once the app pool has been recycled.
My question is how can I handle this scenario so that all clients are aware of the server unavailability/delay due to app pool recycle, notify the user to wait for a while and then resume operation on reconnect?
There's two options.
For 0.5.3 you can detect when the client goes into "reconnecting" or is "disconnected" and notify the user that there is server issues. Keep in mind in most situations the client will not actually know that it is disconnected if the server just goes away.
OR
If you wait for the next release (1.0alpha) we will take care of the bulk of this for you. On lack of a server we will trigger an onConnectionSlow event which will then result in the client shifting into the "reconnect" (if it does not receive any info) state until the server comes back online. The client will also know if the server goes away (we're adding this functionality) for edge cases such as an app pool recycle.
Hope this helps!
I have a chat site (http://www.pitput.com) that connects user via socket connections.
I have in the client side a flash object that opens a connection to a port in my server.
In the server i have a service that is listening to that port in an async matter.
All is working fine except when i talk to someone after an unknown period of time(about couple of minutes) the server is closing my connection and i get an error in the server :
" A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond".
I dont know how exactly the tcp socket works. does it checking for "live" connection every couple of seconds? how does it decide when to close the connection? Im pretty sure that the close operation is not coming from the client side.
Thanks.
Sounds like the server is handling the connection but not responding. This is the point where I usually pull out WireShark to find out what's going on.
TCP/IP does have an option for checking for live connections; it's called "keepalive." Keepalives are hardly ever used. They're not enabled by default. They can be enabled on a system-wide basis by tweaking the Registry, but IIRC the lowest timeout is 1 hour. They can also be enabled on a single socket (with a timeout in minutes), but you would know if your application does that.
If you are using a web service and your client is connecting to an HTTP/HTTPS port, then it may be getting closed by the HTTP server (which usually close their connections after a couple minutes of idle time). It is also possible that an intermediate router may be closing it on your behalf after an amount of idle time (this is not default behavior, but corporate routers are sometimes configured with such "helpful" settings).
If you are using a Win32 service, then it does in fact sound like the client side is dropping the connection or losing their network (e.g., moving outside the range of a wireless router). In the latter case, it's possible that the client remains oblivious to the fact that the connection has been closed (this situation is called "half-open"); the server sees the close but the client thinks the connection is still there.
Is this an ASP web service hosted with some company? If so, the server generally recycles apps every 10 to 20 minutes. You cannot have a web service running indefinitely, unless it's your own server (I believe).
The TIBCO EMS .NET reference guide says (pg 134)
To enable reconnection behavior and fault tolerance, the serverURL parameter must be a comma-separated list of two or more URLs. In a situation with only one server, you may supply two copies of that server’s URL to enable client reconnection (for example, tcp://localhost:7222,tcp://localhost:7222).
The TIBCO EMS user's guide (pg 292) talks about failover scenarios, client notification, and automatic transfer of clients to the backup server, but nothing specifically "reconnect" related.
In a "reconnect" scenario, does the server handle everything? or does the client have to do something with it's TIBCO.EMS.Connection instances?
Looks like from our testing that the there are settings on both the server and the client that enable this feature. On the client side, the SetReconnAttemptCount, Delay, Timeout govern the attempts the client tries to reconnect once its aware of a server failover / connection failover.
In our testing, we used a single server environment, listed the server twice in the connection string (using the trick you outlined above) and when that server was taken offline, we received a client notification of the failover process taking affect (we enabled Tibems.SetExceptionOnFTSwitch(true)) and when the server was brought back online, our client seemlessly reconnected without missing a beat. We didn't need to code anything, the internal reconnect logic worked its magic.
On the server side, fault tolerance needs to be enabled and I believe server-client and client-server heartbeats need to be enabled (though this has not yet been verified).
Hope this helps.