I have a client and server, based on TcpListener and TcpClient. The client connects to the server and they exchange some data. Everything works just fine when I run locally.
But when I put the server in a Docker container on Azure Container Services, and connect the client to it, the following happens:
Client connects successfully to server
Client and server perform successful handshake
Data transfer begins
Approximately 20 seconds later (this is supposed to take several minutes) the whole thing blows up. The server reports "connection reset by peer" and the client reports "error reading past the end of the stream."
Each side seems to think the other side is the one with the problem. When I'm running locally, everything works as expected, which leads me to believe that the problem is somewhere in between.
There isn't a fundamental issue with establishing the connection, such as a firewall getting in the way, because I've verified at both ends that they're connecting and performing the handshake. The client is not "slamming the phone down"; it's expecting more data from the server. But "connection reset by peer" means that someone somewhere is intentionally sending a RST packet.
Is there any good way to figure out what's interfering with my data transfer?
For Azure Container Services (be it Azure Container Instances or Azure Kubernetes Service), the major cause for intermittent connection issues is hitting a limit while making new outbound connections. The limits you can hit include:
TCP Connections
SNAT ports
Please see:
Detecting SNAT port exhaustion on Azure Kubernetes Service
Troubleshooting intermittent outbound connection errors in Azure App Service (even if it is about Azure App Service most of them still apply)
More info:
kube-proxy Subtleties: Debugging an Intermittent Connection Reset
Fix a random network Connection Reset issue in Docker/Kubernetes
Related
My system is composed of
SignalR server
Multiple C# clients connecting to said SignalR server.
As I understand, once connected each of these clients would have its own associated ConnectionId.
Now, I want to implement a resiliency strategy where after the SignalR server is restarted, it should still retain the Groups and Connections it used to have in the Hub.
I was thinking of achieving this by storing the Groups and ConnectionIds in an external storage (e.g. database), and restore it when the application starts up.
When the server goes down, the clients' connection might have dropped. But this can be mitigated somewhat by making the client always attempt to reconnect on disconnection. Once the server is up, the client would reconnect.
However, this solution feels rather flaky. In particular, I'm not sure whether once the client reconnects it will retain the same ConnectionId.
Does this approach make sense? Is there a better way to do it?
Yes, client-reconnects ALWAYS happen with the same connectionID. The connectionID is renewed ONLY in case:
the connection.stop() method is called from the client,
the server has detected client disconnection and does a disconnect on the server + sends a disconnect message to clients that happen to reconnect too late. Then those clients will close connection. So client-server are synced again.
If the client is connected to a server that is about to reboot, the clients will notice upon disconnection and try to reconnect to the server with the same connectionID, all within a given timeframe, defined by the connection-timeout. If the server, then, reboots within the connection timeout frame, the client reconnects to the server, with the existing ID.
In this case the Reconnect() event is fired on the server, without the OnConnected() event is happened. This is an exceptional signalr case.
Code your Reconnect() events very defensively.
link to official documentation explaining this issue
chapter: Server disconnection scenarios
Created .net custom tcp server and client using tcplistener. Taks is simple to exchange the data. I have Windows server, on which I am able to communicate. Within network also I am able to communicate. When I deploy this same on our web server, public IP is not accepting connections. At client side it throws time out error:
A connection attempt failed because the connected party did not
properly respond after a period of time, or established connection
failed because connected host has failed to respond .....
Trace route shows request is going there till server and at the end its not showing time out. Server is VM on azure.
Add the endpoint to the Azure configuration:
https://azure.microsoft.com/en-us/documentation/articles/virtual-machines-set-up-endpoints/
Thanks for your help. It was firewall issue. It is resolved but I am clueless about the problem. Firewall ports opened earlier did not worked. so I stopped firewall and I found its working. later on I started and firewall it again worked. strange....fire wall is not updated it seems.
Thanks.
Kishor
I'm trying to configure IIS 7.5 so that it can serve large (400mb) files via HTTP. The client is a C# client using WebClient.DownloadFile() (basically wrapping an HTTP GET request).
However when downloading, I occasionally get the error from the client:
An established connection was aborted by the software in your host machine
Anything I can do in either the client or IIS to prevent this error?
After much digging around, it looks like a dodgy internet connection on my side. All clients on my site disconnect at the same time, but clients on another site stay connected. Unfortunately the internet router that I have here doesnt have any connectivity logs (its a top-of-the-range linksys.. go figure) so it wasnt easy to see what the problem was.
I have a chat site (http://www.pitput.com) that connects user via socket connections.
I have in the client side a flash object that opens a connection to a port in my server.
In the server i have a service that is listening to that port in an async matter.
All is working fine except when i talk to someone after an unknown period of time(about couple of minutes) the server is closing my connection and i get an error in the server :
" A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond".
I dont know how exactly the tcp socket works. does it checking for "live" connection every couple of seconds? how does it decide when to close the connection? Im pretty sure that the close operation is not coming from the client side.
Thanks.
Sounds like the server is handling the connection but not responding. This is the point where I usually pull out WireShark to find out what's going on.
TCP/IP does have an option for checking for live connections; it's called "keepalive." Keepalives are hardly ever used. They're not enabled by default. They can be enabled on a system-wide basis by tweaking the Registry, but IIRC the lowest timeout is 1 hour. They can also be enabled on a single socket (with a timeout in minutes), but you would know if your application does that.
If you are using a web service and your client is connecting to an HTTP/HTTPS port, then it may be getting closed by the HTTP server (which usually close their connections after a couple minutes of idle time). It is also possible that an intermediate router may be closing it on your behalf after an amount of idle time (this is not default behavior, but corporate routers are sometimes configured with such "helpful" settings).
If you are using a Win32 service, then it does in fact sound like the client side is dropping the connection or losing their network (e.g., moving outside the range of a wireless router). In the latter case, it's possible that the client remains oblivious to the fact that the connection has been closed (this situation is called "half-open"); the server sees the close but the client thinks the connection is still there.
Is this an ASP web service hosted with some company? If so, the server generally recycles apps every 10 to 20 minutes. You cannot have a web service running indefinitely, unless it's your own server (I believe).
The TIBCO EMS .NET reference guide says (pg 134)
To enable reconnection behavior and fault tolerance, the serverURL parameter must be a comma-separated list of two or more URLs. In a situation with only one server, you may supply two copies of that server’s URL to enable client reconnection (for example, tcp://localhost:7222,tcp://localhost:7222).
The TIBCO EMS user's guide (pg 292) talks about failover scenarios, client notification, and automatic transfer of clients to the backup server, but nothing specifically "reconnect" related.
In a "reconnect" scenario, does the server handle everything? or does the client have to do something with it's TIBCO.EMS.Connection instances?
Looks like from our testing that the there are settings on both the server and the client that enable this feature. On the client side, the SetReconnAttemptCount, Delay, Timeout govern the attempts the client tries to reconnect once its aware of a server failover / connection failover.
In our testing, we used a single server environment, listed the server twice in the connection string (using the trick you outlined above) and when that server was taken offline, we received a client notification of the failover process taking affect (we enabled Tibems.SetExceptionOnFTSwitch(true)) and when the server was brought back online, our client seemlessly reconnected without missing a beat. We didn't need to code anything, the internal reconnect logic worked its magic.
On the server side, fault tolerance needs to be enabled and I believe server-client and client-server heartbeats need to be enabled (though this has not yet been verified).
Hope this helps.