So I've been getting this exception for about a week now, and I've finally managed to corner it into a code snippet that can be easily read.
As a background, I am programming an app for Windows RT and I am trying to use basic sockets.
For the sake of testing, I've created a local socket listener to act as a server. Both the server and client need to be able to read/write on the socket.
Neither the client nor the server can (or should) know how much data will come across the wire (if any). This is an absolute requirement. The server should be able to process an arbitrary amount of data on demand.
Here is an example. It is presented as a Unit Test, simply because that is where I consistently encounter the error. Removing any single line from this example causes the error to go away:
[TestMethod]
public async Task TestSomething()
{
// Setup local server
//
StreamSocketListener listener = new StreamSocketListener();
listener.ConnectionReceived += async (sender, args) =>
{
DataReader serverReader = new DataReader(args.Socket.InputStream);
await serverReader.LoadAsync(4096); // <-- Exception on this line
};
await listener.BindServiceNameAsync("10181");
// Setup client
//
using (StreamSocket socket = new StreamSocket())
{
await socket.ConnectAsync(new HostName("localhost"), "10181");
DataReader reader = new DataReader(socket.InputStream);
Task readTask = Listen(reader);
}
}
public async Task Listen(DataReader reader)
{
await reader.LoadAsync(4096);
}
The exception happens on the line where the server calls LoadAsync(...), and the exception is thrown when the unit test quits.
The exception is (seemingly) simple:
An existing connection was forcibly closed by the remote host.
(Exception from HRESULT: 0x80072746)
Any clues would be greatly appreciated.
With the new WinRT socket types, it's easier than ever to program sockets correctly, but make no mistake: they are still complex beasts.
The "forcibly closed" (WSAECONNRESET / 10054) error is when the remote side (in this case, the client) aborts its connection, which it does by disposing its StreamSocket. This is reported as an error but is not uncommon and should be handled gracefully. I.e., if the server has sent all its data and is just waiting to receive more (optional) data, then it should treat WSAECONNRESET as a regular close.
Tip: If you pass Exception.HResult to SocketError.GetStatus, you should see it's SocketErrorStatus.ConnectionResetByPeer. That way you can avoid magic values in your error handling code.
P.S. I have a blog post describing some of the more common socket errors and socket error handling in general.
Related
I am testing how .NET WebSockets work when the client can't process data from the server side fast enough. For this purpose, I wrote an application that sends data continuously to a WebSocket, but includes an artificial delay in the receive loop. As expected, once the TCP window and other buffers fill, the SendAsync calls start to take long to return. But after a few minutes, one of these exceptions is thrown by SendAsync:
System.Net.HttpListenerException: The device does not recognize the command
System.Net.HttpListenerException: The I/O operation has been aborted because of either a thread exit or an application request.
What's weird is, that this only happens with certain message sizes and certain timing. When the client is allowed to read all data unrestricted, the connection is stable. Also, when the client is blocked completely and does not read at all, the connection stays open.
Examining the data flow through Wireshark revealed that it is the server that is resetting the TCP connection while the client's TCP window is exhausted.
I tried to follow this answer (.NET WebSockets forcibly closed despite keep-alive and activity on the connection) without success. Tweaking the WebSocket keep alive interval has no effect. Also, I know that the final application needs to be able to handle unexpected disconnections gracefully, but I do not want them to occur if they can be avoided.
Did anybody encounter this? Is there some timeout tweaking that I can do? Running this should produce the error between a minute and half to three minutes:
class Program
{
static void Main(string[] args)
{
System.Net.ServicePointManager.MaxServicePointIdleTime = Int32.MaxValue; // has no effect
HttpListener httpListener = new HttpListener();
httpListener.Prefixes.Add("http://*/ws/");
Listen(httpListener);
Thread.Sleep(500);
Receive("ws://localhost/ws/");
Console.WriteLine("running...");
Console.ReadKey();
}
private static async void Listen(HttpListener listener)
{
listener.Start();
while (true)
{
HttpListenerContext ctx = await listener.GetContextAsync();
if (!ctx.Request.IsWebSocketRequest)
{
ctx.Response.StatusCode = (int)HttpStatusCode.NotImplemented;
ctx.Response.Close();
return;
}
Send(ctx);
}
}
private static async void Send(HttpListenerContext ctx)
{
TimeSpan keepAliveInterval = TimeSpan.FromSeconds(5); // tweaking has no effect
HttpListenerWebSocketContext wsCtx = await ctx.AcceptWebSocketAsync(null, keepAliveInterval);
WebSocket webSocket = wsCtx.WebSocket;
byte[] buffer = new byte[100];
while (true)
{
await webSocket.SendAsync(new ArraySegment<byte>(buffer), WebSocketMessageType.Binary, true, CancellationToken.None);
}
}
private static async void Receive(string serverAddress)
{
ClientWebSocket webSocket = new ClientWebSocket();
webSocket.Options.KeepAliveInterval = TimeSpan.FromSeconds(5); // tweaking has no effect
await webSocket.ConnectAsync(new Uri(serverAddress), CancellationToken.None);
byte[] receiveBuffer = new byte[10000];
while (true)
{
await Task.Delay(10); // simulate a slow client
var message = await webSocket.ReceiveAsync(new ArraySegment<byte>(receiveBuffer), CancellationToken.None);
if (message.CloseStatus.HasValue)
break;
}
}
I'm not a .NET developer but as far as I have seen these kind of problems in websocket topic and in my own opinion, these can be the reasons:
Very short timeout setting on websocket on both sides.
Client/Server side runtime exceptions (beside of logging, must check onError and onClose methods to see why)
Internet or connection failures. Websocket sometimes goes into IDLE mode too. You have to implement a heartbeat system on websockets to keep them alive. Use ping and pong packets.
check maximum binary or text message size on server side. Also set some buffers to avoid failure when message is too big.
As you said your error usually happens within a certain time, 1 and 2 must help you. Again sorry if I cant provide you codes, but I have had same problems in java and I found out these are the settings that must be set in order to work with websockets. Search how to set these in your client and server implementations and you must be fine after that.
Apparently, I was hitting an HTTP.SYS low speed connection attack countermeasure, as roughly described in KB 3137046 (https://support.microsoft.com/en-us/help/3137046/http-sys-forcibly-disconnects-http-bindings-for-wcf-self-hosted-servic):
By default, Http.sys considers any speed rate of less than 150 bytes per second as a potential low speed connection attack, and it drops the TCP connection to release the resource.
When HTTP.SYS does that, there is a trace entry in the log at %windir%\System32\LogFiles\HTTPERR
Switching it off was simple from code:
httpListener.TimeoutManager.MinSendBytesPerSecond = UInt32.MaxValue;
Background: External factors force me to implement a simple websocket client in C# and I would like to keep the number of third party dependencies low. So it would be nice to stick with System.Net.WebSockets if possible.
Objective: Send a series of JSON objects as individual messages to a websocket server. Throw an exception if sending fails with any reasonable timeout. Throw an exception if the underlying TCP connection breaks between new objects becoming available.
Sending code:
// ws is an instance of System.Net.WebSockets.ClientWebSocket
ArraySegment<byte> json = new ArraySegment<byte>
Encoding.UTF8.GetBytes(JsonConvert.SerializeObject(data)));
if (!ws.SendAsync(json, WebSocketMessageType.Binary, true,
CancellationToken.None).Wait(10000))
{
throw new Exception("Error: Websocket send timeout");
}
Console.Write("Sent")
My problem with this code is that when I see the client program printing "Sent", the server will not receive that message until several more invocations of SendAsync. My impression is that it is stuck in some kind of internal message buffer until the buffer spills over.
Similarly, breaking the underlying TCP connection does not result in the next SendAsync throwing an exception until several messages later.
Checking if websocket is alive without sending:
The keepalive is set to the default value 30s. Yet I have never seen ws.State return anything but WebSocketState.Open, even minutes after disconnecting the Server if I don't have any messages to send in between.
Am I fundamentally confused about how ClientWebSocket is supposed to be used, does it simply not work, or did I miss some detail in the documentation?
You should await your async method in a Task.Run :
Task.Run(() => {
if (!await ws.SendAsync(json, WebSocketMessageType.Binary, true,
CancellationToken.None))
{
throw new Exception("Error: Websocket send timeout");
}
Console.Write("Sent");
} ).Wait(10000);
I've tried checking the server:port with telnet and I'm getting the expected results. So either writer.Write() or reader.ReadLine() isn't working cause I get nothing from the server.
TcpClient socket = new TcpClient(hostname, port);
if (!socket.Connected) {
Console.WriteLine("Failed to connect!");
return;
}
TextReader reader = new StreamReader(socket.GetStream());
TextWriter writer = new StreamWriter(socket.GetStream());
writer.Write("PING");
writer.Flush();
String line = null;
while ((line = reader.ReadLine()) != null) {
Console.WriteLine(line);
}
Console.WriteLine("done");
EDIT: I might have found the issue. This code was based off examples I found on the web. I tried another irc server: open.ircnet.net:6669 and I got a response:
:openirc.snt.utwente.nl 020 * :Please wait while we process your connection.
It seems as if I probably need to run the reader in a Thread so it can just constantly wait for a response. However it does seem weird that the program got caught on the while loop without ever printing done to the console.
I think you need to provide further details. I'm just going to assume that because you can easily telnet to the server using the same port your problem lies in the evaluation of the Connected property...
if (!socket.Connected) {
Console.WriteLine("Failed to connect!");
return;
}
this is wrong because Microsoft clearly specifies in the documentation that the Connected property is not reliable
Because the Connected property only reflects the state of the connection as of the most recent operation, you should attempt to send or receive a message to determine the current state. After the message send fails, this property no longer returns true. Note that this behavior is by design. You cannot reliably test the state of the connection because, in the time between the test and a send/receive, the connection could have been lost. Your code should assume the socket is connected, and gracefully handle failed transmissions.
That said, you should not use this property to determine the state of the connection. Needless to say that using this property to control the flow of your console app will result in unexpected results.
Suggestion
Remove the evaluation of the Connected property
Wrap your GetStream and Write method calls in a try/catch block to handle network communication errors
reader.ReadLine() will just wait for any data to arrive. If no data arrive, it seems to hang. That's a feature of tcp (I don't like it either). You need to find out how the end of the message is defined and stop based on that end criterion. Be careful, the end of message identifier may be split into two or more lines...
RFC for ping says that the server may not respond to it & such connections has to be closed after a time. Please check the RFC: https://www.rfc-editor.org/rfc/rfc1459#section-4.6.2
I am new to both C# and to client-server programming. Right now, for class, I'm attempting to create an FTP client without using any pre-established FTP libraries. I feel like I have the project down for the most part, however I'm running into a problem when I make more than one call that requires use of the data port (list, retr, etc.) Here is a sample of the code that is breaking:
writer.WriteLine(portcmd);
writer.Flush();
GetServerMessage(stream);
writer.WriteLine("list ");
writer.Flush();
tmpserver = new TcpListener(IPAddress.Any, 3128);
tmpserver.Start();
tmpclient = tmpserver.AcceptTcpClient();
Console.WriteLine("gothere");
if (!tmpclient.Connected)
{
tmpserver.Start();
}
StreamReader tmpreader = new StreamReader(tmpclient.GetStream());
GetServerMessage(stream);
while (tmpreader.Peek() != -1)
{
Console.WriteLine(tmpreader.ReadLine());
}
tmpclient.Close();
tmpserver.Stop();
GetServerMessage(stream);
Getservermessage is a method that takes a network stream and prints out everything available within a .5 second timeout, stream is the NetworkStream for the current connection to the FTP server, and writer is that same network stream wrapped in a StreamReader for ease of writing ASCII characters to the server. In case you are wondering why I use a stream reader to read from the data connection, it is because the server closes the connection after it transmits the data so I could easily get an eof notification. My GetServerMessage method was for some reason breaking when I used the closed network stream.
This code is sending the port command to the FTP server to inform it that I will be requiring a data connection (first 2 lines) Then sending the list command, establishing the data connection to the server, getting the desired information, and then terminating the data connection (the rest of the code).
This code will execute without flaw the first time I run it but if I try it again, it hangs on the 'tmpclient = tmpserver.AcceptTcpClient();' line. It never reaches the "gothere" print statement. I believe this is because I am receiving the client from the same machine on the same port but I'm not sure. I tried adding a Boolean value to make sure the AcceptTcpClient() only ran once but then I got a runtime error and visual studio informed me that I may have 'released resources before I was done with them' I predicted this would be a problem but how can I tell if the server reestablishes the connection after it has closed it once?
At the end of the given code I stop tmpserver and close tmpclient. I originally did this because I knew the FTP server would close the connection when it was finished transmitting and thought it was the proper thing to do. I find if I comment out these lines, the code will execute more than once but the streams appear to be empty... I'm not sure if this information is helpful but I figured I'd mention it.
I apologize if I am unclear at all but my lack of knowledge with the subject makes it difficult to articulate my problem. If there is any confusion over what the problem is I'd be happy to attempt to clear it up.
To be able to accept another client you should execute tmpclient = tmpserver.AcceptTcpClient(); and waiting for the first client to finish its works(before accepting second client) may not be a good idea
Here is a sample server code that waits for the connections and echoes strings sent from each client. You can test it with telnet localhost 3128
Thread t = new Thread(Server);
t.IsBackground = true;
t.Start();
-
void Server()
{
TcpListener listener = new TcpListener(IPAddress.Any, 3128);
listener.Start();
while (true)
{
var client = listener.AcceptTcpClient();
new Thread(() =>
{
using (client)
{
var reader = new StreamReader(client.GetStream());
var writer = new StreamWriter(client.GetStream());
while (true)
{
string line = reader.ReadLine();
if (line == "QUIT") break;
writer.WriteLine("From Thread[" + Thread.CurrentThread.ManagedThreadId + "] > " + line);
writer.Flush();
}
}
}).Start();
}
}
OK, it's like this. To do a server in a simple manner, you need to thread off the code that handles the client socket. When the accept returns, create and start a thread, passing it the 'tmpclient' and then loop around to the accept call again so that any new client can connnct up. In the newly-spawned server<> client thread, read and write to the passed socket in a loop to communicate with the client.
Once you close your tcp client stream, you can no longer read from the stream you pulled from it.
var stream = tcpClient.GetStream();
...
tcpclient.Close();
...
stream.Read .. fail
The client would have to request another connection,
OR
You should keep your tcp client sockets open.
More complex servers will keep some metadata (state) cached about the client so when sockets unexpectedly close - and the client quickly tries to reconnect, the server can continue processing the smoothly.
I'm trying to reuse a socket in an asynchronous HTTP client, but I'm not able to connect to the host the second time around. I basically treat my asynchronous HTTP client as a state machine with the following states:
Available: the socket is available for use
Connecting: the socket is connecting to the endpoint
Sending: the socket is sending data to the endpoint
Receiving: the socket is receiving data from the endpoint
Failed: there was a socket failure
Clean Up: cleaning up the socket state
In the connecting state I call BeginConnect:
private void BeginConnect()
{
lock (_sync) // re-entrant lock
{
IPAddress[] addersses = Dns.GetHostEntry(_asyncTask.Host).AddressList;
// Connect to any available address
IAsyncResult result = _reusableSocket.BeginConnect(addersses, _asyncTask.Port, new AsyncCallback(ConnectCallback), null);
}
}
The callback method changes the state to Sending once a successful connection has been established:
private void ConnectCallback(IAsyncResult result)
{
lock (_sync) // re-entrant lock
{
try
{
_reusableSocket.EndConnect(result);
ChangeState(EClientState.Sending);
}
catch (SocketException e)
{
Console.WriteLine("Can't connect to: " + _asyncTask.Host);
Console.WriteLine("SocketException: {0} Error Code: {1}", e.Message, e.NativeErrorCode);
ThreadPool.QueueUserWorkItem(o =>
{
// An attempt was made to get the page so perform a callback
ChangeState(EClientState.Failed);
});
}
}
}
In the cleanup I Shutdown the socket and Disconnect with a reuse flag:
private void CleanUp()
{
lock (_sync) // re-entrant lock
{
// Perform cleanup
if (_reusableSocket.Connected)
{
_reusableSocket.Shutdown(SocketShutdown.Both);
_reusableSocket.Disconnect(true);
}
ChangeState(EClientState.Available);
}
}
Subsequent calls to BeginConnect result in a timeout and an exception:
SocketException: A connection attempt
failed because the connected party did
not properly respond after a period of
time, or established connection failed
because connected host has failed to
respond XX.XXX.XX.XX:80
Error Code: 10060
Here is the state trace:
Initializing...
Change State: Connecting
Change State: Sending
Change State: Receiving
Change State: CleanUp
Callback: Received data from client 0 // <--- Received the first data
Change State: Available
Change State: Connecting // <--- Timeout when I try to reuse the socket to connect to a different endpoint
What do I have to do to be able to reuse the socket to connect to a different host?
Note: I have not tried to re-connect to the same host, but I assume the same thing happens (i.e. fails to connect).
Update
I found the following note in the documentation of BeginConnect:
If this socket has previously been disconnected, then BeginConnect must be called on a thread that will not exit until the operation is complete. This is a limitation of the underlying provider. Also the EndPoint that is used must be different.
I'm starting to wonder if my issue has something to do with that... I am connecting to a different EndPoint, but what do they mean that the thread from which we call BeginConnect must not exit until the operation is complete?
Update 2.0:
I asked a related question and I tried using the "Async family" calls instead of the "Begin family" calls, but I get the same problem!!!
I commented on this question: what is benefit from socket reuse in C# about socket reuse using Disconnect(true)/DisconnectEx() and this may help you.
Personally I think it's an optimisation too far in client code.
Re update 1 to your question; no, you'd get an AbortedOperation exception if that were the case (see here: VB.NET 3.5 SocketException on deployment but not on development machine) and the docs are wrong if you're running on Vista or later as it doesn't enforce the "thread must exist until after overlapped I/O completes" rule that previous operating systems enforce.
As I've already said in the reply to the linked question; there's very little point in using this functionality for outbound connection establishment. It's likely that it was originally added to the Winsock API to support socket reuse for AcceptEx() on inbound connections, where, on a very busy web server that was using TransmitFile() to send files to clients (which is where disconnect for reused seems to have originated). The docs state that it doesn't play well with TIME_WAIT and so using it for connections where you initiate the active close (and thus put the socket into TIME_WAIT, see here) doesn't really make sense.
Can you explain why you think this micro optimisation is actually necessary in your case?
have you checked the MaxConnections Setting?
http://msdn.microsoft.com/de-de/library/system.servicemodel.nettcpbinding.maxconnections.aspx