I've written a relatively basic asynchronous server which boots up a task to accept clients and each client then boots up a task to accept incoming packets, the code for which is as follows:
MessageListeningTask = new Task(async () => {
while (true) {
byte[] buffer = new byte[256];
try {
await tcpClient.GetStream().ReadAsync(buffer, 0, buffer.Length);
} catch {
break;
}
string data = Encoding.UTF8.GetString(buffer).Trim('\0', '\n', '\r', '\t', ' ');
OnMessageReceived(data);
}
});
This seems to work pretty well for most things, and after routing it through a class that splits the tokens based on a token at the start, is quite an effective listener.
Except, given my naivety to the topic, I seem to have done something stupidly somewhere in my implementation, and checking with this tool: http://www.yougetsignal.com/tools/open-ports/ seems to break this loop and cause it to trigger OnMessageReceived constantly with no data.
I'm not entirely sure what procedures to take to help diagnose this issue, and figure it's probably something to do with how the information stream operates, so I was hoping someone with experience in the topic could help me solve my issue but also explain what is causing it. If it's relevant, it's running under Mono on Ubuntu, but it usually runs perfectly so I can't see this being the issue.
I am happy to provide any additional information, or to check anything.
Thanks!
From the documentation of ReadAsync:
https://msdn.microsoft.com/en-us/library/hh137813(v=vs.110).aspx
"Return Value
Type: System.Threading.Tasks.Task
A task that represents the asynchronous read operation. The value of the TResult parameter contains the total number of bytes read into the buffer. The result value can be less than the number of bytes requested if the number of bytes currently available is less than the requested number, or it can be 0 (zero) if the end of the stream has been reached."
In the loop ReadAsync must be constantly returning 0 because it reached the end of stream.
Related
Need: long-running program with TCP connections
A C# 4.0 (VS1010, XP) program needs to connect to a host using TCP, send and receive bytes, sometimes close the connection properly and reopen it later. Surrounding code is written using Rx.Net Observable style. The volume of data is low but the program should runs continuously (avoid memory leak by taking care of properly disposing resources).
The text below is long because I explain what I searched and found. It now appears to work.
Overall questions are: since Rx is sometime unintuitive, are the solutions good? Will that be reliable (say, may it run for years without trouble)?
Solution so far
Send
The program obtains a NetworkStream like this:
TcpClient tcpClient = new TcpClient();
LingerOption lingerOption = new LingerOption(false, 0); // Make sure that on call to Close(), connection is closed immediately even if some data is pending.
tcpClient.LingerState = lingerOption;
tcpClient.Connect(remoteHostPort);
return tcpClient.GetStream();
Asynchronous sending is easy enough. Rx.Net allows to handle this with much shorter and cleaner code than traditional solutions. I created a dedicated thread with an EventLoopScheduler. The operations needing a send are expressed using IObservable. Using ObserveOn(sendRecvThreadScheduler) guarantee that all send operations are done on that thread.
sendRecvThreadScheduler = new EventLoopScheduler(
ts =>
{
var thread = new System.Threading.Thread(ts) { Name = "my send+receive thread", IsBackground = true };
return thread;
});
// Loop code for sending not shown (too long and off-topic).
So far this is excellent and flawless.
Receive
It seems that to receive data, Rx.Net should also allow shorter and cleaner code that traditional solutions.
After reading several resources (e.g. http://www.introtorx.com/ ) and stackoverflow, it seems that a very simple solution is to bridge the Asynchronous Programming to Rx.Net like in https://stackoverflow.com/a/14464068/1429390 :
public static class Ext
{
public static IObservable<byte[]> ReadObservable(this Stream stream, int bufferSize)
{
// to hold read data
var buffer = new byte[bufferSize];
// Step 1: async signature => observable factory
var asyncRead = Observable.FromAsyncPattern<byte[], int, int, int>(
stream.BeginRead,
stream.EndRead);
return Observable.While(
// while there is data to be read
() => stream.CanRead,
// iteratively invoke the observable factory, which will
// "recreate" it such that it will start from the current
// stream position - hence "0" for offset
Observable.Defer(() => asyncRead(buffer, 0, bufferSize))
.Select(readBytes => buffer.Take(readBytes).ToArray()));
}
}
It mostly works. I can send and receive bytes.
Close time
This is when things start to go wrong.
Sometimes I need to close the stream and keep things clean. Basically this means: stop reading, end the byte-receiving observable, open a new connection with a new one.
For one thing, when connection is forcibly closed by remote host, BeginRead()/EndRead() immediately loop consuming all CPU returning zero bytes. I let higher level code notice this (with a Subscribe() to the ReadObservable in a context where high-level elements are available) and cleanup (including closing and disposing of the stream). This works well, too, and I take care of disposing of the object returned by Subscribe().
someobject.readOneStreamObservableSubscription = myobject.readOneStreamObservable.Subscribe(buf =>
{
if (buf.Length == 0)
{
MyLoggerLog("Read explicitly returned zero bytes. Closing stream.");
this.pscDestroyIfAny();
}
});
Sometimes, I just need to close the stream. But apparently this must cause exceptions to be thrown in the asynchronous read. c# - Proper way to prematurely abort BeginRead and BeginWrite? - Stack Overflow
I added a CancellationToken that causes Observable.While() to end the sequence. This does not help much to avoid these exceptions since BeginRead() can sleep for a long time.
Unhandled exception in the observable caused the program to exit. Searching provided .net - Continue using subscription after exception - Stack Overflow which suggested to add a Catch that resumes the broken Observable with an empty one, effectively.
Code looks like this:
public static IObservable<byte[]> ReadObservable(this Stream stream, int bufferSize, CancellationToken token)
{
// to hold read data
var buffer = new byte[bufferSize];
// Step 1: async signature => observable factory
var asyncRead = Observable.FromAsyncPattern<byte[], int, int, int>(
stream.BeginRead,
stream.EndRead);
return Observable.While(
// while there is data to be read
() =>
{
return (!token.IsCancellationRequested) && stream.CanRead;
},
// iteratively invoke the observable factory, which will
// "recreate" it such that it will start from the current
// stream position - hence "0" for offset
Observable.Defer(() =>
{
if ((!token.IsCancellationRequested) && stream.CanRead)
{
return asyncRead(buffer, 0, bufferSize);
}
else
{
return Observable.Empty<int>();
}
})
.Catch(Observable.Empty<int>()) // When BeginRead() or EndRead() causes an exception, don't choke but just end the Observable.
.Select(readBytes => buffer.Take(readBytes).ToArray()));
}
What now? Question
This appears to work well. Conditions where remote host forcibly closed the connection or is just no longer reachable are detected, causing higher level code to close the connection and retry. So far so good.
I'm unsure if things feel quite right.
For one thing, that line:
.Catch(Observable.Empty<int>()) // When BeginRead() or EndRead() causes an exception, don't choke but just end the Observable.
feels like the bad practice of empty catch block in imperative code. Actual code does log the exception, and higher level code detect the absence of reply and correctly handle, so it should be considered fairly okay (see below)?
.Catch((Func<Exception, IObservable<int>>)(ex =>
{
MyLoggerLogException("On asynchronous read from network.", ex);
return Observable.Empty<int>();
})) // When BeginRead() or EndRead() causes an exception, don't choke but just end the Observable.
Also, this is indeed shorter than most traditional solutions.
Are the solutions correct or did I miss some simpler/cleaner ways?
Are there some dreadful problems that would look obvious to wizards of Reactive Extensions?
Thank you for your attention.
I'm working on an application where a client connects with a TCP connection which then triggers an amount of work that may potentially take a lot of time to complete. This work must be cancelled if the user drops the TCP connection.
Currently, what I'm doing is starting up a timer that periodically checks the networks streams connectivity by doing this:
// stream is a Stream instance
var abort = false;
using (new Timer(x => {
try
{
stream.Write(new byte[0], 0, 0);
}
catch (Exception)
{
abort = true;
}
}, null, 1000, 1000))
{
// Do expensive work here and check abort periodically
}
I would have liked to read the CanWrite, CanRead or Connected but they report the last status of the stream. Is writing zero bytes a reliable way of testing connectivity, or can this itself cause issues? I cannot write or read any real data on the stream since that would mess up the client.
Let's just say that I have known it to work, decades ago, but there is no intrinsic reason why it should. Any of the API layers between you and the TCP stack is entitled to suppress the call to the next layer down, and even if it gets all the way into the stack it will only return an error if:
It checks for network errors before checking for zero length, which is implementation-dependent, and
There already was a network error, caused by some previous operation, or an incoming RST.
If you're expecting it to magically probe the network all the way to the other end, it definitely won't.
I'm writing a message layer for my distributed system. I'm using IOCP, ie the Socket.XXXAsync methods.
Here's something pretty close to what I'm doing (in fact, my receive function is based on his):
http://vadmyst.blogspot.com/2008/05/sample-code-for-tcp-server-using.html
What I've found now is that at the start of the program (two test servers talking to each other) I each time get a number of SAEA objects where the .Buffer is entirely filled with zeroes, yet the .BytesTransferred is the size of the buffer (1024 in my case).
What does this mean? Is there a special condition I need to check for? My system interprets this as an incomplete message and moves on, but I'm wondering if I'm actually missing some data. I was under the impression that if nothing was being received, you'd not get a callback. In any case, I can see in WireShark that there aren't any zero-length packets coming in.
I've found the following when I Googled it, but I'm not sure my problem is the same:
http://social.msdn.microsoft.com/Forums/en-US/ncl/thread/40fe397c-b1da-428e-a355-ee5a6b0b4d2c
http://go4answers.webhost4life.com/Example/socketasynceventargs-buffer-not-ready-121918.aspx
I am sure not what is going on in the linked example. It appears to be using asynchronous sockets in a synchronous way. I cannot see any callbacks or similar in the code. You may need to rethink whether you need synchronous or asynchronous sockets :).
To the problem at hand stems from the possibility that your functions are trying to read/write to the buffer before the network transmit/receive has been completed. Try using the callback functionality included in the async Socket. E.g.
// This goes into your accept function, to begin receiving the data
socketName.BeginReceive(yourbuffer, 0, yourbuffer.Length,
SocketFlags.None, new AsyncCallback(OnRecieveData), socketName);
// In your callback function you know that the socket has finished receiving data
// This callback will fire when the receive is complete.
private void OnRecieveData(IAsyncResult input) {
Socket inSocket = (Socket)input.AsyncState; // This is just a typecast
inSocket.EndReceive(input);
// Pull the data out of the socket as you already have before.
// state.Data.Write ......
}
I am trying to interface an ancient network camera to my computer and I am stuck at a very fundamental problem -- detecting the end of stream.
I am using TcpClient to communicate with the camera and I can actually see it transmitting the command data, no problems here.
List<int> incoming = new List<int>();
TcpClient clientSocket = new TcpClient();
clientSocket.Connect(txtHost.Text, Int32.Parse(txtPort.Text));
NetworkStream serverStream = clientSocket.GetStream();
serverStream.Flush();
byte[] command = System.Text.Encoding.ASCII.GetBytes("i640*480M");
serverStream.Write(command, 0, command.Length);
Reading back the response is where the problem begins though. I initially thought something simple like the following bit of code would have worked:
while (serverStream.DataAvailable)
{
incoming.Add(serverStream.ReadByte());
}
But it didn't, so I had a go another version this time utilising ReadByte(). The description states:
Reads a byte from the stream and
advances the position within the
stream by one byte, or returns -1 if
at the end of the stream.
so I thought I could implement something along the lines of:
Boolean run = true;
int rec;
while (run)
{
rec = serverStream.ReadByte();
if (rec == -1)
{
run = false;
//b = (byte)'X';
}
else
{
incoming.Add(rec);
}
}
Nope, still doesn't work. I can actually see data coming in and after a certain point (which is not always the same, otherwise I could have simply read that many bytes every time) I start getting 0 as the value for the rest of the elements and it doesn't halt until I manually stop the execution. Here's what it looks like:
So my question is, am I missing something fundamental here? How can I detect the end of the stream?
Many thanks,
H.
What you're missing is how you're thinking of a TCP data stream. It is an open connection, like an open phone line - someone on the other end may or may not be talking (DataAvailable), and just because they paused to take a breath (DataAvailable==false) it doesn't mean they're actually DONE with their current statement. A moment later they could start talking again (DataAvailable==true)
You need to have some kind of defined rules for the communication protocol ABOVE TCP, which is really just a transport layer. So for instance perhaps the camera will send you a special character sequence when it's current image transmission is complete, and so you need to examine every character sent and determine if that sequence has been sent to you, and then act appropriately.
Well you can't exactly says EOS on a network communication ( unless the other party drop the connection ) usually the protocol itself contains something to signal that the message is complete ( sometimes a new line, for example ). So you read the stream and feed a buffer, and you extract complete message by applying these strategies.
There is a a single-threaded server using .NET Socket with TCP protocol, and Socket.Pool(), Socket.Select(), Socket.Receive().
To send, I used:
public void SendPacket(int clientid, byte[] packet)
{
clients[clientid].socket.Send(packet);
}
But it was very slow when sending a lot of data to one client (halting the whole main thread), so I replaced it with this:
public void SendPacket(int clientid, byte[] packet)
{
using (SocketAsyncEventArgs e = new SocketAsyncEventArgs())
{
e.SetBuffer(packet, 0, packet.Length);
clients[clientid].socket.SendAsync(e);
}
}
It works fine on Windows with .NET (I don't know if it's perfect), but on Linux with Mono, packets are either dropped or reordered (I don't know). Reverting to slow version with Socket.Send() works on Linux. Source for whole server.
How to write non-blocking SendPacket() function that works on Linux?
I'm going to take a guess that it has to do with your using statement and your SendAsync call. Perhaps e falls out of scope and is being disposed while SendAsync is still processing the buffer. But then this might throw an exception. I am really just taking a guess. Try removing the using statement and see what happens.
I would say by not abusing the async method. YOu will find nowhere a documentation stating that this acutally is forced to maintain order. it queues iem for a scheuler which get distributed to threads, and by ignoring that the oder is not maintained per documentation you open yourself up to implementation details.
The best possibly is to:
Have a queue per socket.
When you write dasta into this queue, and there is no worker thread, start a work item (ThreadPool) to process the thread.
This way you have separate distinct queues that maintain order. Only one thread will ever process one queue / socket.
I got the same problem; Linux and windows react not in the same way with SendAsync. Sometimes linux truncate the data, but there is a workaround. First of all you need to use a queue. Each time you use SendAsync you have to check the callback.
If e.Offset + e.BytesTransferred < e.Buffer.Length, you just have to e.SetBuffer(e.Offset + e.BytesTransferred, e.Buffer.Length - e.BytesTransferred - e.Offset); and call SendAsync again.
I dont know why mono-linux believe it's completed before sending all the data and it's strange but i'm sure he does.
just like #mathieu, 10y later, I can confirm on Unity Mono+Linux complete callback is called without all bytes being sent in some cases. For me it was large packets only.