I am using a networkstream to pass short strings around the network.
Now, on the receiving side I have encountered an issue:
Normally I would do the reading like this
see if data is available at all
get count of data available
read that many bytes into a buffer
convert buffer content to string.
In code that assumes all offered methods work as probably intended, that would look something like this:
NetworkStream stream = someTcpClient.GetStream();
while(!stream.DataAvailable)
;
byte[] bufferByte;
stream.Read(bufferByte, 0, stream.Lenght);
AsciiEncoding enc = new AsciiEncoding();
string result = enc.GetString(bufferByte);
However, MSDN says that NetworkStream.Length is not really implemented and will always throw an Exception when called.
Since the incoming data are of varying length I cannot hard-code the count of bytes to expect (which would also be a case of the magic-number antipattern).
Question:
If I cannot get an accurate count of the number of bytes available for reading, then how can I read from the stream properly, without risking all sorts of exceptions within NetworkStream.Read?
EDIT:
Although the provided answer leads to a better overall code I still want to share another option that I came across:
TCPClient.Available gives the bytes available to read. I knew there had to be a way to count the bytes in one's own inbox.
There's no guarantee that calls to Read on one side of the connection will match up 1-1 with calls to Write from the other side. If you're dealing with variable length messages, it's up to you to provide the receiving side with this information.
One common way to do this is to first work out the length of the message you're going to send and then send that length information first. On the receiving side, you then obtain the length first and then you know how big a buffer to allocate. You then call Read in a loop until you've read the correct number of bytes. Note that, in your original code, you're currently ignoring the return value from Read, which tells you how many bytes were actually read. In a single call and return, this could be as low as 1, even if you're asking for more than 1 byte.
Another common way is to decide on message "formats" - where e.g. message number 1 is always 32 bytes in length and has X structure, and message number 2 is 51 bytes in length and has Y structure. With this approach, rather than you sending the message length before sending the message, you send the format information instead - first you send "here comes a message of type 1" and then you send the message.
A further common way, if applicable, is to use some form of sentinels - if your messages will never contain, say, a byte with value 0xff then you scan the received bytes until you've received an 0xff byte, and then everything before that byte was the message you wanted to receive.
But, whatever you want to do, whether its one of the above approaches, or something else, it's up to you to have your sending and receiving sides work together to allow the receiver to discover each message.
I forgot to say but a further way to change everything around is - if you want to exchange messages, and don't want to do any of the above fiddling around, then switch to something that works at a higher level - e.g. WCF, or HTTP, or something else, where those systems already take care of message framing and you can, then, just concentrate on what to do with your messages.
You could use StreamReader to read stream to the end
var streamReader = new StreamReader(someTcpClient.GetStream(), Encoding.ASCII);
string result = streamReader.ReadToEnd();
Related
[SOLVED, the question is based on incorrect assumptions]
While working with TCP I came across an issue with NetworkStream.Read returning value 0 in two separate cases, which I have trouble distinguishing.
A bit of background - I have a working client-server solution communicating via TCP using length-prefixed messages. However, since most of the communication (apart from some initial messages exchange) happens from Client to Server, the Server doesn't have a good way to know whether a Client is still connected or not. One way to find this out is to send something to the Client from time to time, and that is what I decided to do.
I know that I can add a dedicated "ping" message to my protocol, and simply ignore it in the Client, but I was also testing for other possibilities. One thing I tried was to send an empty byte array to the Client like so:
networkStream.Write(new byte[0], 0, 0);
All looks good, it seems to be sending a TCP packet with no data in it... however! My client code does expect data from the Server from time to time, so it has a thread that blocks on networkStream.Read, like so:
int bytesRead = networkStream.Read(buffer, 0, 4);
if (bytesRead == 0)
break;
According to the docs, Socket.Read (or NetworkStream.Read) returns 0 if the other end closes the connection. This is true, but in my case, after sending the empty byte array, Read(...) also returns 0.
So far I was unable to distinguish those two situations. Socket.Connected checked after Read is true in both cases (connection closed and the empty byte array). Is there any other way to handle this?
Again, I do know that sending this empty array is almost the same as adding a new type of message for that purpose. I am not asking for a solution here... just wanted to know if .NET's Socket can distinguish between an empty byte array and connection closing.
EDIT:
I am sorry for bothering everyone with a question, that in the end was based on incorrect assumptions. My tests were not done on my production code, and were too sloppy. This caused me to draw incorrect conclusions.
Basically, what I was testing is if I do a Write(new byte[0]...) on one end, the other end's Read(...) will return 0. It did, but not due to the send. The TcpClient I used to test was falling out of scope, which (I assume) caused it to be Disposed by GC, thus the connection got closed, which caused the Read to return 0. I did repeat the test with TcpClient not being disposed/lost, and the Read does not return anything, no matter how many empty byte arrays I send.
At first, I expected the Nagle's algorithm to mess things up, but it doesn't in this case - 1-byte arrays arrive without delays, as I was testing on localhost. I may do a different test, using Sockets, and explicitly disabling Nagle's algorithm, but I don't think this will change anything.
Now I just need to check whether sending such an array will actually allow me to detect a disconnection, but that is a different story, not in the scope of this question.
EDIT 2:
I did some more tests regarding this, and found out, that despite several suggestions, e.g. here (which seems like a valid source of information), doing an empty Send does not recognize a broken connection. I physically disconnected a network cable, and my Server is doing those empty sends every 5 seconds. It's been going like that for several minutes, and no disconnection was detected. If I decide to send any data (even a single byte), the disconnection gets detected after 20 seconds at most.
From MSDN's NetworkStream.Read Method (Byte[], Int32, Int32):
The Read operation reads as much data as is available, up to the
number of bytes specified by the size parameter. If no data is
available for reading, the Read method returns 0.
While sending data, you're sending an empty byte array and writing zero bytes using:
networkStream.Write(new byte[0], 0, 0);
Whereas, while reading the data, you claim that
"My client code does expect data from the Server from time to time, so
it has a thread that blocks on networkStream."
int bytesRead = networkStream.Read(buffer, 0, 4);
if (bytesRead == 0)
break;
But, then again you're trying to read 4 bytes to a byte array, which would obviously keep on waiting. So, what else do you expect!
"...just wanted to know if .NET's Socket can distinguish between an empty
byte array and connection closing."
Connection closing is totally a different story, which involves several steps before closing of a Socket connection. So, obviously, it is not the same as sending or receiving zero bytes!
-> Lastly, as hinted by #WithMetta in the comments, please check whether the data is available to be read or not using NetworkStream.DataAvailable Property.
while(networkStream.DataAvailable) { // your code logic}
I want to send JSON strings back and forth over a socket connection in c# (xamarin).
I want to know, how does the receiver know how many bytes to read from the socket in order to receive the complete JSON string because the string will vary in size.
Do I have to send a length first in binary (maybe one or two bytes), then the JSON string? What is the standard way to do it so that the receiver knows how many bytes to read from the socket each time it get a complete JSON string.
It has to know how many bytes per string because each string is a separate packet, and if many packets are send back to back, if the length of each string is not known exactly, it will read past the end of one string and into the beginning of another, or not read the whole string, either way it will crash while decoding the malformed string.
Another problem, if I send the length first in binary, then if anything should happen where the receiver gets out of sync with the sender, then it wont know which byte is the length anymore because it cant tell where the strings start, and which incoming data represents the length, it will just receive a bunch of bytes and it wont know where is the start from where is the end etc.
Anybody knows the proper way to do it without writing a megabyte of code?
Thanks
If it's a string based message(as you mentioned JSON), you can use a StringBuilder to concat each packet you received, and check at every receive step for an End of File tag(which is defined by yourself, e.g. <EOF>).
Here is an example on MSDN
Client and Server implementations: Client sends messages ending with <EOF> tag and server checks for it to make sure each message is completed.
I'm encountering a pretty difficult problem in an asynchronous RSS feed aggregator application I am creating.
The system is built from a communicating client and server. The server collects RSS feed items from different sources and then distributes them to clients based on their subscriptions.
The issue that I am having is - My specification states that I must implement a pre-defined byte based protocol to communicate between the client and server. Part of this protocol is that when the server sends a payload to the client, that payload must have a checksum created and sent with it, then when it is received by the client the checksum is equated based on the received payload. We then compare the two checksums and if they do not match then we have an issue.
The thing that is baffling me is that, sometimes the checksums will match perfectly and other times they will not, using the same algorithm for all sending and receiving operations.
Here is the method that checks incoming ->
private bool CheckChecksum(uint receivedChecksum, IEnumerable<byte> bytes) {
uint ourChecksum = 0;
foreach (var b in bytes)
ourChecksum += b;
if (ourChecksum != receivedChecksum) {
Debug.WriteLine("received {0}, calculated {1}", receivedChecksum, ourChecksum);
_writeOutToFile = true;
}
return receivedChecksum == ourChecksum;
}
and when calculating it before sending ->
uint checksum = payloadBytes.Aggregate<byte, uint>(0,
(currentChecksum, currentByte) => currentChecksum + currentByte);
Since the behaviour seems to occur on updates that have very large checksums (generally 2 million+) then the only thing I can think that would be causing it is these large byte[] sizes.
As you can see above, I write out to a file the contents of the payload when the checksums do not match. What I found was that the byte[] just ends early (despite the fact that the reading/writing lengths match on both the client and server). The end of the byte[] is just filled with empty spaces.
I am using NetworkStream.ReadAsync and WriteAsync to do all of my I/O operations. I thought that it may be a fault with the way I am using or understanding these.
I realise that this is a very difficult and vague problem (because I'm not sure what is going wrong myself) but if you need any further information I will provide it.
Here is some extra information:
The checksums are of type uint, and their endianness is encoded correctly at both ends of the system.
Payloads are strings which are encoded into ASCII bytes and decoded on the client-side.
All messages are sent with a checksum and a payload length. The payload length represents how many bytes to read off the stream. Since the I read payload length amount of bytes from the networkstream, when the checksums do not match the payload becomes white space after a varying length (but is correct before this point).
Sometimes (rarely) the checksums will match even when they are large.
I am running both the client and server locally on one machine.
I have tried having a Thread.Sleep(5000) before the read and this makes no difference.
Here is a sample of sent and received data that fails.
Sent from server - http://pastebin.com/jvbCbQmJ
Received by client - http://pastebin.com/eNkWymwi
Your receiving code tries to read before all the bytes have been posted.
When you try to send huge chunks of data on only one package, your readAsync will detect there's something to read, try to read for full lenght, which may not be available, and fill the part that was not posted yet with 0s
You can either divide your message on the server, read by parts on the client, or try to read what's available until you have received everything or some time has passed
I want to use the stream class to read/write data to/from a serial port. I use the BaseStream to get the stream (link below) but the Length property doesn't work. Does anyone know how can I read the full buffer without knowing how many bytes there are?
http://msdn.microsoft.com/en-us/library/system.io.ports.serialport.basestream.aspx
You can't. That is, you can't guarantee that you've received everything if all you have is the BaseStream.
There are two ways you can know if you've received everything:
Send a length word as the first 2 or 4 bytes of the packet. That says how many bytes will follow. Your reader then reads that length word, reads that many bytes, and knows it's done.
Agree on a record separator. That works great for text. For example you might decide that a null byte or a end-of-line character signals the end of the data. This is somewhat more difficult to do with arbitrary binary data, but possible. See comment.
Or, depending on your application, you can do some kind of timing. That is, if you haven't received anything new for X number of seconds (or milliseconds?), you assume that you've received everything. That has the obvious drawback of not working well if the sender is especially slow.
Maybe you can try SerialPort.BytesToRead property.
Whats the best way to format a message to a server, at moment I'm serilising an object using the binaryformatter and then sending it to the server.
At the server end its listening in an async fashion and then when the buffer size recieved is not 100% it assumes that the transfer has complete.
This is working and the moment, and I can deserialise the object at the other end, I'm just concerned that if I start sending async this method will fail has message's could be blurred.
I know that I need to mark the message somehow as to say that's the end of message one, this other bit belongs to message 2, but I'm unsure of the correct way to do this.
Could anyone point me in the right direction and maybe give me some examples?
Thanks
You could always serialize it to a memory stream, see how big it is, send the length as a 4-byte binary number then send the stream's contents.
On the other side you can just sit and wait for 4 bytes, combine them into an integer then sit and wait for that number of bytes.
When your read function returns (make them blocking reads), you know you have the entire object into a buffer, so you just deserialize it and cast it to your common interface type.
edit: this is the answer to your specific question. This being said you would be better off using a higher-level library than pure tcp.
If your object has a fixed length you can received specified amount of bytes on the other end and then create your object.
Otherwise you can send delimiters (symbol or sequence of symbols that you are not using in your object) between your objects and keep reading received data byte by byte until you see the delimiter.
You might want to take a look at Protocol Buffers (http://code.google.com/p/protobuf), a (implementation language independent) data interchange format/framework. There exist at least two .NET implementations for it (see http://code.google.com/p/protobuf/wiki/ThirdPartyAddOns).