I'm trying to understand the use of a buffer in the socket scheme of things. Here is what I understand, and if someone could tell me if I'm correct or give feedback to where I might be misunderstanding that would be great.
Using the Asynchronous Sockets detailed on MSDN example (all below is with reference to that example)
Below describes a state object of which I understand the point of:
// State object for reading client data asynchronously
public class StateObject
{
// Client socket.
public Socket workSocket = null;
// Size of receive buffer.
public const int BufferSize = 1024;
// Receive buffer.
public byte[] buffer = new byte[BufferSize];
// Received data string.
public StringBuilder sb = new StringBuilder();
}
Now it says the buffer is 1024 bytes. When you recieve data, you are only asking for 1024 bytes at a time? And in the async method ReceiveCallback, it only ask for 1024 and then says this may not be all of the data, so now retrieve the next 1024.
So basically, the input stream sent to the socket is read in 1024 byte chunks and you keep reading till you've hit an end point (by you're own definition)?
So basically, the input stream sent to the socket is read in 1024 byte chunks and you keep reading till you've hit an end point (by you're own definition)?
Correct. Since TCP is just a stream of data there is really no way of know the length of each if your messages in the stream (unless you always have a fixed length). hence you need to use some kind of buffer to read from the stream to be able to detect the end of your messages.
Related
I have an exe which simulates a video stream. I connect to it and very occasionally read the expected data, but usually I get only the first 28 bytes and then 65508 bytes of zeros. Assume the video stream is working correctly.
TcpClient tcpClient = new TcpClient ();
int port = 13000;
myIP = IPAddress.Loopback.ToString();
tcpClient.Connect (myIP, port);
NetworkStream netStream = tcpClient.GetStream ();
byte[] bytes = new byte[tcpClient.ReceiveBufferSize];
netStream.Read (bytes, 0, (int)tcpClient.ReceiveBufferSize);
string dataString = Encoding.ASCII.GetString (bytes);
Console.WriteLine("\ndataString: "+dataString.Substring(0,1000));
Console.WriteLine("\nnumber of bytes read: "+bytes.Length);
tcpClient.Close ();
// Closing the tcpClient instance does not close the network stream.
netStream.Close();
How can I make it so that I get the expected output every time?
TCP represents a (bi-directional) stream of data. You're supposed to keep reading from it in a loop, and parsing the data as you need them. It doesn't have a concept of messages - ten writes on one side can result in a single read on the other side just as easily as one write on one side can result in ten reads on the other side.
The contract you have with TCP is as follows:
If there is data in the receive buffer, Read returns immediately, filling the buffer you provided with as much data as is available, up to the length of the buffer. The number of bytes read is the return value of Read.
If there is no data in the receive buffer, Read will block until there's at least a single byte of data. Then it follows as in the first case.
If the other sides shuts down the socket, Read will return zero.
So to get TCP working, you need a loop. How exactly you form the loop depends on what you're trying to do. If you're really working with data that is logically a stream (e.g. audio data), just keep reading as fast as you can and process whatever data you get as it comes in. If you need to send messages, you need to implement a message protocol. If you need a one-off message, you can just keep reading until Read returns zero.
Your case can be handled with the first approach - keep reading until the stream closes, and push the received data forward. Assuming the data is actually a UTF8 stream of text, the basic receiver would look something like this:
using (var client = new TcpClient())
{
tcpClient.Connect(myIP, port);
var stream = client.GetStream();
var buffer = new byte[4096]; // Adapt the size based on what you want to do with the data
var charBuffer = new char[4096];
var decoder = Encoding.UTF8.GetDecoder();
int bytesRead;
while ((bytesRead = stream.Read(buffer, 0, buffer.Length)) != 0)
{
var expectedChars = decoder.GetCharCount(buffer, 0, bytesRead);
if (charBuffer.Length < expectedChars) charBuffer = new char[expectedChars];
var charCount = decoder.GetChars(buffer, 0, bytesRead, charBuffer, 0);
Console.Write(new string(charBuffer, 0, charCount));
}
client.Close();
}
Note that this does no error handling, so don't use it as is in any production code. You would probably also want to use Async methods if you're expecting more than a few simultaneous connections. It's just to illustrate the basic way one would handle a stream of data being received over TCP.
If you want some more insight into dealing with TCP, I have a few very simple examples at https://github.com/Luaancz/Networking. I haven't found any good tutorials or code samples for C#, so if this isn't enough, you'll probably have to dig deeper into the documentation around sockets, TCP and all that.
Or just use an existing networking library, rather than trying to write your own :) TCP is still very low level.
I have a network stream over TCP which is a Zlib stream in both ways (client to server and server to client).
For Zlib, I am using the latest Ionic.Zlib.dll.
For the first couple packets that get sent/received, it works fine, but when receiving the 3rd packet or so, it starts misbehaving.
Every read operation on the ZlibStream blocks until the packet is repeated by the other end.
For example:
// Where reader is a BinaryReader on the ZlibStream on the NetworkStream
int a = reader.ReadInt32(); // blocks until initial packet that is 12 bytes is received
int b = reader.ReadInt32(); // blocks until the packet is repeated
int c = reader.ReadInt32(); // blocks until the packet is repeated again
After that code, the packet has actually been sent 3 times and received only once.
Why is it blocking there? How can I get it to not block and continue as expected?
Could flushing in Zlib have anything to do with it? (The stream is flushed after every packet.)
Zlib, as well as other compression algorithms, operates with blocks of data. It bufferes some data from the source stream and then tries to compress it.
If you put in the ZlibStream data that is less than buffer size, nothing will actually come out of ZlibStream right away. Your data will be sitting there until buffer fills. To get that data compressed, flushing your original stream, and probably even compressed stream, would not be enough.
All your readInts should receive the right data if serve closes the ZlibStream.
If you want to receive packets from server in real-time, try completely compressing the packets first and then send them through the network stream.
First, you say
"Every read operation on the ZlibStream blocks until the packet is
repeated by the other end."
That means that there's a read on your server... E.g. the pattern should be like...
Client: Connects to server, asks for data
Server: Sends packet
Client: Reads packet
Client: Sends same packet back
Server: Reads Packet
Server: Sends new packet?
If that's the case, the server will always block since you're ReadInt32()ing 3 times. Are you positive the other int packets are sent by your server?
Second, I wouldn't use ZlibStream to read directly from a network socket. I'd read in the raw data from the network stream, buffer it, and have another method fire when data was in the buffer that decompressed it...
In psuedocode:
byte[] compressedDataBuffer
bool canReadFromBuffer = false;
ASYNC:
while(true):
get current network stream, see if there's data
if it has data:
add networkStream's current data to compressedDataBuffer
lock:
if compressedDataBuffer has new data/has enough data
canReadFromBuffer = true
else
canReadFromBuffer = false
else:
sleep(0); //yield cycles
SYNC:
while(true):
if canReadFromBuffer:
create a memory stream of compressedDataBuffer
create a zlibstream around that memory stream
perform operations on the zlibstream that are required.
I have an asynchronous TCP socket server in C# that I've made using the TcpListener/TcpClient wrappers for Socket. I'm pretty new to networking in general so I was unaware of how TCP treats data sent, and how it doesn't preserve the message boundaries. After a little research I think I've come up with a solid solution but I'm wondering if anyone has more advice for me.
Currently the data I'm sending is a byte[] of a serialized class object using protobuf (Google's data exchange library) https://code.google.com/p/protobuf/
After I serialize my data, and before it's sent, I've decided to append a 4 byte Int32 at the beginning of the byte array. My idea is that when the packet is sent to the server, it will parse out the Int32 and then wait until it's received that number of bytes before doing anything with the data, otherwise just calling BeginRead again.
Here is the code it's ran through before I write the data, and it seems to work fine, but I'm open to any performance suggestions:
public byte[] PackByteArrayForSending(byte[] data)
{
// [0-3] Packet Length
// [3-*] original data
// Get Int32 of the packet length
byte[] packetLength = BitConverter.GetBytes(data.Length);
// Allocate a new byte[] destination array the size of the original data length plus the packet length array size
byte[] dest = new byte[packetLength.Length + data.Length];
// Copy the packetLength array to the dest array
Buffer.BlockCopy(packetLength, 0, dest, 0, packetLength.Length);
// Copy the data array to the dest array
Buffer.BlockCopy(data, 0, dest, packetLength.Length, data.Length);
return dest;
}
I'm a little stuck on the server end. I have it reading the packetLength variable by using Buffer.BlockCopy to copy the first 4 bytes and then BitConverter.ToInt32 to read the length I should be getting. I'm not sure if I should constantly read the incoming data into a client-specific Stream object, or just use a while loop. Here's an example of the code I have on the server end so far:
NetworkStream networkStream = client.NetworkStream;
int bytesRead = networkStream.EndRead(ar);
if (bytesRead == 0)
{
Console.WriteLine("Got 0 bytes from {0}, marking as OFFLINE.", client.User.Username);
RemoveClient(client);
}
Console.WriteLine("Received {0} bytes.", bytesRead);
// Allocate a new byte array the size of the data that needs to be deseralized.
byte[] data = new byte[bytesRead];
// Copy the byte array into the toDeserialize buffer
Buffer.BlockCopy(
client.Buffer,
0,
data,
0,
bytesRead);
// Read the first Int32 tp get the packet length and then use the byte[4] to get the packetLength
byte[] packetLengthBytes = new byte[4];
Buffer.BlockCopy(
data,
0,
packetLengthBytes,
0,
packetLengthBytes.Length);
int packetLength = BitConverter.ToInt32(packetLengthBytes, 0);
// Now what do you recommend?
// If not all data is received, call
// networkStream.BeginRead(client.Buffer, 0, client.Buffer.Length, ReadCallback, client);
// and preserve the initial data in the client object
Thanks for your time and advice, I look forward to learning more about this subject.
TCP guarantees that the bytes stuffed into the stream at one end will fall out the other end in the same order and without loss or duplication. Expect nothing more, certainly not support for entities larger than a byte.
Typically I have prepended a header with a message length, type and subtype. There is often a correlation id provided to match requests to responses.
The basic pattern is to get bytes and append them to a buffer. If the data in the buffer is sufficient to contain a message header then extract the message length. If the data in the buffer is sufficient to contain the message then remove the message from the buffer and process it. Repeat with any remaining data until there are no complete messages to process. Depending on your application this may be the point to wait on a read or check the stream for additional data. Some applications may need to keep reading the stream from a separate thread to avoid throttling the sender.
Note that you cannot assume that you have a complete message length field. You may have three bytes left after processing a message and cannot extract an int.
Depending on your messages it may be more efficient to use a circular buffer rather than shuffling any remaining bytes each time a message is processed.
This question already has answers here:
Receiving data in TCP
(10 answers)
Closed 2 years ago.
I have written a simple TCP client and server. The problem lies with the client.
I'm having some trouble reading the entire response from the server. I must let the thread sleep to allow all the data be sent.
I've tried a few times to convert this code into a loop that runs until the server is finished sending data.
// Init & connect to client
TcpClient client = new TcpClient();
Console.WriteLine("Connecting.....");
client.Connect("192.168.1.160", 9988);
// Stream string to server
input += "\n";
Stream stm = client.GetStream();
ASCIIEncoding asen = new ASCIIEncoding();
byte[] ba = asen.GetBytes(input);
stm.Write(ba, 0, ba.Length);
// Read response from server.
byte[] buffer = new byte[1024];
System.Threading.Thread.Sleep(1000); // Huh, why do I need to wait?
int bytesRead = stm.Read(buffer, 0, buffer.Length);
response = Encoding.ASCII.GetString(buffer, 0, bytesRead);
Console.WriteLine("Response String: "+response);
client.Close();
The nature of streams that are built on top of sockets is that you have an open pipeline that transmits and receives data until the socket is closed.
However, because of the nature of client/server interactions, this pipeline isn't always guaranteed to have content on it to be read. The client and server have to agree to send content over the pipeline.
When you take the Stream abstraction in .NET and overlay it on the concept of sockets, the requirement for an agreement between the client and server still applies; you can call Stream.Read all you want, but if the socket that your Stream is connected to on the other side isn't sending content, the call will just wait until there is content.
This is why protocols exist. At their most basic level, they help define what a complete message that is sent between two parties is. Usually, the mechanism is something along the lines of:
A length-prefixed message where the number of bytes to be read is sent before the message
A pattern of characters used to mark the end of a message (this is less common depending on the content that is being sent, the more arbitrary any part of the message can be, the less likely this will be used)
That said you aren't adhering to the above; your call to Stream.Read is just saying "read 1024 bytes" when in reality, there might not be 1024 bytes to be read. If that's the case, the call to Stream.Read will block until that's been populated.
The reason the call to Thread.Sleep probably works is because by the time a second goes by, the Stream has 1024 bytes on it to read and it doesn't block.
Additionally, if you truly want to read 1024 bytes, you can't assume that the call to Stream.Read will populate 1024 bytes of data. The return value for the Stream.Read method tells you how many bytes were actually read. If you need more for your message, then you need to make additional calls to Stream.Read.
Jon Skeet wrote up the exact way to do this if you want a sample.
Try to repeat the
int bytesRead = stm.Read(buffer, 0, buffer.Length);
while bytesRead > 0. It is a common pattern for that as i remember.
Of course don't forget to pass appropriate params for buffer.
You dont know the size of data you will be reading so you have to set a mechanism to decide. One is timeout and another is using delimiters.
On your example you read whatever data from just one iteration(read) because you dont set the timeout for reading and using default value thats "0" milisecond. So you have to sleep just 1000 ms. You get same effect with using recieve time out to 1000 ms.
I think using lenght of data as prefix is not the real solution because when socket is closed by both sides, socket time-wait situation can not handled properly. Same data can be send to server and cause server to get exception . We used prefix-ending character sequence. After every read we check the data for start and end character sequence, if we cant get end characters, we call another read. But of course this works only if you have the control of server side and client side code.
In the TCP Client / Server I just wrote I generate the packet I want to send to a memory stream, then take the length of that stream and use it as a prefix when sending the data. That way the client knows how many bytes of data it's going to need to read for a full packet.
I'm trying to write a simple SNPP (Simple Network Paging Protocol) client using sockets. Everything seems to be working well, except for a small inconsistency between servers.
When I send a command, I need to read the reply, which is usually a single chunk of data. However, Sprint's SNPP server sends replies in two parts. The first chunk of data is the first digit of the status code. The second chunk is the remainder. For example, when I attempt to receive the "220 Gateway ready" reply, it arrives like this:
2
I have to send another empty command to retrieve the rest:
20 Gateway ready
For the moment, I'm using:
byte[] buffer = new byte[256];
socket.Receive(buffer);
How can I make sure that I receive all of the available data after issuing a command without allocating a separate buffer for each chunk of data?
For chunked responses I would recommend you reading data like this:
using (var resultStream = new MemoryStream())
{
const int CHUNK_SIZE = 2 * 1024; // 2KB, could be anything that fits your needs
byte[] buffer = new byte[CHUNK_SIZE];
int bytesReceived;
while ((bytesReceived = socket.Receive(buffer, buffer.Length, SocketFlags.None)) > 0)
{
byte[] actual = new byte[bytesReceived];
Buffer.BlockCopy(buffer, 0, actual, 0, bytesReceived);
resultStream.Write(actual, 0, actual.Length);
}
// Do something with the resultStream, like resultStream.ToArray() ...
}
Try to check Socket.Available property to determine if do you need to call Receive again.
I think I understand your question. This overload of Socket.Receive allows you to pass an integer to specify an offset position to start placing data. If you got 1 byte, on the first call as in your example, you can call this overload with an offset of 1 and use the same buffer.
Socket.Receive returns an integer that is the number of bytes that were received. You could check if this is 1 and call Receive again.
byte[] buffer = new byte[256];
int len = socket.Receive(buffer);
if (len == 1)
socket.Receive(buffer, 1, buffer.Length - 1, SocketFlags.None);
All,
When a web-server is sending data in "chunks" it precedes each chunk with it's length (as a string indicating the hex value).
chunk-size[;chunk-extensions]
chunk-data
eg: to chunk 15 bytes:
F;
123456789ABCDEF
The biggest issue with receiving HTTP data from a Socket is determining how much data to read: if you have received all the available data and call the Recieve method again, then the Recieve method will block until the remote socket sends more data, which for a HTTP/1.0 connection will never happen.
You need to implement a reader that wraps around the socket; it should receive data into a buffer, and provide a "ReadLine" method that reads the contents of the buffer until it reads 13 followed by 10 (CRLF)