If i send 1000 bytes in TCP, does it guarantee that the receiver will get the entire 1000 bytes "togther"? or perhaps he will first only get 500 bytes, and later he'll receive the other bytes?
EDIT: the question comes from the application's point of view. If the 1000 bytes are reassembles into a single buffer before they reach the application .. then i don't care if it was fragmented in the way..
See Transmission Control Protocol:
TCP provides reliable, ordered delivery of a stream of bytes from a program on one computer to another program on another computer.
A "stream" means that there is no message boundary from the receiver's point of view. You could get one 1000 byte message or one thousand 1 byte messages depending on what's underneath and how often you call read/select.
Edit: Let me clarify from the application's point of view. No, TCP will not guarantee that the single read would give you all of the 1000 bytes (or 1MB or 1GB) packet the sender may have sent. Thus, a protocol above the TCP usually contains fixed length header with the total content length in it. For example you could always send 1 byte that indicates the total length of the content in bytes, which would support up to 255 bytes.
As other answers indicated, TCP is a stream protocol -- every byte sent will be received (once and in the same order), but there are no intrinsic "message boundaries" -- whether all bytes are sent in a single .send call, or multiple ones, they might still be received in one or multiple .receive calls.
So, if you need "message boundaries", you need to impose them on top of the TCP stream, IOW, essentially, at application level. For example, if you know the bytes you're sending will never contain a \0, null-terminated strings work fine; various methods of "escaping" let you send strings of bytes which obey no such limitations. (There are existing protocols for this but none is really widespread or widely accepted).
Basically as far as TCP goes it only guarantees that the data sent from one end to the other end will be sent in the same order.
Now usually what you'll have to do is have an internal buffer that keeps looping until it has received your 1000 byte "packet".
Because the recv command as mentioned returns how much has actually been received.
So usually you'll have to then implement a protocol on top of TCP to make sure you send data at an appropriate speed. Because if you send() all the data in one run through it will overload the under lying networking stack, and which will cause complications.
So usually in the protocol there is a tiny acknowledgement packet sent back to confirm that the packet of 1000 bytes are sent.
You decide, in your message that how many bytes your message shall contain. For instance in your case its 1000. Following is up and running C# code to achieve the same. The method returns with 1000 bytes. The abort code is 0 bytes; you can tailor that according to your needs.
Usage:
strMsg = ReadData(thisTcpClient.Client, 1000, out bDisconnected);
Following is the method:
string ReadData(Socket sckClient, int nBytesToRead, out bool bShouldDisconnect)
{
bShouldDisconnect = false;
byte[] byteBuffer = new byte[nBytesToRead];
Array.Clear(byteBuffer, 0, byteBuffer.Length);
int nDataRead = 0;
int nStartIndex = 0;
while (nDataRead < nBytesToRead)
{
int nBytesRead = sckClient.Receive(byteBuffer, nStartIndex, nBytesToRead - nStartIndex, SocketFlags.None);
if (0 == nBytesRead)
{
bShouldDisconnect = true;
//0 bytes received; assuming disconnect signal
break;
}
nDataRead += nBytesRead;
nStartIndex += nBytesRead;
}
return Encoding.Default.GetString(byteBuffer, 0, nDataRead);
}
Let us know this didn't help you (0: Good luck.
Yes, there is a chance for receiving packets part by part. Hope this msdn article and following example (taken from the article in msdn for quick review) would be helpful to you if you are using windows sockets.
void CChatSocket::OnReceive(int nErrorCode)
{
CSocket::OnReceive(nErrorCode);
DWORD dwReceived;
if (IOCtl(FIONREAD, &dwReceived))
{
if (dwReceived >= dwExpected) // Process only if you have enough data
m_pDoc->ProcessPendingRead();
}
else
{
// Error handling here
}
}
TCP guarantees that they will recieve all 1000 bytes, but not necessarily in order (though, it will appear so to the recieving application) and not necessarily all at once (unless you craft the packet yourself and make it so.).
That said, for a packet as small as 1000 bytes, there is a good chance it'll send in one packet as long as you do it in one call to send, though for larger transmissions it may not.
The only thing that the TCP layer guarantees is that the receiver will receive:
all the bytes transmitted by the sender
in the same order
There are no guarantees at all about how the bytes might be split up into "packets". All the stuff you might read about MTU, packet fragmentation, maximum segment size, or whatever else is all below the layer of TCP sockets, and is irrelevant. TCP provides a stream service only.
With reference to your question, this means that the receiver may receive the first 500 bytes, then the next 500 bytes later. Or, the receiver might receive the data one byte at a time, if that's what it asks for. This is the reason that the recv() function takes a parameter that tells it how much data to return, instead of it telling you how big a packet is.
The transmission control protocol guarantees successful delivery of all packets by requiring acknowledgment of the successful delivery of each packet to the sender by the receiver. By this definition the receiver will always receive the payload in chunks when the size of the payload exceeds the MTU (maximum transmission unit).
For more information please read Transmission Control Protocol.
The IP packets may get fragmented during retransmission.
So the destination machine may receive multiple packets - which will be reassembled back by TCP/IP stack. Depending on the network API you are using - the data will be given to you either reassembled or in RAW packets.
It depends of the stablished MTU (Maximum transfer unit). If your stablished connection (once handshaked) refers to a MTU of 512 bytes you will need two or more TCP packets to send 1000 bytes.
What is the behaviour of the NetworkStream.Write() method, if I send data via a TcpClient's NetworkStream, and the TcpClient.SendBufferSize is smaller than the data?
The MSDN documentation for SendBufferSize says
If the network buffer is smaller than the amount of data you provide
the Write method, several network send operations will be performed
for every call you make to the Write method. You can achieve greater
data throughput by ensuring that your network buffer is at least as
large as your application buffer.
So I know that the data will be sent in multiple operations, and the receiving TCP-Stack should reassemble it into one continuous stream transparently.
But what happens exactly in my program during this time?
If there is enough space in the SendBuffer, TcpClient.GetStream().Write() will not block at all, and return immediately and so will NetworkStream.Flush().
If I set the TcpClient.SendBufferSize to a value smaller than the data, will the Write() block until
either the first part of the data has been received and ACKed,
or the TcpClient.SendTimeout has expired?
Or does it work in some other way? Does it actually wait for a TCP ACK?
Are there any other drawbacks besides higher overhead to such a smaller buffer size? Are there problems with changing the SendBufferSize on the fly?
Example:
byte[] data = new byte[20] // 20 byte data
tcpClient.SendBufferSize= 10; // 10 byte buffer
tcpClient.SendTimeout = 1000; // 1s timeout
tcpClient.GetStream().Write(data,0,20);
// will this block until the first 10 bytes have been full transmitted?
// does it block until a TCP ACK has been received? or something else?
// will it throw if the first 10 bytes have not been received in 1 second?
tcpClient.GetStream().Flush(); // would this make any difference?
My goal here is mainly getting a better understanding of the network internals.
Also , I was wondering if this could be abused to react more quickly to a network failure. If data is sent only infrequently, and each data packet is small enough to be transmitted at once, and there are no receipt messages in a given message protocol, it could take a long time after a network error until the next Write() is called; so a long time until an exception is thrown.
If the SentBuffer is very small, would an error be noticed more quickly, unless it happened at end of the data?
Could I abuse this to measure the time it takes for a single packet to be transmitted and ACKed?
I'm encountering a pretty difficult problem in an asynchronous RSS feed aggregator application I am creating.
The system is built from a communicating client and server. The server collects RSS feed items from different sources and then distributes them to clients based on their subscriptions.
The issue that I am having is - My specification states that I must implement a pre-defined byte based protocol to communicate between the client and server. Part of this protocol is that when the server sends a payload to the client, that payload must have a checksum created and sent with it, then when it is received by the client the checksum is equated based on the received payload. We then compare the two checksums and if they do not match then we have an issue.
The thing that is baffling me is that, sometimes the checksums will match perfectly and other times they will not, using the same algorithm for all sending and receiving operations.
Here is the method that checks incoming ->
private bool CheckChecksum(uint receivedChecksum, IEnumerable<byte> bytes) {
uint ourChecksum = 0;
foreach (var b in bytes)
ourChecksum += b;
if (ourChecksum != receivedChecksum) {
Debug.WriteLine("received {0}, calculated {1}", receivedChecksum, ourChecksum);
_writeOutToFile = true;
}
return receivedChecksum == ourChecksum;
}
and when calculating it before sending ->
uint checksum = payloadBytes.Aggregate<byte, uint>(0,
(currentChecksum, currentByte) => currentChecksum + currentByte);
Since the behaviour seems to occur on updates that have very large checksums (generally 2 million+) then the only thing I can think that would be causing it is these large byte[] sizes.
As you can see above, I write out to a file the contents of the payload when the checksums do not match. What I found was that the byte[] just ends early (despite the fact that the reading/writing lengths match on both the client and server). The end of the byte[] is just filled with empty spaces.
I am using NetworkStream.ReadAsync and WriteAsync to do all of my I/O operations. I thought that it may be a fault with the way I am using or understanding these.
I realise that this is a very difficult and vague problem (because I'm not sure what is going wrong myself) but if you need any further information I will provide it.
Here is some extra information:
The checksums are of type uint, and their endianness is encoded correctly at both ends of the system.
Payloads are strings which are encoded into ASCII bytes and decoded on the client-side.
All messages are sent with a checksum and a payload length. The payload length represents how many bytes to read off the stream. Since the I read payload length amount of bytes from the networkstream, when the checksums do not match the payload becomes white space after a varying length (but is correct before this point).
Sometimes (rarely) the checksums will match even when they are large.
I am running both the client and server locally on one machine.
I have tried having a Thread.Sleep(5000) before the read and this makes no difference.
Here is a sample of sent and received data that fails.
Sent from server - http://pastebin.com/jvbCbQmJ
Received by client - http://pastebin.com/eNkWymwi
Your receiving code tries to read before all the bytes have been posted.
When you try to send huge chunks of data on only one package, your readAsync will detect there's something to read, try to read for full lenght, which may not be available, and fill the part that was not posted yet with 0s
You can either divide your message on the server, read by parts on the client, or try to read what's available until you have received everything or some time has passed
A couple of questions about TCP.
I know that the packet will be "split" or fragmented if it hits a network device with lower MTU etc. But the problems I have is understanding how I "rebuild" my payload in the application. (I've been trying for 2-3 hours now, but can't seem to get it right)
First things first. When sending the packet, what are the pros/cons of the two following options;
NetworkStream ns = client.GetStream();
ns.Write(BitConverter.GetBytes(100));
ns.Write(BitConverter.GetBytes("Test"));
or
NetworkStream ns = client.GetStream();
byte[] payload = BitConverter.GetBytes(100).Concat(BitConverter.GetBytes("Test"));
ns.Write(payload); //, 0, payload.Length);
And how do I rebuild the payload at the recv. part if it has been split? I would love a spoon-feeding on this one as I seem to miss some very important, but not so obvious, part in my current application.
You don't care at all about MTU or fragmentation. It is the responsibility of the OS's TCP/IP stack to re-assemble a fragmented packet.
You have to remember, it is a TCP "stream", so it doesn't really matter how you read from or write to it; it's just a continual flow of data. It's up to your definition of the packets to provide for data structures passed through that stream.
In many structured protocols, there is a user-defined header and the first field specifies the length of the packet. So you first read X bytes where X is the size of the header. You interpret that header so you know how large the rest of the packet is. Then you read an appropriate number of bytes to get the rest of the packet.
So, of your two examples, it doesn't really matter. Go with whatever is easiest (in this case, the first one.)
I have a client-server program.
I'm sending data like this:
private void Sender(string s,TcpClient sock)
{
try
{
byte[] buffer = Encoding.UTF8.GetBytes(s);
sock.Client.Send(buffer);
}catch{}
}
and on the client side receiving like this:
byte[] buffer = new byte[PacketSize];
int size = client.Client.Receive(buffer);
String request = Encoding.UTF8.GetString(buffer, 0, size);
The problem is that data is not fully received always, sometimes it's only part of what I have sent. PacketSize is 10240 which is more than the bytes I send. I have also set SendBufferSize and ReceiveBufferSize at both sides.
The worst part is that sometimes data is fully received!
What might be the problem?
The size value returned by TcpClient.Receive is not the same as the length of the buffered string you sent. This is because there is no guarantee that when calling Receive once you will get back all the data that you sent with Send call. This behavior is intrinsic to the way TCP works (it's a stream-, not a message-based data protocol).
You cannot solve the problem by using bigger buffers, as the buffers you provide can only limit the amount of data that Receive returns. Even if you provide a 1MB buffer and there is 1MB of data to read, Receive can legitimately return any number of bytes (even just 1).
What you need to do is make sure that you have buffered all the data before calling Encoding.GetString. To do that, you need to know how much data there is in the first place. So at the very least, you need to write the length of the string bytes when sending:
byte[] buffer = Encoding.UTF8.GetBytes(s);
byte[] length = BitConverter.GetBytes(buffer.Length);
sock.Client.Send(length);
sock.Client.Send(buffer);
When receiving, you will first read the length (which has a known fixed size: 4 bytes) and then start buffering the rest of the data in a temp buffer until you have length bytes (this might take any number of Receive calls, so you 'll need a while loop). Only then can you call Encoding.GetString and get your original string back.
Explanation of the behavior you observe:
Even though the network stack of the OS makes pretty much no guarantees, in practice it will usually give you the data one TCP packet brings with one Receive call. Since the MTU (maximum packet size) for TCP allows around 1500 bytes for payload, naive code will work fine as long as the strings are less than this size. More than that and it will get split into multiple packets, and one Receive will then return only part of the data.