Develop Reference

Why does SslStream.Read always set TcpClient.Available to 0, when NetworkStream.Read doesn't

I have a TcpClient client connected to a server, that send a message back to the client.
When reading this data using the NetworkStream.Read class I can specify the amount of bytes I want to read using the count parameter, which will decrease the TcpClient.Available by count after the read is finished. From the docs:
count Int32
The maximum number of bytes to be read from the current stream.
In example:
public static void ReadResponse()
{
if (client.Available > 0) // Assume client.Available is 500 here
{
byte[] buffer = new byte[12]; // I only want to read the first 12 bytes, this could be a header or something
var read = 0;
NetworkStream stream = client.GetStream();
while (read < buffer.Length)
{
read = stream.Read(buffer, 0, buffer.Length);
}
// breakpoint
}
}
This reads the first 12 bytes of the 500 available on the TcpClient into buffer, and inspecting client.Available at the breakpoint will yield (the expected) result of 488 (500 - 12).
Now when I try to do the exact same thing, but using an SslStream this time, the results are rather unexpected to me.
public static void ReadResponse()
{
if (client.Available > 0) // Assume client.Available is 500 here
{
byte[] buffer = new byte[12]; // I only want to read the first 12 bytes, this could be a header or something
var read = 0;
SslStream stream = new SslStream(client.GetStream(), false, new RemoteCertificateValidationCallback(ValidateServerCertificate), null);
while (read < buffer.Length)
{
read = stream.Read(buffer, 0, buffer.Length);
}
// breakpoint
}
}
This code will read the first 12 bytes into buffer, as expected. However when inspecting the client.Available at the breakpoint now will yield a result of 0.
Like the normal NetworkStream.Read the documentation for SslStream.Read states that count indicates the max amount of bytes to read.
count Int32
A Int32 that contains the maximum number of bytes to read from this stream.
While it does only read those 12 bytes, and nothing more I am wondering where the remaining 488 bytes go.
In the docs for either SslStream or TcpClient I couldn't find anything indicating that using SslStream.Read flushes the stream or otherwise empties the client.Available. What is the reason for doing this (and where is this documented)?
There is this question that asks for an equivalent of TcpClient.Available, which is not what i'm asking for. I want to know why this happens, which isn't covered there.

Remember that the SslStream might be reading large chunks from the underlying TcpStream at once and buffering them internally, for efficiency reasons, or because the decryption process doesn't work byte-by-byte and needs a block of data to be available. So the fact that your TcpClient contains 0 available bytes means nothing, because those bytes are probably sitting in a buffer inside the SslStream.
In addition, your code to read 12 bytes is incorrect, which might be affecting what you're seeing.
Remember that Stream.Read can return fewer bytes than you were expecting. Subsequent calls to Stream.Read will return the number of bytes read during that call, and not overall.
So you need something like this:
int read = 0;
while (read < buffer.Length)
{
int readThisTime = stream.Read(buffer, read, buffer.Length - read);
if (readThisTime == 0)
{
// The end of the stream has been reached: throw an error?
}
read += readThisTime;
}

When you're reading from a TLS stream, it over-reads, maintaining an internal buffer of data that is yet to be decrypted - or which has been decrypted but not yet consumed. This is a common approach used in streams especially when they mutate the content (compression, encryption, etc), because there is not necessarily a 1:1 correlation between input and output payload sizes, and it may be necessary to read entire frames from the source - i.e. you can't just read 3 bytes - the API needs to read the entire frame (say, 512 bytes), decrypt the frame, give you the 3 you wanted, and hold onto the remaining 509 to give you next time(s) you ask. This means that it often needs to consume more from the source (the socket in this case) than it gives you.
Many streaming APIs also do the same for performance reasons, for example StreamReader over-reads from the underlying Stream and maintains internally both a byteBuffer of bytes not yet decoded, and a charBuffer of decoded characters available for consuming. Your question would then be comparable to:
When using StreamReader, I've only read 3 characters, but my Stream has advanced 512 bytes; why?

Do TCP sockets automatically close after 64kB send?

I'm trying to send multiple files over TCP using C# TcpClient, for a file below 64kB it works great, but when I have more it throws an exeption that host-computer disconnected.
Here is my code:
Server side(sending).
1) SocketServer class
public abstract class SocketServer
{
private Socket serverSocket;
public SocketServer(IPEndPoint localEndPoint)
{
serverSocket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
serverSocket.Bind(localEndPoint);
serverSocket.Listen(0);
serverSocket.BeginAccept(BeginAcceptCallback, null);
}
private void BeginAcceptCallback(IAsyncResult ar)
{
Socket clientSocket = serverSocket.EndAccept(ar);
Console.WriteLine("Client connected");
ClientConnection clientConnection = new ClientConnection(this, clientSocket);
Thread clientThread = new Thread(new ThreadStart(clientConnection.Process));
clientThread.Start();
serverSocket.BeginAccept(BeginAcceptCallback, null);
}
internal abstract void OnReceiveMessage(ClientConnection client, byte header, byte[] data);
}
Here goes ClientConnection:
public class ClientConnection
{
private SocketServer server;
private Socket clientSocket;
private byte[] buffer;
private readonly int BUFFER_SIZE = 8192;
public ClientConnection(SocketServer server, Socket clientSocket)
{
this.server = server;
this.clientSocket = clientSocket;
buffer = new byte[BUFFER_SIZE];
}
public void Process()
{
clientSocket.BeginReceive(buffer, 0, buffer.Length, SocketFlags.Peek, BeginReceiveCallback, null);
}
private void BeginReceiveCallback(IAsyncResult ar)
{
int bytesReceived = clientSocket.EndReceive(ar);
if (bytesReceived >= 4)
{
clientSocket.Receive(buffer, 0, 4, SocketFlags.None);
// message size
int size = BitConverter.ToInt32(buffer, 0);
// read message
int read = clientSocket.Receive(buffer, 0, size, SocketFlags.None);
// if data still fragmented, wait for it
while (read < size)
{
read += clientSocket.Receive(buffer, read, size - read, SocketFlags.None);
}
ProcessReceivedData(size);
}
clientSocket.BeginReceive(buffer, 0, buffer.Length, SocketFlags.Peek, BeginReceiveCallback, null);
}
private void ProcessReceivedData(int size)
{
using (PacketReader pr = new PacketReader(buffer))
{
// message header = 1 byte
byte header = pr.ReadByte();
// next message data
byte[] data = pr.ReadBytes(size - 1);
server.OnReceiveMessage(this, header, data);
}
}
public void Send(byte[] data)
{
// first of all, send message length
clientSocket.Send(BitConverter.GetBytes(data.Length), 0, 4, SocketFlags.None);
// and then message
clientSocket.Send(data, 0, data.Length, SocketFlags.None);
}
}
Implementation of FileServer class:
public class FileServer : SocketServer
{
string BinaryPath;
public FileServer(IPEndPoint localEndPoint) : base(localEndPoint)
{
BinaryPath = Path.GetDirectoryName(System.Reflection.Assembly.GetExecutingAssembly().Location);
}
internal override void OnReceiveMessage(ClientConnection client, byte hdr, byte[] data)
{
// Convert header byte to ENUM
Headers header = (Headers)hdr;
switch(header)
{
case Headers.Queue:
Queue(client, data);
break;
default:
Console.WriteLine("Wrong header received {0}", header);
break;
}
}
private void Queue(ClientConnection client, byte[] data)
{
// this message contains fileName
string fileName = Encoding.ASCII.GetString(data, 1, data.Length - 1);
// combine path with assembly location
fileName = Path.Combine(BinaryPath, fileName);
if (File.Exists(fileName))
{
FileInfo fileInfo = new FileInfo(fileName);
long fileLength = fileInfo.Length;
// pass the message that is now start a file transfer, contains:
// 1 byte = header
// 16 bytes = file length
using (PacketWriter pw = new PacketWriter())
{
pw.Write((byte)Headers.Start);
pw.Write(fileLength);
client.Send(pw.GetBytes());
}
//
using (FileStream fs = new FileStream(fileName, FileMode.Open, FileAccess.Read))
{
int read = 0, offset = 0;
byte[] fileChunk = new byte[8191];
while (offset < fileLength)
{
// pass message with file chunks, conatins
// 1 byte = header
// 8195 bytes = file chunk
using (PacketWriter pw = new PacketWriter())
{
fs.Position = offset;
read = fs.Read(fileChunk, 0, fileChunk.Length);
pw.Write((byte)Headers.Chunk);
pw.Write(fileChunk, 0, read);
client.Send(pw.GetBytes());
}
offset += read;
}
}
}
}
}
And helper classes:
public class PacketWriter : BinaryWriter
{
private MemoryStream memoryStream;
public PacketWriter() : base()
{
memoryStream = new MemoryStream();
OutStream = memoryStream;
}
public byte[] GetBytes()
{
byte[] data = memoryStream.ToArray();
return data;
}
}
public class PacketReader : BinaryReader
{
public PacketReader(byte[] data) : base(new MemoryStream(data))
{
binaryFormatter = new BinaryFormatter();
}
}
Client have the almost same code. Except receive:
internal override void OnReceiveMessage(ServerConnection client, byte hdr, byte[] data)
{
Headers header = (Headers)hdr;
switch(header)
{
case Headers.Start:
Start(data); // save length of file and prepare for receiving it
break;
case Headers.Chunk:
Chunk(data);
break;
default:
Console.WriteLine("Wrong header received {0}", header);
break;
}
}
private void Chunk(byte[] data)
{
// Process reveived data, write it into a file
}

Do TCP sockets automatically close after 64kB send?
No, TCP sockets do not automatically close after 64 KB of sent data, nor after any amount of sent data. They remain open until either end closes the connection or a network error of some sort happens.
Unfortunately, your question is extremely vague, providing practically no context. But I will point out one potentially serious bug in your client code: you are not restricting the data read by the client to that which you expect for the current file. If the server is sending more than one file, it is entirely possible and likely that on reaching the end of the data for the current file, a read containing that final sequence of bytes will also contain the initial sequence of bytes for the next file.
Thus, the expression totalBytesRead == fileLenght will fail to ever be true and your code will attempt to read all of the data the server sends as if it's all part of the first file.
One easy way to check for this scenario is to look at the size of the file that was written by the client. It will be much larger than expected, if the above is the issue.
You can fix the code by only ever receiving at most the number of bytes you actually expect to be remaining:
while ((bytesRead = binaryReader.Read(
buffer, 0, Math.Min(fileLenght - totalBytesRead, buffer.Length))) > 0)
{
fs.Write(buffer, 0, bytesRead);
totalBytesRead += bytesRead;
if (totalBytesRead == fileLenght)
{
break;
}
}
While you're at it, you might fix the spelling of the variable named fileLenght. I know Intellisense makes it easy to type the variable name whether it's spelled right or not, but if you one day have to grep through the source code looking for code that involves itself with "length" values, you'll miss that variable.
EDIT:
Having pointed out the problem in the code you posted with your question originally, I now find myself looking at completely different code, after your recent edit to the question. The example is still not a good, minimal, complete code example and so advice will still have to be of a limited nature. But I can see at least two major problems in your receive handling:
You are mixing Receive() and BeginReceive(). This is probably not too terrible in general; what makes it really bad in your case is that in the completion handler for the initial asynchronous receive, you are blocking the thread while you use the synchronous Receive() to (attempt to) receive the rest of the data, and then to make matters worse, in the FileServer case you continue to block that thread while you transmit the file data to the client.I/O completion handlers must not block the thread for any significant amount of time; they must retrieve the data for the completed operation, and return as quickly as is feasible. There's a lot of wiggle room, but not enough to justify tying up the IOCP thread for the entire file transfer.
While the first issue is bad, it's probably not bad enough to actually cause the connection to reset. In a low-volume scenario, and especially if you're just dealing with one client at a time at the moment, I doubt blocking the IOCP thread that's being used to call your I/O completion callback will cause any easily observable problems.What's much worse is that you allocate the buffer array once, with a length of 8192, and then proceed to attempt to stuff the entire counted-byte stream of bytes being sent. On the assumption that the client will be sending short messages, this is probably fine on the server side. But on the client side, the value of size can easily be greater than the 8192 bytes allocated in the buffer array.I find it very likely that your BeginReceiveCallback() method is throwing an ArgumentOutOfRangeException after receiving the first 8192 bytes, due to the fact that you are passing the Receive() method a size value that exceeds the length of the buffer array. I can't prove that, because the code example is incomplete. You'll have to follow up on that theory yourself.
As long as I'm reviewing the code you posted, some other irregularities stood out as I scanned through it:
It's a bit odd to be using a buffer length of 8191 when transmitting the file.
Your BeginReceiveCallback() method discards whatever bytes were sent initially. You call EndReceive(), which returns the count of bytes received, but the code that follows assumes that the size value in the stream is still waiting to be read, when it very easily could have been sent in the first block of bytes. Indeed, your comparison bytesReceived >= 4 seems to be intended to refrain from processing received data until you have that length value. Which is wrong for another reason:
If your header is 1 byte followed by a 4-byte count-of-bytes value, then you really need bytesReceived > 4. But worse, TCP is allowed to drip out a single byte at a time if it really wants to (it won't, but if you write your code correctly, it won't matter). Since you only process received data if you get at least 4 bytes, and since if you don't get 4 bytes, you just ignore whatever was sent so far, you could theoretically wind up ignoring all of the data that was sent.
That you have the risk that any data could be ignored at all is terrible. But #2 above guarantees it, and #3 just adds insult to injury.
When receiving the size value, you are reading a 32-bit integer, but when sending the size value (i.e. for the file itself), you are writing a 64-bit integer. You'll get the right number on the receiving end, because BinaryWriter uses little-endian and that means that the first four bytes of any 64-bit integer less than 2^31 are identical to the four bytes of the 32-bit integer representation of the same number. But you'll still wind up with four extra zero bytes after that; if treated as file data as they likely would, that would corrupt your file.

TcpClient's NetworkStream reads incomplete data unless Thread.Sleep(1) is called

I have a method that reads some data through a TcpClient's NetworkStream, and it has been giving me some errors.
During investigation, I discovered that it actually worked fine... but only if I stepped through the code with Visual Studio 2012's debugger, using a breakpoint.
Here's my code:
public static byte[] DownloadStream(string hostname, int port,
byte[] requestBytes, int bufferSize = 4096)
{
byte[] responseBytes = null;
var client = new System.Net.Sockets.TcpClient(hostname, port);
if (client.Connected)
{
using (var stream = client.GetStream())
{
stream.Write(requestBytes, 0, requestBytes.Length);
stream.Flush();
if (stream.CanRead)
{
var responseStream = new System.IO.MemoryStream();
byte[] buffer = new byte[bufferSize];
int bytesRead = 0;
do
{
bytesRead = stream.Read(buffer, 0, buffer.Length);
responseStream.Write(buffer, 0, bytesRead);
}
while (stream.DataAvailable);
responseBytes = responseStream.ToArray();
}
}
}
client.Close();
return responseBytes;
}
This is pretty frustrating, since there's no real error. It apparently just needs the debugger to hold its hand while it reads the NetworkStream.
Does anybody know why this is happening? How can I fix it?
Edit:
For some reason making this change eliminates the problem:
do
{
bytesRead = stream.Read(buffer, 0, buffer.Length);
responseStream.Write(buffer, 0, bytesRead);
System.Threading.Thread.Sleep(1); //added this line
}
while (stream.DataAvailable);
Any insight on this?

The NetworkStream.DataAvailable property is not a reliable way to detect the end of the response; if the response is split into multiple TCP packets and a packet has not yet been delivered at the moment you check DataAvailable, then the property will return false, terminating your loop prematurely.
Apparently, your network connection is fast enough that Thread.Sleep(1) provides sufficient time for each successive packet to arrive. On a slower connection, though, it might not be adequate.
For reliable communication, the client and the server need to agree on a way to signal the end of the response. For example:
The server can send the length of the response as a prefix before the actual response. The client reads the length, and then it reads from the stream until it has received that number of bytes.
The server can send a special terminator as a suffix after the actual response. The client reads from the stream until it encounters the terminator. (Obviously, the terminator cannot appear anywhere in the response itself.)
The server can close the connection after it sends the entire response. The client reads from the stream until the connection is closed.

When NetworkStream.Read(byte[], int, int) needs to return -1, it abort()s the thread it is running on instead?

NetworkStream stream = socket.GetStream();
if (stream.CanRead)
{
while (true)
{
int i = stream.Read(buf, 0, 1024);
result += Encoding.ASCII.GetString(buf, 0, i);
}
}
Above code was designed to retrieve message from a TcpClient while running on a separate thread. Read Method works fine until it is supposed to return -1 to indicate there is nothing to read anymore; instead, it just terminates the thread it is running on without any apparent reason - tracing each step using the debugger shows that it just stops running right after that line.
Also I tried encapsulating it with a try ... catch without much success.
What could be causing this?
EDIT: I tried
NetworkStream stream = socket.GetStream();
if (stream.CanRead)
{
while (true)
{
int i = stream.Read(buf, 0, 1024);
if (i == 0)
{
break;
}
result += Encoding.ASCII.GetString(buf, 0, i);
}
}
thanks to #JonSkeet, but the problem is still there. The thread terminates at that read line.
EDIT2: I fixed the code like this and it worked.
while (stream.DataAvailable)
{
int i = stream.Read(buf, 0, 1024);
result += Encoding.ASCII.GetString(buf, 0, i);
}
I think the problem was simple, I just didn't think thoroughly enough. Thanks everyone for taking a look at this!

No, Stream.Read returns 0 when there's nothing to read, not -1:
Return value
The total number of bytes read into the buffer. This can be less than the number of bytes requested if that many bytes are not currently available, or zero (0) if the end of the stream has been reached.
My guess is that actually, no exception is being thrown and the thread isn't being aborted - but it's just looping forever. You should be able to see this if you step through in the debugger. Whatever's happening, your "happy" termination condition will never be hit...

Since you're trying to read ASCII characters, from a stream, take a look at the following as a potentially simpler way to do it:
public IEnumerable<string> ReadLines(Stream stream)
{
using (StreamReader reader = new StreamReader(stream, Encoding.ASCII))
{
while (!reader.EndOfStream)
yield return reader.ReadLine();
}
}
While this may not be exactly what you want, the salient points are:
Use a StreamReader to do all the hard work for you
Use a while loop with !reader.EndOfStream to loop through the stream
You can still use reader.Read(buffer, 0, 1024) if you'd prefer to read chunks into a buffer, and append to result. Just note that these will be char[] chunks not byte[] chunks, which is likely what you want.

It looks to me like it is simply blocking - i.e. waiting on the end of the stream. For it to return a non-positive number, it is necessary that the stream be closed, i.e. the the caller has not only sent data, but has closed their outbound socket. Otherwise, the system cannot distinguish between "waiting for a packet to arrive" and "the end of the stream".
If the caller is sending one message only, they should close their outbound socket after sending (they can keep their inbound socket open for a reply).
If the caller is sending multiple messages, then you must use a framing approach to read individual sub-messages. In the case of a text-based protocol this usually means "hunt the newline".

TCP server is not reading fixed size messages correctly

I'm sending sensor data from an android device to a TCP server in C#. The android client sends the data in fixed size chunks of 32 bytes.
In the server I read the data expecting it would come in full packs, but since TCP is a stream protocol some messages arrive at the server split in two parts. I know that because I can watch it with a software called SocketSniff.
The problem is that I don't know how to handle it on my server.
All examples I found use the NetworkStream.Read(), in this method I have to pass a array of bytes to store the data read, an offset and the number of bytes to read. This array of bytes must have a know size, in my case 32.
I don't know the real size of the message that arrived on my server, but it could be one of the following situations.
If the received data size is 32 bytes, it's all OK.
If the received data size if greater than 32 bytes, I think I'm loosing data.
If the received data size is less than 32 bytes, lets say 20 bytes, these bytes are stored in my array and the last 12 bytes of the array remain with the value of zero. Since I may be really receiving some zeros there's no way to know the size I really received, so I can't merge it with the remaining data which should come in the next reading.
My code which handles the receiving is the following:
int buffer = 32;
...
private void HandleClientComm(object client)
{
TcpClient tcpClient = (TcpClient)client;
NetworkStream clientStream = tcpClient.GetStream();
byte[] message = new byte[buffer];
int bytesRead;
while (true)
{
bytesRead = 0;
try
{
bytesRead = clientStream.Read(message, 0, message.Length);
}
catch
{
break;
}
if (bytesRead == 0)
{
// Connection closed
break;
}
SensorData sensorData = ProcessTcpPacket(message);
}
tcpClient.Close();
}
Is there any way to know the size of the data I'm receiving in the socket?

Well, yes, you have the bytesRead variable - it holds the number of bytes read from the stream. You will read at most message.Length bytes, but you may read less.
Note that if there are more bytes available, you will not lose them by reading just message.Length bytes. Rather, they will be available for you next time you read from the stream.
What you need to do is add another while loop liked this:
int messageRead = 0;
while(messageRead < message.Length)
{
int bytesRead = clientStream.Read(message, messageRead, message.Length - messageRead);
messageRead += bytesRead;
if(bytesRead==0)
return; // The socket was closed
}
// Here you have a full message