Develop Reference

TcpListener and NetworkStream doesn't get data

I am trying to receive a message from an equipment. This equipment is an authentication terminal, and it will send the message as soon as the user set his credentials.
Also, the manual of the equipment says the message will be sent in the ILV format, standing I for identification, L for length and V for value.
a normal message would be:
I -> 0x00 (byte 0 indicating success)
L -> 0x04 0x00 (two bytes for length, being 4 the length in this case)
V -> 0x35 0x32 0x38 0x36 (the message itself)
The message is sent in TCP protocol, so I created a socket using the TcpListener class, following this sample from Microsoft:
https://msdn.microsoft.com/en-us/library/system.net.sockets.tcplistener(v=vs.110).aspx
new Thread(() =>
{
TcpListener server = null;
try
{
Int32 port = 11020;
IPAddress localAddr = IPAddress.Parse("192.168.2.2");
server = new TcpListener(localAddr, port);
server.Server.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReuseAddress, true);
server.Start();
byte[] bytes = new byte[256];
String data = null;
while (true)
{
TcpClient client = server.AcceptTcpClient();
data = null;
NetworkStream stream = client.GetStream();
int i = 0;
while((i = stream.Read(bytes, 0, bytes.Length)) != 0)
{
// this code is never reached as the stream.Read above runs for a while and receive nothing
data = System.Text.Encoding.ASCII.GetString(bytes, 0, i);
}
client.Close();
}
}
catch (SocketException ex)
{
// Actions for exceptions.
}
finally
{
server.Stop();
}
}).Start();
If the stream.Read is removed, then the code flows (although I got nothing either), however if I put any stream.Read statement the execution holds for a while like it was waiting for some response, and then it ends with no response, all bytes read is zero.
I am running Wireshark on the computer and the data is being sent.
Anybody knows what I am doing wrong?

I think the problem is right in the Read method, which halts until the buffer fills completely, but that's obviously won't happen anytime.
In that while-loop, you should spin by checking for available data first, then read them and append to the byte-array. Moreover, since the loop become too "tight", it's better to yield the control to the task scheduler for a bit.
Here is an example:
while (true)
{
if (stream.DataAvailable)
{
int count = stream.Read(bytes, i, bytes.Length);
i += count;
// ...add something to detect the end of message
}
else
{
Thread.Sleep(0);
}
}
It's worthwhile to notice that there's no timeout, but that's a very common feature in order to quit the loop when no data (or broken) are incoming.
Furthermore, the above pattern is not the best way to accumulate data, because you may get an exception when too many bytes are received. Consider a spurious stream of 300 bytes: that will overflow the available buffer. Either use the "bytes" as a temporary buffer for what the Read method gives, or provide a safety check before calling the Read method (so that you may provide the best byte count to read).
Useful links here:
https://learn.microsoft.com/en-us/dotnet/api/system.net.sockets.networkstream.read?view=netframework-4.7.1#System_Net_Sockets_NetworkStream_Read_System_Byte___System_Int32_System_Int32_

Best Way to write large data over network stream of TcpClient

We have a requirement to upload large firmware files to printers to upgrade the firmware of the device. The printer device is in the same network as my server, and the size of firmware that we are trying to upload is approximately to 200 - 500 MB. The approach that we have chosen is to load the firmware (.bin file) into Memory stream and write it in chunks over the network using TcpClient.
Based on the response from the network stream, we are displaying the status of firmware upgrade to the client. Following is the code snippet that we have used for firmware upgrade. I want to know if it is the best approach, as wrong one may harm the device.
EDIT:
class MyClass
{
int port = 9100;
string _deviceip;
byte[] m_ReadBuffer = null;
TcpClient _tcpclient;
NetworkStream m_NetworkStream;
static string CRLF = "\r\n";
public event EventHandler<DeviceStatus> onReceiveUpdate;
public async Task<bool> UploadFirmware(Stream _stream)
{
bool success = false;
try
{
_tcpclient = new TcpClient();
_tcpclient.Connect(_deviceip, port);
_stream.Seek(0, SeekOrigin.Begin);
m_NetworkStream = _tcpclient.GetStream();
byte[] buffer = new byte[1024];
m_ReadBuffer = new byte[1024];
int readcount = 0;
m_NetworkStream.BeginRead(m_ReadBuffer, 0, m_ReadBuffer.Length,
new AsyncCallback(EndReceive), null);
await Task.Run(() =>
{
while ((readcount = _stream.Read(buffer, 0, buffer.Length)) > 0)
{
m_NetworkStream.Write(buffer, 0, readcount);
m_NetworkStream.Flush();
}
});
success = true;
}
catch (Exception ex)
{
upgradeStatus = false;
}
return success;
}
private void EndReceive(IAsyncResult ar)
{
try
{
int nBytes;
nBytes = m_NetworkStream.EndRead(ar);
if (nBytes > 0)
{
string res = Encoding.UTF8.GetString(m_ReadBuffer, 0, nBytes);
DeviceStatus status = new DeviceStatus();
string[] readlines = res.Split(new string[] { CRLF },
StringSplitOptions.RemoveEmptyEntries);
foreach (string readline in readlines)
{
if (readline.StartsWith("CODE"))
{
//read readline string here
break;
}
}
}
if (m_NetworkStream.CanRead)
{
do
{
m_NetworkStream.BeginRead(m_ReadBuffer, 0, m_ReadBuffer.Length, new
AsyncCallback(EndReceive), null);
} while (m_NetworkStream.DataAvailable);
}
}
catch (ObjectDisposedException ods)
{
return;
}
catch (System.IO.IOException ex)
{
}
}
}
Any help will be really appreciated.

Your code is basically fine with a few issues:
m_NetworkStream.Flush(); AFAIK this does nothing. If it did something it would harm throughput. So delete that.
_stream.Seek(0, SeekOrigin.Begin); seeking is the callers concern, remove that. This is a layering violation.
Use bigger bigger buffers. Determine the right size experimentally. I usually start at 64KB for bulk transfers. This makes the IOs less chatty.
Turn on nagling which helps with bulk transfers because it saves you from spurious small packets.
You can replace the entire read-write-loop with Stream.Copy.
The way you report exceptions to the callers hides a lot of information. Just let the exception bubble out. Don't return a bool.
Use using for all resource to ensure they are cleaned up in the error case.
nBytes = m_NetworkStream.EndRead(ar); here, you assume that a single read will return all data that will be coming. You might receive just the first byte, though. Probably, you should use StreamReader.ReadLine in a loop until you know you are done.
catch (System.IO.IOException ex) { } What is that about? If firmware updates are such a critical thing suppressing errors appears very dangerous. How else can you find out about bugs?
I would convert the reading code to use await.

As the maximum length for a TcpPacket is 65535 (2^16-1), if you send any packet breaching this lenght it will be truncated. If I were you, I think the best way of sending large packets, is setting a Header of every packet and enumerating them. For example:
C->S ; [P1] <content>
and then the same structure, just plus 1 [P2] <content>
To do so, just use few substrings to truncate the data and sending them.
Cheers!

SSLStream reads invalid data + KB3147458 SSLStream bug (?)

I'm having an issue with SSLStream returning some data when the remote client did not send anything. I am having this issue when the server is listening for a new command.
If the server doesn't receive a new request, the ReadMessage() function should catch an IOException due to the Read timeout of the SSLStream. The problem happens when the sslStream.Read() is executed the second time it seems to read 5 bytes which were not sent by the client. So the problem happens in this sequence:
-> ReadMessage() -> sslstream.Read() -> timeout exception caught as expected
-> ReadMessage() -> sslstream.Read() -> timeout exception NOT caught, 5 bytes read even even though the client did not send anything
-> ReadMessage() -> sslstream.Read() -> timeout exception caught as expected
-> ReadMessage() -> sslstream.Read() -> timeout exception NOT caught, 5 bytes read even though client did not send anything...
and so on..
public void ClientHandle(object obj)
{
nRetry = MAX_RETRIES;
// Open connection with the client
if (Open() == OPEN_SUCCESS)
{
String request = ReadMessage();
String response = null;
// while loop for the incoming commands from client
while (!String.IsNullOrEmpty(request))
{
Console.WriteLine("[{0}] {1}", RemoteIPAddress, request);
response = Execute(request);
// If QUIT was received, close the connection with the client
if (response.Equals(QUIT_RESPONSE))
{
// Closing connection
Console.WriteLine("[{0}] {1}", RemoteIPAddress, response);
// Send QUIT_RESPONSE then return and close this thread
SendMessage(response);
break;
}
// If another command was received, send the response to the client
if (!response.StartsWith("TIMEOUT"))
{
// Reset nRetry
nRetry = MAX_RETRIES;
if (!SendMessage(response))
{
// Couldn't send message
Close();
break;
}
}
// Wait for new input request from client
request = ReadMessage();
// If nothing was received, SslStream timeout occurred
if (String.IsNullOrEmpty(request))
{
request = "TIMEOUT";
nRetry--;
if (nRetry == 0)
{
// Close everything
Console.WriteLine("Client is unreachable. Closing client connection.");
Close();
break;
}
else
{
continue;
}
}
}
Console.WriteLine("Stopped");
}
}
public String ReadMessage()
{
if (tcpClient != null)
{
int bytes = -1;
byte[] buffer = new byte[MESSAGE_SIZE];
try
{
bytes = sslStream.Read(buffer, 0, MESSAGE_SIZE);
}
catch (ObjectDisposedException)
{
// Streams were disposed
return String.Empty;
}
catch (IOException)
{
return String.Empty;
}
catch (Exception)
{
// Some other exception occured
return String.Empty;
}
if (bytes != MESSAGE_SIZE)
{
return String.Empty;
}
// Return string read from the stream
return Encoding.Unicode.GetString(buffer, 0, MESSAGE_SIZE).Replace("\0", String.Empty);
}
return String.Empty;
}
public bool SendMessage(String message)
{
if (tcpClient != null)
{
byte[] data = CreateMessage(message);
try
{
// Write command message to the stream and send it
sslStream.Write(data, 0, MESSAGE_SIZE);
sslStream.Flush();
}
catch (ObjectDisposedException)
{
// Streamers were disposed
return false;
}
catch (IOException)
{
// Error while trying to access streams or connection timedout
return false;
}
catch (Exception)
{
return false;
}
// Data sent successfully
return true;
}
return false;
}
private byte[] CreateMessage(String message)
{
byte[] data = new byte[MESSAGE_SIZE];
byte[] messageBytes = Encoding.Unicode.GetBytes(message);
// Can't exceed MESSAGE_SIZE parameter (max message size in bytes)
if (messageBytes.Length >= MESSAGE_SIZE)
{
throw new ArgumentOutOfRangeException("message", String.Format("Message string can't be longer than {0} bytes", MESSAGE_SIZE));
}
for (int i = 0; i < messageBytes.Length; i++)
{
data[i] = messageBytes[i];
}
for (int i = messageBytes.Length; i < MESSAGE_SIZE; i++)
{
data[i] = messageBytes[messageBytes.Length - 1];
}
return data;
}
The very same ReadMessage(), SendMessage() and CreateMessage() functions are used also by the client to send messages to the server. MESSAGE_SIZE constant is also the same and it's set to 2048.

The problem was that I re-used the SSLStream after a timeout. So I solved the problem just by removing the nRetry variable and set a longer timeout. The related MSDN article says that SSLStream will return garbage after a timeout exception (https://msdn.microsoft.com/en-us/library/system.net.security.sslstream(v=vs.110).aspx):
SslStream assumes that a timeout along with any other IOException when one is thrown from the inner stream will be treated as fatal by its caller. Reusing a SslStream instance after a timeout will return garbage. An application should Close the SslStream and throw an exception in these cases.
Another issue is that Windows update KB3147458 (Windows 10 April's update) changes the something in the behaviour of the Read function. It looks like something in the SSLStream implementation changed and now it returns data in 2 parts, 1 byte and the rest of the bytes every single time. Actually the MSDN document doesn't say that the Read() function will return all the requested bytes in one step and the provided example uses a do-while loop in order to read the exact number of bytes. So I suppose that the Read() function doesn't guarantee to read the exact requested number of bytes all at once, more read iterations might be required.
SSLstream works properly so it's NOT BROKEN. You just need to pay attention and use of a do-while loop and check that all the bytes are read correctly.
I changed the code as shown here to address the bugs I had.
public String ReadMessage()
{
if (tcpClient != null)
{
int bytes = -1, offset = 0;
byte[] buffer = new byte[MESSAGE_SIZE];
try
{
// perform multiple read iterations
// and check the number of bytes received
while (offset < MESSAGE_SIZE)
{
bytes = sslStream.Read(buffer, offset, MESSAGE_SIZE - offset);
offset += bytes;
if (bytes == 0)
{
return String.Empty;
}
}
}
catch (Exception)
{
// Some exception occured
return String.Empty;
}
if (offset != MESSAGE_SIZE)
{
return String.Empty;
}
// Return string read from the stream
return Encoding.Unicode.GetString(buffer, 0, MESSAGE_SIZE).Replace("\0", String.Empty);
}
return String.Empty;
}

With regard to the SslStream returning five bytes on a Read() after a timeout, this is because the SslStream class doesn't gracefully handle any IOException from the underlying stream, and this is documented as previously noted.
SslStream assumes that a timeout along with any other IOException when one is thrown from the inner stream will be treated as fatal by its caller. Reusing a SslStream instance after a timeout will return garbage. An application should Close the SslStream and throw an exception in these cases.
https://msdn.microsoft.com/en-us/library/system.net.security.sslstream(v=vs.110).aspx
However, you can fix the problem by creating a wrapper class that sits between the Tcp NetworkStream and the SslStream which catches and suppresses the harmless timeout exception, with (seemingly) no loss of functionality.
The full code of this is in my answer on a similar thread, here https://stackoverflow.com/a/48231248/8915494
With regard to the Read() method returning only part of the payload on each Read(), your answer already fixes this correctly. While this is "recent" behaviour for SslStream, it is unfortunately expected behaviour for all networking and all code needs to create some form of buffer to store the fragments until you have a complete packet. For example, if your data exceeds 1500 bytes (the maximum packet size for most Ethernet adapters, assuming Ethernet transport), you are very likely to receive the data in multiple parts and have to reassemble it yourself.
Hope this helps

How to correctly read zipped data from NetworkStream with BeginRead/EndRead?

Here is the AsyncCallback that I'm using to read data from GZipStream created on top of NetworkStream:
void ReadCompressedDataCallback(IAsyncResult ar)
{
var state = ar.AsyncState as ReadCompressedDataState;
try
{
switch (state.State)
{
case EReadCompressedDataState.BeginRead:
state.State = EReadCompressedDataState.EndRead;
state.InputStream.BeginRead(state.Buf, 0, state.Buf.Length, ReadCompressedDataCallback, state);
break;
case EReadCompressedDataState.EndRead:
var read = state.InputStream.EndRead(ar);
if (read==0)
{
state.OnCompleted();
return;
}
state.Result.Write(state.Buf,0,read);
state.State = EReadCompressedDataState.BeginRead;
ReadCompressedDataCallback(ar);
break;
default:
throw new ArgumentOutOfRangeException();
}
}
catch (Exception e)
{
state.LastException = e;
state.OnCompleted();
}
}
state.InputStream is something like new GZipStream(NetworkStream).
My problem is that first time I call read = state.InputStream.EndRead(ar);, read is always 0, and the reading stops. When, in the debugger, I start reading again, some data is read.
According to documentation I've read, NetworkStream returns 0 bytes on reading only when there are no more bytes to read, yet here 0 bytes is returned and than some data follows on the next read.
How do I correctly read all gzipped data without risking "deadlocking" the app into waiting for data that never arrives?

Your sample code looks a bit unusual.
I think another answer might help you Calling BeginRead from a NetworkStream

C# TCP Server Reply Packet Issue

I have been banging my head against my code for the better part of the day, and I am completely stumped. Basically, the source game engine has a documented protocol for its RCON (Remote Console Over Network?) which I am trying to reproduce. There are hundreds of examples, but all of them are from the client side (establishing a connection to the game server's RCON) where as I am trying to actually re-create the server portion to reply to clients.
Here is the information on the RCON Protocol. The problem I am having with the code is, when I receive the Authentication request everything is fine. When I attempt to reply to it and okay the connection, the connection fails. So I am doing something wrong when replying but not sure what.
http://developer.valvesoftware.com/wiki/Source_RCON_Protocol
private void ReadClientPacket(object client)
{
TcpClient tcpClient = (TcpClient)client;
NetworkStream clientStream = tcpClient.GetStream();
while (true)
{
try
{
int packetsize;
// Create a new Packet Object and fill out the data from the incoming TCP Packets
RCONPacket packet = new RCONPacket();
using (BinaryReader reader = new BinaryReader(clientStream))
{
// First Int32 is Packet Size
packetsize = reader.ReadInt32();
packet.RequestId = reader.ReadInt32();
packet.RconDataReceived = (RCONPacket.RCONDATA_rec)reader.ReadInt32();
Console.WriteLine("Packet Size: {0} RequestID: {1} ServerData: {2}", packetsize, packet.RequestId, packet.RconDataReceived);
// Read first and second String in the Packet (UTF8 Null Terminated)
packet.String1 = ReadBytesString(reader);
packet.String2 = ReadBytesString(reader);
Console.WriteLine("String1: {0} String2: {1}", packet.String1, packet.String2);
switch (packet.RconDataReceived)
{
case RCONPacket.RCONDATA_rec.SERVERDATA_AUTH:
{
ReplyAuthRequest(packet.RequestId, tcpClient);
break;
}
case RCONPacket.RCONDATA_rec.SERVERDATA_EXECCOMMAND:
{
//ReplyExecCommand(packet.RequestId, tcpClient);
break;
}
default:
{
break;
}
}
}
break;
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
break;
}
}
tcpClient.Close();
}
private void ReplyAuthRequest(int RequestID, TcpClient client)
{
Console.WriteLine("Replying to Auth Request");
// Authentication Reply
using (NetworkStream clientStream = client.GetStream())
using (MemoryStream stream = new MemoryStream())
using (BinaryWriter writer = new BinaryWriter(stream))
{
writer.Write((int)10); // Packet Size
writer.Write(RequestID); // Mirror RequestID if Authenticated, -1 if Failed
writer.Write((int)RCONPacket.RCONDATA_sent.SERVERDATA_AUTH_RESPONSE);
writer.Write(ConvertStringToByteArray("" + char.MinValue));
writer.Write(ConvertStringToByteArray("" + char.MinValue));
byte[] buffer = stream.ToArray();
Console.WriteLine("size of full auth response packet is {0}", buffer.Length);
clientStream.Write(buffer, 0, buffer.Length);
clientStream.Flush();
}
}

Have you been using Wireshark?
This tool is essential when you try to copy/reverse-engineer existing protocols. Just do a normal authentication with a known-working client and save the log. Then use your own code and try to see where the bits sent on the wire are different from those in the log.
Sometimes its pretty difficult things to see just by looking at the code, like a '\n' getting inserted at the wrong point, or an extra line after the whole message

I've had a similar problem a while ago
In your Reply function :
using (NetworkStream clientStream = client.GetStream())
(...)
The disposal of the clientStream could be the culprit. In my case the disposal of the stream caused the connection termination, GetStream() doesn't return a new instance of a stream, it returns the Stream that is owned by the TCPClient. See if that helps.