Develop Reference

Read from C# Stream without busy waiting

I need to read content from Stream in C#. I do not know which kind of Stream it would be. Let's assume it is Network Stream and network is very slow. So I can't read all bytes immediately. I know that 4 bytes in stream is integer and this int defines content length that we need to get from stream. How to do it without busy waiting (looping)?
private (int headerValue, bool isSuccess) ReadHeader()
{
var bytesRead = 0;
var headerBuffer = new byte[BufferHeaderLength];
var headerIsReady = false;
while (!headerIsReady)
{
try
{
var availableBytesToRead = Math.Min(BufferHeaderLength - bytesRead, underlyingStream.Length); // should be a number from 0 to 4
bytesRead += underlyingStream.Read(headerBuffer, bytesRead, (int)availableBytesToRead);
if (bytesRead == 0)
{
break;
}
}
catch (Exception)
{
break;
}
headerIsReady = bytesRead == BufferHeaderLength;
}
var headerValue = headerIsReady ? BitConverter.ToInt32(headerBuffer, 0) : 0;
return (headerValue, headerIsReady);
}

Is it possible to use asynchronous receiver for unknown number of received bytes?

My app is built on C# synchronous sockets, and I am trying to replace them with asynchronous ones, in order to improve efficiency (if possible). So far, I have replaced on the server the Send() and Accept() with the asynchronous ones (based on the MSDN documents), but I have some issues to implement the Receive().
Original (synchronous):
Socket _socket = new Socket(SocketType.Stream, ProtocolType.Tcp);
bool _flag = true;
CheckFun()
{
while (_flag)
{
Thread.Sleep(10); // every ~10ms check
byte[] _buffer;
if (_socket.Available != 0)
{
_buffer = new byte[1]; // just check 1st byte
_socket.Receive(_buffer, 1, SocketFlags.None);
if (_buffer[0] == 1)
{
ReceiveFun();
}
}
}
}
ReceiveFun()
{
int hdrSize = 12;
byte[] buffer = new byte[hdrSize];
_socket.Receive(buffer, hdrSize, SocketFlags.None);
int _dataLength = buffer[0]; // the 1st byte has already been removed before calling the ReceiveFun()
buffer = new byte[_dataLength];
int _bytesRead = 0;
while (_bytesRead != _dataLength)
{
while (_socket.Available == 0)
{
if (!_socket.Connected)
return 0;
}
_bytesRead += _socket.Receive(buffer, _bytesRead, _dataLength - _bytesRead, SocketFlags.None);
}
//read bytes blah blah...
}
My question is how to turn this operation into an asynchronous one and concatenate the bytes received until I receive all information? And then again wait for the next one?
EDIT
Async
public class State
{
public int nToReadBytes = 0;
public int nBytesRead = 0;
public const int BufferSize = 1024;
public byte[] buffer = new byte[BufferSize]; // Receive buffer.
}
List<byte> lReceivedBytes = new List<byte>();
int hdrSize = 12;
public void ReadCallback(IAsyncResult ar)
{
// Retrieve the state object and the handler socket
// from the asynchronous state object.
var state = ar.AsyncState as State;
// Read data from the client socket.
int availableBytes = oSocket.EndReceive(ar);
if (availableBytes > 0)
{
if (lReceivedBytes.Count == 0)
{
if (state.buffer[0] == 1)
{
// the first field of the header has been successfully decoded
if (availableBytes > 1)
{
state.nToReadBytes = BitConverter.ToInt32(state.buffer, 1) + hdrSize;
int _bytesCopy = Math.Min(state.nToReadBytes, state.buffer.Length); //in case that the data is less than the State.BufferSize (unlikely)
state.nBytesRead += _bytesCopy;
lReceivedBytes.AddRange(state.buffer);
}
}
else if (state.buffer[0] == 2)
{
// the first field of the header has been successfully decoded but do nothing!
_socket.BeginReceive(state.buffer, 0, State.BufferSize, 0, new AsyncCallback(ReadCallback), state);
return;
}
else
throw new InvalidDataException("Invalid hdr field [1-2]: " + state.buffer[0]);
}
else
{
state.nBytesRead += state.buffer.Length;
lReceivedBytes.AddRange(state.buffer);
}
if (lReceivedBytes.Count == state.nToReadBytes)
{
//read all information and clear list and States in the end (?)
// ...
lReceivedBytes.Clear();
state.nToReadBytes = 0;
state.nBytesRead = 0;
_socket.BeginReceive(state.buffer, 0, State.BufferSize, 0, new AsyncCallback(ReadCallback), state);
}
else
{
//int _newSize = Math.Min(state.nToReadBytes - state.nBytesRead, State.BufferSize); // for restriction (?)
int _newSize = state.nToReadBytes - state.nBytesRead; // for now don't check
_socket.BeginReceive(state.buffer, 0, _newSize, 0, new AsyncCallback(ReadCallback), state); //shall I increase the size (it could be between 90 kB - 170kB, until all info is received)
}
}
else
_socket.BeginReceive(state.buffer, 0, State.BufferSize, 0, new AsyncCallback(ReadCallback), state);
}

The Socket class has two main asynchronous paradigms: the original callback-based Asynchronous Programming Model (APM) and the only-slightly-newed Event-based Asynchronous Pattern (EAP). Both of these can be somewhat unwieldy to implement as compared to the synchronous appproach, as they require you to adjust your thinking to connection state instead of just local method logic, breaking up what would be a single method into parts dealing with the initiation and completion of the operation.
Fortunately, the newer Task Parallel Library model leverages async and await to allow asynchronous code to be written in almost identical fashion to the equivalent synchronous version. For the Socket class, to take advantage of this, you need to wrap its asynchronous API in a TPL-compatible way. .NET does provide a general-purpose way to take existing APM APIs and wrap them in tasks (see https://learn.microsoft.com/en-us/dotnet/standard/parallel-programming/tpl-and-traditional-async-programming), but IMHO it is simpler to take advantage of the NetworkStream class, which wraps a socket in a Stream object.
Since Stream already has received the "TPL love", i.e. has ReceiveAsync() and similar methods to allow TPL-based operations, I find this somewhat easier than dealing with the wrapper methods designed to map APM to TPL.
In your code, that would wind up looking something like this:
// Somewhere appropriate -- your code example isn't specific or complete enough
// to offer anything more detailed than this
NetworkStream _stream = new NetworkStream(_socket);
async Task ReceiveFun()
{
int _bytesRead = 0, hdrSize = 12;
byte[] buffer = new byte[hdrSize];
while (_bytesRead < hdrSize)
{
int bytesRead = await _stream.ReadAsync(buffer, _bytesRead, hdrSize - _bytesRead);
if (bytesRead == 0) throw new InvalidDataException("unexpected end-of-stream");
_bytesRead += bytesRead;
}
int _dataLength = buffer[0]; // the 1st byte has already been removed before calling the ReceiveFun()
buffer = new byte[_dataLength];
_bytesRead = 0;
while (_bytesRead < _dataLength)
{
int bytesRead = await _stream.ReadAsync(buffer, _bytesRead, _dataLength - _bytesRead);
if (bytesRead == 0) throw new InvalidDataException("unexpected end-of-stream");
_bytesRead += bytesRead;
}
//read bytes blah blah...
}
Note that your original synchronous code had a bug: you were not using the returned byte count when reading the header, and so had no way to know if the full header was actually being returned in a single call or not. You must always look at the count of bytes read; this can always be as few as 1 when data is available, and of course will be 0 when the end of stream is reached (i.e. the remote endpoint uses the shutdown operation).
Your code also did not check for end-of-stream correctly. Just look at the bytes returned when you try to read from the socket. I've fixed that in the above as well.

Read asynchronously data from NetworkStream with huge amount of packets

In my application every packet has 2 bytes length on the start. However after some time application starts receiving length less than zero. In synchronous client everything works correctly, but it's too slow. I'm 100% sure in Server everything is correct.
Connect:
public void Connect(IPAddress ip, int port)
{
tcpClient.Connect(ip, port);
stream = tcpClient.GetStream();
byte[] len_buffer = new byte[2];
stream.BeginRead(len_buffer, 0, len_buffer.Length, OnDataRead, len_buffer);
}
OnDataRead:
private void OnDataRead(IAsyncResult ar)
{
byte[] len = ar.AsyncState as byte[];
int length = BitConverter.ToInt16(len, 0);
byte[] buffer = new byte[length];
int remaining = length;
int pos = 0;
while (remaining != 0)
{
int add = stream.Read(buffer, pos, remaining);
pos += add;
remaining -= add;
}
Process(buffer);
len = new byte[2];
stream.EndRead(ar);
stream.BeginRead(len, 0, len.Length, OnDataRead, len);
}

As I can see, you're mixing up synchronious and asynchronious. That's a bad practice.
What you want is something like:
var header = ReadHeader(); // 2 bytes
var data = ReadData(header.DataSize);
I didn't use the network stream, but....
Here's an example of my async SocketReader:
public static class SocketReader
{
// This method will continues read until count bytes are read. (or socket is closed)
private static void DoReadFromSocket(Socket socket, int bytesRead, int count, byte[] buffer, Action<ArraySegment<byte>> endRead)
{
// Start a BeginReceive.
try
{
socket.BeginReceive(buffer, bytesRead, count - bytesRead, SocketFlags.None, (asyncResult) =>
{
// Get the bytes read.
int read = 0;
try
{
// if this goes wrong, the read remains 0
read = socket.EndReceive(asyncResult);
}
catch (ObjectDisposedException) { }
catch (Exception exception)
{
Trace.TraceError(exception.Message);
}
// if zero bytes received, the socket isn't available anymore.
if (read == 0)
{
endRead(new ArraySegment<byte>(buffer, 0, 0));
return;
}
// increase the bytesRead, (position within the buffer)
bytesRead += read;
// if all bytes are read, call the endRead with the buffer.
if (bytesRead == count)
// All bytes are read. Invoke callback.
endRead(new ArraySegment<byte>(buffer, 0, count));
else
// if not all bytes received, start another BeginReceive.
DoReadFromSocket(socket, bytesRead, count, buffer, endRead);
}, null);
}
catch (Exception exception)
{
Trace.TraceError(exception.Message);
endRead(new ArraySegment<byte>(buffer, 0, 0));
}
}
public static void ReadFromSocket(Socket socket, int count, Action<ArraySegment<byte>> endRead)
{
// read from socket, construct a new buffer.
DoReadFromSocket(socket, 0, count, new byte[count], endRead);
}
public static void ReadFromSocket(Socket socket, int count, byte[] buffer, Action<ArraySegment<byte>> endRead)
{
// if you do have a buffer available, you can pass that one. (this way you do not construct new buffers for receiving and able to reuse buffers)
// if the buffer is too small, raise an exception, the caller should check the count and size of the buffer.
if (count > buffer.Length)
throw new ArgumentOutOfRangeException(nameof(count));
DoReadFromSocket(socket, 0, count, buffer, endRead);
}
}
Usage:
SocketReader.ReadFromSocket(socket, 2, (headerData) =>
{
if(headerData.Count == 0)
{
// nothing/closed
return;
}
// Read the length of the data.
int length = BitConverter.ToInt16(headerData.Array, headerData.Offset);
SocketReader.ReadFromSocket(socket, length, (dataBufferSegment) =>
{
if(dataBufferSegment.Count == 0)
{
// nothing/closed
return;
}
Process(dataBufferSegment);
// extra: if you need a binaryreader..
using(var stream = new MemoryStream(dataBufferSegment.Array, dataBufferSegment.Offset, dataBufferSegment.Count))
using(var reader = new BinaryReader(stream))
{
var whatever = reader.ReadInt32();
}
}
});
You can optimize the receive buffer by passing a buffer (look at the overloads)
Continues receiving: (reusing receivebuffer)
public class PacketReader
{
private byte[] _receiveBuffer = new byte[2];
// This will run until the socket is closed.
public void StartReceiving(Socket socket, Action<ArraySegment<byte>> process)
{
SocketReader.ReadFromSocket(socket, 2, _receiveBuffer, (headerData) =>
{
if(headerData.Count == 0)
{
// nothing/closed
return;
}
// Read the length of the data.
int length = BitConverter.ToInt16(headerData.Array, headerData.Offset);
// if the receive buffer is too small, reallocate it.
if(_receiveBuffer.Length < length)
_receiveBuffer = new byte[length];
SocketReader.ReadFromSocket(socket, length, _receiveBuffer, (dataBufferSegment) =>
{
if(dataBufferSegment.Count == 0)
{
// nothing/closed
return;
}
try
{
process(dataBufferSegment);
}
catch { }
StartReceiving(socket, process);
});
});
}
}
Usage:
private PacketReader _reader;
public void Start()
{
_reader = new PacketReader(socket, HandlePacket);
}
private void HandlePacket(ArraySegment<byte> packet)
{
// do stuff.....
}

Reading all data from stream of TcpClient

I am working on a new machine at my works which is controlled via a PC. Current stuck on talking to a camera system that is connected via Ethernet.
Here is my code of opening a connection.
TcpClient client;
NetworkStream stream;
public bool OpenConnection()
{
client = new TcpClient();
try
{
//Camera.Open();
client.Connect("192.168.0.10", 8500);
stream = client.GetStream();
}
catch (Exception ex)
{
return false;
}
return true;
}
So far so good. Once the connection is open I then request some information from the camera
The message format I am using has STX(0x02) to indicate the start of the message and ETX(0x03) as the end of the message.
char StartOfPacket = (char)0x02;
char EndOfPacket= (char)0x03;
public bool RetrieveDigits(out string Digits)
{
// Send the trigger cammand to the vision system
Digits = "";
bool EverythingOK = true;
string DataToSend = StartOfPacket + "T1" + EndOfPacket;
byte[] buff = Encoding.ASCII.GetBytes(DataToSend);
if (LogCameraEvent != null)
LogCameraEvent(">> " + DataToSend);
try
{
stream.Write(buff, 0, buff.Length);
}
catch (Exception ex)
{
Logging.Instance.LogExceptionToFile(MethodBase.GetCurrentMethod(), ex);
EverythingOK = false;
}
Thread.Sleep(100);
byte[] buffer;
if (EverythingOK)
{
// Check the response
buffer = ReadCamera(10);
// Process the packets
string[] packets = ProcessPackets(buffer);
if (packets != null)
{
if (packets.Length > 0)
{
bool TriggerFound = false;
for (int i = 0; i < packets.Length; i++)
{
if (packets[i] == "T1")
{
TriggerFound = true;
continue;
}
else if (TriggerFound)
{
// If we are here then we should now be the data that was requested
if (string.IsNullOrEmpty(packets[i]))
{
Digits = packets[i-1]; // previous packet may have data from previous trigger. Need to look into why this happens.
}
else
Digits = packets[i];
EverythingOK = true;
break;
}
else
EverythingOK = false;
}
}
else
{
Console.WriteLine("No Packets Recieved");
EverythingOK = false;
}
}
else
{
Console.WriteLine("No Packets Recieved");
EverythingOK = false;
}
}
return EverythingOK;
}
Here is the part where I think the issue may lie, getting the response.
private byte[] ReadCamera(int ExpectedLength)
{
if(ExpectedLength < 1)
{
ExpectedLength = 100;
}
byte[] Buffer = new byte[ExpectedLength];
int read = 0;
int chunk;
while(stream.DataAvailable)
{
chunk = stream.Read(Buffer, read, Buffer.Length-read);
read += chunk;
// If we have reached the end of our buffer, check to see if theres any more information
if(read == Buffer.Length)
{
if(!stream.DataAvailable)
return Buffer;
// Nope. Resize the buffer, put the byte we've just read and continue
byte[] newBuffer = new byte[Buffer.Length * 2];
Array.Copy(Buffer, newBuffer, Buffer.Length);
Buffer = newBuffer;
//read++;
}
else if(!stream.DataAvailable)
return Buffer;
Thread.Sleep(50);
}
// Buffer is now too big, shrink it
byte[] ret = new byte[read];
Array.Copy(Buffer, ret, read);
return ret;
}
private string[] ProcessPackets(byte[] data)
{
// look for the stat char
List<string> Packets = new List<string>();
byte Start = 0x02;
bool StartFound = false;
byte End = 0x03;
StringBuilder sb = new StringBuilder();
for(int i =0; i<data.Length; i++)
{
if(StartFound)
{
// Check to see if its the end
if (data[i] == End)
{
Packets.Add(sb.ToString());
sb.Clear();
}
else
sb.Append(Encoding.ASCII.GetChars(data, i, 1));
}
// Find the start
if (!StartFound)
{
if (data[i] == Start)
{
StartFound = true;
}
}
}
return Packets.ToArray();
}
Let me explain what should happen and then what is happening. Through RetrieveDigits method I am sending "T1" to the camera. What the camera will do is respond with "T1" followed by 2 ascii characters, for now we will say AA. Using a diagnostic app on the camera system I can monitor the Ethernet and I see the following.
(>> means Received by Camera)
(<< means Sent from Camera)
>>[STX]T1[ETX]
<<[STX]T1[ETX][STX]AA[ETX]
So I see what the camera has sent. I confirmed the PC has has received the data using wire shark.
Now further down in the RetrieveDigits method you can see I process the packets received, loop through the packets until I find "T1" and then assume the next packet will be the data I am after and I set Digits to this value.
What I am finding is that sometimes when I run the app I see that Digits is set to "".
I am also finding that sometimes my data received will be "AA" then "T1" rather than "T1" then "AA". What I suspect is happening is that when its back to front, the "AA" is actually from the previous data sent from the camera and for some reason it was missed when reading from the stream.
Any idea why this could be happening as I am reading the data until Stream.Available is false.
Edit:
Modified the ReadCamera code to process the packets, reading 1 byte at a time.
private string[] ReadCamera(int ExpectedLength, int ExpectedPackets)
{
List<string> Packets = new List<string>();
bool StartFound = false;
StringBuilder sb = new StringBuilder();
if(ExpectedLength < 1)
{
ExpectedLength = 100;
}
byte[] Buffer = new byte[ExpectedLength];
int read = 0;
while (true)
{
read += stream.Read(Buffer, read, 1);
// Check to see if the byte read is the start of a packet
if (Buffer[read - 1] == StartOfPacket)
{
StartFound = true;
}
else if (StartFound)
{
// Check to see if the byte read is the end of a packet
if (Buffer[read - 1] == EndOfPacket)
{
Packets.Add(sb.ToString());
sb.Clear();
StartFound = false;
if (Packets.Count == ExpectedPackets)
break;
}
else
{
sb.Append(Encoding.ASCII.GetChars(Buffer, read - 1, 1));
}
}
}
// For Debuggin purposes
foreach(string s in Packets)
if (LogCameraEvent != null)
LogCameraEvent("<< " + s);
return Packets.ToArray();
}
and modified calling the method like so
// Check the response
string[] packets = ReadCamera(10,2);
// Process the packets
//string[] packets = ProcessPackets(buffer);

Sending lots data disconnects the server

I'm building a network library to use in further projects, I know a bit of networking (in general and programming) but I still encounter problems when I want to transfer larger packets on a high frequency. How would this be possible in TCP?
One possible solution would be chuncking, that is not a problem at all I'll just put them in a buffer and have the whole packet again. This causes overhead at the server, handling lots of data is consuming lots of resources. And it affects the stability because the server get somewhere stuck and the client throws an ObjectDisposedException at BeginSend (connection is closed). Of course its probably my fault but consider a 'relative' small packet of 2048 bytes (2 kB) which get chunked in 512 bytes (4 chunks):
//main.cs
static void Main(string[] args)
{
client = new Client("82.72.201.150", 2345);
client.SendCompleted += OnSend;
Console.WriteLine(client.Connect());
var bytes = new byte[2048];
for (int i = 0; i < bytes.Length; i++)
{
bytes[i] = (byte)((i*i + 0x25)%byte.MaxValue);
}
while (true)
{
SendChuncked(bytes);
Thread.Sleep(500);
}
}
static void SendChuncked(byte[] buffer, int chunksize = 512)
{
var chunks = buffer.Length/chunksize;
var rest = buffer.Length%chunksize;
var size = BitConverter.GetBytes(buffer.Length);
client.Send(size, 0, size.Length);
for (int i = 0; i < chunks; i++)
{
client.Send(buffer, i * chunksize, chunksize);
}
if (rest > 0)
client.Send(buffer, chunks * chunksize, rest);
}
//....
//client.cs
public struct TransferState
{
public byte[] buffer;
public int offset;
public int count;
public int handled;
public ManualResetEvent waitHandle;
public bool wait;
}
public void Send(byte[] buffer, int offset, int count)
{
var size = BitConverter.GetBytes(count);
Send(size, 0, size.Length, true);
Send(buffer, offset, count, false);
}
private void Send(byte[] buffer, int offset, int count, bool wait, TransferState? state = null)
{
state = state ?? new TransferState
{
buffer = buffer,
offset = offset,
count = count,
handled = 0,
wait = wait,
waitHandle = new ManualResetEvent(false)
};
socket.BeginSend(buffer, offset, count, SocketFlags.None, SendCallback, state);
if (wait)
{
((TransferState) state).waitHandle.WaitOne();
}
}
private void Send(byte[] buffer, int offset, int count, bool wait, TransferState? state = null)
{
state = state ?? new TransferState
{
buffer = buffer,
offset = offset,
count = count,
handled = 0,
wait = wait,
waitHandle = new ManualResetEvent(false)
};
socket.BeginSend(buffer, offset, count, SocketFlags.None, SendCallback, state);
if (wait)
{
((TransferState) state).waitHandle.WaitOne();
}
}
private void SendCallback(IAsyncResult result)
{
if (result.AsyncState is TransferState == false)
throw new ArgumentException("Invalid type of state.", "state");
var state = (TransferState)result.AsyncState;
var sent = socket.EndSend(result);
state.handled += sent;
var tosent = state.count - state.handled;
var offset = state.offset + state.handled;
if (tosent > 0)
{
Send(state.buffer, offset, tosent, state.wait, state);
}
else
{
state.waitHandle.Set();
SendCompleted(this, new TransferCompletedArgs(this, state));
}
}
//....
public void Receive(TransferState? state = null)
{
if (state == null)
{
var buffer = new byte[sizeof(int)];
socket.BeginReceive(buffer, 0, sizeof (int), SocketFlags.None, ReceiveCallback, buffer);
}
else
{
var transferState = state.Value;
socket.BeginReceive(transferState.buffer, transferState.offset, transferState.count - transferState.handled, SocketFlags.None, ReceiveCallback, transferState);
}
}
private void ReceiveCallback(IAsyncResult result)
{
//receiving the amount to receive
if (result.AsyncState is byte[])
{
var buffer = (byte[])result.AsyncState;
var rec = socket.EndReceive(result);
if (rec != 4) //TODO: HANDLE MORE PROPERLY
throw new NotImplementedException("Error while receiving the amoount to receive.");
var toreceive = BitConverter.ToInt32(buffer, 0);
var state = new TransferState
{
buffer = new byte[toreceive],
count = toreceive,
wait = false
};
Receive(state);
}
//know we know the amount we can receive it till the end
else if (result.AsyncState is TransferState)
{
var state = (TransferState)result.AsyncState;
var rec = socket.EndReceive(result);
state.offset += rec;
state.handled += rec;
var toreceive = state.count - state.handled;
if (toreceive > 0)
{
Receive(state);
Debug.WriteLine("[{2}] size mismatch: {0}/{1}", state.handled, state.count, DateTime.Now.ToString("mm:ss.fff"));
}
else
{
ReceiveCompleted(this, new TransferCompletedArgs(this, state));
Receive();
}
}
else
{
throw new ArgumentException("State is not typeof byte[] or TransferState.");
}
}
So if its hard to follow:
The client tries to connect with the server
The server accepts
the client
The client first sends how long the buffer is
The
client starts sending the buffer (and loops this)
The server reads
how long the buffer is and allocates a buffer for it
The server
reads until the buffer is read and invokes an event (and loops this
proces as well)
So I don't know what actually happens, but the server stops receiving. While connected clients still sending data, after couple seconds an ObjectDisposedException is thrown at BeginSend. How would I resolve this from happening? (And have a stable server that can handle lots of traffic.)