Develop Reference

In C#, what's the best way to deal with partially received messages using SocketAsyncEventArgs Buffer

In order to clean some messy code and get a better understanding of the SocketAsyncEventArgs class, I'd to know what's the most efficient technique to reassemble partially received messages from SocketAsyncEventArgs buffers.
To give you the big picture, I'm connected to a TCP server using a C# Socket client that will essentially receive data. The data received is message-based delimited by a \n character.
As you're probably already aware of, when using the ReceiveAsync method, this is almost a certitude that the last received message will be uncompleted such as you'll have to locate the index of the last complete message, copy the incomplete buffer section and keep it as start for the next received buffer and so on.
The thing is, I wish to abstract this operation from the upper layer and call the ProcessReceiveDataImpl as soon I get completed messages in the _tmpBuffer. I found that my Buffer.BlockCopy is not much readable (very old code also (-:) but anyway I wish to know what are you doing in this typical use case?
Code to reassemble messages:
public class SocketClient
{
private const int _receiveBufferSize = 8192;
private byte[] _remBuffer = new byte[2 * _receiveBufferSize];
private byte[] _tmpBuffer = new byte[2 * _receiveBufferSize];
private int _remBufferSize = 0;
private int _tmpBufferSize = 0;
private void ProcessReceiveData(SocketAsyncEventArgs e)
{
// the buffer to process
byte[] curBuffer = e.Buffer;
int curBufferSize = e.BytesTransferred;
int curBufferOffset = e.Offset;
int curBufferLastIndex = e.BytesTransferred - 1;
int curBufferLastSplitIndex = int.MinValue;
if (_remBufferSize > 0)
{
curBufferLastSplitIndex = GetLastSplitIndex(curBuffer, curBufferOffset, curBufferSize);
if (curBufferLastSplitIndex != curBufferLastIndex)
{
// copy the remain + part of the current into tmp
Buffer.BlockCopy(_remBuffer, 0, _tmpBuffer, 0, _remBufferSize);
Buffer.BlockCopy(curBuffer, curBufferOffset, _tmpBuffer, _remBufferSize, curBufferLastSplitIndex + 1);
_tmpBufferSize = _remBufferSize + curBufferLastSplitIndex + 1;
ProcessReceiveDataImpl(_tmpBuffer, _tmpBufferSize);
Buffer.BlockCopy(curBuffer, curBufferLastSplitIndex + 1, _remBuffer, 0, curBufferLastIndex - curBufferLastSplitIndex);
_remBufferSize = curBufferLastIndex - curBufferLastSplitIndex;
}
else
{
// copy the remain + entire current into tmp
Buffer.BlockCopy(_remBuffer, 0, _tmpBuffer, 0, _remBufferSize);
Buffer.BlockCopy(curBuffer, curBufferOffset, _tmpBuffer, _remBufferSize, curBufferSize);
ProcessReceiveDataImpl(_tmpBuffer, _remBufferSize + curBufferSize);
_remBufferSize = 0;
}
}
else
{
curBufferLastSplitIndex = GetLastSplitIndex(curBuffer, curBufferOffset, curBufferSize);
if (curBufferLastSplitIndex != curBufferLastIndex)
{
// we must copy the unused byte into remaining buffer
_remBufferSize = curBufferLastIndex - curBufferLastSplitIndex;
Buffer.BlockCopy(curBuffer, curBufferLastSplitIndex + 1, _remBuffer, 0, _remBufferSize);
// process the msg
ProcessReceiveDataImpl(curBuffer, curBufferLastSplitIndex + 1);
}
else
{
// we can process the entire msg
ProcessReceiveDataImpl(curBuffer, curBufferSize);
}
}
}
protected virtual void ProcessReceiveDataImpl(byte[] buffer, int bufferSize)
{
}
private int GetLastSplitIndex(byte[] buffer, int offset, int bufferSize)
{
for (int i = offset + bufferSize - 1; i >= offset; i--)
{
if (buffer[i] == '\n')
{
return i;
}
}
return -1;
}
}
Your input is very important and appreciated!
Thank you!
Updated:
Also, rather then calling the ProcessReceiveDataImpl and block further receive operations, will it be useful to queue completed messages and make them available to the consumer?

C# Socket ReceiveAsync

I am used to sync sockets and had a few headaches to get to the point where I am now, especially with Socket.Receive(..) not always receiveing all bytes
Here is my code what I used to use
public byte[] Receive(int size)
{
var buffer = new byte[size];
var r = 0;
do
{
// ReSharper disable once InconsistentlySynchronizedField
var c = _clientSocket.Receive(buffer, r, size - r, SocketFlags.None);
if (c == 0)
{
throw new SocketExtendedException();
}
r += c;
} while (r != buffer.Length);
return buffer;
}
Now I started to use sockets in Windows Phone BUT .Receive(..) is not available and I managed to get Socket.ReceiveAsync(..) working but I am concerned (no problems happened so far) here is my new code, I have not implemented the checking if all bytes has been recieved or not nor do I know if I have to with the following code
private byte[] ReadBySize(int size = 4)
{
var readEvent = new AutoResetEvent(false);
var buffer = new byte[size];
var recieveArgs = new SocketAsyncEventArgs()
{
UserToken = readEvent
};
recieveArgs.SetBuffer(buffer, 0, size);
recieveArgs.Completed += recieveArgs_Completed;
_connecter.ReceiveAsync(recieveArgs);
readEvent.WaitOne();
if (recieveArgs.BytesTransferred == 0)
{
if (recieveArgs.SocketError != SocketError.Success)
throw new SocketException((int)recieveArgs.SocketError);
throw new CommunicationException();
}
return buffer;
}
void recieveArgs_Completed(object sender, SocketAsyncEventArgs e)
{
var are = (AutoResetEvent)e.UserToken;
are.Set();
}
This is my first use of ReceiveAsync can someone point out anything I might have done wrong or need to change

Ok I went and took a large buffer and send it in batches with a sleep interval in between to replicate 'not all bytes received' So my code above doesn't recieve all bytes. for those who also use ReceiveAsync(..) here is my code that works
private byte[] ReadBySize(int size = 4)
{
var readEvent = new AutoResetEvent(false);
var buffer = new byte[size]; //Receive buffer
var totalRecieved = 0;
do
{
var recieveArgs = new SocketAsyncEventArgs()
{
UserToken = readEvent
};
recieveArgs.SetBuffer(buffer, totalRecieved, size - totalRecieved);//Receive bytes from x to total - x, x is the number of bytes already recieved
recieveArgs.Completed += recieveArgs_Completed;
_connecter.ReceiveAsync(recieveArgs);
readEvent.WaitOne();//Wait for recieve
if (recieveArgs.BytesTransferred == 0)//If now bytes are recieved then there is an error
{
if (recieveArgs.SocketError != SocketError.Success)
throw new ReadException(ReadExceptionCode.UnexpectedDisconnect,"Unexpected Disconnect");
throw new ReadException(ReadExceptionCode.DisconnectGracefully);
}
totalRecieved += recieveArgs.BytesTransferred;
} while (totalRecieved != size);//Check if all bytes has been received
return buffer;
}
void recieveArgs_Completed(object sender, SocketAsyncEventArgs e)
{
var are = (AutoResetEvent)e.UserToken;
are.Set();
}
The way I work with my Socket applications is to send a Buffer that consist of some variables
[0] -> 0,1,2 0 is keep alive, 1 means there are data, 2 means a type off error occured
[1,2,3,4] size of the actual buffer I am sending
[x(size of 1,2,3,4)] the actual 'Serialized' data buffer

You could create a socket extension like:
public static Task<int> ReceiveAsync(this Socket socket,
byte[] buffer, int offset, int size, SocketFlags socketFlags)
{
if (socket == null) throw new ArgumentNullException(nameof(socket));
var tcs = new TaskCompletionSource<int>();
socket.BeginReceive(buffer, offset, size, socketFlags, ar =>
{
try { tcs.TrySetResult(socket.EndReceive(ar)); }
catch (Exception e) { tcs.TrySetException(e); }
}, state: null);
return tcs.Task;
}
And then a method to read the size you want like this:
public static async Task<byte[]> ReadFixed(Socket socket, int bufferSize)
{
byte[] ret = new byte[bufferSize];
for (int read = 0; read < bufferSize; read += await socket.ReceiveAsync(ret, read, ret.Length - read, SocketFlags.None)) ;
return ret;
}

How Can I Edit Bytes As They Pass Through A Stream?

My goal is to have a file stream open up a user-chosen file, then, it should stream the files bytes through in chunks (buffers) of about 4mb (this can be changed it's just for fun). As the bytes travel (in chunks) through the stream, I'd like to have a looping if-statement see if the bytes value is contained in an array I have declared elsewhere. (The code below will build a random array for replacing bytes), and the replacement loop could just say something like the bottom for-loop. As you can see I'm fairly fluent in this language but for some reason the editing and rewriting of chunks as they are read from a file to a new one is eluding me. Thanks in advance!
private void button2_Click(object sender, EventArgs e)
{
GenNewKey();
const int chunkSize = 4096; // read the file by chunks of 4KB
using (var file = File.OpenRead(textBox1.Text))
{
int bytesRead;
var buffer = new byte[chunkSize];
while ((bytesRead = file.Read(buffer, 0, buffer.Length)) > 0)
{
byte[] newbytes = buffer;
int index = 0;
foreach (byte b in buffer)
{
for (int x = 0; x < 256; x++)
{
if (buffer[index] == Convert.ToByte(lst[x]))
{
try
{
newbytes[index] = Convert.ToByte(lst[256 - x]);
}
catch (System.Exception ex)
{
//just to show why the error was thrown, but not really helpful..
MessageBox.Show(index + ", " + newbytes.Count().ToString());
}
}
}
index++;
}
AppendAllBytes(textBox1.Text + ".ENC", newbytes);
}
}
}
private void GenNewKey()
{
Random rnd = new Random();
while (lst.Count < 256)
{
int x = rnd.Next(0, 255);
if (!lst.Contains(x))
{
lst.Add(x);
}
}
foreach (int x in lst)
{
textBox2.Text += ", " + x.ToString();
//just for me to see what was generated
}
}
public static void AppendAllBytes(string path, byte[] bytes)
{
if (!File.Exists(path + ".ENC"))
{
File.Create(path + ".ENC");
}
using (var stream = new FileStream(path, FileMode.Append))
{
stream.Write(bytes, 0, bytes.Length);
}
}
Where textbox1 holds the path and name of file to encrypt, textBox2 holds the generated cipher for personal debugging purposes, button two is the encrypt button, and of course I am using System.IO.

Indeed you have a off by one error in newbytes[index] = Convert.ToByte(lst[256 - x])
if x is 0 then you will have lst[256], however lst only goes between 0-255. Change that to 255 should fix it.
The reason it freezes up is your program is EXTREMELY inefficient and working on the UI thread (and has a few more errors like you should only go up to bytesRead in size when processing buffer, but that will just give you extra data in your output that should not be there. Also you are reusing the same array for buffer and newbytes so your inner for loop could modify the same index more than once because every time you do newbytes[index] = Convert.ToByte(lst[256 - x]) you are modifying buffer[index] which will get checked again the next itteration of the for loop).
There is a lot of ways you can improve your code, here is a snippet that does similar to what you are doing (I don't do the whole "find the index and use the opposite location", I just use the byte that is passed in as the index in the array).
while ((bytesRead = file.Read(buffer, 0, buffer.Length)) > 0)
{
byte[] newbytes = new byte[bytesRead];
for(int i = 0; i < newbytes.Length; i++)
{
newbytes[i] = (byte)lst[buffer[i]]))
}
AppendAllBytes(textBox1.Text + ".ENC", newbytes);
}
This may also lead to freezing but not as much, to solve the freeing you should put all of this code in to a BackgroundWorker or similar to run on another thread.

SerialPort in c# , Data Receive Incomplete Message

I'm working on Serialport. I'm facing a new problem that once I receive data my data are incomplete. How can I check if my data are complete then process them, and if not, don't process them?
Here are my data receive and my send function:
private void Send(byte[] cmd)
{
bResponse = new byte[0];
Write(cmd);
}
void comPort_DataReceived(object sender, SerialDataReceivedEventArgs e)
{
int iCount = comPort.BytesToRead;
byte[] bBuffer = new byte[iCount];
comPort.Read(bBuffer, 0, iCount)
if (bBuffer.Length == 1 && bBuffer[0] == ACK)
Write(new byte[] { ENQ });
else if (bBuffer.Length == 1 && bBuffer[0] == NAK)
{
Debug.WriteLine("Incomplete Message detected!");
}
else
{
bResponse = bResponse.Concat(bBuffer).ToArray();
rResponse = Decode(bResponse);
Write(new byte[] { ACK });
}
}
I know my data are received in a few packages and I need to wait until the response is complete, but I don't know based on the code above. How should I check whether the data are complete to determine whether to wait? (P.S: The size of the received response varies.)

There is no built-in concept of completeness or packet size.
You'll have to append to a buffer until you see some recognizable end-of-packet pattern that you (or someone else) defined as part of the protocol specification. - And then probably time out after a while if you haven't seen what you are looking for.

Example of old project, notice the firstindex, lastindex, you put in a character to know the length, the start/end character is predefined and can be any character you choose, just be sure not to take any common characters
This is for tcp/ip, but same principle can be used for serialport
public void ReceiveMessage(IAsyncResult ar)
{
int bytesRead;
try
{
lock (client1.GetStream())
{
bytesRead = client1.GetStream().EndRead(ar);
}
string messageReceived = System.Text.Encoding.ASCII.GetString(data, 0, bytesRead);
received = messageReceived;
int firstindex = received.IndexOf((char)2);
int lastindex = received.IndexOf((char)3);
if (firstindex > 0 && lastindex > 0)
{
string first = received.Substring(firstindex, 1);
string last = received.Substring(lastindex, 1);
}
lock (client1.GetStream())
{
client1.GetStream().BeginRead(data, 0, System.Convert.ToInt32(client1.ReceiveBufferSize), ReceiveMessage, null);
}
}
catch (Exception ex)
{
MessageBox.Show(ex.ToString());
}
}

I have some code for you.
First, you implement the DataReceived Event (as you have done already). This event is only called when there is data to process. While I would not call it interrupt-based (as in "realtime capable") is is definitely not polling. Which is good.
Second: When the event is called you may have only one byte, but there may be more bytes. To capture each packet you need to implement an custom buffer.
Third: After you append one byte to your buffer, you check whether the buffer contains a valid packet. If so, process it. If not: Append another one. If no bytes are left, wait for the event to be called again.
In code it looks like this:
const BUFFERLENGTH = 17; // Bytes
byte[] buffer = new byte[BUFFERLENGTH];
private void COM_Port_DataReceived(object sender, SerialDataReceivedEventArgs e)
{
var port = (SerialPort)sender;
while (port.BytesToRead > 0)
{
var data = (byte)port.ReadByte();
Read(data);
}
}
private void Read(byte value)
{
// Append Byte to buffer
System.Buffer.BlockCopy(buffer, 1, buffer, 0, BUFFERLENGTH- 1);
buffer[BUFFERLENGTH - 1] = value;
// Check for valid Packet
if (IsValidPacket(buffer))
{
// Yeah! Gotcha :-)
// Now copy your Packet from the Buffer to your struct/whatever
}
}
private bool IsValidPacket(byte[] buffer)
{
// Todo: Check whether buffer contains a valid Packet.
// Some think like:
// return buffer != null && buffer[0] == 0xF4 && buffer[2] == buffer.length
throw new NotImplementedException();
}
Note that I did not "append the byte to the buffer". I discarded the first byte, shifted every byte by one position and inserted the new byte at the end. If a valid Packet was found I could just copy it in one block into a struct. So the buffer size is always constant and exactly as long as one packet.
This may not be the fastest code out there (because it's reading each byte separately) but it works well for me :-)
Oh, and remember to use Begininvoke() if you want to display that stuff in your GUI.

Error in recieving data from TCP Client

I've written an application which listens to a port and receives some packets,according to my customized protocol, the packets are either 49 byte to 1500 byte, which i can tell from data length in the packet. the way i should interpret and deal with data in 49 byte packets and bigger packets are different.
The problem is that when i receive packets less than 1374 byte everything is ok, but when the packet length gets more, i receive the following exception and i also lose 4 last bytes of my data(i've tested with a 1384byte packet and i lost the last 4 bytes)
Exception which is raised:
Index was out of range. Must be non-negative and less than the size of the collection.
Parameter name: startIndex
each 49 byte packet has 35 byte of data, and the data length of bigger packets are non-deterministic(because of compression).
i found out sometimes the last 4 bytes are in a seperate "bytes" and "result" variables,meaning they are being treated like new packets and are not being attached to the packet they belong to.
here's the code for receiving data:
TcpClient Client = obj as TcpClient;
EndPoint ep = Client.Client.RemoteEndPoint;
List<Byte> result = new List<byte>();
result.Capacity = 2000;
try
{
NetworkStream stream = Client.GetStream();
int i = 49;
while ((i = stream.Read(bytes, 0,49)) != 0)
{
for (int id = 0; id < i; id++)
{
result.Add(bytes[id]);
}
//reading data length to determine packet length
byte[] tmp = new byte[2];
tmp = BitConverter.GetBytes(BitConverter.ToUInt16(result.ToArray(), 9));
if (BitConverter.IsLittleEndian)
{
Array.Reverse(tmp);
}
Int16 l = BitConverter.ToInt16(tmp, 0);
if (l>35)
{
stream.Read(bytes, result.Count, l - 35);
for (int id = 49; id <((l-35)+49); id++)
{
result.Add(bytes[id]);
}
if (this.TCPDataReceivedHandler != null)
{
this.TCPDataReceivedHandler(ep, result.Count, result.ToArray());
result.Clear();
Array.Clear(bytes, 0, 2000);
result.Capacity = 2000;
}
}
else
{
if (this.TCPDataReceivedHandler != null)
{
this.TCPDataReceivedHandler(ep, result.Count, result.ToArray());
result.Clear();
Array.Clear(bytes, 0, 2000);
result.Capacity = 2000;
}
}
}
System.Diagnostics.Debug.WriteLine("client Close");
Client.Close();
}
catch (System.Exception ex)
{
throw ex;
}
finally
{
Client.Close();
this.clients.Remove(Client);
}
According to Greg Suggestion and my researches,i also tried using following method:
NetworkStream stream = Client.GetStream();
int bytesread = 0, OffsetTemp = 0;
while (stream.CanRead)
{
OffsetTemp = 0;
bytesread += stream.Read(bytess, OffsetTemp, 11);
OffsetTemp = OffsetTemp + 11;
byte[] tmp = new byte[2];
tmp = BitConverter.GetBytes(BitConverter.ToUInt16(bytess.ToArray(), 9));
if (BitConverter.IsLittleEndian)
{
Array.Reverse(tmp);
}
Int16 l = BitConverter.ToInt16(tmp, 0);
bytesread += stream.Read(bytess, OffsetTemp++, 11 + l + 3);
for (int id = 0; id < l + 14; id++)
{
result.Add(bytess[id]);
}
if (this.TCPDataReceivedHandler != null)
{
this.TCPDataReceivedHandler(ep, result.Count, result.ToArray());
result.Clear();
Array.Clear(bytess, 0, 2000);
}
}

When using TCP, you must keep in mind that the size of the data blocks you put in is not guaranteed to be the same size of data blocks you get out on the receiving side. TCP is a stream protocol, so it guarantees that the same bytes get to the other side in the same order as they were sent (if not, the socket connection will be reset). TCP does not maintain any kind of block boundaries between calls to send(), and blocks may be arbitrarily split or coalesced depending on network conditions.
Your receiver must be prepared to handle receiving any amount of data from the calls to stream.Read(), but your code does not appear to do this. For example, correctly written receiver code should continue to work correctly even if stream.Read() receives only one byte at a time.