Develop Reference

In C#, what's the best way to deal with partially received messages using SocketAsyncEventArgs Buffer

In order to clean some messy code and get a better understanding of the SocketAsyncEventArgs class, I'd to know what's the most efficient technique to reassemble partially received messages from SocketAsyncEventArgs buffers.
To give you the big picture, I'm connected to a TCP server using a C# Socket client that will essentially receive data. The data received is message-based delimited by a \n character.
As you're probably already aware of, when using the ReceiveAsync method, this is almost a certitude that the last received message will be uncompleted such as you'll have to locate the index of the last complete message, copy the incomplete buffer section and keep it as start for the next received buffer and so on.
The thing is, I wish to abstract this operation from the upper layer and call the ProcessReceiveDataImpl as soon I get completed messages in the _tmpBuffer. I found that my Buffer.BlockCopy is not much readable (very old code also (-:) but anyway I wish to know what are you doing in this typical use case?
Code to reassemble messages:
public class SocketClient
{
private const int _receiveBufferSize = 8192;
private byte[] _remBuffer = new byte[2 * _receiveBufferSize];
private byte[] _tmpBuffer = new byte[2 * _receiveBufferSize];
private int _remBufferSize = 0;
private int _tmpBufferSize = 0;
private void ProcessReceiveData(SocketAsyncEventArgs e)
{
// the buffer to process
byte[] curBuffer = e.Buffer;
int curBufferSize = e.BytesTransferred;
int curBufferOffset = e.Offset;
int curBufferLastIndex = e.BytesTransferred - 1;
int curBufferLastSplitIndex = int.MinValue;
if (_remBufferSize > 0)
{
curBufferLastSplitIndex = GetLastSplitIndex(curBuffer, curBufferOffset, curBufferSize);
if (curBufferLastSplitIndex != curBufferLastIndex)
{
// copy the remain + part of the current into tmp
Buffer.BlockCopy(_remBuffer, 0, _tmpBuffer, 0, _remBufferSize);
Buffer.BlockCopy(curBuffer, curBufferOffset, _tmpBuffer, _remBufferSize, curBufferLastSplitIndex + 1);
_tmpBufferSize = _remBufferSize + curBufferLastSplitIndex + 1;
ProcessReceiveDataImpl(_tmpBuffer, _tmpBufferSize);
Buffer.BlockCopy(curBuffer, curBufferLastSplitIndex + 1, _remBuffer, 0, curBufferLastIndex - curBufferLastSplitIndex);
_remBufferSize = curBufferLastIndex - curBufferLastSplitIndex;
}
else
{
// copy the remain + entire current into tmp
Buffer.BlockCopy(_remBuffer, 0, _tmpBuffer, 0, _remBufferSize);
Buffer.BlockCopy(curBuffer, curBufferOffset, _tmpBuffer, _remBufferSize, curBufferSize);
ProcessReceiveDataImpl(_tmpBuffer, _remBufferSize + curBufferSize);
_remBufferSize = 0;
}
}
else
{
curBufferLastSplitIndex = GetLastSplitIndex(curBuffer, curBufferOffset, curBufferSize);
if (curBufferLastSplitIndex != curBufferLastIndex)
{
// we must copy the unused byte into remaining buffer
_remBufferSize = curBufferLastIndex - curBufferLastSplitIndex;
Buffer.BlockCopy(curBuffer, curBufferLastSplitIndex + 1, _remBuffer, 0, _remBufferSize);
// process the msg
ProcessReceiveDataImpl(curBuffer, curBufferLastSplitIndex + 1);
}
else
{
// we can process the entire msg
ProcessReceiveDataImpl(curBuffer, curBufferSize);
}
}
}
protected virtual void ProcessReceiveDataImpl(byte[] buffer, int bufferSize)
{
}
private int GetLastSplitIndex(byte[] buffer, int offset, int bufferSize)
{
for (int i = offset + bufferSize - 1; i >= offset; i--)
{
if (buffer[i] == '\n')
{
return i;
}
}
return -1;
}
}
Your input is very important and appreciated!
Thank you!
Updated:
Also, rather then calling the ProcessReceiveDataImpl and block further receive operations, will it be useful to queue completed messages and make them available to the consumer?

C# partial UTF-8 byte stream conversion

I have wrote the following simple test:
[Test]
public void TestUTF8()
{
var c = "abc☰def";
var b = Encoding.UTF8.GetBytes(c);
Assert.That(b.Length, Is.EqualTo(9));
//Assuming, you are reading a byte stream and got partial result with the first 5 bytes
var p = Encoding.UTF8.GetChars(b, 0, 5);
Trace.WriteLine(new string(p));
Assert.That(p.Length, Is.EqualTo(3));
}
The Trace outputs abc� and the last assert fails because p.Length is 4.
However, I wanted Trace outputs abc and the last assert passes, since in reality I know the stream will have valid chars and when it is not the case for the last few bytes, just leave them there waiting for more data to come.
So how can I achieve this in C#?

Encoding.GetChars isn't really designed for bytes coming from a stream where some state needs to be kept track of during the decoding process because a single character might span multiple buffer segments. To do that work you should use a Decoder obtained from Encoding.GetDecoder. However, Decoder.Convert is really low-level allowing you control over both the input and output buffers and somewhat difficult to use. Decoder.GetChars is somewhat easier to use and does the important work of storing state between calls. We can easily expand on Peter Duniho's answer for arbitrary buffer size:
public static void Main(string[] args)
{
var c = "abc☰def";
var b = Encoding.UTF8.GetBytes(c);
var result = DecodeFromStream(new MemoryStream(b), Encoding.UTF8, 3);
Console.WriteLine(result);
Console.WriteLine(c == result);
}
private static string DecodeFromStream(Stream dataStream, Encoding encoding, int bufferSize)
{
Decoder decoder = encoding.GetDecoder();
StringBuilder sb = new StringBuilder();
int inputByteCount;
byte[] inputBuffer = new byte[bufferSize];
char[] charBuffer = new char[encoding.GetMaxCharCount(inputBuffer.Length)];
while ((inputByteCount = dataStream.Read(inputBuffer, 0, inputBuffer.Length)) > 0)
{
int readChars = decoder.GetChars(inputBuffer, 0, inputByteCount, charBuffer, 0);
if (readChars > 0)
sb.Append(charBuffer, 0, readChars);
}
return sb.ToString();
}

SerialPort in c# , Data Receive Incomplete Message

I'm working on Serialport. I'm facing a new problem that once I receive data my data are incomplete. How can I check if my data are complete then process them, and if not, don't process them?
Here are my data receive and my send function:
private void Send(byte[] cmd)
{
bResponse = new byte[0];
Write(cmd);
}
void comPort_DataReceived(object sender, SerialDataReceivedEventArgs e)
{
int iCount = comPort.BytesToRead;
byte[] bBuffer = new byte[iCount];
comPort.Read(bBuffer, 0, iCount)
if (bBuffer.Length == 1 && bBuffer[0] == ACK)
Write(new byte[] { ENQ });
else if (bBuffer.Length == 1 && bBuffer[0] == NAK)
{
Debug.WriteLine("Incomplete Message detected!");
}
else
{
bResponse = bResponse.Concat(bBuffer).ToArray();
rResponse = Decode(bResponse);
Write(new byte[] { ACK });
}
}
I know my data are received in a few packages and I need to wait until the response is complete, but I don't know based on the code above. How should I check whether the data are complete to determine whether to wait? (P.S: The size of the received response varies.)

There is no built-in concept of completeness or packet size.
You'll have to append to a buffer until you see some recognizable end-of-packet pattern that you (or someone else) defined as part of the protocol specification. - And then probably time out after a while if you haven't seen what you are looking for.

Example of old project, notice the firstindex, lastindex, you put in a character to know the length, the start/end character is predefined and can be any character you choose, just be sure not to take any common characters
This is for tcp/ip, but same principle can be used for serialport
public void ReceiveMessage(IAsyncResult ar)
{
int bytesRead;
try
{
lock (client1.GetStream())
{
bytesRead = client1.GetStream().EndRead(ar);
}
string messageReceived = System.Text.Encoding.ASCII.GetString(data, 0, bytesRead);
received = messageReceived;
int firstindex = received.IndexOf((char)2);
int lastindex = received.IndexOf((char)3);
if (firstindex > 0 && lastindex > 0)
{
string first = received.Substring(firstindex, 1);
string last = received.Substring(lastindex, 1);
}
lock (client1.GetStream())
{
client1.GetStream().BeginRead(data, 0, System.Convert.ToInt32(client1.ReceiveBufferSize), ReceiveMessage, null);
}
}
catch (Exception ex)
{
MessageBox.Show(ex.ToString());
}
}

I have some code for you.
First, you implement the DataReceived Event (as you have done already). This event is only called when there is data to process. While I would not call it interrupt-based (as in "realtime capable") is is definitely not polling. Which is good.
Second: When the event is called you may have only one byte, but there may be more bytes. To capture each packet you need to implement an custom buffer.
Third: After you append one byte to your buffer, you check whether the buffer contains a valid packet. If so, process it. If not: Append another one. If no bytes are left, wait for the event to be called again.
In code it looks like this:
const BUFFERLENGTH = 17; // Bytes
byte[] buffer = new byte[BUFFERLENGTH];
private void COM_Port_DataReceived(object sender, SerialDataReceivedEventArgs e)
{
var port = (SerialPort)sender;
while (port.BytesToRead > 0)
{
var data = (byte)port.ReadByte();
Read(data);
}
}
private void Read(byte value)
{
// Append Byte to buffer
System.Buffer.BlockCopy(buffer, 1, buffer, 0, BUFFERLENGTH- 1);
buffer[BUFFERLENGTH - 1] = value;
// Check for valid Packet
if (IsValidPacket(buffer))
{
// Yeah! Gotcha :-)
// Now copy your Packet from the Buffer to your struct/whatever
}
}
private bool IsValidPacket(byte[] buffer)
{
// Todo: Check whether buffer contains a valid Packet.
// Some think like:
// return buffer != null && buffer[0] == 0xF4 && buffer[2] == buffer.length
throw new NotImplementedException();
}
Note that I did not "append the byte to the buffer". I discarded the first byte, shifted every byte by one position and inserted the new byte at the end. If a valid Packet was found I could just copy it in one block into a struct. So the buffer size is always constant and exactly as long as one packet.
This may not be the fastest code out there (because it's reading each byte separately) but it works well for me :-)
Oh, and remember to use Begininvoke() if you want to display that stuff in your GUI.

.NET newbie socket problem

I have a C# client/server network program I've written using
TCPListener and TCPClient classes. The server is reading everything
from the client (small amounts of xml) just fine until I try to send a
large file (100k) back to the client.
I'm using stream functions for
both client and server with non-blocking socket functions. When I do a
socket.SendFile("filename") back to the client, the file is getting
cut off - I've set the receive buffer size on the client to well past
100k but it still gets cut off around 25k and the communication
between client and server is unreliable afterwords.
My basic question
is what happens if data is somehow left in the pipe ? i.e.. will it be
read by the next socket.Read... Does every Send call require exactly
one and only one Read ? Maybe I'm not giving the client enough time to
read the file but their both on the same machine and I've tried
sleeping for a few seconds in various places w/o success.

It is very possible that you cannot read the entire message through one Read call (perhaps all data has not arrived yet). In network programming, you would often place the call to Read in a while loop and simply Read() until you have received the entire expected message.

1 Send call might take more than one Read call to receive, and 1 Read call might read the data send by several Send call.
TCP just provides a stream, so it's up to you to define how the data or messages you send are partitioned.
In this case, you probably just need to loop ,doing Read until the stream is closed.

You probably want something like this:
socket.Blocking = false;
const int ChunkSize = 1492;
const int ReceiveTimeout = 10000;
const int SendTimeout = 10000;
public void Send(Byte[] data)
{
var sizeBytes = BitConverter.GetBytes(data.Length);
SendInternal(sizeBytes);
SendInternal(data);
}
public Byte[] Receive()
{
var sizeBytes = ReceiveInternal(4);
var size = BitConverter.ToInt32(sizeBytes, 0);
var data = ReceiveInternal(size);
return data;
}
private void SendInternal(Byte[] data)
{
var error = SocketError.Success;
var lastUpdate = Environment.TickCount;
var size = data.Length;
var count = 0;
var sent = 0;
while (sent < size)
{
count = Math.Min(ChunkSize, size - sent);
count = socket.Send(data, sent, count, SocketFlags.None, out error);
if (count > 0)
{
sent += count;
lastUpdate = Environment.TickCount;
}
if (error != SocketError.InProgress && error != SocketError.Success && error != SocketError.WouldBlock)
throw new SocketException((Int32)error);
if (Environment.TickCount - lastUpdate > SendTimeout)
throw new TimeoutException("Send operation timed out.");
if (count == 0 && !socket.Poll(100, SelectMode.SelectWrite))
throw new SocketException((Int32)SocketError.Shutdown);
}
}
private Byte[] ReceiveInternal(Int32 size)
{
var error = SocketError.Success;
var lastUpdate = Environment.TickCount;
var buffer = new Byte[ChunkSize];
var count = 0;
var received = 0;
using (var ms = new MemoryStream(size))
{
while (received < size)
{
count = Math.Min(ChunkSize, size - received);
count = socket.Receive(buffer, 0, count, SocketFlags.None, out error);
if (count > 0)
{
ms.Write(buffer, 0, count);
received += count;
lastUpdate = Environment.TickCount;
}
if (error != SocketError.InProgress && error != SocketError.Success && error != SocketError.WouldBlock)
throw new SocketException((Int32)error);
if (Environment.TickCount - lastUpdate > ReceiveTimeout)
throw new TimeoutException("Receive operation timed out.");
if (count == 0 && socket.Poll(100, SelectMode.SelectRead) && socket.Available == 0)
throw new SocketException((Int32)SocketError.Shutdown);
}
return ms.ToArray();
}
}

What I would usually do is create a header structure that is sent
Header Size (int, 4 bytes)
File Name Offset (int, 4 bytes)
File Name Size (int , 4 bytes)
File Data Offset (int, 4 bytes)
File Data Size (int , 4 bytes)
[ message data here]
and then that header is read using either a BinaryReader or copying the bytes to a struct using Marshal. This way you always know what data is arriving and how many times you need to call Read().
The header size field is also helps with versioning the protocol (keep the structure the same but add to it for later clients so you can keep backwards compatibility). If you define the structure in C# be sure to do it like so:
[StructLayout LayoutKind.Sequential]
struct MessageHeader
{
public int HeaderSize;
public int FileNameOffset;
public int FileNameSize;
public int FileDataOffset;
public int FileDataSize;
}
Then Marshal.PtrToStructure will allow you do create an instance of this struct right from the byte[] you read from the socket

Try sending the chunk from the server side in chunks. Just as the other said, posting the code would be of great help to us.

What does a working example of dealing with packet fragmentation in C# look like when using the Begin* End* method?

After enough playing with asynchronous socket programming I noticed that the server was receiving chunked payloads (ie: more than one complete payload sitting in the same buffer). So I came up with the following:
if (bytes_to_read > 0)
{
while (bytes_to_read > 0)
// Get payload size as int.
// Get payload in byte format.
// Do something with payload.
// Decrease the amount of bytes to read.
}
// Wait for more data.
}
And then I noticed packet fragmentation (ie: what I thought were complete payloads chunked together wasn't always so) which changed the previous code to something like:
if (bytes_to_read > 0)
{
while (bytes_to_read > 0)
{
// Get payload size as int.
// Check if the payload size is less than or equal to the amount of bytes left to read.
if (payload_size <= bytes_to_read)
{
// Get payload in byte format.
// Do something with payload.
// Decrease the amount of bytes to read.
}
else
{
// We received a fragmented payload.
break;
}
}
if (bytes_to_read == 0)
{
// Wait for more data.
}
else if (bytes_to_read > 0)
{
// Wait for more data where we left off. ***
}
else
{
// Something awful happened.
}
}
*** I don't even know how to go about this and would like to see code for it. I had an idea that it involved copying the in-completed payload to the beginning of the buffer and then picking up from there.
The pseudo code I included is based on the Begin* End* method I am using (I'm aware that I should be using the *Async set of methods found here -> http://msdn.microsoft.com/en-us/library/system.net.sockets.socketasynceventargs.aspx <- but I think my overall question still applies).
I am seeking the answers to 2 questions--namely:
Is this approach correct or am I
missing something?
What does a working example of
dealing with packet fragmentation in
C# look like?
EDIT: I'm using raw sockets.
Thanks in advance for all your help.
EDIT: John Saunders and Greg Hewgill have brought up the point of treating data as a stream but that does not provide me with a concrete example of how to deal with the last chunked payload sometimes being fragmented.
EDIT: I have read Jon Skeet's answer here which is basically along the same lines as the other answers I have seen but it doesn't help me much as I already get what I have to do but not how to do it.
EDIT: To elaborate on what I mean by fragmentation, consider the following the receive buffers:
224TEST3foo3bar
224TEST3foo3bar224TEST3foo3bar
224TEST3foo3bar224TEST3foo
3bar224TEST3foo3bar
EDIT: I found this and this which lead me here. Vadym Stetsiak has cleared nearly everything up (his was one of the answers I was looking for).

This may or may not have anything to do with fragmentation.
In general, the socket will pass you as many bytes at a time as it feels like. Your job is to know how many bytes are in your overall message, and to read them all. Just keep looping until you have all the bytes you need, or until there's an exception.
The following code is untested right now. I thought I'd post it before writing the server side of it and testing both.
private static string ReceiveMessage(Socket socket)
{
const int BUFFER_SIZE = 1024;
var inputBuffer = new byte[BUFFER_SIZE];
var offset = 0;
var bytesReceived = socket.Receive(
inputBuffer, offset, BUFFER_SIZE - offset, SocketFlags.None);
if (bytesReceived < 2)
{
throw new InvalidOperationException("Receive error");
}
var inputMessageLength = inputBuffer[0]*256 + inputBuffer[1];
offset += bytesReceived;
var totalBytesReceived = bytesReceived;
while (bytesReceived > 0 &&
totalBytesReceived < inputMessageLength + 2)
{
bytesReceived = socket.Receive(
inputBuffer, offset, BUFFER_SIZE - offset, SocketFlags.None);
offset += bytesReceived;
totalBytesReceived += bytesReceived;
}
return Encoding.UTF8.GetString(
inputBuffer, 2, totalBytesReceived - 2);
}
Note that the receipt of the message length is wrong. The socket layer could give it to me a byte at a time. I'm going to revisit that as part of a refactoring that will receive the count into a separate two-byte buffer, and change the loop into a single do/while.

When you have to do it yourself, it can be done like so (reference here):
///
/// Server state holds current state of the client socket
///
class AsyncServerState
{
public byte[] Buffer = new byte[512]; //buffer for network i/o
public int DataSize = 0; //data size to be received by the server
//flag that indicates whether prefix was received
public bool DataSizeReceived = false;
public MemoryStream Data = new MemoryStream(); //place where data is stored
public SocketAsyncEventArgs ReadEventArgs = new SocketAsyncEventArgs();
public Socket Client;
}
///
/// Implements server receive logic
///
private void ProcessReceive(SocketAsyncEventArgs e)
{
//single message can be received using several receive operation
AsyncServerState state = e.UserToken as AsyncServerState;
if (e.BytesTransferred <= 0 || e.SocketError != SocketError.Success)
{
CloseConnection(e);
}
int dataRead = e.BytesTransferred;
int dataOffset = 0;
int restOfData = 0;
while (dataRead > 0)
{
if (!state.DataSizeReceived)
{
//there is already some data in the buffer
if (state.Data.Length > 0)
{
restOfData = PrefixSize - (int)state.Data.Length;
state.Data.Write(state.Buffer, dataOffset, restOfData);
dataRead -= restOfData;
dataOffset += restOfData;
}
else if (dataRead >= PrefixSize)
{ //store whole data size prefix
state.Data.Write(state.Buffer, dataOffset, PrefixSize);
dataRead -= PrefixSize;
dataOffset += PrefixSize;
}
else
{ // store only part of the size prefix
state.Data.Write(state.Buffer, dataOffset, dataRead);
dataOffset += dataRead;
dataRead = 0;
}
if (state.Data.Length == PrefixSize)
{ //we received data size prefix
state.DataSize = BitConverter.ToInt32(state.Data.GetBuffer(), 0);
state.DataSizeReceived = true;
state.Data.Position = 0;
state.Data.SetLength(0);
}
else
{ //we received just part of the headers information
//issue another read
if (!state.Client.ReceiveAsync(state.ReadEventArgs))
ProcessReceive(state.ReadEventArgs);
return;
}
}
//at this point we know the size of the pending data
if ((state.Data.Length + dataRead) >= state.DataSize)
{ //we have all the data for this message
restOfData = state.DataSize - (int)state.Data.Length;
state.Data.Write(state.Buffer, dataOffset, restOfData);
Console.WriteLine("Data message received. Size: {0}",
state.DataSize);
dataOffset += restOfData;
dataRead -= restOfData;
state.Data.SetLength(0);
state.Data.Position = 0;
state.DataSizeReceived = false;
state.DataSize = 0;
if (dataRead == 0)
{
if (!state.Client.ReceiveAsync(state.ReadEventArgs))
ProcessReceive(state.ReadEventArgs);
return;
}
else
continue;
}
else
{ //there is still data pending, store what we've
//received and issue another BeginReceive
state.Data.Write(state.Buffer, dataOffset, dataRead);
if (!state.Client.ReceiveAsync(state.ReadEventArgs))
ProcessReceive(state.ReadEventArgs);
dataRead = 0;
}
}
}
I did not do it exactly this way myself but it helped.