Develop Reference

Awaiting data from Serial Port in C#

I have an application that receives data from a wireless radio using RS-232. These radios use an API for communicating with multiple clients. To use the radios I created a library for communicate with them that other software can utilize with minimal changes from a normal SerialPort connection. The library reads from a SerialPort object and inserts incoming data into different buffers depending on the radio it receives from. Each packet that is received contains a header indicating its length, source, etc.
I start by reading the header, which is fixed-length, from the port and parsing it. In the header, the length of the data is defined before the data payload itself, so once I know the length of the data, I then wait for that much data to be available, then read in that many bytes.
Example (the other elements from the header are omitted):
// Read header
byte[] header = new byte[RCV_HEADER_LENGTH];
this.Port.Read(header, 0, RCV_HEADER_LENGTH);
// Get length of data in packet
short dataLength = header[1];
byte[] payload = new byte[dataLength];
// Make sure all the payload of this packet is ready to read
while (this.Port.BytesToRead < dataLength) { }
this.Port.Read(payload, 0, dataLength);
Obviously the empty while port is bad. If for some reason the data never arrives the thread will lock. I haven't encountered this problem yet, but I'm looking for an elegant way to do this. My first thought is to add a short timer that starts just before the while-loop, and sets an abortRead flag when it elapses that would break the while loop, like this:
// Make sure all the payload of this packet is ready to read
abortRead = false;
readTimer.Start();
while (this.Port.BytesToRead < dataLength && !abortRead) {}
This code needs to handle a constant stream of incoming data as quickly as it can, so keeping overhead to a minimum is a concern, and am wondering if I am doing this properly.

You don't have to run this while loop, the method Read would either fill the buffer for you or would throw a TimeoutException if buffer wasn't filled within the SerialPort.ReadTimeout time (which you can adjust to your needs).
But some general remark - your while loop would cause intensive CPU work for nothing, in the few milliseconds it would take the data to arrive you would have thousends of this while loop iterations, you should've add some Thread.Sleep inside.

If you want to truly adress this problem, you need to run the code in the background. There are different options to do that; you can start a thread, you start a Task or you can use async await.
To fully cover all options, the answer would be endless. If you use threads or tasks with the default scheduler and your wait time is expected to be rather short, you can use SpinWait.SpinUntil instead of your while loop. This will perform better than your solution:
SpinWait.SpinUntil(() => this.Port.BytesToRead >= dataLength);
If you are free to use async await, I would recommend this solution, since you need only a few changes to your code. You can use Task.Delay and in the best case you pass a CancellationToken to be able to cancel your operation:
try {
while (this.Port.BytesToRead < dataLength) {
await Task.Delay(100, cancellationToken);
}
}
catch(OperationCancelledException) {
//Cancellation logic
}

I think I would do this asynchronously with the SerialPort DataReceived event.
// Class fields
private const int RCV_HEADER_LENGTH = 8;
private const int MAX_DATA_LENGTH = 255;
private SerialPort Port;
private byte[] PacketBuffer = new byte[RCV_HEADER_LENGTH + MAX_DATA_LENGTH];
private int Readi = 0;
private int DataLength = 0;
// In your constructor
this.Port.DataReceived += new SerialDataReceivedEventHandler(DataReceivedHandler);
private void DataReceivedHandler(object sender, SerialDataReceivedEventArgs e)
{
if (e.EventType != SerialData.Chars)
{
return;
}
// Read all available bytes.
int len = Port.BytesToRead;
byte[] data = new byte[len];
Port.Read(data, 0, len);
// Go through each byte.
for (int i = 0; i < len; i++)
{
// Add the next byte to the packet buffer.
PacketBuffer[Readi++] = data[i];
// Check if we've received the complete header.
if (Readi == RCV_HEADER_LENGTH)
{
DataLength = PacketBuffer[1];
}
// Check if we've received the complete data.
if (Readi == RCV_HEADER_LENGTH + DataLength)
{
// The packet is complete add it to the appropriate buffer.
Readi = 0;
}
}
}

Serial port under-runs occurring even though code is built to try and keep Tx buffer full.

OK, Here's the deal. I am using the serial port to modulate an LED light array (making the array dimmable). Problem is the array flashes periodically, a quick flash every 2-4 seconds.
I use a System.Timer to check the Tx buffer state every 10 mS. If the buffer doesn't have a minimum number of characters pending, the routine adds more. In theory the buffer should never run out of characters, but it does, aka LED array flashes. I detect and count these events, so I know under-runs are occurring. Furthermore, I measure the time taken to refill the buffer and it sometimes takes over 1000 mS to add more characters. TxBuffer size is set to 4096, and my desired buffer count is 2000. For the life of me I don't get why the serial port blocks for as long as a second when in theory it shouldn't block at all...
Setup code:
charsSent = 0;
myPort.PortName = commPort;
myPort.Encoding = new UTF8Encoding ();
myPort.BaudRate = 19200;
myPort.DataBits = 8;
myPort.Parity = Parity.None;
myPort.StopBits = StopBits.One;
myPort.Handshake = Handshake.None;
myPort.DtrEnable = false;
myPort.RtsEnable = false;
myPort.ReadTimeout = 500; // SerialPort.InfiniteTimeout;
myPort.WriteTimeout = 500; // SerialPort.InfiniteTimeout;
myPort.ReadBufferSize = 4096;
myPort.WriteBufferSize = 4096;
myPort.Open ();
Timer code:
myPort.BreakState = false;
int txBufferCount = myPort.BytesToWrite;
if (txBufferCount == 0)
++underRuns;
// refill the xmit buffer if needed...
int bytesNeeded = (txBufferCount > MaxTxLoadCount) ? 0 : MaxTxLoadCount - txBufferCount;
DateTime start = DateTime.Now;
while (bytesNeeded > 0)
{
myPort.Write (charbuf, 0, 5);
charsSent += 5;
bytesNeeded -= 5;
}
if (DateTime.Now - start > maxWriteTime)
maxWriteTime = DateTime.Now - start;
Any ideas?

For my usage I found a workaround. Basically I allow for two instances of my timer service route to exists at any one time. Furthermore I protect the buffer refill code with a lock, so that while one instance is inside the lock section (and being held up by the bull shit blocking write) the other is being held just outside of the lock section by the lock.
Net result is that as soon as the instance being write blocked, unblocks and exits the lock section, the other instance enters the lock and refills the buffer. Net net the LEDs might still be flashing (depending upon exactly when the driver unblocks (UART's Txbuffer = empty, vs. UART's TxShiftRegister = empty).
It's just now you can't see them do it, so end the end it works for me...
cheers,
Keith

Serial Port Polling and Data handling

I am trying to read from several serial ports from sensors through microcontrollers. Each serial port will receive more than 2000 measurements (each measurement is 7 bytes, all in hex). And they are firing at the same time. Right now I am polling from 4 serial ports. Also, I translate each measurement into String and append it to a Stringbuilder. When I finish receiving data, they will be ouput in to a file. The problem is the CPU consumption is very high, ranging from 80% to 100%.
I went though some articles and put Thread.Sleep(100) at the end. It reduces CPU time when there is no data coming. I also put Thread.Sleep at the end of each polling when the BytesToRead is smaller than 100. It only helps to a certain extent.
Can someone suggest a solution to poll from serial port and handle data that I get? Maybe appending every time I get something causes the problem?
//I use separate threads for all sensors
private void SensorThread(SerialPort mySerialPort, int bytesPerMeasurement, TextBox textBox, StringBuilder data)
{
textBox.BeginInvoke(new MethodInvoker(delegate() { textBox.Text = ""; }));
int bytesRead;
int t;
Byte[] dataIn;
while (mySerialPort.IsOpen)
{
try
{
if (mySerialPort.BytesToRead != 0)
{
//trying to read a fix number of bytes
bytesRead = 0;
t = 0;
dataIn = new Byte[bytesPerMeasurement];
t = mySerialPort.Read(dataIn, 0, bytesPerMeasurement);
bytesRead += t;
while (bytesRead != bytesPerMeasurement)
{
t = mySerialPort.Read(dataIn, bytesRead, bytesPerMeasurement - bytesRead);
bytesRead += t;
}
//convert them into hex string
StringBuilder s = new StringBuilder();
foreach (Byte b in dataIn) { s.Append(b.ToString("X") + ","); }
var line = s.ToString();
var lineString = string.Format("{0} ---- {2}",
line,
mySerialPort.BytesToRead);
data.Append(lineString + "\r\n");//append a measurement to a huge Stringbuilder...Need a solution for this.
////use delegate to change UI thread...
textBox.BeginInvoke(new MethodInvoker(delegate() { textBox.Text = line; }));
if (mySerialPort.BytesToRead <= 100) { Thread.Sleep(100); }
}
else{Thread.Sleep(100);}
}
catch (Exception ex)
{
//MessageBox.Show(ex.ToString());
}
}
}

this is not a good way to do it, it far better to work on the DataReceived event.
basically with serial ports there's a 3 stage process that works well.
Receiving the Data from the serial port
Waiting till you have a relevant chunk of data
Interpreting the data
so something like
class DataCollector
{
private readonly Action<List<byte>> _processMeasurement;
private readonly string _port;
private SerialPort _serialPort;
private const int SizeOfMeasurement = 4;
List<byte> Data = new List<byte>();
public DataCollector(string port, Action<List<byte>> processMeasurement)
{
_processMeasurement = processMeasurement;
_serialPort = new SerialPort(port);
_serialPort.DataReceived +=SerialPortDataReceived;
}
private void SerialPortDataReceived(object sender, SerialDataReceivedEventArgs e)
{
while(_serialPort.BytesToRead > 0)
{
var count = _serialPort.BytesToRead;
var bytes = new byte[count];
_serialPort.Read(bytes, 0, count);
AddBytes(bytes);
}
}
private void AddBytes(byte[] bytes)
{
Data.AddRange(bytes);
while(Data.Count > SizeOfMeasurement)
{
var measurementData = Data.GetRange(0, SizeOfMeasurement);
Data.RemoveRange(0, SizeOfMeasurement);
if (_processMeasurement != null) _processMeasurement(measurementData);
}
}
}
Note: Add Bytes keeps collecting data till you have enough to count as a measurement, or if you get a burst of data, splits it up into seperate measurements.... so you can get 1 byte one time, 2 the next, and 1 more the next, and it will then take that an turn it into a measurement. Most of the time if your micro sends it in a burst, it will come in as one, but sometimes it will get split into 2.
then somewhere you can do
var collector = new DataCollector("COM1", ProcessMeasurement);
and
private void ProcessMeasurement(List<byte> bytes)
{
// this will get called for every measurement, so then
// put stuff into a text box.... or do whatever
}

First of all consider reading Using Stopwatches and Timers in .NET. You can break down any performance issue with this and tell exactly which part of Your code is causing the problem.
Use SerialPort.DataReceived Event to trigger data receiving process.
Separate receiving process and data manipulation process. Store Your data first then process.
Do not edit UI from reading loop.

I guess what you should be doing is adding an event handler to process incoming data:
mySerialPort.DataReceived += new SerialDataReceivedEventHandler(mySerialPort_DataReceived);
This eliminates the need to run a separate thread for each serial port you listen to. Also, each DataReceived handler will be called precisely when there is data available and will consume only as much CPU time as is necessary to process the data, then yield to the application/OS.
If that doesn't solve the CPU usage problem, it means you're doing too much processing. But unless you've got some very fast serial ports I can't imagine the code you've got there will pose a problem.

Stop to wait for a packet from serial port

I'm receiving periodically some data via Serial Port, in order to plot it and do some more stuff. In order to achive this purpose, I send the data from my microcontroller to my computer with a header, which specifies the length of each packet.
I have the program running and working perfectly except a one last detail. When the header specifies a lenght, my program will not stop until it reachs that amount of bytes. So if, for some reason, some data from one packet is missed, the program wait and take the beginning of the next packet...and then start the real problems. Since that moment, every fails.
I thought about rising a Timer every 0.9 seconds ( the packages come every second) who will give a command in order to comeback to wait and reset variables. But I don't know how to do it, I tried but I obtain errors while running. Since IndCom ( see next code) resets in the midle of some function and errors as "Index out of bounds" arises.
I attach my code ( without timer)
private void routineRx(object sender, System.IO.Ports.SerialDataReceivedEventArgs e)
{
try
{
int BytesWaiting;
do
{
BytesWaiting = this.serialPort.BytesToRead;
//Copy it to the BuffCom
while (BytesWaiting > 0)
{
BuffCom[IndCom] = (byte)this.serialPort.ReadByte();
IndCom = IndCom + 1;
BytesWaiting = BytesWaiting - 1;
}
} while (IndCom < HeaderLength);
//I have to read until I got the whole Header which gives the info about the current packet
PacketLength = getIntInfo(BuffCom,4);
while (IndCom < PacketLength)
{
BytesWaiting = this.serialPort.BytesToRead;
//Copy it to the BuffCom
while (BytesWaiting > 0)
{
BuffCom[IndCom] = (byte)this.serialPort.ReadByte();
IndCom = IndCom + 1;
BytesWaiting = BytesWaiting - 1;
}
}
//If we have a packet--> check if it is valid and, if so, what kind of packet is
this.Invoke(new EventHandler(checkPacket));
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
}
}
I'm new in object-oriented programming and c#, so be clement, please! And thank you very much

What you might do is use a Stopwatch.
const long COM_TIMEOUT = 500;
Stopwatch spw = new Stopwatch();
spw.Restart();
while (IndCom < PacketLength)
{
//read byte, do stuff
if (spw.ElapsedMilliseconds > COM_TIMEOUT) break; //etc
}
Restart the stopwatch at the beginning and check the time in each while loop, then break out(and clean up) if the timeout hits. 900ms is probably too much, even, if you're only expecting a few bytes. Com traffic is quite fast - if you don't get the whole thing immediately it's probably not coming.
I like to use termination characters in communication protocols (like [CR], etc). This allows you to read until you find the termination character, then stop. This prevents reading into the next command. Even if you don't want to use termination characters, changing your code to something like this :
while (IndCom < PacketLength)
{
if (serialPort.BytesToRead > 0)
{
BuffCom[IndCom] = (byte)this.serialPort.ReadByte();
IndCom++;
}
}
it allows you to stop when you reach your packet size, leaving any remaining characters in the buffer for the next round through (ie: the next command). You can add the stopwatch timeout in the above also.
The other nice thing about termination characters is that you don't have to know in advance how long the packet should be - you just read until you reach the termination character and then process/parse the whole thing once you've got it. It makes your two-step port read into a one-step port read.

Serial Port; Missing Packets after few hours

I am new to Visual C#. I have to receive a packet of 468 bytes every second from a embedded device serially. The header of the packet is 0xbf, 0x13, 0x97, 0x74. After check validating the packet header i am saving this packet , process it, and display it graphically.
The problem is that i start losing packets after few hours. (Other software was logging the same data for the whole week and is working well).
The code is here...
private void DataRec(object sender, System.IO.Ports.SerialDataReceivedEventArgs e)
{
rtTotBytes = comport.BytesToRead;
rtTotBytesRead = comport.Read(rtSerBuff, 0, rtTotBytes);
this.Invoke(new ComportDelegate(ComportDlgtCallback), rtSerBuff, rtTotBytesRead);
}
//Delegate
delegate void ComportDelegate(byte[] sBuff, int sByte);
//Callback Function to Delegate
private void ComportDlgtCallback(byte[] SerBuff, int TotBytes)
{
for (int k = 0; k < TotBytes; k++)
{
switch (rtState)
{
case 0:
if (SerBuff[k] == 0xbf) { rtState = 1; TempBuff[0] = 0xbf; }
else rtState = 0;
break;
case 1:
if (SerBuff[k] == 0x13) { rtState = 2; TempBuff[1] = 0x13; }
else rtState = 0;
break;
case 2:
if (SerBuff[k] == 0x97) { rtState = 3; TempBuff[2] = 0x97; }
else rtState = 0;
break;
case 3:
if (SerBuff[k] == 0x74) { rtState = 4; TempBuff[3] = 0x74; rtCnt = 4; }
else rtState = 0;
break;
case 4:
if (rtCnt == 467)
{
TempBuff[rtCnt] = SerBuff[k];
TempBuff.CopyTo(PlotBuff, 0);
ProcessPacket(PlotBuff);
rtState = 0; rtCnt = 0;
}
else
TempBuff[rtCnt++] = SerBuff[k];
break;
}
}
}
Another question: can the BytesToRead be zero if a DataReceivedEvent had occured? Do you have to check (BytesToRead>0) in DataRecievedEvent?

Serial port input data must be treated as a stream, and not series of packets. For example, when device sends 0xbf, 0x13, 0x97, 0x74 packet, DataRec function may be called once with the whole packet, or twice with 0xbf, 0x13 and 0x97, 0x74 packets, or 4 times with one byte, etc. The program must be flexible enough to handle input stream using some parser. Your current program doesn't do this, it can miss logical packets which are received in a several function calls. Another situation is possible, when several packets are received in one DataRec function call - your program is not ready also for such situation.
Edit.
Typical serial port input stream handling algorithm should look like this:
DataRec function adds received data to input queue and calls parser.
Input queue is some byte array, which contains the data already received, but not parsed yet. New data is added to the end, and parsed packets are removed from the beginning of this queue.
Parser reads the input queue, handles all recognized packets and removes them from the queue, leaving all unrecognized data for the next call.

I think a problem could be that you can't be sure that you receive a full package within the DataReceived event. It is possible that you just got the first half of the packet and half a second later the second half.
So you should implement another layer where you put the data into a buffer. The further proceeding depends on the data format.
If you receive additionally informations like an end mark or the length of the data you could check if the buffer already contains these informations. If yes advance this full package to your routine.
If you don't have this information you have to wait till you receive the next header and forward the data within your buffer till this new header.

Have you checked the memory usage of the program?
Maybe you have a small interop class, memory or something which is not properly freed, adds up after a few hours and make the program run sluggish, causing it to lose data.
I'd use process explorer to check how memory and cpu use change after a few hours. Maybe check for hdd activity, too.
If this does not lead to results, use a full blown profiler like ANTS and try to run the program under the profiler to check for problems.

As Alex Farber points out, there's no guarantee that when your DataReceived handler is invoked, all the bytes are there.
If your buffers are always a fixed size, and at a low rate, you can use the Read function directly, rather than relying on the DataReceived event. Conceptually:
packetSize = 468;
...initialization...
comport.ReadTimeout = 2000; //packets expected every 1000 milliseconds, so give it some slack
while (captureFlag) {
comport.Read(rtSerBuff, 0, packetSize);
...do stuff...
}
This can be put into its own worker thread if you want.
Another approach would be to use the ReadLine method. You mention that the packets have a known starting signature. Do they also have a known ending signature that is guaranteed to not be repeated in the packet? If so, you can set the NewLine property to this ending signature and use ReadLine. Again, you can put this in a worker thread,