Develop Reference

Awaiting data from Serial Port in C#

I have an application that receives data from a wireless radio using RS-232. These radios use an API for communicating with multiple clients. To use the radios I created a library for communicate with them that other software can utilize with minimal changes from a normal SerialPort connection. The library reads from a SerialPort object and inserts incoming data into different buffers depending on the radio it receives from. Each packet that is received contains a header indicating its length, source, etc.
I start by reading the header, which is fixed-length, from the port and parsing it. In the header, the length of the data is defined before the data payload itself, so once I know the length of the data, I then wait for that much data to be available, then read in that many bytes.
Example (the other elements from the header are omitted):
// Read header
byte[] header = new byte[RCV_HEADER_LENGTH];
this.Port.Read(header, 0, RCV_HEADER_LENGTH);
// Get length of data in packet
short dataLength = header[1];
byte[] payload = new byte[dataLength];
// Make sure all the payload of this packet is ready to read
while (this.Port.BytesToRead < dataLength) { }
this.Port.Read(payload, 0, dataLength);
Obviously the empty while port is bad. If for some reason the data never arrives the thread will lock. I haven't encountered this problem yet, but I'm looking for an elegant way to do this. My first thought is to add a short timer that starts just before the while-loop, and sets an abortRead flag when it elapses that would break the while loop, like this:
// Make sure all the payload of this packet is ready to read
abortRead = false;
readTimer.Start();
while (this.Port.BytesToRead < dataLength && !abortRead) {}
This code needs to handle a constant stream of incoming data as quickly as it can, so keeping overhead to a minimum is a concern, and am wondering if I am doing this properly.

You don't have to run this while loop, the method Read would either fill the buffer for you or would throw a TimeoutException if buffer wasn't filled within the SerialPort.ReadTimeout time (which you can adjust to your needs).
But some general remark - your while loop would cause intensive CPU work for nothing, in the few milliseconds it would take the data to arrive you would have thousends of this while loop iterations, you should've add some Thread.Sleep inside.

If you want to truly adress this problem, you need to run the code in the background. There are different options to do that; you can start a thread, you start a Task or you can use async await.
To fully cover all options, the answer would be endless. If you use threads or tasks with the default scheduler and your wait time is expected to be rather short, you can use SpinWait.SpinUntil instead of your while loop. This will perform better than your solution:
SpinWait.SpinUntil(() => this.Port.BytesToRead >= dataLength);
If you are free to use async await, I would recommend this solution, since you need only a few changes to your code. You can use Task.Delay and in the best case you pass a CancellationToken to be able to cancel your operation:
try {
while (this.Port.BytesToRead < dataLength) {
await Task.Delay(100, cancellationToken);
}
}
catch(OperationCancelledException) {
//Cancellation logic
}

I think I would do this asynchronously with the SerialPort DataReceived event.
// Class fields
private const int RCV_HEADER_LENGTH = 8;
private const int MAX_DATA_LENGTH = 255;
private SerialPort Port;
private byte[] PacketBuffer = new byte[RCV_HEADER_LENGTH + MAX_DATA_LENGTH];
private int Readi = 0;
private int DataLength = 0;
// In your constructor
this.Port.DataReceived += new SerialDataReceivedEventHandler(DataReceivedHandler);
private void DataReceivedHandler(object sender, SerialDataReceivedEventArgs e)
{
if (e.EventType != SerialData.Chars)
{
return;
}
// Read all available bytes.
int len = Port.BytesToRead;
byte[] data = new byte[len];
Port.Read(data, 0, len);
// Go through each byte.
for (int i = 0; i < len; i++)
{
// Add the next byte to the packet buffer.
PacketBuffer[Readi++] = data[i];
// Check if we've received the complete header.
if (Readi == RCV_HEADER_LENGTH)
{
DataLength = PacketBuffer[1];
}
// Check if we've received the complete data.
if (Readi == RCV_HEADER_LENGTH + DataLength)
{
// The packet is complete add it to the appropriate buffer.
Readi = 0;
}
}
}

Voice Chat, Increased Delay(Buffer) - Can it be solved?

Okay so i am making a Voice chat software.
I am using NAudio for it, an excellent library.
But i got a problem. The buffer can go up when something happens. I guess it´s from example, when the OS loads something and the Voice Chat application is put on "hold" for a sec. During that time, it adds the data in the buffer, making the current data get delayed.
And as the receiver is playing at the same pace all the time, it will always be delayed.
Now i have a "solution" for this, which is to clear the buffer when it reaches a certain length. Though this is not ideal at all, and is more of a trick than a solution.
Now to the code parts.
First i initialize the things i use.
private NAudio.Wave.WaveInEvent SendStream = new WaveInEvent();
private NAudio.Wave.AsioOut Aut;
private NAudio.Wave.WaveFormat waveformat = new WaveFormat(48000, 16, 2);
private WasapiLoopbackCapture Waloop = new WasapiLoopbackCapture();
private NAudio.Wave.BufferedWaveProvider waveProvider;
waveProvider = new NAudio.Wave.BufferedWaveProvider(waveformat);
waveProvider.DiscardOnBufferOverflow = true;
SendStream.WaveFormat = waveformat;
waveformat is used just so i don´t have to rewrite it all the time.
DiscardOnBufferOverflow is used so if i set a certain lenght on the buffer, for example 20ms. It will Discard anything above, else it will return an exception. I think however it doesn´t do anything if i don´t set a length, it´s probably infinite at default.
And not much else, SendStream is a WaveInEvent, meaning it will run on a BackgroundThread when i use DataAvailable. Waloop is pretty much the same except it´s a loopback.
waveprovider is used in the receiving part to play back the audio.
Waveformat is, well waveformat, it´s importat to set it out, and have all the same, at least in my application.
Here is the receiving part. As you can se, it puts the data in a byte array, then plays it. nothing weird.
byte[] byteData = udpClient.Receive(ref remoteEP);
waveProvider.AddSamples(byteData, 0, byteData.Length);
Here is the sending/recording part.
private void Sendv2()
{
try
{
if (connect == true)
{
if (AudioDevice == "Wasapi Loopback")
{
SendStream.StopRecording();
Waloop.StartRecording();
}
else
{
Waloop.StopRecording();
SendStream.StartRecording();
}
}
}
catch (Exception e)
{
MessageBox.Show(e.Message);
}
}
void Sending(object sender, NAudio.Wave.WaveInEventArgs e)
{
if (connect == true && MuteMic.Checked == false)
{
udpClient.Send(e.Buffer, e.BytesRecorded, otherPartyIP.Address.ToString(), 1500);
}
}
void SendWaloop(object sender, NAudio.Wave.WaveInEventArgs e)
{
byte[] newArray16Bit = new byte[e.BytesRecorded / 2];
short two;
float value;
for (int i = 0, j = 0; i < e.BytesRecorded; i += 4, j += 2)
{
value = (BitConverter.ToSingle(e.Buffer, i));
two = (short)(value * short.MaxValue);
newArray16Bit[j] = (byte)(two & 0xFF);
newArray16Bit[j + 1] = (byte)((two >> 8) & 0xFF);
}
if (connect == true && MuteMic.Checked == false)
{
udpClient.Send(newArray16Bit, newArray16Bit.Length, otherPartyIP.Address.ToString(), 1500);
}
}
Waloop is a Loopback, so it goes through another "channel", but it´s not really important here.
Very simple, When data is available (when it´s recording) and if the connect is true etc, it will just send the buffer.
So pretty much like the receiver part but other way around.
Now how i currently solve this is like this:
if (waveProvider.BufferedDuration.Milliseconds > 40)
{
waveProvider.ClearBuffer();
TimesBufferClear++;
}
So i am clearing the buffer if it´s above 40ms (this is in a Timer at 600ms interval).
(TimesBufferClear++; is just so i can keep track of the times it had been cleared)
Now sadly, i have no idea how to prevent the buffer to be increased, and setting it to a forced state (20ms etc) will just cause the playback to be worse and worse the higher up the buffer goes as it doesn´t really stop, it just ignores the part above i think.
Here is the creationg of the input devices. It is a bit different from ASIO and Wasapi in my implementation, but it pretty much works the same, only real difference is that i tell the UI that ASIO is on or off as you can see in the code, at the end i add the DataAvailable event´s to both SendStream (any input, Microphone etc) and Waloop (Loopback sound that´s being played).
private void CheckAsio()
{
if (NAudio.Wave.AsioOut.isSupported())
{
Aut = new NAudio.Wave.AsioOut();
ASIO.Text += "\nSupported: " + Aut.DriverName;
ASIO.ForeColor = System.Drawing.Color.Green;
Aut.Init(waveProvider);
Aut.Play();
SendStream.NumberOfBuffers = 2;
SendStream.BufferMilliseconds = 10;
}
else
{
AsioSettings.Enabled = false;
ASIO.Text += "\n Not Supported: Wasapi used";
ASIO.ForeColor = System.Drawing.Color.DarkGray;
Wasout = new WasapiOut(AudioClientShareMode.Shared, 0);
Wasout.Init(waveProvider);
Wasout.Play();
SendStream.NumberOfBuffers = 2;
SendStream.BufferMilliseconds = 9;
}
SendStream.DataAvailable += Sending;
Waloop.DataAvailable += SendWaloop;
}
I am not sure if this even can be solved. But as i don´t see other voice chat programs have it, i am guessing there must be something that can be done.

The way this appears to be handled in most applications is to send blocks of data a defined rate (in samples/sec), and drop blocks of data that exceed that rate. If the sender is resource-limited and is not able to maintain the rate, the stream will have audio gaps. This used to happen in audio calls over dial-up when the transmission rate was locked higher than the network connection could handle, or when the CODEC code was taking up too much time.
But from the sound of things the buffering and skips are symptoms, not causes. The root of the problem is that your process is getting shelved for other operations. You can address this by running at a higher process and/or thread priority. The higher your priority the less interruptions you'll have, which will reduce the likelihood of data queuing up to be processed.
In .NET you can raise your process and/or thread priority fairly simply. For process priority:
using System.Diagnostics;
...
Process.GetCurrentProcess().PriorityClass = PriorityClass.Highest;
Or for a thread:
using System.Threading;
...
Thread.CurrentThread.Priority = ThreadPriority.Highest;
This is not a complete solution, since the OS will still steal time-slices from your application under various circumstances, but in a multi-CPU/core system with plenty of memory you should get a fairly good shot at a stable recording environment.
Of course there are no fool-proof methods, and there's always that one slow computer that will mess you up, so you should allow the system to drop excess samples when necessary. Keep track of how much data you're sending out and when it starts to back up, drop anything over your maximum samples/sec. That way your server (or client) isn't going to be buffering increasing amounts of data and lagging further and further behind real-time.
One option there is to time-stamp each packet you send so that the client can choose when to start dropping data to catch up. Better to lose a few milliseconds of output here and there than to drift further and further out of sync.

Serial Port Polling and Data handling

I am trying to read from several serial ports from sensors through microcontrollers. Each serial port will receive more than 2000 measurements (each measurement is 7 bytes, all in hex). And they are firing at the same time. Right now I am polling from 4 serial ports. Also, I translate each measurement into String and append it to a Stringbuilder. When I finish receiving data, they will be ouput in to a file. The problem is the CPU consumption is very high, ranging from 80% to 100%.
I went though some articles and put Thread.Sleep(100) at the end. It reduces CPU time when there is no data coming. I also put Thread.Sleep at the end of each polling when the BytesToRead is smaller than 100. It only helps to a certain extent.
Can someone suggest a solution to poll from serial port and handle data that I get? Maybe appending every time I get something causes the problem?
//I use separate threads for all sensors
private void SensorThread(SerialPort mySerialPort, int bytesPerMeasurement, TextBox textBox, StringBuilder data)
{
textBox.BeginInvoke(new MethodInvoker(delegate() { textBox.Text = ""; }));
int bytesRead;
int t;
Byte[] dataIn;
while (mySerialPort.IsOpen)
{
try
{
if (mySerialPort.BytesToRead != 0)
{
//trying to read a fix number of bytes
bytesRead = 0;
t = 0;
dataIn = new Byte[bytesPerMeasurement];
t = mySerialPort.Read(dataIn, 0, bytesPerMeasurement);
bytesRead += t;
while (bytesRead != bytesPerMeasurement)
{
t = mySerialPort.Read(dataIn, bytesRead, bytesPerMeasurement - bytesRead);
bytesRead += t;
}
//convert them into hex string
StringBuilder s = new StringBuilder();
foreach (Byte b in dataIn) { s.Append(b.ToString("X") + ","); }
var line = s.ToString();
var lineString = string.Format("{0} ---- {2}",
line,
mySerialPort.BytesToRead);
data.Append(lineString + "\r\n");//append a measurement to a huge Stringbuilder...Need a solution for this.
////use delegate to change UI thread...
textBox.BeginInvoke(new MethodInvoker(delegate() { textBox.Text = line; }));
if (mySerialPort.BytesToRead <= 100) { Thread.Sleep(100); }
}
else{Thread.Sleep(100);}
}
catch (Exception ex)
{
//MessageBox.Show(ex.ToString());
}
}
}

this is not a good way to do it, it far better to work on the DataReceived event.
basically with serial ports there's a 3 stage process that works well.
Receiving the Data from the serial port
Waiting till you have a relevant chunk of data
Interpreting the data
so something like
class DataCollector
{
private readonly Action<List<byte>> _processMeasurement;
private readonly string _port;
private SerialPort _serialPort;
private const int SizeOfMeasurement = 4;
List<byte> Data = new List<byte>();
public DataCollector(string port, Action<List<byte>> processMeasurement)
{
_processMeasurement = processMeasurement;
_serialPort = new SerialPort(port);
_serialPort.DataReceived +=SerialPortDataReceived;
}
private void SerialPortDataReceived(object sender, SerialDataReceivedEventArgs e)
{
while(_serialPort.BytesToRead > 0)
{
var count = _serialPort.BytesToRead;
var bytes = new byte[count];
_serialPort.Read(bytes, 0, count);
AddBytes(bytes);
}
}
private void AddBytes(byte[] bytes)
{
Data.AddRange(bytes);
while(Data.Count > SizeOfMeasurement)
{
var measurementData = Data.GetRange(0, SizeOfMeasurement);
Data.RemoveRange(0, SizeOfMeasurement);
if (_processMeasurement != null) _processMeasurement(measurementData);
}
}
}
Note: Add Bytes keeps collecting data till you have enough to count as a measurement, or if you get a burst of data, splits it up into seperate measurements.... so you can get 1 byte one time, 2 the next, and 1 more the next, and it will then take that an turn it into a measurement. Most of the time if your micro sends it in a burst, it will come in as one, but sometimes it will get split into 2.
then somewhere you can do
var collector = new DataCollector("COM1", ProcessMeasurement);
and
private void ProcessMeasurement(List<byte> bytes)
{
// this will get called for every measurement, so then
// put stuff into a text box.... or do whatever
}

First of all consider reading Using Stopwatches and Timers in .NET. You can break down any performance issue with this and tell exactly which part of Your code is causing the problem.
Use SerialPort.DataReceived Event to trigger data receiving process.
Separate receiving process and data manipulation process. Store Your data first then process.
Do not edit UI from reading loop.

I guess what you should be doing is adding an event handler to process incoming data:
mySerialPort.DataReceived += new SerialDataReceivedEventHandler(mySerialPort_DataReceived);
This eliminates the need to run a separate thread for each serial port you listen to. Also, each DataReceived handler will be called precisely when there is data available and will consume only as much CPU time as is necessary to process the data, then yield to the application/OS.
If that doesn't solve the CPU usage problem, it means you're doing too much processing. But unless you've got some very fast serial ports I can't imagine the code you've got there will pose a problem.

Consuming a HTTP stream without reading one byte at a time

I have been trying to read data from the Twitter stream API using C#, and since sometimes the API will return no data, and I am looking for a near-realtime response, I have been hesitant to use a buffer length of more than 1 byte on the reader in case the stream doesn't return any more data for the next day or two.
I have been using the following line:
input.BeginRead(buffer, 0, buffer.Length, InputReadComplete, null);
//buffer = new byte[1]
Now that I plan to scale the application up, I think a size of 1 will result in a lot of CPU usage, and want to increase that number, but I still don't want the stream to just block. Is it possible to get the stream to return if no more bytes are read in the next 5 seconds or something similar?

Async Option
You can use a timer in the async callback method to complete the operation if no bytes are received for e.g. 5 seconds. Reset the timer every time bytes are received. Start it before BeginRead.
Sync Option
Alternatively, you can use the ReceiveTimeout property of the underlying socket to establish a maximum time to wait before completing the read. You can use a larger buffer and set the timeout to e.g. 5 seconds.
From the MSDN documentation that property only applies to a synchronous read. You could perform a synchronous read on a separate thread.
UPDATE
Here's rough, untested code pieced together from a similar problem. It will probably not run (or be bug-free) as-is, but should give you the idea:
private EventWaitHandle asyncWait = new ManualResetEvent(false);
private Timer abortTimer = null;
private bool success = false;
public void ReadFromTwitter()
{
abortTimer = new Timer(AbortTwitter, null, 50000, System.Threading.Timeout.Infinite);
asyncWait.Reset();
input.BeginRead(buffer, 0, buffer.Length, InputReadComplete, null);
asyncWait.WaitOne();
}
void AbortTwitter(object state)
{
success = false; // Redundant but explicit for clarity
asyncWait.Set();
}
void InputReadComplete()
{
// Disable the timer:
abortTimer.Change(System.Threading.Timeout.Infinite, System.Threading.Timeout.Infinite);
success = true;
asyncWait.Set();
}

How can I improve this underperforming, terrible Serial Port code?

I have an ugly piece of Serial Port code which is very unstable.
void port_DataReceived(object sender, SerialDataReceivedEventArgs e)
{
Thread.Sleep(100);
while (port.BytesToRead > 0)
{
var count = port.BytesToRead;
byte[] buffer = new byte[count];
var read = port.Read(buffer, 0, count);
if (DataEncapsulator != null)
buffer = DataEncapsulator.UnWrap(buffer);
var response = dataCollector.Collect(buffer);
if (response != null)
{
this.OnDataReceived(response);
}
Thread.Sleep(100);
}
}
If I remove either Thread.Sleep(100) calls the code stops working.
Of course this really slows things down and if lots of data streams in,
it stops working as well unless I make the sleep even bigger.
(Stops working as in pure deadlock)
Please note the DataEncapsulator and DataCollector are components
provided by MEF, but their performance is quite good.
The class has a Listen() method which starts a background worker to
receive data.
public void Listen(IDataCollector dataCollector)
{
this.dataCollector = dataCollector;
BackgroundWorker worker = new BackgroundWorker();
worker.DoWork += new DoWorkEventHandler(worker_DoWork);
worker.RunWorkerAsync();
}
void worker_DoWork(object sender, DoWorkEventArgs e)
{
port = new SerialPort();
//Event handlers
port.ReceivedBytesThreshold = 15;
port.DataReceived += new SerialDataReceivedEventHandler(port_DataReceived);
..... remainder of code ...
Suggestions are welcome!
Update:
*Just a quick note about what the IDataCollector classes do.
There is no way to know if all bytes of the data that has been sent
are read in a single read operation. So everytime data is read it is
passed to the DataColllector which returns true when a complete and
valid protocol message has been received. In this case here it just
checks for a sync byte, length , crc and tail byte. The real work
is done later by other classes.
*
Update 2:
I replaced the code now as suggested, but still there is something wrong:
void port_DataReceived(object sender, SerialDataReceivedEventArgs e)
{
var count = port.BytesToRead;
byte[] buffer = new byte[count];
var read = port.Read(buffer, 0, count);
if (DataEncapsulator != null)
buffer = DataEncapsulator.UnWrap(buffer);
var response = dataCollector.Collect(buffer);
if (response != null)
{
this.OnDataReceived(response);
}
}
You see this works fine with a fast and stable connection.
But OnDataReceived is NOT called every time data is received.
(See the MSDN docs for more). So if the data gets fragmented
and you only read once within the event data gets lost.
And now I remember why I had the loop in the first place, because
it actually does have to read multiple times if the connection is slow or unstable.
Obviously I can't go back to the while loop solution, so what can I do?

My first concern with the original while-based code fragment is the constant allocation of memory for the byte buffer. Putting a "new" statement here specifically going to the .NET memory manager to allocate memory for the buffer, while taking the memory allocated in the last iteration and sending it back into the unused pool for eventual garbage collection. That seems like an awful lot of work to do in a relatively tight loop.
I am curious as to the performance improvement you would gain by creating this buffer at design-time with a reasonable size, say 8K, so you don't have all of this memory allocation and deallocation and fragmentation. Would that help?
private byte[] buffer = new byte[8192];
void port_DataReceived(object sender, SerialDataReceivedEventArgs e)
{
Thread.Sleep(100);
while (port.BytesToRead > 0)
{
var count = port.BytesToRead;
var read = port.Read(buffer, 0, count);
// ... more code
}
}
My other concern with re-allocating this buffer on every iteration of the loop is that the reallocation may be unnecessary if the buffer is already large enough. Consider the following:
Loop Iteration 1: 100 bytes received; allocate buffer of 100 bytes
Loop Iteration 2: 75 bytes received; allocate buffer of 75 bytes
In this scenario, you don't really need to re-allocate the buffer, because the buffer of 100 bytes allocated in Loop Iteration 1 is more than enough to handle the 75 bytes received in Loop Iteration 2. There is no need to destroy the 100 byte buffer and create a 75 byte buffer. (This is moot, of course, if you just statically create the buffer and move it out of the loop altogether.)
On another tangent, I might suggest that the DataReceived loop concern itself only with the reception of the data. I am not sure what those MEF components are doing, but I question if their work has to be done in the data reception loop. Is it possible for the received data to be put on some sort of queue and the MEF components can pick them up there? I am interested in keeping the DataReceived loop as speedy as possible. Perhaps the received data can be put on a queue so that it can go right back to work receiving more data. You can set up another thread, perhaps, to watch for data arriving on the queue and have the MEF components pick up the data from there and do their work from there. That may be more coding, but it may help the data reception loop be as responsive as possible.

And it can be so simple...
Either you use DataReceived handler but without a loop and certainly without Sleep(), read what data is ready and push it somewhere (to a Queue or MemoryStream),
or
Start a Thread (BgWorker) and do a (blocking) serialPort1.Read(...), and again, push or assemble the data you get.
Edit:
From what you posted I would say: drop the eventhandler and just Read the bytes inside Dowork(). That has the benefit you can specify how much data you want, as long as it is (a lot) smaller than the ReadBufferSize.
Edit2, regarding Update2:
You will still be much better of with a while loop inside a BgWorker, not using the event at all. The simple way:
byte[] buffer = new byte[128]; // 128 = (average) size of a record
while(port.IsOpen && ! worker.CancelationPending)
{
int count = port.Read(buffer, 0, 128);
// proccess count bytes
}
Now maybe your records are variable-sized and you don't don't want to wait for the next 126 bytes to come in to complete one. You can tune this by reducing the buffer size or set a ReadTimeOut. To get very fine-grained you could use port.ReadByte(). Since that reads from the ReadBuffer it's not really any slower.

If you want to write the data to a file and the serial port stops every so often this is a simple way to do it. If possible make your buffer large enough to hold all the bytes that you plan to put in a single file. Then write the code in your datareceived event handler as shown below. Then when you get an oportunity write the whole buffer to a file as shown below that. If you must read FROM your buffer while the serial port is reading TO your buffer then try using a buffered stream object to avoid deadlocks and race conditions.
private byte[] buffer = new byte[8192];
var index = 0;
void port_DataReceived(object sender, SerialDataReceivedEventArgs e)
{
index += port.Read(buffer, index, port.BytesToRead);
}
void WriteDataToFile()
{
binaryWriter.Write(buffer, 0, index);
index = 0;
}