Develop Reference

Data errors with my serial receive method

I'm taking data from a serial instrument for plotting on a chart. The data stream is 230 kbps, and the serial pipeline is less than 50% full, data arrives about 100 kbps and actually doesn't vary really and rate or quantity.
Having used just a serial terminal program, like Teraterm, on the same computer; I can capture data and prove that both the source of the data as well as the test reception method are fine and I see no errors to the captured data.
The Windows Forms application I'm developing loses data. I've reduced it from receiving, capturing (in parallel), parsing, and plotting, to just receiving and capturing. And have found that I still see lost data in the capture.
I'm not a long experienced Windows person, so therefore may not know of better ways to accomplish the same functions. Here are the actions I'm taking to perform receive actions:
I'm using a System.IO.Ports.SerialPort class.
I modify the .DataReceived event via:
+= new SerialDataReceivedEventHandler(comPort_DataReceive);
I then call the open() method.
Note: I may be doing something incorrect here, I never clear the .DataReceived event with a -= at any point, instead each time I open, the event is added yet again. Nevertheless, these problems occur even when I've only talked to the port once.
Here's my code for the data receive function. RxString is a string.
private void comPort_DataReceive(object sender, SerialDataReceivedEventArgs e)
{
RxString = comPort.ReadExisting();
this.Invoke(new EventHandler(ParseData));
}
private void ParseData(object sender, EventArgs e)
{
// Save to capture file, if capture is enabled
if ((WriteToFileEnabled == true) && (WriteToFileName != null))
{
writeFileHandle.Write(RxString);
}
return;
// Previously would parse and plot data
}
So, how would persons execute a receive in this situation to get this data without losing it?
Follow on questions are things like: How big is the buffer for serial receive, or do I need to worry about that if I have a reasonably responsive application? Flow control is irrelevant, the remote device is going to send data no matter what, so it would be up to my computer to take that data and process it or ignore it. But how would I know if I've lost data or experienced framing errors and stuff? (I ask that last one without having searched much on the SerialPort class structure, sorry.)

Lets assume that your device is sending messages that are 85 bytes in length. The DataReceive event handler may or may not fire once to receive those 85 bytes. Since it might fire more than once your code must account for that. The DataReceive event handler should read the bytes available and append them to a buffer that is processed later.
Also, only one of the events raised by the SerialPort class can execute at a time. In the example assume the handler has to fire three times to receive the 85 bytes. While processing the first part the other two can't execute. If while processing the first part one of the other events, PinChanged or ErrorReceived, are needed they can't be executed either.
My first two experiences with the SerialPort class were a 9600 bps terminal and a 1 Mbps bluetooth device. What worked for the slower did not work for the faster, but when I figured out how to get the faster to work the slower could use the same methodology.
My methodology:
Before opening the serial port I start two other background threads that run in a do loop. The first one(Receive) reads all available bytes from the serial port, adds them to a buffer, and signals the second thread on every read. The second one(Protocol) determines if a full message has arrived, does any byte to string conversion, updates the UI, etc. Depending on the application I may start a third thread that handles errors and pin changes. All of these threads are throttled by a Threading AutoResetEvent.
My DataReceive event handler has one line in it, a Set on the AutoResetEvent that is throttling Receive.
A VB example of this can be found here SerialPort Methodology. Since adopting this methodology I have not had any of the problems that seem to plague other SerialPort users and have used it successfully with speeds up to 2Mbps.

What is a best approach for multithreading on SerialPort

as I am new in multithreaded application I would like to have some advice from more experienced people before starting to write the code...
I need to queue data received on serial port in serial port event for further processing.
So I have the following event handler:
void jmPort_ReceivedEvent(object source, SerialEventArgs e)
{
SetStatusLabel("Iddle...", lbStatus);
SetPicVisibility(ledNotReceiving, true);
SetPicVisibility(ledReceiving, false);
String st = jmPort.ReadLine();
if (st != null)
{
lines.Enqueue(st); //"lines" is the ConcurrentQueue<string> object
StartDataProcessing(lines); //???
SetStatusLabel("Receiving data...", lbStatus);
SetPicVisibility(ledNotReceiving, false);
SetPicVisibility(ledReceiving, true);
}
else
{
jmPort.Close();
jmPort.Open();
}
}
Within the StartDataProcessing I need to dequeue strings and update MANY UI controlls (using the InvokeRequired...this I already know :-)).
What is the best approach and colision free (without deadlock) approach to achieve this?
How to call StartDataProcessing method in more threads and safely dequeue (TryDequeue) the lines queue, make all needed computations and update UI controlls?
I have to appoint that the communication is very fast and that I am not using the standard SerialPort class. If I simply write all received strings without further processing to console window it works just well.
I am working in .NET 4.5.
Thank you for any advice...
Updated question: Ok, so what will be the best way to run the task from the datareceived event using TPL? Is it necessary to create another class (object) that will process data and use callbacks to update UI or it is possible to load some form method from the event? I'll could be very happy if someone can give me the direction what exactly to do within the datareceived event. What to do as the first step because studying all possible ways is not the solution I have time for. I need to begin with some particular way... There is so many different possible multithreading approaches and after reading about them I am still more confused and I don't know what will be the best a fastest solution... Usual Thread(s), BackgroundWorker, TPL, async-await...? :-( Because my application uses .NET 4.5 I would like to use some state-of-the-art solution :-) Thank you for any advice...

So after a lot of trying it is working to my satisfaction now.
Finally I've used the standard .NET SerialPort class as the third-party Serial class causes somae problems with higher baudrates (115200). It uses WinAPI directly so the finall code was mixed - managed and unmanaged. Now, even the standard .NET 4.5 SerialPort class works well (I've let my application successfully running through a whole night).
So, for everyone that need to deal with C#, SerialPort and higher rates (only for clarification - the device sending messages to PC is the STM32F407 /using USART 2/. I've tried it also with Arduino Due and it works as well) my datareceived event is in the following form now:
private void serialPort1_DataReceived(object sender, System.IO.Ports.SerialDataReceivedEventArgs e)
{
//the SetXXXXX functions are using the .InvokeRequired approach
//because the UI components are updated from another thread than
//the thread they were created in
SetStatusLabel("Iddle...", lbStatus);
SetPicVisibility(Form1.frm.ledNotReceiving, true);
SetPicVisibility(Form1.frm.ledReceiving, false);
String st = serialPort1.ReadLine();
if (st != null)
{
lines.Enqueue(st);
Task.Factory.StartNew(() => StartDataProcessing(lines)); // lines is global ConcurrentQueue object so in fact there is no need to pass it as parameter
SetStatusLabel("Receiving data...", lbStatus);
SetPicVisibility(Form1.frm.ledNotReceiving, false);
SetPicVisibility(Form1.frm.ledReceiving, true);
}
}
Within the StartDataProcessing function:
1. TryDequeue(lines, out str)
2. Use the ThreadPool.QueueUserWorkItem(lCallBack1, tmp); where tmp is needed part of the str (without EOF, without the message number etc.)
lCallBack1 = new WaitCallback(DisplayData);
Within the DisplayData function all the UI controls are updated
This approach mixes the ThreadPool and TPL ways but it is not a problem because the ThreadPool is used by TPL in background operation anyway.
Another working method I've tried was the following:
ThreadPool.QueueUserWorkItem(lCallBack, lines);
instead of :
Task.Factory.StartNew(() => StartDataProcessing(lines));
This method was working well but I've not tested it in over night run.
By my subjective perception the Task.... method updated the controls more smoothly but it can be only my personal feeling :-)
So, I hope this answer will help someone as I know from forums that many people are dealing with with unreliable communication based on the micocontroller <--> PC
My (surprising :-) ) conclusion is that the standard .NET SerialPort is able to handle messages even at higher baudrates. If you still run into troubles with buffer overrun then try to play with the SerialPort buffer size and SerialPort threshold. For me the settings 1024/500 are satisfactory (max size of the message send by microcontroller is 255 bytes so 500 bytes means that 2 messages are in buffer before the event is fired.)
You can also remove all SetXXXX calls from the datareceived event as they are not really needed and they can slow down the communication a little...
I am very close to real-time data capturing now and it is exactly what I've needed.
Good luck to everyone :-)

Within the StartDataProcessing I need to dequeue strings and update MANY UI controlls
No, you do not. You need to dequeue strings and then enqueue them again into the multiple queues for the different segments of the UI.
If you want to be fast, you scatter all operations and definitely the UI into separate windows that run their own separate message pumps and thus can update independently in separate UI threads.
The general process would be:
1 thread handles the serial port and takes the data and queues it.
Another one dequeues it and distributes it to separate processing threads from which
the data goes to multiple output queues all responsible for one part of the UI (depending on whether the UI Will turn a bottleneck).
There is no need to be thread safe in dequeuing. How serial is the data? Can you skip data when another update for the same piece arrives?
Read up on TPL and tasks - there are base libraries for parallel processing which come with a ton of documentation.

Is it not possible for two timer_tick events to work in parallel in reading the same data but performing different tasks?

I developed a C# application for reading calls from GSM Modem. I used a timer to read the port regularly and notify me when there is a incoming call.
Now i am trying to use another timer which will write AT+CSQ -To know the signal quality, on the port and read the port for the quality value. In both the timers i used regular expressions to match and separate the data i need. Now the problem is that only my timer2 which is reading the signal quality is only working but not timer reading for incoming calls.
Timer reading signal strength:
private void tmr_sig_quality_Tick(object sender, EventArgs e)
{
if (port.IsOpen)
{
port.WriteLine("AT+CSQ");
string s= port.ReadExisting();
var match= Regex.Match(s,#"\+CSQ: (\d+),(\d+)");
if (match.Success)
{
progressBar1.Value = int.Parse(match.Groups[1].Value);
}
}
}
Timer reading incoming calls:
private void timer1_Tick(object sender, EventArgs e)
{
s = port.ReadExisting();
var match = Regex.Match(s, "RING[^\\+]*\\+CLIP:\\s*\"(?<phone>[^\"]*)\",(\\d+),\"([^\"]*)\",(\\w*),\"(\\w*)\",(\\w*)");
if (match.Success && s.Contains("RING"))
{
incall_status.Text = "Incoming Call...." + match.Groups["phone"].Value;
incall_status.Visible = true;
}
}
Why is this happening and solution please?

Two major problems. First is ReadExisting(), that will always return an empty string. Except when you are debugging and stepping through the code line by line. That gives the modem enough time to send the response. But this won't work when you run at full speed, you'll need to make a blocking call that ensures your program waits long enough to get all the returned characters. Use ReadLine() instead.
Second killer problem is that you are mixing commands. Your tmr_sig_quality_Tick() method is likely to read whatever timer1_Tick() was supposed to read. You'll need to re-think this approach. Something like a queue of commands that doesn't advance until the response to a command is received.

Not an actual answer to your question, but a general advice for communicating with a GSM modem:
Keep in mind that you have only one serial port, thus only one communication channel. A GSM modem can send spontaneous events, such as the RING, FAX, VOICE events, so there is no guarantee that when you write AT+CSQ the first reply is what you expect, i.e. the signal quality.
Having two timers like you're intending is not a good idea since you'll eventually end up with the response of one command in timer A when it was expected in B, because A read one of the spontaneous events...etc.
A much better and robust way is to have a worker thread that reads and interprets the incoming data and then distributes that in you application as needed. It can also handle the outgoing data. Use concurrent queues and some signaling mechanism (i.e. AutoResetEvent) for exchanging data with this thread, that way you get a cleaner interface to your hardware device and don't need to worry about timing and such in your application.

How can I read from a socket repeatedly?

To start I am coding in C#. I am writing data of varying sizes to a device through a socket. After writing the data I want to read from the socket because the device will write back an error code/completion message once it has finished processing all of the data. Currently I have something like this:
byte[] resultErrorCode = new byte[1];
resultErrorCode[0] = 255;
while (resultErrorCode[0] == 255)
{
try
{
ReadFromSocket(ref resultErrorCode);
}
catch (Exception)
{
}
}
Console.WriteLine(ErrorList[resultErrorCode[0] - 48]);
I use ReadFromSocket in other places, so I know that it is working correctly. What ends up happening is that the port I am connecting from (on my machine) changes to random ports. I think that this causes the firmware on the other side to have a bad connection. So when I write data on the other side, it tries to write data to the original port that I connected through, but after trying to read several times, the connection port changes on my side.
How can I read from the socket continuously until I receive a completion command? If I know that something is wrong with the loop because for my smallest test file it takes 1 min and 13 seconds pretty consistently. I have tested the code by removing the loop and putting the code to sleep for 1 min and 15 seconds. When it resumes, it successfully reads the completion command that I am expecting. Does anyone have any advice?

What you should have is a separate thread which will act like a driver of your external hardware. This thread will receive all data, parse it and transmit the appropriate messages to the rest of your application. This portion of code will give you an idea of how receive and parse data from your hardware.
public void ContinuousReceive(){
byte[] buffer = new byte[1024];
bool terminationCodeReceived = false;
while(!terminationCodeReceived){
try{
if(server.Receive(buffer)>0){
// We got something
// Parse the received data and check if the termination code
// is received or not
}
}catch (SocketException e){
Console.WriteLine("Oops! Something bad happened:" + e.Message);
}
}
}
Notes:
If you want to open a specific port on your machine (some external hardware are configured to talk to a predefined port) then you should specify that when you create your socket
Never close your socket until you want to stop your application or the external hardware API requires that. Keeping your socket open will resolve the random port change
using Thread.Sleep when dealing with external hardware is not a good idea. When possible, you should either use events (in case of RS232 connections) or blocking calls on separate threads as it is the case in the code above.

C# Sending Data over serial port between 2 PC using own Protocol

I have an assignment where I need to load some data like user (pouzivatel) and some int(stav odberu) through link modem with the serial port and store it in my local database. I know how to load data, send data over the serial port, but I need to make it happen in a structure on the image.
First I dial the telephone number of the device with AT command, btw this is working, but I do not know now how to stop and wait for SOH+adresa objektu (SOH+some string about address). Then send data about confirmation (ACK) and wait for new data to come.
The wait sequence is my biggest problem. How do I stop and wait for data being received.

Using the component and utilizing its DataReceived event as suggested in the comments would probably solve your problem easy and effectively. But you may have been looking for something more low-level to do it yourself.
If you want/need to do it in-line without any fancy event based system that would assume you are already in some message queue based environment like WinForms, you could do something like this.
while (true)
{
// check for new data
...
// if you got some, respond to it
...
if (someConditionThatTellsYouYouAreDoneOrSupposedToTerminate) break;
System.Threading.Thread.Sleep(50);
}