I am attempting to replace a C library used for communications to a device over RS-485. My code works well on some serial devices and all USB-to-RS485 and IP-to-RS485 devices that I have tried. However, on other serial devices, it fails to correctly receive packets that are longer than about 28 bytes. After about that many bytes, it receives some garbage characters and times out (as it never receives enough characters).
On the same hardware, I can receive data through the C library without issue. (So, the hardware is clearly capable).
Initially, I was calling SerialPort.BaseStream to get a stream and then Stream.ReadByte on the stream to receive the data. I found an offhand reference in the documentation indicating that BaseStream is not buffered. So, I switched to calling SerialPort.ReadByte - which made no difference.
In an attempt to be more efficient, I switched the code to call SerialPort.Read for larger chunks of data. This (counter-intuitively) made things worse - failing after the 20th character.
As SerialPort.ReadBufferSize defaults to 4096 bytes (and won't even accept anything smaller) I am confused as to how the packets can overrun at 20-ish bytes.
How does one go about receiving data on a serial port such that the read buffer comes into play?
Related
I would like to read some data from a serial port. However, the packets of data are not well defined - there is no fixed length or end character. What I do to solve this is I wait until the serial port stops sending data for a short time, say 50ms, and then I say one transmission is complete. In Python, I can use the inter_byte_timeout feature of pyserial for this. I am confident the device will send all bytes in one go, so this works.
How do I do it in C#? I have been using System.IO.Ports.SerialPort, but I am open to other solutions.
What doesn't work is the normal ReadTimeout property. That only waits until the first byte. Read will then return an arbitrary amount of data. I also can't use ReceivedBytesThreshold, since I don't know how many bytes it will be in total.
I've read around and i'm hearing mixed things about this. Do you have to split a file into chunks to send it over a stream? or does the OS do that for you?
I have a byte array of about 320,000 values, which i need to get across a network. I can get the first several thousand over but anything after that, it's just set to 0.
I'm using the NetworkStream class, creating a TcpListener / TcpClient, getting the stream from the listener once connected and writing the array to the stream then flushing. Without Success.
Any help would be appreciated.
Cheers,
When using TCP sockets, sending 1024 bytes may or may not be split into chunks by the OS. This behavior at our level should be considered undefined and the receiver should be able to handle a situation like that. What most protocols do is specify a certain (known) message size that contains information such as file size, what range of data it should read, etc. Each message the server constructs will have this header. You as the programmer can specify your chunk sizes, and each chunk must be reconstructed at the receiver level.
Here's a walkthrough:
Server sends a command to the client with information about the file, such as total size, file name, etc.
Client knows how big the command is based on a pre-programmed agreement of header size. If the command is 512 bytes, then the client will keep receiving data until it has filled a 512 byte buffer. Behind the scenes, the operating system could have picked that data up in multiple chunks, but that shouldn't be a worry for you. After all, you only care about reading exactly 512 bytes.
Server begins sending more commands, streaming a file to the client chunk by chunk (512 bytes at a time).
The client receives these chunks and constructs the file over the course of the connection.
Since the client knows how big the file is, it no longer reads on that socket.
The server terminates the connection.
That example is pretty basic, but it's a good groundwork on how communication works.
Recently I started working with sockets. I realized that when reading from a network stream, you can not know how much data is coming in. So either you know in advance how many bytes have to be recieved or you know which bytes.
Since I am currently trying to implement a C# WebSocket server I need to process HTTP requests. A HTTP request can have arbitrary length, so knowing in advance how many bytes is out of the question. But a HTTP request always has a certain format. It starts with the request-line, followed by zero or more headers, etc. So with all this information it should be simple, right?
Nope.
One approach I came up with was reading all data until a specific sequence of bytes was recognized. The StreamReader class has the ReadLine method which, I believe, works like this. For HTTP a reasonable delimiter would be the empty line separating the message head from the body.
The obvious problem here is the requirement of a (preferrably short) termination sequence, like a line break. Even the HTTP specification suggests that these two adjacent CRLFs are not a good choice, since they could also occur at the beginning of the message. And after all, two CRLFs are not a simple delimiter anyways.
So expanding the method to arbitrary type-3 grammars, I concluded the best choice for parsing the data is a finite state machine. I can feed the data to the machine byte after byte, just as I am reading it from the network stream. And as soon as the machine accepts the input I can stop reading data. Also, the FSM could immediately capture the significant tokens.
But is this really the best solution? Reading byte after byte and validating it with a custom parser seems tedious and expensive. And the FSM would be either slow or quite ugly. So...
How do you process data from a network stream when the form is known but not the size?
How can classes like the HttpListener parse the messages and be fast at it too?
Did I miss something here? How would this usually be done?
HttpListener and other such components can parse the messages because the format is deterministic. The Request is well documented. The request header is a series of CRLF-terminated lines, followed by a blank line (two CRLF in a row).
The message body can be difficult to parse, but it's deterministic in that the header tells you what encoding is used, whether it's compressed, etc. Even multi-part messages are not terribly difficult to parse.
Yes, you do need a state machine to parse HTTP messages. And yes you have to parse it byte-by-byte. It's somewhat involved, but it's very fast. Typically you read a bunch of data from the stream into a buffer and then process that buffer byte-by-byte. You don't read the stream one byte at a time because the overhead will kill performance.
You should take a look at the HttpListener source code to see how it all works. Go to http://referencesource.microsoft.com/netframework.aspx and download the .NET 4.5 Update 1 source.
Be prepared to spend a lot of time digging through that and through the HTTP spec.
By the way, it's not difficult to create a program that handles a small subset of HTTP requests. But I wonder why you'd want to do that when you can just use HttpListener and have all the details handled for you.
Update
You are talking about two different protocols. HTTP and WebSocket are two entirely different things. As the Wikipedia article says:
The WebSocket Protocol is an independent TCP-based protocol. Its only relationship to HTTP is that its handshake is interpreted by HTTP servers as an Upgrade request.
With HTTP, you know that the server will send the stream and then close the connection; it's a stream of bytes with a defined end. WebSocket is a message-based protocol; it enables a stream of messages. Those messages have to be delineated in some way; the sender has to tell the receiver where the end of the message is. That can be implicit or explicit. There are several different ways this is done:
The sender includes the length of message in the first few bytes of the message. For example, the first four bytes are a binary integer that says how many bytes follow in that message. So the receiver reads the first four bytes, converts that to an integer, and then reads that many bytes.
The length of the message is implicit. For example, sender and receiver agree that all messages are 80 bytes long.
The first byte of the message is a message type, and each message type has a defined length. For example, message type 1 is 40 bytes, message type 2 is 27 bytes, etc.
Messages have some terminator. In a line-oriented message system, for example, messages are terminated by CRLF. The sender sends the text and then CRLF. The receiver reads bytes until it receives CRLF.
Whatever the case, sender and receiver must agree on how messages are structured. Otherwise the case that you're worried about does crop up: the receiver is left waiting for bytes that will never be received.
In order to handle possible communications problems you set the ReceiveTimeout property on the socket, so that a Read will throw SocketException if it takes too long to receive a complete message. That way, your program won't be left waiting indefinitely for data that is not forthcoming. But this should only happen in the case of communications problems. Any reasonable message format will include a way to determine the length of a message; either you know how much data is coming, or you know when you've reached the end of a message.
If you want to send a message you can just pre-pend the size of the message to it. Get the number of bytes in the message, pre-pend a ulong to it. At the receiver, read the size of a ulong, parse it, then read that amount of bytes from the stream and then close it.
In a HTTP header you can read: Content-Length The length of the request body in octets (8-bit bytes)
I am trying to implement serial communications between a micro-controller and a c# windows application.
I have everything working correctly in the computer to micro-controller direction. But am having trouble working out how exactly to implement communications in the other direction.
My messages are composed of 4 bytes
B0 – Address/Name of the value being sent
B1 – High Byte
B2 – Low Byte
B3 – Checksum = Addition of bytes 0-2
To make sure complete messages are received, I have the micro-controller give up on the current message being received if there is longer than 20ms between bytes, this appears to work well and can tolerate faults in communications that may cause a loss in synchronisation.
I am not sure how I can implement this delay if at all from within a c# application, as I know you have a lot less fine control over timing.
I have seen other ASCII protocols that send a start and stop character but am not sure how to implement this when sending binary data where my values can take any value possible in the byte, and might happen to be whatever the start or stop character is.
I need to keep the micro-controller side basic as I have limited resources, and the controllers primary task requires very precise (sub us range) that converting ascii to decimal may have.
Does anybody have recommendations on how I should be implementing this from either the microprocessor or the computer side.
EDIT
I have looked at some of the other questions on here but they all seem to refer to much larger ASCII based messages.
This is a pretty bad idea. It assumes that the machine on the other end of the wire is capable of providing the same guarantee. 20 msec is no issue at all on a micro-controller. A machine that boots Linux or Windows, no way. A thread that's busy writing to the serial port can easily lose the processor for hundreds of milliseconds. A garbage collection in the case of C#.
Just don't optimize for the exceptional case, there's no point. Timeouts should be in the second range, ten times larger than the worst case you expect. Make a protocol further reliable by framing it. Always a start byte that give the receiver a chance to re-synchronize. Might as well include a length byte, you are bound to need it sooner or later. Favor a CRC over a checksum. Check out RFC916 for a recoverable protocol suggestion, albeit widely ignored. Worked well when I used it, although it needed extra work to get connection attempts reliable, you have to flush the receive buffer.
You can set the read timeout to 20ms by using the following command,
serialPort.ReadTimeout = 20;
This will make the read operation time out after 20ms, in which case you can do what you want.
Don't use ReadExisting with this timeout as it does not rely on the read timeout,
instead use Read() or readByte() and check for a Timeout Exception
incedently the same can be done with WriteTimeout even on successful writes. so take care on that.
C# socket server, which has roughly 200 - 500 active connections, each one constantly sending messages to our server.
About 70% of the time the messages are handled fine (in the correct order etc), however in the other 30% of cases we have jumbled up messages and things get screwed up. We should note that some clients send data in unicode and others in ASCII, so that's handled as well.
Messages sent to the server are a variable length string which end in a char3, it's the char3 that we break on, other than that we keep receiving data.
Could anyone shed any light on our ProcessReceive code and see what could possibly be causing us issues and how we can solve this small issue (here's hoping it's a small issue!)
Code below:
Firstly, I'm sure you know, but it's always worth repeating; TCP is a stream of bytes. It knows nothing of any application level "messages" that you may determine exist in that stream of bytes. All successful socket Recv calls, whether sync or async, can return any number of bytes between 1 and the size of the buffer supplied.
With that in mind you should really be dealing with your message framing (i.e. looking for your delimiter) before you do anything else. If you don't find a delimiter then simply reissue the read using the same SocketAsyncEventArgs, the same buffer and set the offset to where you currently are, this will read some more data into the buffer and you can take another look for the delimiter once the next read has completed... Ideally you'd keep track of where you last got to when searching for a delimiter in this buffer to reduce repeated scanning...
Right now you're not doing that and your use of e.Buffer[e.Offset] == 255 will fail if you get a message that arrives in pieces as you could be referring to any byte in the message if the message is split over multiple reads.
The problem I am seeing is that you are calling Encoding.Unicode.GetString() on a buffer you received in the current read from socket. However, the contents of that buffer might not be a valid unicode encoding of a string.
What you need to do is to buffer your entire stream, and then decode it as a string in one final operation, after you have received all the data.