I am writing a program in C# under Windows 10 to capture SysEx messages from MIDI devices. I'm using a callback function as follows:
private delegate void MidiInProc(
int handle,
uint msg,
int instance,
int param1,
int param2);
[DllImport("winmm.dll")]
private static extern int midiInOpen(
out int handle,
int deviceID,
MidiInProc proc,
int instance,
int flags);
private int hHandle;
public MidiInput(int deviceID)
{
MidiInProc midiInProc = MidiInProcess;
int hResult = midiInOpen(
out hHandle,
deviceID,
midiInProc,
0,
CALLBACK_FUNCTION | MIDI_IO_STATUS);
for (int i = 0; i < NUM_MIDIHDRS; ++i)
{
InitMidiHdr(i);
}
}
private void MidiInProcess(int hMidiIn, uint uMsg, int dwInstance, int dwParam1, int dwParam2)
{
switch (uMsg)
{
case MIM_OPEN:
break;
case MIM_CLOSE:
break;
case MIM_DATA:
QueueData(dwParam1);
break;
case MIM_MOREDATA:
QueueData(dwParam1);
break;
case MIM_LONGDATA:
QueueLongData((IntPtr)dwParam1);
break;
case MIM_ERROR:
throw new ApplicationException(string.Format("Invalid MIDI message: {0}", dwParam1));
case MIM_LONGERROR:
throw new ApplicationException("Invalid SysEx message.");
default:
throw new ApplicationException(string.Format("unexpected message: {0}", uMsg));
}
}
The InitMidiHdr method creates NUM_MIDIHDRS headers and buffers on the heap and passes their addresses to the driver with calls to MidiInPrepareHeader and MidiInAddBuffer. When SysEx data is received, the callback switches to the MIM_LONGDATA case and queues the buffer address to another thread which dequeues it, processes it, and then passes it back to the driver for further use via another call to MidiInAddBuffer.
Now, some programs do not use a separate thread, instead processing SysEx data on the callback's thread. From what I have read, this works most of the time, but not always, and is against the MSDN's advice: "Applications should not call any multimedia functions from inside the callback function[.]" This may not be a problem with contemporary drivers, but some users have, in the past, reported deadlocks when calling MidiInAddBuffer directly from the callback.
But...
When I use a worker thread to call MidiInAddBuffer, I can't think of a way to guarantee that it will keep up with calls to the callback. Sure, if the only thing the worker does is return the buffer, it will probably stay ahead of the callback, but relying on one thread to stay ahead of another without explicit synchronization is a bad practice. (Having the callback wait on a signal from the worker that it has returned the buffer doesn't work, as MidiInAddBuffer blocks when called from another thread until the callback returns, which leads to a deadlock if you condition its return on a return from MidiInAddBuffer in the worker thread.) Thus, it's possible in that scenario for the driver to run out of buffers while the MIDI device is still sending SysEx data. (Indeed, even in the scenario where the callback does all the processing itself, it too might fall behind, with the driver using up all of its buffers before the device stops sending SysEx data.)
I have found a couple of open-source projects that use a worker to call MidiInAddBuffer, but both appear to rely on the worker to stay ahead of calls to the callback. In fact, in one case, I added a 50ms sleep call to the worker, with the result being that it fell far behind the callback thread, which ultimately meant that only about half of the SysEx messages made it to the callback. Apparently, the driver doesn't buffer SysEx data internally and, when it has no buffers left to use, throws SysEx data away until it gets another buffer.
Now, memory being cheap and all that, one "solution" is to have a lot of buffers. My Behringer BCF2000, however, sends almost 500 distinct SysEx messages in a single dump, each in its own buffer. That's not an impossible number to supply, but it requires guesswork as to how many is really enough (and, given some of the exotic things people use SysEx messages to do, like pass around audio samples, such an approach could get unwieldy).
Alas, the MIM_LONGERROR case never gets called when my tests show that my code isn't keeping up, so that's no help.
So, here's my question: If my code is unable to keep up with the rate at which SysEx buffers are consumed by a MIDI device's driver and I end up missing some SysEx data, is there any way to at least detect that I have missed that data?
There is no reporting mechanism for missed SysEx buffers.
At the MIDI speed of 3125 bytes/s, the 50 ms delay corresponds to a buffer of 156 bytes; if the actual messages are shorter, you are guaranteed to fall behind.
But real code will not have such a consistent delay, so this is not a realistic test. Your threads might get random scheduling delays, but this is no problem as long as you have enough buffers queued. (And if some other code with higher priority prevents your code from being executed at all, there's nothing you can do anyway.)
Related
One of our programs suffered from a severe memory leak: its process memory rose by 1 GB per day at a customer site.
I could set up the scenario in our test center, and could get a memory leak of some 700 MB per day.
This application is a Windows service written in C# which communicates with devices over a CAN bus.
The memory leak does not depend on the rate of data the application writes to the CAN bus. But it clearly depends on the number of messages received.
The "unmanaged" side of reading the messages is:
[StructLayout(LayoutKind.Sequential, Pack = 1)]
public struct CAN_MSG
{
public uint time_stamp;
public uint id;
public byte len;
public byte rtr;
[MarshalAs(UnmanagedType.ByValArray, SizeConst = 8)]
public byte[] a_data;
}
[DllImport("IEICAN02.dll", EntryPoint = "#3")]
public static extern int CAN_CountMsgs(ushort card_idx, byte can_no, byte que_type);
//ICAN_API INT32 _stdcall CAN_CountMsgs(UINT16 card_idx, UINT8 can_no,UINT8 que_type);
[DllImport("IEICAN02.dll", EntryPoint = "#10")]
public static extern int CAN_ReadMsg(ushort card_idx, byte can_no, ushort count, [MarshalAs(UnmanagedType.LPArray), Out()] CAN_MSG[] msg);
//ICAN_API INT32 _stdcall CAN_ReadMsg(UINT16 card_idx, UINT8 can_no, UINT16 count, CAN_MSG* p_obj);
We use essentially as follows:
private void ReadMessages()
{
while (keepRunning)
{
// get the number of messages in the queue
int messagesCounter = ICAN_API.CAN_CountMsgs(_CardIndex, _PortIndex, ICAN_API.CAN_RX_QUE);
if (messagesCounter > 0)
{
// create an array of appropriate size for those messages
CAN_MSG[] canMessages = new CAN_MSG[messagesCounter];
// read them
int actualReadMessages = ICAN_API.CAN_ReadMsg(_CardIndex, _PortIndex, (ushort)messagesCounter, canMessages);
// transform them into "our" objects
CanMessage[] messages = TransformMessages(canMessages);
Thread thread = new Thread(() => RaiseEventWithCanMessages(messages))
{
Priority = ThreadPriority.AboveNormal
};
thread.Start();
}
Thread.Sleep(20);
}
}
// transformation process:
new CanMessage
{
MessageData = (byte[])messages[i].a_data.Clone(),
MessageId = messages[i].id
};
The loop is executed once per every ~30 milliseconds.
When I call RaiseEventWithCanMessages(messages) in the same thread, the memory leak disappears (well, not completely, some 10 MB per day - i.e. about 1% of the original leak - remain, but that other leak is likely unrelated).
I do not understand how this creation of threads can lead to a memory leak. Can you provide me with some information how the memory leak is caused?
Addendum 2018-08-16:
The application starts of with some 50 MB of memory, and crashes at some 2GB. That means, that Gigabytes of memory are available for most of the time.
Also, CPU is at some 20% - 3 out of 4 cores are idle.
The number of threads used by the application remains rather constant around ~30 threads.
Overall, there are plenty of resources available for the Garbage Collection. Still, GC fails.
With some 30 threads per second, and a memory leak of 700 MB per day, on average ~300 bytes of memory leak per freshly created thread; with ~5 messages per new thread, some ~60bytes per message. The "unmanaged" struct does not make it into the new thread, its contents are copied into a newly instantiated class.
So: why does GC fail despite the enormous amount of resources available for it?
You're creating 2 arrays and a thread every ~30 milliseconds, without any coordination between them. The arrays could be a problem, but frankly I'm much more worried about the thread - creating threads is really, really expensive. You should not be creating them this frequently.
I'm also concerned about what happens if the read loop is out-pacing the thread - i.e. if RaiseEventWithCanMessages takes more time than the code that does the query/sleep. In that scenario, you'd have a constant growth of threads. And you'd probably also have all the various RaiseEventWithCanMessages fighting with each-other.
The fact that putting RaiseEventWithCanMessages inline "fixes" it suggests that the main problem here is either the sheer number of threads being created (bad), or the many overlapping and growing numbers of concurrent RaiseEventWithCanMessages.
The simplest fix would be: don't use the extra threads here.
If you actually want concurrent operations, I would have exactly two threads here - one that does the query, and one that does whatever RaiseEventWithCanMessages is, both in a loop. I would then coordinate between the threads such that the query thread waits for the previous RaiseEventWithCanMessages thing to be complete, such that it hands it over in a coordinated style - so there is always at most one outstanding RaiseEventWithCanMessages, and you stop running queries if it isn't keeping up.
Essentially:
CanMessage[] messages = TransformMessages(canMessages);
HandToConsumerBlockingUntilAvailable(messages); // TODO: implement
with the other thread basically doing:
var nextMessages = BlockUntilAvailableFromProducer(); // TODO: implement
A very basic implementation of this could be just:
void HandToConsumerBlockingUntilAvailable(CanMessage[] messages) {
lock(_queue) {
if(_queue.Length != 0) Monitor.Wait(_queue); // block until space
_queue.Enqueue(messages);
if(queue.Length == 1) Monitor.PulseAll(_queue); // wake consumer
}
}
CanMessage[] BlockUntilAvailableFromProducer() {
lock(_queue) {
if(_queue.Length == 0) Monitor.Wait(_queue); // block until work
var next = _queue.Dequeue();
Monitor.Pulse(_queue); // wake producer
return _next;
}
}
private readonly Queue<CanMessage[]> _queue = new Queue<CanMessage[]>;
This implementation enforces that there is no more than 1 outstanding unprocessed Message[] in the queue.
This addresses the issues of creating lots of threads, and the issues of the query loop out-pacing the RaiseEventWithCanMessages code.
I might also look into using the ArrayPool<T>.Shared for leasing oversized arrays (meaning: you need to be careful not to read more data than you've actually written, since you might have asked for an array of 500 but been given one of size 512), rather than constantly allocating arrays.
I'm taking data from a serial instrument for plotting on a chart. The data stream is 230 kbps, and the serial pipeline is less than 50% full, data arrives about 100 kbps and actually doesn't vary really and rate or quantity.
Having used just a serial terminal program, like Teraterm, on the same computer; I can capture data and prove that both the source of the data as well as the test reception method are fine and I see no errors to the captured data.
The Windows Forms application I'm developing loses data. I've reduced it from receiving, capturing (in parallel), parsing, and plotting, to just receiving and capturing. And have found that I still see lost data in the capture.
I'm not a long experienced Windows person, so therefore may not know of better ways to accomplish the same functions. Here are the actions I'm taking to perform receive actions:
I'm using a System.IO.Ports.SerialPort class.
I modify the .DataReceived event via:
+= new SerialDataReceivedEventHandler(comPort_DataReceive);
I then call the open() method.
Note: I may be doing something incorrect here, I never clear the .DataReceived event with a -= at any point, instead each time I open, the event is added yet again. Nevertheless, these problems occur even when I've only talked to the port once.
Here's my code for the data receive function. RxString is a string.
private void comPort_DataReceive(object sender, SerialDataReceivedEventArgs e)
{
RxString = comPort.ReadExisting();
this.Invoke(new EventHandler(ParseData));
}
private void ParseData(object sender, EventArgs e)
{
// Save to capture file, if capture is enabled
if ((WriteToFileEnabled == true) && (WriteToFileName != null))
{
writeFileHandle.Write(RxString);
}
return;
// Previously would parse and plot data
}
So, how would persons execute a receive in this situation to get this data without losing it?
Follow on questions are things like: How big is the buffer for serial receive, or do I need to worry about that if I have a reasonably responsive application? Flow control is irrelevant, the remote device is going to send data no matter what, so it would be up to my computer to take that data and process it or ignore it. But how would I know if I've lost data or experienced framing errors and stuff? (I ask that last one without having searched much on the SerialPort class structure, sorry.)
Lets assume that your device is sending messages that are 85 bytes in length. The DataReceive event handler may or may not fire once to receive those 85 bytes. Since it might fire more than once your code must account for that. The DataReceive event handler should read the bytes available and append them to a buffer that is processed later.
Also, only one of the events raised by the SerialPort class can execute at a time. In the example assume the handler has to fire three times to receive the 85 bytes. While processing the first part the other two can't execute. If while processing the first part one of the other events, PinChanged or ErrorReceived, are needed they can't be executed either.
My first two experiences with the SerialPort class were a 9600 bps terminal and a 1 Mbps bluetooth device. What worked for the slower did not work for the faster, but when I figured out how to get the faster to work the slower could use the same methodology.
My methodology:
Before opening the serial port I start two other background threads that run in a do loop. The first one(Receive) reads all available bytes from the serial port, adds them to a buffer, and signals the second thread on every read. The second one(Protocol) determines if a full message has arrived, does any byte to string conversion, updates the UI, etc. Depending on the application I may start a third thread that handles errors and pin changes. All of these threads are throttled by a Threading AutoResetEvent.
My DataReceive event handler has one line in it, a Set on the AutoResetEvent that is throttling Receive.
A VB example of this can be found here SerialPort Methodology. Since adopting this methodology I have not had any of the problems that seem to plague other SerialPort users and have used it successfully with speeds up to 2Mbps.
I'm embarrassed to have to ask such a question, but I'm having a rough time figuring out how to reliably read data over a serial port with the .NET SerialPort class.
My first approach:
static void Main(string[] args)
{
_port = new SerialPort
{
PortName = portName,
BaudRate = 57600,
DataBits = 8,
Parity = Parity.None,
StopBits = StopBits.One,
RtsEnable = true,
DtrEnable = false,
WriteBufferSize = 2048,
ReadBufferSize = 2048,
ReceivedBytesThreshold = 1,
ReadTimeout = 5000,
};
_port.DataReceived += _port_DataReceived;
_port.Open();
// whatever
}
private void _port_DataReceived(object sender, SerialDataReceivedEventArgs e)
{
var buf = new byte[_port.BytesToRead];
var bytesRead = _port.Read(buf, 0, buf.Length);
_port.DiscardInBuffer();
for (int i = 0; i < bytesRead; ++i)
{
// read each byte, look for start/end values,
// signal complete packet event if/when end is found
}
}
So this has an obvious problem; I am calling DiscardInBuffer, so any data which came in after the event was fired is discarded, i.e., I'm dropping data.
Now, the documentation for SerialPort.Read() does not even state if it advances the current position of the stream (really?), but I have found other sources which claim that it does (which makes sense). However, if I do not call DiscardInBuffer I eventually get an RXOver error, i.e., I'm taking too long to process each message and the buffer is overflowing.
So... I'm really not a fan of this interface. If I have to process each buffer on a separate thread I'll do that, but that comes with its own set of problems, and I'm hoping that I am missing something as I don't have much experience with this interface.
Jason makes some good points about reducing UI access from the worker thread, but an even better option is to not receive the data on a worker thread in the first place.
Use port.BaseStream.ReadAsync to get your data, event-driven, on the thread where you want it. I've written more about this approach at http://www.sparxeng.com/blog/software/must-use-net-system-io-ports-serialport
To correctly handle data from a serial port you need to do a couple of things.
First, don't handle the data in your receive event. Copy the data somewhere else and do any processing on another thread. (This is true of most events - it is a bad idea to do any time-consuming processing in an event handler as it delays the caller and can introduce problems. You also need to be careful as your event is raised on a different thread to your main application)
Secondly, you can't guarantee that you will receive exactly one packet, or a complete packet when you receive data - it may come to you in small fragments.
So the upshot of this is that you should create your own buffer (big enough to hold several packets), and when you receive data, append it to your buffer. Then in another thread you can process the buffer, looking to see if you can decode a packet from it and then consume that data. You may have to skip the end of a partial packet before you find the start of a valid one. If you don't have enough data to build a full packet, then you may need to wait for a bit until more data arrives.
You shouldn't call Discard on the port - just read the data and consume it. Each time you are called, there will be another fragment of data to process. It does not remember the data from previous calls - each time your event is called, it is given a small burst of data that has arrived since you were last called. Just use the data you've been given and return.
As a last suggestion: Don't change any settings for the port unless you specifically need to for it to operate properly. So you must set the baud rate, data/stop bits and parity, but avoid trying to change properties like the Rts/Dtr, buffer sizes and read thresholds unless you have a good reason to think you know better than the author of the serial port. Most serial devices work in an industry standard manner these days, and changing these low-level options is very likely to cause trouble unless you're talking to some unusual equipment and you intimately know the hardware.
In particular setting the ReceivedBytesThreshold to 1 is probably what is causing the failure you've mentioned, because you are asking the serial port to call your event handler with only one byte at a time, 57,600 times per second - giving your event handler only 0.017 milliseconds to process each byte before you'll start to get re-entrant calls.
DiscardInBuffer is typically only used immediately after opening a serial port. It is not required for standard serial port communication so you should not have it in your dataReceived handler.
I'm writing a message layer for my distributed system. I'm using IOCP, ie the Socket.XXXAsync methods.
Here's something pretty close to what I'm doing (in fact, my receive function is based on his):
http://vadmyst.blogspot.com/2008/05/sample-code-for-tcp-server-using.html
What I've found now is that at the start of the program (two test servers talking to each other) I each time get a number of SAEA objects where the .Buffer is entirely filled with zeroes, yet the .BytesTransferred is the size of the buffer (1024 in my case).
What does this mean? Is there a special condition I need to check for? My system interprets this as an incomplete message and moves on, but I'm wondering if I'm actually missing some data. I was under the impression that if nothing was being received, you'd not get a callback. In any case, I can see in WireShark that there aren't any zero-length packets coming in.
I've found the following when I Googled it, but I'm not sure my problem is the same:
http://social.msdn.microsoft.com/Forums/en-US/ncl/thread/40fe397c-b1da-428e-a355-ee5a6b0b4d2c
http://go4answers.webhost4life.com/Example/socketasynceventargs-buffer-not-ready-121918.aspx
I am sure not what is going on in the linked example. It appears to be using asynchronous sockets in a synchronous way. I cannot see any callbacks or similar in the code. You may need to rethink whether you need synchronous or asynchronous sockets :).
To the problem at hand stems from the possibility that your functions are trying to read/write to the buffer before the network transmit/receive has been completed. Try using the callback functionality included in the async Socket. E.g.
// This goes into your accept function, to begin receiving the data
socketName.BeginReceive(yourbuffer, 0, yourbuffer.Length,
SocketFlags.None, new AsyncCallback(OnRecieveData), socketName);
// In your callback function you know that the socket has finished receiving data
// This callback will fire when the receive is complete.
private void OnRecieveData(IAsyncResult input) {
Socket inSocket = (Socket)input.AsyncState; // This is just a typecast
inSocket.EndReceive(input);
// Pull the data out of the socket as you already have before.
// state.Data.Write ......
}
There is a a single-threaded server using .NET Socket with TCP protocol, and Socket.Pool(), Socket.Select(), Socket.Receive().
To send, I used:
public void SendPacket(int clientid, byte[] packet)
{
clients[clientid].socket.Send(packet);
}
But it was very slow when sending a lot of data to one client (halting the whole main thread), so I replaced it with this:
public void SendPacket(int clientid, byte[] packet)
{
using (SocketAsyncEventArgs e = new SocketAsyncEventArgs())
{
e.SetBuffer(packet, 0, packet.Length);
clients[clientid].socket.SendAsync(e);
}
}
It works fine on Windows with .NET (I don't know if it's perfect), but on Linux with Mono, packets are either dropped or reordered (I don't know). Reverting to slow version with Socket.Send() works on Linux. Source for whole server.
How to write non-blocking SendPacket() function that works on Linux?
I'm going to take a guess that it has to do with your using statement and your SendAsync call. Perhaps e falls out of scope and is being disposed while SendAsync is still processing the buffer. But then this might throw an exception. I am really just taking a guess. Try removing the using statement and see what happens.
I would say by not abusing the async method. YOu will find nowhere a documentation stating that this acutally is forced to maintain order. it queues iem for a scheuler which get distributed to threads, and by ignoring that the oder is not maintained per documentation you open yourself up to implementation details.
The best possibly is to:
Have a queue per socket.
When you write dasta into this queue, and there is no worker thread, start a work item (ThreadPool) to process the thread.
This way you have separate distinct queues that maintain order. Only one thread will ever process one queue / socket.
I got the same problem; Linux and windows react not in the same way with SendAsync. Sometimes linux truncate the data, but there is a workaround. First of all you need to use a queue. Each time you use SendAsync you have to check the callback.
If e.Offset + e.BytesTransferred < e.Buffer.Length, you just have to e.SetBuffer(e.Offset + e.BytesTransferred, e.Buffer.Length - e.BytesTransferred - e.Offset); and call SendAsync again.
I dont know why mono-linux believe it's completed before sending all the data and it's strange but i'm sure he does.
just like #mathieu, 10y later, I can confirm on Unity Mono+Linux complete callback is called without all bytes being sent in some cases. For me it was large packets only.