I'm working on a Unity game that receives OSC messages from the Muse EEG headset. I've tried two 3rd party C# libraries to handle the OSC communication, UnityOSC and unity-OSC-receiver. Both implement the OSC communication with an underlying System.Net.Sockets.UdpClient. Everything is running smoothly on Windows, but on OSX, after a while, I just stop receiving messages every time. No exceptions or error messages, no indication of what went wrong at all, just silence.
My application roughly works as follows:
Start a thread that spawns a process that runs Muse-IO. This makes the headset start sending messages. After starting the process, this thread is just chilling on process.WaitforExit()
Another thread runs a while loop - not in MonoBehavior.Update(), that's not fast enough - that keeps receiving and processing OSC messages. In both libraries, this essentially boils down to calling UdpClient.Receive()
Game uses the processed messages in the normal Unity update cycle.
Some 120 to 140 seconds after the connection is initialized, the stream of messages just stops, and so far I haven't been able to figure out why. The connection indicator light on the headset stays on, but nothing indicates it's actually still sending data.
Things I've ruled out:
It's not because the number of messages or the size of the messages. If I modify the command to the headset to only send some categories of messages, cutting the total in half (from about 600/s to 300/s), the timeout still happens at the same time.
It's not the OSC library. I get the exact same results with both OSC libraries.
It's not the firewall. The firewall is off.
It's probably not the port being used by something else. I tried different ports with the same result.
It doesn't appear to be Muse's OSX driver. When I use their GUI to visualize the incoming data, it keeps receiving data for as long as I want.
I suspect that Mono, Unity or OSX might be shutting down (garbage-collecting?) the Muse-IO process or thread, because the time before the problem occurs seems to be pretty much constant regardless of what I try. But I'm unsure how to further diagnose, let alone fix this now. Any clues, suggestions or amazing solutions would be most welcome.
I found the cause.
After spawning the I/O process, the thread would do
print("Process started!");
process.PriorityClass = ProcessPriorityClass.High;
process.WaitforExit();
In hindsight, that print statement is really poorly placed, oh well. It worked fine on Windows. Changing process priority only requires admin privilege if you're increasing it to Realtime, according to the docs. Not so on Mac though. apparently setting it to High also requires elevated rights on OSX. The resulting exception was silent/undetected/uncaught because it happens outside the main thread.
Then, several minutes later, it seems the thread is garbage collected, including its child process, even though that's still running. That delay really threw me off, making me look for the cause in all the wrong places.
Lessons learned:
Be more careful with possible exceptions when multithreading,
Don't mess with process priority if you don't absolutely have to,
And never trust the docs.
Related
I've got quite an abstract question. I'm working on a project that requires constant device communication. I'm integrating multiple devices onto an external processing unit with a touchpanel to execute certain methods. I.e. the "start videocall" button on the touchpanel activates a relay, turns a display-device, camera-device and microphone-device on, etc.
On the flipside, I'm also trying to monitor these devices. What status do they currently have? Are they enabled/disabled ? What input is the display device currently on?
So far, I've come up with two solutions to prevent a bottleneck in the communication where I'm constantly polling (i.e. every two to five seconds to keep an acurate and up-to-date status) the on-state and input-state of the display-device.
Make use of threading so I can enqueue the different commands and execute them async. By also reading the response async, all communication should be nicely spaced out but I'd have a very "busy" communication line, taking it's toll on the processing unit.
With the help of events have the display-device notify the processor of it's changed status. This would take a lot of stress off of the communication line, but I feel like this is very easily disrupted. If the device doesn't throw it's events correctly (or the events are missed out on) the monitored state does not correspond with the actual state.
I'm curious if there are other ways of going about this issue. As of now, I'm leaning towards the second one because it stresses the processing unit a whole lot less, I just feel like I should be building in a lot of safeguards to prevent an inacurate representation of the actual device-states.
The project runs in C# on .Net 3.5.
Polling works, but it isn't fun or optimal. Reactive is best but as you've mentioned there may be a hiccup insuring your still listening to to the device and not just standing by for nothing. In this situation it makes since to optimize both processes. Poll when you're waiting or haven't heard a response in so long and listen when your polling returns good info, passing the polling.
That said, you shouldn't worry about taxing the unit too much with polling on various threads. This sounds like a purpose device so as long as you're not running it hot or stressing it to max all the time then using your resources are perfectly fine.
While trying to read data from a TCP socket using ReceiveAsync.Completed, I experience a few ms delay (1-5ms, once in a while) between the time the data arrives to my application and the time it arrives to the machine. I use Wireshark to compare the timestamps, PSH bit is turned on for those messages as well and the application is not busy or blocking anything. What am I missing ?
Thanks.
With such a small delay it could come from anywhere, the garbage collector kicking in could delay you that much easily.
Wireshark isn't operating on the same level as your application. It could be that it's getting the data at different times than you just because the system is supplying it a little faster. Or maybe their times are off. Maybe your times are off. With such a small difference its hard to tell.
I bet you're not doing anything wrong.
I have written my network code for my app using Sockets. I've tested the library on a windows PC and it is very fast. Whether it's many small "packets" (by packets I mean send operations, I am using TCP which is streaming) of data in a small amount of time, or a few large ones, it works perfectly.
I moved the code into a test app for the iPhone. Ran the test, great speeds again. about 5 MB sent over wifi between two phones in about 3 seconds.
Im using synchronous Socket.Send() operations in a threadpool thread, and using ReceiveAsync() for receiving. (I've also tried the BeginReceive() style, but it behaves the same.
However, once I move the code into my app, I start to encounter problems. The general problem is that the receiving app doesnt seem to receive consistently. I could send several "packets" of data from the host phone, and it can be anywhere from instantly, to a few seconds to 10 minutes before they appear on the receiving end. I've been on this problem for 2 weeks now (evenings/weekends) and i've spent days testing it to try understand exactly what I can do to reproduce it, but its never the same twice.
At the moment, im putting it down to threadpool threads being exhausted. I've used
ThreadPool.SetMaxThreads()
to increase the thread count drastically, but it doesnt make any difference. Its like as if the completed callback in SocketAsyncEventArgs cannot get a thread to operate on, so it just sits there. I've gone through my code and refactored anything that was un-necessarily using threads with a loop performing periodic tasks and changed them to timers, but the problem remains.
I have literally no idea where to turn with this one. Im hoping its maybe a bug in Monotouch (not that Im trying to blame those guys!).
Im not sure what code to post, as the network code has been tested on its own and operates fine. I've tested it with 1,000,000 send/receives to check if there was some kind of leak but found no problems.
It seems like the data is getting to the recipient, but the callback is somehow getting severely delayed in getting called, sometimes by several minutes.
Can anyone point me in a direction of why this might be happening?
Thank you.
My problem with this was caused by having a GKSession also initialized. I hope this is a bug in Monotouch/Mono that can be fixed, as I do need both network features enabled. As soon as i disabled the GKSession, the socket code flows freely.
Is there a way to make my Windows Mobile 6.1 application react to system reset?
If it matters, I am using Compact Framework 3.5.
A system reset (i.e. a call to the kernel with IOCTL_HAL_REBOOT) goes through the power manager (well it should, the OEM might have opted otherwise but that would be rare). During the reset, the PM broadcasts a power manegment notification which you can requent to receive by calling the RequestPowerNotifications APIs. One of the last things the power manager does is goes single-threaded and calls all drivers' Xxx_PowerDown methods. The drivers at this point have the option of doing clean up, final flushing, etc. to get into a stable state and then the device restarts.
From an app perspective, the RequestPowerNotifications call is the only thing you can do. Be aware that the PM broadcasts the notification and just keeps on moving. It does not wait for anyone to complete any activity (like it does in calling Xxx_PowerDown). This often means that your application will never get the notification, or if it does you may only get the opportunity to run a couple lines of code (I think the most I've seen was about 4 lines). On a suspend/resume, the app code typically runs after you wake back up. With a reset, since RAM and processor are reset, you don't get that opportunity.
So, after that long-winded response, the short answer is that you might be able to get the notification, but even if you do it's unlikely you'll actually be able to do anything in response to it.
I know this is probably the canonical "It depends..." question but I'd appreciate any pointers as to where to start looking.
I have a client/server app talking over ethernet. In one computer I run the server and a client and on another just the client. One runs Vista and one runs XP. After an uptime of about 3 weeks the entire computer freezes and nothing works, not mouse, not keyboard, nothing -just power off. Every ten seconds the server sends a ping message to see if the clients are alive, other than that just a few small messages go back and forth every day.
I'm trying to find out if it's me causing it or something else. I've started a session and after a few days I thought I'd check for strange increases in memory use but beyond that I have very few ideas.
Some thoughts to consider:
You know the computer doesn't respond, but that doesn't mean it's hung. Does it respond to a ping?
Maybe the disk activity light is on all the time?
You say "no keyboard" - do you mean no caps lock or num lock lights?
Although the .NET application may be the only one you're running at the time, that does not imply it is the cause of the problem. Some background job could be doing it.
For example, I notice that Retrospect backup, when it is creating a snapshot, freezes the entire system for 10-15 minutes. I mean, no caps lock, the clock in the task bar doesn't update, no CTRL-ALT-DEL, can't type into an "Answer" text box in SO, nothing. It had nothing to do with what I was doing at the time, which was answering a question on SO.
After it came back, SO asked if I was a human. My feelings were hurt. ;-)
You could attach a kernel debugger to the OS. That way you should be able to inspect the state of the OS and your process even if the OS is completely unresponsive. (Unfortunately, it's a lot harder than just hitting "break" in VS. I suggest reading John Robbin's "Debugging Applications for .NET and Windows" before trying that.)
You could also try to create memory dumps of your application in regular intervals. You might have to do a little scripting for that, though. (usually, you'd create a dump with a keystroke, using a tool like userdump or adplus, but if the OS is not responding to keystrokes, that won't work.) That way, you know what state your process is in during or shortly before a hang.
This page: http://blogs.msdn.com/debuggingtoolbox/default.aspx is a good starting point for scripting WinDbg. (If you don't know what to do with a memory dump, I'd again suggest John Robbin's excellent book on debugging!)
Other than that, I can only think of standard debugging tricks: does the problem occur on every PC? Does it happen if there are no client requests? Does it happen sooner if there are more client requests? Does it happen sooner if there is less available physical memory? Try removing parts of your application (maybe on a separate server for testing) and see if the problem still occurs, and so on. Try running it in a VM so you can see if it uses the CPU, harddisk, or network during those "hangs".
This isn't going to be the answer, but I'd advise starting by checking your OS event logs and running a perfmon to keep track of memory, cpu usage etc.
Which computer freezes, the server or client? And what OSes are they running respectively?
As Daniel L noted, tight polling loops can really kill the CPU. If you can, change your code to use event handlers, it's a much more robust solution.
Finally, are you certain there's not a hardware problem on the freezing computer?