I have a C# (Mono) app that gets data from a sensor and passes it to a Python script for processing. It runs on a Raspberry Pi/Raspbian. The script is started at program initialization and is running in background waiting for data to be passed to it.
The data consists of 1000 samples, 32 bit double (in total, 32 kbits). Currently I write the data in a file and pass the file path to the script.
I was thinking of speeding up processing time by passing the data directly to the script, avoiding writing the file. However, I see that the number of characters of command line arguments is a limited.
Is there a solution to pass the data to the Python script by avoiding file writing/reading? I've read something about memory mapping, but I do not know if it's a solution.
Writing data to a file relies on the speed of the Micro SDCard used in the raspberry pi. It is good practice to run your program in one application so that all the operation is using the same memory pool.
If not possible, to utilize the RAM you can use communication protocol like MQTT. it is lightweight for Raspberry Pi.
MQTT requires broker (server), and you can start by using mosquitto (http://mosquitto.org/). install it using apt install mosquitto for Raspbian.
In your python and mono source code you can use mqtt client library of your choice. An example of client library is paho-mqtt (https://www.eclipse.org/paho/).
you could also use raw socket but please note that there's maximum data per packet depends on your environment, typically 8KB. https://docs.python.org/3/library/socket.html#socket.socket.recv
Is there a solution to pass the data to the Python script by avoiding
file writing/reading?
I can only think of two approaches:
You could open a socket for communication between both programs through localhost. The C# program would send data to that socket, and the Python program would read it.
Write both programs in Python or C#. The single program would capture
data and process it.
I've read something about memory mapping, but I do not know if it's a
solution.
Memory mapping is about loading a file in memory, and once the work on it is finished, write it back at once. Since you have two different processes, I don't think it is applyable here.
Hope this helps.
Related
I am trying to send over image data from a compiled C++ process to a compiled C# process. The C++ process is accessing the webcam and doing some processing on the image. The image is represented by an 2D array of pixels with each pixel value being an 8 bit value (0-255) which is the gray-scale value of that pixel.
The image size is 640 by 480.
The C# application does some more processing and displays this image onto the screen. The processes are both running at the same time on my laptop (Windows 7 OS) but I cannot make a single process that does all the steps which is why I need my C++ and C# code to communicate.
I was wondering what is the best way to do this? I read about writing a UDP or TCP server in the C# part and a client on the C++ part, I can then send over the image data as a datagram. I was wondering if this is the best way and if it is whether UDP or TCP would be better?
EDIT: The C++ process is unmanaged C++, I don't have the option to run it as a managed DLL. Could I use named pipes to send over the image?
Finally is UDP guaranteed in order if it is communicating locally? I realise the image would be over the limit for UDP but if it is inorder I should be able to split the images up to send over.
Interprocess communication can be done via sockets or pipes.
With sockets(TCP and UDP) you're essentially sending the data over the internet to yourself. Luckily since your comp knows itself, the data shouldn't leave the comp so this should be pretty quick. TCP is guaranteed to be in order and has a bunch of other nice features while UDP is pretty much slap some headers onto the data and hope for the best. For this application TCP should be fine. UDP adds unneeded complexity.
Pipes are the other way to have two processes to communicate. You basically have the C++ or C# process create a pipe and start the other process. You just use the pipe like a file: write to and read from it. This can be done in C/C++ using a combination of the pipe, fork, and exec functions or simply using the popen function. C# probably has similar functions.
I suggest using a pipe using _popen, (popen for windows) and writing a series of ints to the pipe and reading it from the other side. This is probably the easiest way... besides using one language of course...
If you are writing both of the programs, you can compile C++ one as DLL, and call function that returns an array or some structure from your C# program with DllImport Attribute in System.Runtime.InteropServices namespace.
Why can't you do it in the same process? Is it because you need to mix C# and C++? In that case C++/CLI can be used as a bridge between the environments to have both C# code for the .NET CLR and C++ code compiled natively in one process.
If you really need two processes there are several options when running on a local machine, but a small TCP-based service is probably best. The size of each image will be 307kb which is larger than the 65kb limit of UDP.
I was wondering if this is the best way and if it is whether UDP or TCP would be better?
You usually resort to UDP as a speed optimization when TCP is not fast enough and packet loss is inconvenient rather than when it can't be handled. If you can't handle losing part of the image in the transmission I doubt you can resort to UDP.
Moreover, UDP is unlikely to give a performance boost in your case since you'll be using the loopback interface. This means that all TCP packets are likely to arrive in order and without loss, making TCP extra cheap.
If you write your application using TCP and in the future, for some reason, you decide the processes no longer run on the same machine, you won't have to change your code.
Finally, TCP sockets are just easier to use, so unless TCP is not fast enough on your machine, I would stick with TCP sockets.
is UDP guaranteed in order if it is communicating locally?
AFAIK, this behavior is not guaranteed. It is very likely to work most of the time, but unless you can find a quote from relevant documentation, I wouldn't count on this.
Could I use named pipes to send over the image?
Yes, named pipes are very similar to sockets, but they're known to be slow.
Once way of doing it apart from sockets would be to save the image data onto the disk from your C++ application and read it off the disk in your C# application. Of course you will need to make sure some sort of read/write synchronisation so that the file is not read before its fully written.
Or you finally decide to use UDP or TCP, try using RTP. RTP uses UDP with an extra layer of time stamps, sequence numbering to ensure correct order of data delivery. You should be able to find C++ and C# implementations of the protocol. Specifically to mention is that you can send images over a RTP/MJPEG stream if your application is producing JPEG images.
Just move to completely managed code :p (To keep it all in the same process)
https://net7mma.codeplex.com/SourceControl/latest has a C# RtspServer and RtpClient
I am attempting to send player information from my Game to my network client to then be sent off to the server.
Currently the ClientNetwork -> ClientGame relationship is held with XML files. They read/write back and forth at very high speeds. If you use just one XML file for this trade, one will "hog" the file at times, making a kind of lag when one cannot read because the other is viciously writing and rewriting.
To fix this I have 2 of each of my XML files. If it cannot read one, it will read the other. In theory they should be using both of them, since it'd be a tradeoff from one to another. Not working up to par.
But my main problem is just the usage in general of XML is very sloppy, dozens of try-catch statements to make sure they're all happy (and my personal favorite, try catches within try catches -- WE HAVE TO GO DEEPER)
I am just curious of if there is a better way to be doing this. I need a static point of variables that can be accessed by both client side programs. I'm afraid someone is going to say databases...
I'd like to state for anyone who is looking into this as well and stumbled across this page that Shared Memory is awesome. Though I have to convert all strings to characters and then to bytes and read them one by one, in the whole it's ALOT better than dealing with things that cannot read/write the same file at the same time. If you wish to further understand it rather than just use it, go to this link, it explains a lot of the messaging varieties and how to use them.
Yes there is!
The term you are looking for is interprocess communication - communication between two processes on the same machine.
There are various methods which allow two processes on the same machine to communicate with each other, including:
Named pipes
Shared memory
Sockets
HTTP
Fortunately C# applications can simply use the WCF framework to perform IPC (interprocess communication) using one of the above, and let the WCF framework take care of the difficult bits! Here are a couple of guides to get you started (there are many more):
WCF Tutorial - Basic Interprocess Communication
Many to One Local IPC using WCF and NetNamedPipeBinding
Also, one of the neat things about WCF is that you can also use it to communicate between different machines simply by changing the "Transport" (i.e. the communication method) to one which works over a network, (e.g. HTTP).
If you are targetting .Net 2.0 then you should look into either .Net remoting or web services instead.
A simple TCP stream jumps out at me. Have the network client open a listening TCP socket, and have the game connect to the network client. You could continue to send the same XML data you're already writing, if you like.
I agree with the tcp/ip socket answer proposed by David. I would simply submit the data to a socket on the local pc and have the other application listen to the socket. You can transmit data easily and quickly using this method and it will work no matter what version of the .net framework you are targeting.
I have a device connected to my computer that sends serial data to the computer every 5 mins. I want to write a basic program to capture this serial data every 5 mins and put it into a database. I was hoping to use C# because I have used C# with databases before and found it quite easy.
Can anybody offer me any advice on how I might do this, I really have no idea where to start and I know in theory it sounds easy but when I started it I actually found it really hard.
Using C#, you can use the System.IO.Ports namespace to communicate over the serial ports - there's a nice article here.
Alternatively, you can use Python and the pySerial module. I've written an app to communicate over the serial port using pySerial - it's quite easy to use, and can run on many different operating systems including OSX and Windows (I'm assuming you're using Windows). Python also has built-in support for SQLite.
The problem with capturing data on a serial port is that serial ports aren't thread-safe, so if there is more than one listener, data will be corrupted.
If you are absolutely sure that you're the only one listening for data on this port, .NET has a built-in wrapper, System.IO.Ports.SerialPort, which you can use to connect to COM1, COM2, etc. You'll need to know the rate in bits/sec at which this device sends data (its baud rate), its error-checking (parity) protocol, and the format of the data it is sending (you'll get it as a byte array, which you must convert byte-by-byte into data you can work with). Then, your program should be able to open the port and listen for DataReceived events with a handler that will read and digest the data. Again, it's VERY important that you never have two threads trying to read at once; the easiest way is to set a volatile boolean indicating that a handler is reading data; if another handler is ever spawned while a previous one is still running, the first thing the new one should do is read that value, and since it's set, exit the new handler immediately.
I'm working with an application, and I am able to make C# scripts to run in this environment. I can import DLLs of any kind into this environment. My problem is that I'd like to enable communication between these scripts. As the environment is controlled and I have no access to the source code of the application, I'm at a loss as to how to do this.
Things I've tried:
File I/O: Just writing the messages that I would like each to read in .txt files and having the other read it. Problem is that I need this scripts to run quite quickly and that took up too much time.
nServiceBus: I tried this, but I just couldn't get it to work in the environment that I'm dealing with. I'm not saying it can't be done, just that I can't get it done.
Does anyone know of a simple way to do this, that is also pretty fast?
Your method of interprocess communication should depend on how important it is that each message get processed.
For instance, if process A tells process B to, say, send an email to your IT staff saying that a server is down, it's pretty important.
If however you're streaming audio, individual messages (packets) aren't critical to the performance of the app, and can be dropped.
If the former, you should consider using persistent storage such as a database to store messages, and let each process poll the database to retrieve its own messages. In this way, if a process is terminated or loses communication with the other processes temporarily, it will be able to retrieve whatever messages it has missed when it starts up again.
The answer is simple;
Since you can import any DLL into the script you may create a custom DLL that will implement communication between the processes in any way you desire: shared memory, named pipe, TCP/UDP.
You could use a form of Interprocess Communication, even within the same process. Treat your scripts as separate processes, and communicate that way.
Named pipes could be a good option in this situation. They are very fast, and fairly easy to use in .NET 3.5.
Alternatively, if the scripts are loaded into a single AppDomain, you could use a static class or singleton as a communication service. However, if the scripts get loaded in isolation, this may not be possible.
Well, not knowing the details of your environment, there is not much I can really offer. You are using the term "C# scripts"...I am not exactly sure what that means, as C# is generally a compiled language.
If you are using normal C#, have you looked into WCF with Named Pipes? If your assemblies are running on the same physical machine, you should be able to easily and quickly create some WCF services hosted with the Named Pipe binding. Named pipes provide a simple, efficient, and quick message transfer mechanism in a local context. WCF itself is pretty easy to use, and is a native component of the .NET framework.
Since you already have the File I/O in place you might get enough speed by placing it on a RAM disk. If you are polling for changes today a FileSystemWatcher could help to get your communication more responsive.
You can use PipeStream. Which are fast than disk IO as they are done using main memory.
XMPP/Jabber is another appraoch take a look at jabber.net.
Another easy way is to open a TCP Socket on a predefined Port, connect to it from the other process and communicate that way.
I have an application in Lua, I want to communicate between the Lua application and a C# program.
I know of several ways to do this in C# (sockets, OS pipes etc) but I can't find any information about these things in Lua (which is a language I'm rather unfamiliar with unfortunately).
The IPC sending and receiving must not block for longer than a few milliseconds, since the Lua script is part of a game and the game will crash if the script takes too long to execute.
Any good links to libraries/tutorials would be greatly appreciated.
The only option I've seen for this would be to use LuaSocket, and communicate via TCP or UDP.