Develop Reference

Communication between Python and C#

I have a Python backend running machine learning algorithms. I want to use the same backend for both an Excel plugin (C#) and a website. I want both interfaces to send my training data (thousands of lines of numbers in arrays) to the same Python application and retrieve the results in the form of another array up to a few thousand lines.
The website would fetch data from a SQL database and send that data to Python, while the Excel plugin would take the data that is in the current worksheet and send that data to Python. I need to be able to create numpy arrays in Python before continuing to process the data. Note that the website would be running on the same machine where the Python application resides. I still haven't decided what I will use to code the website, but I was leaning towards Node.js.
I have done some research and found a few options:
1- Named pipes
2- Sockets
3- RPC server such as gRPC or XML-RPC.
4- Writing the data to a file and reading it back in Python
5- Web Service
Note: I would need the Python "server" to be stateful and keep the session running between calls. So I would need to have a kind of daemon running, waiting for calls.
Which one would you experts recommend and why? I need flexibility to handle several parameters and also large arrays of numbers. Using IronPython is not an option because I am running Keras on Python, which apparently does not support IronPython.

I had the same problem recently.
I used a named pipe to transport data from python to my c# server, hope it helps you.
Python:
import win32pipe, win32file
class PipeServer():
def __init__(self, pipeName):
self.pipe = win32pipe.CreateNamedPipe(
r'\\.\pipe\\'+pipeName,
win32pipe.PIPE_ACCESS_OUTBOUND,
win32pipe.PIPE_TYPE_MESSAGE | win32pipe.PIPE_READMODE_MESSAGE | win32pipe.PIPE_WAIT,
1, 65536, 65536,
0,
None)
#Carefull, this blocks until a connection is established
def connect(self):
win32pipe.ConnectNamedPipe(self.pipe, None)
#Message without tailing '\n'
def write(self, message):
win32file.WriteFile(self.pipe, message.encode()+b'\n')
def close(self):
win32file.CloseHandle(self.pipe)
t = PipeServer("CSServer")
t.connect()
t.write("Hello from Python :)")
t.write("Closing now...")
t.close()
For this code to work you need to install pywin32 (best choice is from binarys): https://github.com/mhammond/pywin32
C#-Server:
using System;
using System.IO;
using System.IO.Pipes;
class PipeClient
{
static void Main(string[] args)
{
using (NamedPipeClientStream pipeClient =
new NamedPipeClientStream(".", "CSServer", PipeDirection.In))
{
// Connect to the pipe or wait until the pipe is available.
Console.Write("Attempting to connect to pipe...");
pipeClient.Connect();
Console.WriteLine("Connected to pipe.");
Console.WriteLine("There are currently {0} pipe server instances open.",
pipeClient.NumberOfServerInstances);
using (StreamReader sr = new StreamReader(pipeClient))
{
// Display the read text to the console
string temp;
while ((temp = sr.ReadLine()) != null)
{
Console.WriteLine("Received from server: {0}", temp);
}
}
}
Console.Write("Press Enter to continue...");
Console.ReadLine();
}
}

You can use Python for .NET (Python.NET). It may require some changes to your code, but then it should work very well, once everything is in good shape.
Python.NET allows two-way communication between CPython and CLR.

Let me give you a neat and quick recipe, in the form of example code.
There are basically two ways to tie python in the backend of C# (or a C# winform app or gui or something similar).
Method1: Iron Python. In this method you install a .net package in your visual studio called IronPython. I would not prefer this, because assuming your machine learning model uses keras or a lot of other libraries. It would be another quest to get you installations ready and working in IronPython. And most importantly, it is not as good as your common virtual env or conda environment.
Method2: (The Good Method): Create a Custom Process in your C# that takes arguments from your GUI, knows the path to your script and your python env. Using all these things, it calls your python code exactly the way you would call it in your terminal and pass arguments to it.
Now the tasty example code (I have used this simple trick and it always helps make my black screen python stuff look good with the cover of C# apps).
Python Part
import sys
a = sys.argv[1]
b = sys.argv[2]
print("The Sum = ", float(a)+float(b))
The C# Part
So here is the python process/function that you need to call on the click event of your sum button in the application
static void PythonProcess()
{
//1) Create Process Info
var psi = new ProcessStartInfo();
//Conda Env Path
psi.FileName = #"C:\Users\jd\.conda\pkgs\py\python.exe";
//2) Provide Script and the Arguments
var script = #"C:\Users\jd\Desktop\script.py";
var a = "15";
var b = "18";
psi.Arguments = $"\"{script}\" \"{a}\" \"{b}\"";
//3) Process Configuration
psi.UseShellExecute = false;
psi.CreateNoWindow = true;
psi.RedirectStandardOutput = true;
psi.RedirectStandardError = true;
//4) Execute Process and get Output.
var errors = "";
var results = "";
using(var process = Process.Start(psi))
{
errors = process.StandardError.ReadToEnd();
results = process.StandardOutput.ReadToEnd();
}
//5) Display Output
Console.WriteLine("ERRORS: ");
Console.WriteLine(errors);
Console.WriteLine();
Console.WriteLine("RESULTS: ");
Console.WriteLine(results);
}

Calling Python from C# is easily possible via Pyrolite where your Python code is running as a Pyro4 server. It should be fast enough to handle "large arrays of numbers" however you didn't specify any performance constraints.

I had the same issue and seem to end up with named pipes. Here is a nice example of how to set it up to talk C# => Python, assuming C# is the server.
It can use the same way to talk back or just Python.net to call directly through CLR as shown here. I use the latter.

How to run command line code from within C# Windows Form?

I am fairly new to coding, but have built a few small things. One thing I figured out on my last project was how to run 2 simple commands normally run from a console, but from within a form application instead. Simply, the form had 2 buttons and clicking one caused ipconfig to run and the other ipconfig /all. It then posted the ip information coming from the command into another form I created as a message box. That is important because I am trying to do something similar and nothing is working now.
I have a form that has a spot for user name and a spot for password. On submit, I want it to essentially run the following:
NET USE F: \\ALPHA\CLIENTAPPS /user:domain\%username% %password% /persistent:no
NET USE O: \\ALPHA\USERS /user:domain\%username% %password% /persistent:no
NET USE S: \\ALPHA\COMPANY /user:domain\%username% %password% /persistent:no
Where %username% and %password% are captured from the form and domain will be our actual domain.
Using similar methods to the aforementioned ipconfig program that is working, this is what I came up with. However, when I click the Submit button, nothing happens, no errors, nor does it actually create the network share:
private void btnSubmit_Click(object sender, EventArgs e)
{
string un = txtUsername.Text;
string pw = txtPassword.Text;
System.Diagnostics.ProcessStartInfo PR = new System.Diagnostics.ProcessStartInfo("cmd", #" /c net use W: \\\\ALPHA\\CLIENTAPPS /user:acsdish\\" + un + " " + pw + "/persistent:no");
PR.RedirectStandardOutput = true;
PR.UseShellExecute = false;
PR.CreateNoWindow = true;
System.Diagnostics.Process StartPR = new System.Diagnostics.Process();
StartPR.StartInfo = PR;
StartPR.Start();
}
What am I missing here, or is there a better way? Thanks.
Mike

System.Diagnostics.ProcessStartInfo PR = new System.Diagnostics.ProcessStartInfo("cmd", #" /c net use W: \\\\ALPHA\\CLIENTAPPS /user:acsdish\\" + un + " " + pw + "/persistent:no");
Try to remove "#" or remove escaping of "\" char
Info here (Verbatim string literals)

nothing happens, no errors, nor does it actually create the network share
You've done a lot to ensure that. "No errors" is easy to explain, you don't check for errors nor do you give a way for the user to see them because you made sure that the console window isn't visible. If the command failed that it won't be visible. Checking Process.ExitCode is a minimal requirement.
Next flaw is that you create the mapping to the share for a particular user. Which is fine, drive mappings are a per-user setting. But you are not actually logged-in as that user so you can't see those mappings. You'll have to hit Ctrl+Alt+Del and switch the user account. But that's a lost cause because you passed /persistent:no. That means "persistent while the user is logged in".
Ultimate flaw is that you leave it up to an another process to take care of it. That always loses critical information, especially errors. You should pinvoke the Windows api function that does this so you know when it doesn't work and don't burn a gazillion cycles to run another process. Pinvoke WNetAddConnection2().

Changing environment variables of the calling process

This one seems trivial but the answer has eluded me for a few days now.
I have a Windows batch file, that calls a C# program to do an extra verification that cannot be done in a batch file. After the verification is complete I need to return a status and a string back to the calling shell.
Now the return value is trivial and my C# console app simply sets a return value (exit code if you will). And I thought the string will also be a piece of cake. I attempted to define a new shell variable using the:
Environment.SetEnvironmentVariable("ERR", "Some text");
This call should (and does) define a shell variable within the current process - that is the very C# process that created the variable. The value is lost as soon as the C# app terminates and the shell that created the C# app knows nothing about the variable. So... A call with no particular use... At all... Unless perhaps if I created a child process from the C3 app, perhaps it would inherit my variables.
The EnvironmentVariableTarget.Machine and EnvironmentVariableTarget.User targets for the SetEnvironmentVariable call don't solve the problem either, as only a newly created process will get these new values from the registry.
So the only working solution I can think of is:
write to stdout
write to a file
encode extra meaning into the return value
The first two are a bit ugly and the last one has its limitations and problems.
Any other ideas (how to set a shell variable in the parent process)? Maybe such shell variable modifications are a security concern (think PATH)...
Thank-you for your time.

I had the same problem as Ryan and the only thing that came to my mind as a work-around was to write a batch in error out to set the variable and to call it from the batch.
ConsoleApplication1.exe:
'put some sensible code here
'put result in variable myResult
Dim myResult As String = Guid.NewGuid().ToString("D").ToUpperInvariant()
Console.WriteLine("Normal output from the consonle app")
Console.Error.WriteLine("#ECHO OFF")
Console.Error.WriteLine("SET zzzResult={0}", myResult)
Test.cmd (the calling batch):
#ECHO OFF
:Jump to folder of batch file
PUSHD %~d0%~p0
:Define a temp file
SET zzzTempFile=%TEMP%\TMP%Random%.CMD
:Call .NET console app
ConsoleApplication1.exe 2>%zzzTempFile%
:Call the generated batch file
CALL %zzzTempFile%
:Clean up temp file
DEL %zzzTempFile%
:Clean up variable
SET zzzTempFile=
:Do something with the result
ECHO Yeah, we finally got it!
ECHO:
ECHO The value is "%zzzResult%".
ECHO:
:Clean up result variable
SET zzzResult=
:Go back to original folder
POPD
That should do the trick. And yes, I do know this is an old post and Ryan is solving other issues by now, but there might be still somebody else out there having the same problem...

What you are asking is to be able to arbitrarily write to the memory space of a running process. For good reason, this is not possible without SeDebugPrivilege.
Any of the three solutions you list will work. Stdout is the standard way to communicate with a batch script.
By the way, you're writing a Windows batch file. I'm pretty sure the ship has already sailed on "a bit ugly".

If you want to put a value of some output into a variable in the batch you can use the following construct:
FOR /F "usebackq tokens=4 delims=\[\] " %i IN (`ver`) DO set VERSION=%i
ECHO %VERSION%
Output on my OS:
6.1.7601
'usebackq' means we are using back quotes which gives the ability to use a fileset in the command quoted with double quotes. You may not need this. 'tokens' means the index in the resulting string array to select (it can be a range M-N). If you need to skip lines use 'skip=X'). 'delims' are the string separators to use (like string-Split() in .Net).
You will put your console app instead of 'ver' and adapt the delimiters and tokens to match your specific output. If you have more variables to fill you will need to make the if a bit more complex but that should make a good start.

My BAT is a bit rusty, but I think it's possible to retrieve the 'exit' code from processes you've run externally, perhaps via %ERRORLEVEL%. If that's the case, make sure to exit your program via
Environment.Exit(123); // where 123 = error code
You can't add any messages, so you'll have to do that in the .bat file.
If this isn't the case, stdout is probably the best way.

After stumbling on this myself as well recently, I came up with this approach. What I did is run the bat file using the Process class, i.e.
// Spawn your process as you normally would... but also have it dump the environment varaibles
Process process = new Process();
process.StartInfo.FileName = mybatfile.bat;
process.StartInfo.Arguments = #"&&set>>envirodump.txt";
process.StartInfo.UseShellExecute = false;
process.StartInfo.RedirectStandardOutput = true;
process.StartInfo.RedirectStandardError = false;
process.Start();
string output = process.StandardOutput.ReadToEnd();
process.WaitForExit();
// Read the environment variable lines into a string array
string[] envirolines = File.ReadAllLines("envirodump.txt");
File.Delete("envirodump.txt");
// Now simply set the environment variables in the parent process
foreach(string line in a)
{
string var = line.Split('=')[0];
string val = line.Split('=')[1];
Environment.SetEnvironmentVariable(var, val);
}
This seems to have worked for me. It's not the cleanest approach, but will work in a bind. :)

C# can I add the command prompt control to my app

Title says it. I know I can use Process or ProcessStartInfo to run arguments, but I mean actually adding a command prompt control to my app (because I use it very often and it'd be convenient if it was already built-in.
Is there any way to do this other than coding a custom control? If not I can live with it, but it would definitely help.

Something like this (not tested):
ProccessInfo pi = new ProccessInfo("cmd.exe");
pi.RedirectStandardError=true;
pi.RedirectStandardInput=true;
pi.RedirectStandardOutput=true;
Process cmd = Process.Start(pi);
cmd.StandardInput.WriteLine("Dir");
textBox1.Text = cmd.StandardOutput.ReadToEnd();
Watch out for deadlocks, those method can be blocking!
You can also use this solution from codeproject.com: http://www.codeproject.com/KB/miscctrl/commandprompt.aspx

See this question and related. Also this source code for example.
You can start a console near your win app's window which you will control and can output some data or get input from a user.

What is the problem with doing Console.ReadLine()/Console.WriteLine() in a loop? It is the most efficient way and you have fully control over what you are doing.

I need to open several VLC clients in order to "monitor" the stream they're receiving

I want to be able to specify how many clients do I want opened, and be able to manually switch between the windows after they're opened- meaning "streaming in background" (if such a thing is possible? ) won't do here.
I need to specify different inputs for the different clients as well.
Additionally -and this is the part I'm totally clueless about as it's VLC-specific - I need the clients to be logging some info re:the stream they're receiving, so as to be able to determine that it has been received completely etc -such as frame rate/total frames' number or similar.
I'd appreciate helpful suggestions for
running the instances+ controlling
them
getting info about
the stream
Language-wise - I know Java, some C#, and wouldn't mind learning some new language for this purpose if it's a better solution .
Thanks!

Depending on your version of VLC, you may need to enable an option to run multiple instances. See here: http://wiki.videolan.org/How_to_play_multiple_instances_of_VLC
It does sound like a 'run windows processes in a loop' thing, which you could do several ways.
You could make a windows batch file (.bat):
"C:\path\to\vlc.exe" -vvv "http://www.whatever.com/mystream.mms"
"C:\path\to\vlc.exe" -vvv "http://www.whatever.com/mystream2.mms"
"C:\path\to\vlc.exe" -vvv "C:\music\whatever.mp3"
Or you could use a real programming language and perhaps open a variable number of instances... C# for example:
using System.Diagnostics;
...
foreach (string stream in streamList) {
Process myProc = new Process();
string myCmd = #"C:\path\to\vlc.exe";
string myArgs = "-vvv \"" + stream + "\"";
ProcessStartInfo myStart = new ProcessStartInfo(myCmd, myArgs);
myStart.UseShellExecute = false;
myProc.StartInfo = myStart;
myProc.Start();
}
See this page for a full list of VLC command line options: http://www.videolan.org/doc/vlc-user-guide/en/ch04.html
Hope this helps.

You'll either need to run several processes (as above) or hook somehow into libvlc and instruct it to start up several players.
A good demo of this is the python wrapper to libvlc--I think--it shows how to sample to know where the stream is--however I've never tried it with multiple things running at the same time but I think it would work.
Another option might be something like http://wiki.videolan.org/Mosaic