Develop Reference

Communication between Python and C#

I have a Python backend running machine learning algorithms. I want to use the same backend for both an Excel plugin (C#) and a website. I want both interfaces to send my training data (thousands of lines of numbers in arrays) to the same Python application and retrieve the results in the form of another array up to a few thousand lines.
The website would fetch data from a SQL database and send that data to Python, while the Excel plugin would take the data that is in the current worksheet and send that data to Python. I need to be able to create numpy arrays in Python before continuing to process the data. Note that the website would be running on the same machine where the Python application resides. I still haven't decided what I will use to code the website, but I was leaning towards Node.js.
I have done some research and found a few options:
1- Named pipes
2- Sockets
3- RPC server such as gRPC or XML-RPC.
4- Writing the data to a file and reading it back in Python
5- Web Service
Note: I would need the Python "server" to be stateful and keep the session running between calls. So I would need to have a kind of daemon running, waiting for calls.
Which one would you experts recommend and why? I need flexibility to handle several parameters and also large arrays of numbers. Using IronPython is not an option because I am running Keras on Python, which apparently does not support IronPython.

I had the same problem recently.
I used a named pipe to transport data from python to my c# server, hope it helps you.
Python:
import win32pipe, win32file
class PipeServer():
def __init__(self, pipeName):
self.pipe = win32pipe.CreateNamedPipe(
r'\\.\pipe\\'+pipeName,
win32pipe.PIPE_ACCESS_OUTBOUND,
win32pipe.PIPE_TYPE_MESSAGE | win32pipe.PIPE_READMODE_MESSAGE | win32pipe.PIPE_WAIT,
1, 65536, 65536,
0,
None)
#Carefull, this blocks until a connection is established
def connect(self):
win32pipe.ConnectNamedPipe(self.pipe, None)
#Message without tailing '\n'
def write(self, message):
win32file.WriteFile(self.pipe, message.encode()+b'\n')
def close(self):
win32file.CloseHandle(self.pipe)
t = PipeServer("CSServer")
t.connect()
t.write("Hello from Python :)")
t.write("Closing now...")
t.close()
For this code to work you need to install pywin32 (best choice is from binarys): https://github.com/mhammond/pywin32
C#-Server:
using System;
using System.IO;
using System.IO.Pipes;
class PipeClient
{
static void Main(string[] args)
{
using (NamedPipeClientStream pipeClient =
new NamedPipeClientStream(".", "CSServer", PipeDirection.In))
{
// Connect to the pipe or wait until the pipe is available.
Console.Write("Attempting to connect to pipe...");
pipeClient.Connect();
Console.WriteLine("Connected to pipe.");
Console.WriteLine("There are currently {0} pipe server instances open.",
pipeClient.NumberOfServerInstances);
using (StreamReader sr = new StreamReader(pipeClient))
{
// Display the read text to the console
string temp;
while ((temp = sr.ReadLine()) != null)
{
Console.WriteLine("Received from server: {0}", temp);
}
}
}
Console.Write("Press Enter to continue...");
Console.ReadLine();
}
}

You can use Python for .NET (Python.NET). It may require some changes to your code, but then it should work very well, once everything is in good shape.
Python.NET allows two-way communication between CPython and CLR.

Let me give you a neat and quick recipe, in the form of example code.
There are basically two ways to tie python in the backend of C# (or a C# winform app or gui or something similar).
Method1: Iron Python. In this method you install a .net package in your visual studio called IronPython. I would not prefer this, because assuming your machine learning model uses keras or a lot of other libraries. It would be another quest to get you installations ready and working in IronPython. And most importantly, it is not as good as your common virtual env or conda environment.
Method2: (The Good Method): Create a Custom Process in your C# that takes arguments from your GUI, knows the path to your script and your python env. Using all these things, it calls your python code exactly the way you would call it in your terminal and pass arguments to it.
Now the tasty example code (I have used this simple trick and it always helps make my black screen python stuff look good with the cover of C# apps).
Python Part
import sys
a = sys.argv[1]
b = sys.argv[2]
print("The Sum = ", float(a)+float(b))
The C# Part
So here is the python process/function that you need to call on the click event of your sum button in the application
static void PythonProcess()
{
//1) Create Process Info
var psi = new ProcessStartInfo();
//Conda Env Path
psi.FileName = #"C:\Users\jd\.conda\pkgs\py\python.exe";
//2) Provide Script and the Arguments
var script = #"C:\Users\jd\Desktop\script.py";
var a = "15";
var b = "18";
psi.Arguments = $"\"{script}\" \"{a}\" \"{b}\"";
//3) Process Configuration
psi.UseShellExecute = false;
psi.CreateNoWindow = true;
psi.RedirectStandardOutput = true;
psi.RedirectStandardError = true;
//4) Execute Process and get Output.
var errors = "";
var results = "";
using(var process = Process.Start(psi))
{
errors = process.StandardError.ReadToEnd();
results = process.StandardOutput.ReadToEnd();
}
//5) Display Output
Console.WriteLine("ERRORS: ");
Console.WriteLine(errors);
Console.WriteLine();
Console.WriteLine("RESULTS: ");
Console.WriteLine(results);
}

Calling Python from C# is easily possible via Pyrolite where your Python code is running as a Pyro4 server. It should be fast enough to handle "large arrays of numbers" however you didn't specify any performance constraints.

I had the same issue and seem to end up with named pipes. Here is a nice example of how to set it up to talk C# => Python, assuming C# is the server.
It can use the same way to talk back or just Python.net to call directly through CLR as shown here. I use the latter.

Test file permissions on Mono

I'm working on crossplatform app with Mono. I want to check permissions for RW-access to user that runs application. On NT I can use .GetAccessControl methods (on Mono throws PlatformNotSupported exception), but what to do with *nix and MacOS? Is there any crossplatfrom solutions?

Mono.Unix has UNIX specific implementations:
var ufi = new UnixFileInfo("/tmp/test.cpp");
ufi.CanAccess(AccessModes.F_OK); // is a file/directory
ufi.CanAccess(AccessModes.R_OK); // accessible for reading
ufi.CanAccess(AccessModes.W_OK); // accessible for writing
ufi.CanAccess(AccessModes.X_OK); // accessible for executing
FileSpecialAttributes sa = ufi.FileSpecialAttributes; //setuid, setgid and sticky bits
FileAccessPermissions fa = ufi.FileAccessPermissions;
FileAccessPermissions is defined as:
[Flags ()]
public enum FileAccessPermissions {
UserReadWriteExecute,
UserRead,
UserWrite,
UserExecute,
GroupReadWriteExecute,
GroupRead,
GroupWrite,
GroupExecute,
OtherReadWriteExecute,
OtherRead,
OtherWrite,
OtherExecute,
DefaultPermissions,
AllPermissions
}

How about just trying to open the file for RW and seeing if it succeeds?

File.GetAttributes(path)
http://msdn.microsoft.com/en-us/library/system.io.fileattributes.aspx
Not as much information as .GetAccessControl, but you can determine if the file is ReadOnly

I need to open several VLC clients in order to "monitor" the stream they're receiving

I want to be able to specify how many clients do I want opened, and be able to manually switch between the windows after they're opened- meaning "streaming in background" (if such a thing is possible? ) won't do here.
I need to specify different inputs for the different clients as well.
Additionally -and this is the part I'm totally clueless about as it's VLC-specific - I need the clients to be logging some info re:the stream they're receiving, so as to be able to determine that it has been received completely etc -such as frame rate/total frames' number or similar.
I'd appreciate helpful suggestions for
running the instances+ controlling
them
getting info about
the stream
Language-wise - I know Java, some C#, and wouldn't mind learning some new language for this purpose if it's a better solution .
Thanks!

Depending on your version of VLC, you may need to enable an option to run multiple instances. See here: http://wiki.videolan.org/How_to_play_multiple_instances_of_VLC
It does sound like a 'run windows processes in a loop' thing, which you could do several ways.
You could make a windows batch file (.bat):
"C:\path\to\vlc.exe" -vvv "http://www.whatever.com/mystream.mms"
"C:\path\to\vlc.exe" -vvv "http://www.whatever.com/mystream2.mms"
"C:\path\to\vlc.exe" -vvv "C:\music\whatever.mp3"
Or you could use a real programming language and perhaps open a variable number of instances... C# for example:
using System.Diagnostics;
...
foreach (string stream in streamList) {
Process myProc = new Process();
string myCmd = #"C:\path\to\vlc.exe";
string myArgs = "-vvv \"" + stream + "\"";
ProcessStartInfo myStart = new ProcessStartInfo(myCmd, myArgs);
myStart.UseShellExecute = false;
myProc.StartInfo = myStart;
myProc.Start();
}
See this page for a full list of VLC command line options: http://www.videolan.org/doc/vlc-user-guide/en/ch04.html
Hope this helps.

You'll either need to run several processes (as above) or hook somehow into libvlc and instruct it to start up several players.
A good demo of this is the python wrapper to libvlc--I think--it shows how to sample to know where the stream is--however I've never tried it with multiple things running at the same time but I think it would work.
Another option might be something like http://wiki.videolan.org/Mosaic

WMEncoder Throws COMException (0x80004005) Unspecified Error

So I'm running this code
public static void ConvertToWma(string inFile, string outFile, string profileName)
{
// Create a WMEncoder object.
WMEncoder encoder = new WMEncoder();
ManualResetEvent stopped = new ManualResetEvent(false);
encoder.OnStateChange += delegate(WMENC_ENCODER_STATE enumState)
{
if (enumState == WMENC_ENCODER_STATE.WMENC_ENCODER_STOPPED)
stopped.Set();
};
// Retrieve the source group collection.
IWMEncSourceGroupCollection srcGrpColl = encoder.SourceGroupCollection;
// Add a source group to the collection.
IWMEncSourceGroup srcGrp = srcGrpColl.Add("SG_1");
// Add an audio source to the source group.
IWMEncSource srcAud = srcGrp.AddSource(WMENC_SOURCE_TYPE.WMENC_AUDIO);
srcAud.SetInput(inFile, "", "");
// Specify a file object in which to save encoded content.
IWMEncFile file = encoder.File;
file.LocalFileName = outFile;
// Choose a profile from the collection.
IWMEncProfileCollection proColl = encoder.ProfileCollection;
proColl.ProfileDirectory = AssemblyInformation.GetExecutingAssemblyDirectory();
proColl.Refresh();
IWMEncProfile pro;
for (int i = 0; i < proColl.Count; i++)
{
pro = proColl.Item(i);
if (pro.Name == profileName)
{
srcGrp.set_Profile(pro);
break;
}
}
// Start the encoding process.
// Wait until the encoding process stops before exiting the application.
encoder.SynchronizeOperation = false;
encoder.PrepareToEncode(true);
encoder.Start();
stopped.WaitOne();
}
And I get a COMException (0x80004005) when encoder.PrepareToEncode gets executed.
Some notes:
1) The process is spawned by an ASP.NET web service so it runs as NETWORK SERVICE
2) inFile and outFile are absolute local paths and their extensions are correct, in addition inFile definitely exists (this has been a source of problems in the past)
3) The program works when I run it as myself but doesn't work in the ASP.NET context.
This says to me its a security permission issue so in addition I've granted Full Control to the directory containing the program AND the directories containing the audio files to NETWORK SERVICE. So I really don't have any idea what more I can do on the security front. Any help?

Running WM Encoder SDK based app in windows service is not supported. It uses hidden windows for various reasons, and there isn't a desktop window in service. DRM would certainly fail with no user profile. Besides, even when you make your service talk to WME instance on a user's desktop, Microsoft only supports 4 concurrent requests per machine because the global lock in WME (I know, not pretty programming, but WME is old). For more scalable solutions, consider Windows Media Format SDK.
You may want to move your WM Encoder based app to Expression Encoder SDK as WM Encoder's support is ending.

What is the Fastest way to read event log on remote machine?

I am working on an application which reads eventlogs(Application) from remote machines. I am making use of EventLog class in .net and then iterating on the Log entries but this is very slow. In some cases, some machines have 40000+ log entries and it takes hours to iterate through the entries.
what is the best way to accomplish this task? Are there any other classes in .net which are faster or in any other technology?

Man, I feel your pain. We had the exact same issue in our app.
Your solution has a branch depending on what server version you're running on and what server version your "target" machine is running on.
If you're both on Vista or Windows Server 2008, you're in luck. You should look at System.Diagnostics.Eventing.Reader.EventLogQuery and System.Diagnostics.Eventing.Reader.EventLogReader. These are new in .net 3.5.
Basically, you can build a query in XML and ship it over to run on the remote computer. Maybe you're just searching for events of a specific type, or maybe just new events from a specific point in time. The search runs on the remote machine, and then you just get back the matching events. The new classes are much faster than the old .net 2.0 way, but again, they are only supported on Vista or Windows Server 2008.
For our app when the target is NOT on Vista/Win2008, we downloaded the raw .evt file from the remote system, and then parsed the file using its binary format. There are several sources of data about the event log format for .evt files (pre-Vista), including link text and an article I recall on codeproject.com that had some c# code.
Vista and Windows Server 2008 machines use a new .evtx format that is a new format, so you can't use the same binary parsing approach across all versions. But the new EventLogQuery and EventLogReader classes are so fast that you won't have to. It's now perfectly speedy to just use the built-in classes.

Event Log Reader is horribly slow... too slow. WTF Microsoft?
Use LogParser 2.2 - Search for C# and LogParser on the Internet (or you can use the log parser commands from the command line). I don't want to duplicate the work already contributed by others.
I pull the log from the remote system by having the log exported as an EVTX file. I then copy the file from the remote system. This process is really quick - even with a network that spans the planet (I had issues with having the log exported to a network resource). Once you have it local, you can do your searches and processing.
There are multiple reasons for having the EVTX - I won't get into the reasons why we do this.
The following is a working example of the code to save a copy of the log as an EVTX:
(Notes: "device" is the network host name or IP. "LogName" is the name of the log desired: "System", "Security", or "Application". outputPathOnRemoteSystem is the path on the remote computer, such as "c:\temp\%hostname%.%LogName%.%YYYYMMDD_HH.MM%.evtx".)
static public bool DumpLog(string device, string LogName, string outputPathOnRemoteSystem, out string errMessage)
{
bool wasExported = false;
string errorMessage = "";
try
{
System.Diagnostics.Eventing.Reader.EventLogSession els = new System.Diagnostics.Eventing.Reader.EventLogSession(device);
els.ExportLogAndMessages(LogName, PathType.LogName, "*", outputPathOnRemoteSystem);
wasExported = true;
}
catch (UnauthorizedAccessException e)
{
errorMessage = "Unauthorized - Access Denied: " + e.Message;
}
catch (EventLogNotFoundException e)
{
errorMessage = "Event Log Not Found: " + e.Message;
}
catch (EventLogException e)
{
errorMessage = "Export Failed: " + e.Message + ", Log: " + LogName + ", Device: " + device;
}
errMessage = errorMessage;
return wasExported;
}

A good Explanation/Example can be found on MSDN.
EventLogSession session = new EventLogSession(Environment.MachineName);
// [System/Level=2] filters out the errors
// Where "Log" is the log you want to get data from.
EventLogQuery query = new EventLogQuery("Log", PathType.LogName, "*[System/Level=2]");
EventLogReader reader = new EventLogReader(query);
for (EventRecord eventInstance = reader.ReadEvent();
null != eventInstance;
eventInstance = reader.ReadEvent())
{
// Output or save your event data here.
}
When waiting 5-20 minutes with the old code this one does it in less than 10 seconds.

Maybe WMI can help you:
WMI with C#

Have you tried using the remoting features in powershell 2.0? They allow you to execute cmdlets (like ones to read event logs) on remote machines and return the results (as objects, of course) to the calling session.

You could place a Program at those machines that save the log to file and sends it to your webapplication i think that would be alot faster as you can do the looping local but im not sure how to do it so i cant ive you any code :(

I recently did such thing via WCF callback interface however my clients interacted with the server through WCF and adding a WCF Callback was easy in my project, full code with examples is available here

Just had the same issue and want to share my solution. It makes a search through application, system and security eventlogs from 260 seconds (using EventLog) about a 100 times faster (using EventLogQuery).
And this in a way where it is possible to check if the event message contains a pattern or any other check without the requirement of FormatDescription().
My trick is to use the same mechanism as PowerShells Get-WinEvent does and then pass it through the result check.
Here is my code to find all events within last 4 days where the event message contains a filter pattern.
string[] eventLogSources = {"Application", "System", "Security"};
var messagePattern = "*Your Message Search Pattern*";
var timeStamp = DateTime.Now.AddDays(-4);
var matchingEvents = new List<EventRecord>();
foreach (var eventLogSource in eventLogSources)
{
var i = 0;
var query = string.Format("*[System[TimeCreated[#SystemTime >= '{0}']]]",
timeStamp.ToUniversalTime().ToString("o"));
var elq = new EventLogQuery(eventLogSource, PathType.LogName, query);
var elr = new EventLogReader(elq);
EventRecord entryEventRecord;
while ((entryEventRecord = elr.ReadEvent()) != null)
{
if ((entryEventRecord.Properties)
.FirstOrDefault(x => (x.Value.ToString()).Contains(messagePattern)) != null)
{
matchingEvents.Add(entryEventRecord);
i++;
}
}
}

Maybe that the remote computers could do a little bit of computing. So this way your server would only deal with relevant information. It would be a kind of cluster using the remote computer to do some light filtering and the server would the the analysis part.