Develop Reference

C# - Send input to child process - console app

Background info
I am writing an integration test that spawns a child process (c# console app). The test is counting some rows in the database after the process is spun up and after the process is closed. The process is closed via process.Kill()
When the process is killed in this manner, it doesn't hit the Stop method within the process. I need to call this stop method to stop threads and remove entries from the database in order for the test to pass.
Original Code
The console app process that I am spawning in my test:
static void Main(string[] args)
{
TaskManager tm = new TaskManagerProcess();
if (Environment.UserInteractive ||
(args.EmptyForNull().Any(a => a.Equals("-RunInteractive", StringComparison.OrdinalIgnoreCase) || a.Equals("/RunInteractive"))))
{
tm.ConsoleStart(args);
Console.WriteLine("Press [Enter] to shut down, any other key to mark");
while (true)
{
ConsoleKeyInfo key = Console.ReadKey(true);
if (key.Key == ConsoleKey.Enter)
break;
Console.WriteLine("========================================================");
Console.Out.Flush();
}
Console.WriteLine("Shutting down...");
tm.ConsoleStop();
}
else
{
ServiceBase.Run(tm);
}
}
}
The test code:
//count before starting child proc
int preCount;
//count after process is spun up
int runningsCount;
//count after stopped
int postCount;
//Get an initial count of the logged in modules before svc host is started
user = ApiMethod.GetLoggedInUsers().Where(x => x.RecId == userRecID).FirstOrDefault();
preCount = user.LoggedInModules.Count;
Process proc = Helper.StartProcess(ConnectionBundle);
//Give process time to spin up leaders and workers
await Task.Delay(TimeSpan.FromSeconds(30));
//Get a count of modules after process is spun up
user = ApiMethod.GetLoggedInUsers().Where(x => x.RecId == userRecID).FirstOrDefault();
runningCount = user.LoggedInModules.Count;
//Write a line terminator to the child svc host process -
//this allows it to shutdown normally
Helper.ProcessInput.WriteLine();
Helper.ProcessInput.Close();
Helper.KillProcess(proc);
await Task.Delay(TimeSpan.FromSeconds(5));
//Get count of logged in modules after process is closed
user = ApiMethod.GetLoggedInUsers().Where(x => x.RecId == userRecID).FirstOrDefault();
postCount = user.LoggedInModules.Count;
Helper is a static class that sets up the process start info(including args) and starts the process. In helper I've redirected the StandardInput and added a property ProcessInput which is set to the StandardInput of the created process.
My goal is to send input of "Enter" from the test to the spawned process so that it will break from the loop and call tm.ConsoleStop()
TaskManagerProcess is a private custom class that controls the process. It does not inherit from System.Diagnostics.Process. As an alternate approach, my test could interact with TaskManagerProcess directly. However, I can't make TaskManagerProcess public and I need to run TaskManagerProcess in its own AppDomain because calling ConsoleStop is disposing objects in the API that I need to finish the test.
Things I've Tried
[DllImport("Kernel32")]
private static extern bool SetConsoleCtrlHandler(CloseProcDelgate handler, bool add);
I tried adding a call to Kernel32.SetConsoleCtrlHandler (and the necessary delegate) to call ConsoleStop when the process is exited. This doesn't seem to work when the process is killed via process.Kill()
With the original process code, I noticed an exception when I wrote to the StandardInput. The exception message told me to use Console.Read instead of Console.ReadKey(). This actually works intermittently! I can sometimes get a breakpoint on int cKey = Console.Read() (with debugger attached to child process) but other times it doesn't hit the breakpoint.
while (true)
{
//Changing this to Console.Read instead of Console.ReadKey
//Allows us to send redirected input to process?
int cKey = Console.Read();
if ((ConsoleKey)cKey == ConsoleKey.Enter)
break;
Console.WriteLine("========================================================");
Console.Out.Flush();
}
Finally, I tried interacting with TaskManagerProcess directly. I made the private class internal, and marked the internals visible to my test assembly. I cannot make the class public.
When I go this route, calling tm.ConsoleStop() blows away some objects in my API so I can't check the count after this method is called. For this reason, I thought I would create a new AppDomain and call AppDomain.CreateInstanceAndUnwrap() on the TaskManagerProcess class. However, I get an exception here, I believe its due to the the fact that the class is internal.
I am really stuck at this point! Any help is appreciated and thanks for taking the time to read this!
Edit
I created a demo project here
that shows what I am trying to do and has both approaches in the Test method.
Initially I thought I couldn't call AppDomain.CreateInstanceAndUnwrap() because the TaskManagerProcess class was internal. However, after playing with my demo project, I think I just can't load the assembly.

I'm guessing here, but I believe your TaskManagerProcess is a service application. If it is not, please ignore this. If it is, be advised of including details like this in your question. Debugging service applications can be complicated, believe me, I've been there. But before proceed, more advise.
Test the methods in your modules, no whole running programs, as Michael Randall just said.
Unless absolutely necessary, don't do tests against a database. Mock whatever you need to test your code.
You should go back to your alternate approach of interact with TaskManagerProcess directly. From the code of your console app, the only working method I see called is tm.ConsoleStart(args), the rest inside the loop is console writing and reading. So you can't change the acces level of that class, again, I've been there. What I have done in the past to overcome this is to use conditional compilation to create a kind of public facade in my private or internal modules.
Suppose you have:
internal class TaskManagerContainer
{
private class TaskManagerProcess
{
internal void Start()
{
// stuff
}
private void DoSomething(int arg)
{
// more stuff
}
}
}
Change it like this:
#define TEST
// Symbol TEST can also be defined using the GUI of your IDE or compiler /define option
internal class TaskManagerContainer
{
//
#if TEST
public class TaskManagerProcess
#else
private class TaskManagerProcess
#endif
{
internal void Start()
{
// stuff
}
private void DoSomething(int arg)
{
// more stuff
}
#region Methods Facade for Testing
#if TEST
public void Start_Test()
{
Start();
}
private void DoSomething_Test(int arg)
{
DoSomething(arg);
}
#endif
#endregion
}
}
I really hope it will help you making the methods visible to the test assembly and it won't blow objects in you API.

I think I got it with a brute force approach.
while (!testProcess.HasExited)
{
testProcess.StandardInput.WriteLine();
}
Thanks everyone for the input!

How can I break into Python code called from Python.NET on a separate thread?

I would like to use PyCharm's debugger to stop at a breakpoint set in a Python method which is called from C# via Python.NET. However, the breakpoint is never hit, even though the code clearly gets executed.
The issue seems to be that the thread on which the Python method is called was created in C#. I am aware that for a thread to be visible to the debugger, settrace must be called. I thought, calling it once per thread should do the trick, but it does not. Because after calling settrace at the first execution of a Python method, my breakpoints in that method and any methods called by it, are hit. But after control passed from that method back to C#, another call of a Python method on the same thread does not hit the breakpoint.
Here is the Python side of things; imagine breakpoints set at the two print statements.
import sys
import thread
import threading
sys.path.append(r'C:\Program Files (x86)\JetBrains\PyCharm Community Edition 2018.2.4\helpers\pydev')
import pydevd
import clr
from System.Threading import Thread
from System import Action
sys.path.append(r'C:\Code\PythonNetLib\bin\Debug')
clr.AddReference('MyLib')
from MyLib import MyThread, MyThreadBase
def init_debugging():
pydevd.settrace(suspend=False)
# breakpoint here is hit
print 'debugging initialized in thread {0}, managed thread ID {1}'.format(thread.get_ident(), Thread.CurrentThread.ManagedThreadId)
def hello():
# pydevd.settrace(suspend=False)
# breakpoint here is not hit unless pydevd.settrace is called again
print 'Hello from thread {0}, managed thread ID {1}'.format(thread.get_ident(), Thread.CurrentThread.ManagedThreadId))
t = MyThread()
t.ExecuteAsync(Action(init_debugging))
t.ExecuteAsync(Action(hello))
t.ExecuteAsync(Action(t.Stop))
t.Join()
For illustration purposes here is a simplified version of the C# library referenced above containing the MyThread class:
using System;
using System.Collections.Concurrent;
using System.Threading;
namespace MyLib
{
public class MyThread: MyThreadBase
{
private Thread _thread;
public MyThread()
{
_thread = new Thread(Execute);
_thread.Start();
}
public void Join()
{
_thread.Join();
}
}
public class MyThreadBase
{
private ConcurrentQueue<Action> _queue;
private bool _stopped;
public MyThreadBase()
{
_queue = new ConcurrentQueue<Action>();
}
public void Execute()
{
while (!_stopped)
{
if (_queue.TryDequeue(out Action action))
{
action();
}
else
{
Thread.Sleep(TimeSpan.FromSeconds(1));
}
}
}
public void ExecuteAsync(Action action)
{
_queue.Enqueue(action);
}
public void Stop()
{
_stopped = true;
}
}
}
Workaround: I noticed that my breakpoints are hit as expected when I don't create the thread in C# but use a Python thread instead like so:
t = MyThreadBase()
tPy = threading.Thread(target=lambda: t.Execute())
tPy.start()
t.ExecuteAsync(Action(init_debugging))
t.ExecuteAsync(Action(hello))
t.ExecuteAsync(Action(t.Stop))
tPy.join()
This is an option in some cases, but I do not control all thread creations in the C# libraries I want to use.
Question: What would I have to do make C# threads known to the Python debugger permanently such that the debugger doesn't lose sight of the thread after control passes back to C#. Repeatedly calling settrace seems a bit much, particularly since I can't know definitively which methods might all get called from C#.
I am using Python 2.7 and .NET Framework 4.6.1 in case that makes a difference.

Threading A Method Contained In An Object - C#

I'm writing a program that will analyze changes in the stock market.
Every time the candles on the stock charts are updated, my algorithm scans every chart for certain pieces of data. I've noticed that this process is taking about 0.6 seconds each time, freezing my application. Its not getting stuck in a loop, and there are no other problems like exception errors slowing it down. It just takes a bit of time.
To solve this, I'm trying to see if I can thread the algorithm.
In order to call the algorithm to check over the charts, I have to call this:
checkCharts.RunAlgo();
As threads need an object, I'm trying to figure out how to run the RunAlgo(), but I'm not having any luck.
How can I have a thread run this method in my checkCharts object? Due to back propagating data, I can't start a new checkCharts object. I have to continue using that method from the existing object.
EDIT:
I tried this:
M4.ALProj.BotMain checkCharts = new ALProj.BotMain();
Thread algoThread = new Thread(checkCharts.RunAlgo);
It tells me that the checkCharts part of checkCharts.RunAlgo is gives me, "An object reference is required for the non-static field, method, or property "M4.ALProj.BotMain"."
In a specific if statement, I was going to put the algoThread.Start(); Any idea what I did wrong there?

The answer to your question is actually very simple:
Thread myThread = new Thread(checkCharts.RunAlgo);
myThread.Start();
However, the more complex part is to make sure that when the method RunAlgo accesses variables inside the checkCharts object, this happens in a thread-safe manner.
See Thread Synchronization for help on how to synchronize access to data from multiple threads.

I would rather use Task.Run than Thread. Task.Run utilizes the ThreadPool which has been optimized to handle various loads effectively. You will also get all the goodies of Task.
await Task.Run(()=> checkCharts.RunAlgo);

Try this code block. Its a basic boilerplate but you can build on and extend it quite easily.
//If M4.ALProj.BotMain needs to be recreated for each run then comment this line and uncomment the one in DoRunParallel()
private static M4.ALProj.BotMain checkCharts = new M4.ALProj.BotMain();
private static object SyncRoot = new object();
private static System.Threading.Thread algoThread = null;
private static bool ReRunOnComplete = false;
public static void RunParallel()
{
lock (SyncRoot)
{
if (algoThread == null)
{
System.Threading.ThreadStart TS = new System.Threading.ThreadStart(DoRunParallel);
algoThread = new System.Threading.Thread(TS);
}
else
{
//Recieved a recalc call while still calculating
ReRunOnComplete = true;
}
}
}
public static void DoRunParallel()
{
bool ReRun = false;
try
{
//If M4.ALProj.BotMain needs to be recreated for each run then uncomment this line and comment private static version above
//M4.ALProj.BotMain checkCharts = new M4.ALProj.BotMain();
checkCharts.RunAlgo();
}
finally
{
lock (SyncRoot)
{
algoThread = null;
ReRun = ReRunOnComplete;
ReRunOnComplete = false;
}
}
if (ReRun)
{
RunParallel();
}
}

Resource Access by Parallel Threads

I have 2 threads to are triggered at the same time and run in parallel. These 2 threads are going to be manipulating a string value, but I want to make sure that there are no data inconsistencies. For that I want to use a lock with Monitor.Pulse and Monitor.Wait. I used a method that I found on another question/answer, but whenever I run my program, the first thread gets stuck at the Monitor.Wait level. I think that's because the second thread has already "Pulsed" and "Waited". Here is some code to look at:
string currentInstruction;
public void nextInstruction()
{
Action actions = {
fetch,
decode
}
Parallel.Invoke(actions);
_pc++;
}
public void fetch()
{
lock(irLock)
{
currentInstruction = "blah";
GiveTurnTo(2);
WaitTurn(1);
}
decodeEvent.WaitOne();
}
public void decode()
{
decodeEvent.Set();
lock(irLock)
{
WaitTurn(2);
currentInstruction = "decoding..."
GiveTurnTo(1);
}
}
// Below are the methods I talked about before.
// Wait for turn to use lock object
public static void WaitTurn(int threadNum, object _lock)
{
// While( not this threads turn )
while (threadInControl != threadNum)
{
// "Let go" of lock on SyncRoot and wait utill
// someone finishes their turn with it
Monitor.Wait(_lock);
}
}
// Pass turn over to other thread
public static void GiveTurnTo(int nextThreadNum, object _lock)
{
threadInControl = nextThreadNum;
// Notify waiting threads that it's someone else's turn
Monitor.Pulse(_lock);
}
Any idea how to get 2 parallel threads to communicate (manipulate the same resources) within the same cycle using locks or anything else?

You want to run 2 peaces of code in parallel, but locking them at start using the same variable?
As nvoigt mentioned, it already sounds wrong. What you have to do is to remove lock from there. Use it only when you are about to access something exclusively.
Btw "data inconsistencies" can be avoided by not having to have them. Do not use currentInstruction field directly (is it a field?), but provide a thread safe CurrentInstruction property.
private object _currentInstructionLock = new object();
private string _currentInstruction
public string CurrentInstruction
{
get { return _currentInstruction; }
set
{
lock(_currentInstructionLock)
_currentInstruction = value;
}
}
Other thing is naming, local variables name starting from _ is a bad style. Some peoples (incl. me) using them to distinguish private fields. Property name should start from BigLetter and local variables fromSmall.

Multi-process access to database in Windows Phone

I'm writing a windows phone app which stores data in a local database. There are multiple threads in my app that access the database and up until this point I have used the technique described here with an AutoResetEvent to ensure that only one thread can access the database at any one time.
So far this has worked very reliably, but now I want to add a ScheduledTask to do some work in the background so I've potentially got multiple processes now competing for access to the database.
Can anyone advise how I can adapt the AutoResetEvent technique to be used across multiple processes on Windows Phone?
I have seen approaches using a Mutex. If I acquire the Mutex before each DB call and then release it afterwards (similar to the way I'm using AutoResetEvent), will this do the trick? Is there any potential problems with this technique? eg: performance?

Ok so first of all my problem was actually 2 problems:
Need to ensure that if the foreground app is running, the background process won't run
Need to ensure that only one thread can access the databasse at once and this needs to work across processes to cater for the (admittedly rare, but possible) scenario where the foreground app is started while the background process is in progress.
Based on the good work done in this thread, I created a couple of classes to help.
To solve problem (1), I created the SingleInstanceSynchroniser:
/// <summary>
/// Used to ensure only one instance (foreground app or background app) runs at once
/// </summary>
public class SingleInstanceSynchroniser : IDisposable
{
private bool hasHandle = false;
Mutex mutex;
private void InitMutex()
{
string mutexId = "Global\\SingleInstanceSynchroniser";
mutex = new Mutex(false, mutexId);
}
public SingleInstanceSynchroniser()
{
InitMutex();
hasHandle = mutex.WaitOne(0);
}
public void Dispose()
{
if (hasHandle && mutex != null)
mutex.ReleaseMutex();
}
public bool HasExclusiveHandle { get { return hasHandle; } }
}
Usage:
In App.xaml.cs:
...
SingleInstanceSynchroniser singleInstanceSynchroniser;
public App()
{
singleInstanceSynchroniser = new SingleInstanceSynchroniser();
...
In ScheduledAgent.cs:
SingleInstanceSynchroniser singleInstanceSynchroniser;
protected override void OnInvoke(ScheduledTask task)
{
singleInstanceSynchroniser = new SingleInstanceSynchroniser();
if (singleInstanceSynchroniser.HasExclusiveHandle)
{
//Run background process
...
}
else
{ //Do not run if foreground app is running
NotifyComplete();
}
}
To solve problem (2), I created the SingleAccessSynchroniser:
/// <summary>
/// Used to ensure only one call is made to the database at once
/// </summary>
public class SingleAccessSynchroniser : IDisposable
{
public bool hasHandle = false;
Mutex mutex;
private void InitMutex()
{
string mutexId = "Global\\SingleAccessSynchroniser";
mutex = new Mutex(false, mutexId);
}
public SingleAccessSynchroniser() : this(0)
{ }
public SingleAccessSynchroniser(int TimeOut)
{
InitMutex();
if (TimeOut <= 0)
hasHandle = mutex.WaitOne();
else
hasHandle = mutex.WaitOne(TimeOut);
if (hasHandle == false)
throw new TimeoutException("Timeout waiting for exclusive access on SingleInstance");
}
public void Release()
{
if (hasHandle && mutex != null)
{
mutex.ReleaseMutex();
hasHandle = false;
}
}
public void Dispose()
{
Release();
}
}
Usage: In all database calls:
using (var dbSync = new SingleAccessSynchroniser())
{
//Execute your database calls
}
This has been running reliably for a few weeks now. Hope someone else finds it useful.

I ran into some problems using Bens solution on Windows Phone 8. Please see this thread for a complete documentation of the problems.
I was able to resolve the issues by removing "Global\" from "Global\SingleInstanceSynchroniser".

Concurrent access to a database between an agent and an app shouldn't be an issue. In fact, using Linq2SQL is one of the recommended ways for communicating between the app and agent.
In practice, it's rarely necessary for the app and agent to run at the same time and so it may be more appropriate to prevent that happening instead.
Potential performance issues will be dependent upon what you're doing. You'll need to measure this to see if it's really an issue.