Here is the C# code I'm using to launch a subprocess and monitor its output:
using (process = new Process()) {
process.StartInfo.FileName = executable;
process.StartInfo.Arguments = args;
process.StartInfo.UseShellExecute = false;
process.StartInfo.RedirectStandardOutput = true;
process.StartInfo.RedirectStandardInput = true;
process.StartInfo.CreateNoWindow = true;
process.Start();
using (StreamReader sr = process.StandardOutput) {
string line = null;
while ((line = sr.ReadLine()) != null) {
processOutput(line);
}
}
if (process.ExitCode == 0) {
jobStatus.State = ActionState.CompletedNormally;
jobStatus.Progress = 100;
} else {
jobStatus.State = ActionState.CompletedAbnormally;
}
OnStatusUpdated(jobStatus);
}
I am launching multiple subprocesses in separate ThreadPool threads (but no more than four at a time, on a quad-core machine). This all works fine.
The problem I am having is that one of my subprocesses will exit, but the corresponding call to sr.ReadLine() will block until ANOTHER one of my subprocesses exits. I'm not sure what it returns, but this should NOT be happening unless there is something I am missing.
There's nothing about my subprocess that would cause them to be "linked" in any way - they don't communicate with each other. I can even look in Task Manager / Process Explorer when this is happening, and see that my subprocess has actually exited, but the call to ReadLine() on its standard output is still blocking!
I've been able to work around it by spinning the output monitoring code out into a new thread and doing a process.WaitForExit(), but this seems like very odd behavior. Anyone know what's going on here?
The MSDN documents about ProcessStartInfo.RedirectStandardOutput discuss in detail deadlocks that can arise when doing what you are doing here. A solution is provided that uses ReadToEnd but I imagine the same advice and remedy would apply when you use ReadLine.
Synchronous read operations introduce
a dependency between the caller
reading from the StandardOutput stream
and the child process writing to that
stream. These dependencies can cause
deadlock conditions. When the caller
reads from the redirected stream of a
child process, it is dependent on the
child. The caller waits for the read
operation until the child writes to
the stream or closes the stream. When
the child process writes enough data
to fill its redirected stream, it is
dependent on the parent. The child
process waits for the next write
operation until the parent reads from
the full stream or closes the stream.
The deadlock condition results when
the caller and child process wait for
each other to complete an operation,
and neither can continue. You can
avoid deadlocks by evaluating
dependencies between the caller and
child process.
The best solution seems to be async I/O rather than the sync methods:
You can use asynchronous read
operations to avoid these dependencies
and their deadlock potential.
Alternately, you can avoid the
deadlock condition by creating two
threads and reading the output of each
stream on a separate thread.
There is a sample here that ought to be useful to you if you go this route.
I think it's not your code that's the issue. Blocking calls can unblock for a number of reasons, not only because their task was accomplished.
I don't know about Windows, I must admit, but in the Unix world, when a child finishes, a signal is sent to the parent process and this wakes him from any blocking calls. This would unblock a read on whatever input the parent was expecting.
It wouldn't surprise me if Windows worked similarly. In any case, read up on the reasons why a blocking call may unblock.
Related
I have some external exe that I need to run, and read both its output and errors.
prc = new Process();
prc.StartInfo.UseShellExecute = false;
prc.StartInfo.FileName = fileName;
prc.StartInfo.Arguments = arguments;
prc.StartInfo.LoadUserProfile = true;
prc.StartInfo.RedirectStandardOutput = true;
prc.StartInfo.RedirectStandardError = true;
prc.OutputDataReceived += (sendingProcess, outLine) => outputText.AppendLine(outLine.Data);
prc.ErrorDataReceived += (sendingProcess, errorLine) => errorText.AppendLine(errorLine.Data);
prc.Start();
prc.BeginOutputReadLine();
prc.BeginErrorReadLine();
Thread where this happens may get aborted at any time and there is nothing I can change about it, so I can't use prc.WaitForExit(); as it can't be aborted until process itself is killed.
That means that every time thread gets aborted, all execution just hangs up.
So I replaced it with this in separate method:
while (!Thread.CurrentThread.ThreadState.HasFlag(ThreadState.AbortRequested) && !Thread.CurrentThread.ThreadState.HasFlag(ThreadState.Aborted))
{
if (!process.HasExited) continue;
return process.ExitCode;
}
The only problem with this is outputs. Weirdly, there are sometimes strings missing in the end both from output and error, and adding Thread.Sleep solves this problem, meaning that asynchronous writing can't keep up for some reason.
This seems like a trivial problem, yet I can't find a reliable solition. Thread.Sleep is hardly a good practice, and execution time is very important. What is the best way to reliably get all errors and output?
The problem with the output is that the write ends of the anonymous pipes used to carry it to your process may not be flushed until the process exits - it doesn't always happen, but you can't rely on it not happening. Instead, why not run the process stuff, including WaitForExit() (or maybe just WaitForExit()) in a separate thread? Your original thread can wait for this worker to finish in Thread.Abortable manner (e.g. Thread.Join()), and you'll have your complete output. Or, if you aren't worried about ThreadAbortException but need to be able to cancel the WaitForExit() call, you can use WaitHandle.WaitAny(prc.Handle,cancellationToken.WaitHandle).
This is really a streams question, but I'm asking specifically to solve my current problem with processes.
System.Diagnostics.Process exposes StandardOutput as a stream. I want to listen to this stream and process its output line by line. Obviously there's no direct correlation between input and output, but let's add the slightly artificial requirement that we can process output "by line".
So most examples of using it look like this:
using (Process process = Process.Start(start))
{
//
// Read in all the text from the process with the StreamReader.
//
using (StreamReader reader = process.StandardOutput)
{
string result = reader.ReadToEnd();
Console.Write(result);
}
}
Which is of no use as it isn't event driven and assumes the process lives just long enough to return some output. By event driven, it doesn't have to be an event. A lambda, callback, event, whatever, I just want notification when a whole line is output and I want to be able to shutdown cleanly when I finish with the process.
Essentially I'm asking are streams poll only.
Thanks
You can receive data using the Process.OutputDataReceived event. It can be tricky to use. Search Stack Overflow for it and you'll find a few synchronization issues and pitfalls.
In general you never need to poll a stream for data. In fact there is no way to poll as far as I'm aware. You either read synchronously or asynchronously. Your call will only complete when there is data or the stream is depleted.
In the async case you can view the callback you receive as an event. So just call BeginReadLine and that's your event. Or do it with tasks.
Consider this example C# code (irrelevant pieces left out):
using System.Diagnostics.Process;
var process = new Process();
var startInfo = process.StartInfo;
startInfo.RedirectStandardOutput = true;
startInfo.RedirectStandardError = true;
process.EnableRaisingEvents = true;
process.OutputDataReceived += OutputHandler;
process.ErrorDataReceived += ErrorHandler;
process.Exited += ExitHandler;
process.Start();
process.BeginOutputReadLine();
process.BeginErrorReadLine();
Now, I want to notify a listener that the process is finished after there is no more output (stdout/stderr) to read from it. How do I ensure in my ExitHandler method that all remaining stdout/stderr is processed by OutputHandler and ErrorHandler before determining that the process has truly finished?
There is an interlock when you explicitly use Process.WaitForExit(-1). It won't return until the asynchronous readers for stdout and stderr have indicated end-of-file status. Call it in your Exited event handler. You must use a timeout of -1 or this won't work. Or just WaitForExit(). Which is fine, you know it already exited.
The Exited event is called when the process aborts or terminates. There should therefore never be a situation whereby there is still data to read.
When the operating system shuts down a process, any process component
that is waiting for an exit is notified. The component can then access
the associated process information that is still resident in the
operating system memory (such as ExitTime property) by using the
handle that it has to the process.
Because the associated process has exited, the Handle property of the
component no longer points to an existing process resource. Instead,
it can be used only to access the operating system's information about
the process resource. The system is aware of handles to exited
processes that have not been released by Process components, so it
keeps the ExitTime and Handle property information in memory until the
Process component specifically frees the resources.
I have a weird issue:
In my C# app, I am creating another thread, like so:
Thread printThread = new Thread(printWorker);
printThread.Name = "Logger MainThread";
printThread.IsBackground = true;
printThread.Start();
When my main thread finishes, this new thread just keeps on working, although it's marked as Background.
What could be the causes for this?
This object is holding a Mutex object, not sure this may be the reason...
Any ideas anyone?
Here's the code from the printWorker method:
while (loggerIsActive)
{
LogMessage log = LoggerQueue.Dequeue();
if (log.message != null)
{
syncLogObj.WaitOne();
lock (writerobj)
{
StreamWriter sw;
if (!File.Exists(fName))
{
sw = File.CreateText(fName);
}
else
{
sw = new StreamWriter(fName, true);
}
using (sw)
{
if (log.message != "")
{
if (log.message.EndsWith("\r\n"))
{
log.message =
log.message.Substring(0, log.message.Length - 2);
}
sw.WriteLine(string.Format("[{0}][{3}][{1}] | {2}",
log.msgTime,
log.level.ToString(),
log.message,
log.sender.ToString()));
}
sw.Flush();
sw.Close();
}
}
syncLogObj.ReleaseMutex();
}
Thread.Sleep(5);
}
Try this:
Start the app through VS and exit normally. The VS should stay in Debug mode as you described. Click on Pause button (Break all) and then go to Debug->Windows->Threads. Do you see your "Logger MainThread" in the list?
If so, double-click it, it should lead you to the code line that the thread is currently executing. Step-debug from there and see why is it not terminating.
If you don't see it try looking at other threads that have not terminated and try to find the problem.
Otherwise, with those kind of problems it's always useful to monitor the program state via System.Diagnostics.Debug.Print statements (you can see them printing in the VS output window).
kill it.
Not pretty. But this isn't TV. Read on:
1) Not sure you use are using it but it appears you should be locking loggerqueue before you queue(main pgm) or dequeue(thread).
2) No need to lock writerobj with just this setting. But really you should so you can safely kill the thread not during a write:
main thread:
do everything
before close:
-lock writerobj
-printthread.abort
worker thread:
add try catch to handle threadabort exception and just quit
If you're properly doing this, you shouldn't have to use Waits and mutexes. If you are using wait properly anyway you won't need the sleep.
General advice for this application: why not log on main thread? if your logging is that busy, log results will be pretty useless.
But there are rare cases where that might be wrong. Entonces......
General advice to have threads play nice for this problem:
Main program
encapsulate logging (notably, quit flag, queue, and worker thread ref) in an object
'global snobs?' Logging is a rare excuse to use singleton patter.
start worker thread in logger object via method
main thread always calls a single method on logger object to log error
That method locks the queue and adds to it.
Use Monitor/Pulse/Wait, no sleep; full examples abound; it is worth learning
because only this thread is hitting the file anyway, unless you have multiple processes, you don't need waitone/releasemutex.
That logging method monitor.pulses an object
That frees the worker thread's monitor.wait (which is what idles the CPU instead of sleep)
lock the queue, only inside the lock dequeue the object to local ref; nothing else.
Do your normal logging code and 'exit check' loop. Add
Your logic code could leave message unwritten if queue is full on quit:
change to exit check so you can do it without an extra lock of queue:
move declaration of queued object refernce above while; set it to nothing
change logic in while to 'loggerisactive or log != null'
when your main thread finishes, in your exit code:
set the quit flag
pulse the object you're using to wait incase it's not processing the queue
Thread will fall thru.
You have a lot of stuff going on that you're obviously not showing...
Exmaple: you have syncLogObj.WaitOne();, but we don't see where syncLogObj is being declared, or used elsewhere in your program.
Plus, you don't need it... get rid of the syncLogObj thing altogether (including the "ReleaseMutex" garbage)... you already have a lock (blah) { }, and that's all you need (from what code you have displayed).
It's likely that the main thread is NOT ending, likely because of this or some other object that is keeping it open.
So, simple instructions
Get rid of syncLogObj (because you already have the "lock")
Make sure you set loggerIsActive = false somewhere.
Edit: Even more details!
From what I see - you don't need the lock (writerobj) at all, because (I'm quite sure), you only seem to have one thread that is writing to the log.
The "lock" is only there if you have two or more threads that running that code (basically).
If printworker does not finish before your main thread is done, then main will die and your printworker thread will be killed by the OS. If you want main to wait for the thread you created, then you should call printThread.Join() in main. That will get main to wait on your thread.
When main finishes your program dies and your printThread will be destroyed by the OS, It will not keep running.
From here
Background threads are identical to
foreground threads with one exception:
a background thread does not keep the
managed execution environment running.
Once all foreground threads have been
stopped in a managed process (where
the .exe file is a managed assembly),
the system stops all background
threads and shuts down.
Tony the Tiger has the right idea but additional code needs to be added to kill the thread before the application closes.
printThread.Join(1000);
if(printThread!=null && printThread.IsAlive)
printThread.Abort();
Thread.Abort();
Thread.Dispose();
That should do it if I'm not mistaken.
Quick preface of what I'm trying to do. I want to start a process and start up two threads to monitor the stderr and stdin. Each thread chews off bits of the stream and then fires it out to a NetworkStream. If there is an error in either thread, both threads need to die immediately.
Each of these processes with stdout and stdin monitoring threads are spun off by a main server process. The reason this becomes tricky is because there can easily be 40 or 50 of these processes at any given time. Only during morning restart bursts are there ever more than 50 connections, but it really needs to be able to handle 100 or more. I test with 100 simultaneous connections.
try
{
StreamReader reader = this.myProcess.StandardOutput;
char[] buffer = new char[4096];
byte[] data;
int read;
while (reader.Peek() > -1 ) // This can block before stream is streamed to
{
read = reader.Read(buffer, 0, 4096);
data = Server.ClientEncoding.GetBytes(buffer, 0, read);
this.clientStream.Write(data, 0, data.Length); //ClientStream is a NetworkStream
}
}
catch (Exception err)
{
Utilities.ConsoleOut(string.Format("StdOut err for client {0} -- {1}", this.clientID, err));
this.ShutdownClient(true);
}
This code block is run in one Thread which is right now not Background. There is a similar thread for the StandardError stream. I am using this method instead of listening to OutputDataReceived and ErrorDataReceived because there was an issue in Mono that caused these events to not always fire properly and even though it appears to be fixed now I like that this method ensures I'm reading and writing everything sequentially.
ShutdownClient with True simply tries to kill both threads. Unfortunately the only way I have found to make this work is to use an interrupt on the stdErrThread and stdOutThread objects. Ideally peek would not block and I could just use a manual reset event to keep checking for new data on stdOut or stdIn and then just die when the event is flipped.
I doubt this is the best way to do it. Is there a way to execute this without using an Interrupt?
I'd like to change, because I just saw in my logs that I missed a ThreadInterruptException thrown inside Utlities.ConsoleOut. This just does a System.Console.Write if a static variable is true, but I guess this blocks somewhere.
Edits:
These threads are part of a parent Thread that is launched en masse by a server upon a request. Therefore I cannot set the StdOut and StdErr threads to background and kill the application. I could kill the parent thread from the main server, but this again would get sticky with Peek blocking.
Added info about this being a server.
Also I'm starting to realize a better Queuing method for queries might be the ultimate solution.
I can tell this whole mess stems from the fact that Peek blocks. You're really trying to fix something that is fundamentally broken in the framework and that is never easy (i.e. not a dirty hack). Personally, I would fix the root of the problem, which is the blocking Peek. Mono would've followed Microsoft's implementation and thus ends up with the same problem.
While I know exactly how to fix the problem should I be allowed to change the framework source code, the workaround is lengthy and time consuming.
But here goes.
Essentially, what Microsoft needs to do is change Process.StartWithCreateProcess such that standardOutput and standardError are both assigned a specialised type of StreamReader (e.g. PipeStreamReader).
In this PipeStreamReader, they need to override both ReadBuffer overloads (i.e. need to change both overloads to virtual in StreamReader first) such that prior to a read, PeekNamedPipe is called to do the actual peek. As it is at the moment, FileStream.Read() (called by Peek()) will block on pipe reads when no data is available for read. While a FileStream.Read() with 0 bytes works well on files, it doesn't work all that well on pipes. In fact, the .NET team missed an important part of the pipe documentation - PeekNamedPipe WinAPI.
The PeekNamedPipe function is similar to the ReadFile function with the following exceptions:
...
The function always returns immediately in a single-threaded application, even if there is no data in the pipe. The wait mode of a named pipe handle (blocking or nonblocking) has no effect on the function.
The best thing at this moment without this issue solved in the framework would be to roll out your own Process class (a thin wrapper around WinAPI would suffice).
Why dont you just set both Threads to be backround and then kill the app? It would cause an immediate closing of both threads.
You're building a server. You want to avoid blocking. The obvious solution is to use the asynchronous APIs:
var myProcess = Process.GetCurrentProcess();
StreamReader reader = myProcess.StandardOutput;
char[] buffer = new char[4096];
byte[] data;
int read;
while (!myProcess.HasExited)
{
read = await reader.ReadAsync(buffer, 0, 4096);
data = Server.ClientEncoding.GetBytes(buffer, 0, read);
await this.clientStream.WriteAsync(data, 0, data.Length);
}
No need to waste threads doing I/O work :)
Get rid of peek and use the method below to read from the process output streams. ReadLine() returns null when the process ends. To join this thread with your calling thread either wait for the process to end or kill the process yourself. ShutdownClient() should just Kill() the process which will cause the other thread reading the StdOut or StdErr to also exit.
private void ReadToEnd()
{
string nextLine;
while ((nextLine = stream.ReadLine()) != null)
{
output.WriteLine(nextLine);
}
}