Develop Reference

Why does StandardOutput.Read() never return? (deadlock?)

Using C#, I want to automate a third-party Windows command-line program. Usually, it is an interactive console, you send commands, it may prompt for details, send back a result and display a prompt to ask for more commands. Typically:
c:\>console_access.exe
Prompt> version
2.03g.2321
Prompt>
I used .NET classes Process and ProcessStartInfo along with redirections of stdin/stdout/stderr.
public ConsoleAccess()
{
if (!File.Exists(consoleAccessPath)) throw new FileNotFoundException(consoleAccessPath + " not found");
myProcess = new Process();
ProcessStartInfo myProcessStartInfo = new ProcessStartInfo(consoleAccessPath, ""); // even "2>&1" as argument does not work; my code still hangs
myProcessStartInfo.CreateNoWindow = true;
myProcessStartInfo.UseShellExecute = false;
myProcessStartInfo.RedirectStandardOutput = true;
myProcessStartInfo.RedirectStandardError = true;
myProcessStartInfo.RedirectStandardInput = true;
//myProcessStartInfo.ErrorDialog = true; // I tried, to no avail.
myProcess.StartInfo = myProcessStartInfo;
outputQueue = new ConcurrentQueue<string>(); // thread-safe queue
errorQueue = new ConcurrentQueue<string>();
myProcess.Start();
myStandardOutput = myProcess.StandardOutput;
myStandardError = myProcess.StandardError;
myStandardInput = myProcess.StandardInput;
stdOutPumper = new Thread(new ThreadStart(PumpStdOutLoop));
stdOutPumper.Start();
stdErrPumper = new Thread(new ThreadStart(PumpStdErrLoop));
stdErrPumper.Start();
string empty = getResponse(); // check for prompt
string version = getVersion(); // one simple command
}
// [...]
private void PumpStdErrLoop()
{
while (true)
{
string message = myStandardError.ReadLine();
errorQueue.Enqueue(message);
}
}
private void PumpStdOutLoop()
{
while (true)
{
bool done = false;
string buffer = "";
//int blocksize = 1024;
string prompt = "Prompt> ";
while (!done)
{
//char[] intermediaire = new char[blocksize];
//int res = myStandardOutput.Read(intermediaire, 0, blocksize);
//buffer += new string(intermediaire).Substring(0, res);
byte b = (byte)myStandardOutput.Read(); // I go byte per byte, just in case the char[] above is the source of the problem. To no avail.
buffer += (char)b;
done = buffer.EndsWith(prompt);
}
buffer = buffer.Substring(0, buffer.Length - prompt.Length);
outputQueue.Enqueue(buffer);
}
}
Since this program returns "Prompt> " (important : without "\n" at the end) when it's waiting for commands, I can't use myProcess.BeginOutputReadLine();
However, I have to use threads because I must listen stdout AND stderr at the same time.
This is why I used threads and thread-safe queues for a class producer/consumer pattern.
"You can use asynchronous read operations to avoid these dependencies and their deadlock potential. Alternately, you can avoid the deadlock condition by creating two threads and reading the output of each stream on a separate thread." source: http://msdn.microsoft.com/en-us/library/system.diagnostics.process.standardoutput%28v=vs.100%29.aspx
With this design, all sequences like
* cmd -> result with no err (something on stdout, nothing on stderr)
* cmd -> error (something on stderr, nothing on stdout)
works as expected. no problem.
cmd -> result with warning (something on both stderr and stdout)
should work (I'm trying to reproduce this scenario)
however, for one command in particular -- a command that prompts for a password during its execution -- does not work:
main thread principal loops forever on
if (errorQueue.Count == 0 && outputQueue.Count == 0) { System.Threading.Thread.Sleep(500); }
thread pumping stdout waits forever on
byte b = (byte)myStandardOutput.Read();
thread pumping stdout waits a line forever on
string message = myStandardError.ReadLine();
What I don't get is why byte b = (byte)myStandardOutput.Read(); does not pump the message "password:". Nothing happens. I never get the first 'p'.
I feel I hit a deadlock scenario, but I do not understand why.
What's wrong?
(I don't think it is very relevant but I tried the above on .NET 4.0 with MS Visual Studio 2010 on Windows 7 32-bit.)

This is a very common failure mode for these kind of interactive console mode programs. The C runtime library automatically switches the stderr and stdout streams to buffered mode when it detects that output is being redirected. Important to improve throughput. So output goes into that buffer instead of getting directly written to the console. Getting your program to see the output requires the buffer to be flushed.
There are three scenarios where the buffer gets flushed. A flush occurs when the buffer is full, typically around 2 kilobytes. Or when the program writes a line terminator (\n). Or when the program explicitly calls fflush(). The first two scenarios do not occur, not enough output and the program isn't using \n. Which points at the problem, the original programmer forgot to call fflush(). Forgetting this is very common, the programmer simply never intended the program to be used other than in an interactive way.
Nothing can do about it, you'll need to ask the owner or author of the program to add fflush(). Maybe you can limp along by just assuming that the prompt is being written.

Duplicate process (strange issue)

I am trying to prevent opening help file more than once.
This is the method I am using:
public void openHelp()
{
int count = 0;
string helpPath = Environment.GetFolderPath(Environment.SpecialFolder.CommonApplicationData) + #"\MyApp\Help\eHelp.chm";
System.Diagnostics.Process[] helpProcs = System.Diagnostics.Process.GetProcesses();
foreach (System.Diagnostics.Process proc in helpProcs)
{
if (proc.MainWindowTitle == "Sample App Help")
{
count++;
}
}
if (count == 0)
{
System.Diagnostics.Process.Start(helpPath);
}
else
{
}
}
The idea is, if you find the process with the same MainWindowTitle, then do not start a new one.
However, this is not reliable. In some cases it still starts the process, even though one is already running. Is there an issue with a logic?
Thank you.
P.S. Of course the MainWindowTitle is "Sample App Help", at least that is what I see while debugging.
Update:
Issue only occurs when user has minimised help file. So I suspect something happens in the system and I need to check something. Any suggestions?

The Remarks section in Process.MainWindowTitle contains the following note:
The main window is the window that currently has the focus; note that
this might not be the primary window for the process. You must use the
Refresh method to refresh the Process object to get the current main
window handle if it has changed.
Could this perhaps be the cause of your problem?
What about keeping the process id of a newly started help viewer and before starting another one, just check if the old one is still alive.
int id = ...
try
{
var proc = Process.GetProcessById(id);
}
catch
{
// no process running with that id
}

How to redirect output from the NASM command line assembler in C#

Brief Summary
I am creating a lightweight IDE for NASM development in C# (I know kind of an irony). Kinda of like Notepad++ but simpler but with features that make it more than source editor. Since Notepad++ is really just a fancy source editor. I have already implemented features like Project creation (using a project format similar to how Visual Studio organizes projects). Project extension .nasmproj. I am also in the works of hosting it in an open-source place (Codeplex). Although the program is far from finish, and definitely cannot be used in a production environment without proper protection and equipment. In addition, I am working alone with it at this moment, more like a spare time project since I just finished my last Summer final taking Calculus I.
Problem
Right now I am facing a problem, I can build the project but no output from NASM is being fed into the IDE. I have succesfully built a project, and I was able to produce object files. I even tried producing a syntax error to see if I finally see something come up but none and I check the bin folder of the test project I created and I see no object file creating. So definitely NASM is doing its magic. Is it because NASM doesn't want me to see its output. Is there a solution? Any advice would be great. Here is the code which I think is giving Trouble.
Things to Note
I have already checked if events have been invoked. An yes they have but they return empty strings
I have also checked error data and same effect.
Code
public static bool Build(string arguments, out Process nasmP)
{
try
{
ProcessStartInfo nasm = new ProcessStartInfo("nasm", arguments);
nasm.CreateNoWindow = true;
nasm.RedirectStandardError = true;
nasm.RedirectStandardInput = true;
nasm.RedirectStandardOutput = true;
nasm.UseShellExecute = false;
nasmP = new Process();
nasmP.EnableRaisingEvents = true;
nasmP.StartInfo = nasm;
bool predicate = nasmP.Start();
nasmP.BeginOutputReadLine();
return true;
}
catch
{
nasmP = null;
return false;
}
}
//Hasn't been tested nor used
public static bool Clean(string binPath)
{
if (binPath == null || !Directory.Exists(binPath))
{
throw new ArgumentException("Either path is null or it does not exist!");
}
else
{
try
{
DirectoryInfo binInfo = new DirectoryInfo(binPath);
FileInfo[] filesInfo = binInfo.GetFiles();
for (int index = 0; index < filesInfo.Length; index++)
{
try
{
filesInfo[index].Delete();
filesInfo[index] = null;
}
catch
{
break;
}
}
GC.Collect();
return true;
}
catch
{
return false;
}
}
}
}
using (BuildDialog dlg = new BuildDialog(currentSolution))
{
DialogResult result = dlg.ShowDialog();
dlg.onOutputRecieved += new BuildDialog.OnOutputRecievedHandler(delegate(Process _sender, string output)
{
if (result == System.Windows.Forms.DialogResult.OK)
{
outputWindow.Invoke(new InvokeDelegate(delegate(string o)
{
Console.WriteLine("Data:" + o);
outputWindow.Text = o;
}), output);
}
});
}
Edits
I have tried doing synchronously instead of asynchronously but still the same result (and empty string "" is returned) actually by debugging the stream is already at the end. So looks like nothing has been written into the stream.
This is what I tried:
string readToEnd = nasmP.StandardOutput.ReadToEnd();
nasmP.WaitForExit();
Console.WriteLine(readToEnd);
And another interesting thing I have tried was I copied the arguments from the debugger and pasted it in the command line shell and I can see NASM compiling and giving the error that I wanted to see all along. So definitely not a NASM problem. Could it be a problem with my code or the .Net framework.
Here is a nice snapshot of the shell window (although not technically proof; this is what the output should look like in my IDE):
Alan made a very good point, check the sub processes or threads. Is sub process and thread synonymous? But here is the problem. Almost all the properties except a select few and output/error streams are throwing an invalid operation. Here is the debugger information as an image (I wish Visual Studio would allow you to copy the entire information in click):

Okay I finally was able to do it. I just found this control that redirect output from a process and I just looked at the source code of it and got what I needed to do. Here is the the modified code:
public static bool Build(string command, out StringBuilder buildOutput)
{
try
{
buildOutput = new StringBuilder();
ProcessStartInfo startInfo = new ProcessStartInfo("cmd.exe");
startInfo.Arguments = "/C " + " nasm " + command;
startInfo.RedirectStandardError = true;
startInfo.RedirectStandardOutput = true;
startInfo.UseShellExecute = false;
startInfo.CreateNoWindow = true;
Process p = Process.Start(startInfo);
string output = p.StandardOutput.ReadToEnd();
string error = p.StandardError.ReadToEnd();
p.WaitForExit();
if (output.Length != 0)
buildOutput.Append(output);
else if (error.Length != 0)
buildOutput.Append(error);
else
buildOutput.Append("\n");
return true;
}
catch
{
buildOutput = null;
return false;
}
}
Here is how the output is formatted like:
I also wanted to thank Alan for helping me debug my code, although he didn't physically had my code. But he really was helpful and I thank him for it.

Capture output of process synchronously (i.e. "when it happens")

I am trying to start a process and capture the output, have come a far way, but am not quite at the solution I'd want.
Specifically, I am trying to reset the IIS on my development machine from a small utility application that I am writing. I have come to the conclusion, by experimenting, that the safe way to do this is by running iisreset.exe in a child process.
If you run iisreset.exe on a command prompt, you get feedback during the process. Running iisreset takes several seconds, and several lines of feedback is generated, with pauses in between.
I'd like to capture this feedback and present it in my Windows Forms application (in a ListBox), and I have succeeded with that. My remaining concern is that I dont get it until the child process finishes. I'd like to get the output from the child process, line by line, immediately when the lines are created.
I have tried to do my homework, reading/testing things from e.g. these:
How to spawn a process and capture its STDOUT in .NET?
Capturing console output from a .NET application (C#)
http://www.aspcode.net/ProcessStart-and-redirect-standard-output.aspx
and several more with similar content. Most (all?) get the output asynchronously (e.g. with Process.ReadToEnd()). I want the output synchonously, which acording to the MSDN documentation involves establishing an event handler etc and I've tried that. It works, but the event handler does not get called until the process exits. I get the output from iisreset.exe, but not until it has finished.
To rule out the possibility that this has something to do with iisreset.exe in particular, I wrote a small console application that generates some output, pausing in between:
namespace OutputGenerator
{
class Program
{
static void Main(string[] args)
{
System.Console.WriteLine("OutputGenerator starting and pausing for 10 seconds..");
System.Threading.Thread.Sleep(10000);
System.Console.WriteLine("Pausing for another 10 seconds..");
System.Threading.Thread.Sleep(10000);
System.Console.WriteLine("Exiting!");
}
}
}
Testing with this it turns out that I get captured data diretly when I want. So, to some extent it seems that the way iisreset.exe outputs the data come into play here.
Here is the code of the program (a Windows Forms application) that does the capture:
using System;
using System.Windows.Forms;
using System.Diagnostics;
namespace OutputCapturer
{
public partial class Form1 : Form
{
public Form1()
{
InitializeComponent();
}
private void btnRun_Click(object sender, EventArgs e)
{
// Running this will show all output after the process has exited
//String path = #"C:\Windows\system32\iisreset.exe";
// Running this will show all output "when it happens"
String path = #"C:\OutputGenerator.exe";
var p = new Process();
p.StartInfo.FileName = path;
p.StartInfo.UseShellExecute = false; // ShellExecute = true not allowed when output is redirected..
p.StartInfo.RedirectStandardOutput = true;
p.StartInfo.CreateNoWindow = true;
p.OutputDataReceived += OutputDataReceived;
p.Start();
p.BeginOutputReadLine();
}
private delegate void OutputDataToTextboxDelegate(String s);
void OutputDataToTextbox(String s)
{
tbxOutput.Text += s + Environment.NewLine;
tbxOutput.Refresh();
}
private void OutputDataReceived(object sender, DataReceivedEventArgs e)
{
if (e.Data != null && e.Data.ToString() != "")
{
// Must run the update of the textbox in the same thread that created it..
tbxOutput.Invoke(
new OutputDataToTextboxDelegate(OutputDataToTextbox),
DateTime.Now.ToString() + ": " + e.Data.ToString()
);
}
}
}
}
Thinking it was an EOL-encoding problem (the output of iisreset.exe apearing as one line to my app)), I ran a debug session. Nope. The event handler for StandardOutput gets called several times (one time for each output line from iisreset.exe), buth these calls come in one burst after the process exits.
I would LOVE if I could get the output from iisreset.exe "when it happens" so that I can show it as a progress indication.
I've seen one other thread with the same/similar problem, Asynchronous capture from a process output not working properly , but w/o a solution.
I'm sort of stumped.

To do autoflushing of printfs / stdouts
C equivalent of autoflush (flush stdout after each write)?
This saved my ass...

It seems that sixlettervariables is correct, and that this has something to do with iisreset.exe isn't flushing it's buffers for each line. (I still wonder what makes it work on a plain command line - i.e. what does cmd.exe do?)
Anyhow.. I tried what apacay suggested, and wrote this:
private void btnRun_Click(object sender, EventArgs e)
{
// Running this will show the output after the process has finished
//String path = #"C:\Windows\system32\iisreset.exe";
// Running this will show all output "when it happens"
String path = #"C:\OutputGenerator.exe";
var p = new Process();
p.StartInfo.FileName = path;
p.StartInfo.UseShellExecute = false; // ShellExecute = true not allowed when output is redirected..
p.StartInfo.RedirectStandardOutput = true;
p.StartInfo.CreateNoWindow = true;
p.Start();
StreamReader sr = p.StandardOutput;
while (!sr.EndOfStream)
{
String s = sr.ReadLine();
if (s != "")
{
tbxOutput.Text += DateTime.Now.ToString() + ": " + s + Environment.NewLine;
}
tbxOutput.Refresh();
}
}
Notice that I am timestamping when I get each line. For my OutputGenerator I get this:
2011-07-06 17:49:11: OutputGenerator starting and pausing for 10 seconds..
2011-07-06 17:49:21: Pausing for another 10 seconds..
2011-07-06 17:49:31: Exiting!
And for iisreset.exe I get this:
2011-07-06 17:57:11: Attempting stop...
2011-07-06 17:57:11: Internet services successfully stopped
2011-07-06 17:57:11: Attempting start...
2011-07-06 17:57:11: Internet services successfully restarted
Running iisreset.exe on the command line, those lines come with pauses in between, over a span of perhaps 10 seconds.
The case seems more or less closed now. Not that I am all that satisfied, but I'm at roads end it seems. I'll reluctantly live with it..
To summarise: In the general case, it is quite possible to capture output synchronously with when it is generated. This thread presents code for two ways to do that - by establishing an event handler, and by "polling" the stream. In my specific case there is something with how iisreset.exe generates output that prevents this.
Thanks to those who participated and contributed!

Well.... you could kick it old-school. Output can be redirected to the input of another program using old-school DOS commands (foo.exe | bar.exe). Write a program that reads from standard in, and you'll get it every time the stream flushes.
Edit
You could also redirect the ouput to a named pipe and read from that. That would also be "as it happens".

Well, I tried a helper class that I know works: http://csharptest.net/browse/src/Library/Processes/ProcessRunner.cs
ProcessRunner runner = new ProcessRunner("iisreset.exe");
runner.OutputReceived += OutputDataReceived;
runner.Start("/RESTART", "/STATUS");
However, this still doesn't solve the problem with this specific executable. It seems that iisreset was written in such a way that this is not possible. Even running the following from the command line:
iisreset.exe /RESTART /STATUS > temp.txt
Still nothing is written to the text file 'temp.txt' until after all services have been restarted.
As for your example code, I would recommend reading a post I wrote some time ago: How to use System.Diagnostics.Process correctly. Specifically you are not reading the std::err stream or redirecting and closing the std::in stream. This can cause very undesirable results in your program. You can look at the example wrapper class linked above for how to do it with the output events, or if you want to directly read the streams you need to use two of your own threads.
static void Main()
{
ProcessStartInfo psi = new ProcessStartInfo(#"C:\Windows\system32\iisreset.exe", "/RESTART /STATUS");
psi.CreateNoWindow = true;
psi.UseShellExecute = false;
psi.RedirectStandardError = true;
psi.RedirectStandardOutput = true;
psi.RedirectStandardInput = true;
ManualResetEvent output_complete = new ManualResetEvent(false);
ManualResetEvent error_complete = new ManualResetEvent(false);
Process p = Process.Start(psi);
new ReadOutput(p.StandardOutput, output_complete);
new ReadOutput(p.StandardError, error_complete);
p.StandardInput.Close();
p.WaitForExit();
output_complete.WaitOne();
error_complete.WaitOne();
}
private class ReadOutput
{
private StreamReader _reader;
private ManualResetEvent _complete;
public ReadOutput(StreamReader reader, ManualResetEvent complete)
{
_reader = reader;
_complete = complete;
Thread t = new Thread(new ThreadStart(ReadAll));
t.Start();
}
void ReadAll()
{
int ch;
while(-1 != (ch = _reader.Read()))
{
Console.Write((char) ch);
}
_complete.Set();
}
}
I wrote this just to see if anything was coming through. Still got nothing until the end, so I think your just SOL on getting asynchronous output from iisreset.

I've had that problem and had to solve it when my logs where too long to read in a single readtoend.
This is what I've done to solve it. It's been doing Ok so far.
myProcess.StartInfo.FileName = path;
myProcess.StartInfo.Arguments = args;
myProcess.StartInfo.UseShellExecute = false;
myProcess.StartInfo.ErrorDialog = false;
myProcess.StartInfo.CreateNoWindow = true;
myProcess.StartInfo.RedirectStandardError = true;
myProcess.StartInfo.RedirectStandardInput = (stdIn != null);
myProcess.StartInfo.RedirectStandardOutput = true;
myProcess.Start();
int index;
OpenLogFile(myLog); //LOGGGGGGGGGGGGG
if (myProcess.StartInfo.RedirectStandardInput)
{
StreamWriter sw = myProcess.StandardInput;
sw.Write(stdIn + Convert.ToChar(26));
}
StreamReader sr = myProcess.StandardOutput;
/*stdOut = new ArrayLi
*/
while (!sr.EndOfStream)
{ //LOGGGGGGGGGGGGG
Log(sr.ReadLine(), true);
}
Here's OpenLogFile
private void OpenLogFile(string fileName)
{
if (file == StreamWriter.Null)
{
file = new StreamWriter(fileName, true);
file.AutoFlush = true;
}
}
Of course that Log is a function that does something elsewhere. But the solution to you question lies here:
while (!sr.EndOfStream)
{ //LOGGGGGGGGGGGGG
Log(sr.ReadLine(), true);
}
while stream reader is still reading, you can be writing it down as the log comes out.

For my specific situation, the solution is what Mr Moses suggested in a comment above, i.e. run iisreset /stop followed by iisreset /start.
I need a proper answer, rather than a comment, in order to mark it as my "accepted answer", so this answer is more of administrativa than a new contribution. The cred should go to Mr Moses.. :-)

Microsoft.Office.Interop.Word "Cannot activate application"

We are having a problem experienced by a few users when attempting to launch Word from our application via the office interop:
using Word = Microsoft.Office.Interop.Word;
public void ShowWord()
{
_word = new Word.ApplicationClass();
_word.Visible = true;
_word.Activate();
}
If word is not always open a COM exception is thrown stating "Cannot activate application." Adding a Thread.Sleep(1000) before calling _word.Activate() prevents this, but obviously is not ideal.
public void ShowWord()
{
_word = new Word.ApplicationClass();
_word.Visible = true;
Thread.Sleep(1000)
_word.Activate();
}
Has anyone seen this before and knows what is causing this and what the right way to fix this is?

We encountered a similar issue, and it seems that Word is asynchonously waiting for the OS to show its window. The way we resolved this is by waiting until the Visible property returns true:
public void ShowWord()
{
_word = new Word.Application();
_word.Visible = true;
System.Diagnostics.Stopwatch sw = System.Diagnostics.Stopwatch.StartNew();
while (!_word.Visible && sw.ElapsedMilliseconds < 10000)
{ /* Just Wait!! (at most 10s) */}
_word.Activate();
}
Hope this helps somebody.

Does your application have permission to activate the Word COM object?
Check in DCOMCNFG what the local activation security requirements are.
However, not sure why your Thread.Sleep(1000)would allow it to work?