Develop Reference

StreamWriter not flushing

I want to write a file of about 10 million rows. I'm using StreamWriter with a using statement, however it seems that the StreamWriter is not flushing. Here is my code:
public void ExportRecords(IEnumerable<Record> records, string path)
{
using(TextWriter writer = new StreamWriter(path))
{
writer.WriteLine("header");
foreach(var record in records)
{
string line = "";
//Fill line with record properties
writer.WriteLine(line);
}
}
}
At a certain line the StreamWriter doesn't add any rows. So where is the issue?
EDIT
As mentioned is the comments the code is run async.
Here is how it is:
Task.Run(() =>
{
if (!String.IsNullOrEmpty(folderPath))
{
var records = _generator.GenerateData();
_dataService.ExportRecords(records, $"{folderPath}/RECORDS.csv");
//Just a message box
_reporter.ReportMessage($"Generation and exportation finished");
}
else
{
IsGenerating = false;
_reporter.ReportMessage("No output file was created");
}
});
So I did put a breaking point to check if the method starts executing, and it does. I'm just indicating the the code is not stuck in generation method.

Change temporarily the type of your application to Console Application (if it is Windows Application), and add some Console.WriteLine at strategic points so that you have some feedback about the progress of the procedure. Even better you could add logging to your application, using a library like log4net or Serilog. It will be useful in the production phase of your application too, not only in the development/debugging phase.

Flush() has to be manually called at some point if you need it in the middle of the operation, otherwise the stream will only be flushed at the end of the statement, when it is disposed.

How to better implement start and stop video recording Web Services using Axis Media Parser API?

I have a C# Web Service which records data from an IP camera. This service works fine when I record data during a specified amount of time, for example 10 seconds. But my objective is to achieve data recording for an unspecified amount of time and let the user press to stop recording. So I modified my code creating a new Web Service (stopRecording) to change the value of a global variable that acts as a mutex. Obviously this is wrong because I test it but I don´t know how to proceed. Can anybody help me? I would really appreciate it.
Down here I leave the relevant code.
[WebMethod]
public string startRecording(string ipAddress)
{
// Connection preset for H.264 (HTTP API 3.0)
string Url = "axrtsp:://" + ipAddress + "/axis-media/media.amp?videocodec=h264";
string UserName = "username";
string Password = "pass";
string Path = "C:/directory/subdirectory/";
string Filename = "myRecordedFile.bin";
string FilePath = Path + Filename;
// Open binary output file to write parsed video frames into
using (FileStream outFileStream = new FileStream(FilePath, FileMode.Create))
using (outFile = new BinaryWriter(outFileStream))
{
try
{
// Register for events like OnVideoSample, OnAudioSample, OnTriggerData and OnError
...
// Set connection and media properties
...
// Get absolute time from Axis device
...
// Connect to the device
int cookieID;
int numberOfStreams;
object buffer;
parser.Connect(out cookieID, out numberOfStreams, out buffer);
// Write media type information to out file (buffer is an array of bytes)
byte[] mediaTypeBuffer = (byte[])buffer;
outFile.Write(mediaTypeBuffer.Length);
outFile.Write(mediaTypeBuffer, 0, mediaTypeBuffer.Length);
// Start the media stream and registered event handlers will be called
parser.Start();
Debug.WriteLine("Will start recording...");
do {
Debug.WriteLine("recording..."); //want to record during an unspecified time
} while (record); //my mutex variable that doesn´t make the thing even when I call the stopRecording Web Service. The program remains overlooping
System.Diagnostics.Debug.Write("Finish recording... never reached!!!!! ");
// Stop the stream
parser.Stop();
// Unregister event handlers
...
}
catch (COMException e)
{
Console.WriteLine("Exception for {0}, {1}", parser.MediaURL, e.Message);
}
// Inform the GC that COM object no longer will be used
Marshal.FinalReleaseComObject(parser);
parser = null;
Console.WriteLine("Stream stopped");
}
return "Recording from camera " + Url;
}
[WebMethod]
public string stopRecording()
{
System.Diagnostics.Debug.Write("I want to stop recording...");
record = false;
return "Stop";
}

Your record variable is not a Mutex object, but a simple flag... Which is besides the point.
Trouble here is that you code in startRecording() never gives the hand back to the parent class and might be holding the processing thread forever.
If I might suggest, why not create a thread to do your recording ? You have a number of possibilities here ( new Thread(), Action.BeginInvoke(), .. )
This way, you give a chance to your stopRecording to be received and set this record flag to false and leave the recording thread.

How to open a process and make the steam read/writers available for the entire class

I am currently writing an application that manipulates an existing Console app that has already been built. Currently I am able to launch the existing application and then write to the console and receive the output. But I need my app to basically keep the console app running behind the scenes and keep the app open and ready to write a new command to the window to receive more information back. Below is the current code I am using. I am wondering if there is a way to call this code on launch to start the console application.
Code:
private void Button_Click_1(object sender, RoutedEventArgs e)
{
string ApplicationPath = "python";
string ApplicationArguments = "Console/dummy.py";
string returnValue;
//Process PyObj = new Process();
ProcessStartInfo PyObjStartInfo = new ProcessStartInfo();
PyObjStartInfo.FileName = ApplicationPath;
PyObjStartInfo.Arguments = ApplicationArguments;
PyObjStartInfo.RedirectStandardInput = true;
PyObjStartInfo.RedirectStandardOutput = true;
PyObjStartInfo.UseShellExecute = false;
//PyObjStartInfo.CreateNoWindow = true;
//PyObj.StartInfo = PyObjStartInfo;
Thread.Sleep(5000);
using (Process process = Process.Start(PyObjStartInfo))
{
StreamWriter sw = process.StandardInput;
StreamReader sr = process.StandardOutput;
if (sw.BaseStream.CanWrite)
{
sw.WriteLine("auth");
}
sw.Close();
sw.Close();
returnValue = sr.ReadToEnd();
MessageBox.Show(returnValue.ToString());
}
//Thread.Sleep(5000);
//PyObj.WaitForExit();
//PyObj.Close();
}
As you can see this utilizes a button click currently, but I would like the code to run off the bat as soon as my application starts up. Then keep the console app running and in memory so that I can interact with it. Is there any way to do this in C#.net?
For reference. The console application I cam calling is blank and just returns dummy answers for the time being.. here is the Python code below.
python code:
import os, pprint
def main():
keepGoing = True
while keepGoing:
response = menu()
if response == "0":
keepGoing = False
elif response == "auth":
print StartAuthProcess()
elif response == "verify":
print VerifyKey(raw_input(""))
elif response == "get":
print Info()
else:
print "I don't know what you want to do..."
def menu():
'''
print "MENU"
print "0) Quit"
print "1) Start Autentication Process"
print "2) Verify Key"
print "3) Get Output"
return raw_input("What would you like to do? ")
'''
return raw_input();
def StartAuthProcess():
return 1;
def VerifyKey(key):
if(key):
return 1;
else:
return 0;
def Info():
info = "{dummy:stuff}";
return info;
main()

There are a few places you can put code that will run right away. First of all, you'll see a Program.cs that has your static void Main function. That is where your application starts executing. The Form isn't even shown until the call to Application.Run(). This is a good place to put very early initialization stuff.
If you want things to happen when your Form is first opened, you can override the virtual Form.OnShown method:
protected override void OnShown(EventArgs e) {
base.OwnShown(e);
// Call whatever functions you want here.
}
Note that you really shouldn't use any blocking calls like Sleep in the GUI thread (aka your button click handler). This will cause your GUI to hang, and feel unresponsive. I'm not sure exactly how you plan on interacting with the background process (will it be automatic, or user-driven?) But any blocking calls (namely reads from stdout) should happen on a background thread. You can then use Control.Invoke to marshal calls back onto the UI thread to update controls, etc.

Why does StandardOutput.Read() never return? (deadlock?)

Using C#, I want to automate a third-party Windows command-line program. Usually, it is an interactive console, you send commands, it may prompt for details, send back a result and display a prompt to ask for more commands. Typically:
c:\>console_access.exe
Prompt> version
2.03g.2321
Prompt>
I used .NET classes Process and ProcessStartInfo along with redirections of stdin/stdout/stderr.
public ConsoleAccess()
{
if (!File.Exists(consoleAccessPath)) throw new FileNotFoundException(consoleAccessPath + " not found");
myProcess = new Process();
ProcessStartInfo myProcessStartInfo = new ProcessStartInfo(consoleAccessPath, ""); // even "2>&1" as argument does not work; my code still hangs
myProcessStartInfo.CreateNoWindow = true;
myProcessStartInfo.UseShellExecute = false;
myProcessStartInfo.RedirectStandardOutput = true;
myProcessStartInfo.RedirectStandardError = true;
myProcessStartInfo.RedirectStandardInput = true;
//myProcessStartInfo.ErrorDialog = true; // I tried, to no avail.
myProcess.StartInfo = myProcessStartInfo;
outputQueue = new ConcurrentQueue<string>(); // thread-safe queue
errorQueue = new ConcurrentQueue<string>();
myProcess.Start();
myStandardOutput = myProcess.StandardOutput;
myStandardError = myProcess.StandardError;
myStandardInput = myProcess.StandardInput;
stdOutPumper = new Thread(new ThreadStart(PumpStdOutLoop));
stdOutPumper.Start();
stdErrPumper = new Thread(new ThreadStart(PumpStdErrLoop));
stdErrPumper.Start();
string empty = getResponse(); // check for prompt
string version = getVersion(); // one simple command
}
// [...]
private void PumpStdErrLoop()
{
while (true)
{
string message = myStandardError.ReadLine();
errorQueue.Enqueue(message);
}
}
private void PumpStdOutLoop()
{
while (true)
{
bool done = false;
string buffer = "";
//int blocksize = 1024;
string prompt = "Prompt> ";
while (!done)
{
//char[] intermediaire = new char[blocksize];
//int res = myStandardOutput.Read(intermediaire, 0, blocksize);
//buffer += new string(intermediaire).Substring(0, res);
byte b = (byte)myStandardOutput.Read(); // I go byte per byte, just in case the char[] above is the source of the problem. To no avail.
buffer += (char)b;
done = buffer.EndsWith(prompt);
}
buffer = buffer.Substring(0, buffer.Length - prompt.Length);
outputQueue.Enqueue(buffer);
}
}
Since this program returns "Prompt> " (important : without "\n" at the end) when it's waiting for commands, I can't use myProcess.BeginOutputReadLine();
However, I have to use threads because I must listen stdout AND stderr at the same time.
This is why I used threads and thread-safe queues for a class producer/consumer pattern.
"You can use asynchronous read operations to avoid these dependencies and their deadlock potential. Alternately, you can avoid the deadlock condition by creating two threads and reading the output of each stream on a separate thread." source: http://msdn.microsoft.com/en-us/library/system.diagnostics.process.standardoutput%28v=vs.100%29.aspx
With this design, all sequences like
* cmd -> result with no err (something on stdout, nothing on stderr)
* cmd -> error (something on stderr, nothing on stdout)
works as expected. no problem.
cmd -> result with warning (something on both stderr and stdout)
should work (I'm trying to reproduce this scenario)
however, for one command in particular -- a command that prompts for a password during its execution -- does not work:
main thread principal loops forever on
if (errorQueue.Count == 0 && outputQueue.Count == 0) { System.Threading.Thread.Sleep(500); }
thread pumping stdout waits forever on
byte b = (byte)myStandardOutput.Read();
thread pumping stdout waits a line forever on
string message = myStandardError.ReadLine();
What I don't get is why byte b = (byte)myStandardOutput.Read(); does not pump the message "password:". Nothing happens. I never get the first 'p'.
I feel I hit a deadlock scenario, but I do not understand why.
What's wrong?
(I don't think it is very relevant but I tried the above on .NET 4.0 with MS Visual Studio 2010 on Windows 7 32-bit.)

This is a very common failure mode for these kind of interactive console mode programs. The C runtime library automatically switches the stderr and stdout streams to buffered mode when it detects that output is being redirected. Important to improve throughput. So output goes into that buffer instead of getting directly written to the console. Getting your program to see the output requires the buffer to be flushed.
There are three scenarios where the buffer gets flushed. A flush occurs when the buffer is full, typically around 2 kilobytes. Or when the program writes a line terminator (\n). Or when the program explicitly calls fflush(). The first two scenarios do not occur, not enough output and the program isn't using \n. Which points at the problem, the original programmer forgot to call fflush(). Forgetting this is very common, the programmer simply never intended the program to be used other than in an interactive way.
Nothing can do about it, you'll need to ask the owner or author of the program to add fflush(). Maybe you can limp along by just assuming that the prompt is being written.

Capture output of process synchronously (i.e. "when it happens")

I am trying to start a process and capture the output, have come a far way, but am not quite at the solution I'd want.
Specifically, I am trying to reset the IIS on my development machine from a small utility application that I am writing. I have come to the conclusion, by experimenting, that the safe way to do this is by running iisreset.exe in a child process.
If you run iisreset.exe on a command prompt, you get feedback during the process. Running iisreset takes several seconds, and several lines of feedback is generated, with pauses in between.
I'd like to capture this feedback and present it in my Windows Forms application (in a ListBox), and I have succeeded with that. My remaining concern is that I dont get it until the child process finishes. I'd like to get the output from the child process, line by line, immediately when the lines are created.
I have tried to do my homework, reading/testing things from e.g. these:
How to spawn a process and capture its STDOUT in .NET?
Capturing console output from a .NET application (C#)
http://www.aspcode.net/ProcessStart-and-redirect-standard-output.aspx
and several more with similar content. Most (all?) get the output asynchronously (e.g. with Process.ReadToEnd()). I want the output synchonously, which acording to the MSDN documentation involves establishing an event handler etc and I've tried that. It works, but the event handler does not get called until the process exits. I get the output from iisreset.exe, but not until it has finished.
To rule out the possibility that this has something to do with iisreset.exe in particular, I wrote a small console application that generates some output, pausing in between:
namespace OutputGenerator
{
class Program
{
static void Main(string[] args)
{
System.Console.WriteLine("OutputGenerator starting and pausing for 10 seconds..");
System.Threading.Thread.Sleep(10000);
System.Console.WriteLine("Pausing for another 10 seconds..");
System.Threading.Thread.Sleep(10000);
System.Console.WriteLine("Exiting!");
}
}
}
Testing with this it turns out that I get captured data diretly when I want. So, to some extent it seems that the way iisreset.exe outputs the data come into play here.
Here is the code of the program (a Windows Forms application) that does the capture:
using System;
using System.Windows.Forms;
using System.Diagnostics;
namespace OutputCapturer
{
public partial class Form1 : Form
{
public Form1()
{
InitializeComponent();
}
private void btnRun_Click(object sender, EventArgs e)
{
// Running this will show all output after the process has exited
//String path = #"C:\Windows\system32\iisreset.exe";
// Running this will show all output "when it happens"
String path = #"C:\OutputGenerator.exe";
var p = new Process();
p.StartInfo.FileName = path;
p.StartInfo.UseShellExecute = false; // ShellExecute = true not allowed when output is redirected..
p.StartInfo.RedirectStandardOutput = true;
p.StartInfo.CreateNoWindow = true;
p.OutputDataReceived += OutputDataReceived;
p.Start();
p.BeginOutputReadLine();
}
private delegate void OutputDataToTextboxDelegate(String s);
void OutputDataToTextbox(String s)
{
tbxOutput.Text += s + Environment.NewLine;
tbxOutput.Refresh();
}
private void OutputDataReceived(object sender, DataReceivedEventArgs e)
{
if (e.Data != null && e.Data.ToString() != "")
{
// Must run the update of the textbox in the same thread that created it..
tbxOutput.Invoke(
new OutputDataToTextboxDelegate(OutputDataToTextbox),
DateTime.Now.ToString() + ": " + e.Data.ToString()
);
}
}
}
}
Thinking it was an EOL-encoding problem (the output of iisreset.exe apearing as one line to my app)), I ran a debug session. Nope. The event handler for StandardOutput gets called several times (one time for each output line from iisreset.exe), buth these calls come in one burst after the process exits.
I would LOVE if I could get the output from iisreset.exe "when it happens" so that I can show it as a progress indication.
I've seen one other thread with the same/similar problem, Asynchronous capture from a process output not working properly , but w/o a solution.
I'm sort of stumped.

To do autoflushing of printfs / stdouts
C equivalent of autoflush (flush stdout after each write)?
This saved my ass...

It seems that sixlettervariables is correct, and that this has something to do with iisreset.exe isn't flushing it's buffers for each line. (I still wonder what makes it work on a plain command line - i.e. what does cmd.exe do?)
Anyhow.. I tried what apacay suggested, and wrote this:
private void btnRun_Click(object sender, EventArgs e)
{
// Running this will show the output after the process has finished
//String path = #"C:\Windows\system32\iisreset.exe";
// Running this will show all output "when it happens"
String path = #"C:\OutputGenerator.exe";
var p = new Process();
p.StartInfo.FileName = path;
p.StartInfo.UseShellExecute = false; // ShellExecute = true not allowed when output is redirected..
p.StartInfo.RedirectStandardOutput = true;
p.StartInfo.CreateNoWindow = true;
p.Start();
StreamReader sr = p.StandardOutput;
while (!sr.EndOfStream)
{
String s = sr.ReadLine();
if (s != "")
{
tbxOutput.Text += DateTime.Now.ToString() + ": " + s + Environment.NewLine;
}
tbxOutput.Refresh();
}
}
Notice that I am timestamping when I get each line. For my OutputGenerator I get this:
2011-07-06 17:49:11: OutputGenerator starting and pausing for 10 seconds..
2011-07-06 17:49:21: Pausing for another 10 seconds..
2011-07-06 17:49:31: Exiting!
And for iisreset.exe I get this:
2011-07-06 17:57:11: Attempting stop...
2011-07-06 17:57:11: Internet services successfully stopped
2011-07-06 17:57:11: Attempting start...
2011-07-06 17:57:11: Internet services successfully restarted
Running iisreset.exe on the command line, those lines come with pauses in between, over a span of perhaps 10 seconds.
The case seems more or less closed now. Not that I am all that satisfied, but I'm at roads end it seems. I'll reluctantly live with it..
To summarise: In the general case, it is quite possible to capture output synchronously with when it is generated. This thread presents code for two ways to do that - by establishing an event handler, and by "polling" the stream. In my specific case there is something with how iisreset.exe generates output that prevents this.
Thanks to those who participated and contributed!

Well.... you could kick it old-school. Output can be redirected to the input of another program using old-school DOS commands (foo.exe | bar.exe). Write a program that reads from standard in, and you'll get it every time the stream flushes.
Edit
You could also redirect the ouput to a named pipe and read from that. That would also be "as it happens".

Well, I tried a helper class that I know works: http://csharptest.net/browse/src/Library/Processes/ProcessRunner.cs
ProcessRunner runner = new ProcessRunner("iisreset.exe");
runner.OutputReceived += OutputDataReceived;
runner.Start("/RESTART", "/STATUS");
However, this still doesn't solve the problem with this specific executable. It seems that iisreset was written in such a way that this is not possible. Even running the following from the command line:
iisreset.exe /RESTART /STATUS > temp.txt
Still nothing is written to the text file 'temp.txt' until after all services have been restarted.
As for your example code, I would recommend reading a post I wrote some time ago: How to use System.Diagnostics.Process correctly. Specifically you are not reading the std::err stream or redirecting and closing the std::in stream. This can cause very undesirable results in your program. You can look at the example wrapper class linked above for how to do it with the output events, or if you want to directly read the streams you need to use two of your own threads.
static void Main()
{
ProcessStartInfo psi = new ProcessStartInfo(#"C:\Windows\system32\iisreset.exe", "/RESTART /STATUS");
psi.CreateNoWindow = true;
psi.UseShellExecute = false;
psi.RedirectStandardError = true;
psi.RedirectStandardOutput = true;
psi.RedirectStandardInput = true;
ManualResetEvent output_complete = new ManualResetEvent(false);
ManualResetEvent error_complete = new ManualResetEvent(false);
Process p = Process.Start(psi);
new ReadOutput(p.StandardOutput, output_complete);
new ReadOutput(p.StandardError, error_complete);
p.StandardInput.Close();
p.WaitForExit();
output_complete.WaitOne();
error_complete.WaitOne();
}
private class ReadOutput
{
private StreamReader _reader;
private ManualResetEvent _complete;
public ReadOutput(StreamReader reader, ManualResetEvent complete)
{
_reader = reader;
_complete = complete;
Thread t = new Thread(new ThreadStart(ReadAll));
t.Start();
}
void ReadAll()
{
int ch;
while(-1 != (ch = _reader.Read()))
{
Console.Write((char) ch);
}
_complete.Set();
}
}
I wrote this just to see if anything was coming through. Still got nothing until the end, so I think your just SOL on getting asynchronous output from iisreset.

I've had that problem and had to solve it when my logs where too long to read in a single readtoend.
This is what I've done to solve it. It's been doing Ok so far.
myProcess.StartInfo.FileName = path;
myProcess.StartInfo.Arguments = args;
myProcess.StartInfo.UseShellExecute = false;
myProcess.StartInfo.ErrorDialog = false;
myProcess.StartInfo.CreateNoWindow = true;
myProcess.StartInfo.RedirectStandardError = true;
myProcess.StartInfo.RedirectStandardInput = (stdIn != null);
myProcess.StartInfo.RedirectStandardOutput = true;
myProcess.Start();
int index;
OpenLogFile(myLog); //LOGGGGGGGGGGGGG
if (myProcess.StartInfo.RedirectStandardInput)
{
StreamWriter sw = myProcess.StandardInput;
sw.Write(stdIn + Convert.ToChar(26));
}
StreamReader sr = myProcess.StandardOutput;
/*stdOut = new ArrayLi
*/
while (!sr.EndOfStream)
{ //LOGGGGGGGGGGGGG
Log(sr.ReadLine(), true);
}
Here's OpenLogFile
private void OpenLogFile(string fileName)
{
if (file == StreamWriter.Null)
{
file = new StreamWriter(fileName, true);
file.AutoFlush = true;
}
}
Of course that Log is a function that does something elsewhere. But the solution to you question lies here:
while (!sr.EndOfStream)
{ //LOGGGGGGGGGGGGG
Log(sr.ReadLine(), true);
}
while stream reader is still reading, you can be writing it down as the log comes out.

For my specific situation, the solution is what Mr Moses suggested in a comment above, i.e. run iisreset /stop followed by iisreset /start.
I need a proper answer, rather than a comment, in order to mark it as my "accepted answer", so this answer is more of administrativa than a new contribution. The cred should go to Mr Moses.. :-)