This is really a streams question, but I'm asking specifically to solve my current problem with processes.
System.Diagnostics.Process exposes StandardOutput as a stream. I want to listen to this stream and process its output line by line. Obviously there's no direct correlation between input and output, but let's add the slightly artificial requirement that we can process output "by line".
So most examples of using it look like this:
using (Process process = Process.Start(start))
{
//
// Read in all the text from the process with the StreamReader.
//
using (StreamReader reader = process.StandardOutput)
{
string result = reader.ReadToEnd();
Console.Write(result);
}
}
Which is of no use as it isn't event driven and assumes the process lives just long enough to return some output. By event driven, it doesn't have to be an event. A lambda, callback, event, whatever, I just want notification when a whole line is output and I want to be able to shutdown cleanly when I finish with the process.
Essentially I'm asking are streams poll only.
Thanks
You can receive data using the Process.OutputDataReceived event. It can be tricky to use. Search Stack Overflow for it and you'll find a few synchronization issues and pitfalls.
In general you never need to poll a stream for data. In fact there is no way to poll as far as I'm aware. You either read synchronously or asynchronously. Your call will only complete when there is data or the stream is depleted.
In the async case you can view the callback you receive as an event. So just call BeginReadLine and that's your event. Or do it with tasks.
Related
Is there a way to get a Task that completes and returns a key press without a dedicated thread?
// Kernel callback, not a new thread in my process waiting for a keyboard event
var key = await KeyPress();
As Console.ReadKey() is a blocking call and uses a thread only to wait for user input.
You can open the standard input stream, which has asynchronous operations for reading:
using (var stream = Console.OpenStandardInput())
{
var buffer = new byte[1];
var bytesRead = await stream.ReadAsync(buffer, 0, 1);
char c = (char)buffer[0];
Console.WriteLine(c);
}
that is the thing, a thread dedicated only to wait for user input sounds like a waste
(not necessarily a big one, but it feels like it should have an
implementation for this).
I wouldn't be concerned about this for a console app that expects user input.
Anyhow, it might be possible to achieve what you're after by using some underlying Win32 APIs. The docs for ReadConsoleInput say:
A process can specify a console input buffer handle in one of the wait
functions to determine when there is unread console input. When the
input buffer is not empty, the state of a console input buffer handle
is signaled. To determine the number of unread input records in a
console's input buffer, use the GetNumberOfConsoleInputEvents
function. To read input records from a console input buffer without
affecting the number of unread records, use the PeekConsoleInput
function. To discard all unread records in a console's input buffer,
use the FlushConsoleInputBuffer function.
So, in theory, you could use a handle returned by GetStdHandle and pass it to
RegisterWaitForSingleObject. Then you could convert it to an awaitable task using TaskCompletionSource, e.g. as described here.
I haven't verified this in practice. It shoudn't be blocking a thread, but IMO, again, the game isn't worth the candle.
Quick preface of what I'm trying to do. I want to start a process and start up two threads to monitor the stderr and stdin. Each thread chews off bits of the stream and then fires it out to a NetworkStream. If there is an error in either thread, both threads need to die immediately.
Each of these processes with stdout and stdin monitoring threads are spun off by a main server process. The reason this becomes tricky is because there can easily be 40 or 50 of these processes at any given time. Only during morning restart bursts are there ever more than 50 connections, but it really needs to be able to handle 100 or more. I test with 100 simultaneous connections.
try
{
StreamReader reader = this.myProcess.StandardOutput;
char[] buffer = new char[4096];
byte[] data;
int read;
while (reader.Peek() > -1 ) // This can block before stream is streamed to
{
read = reader.Read(buffer, 0, 4096);
data = Server.ClientEncoding.GetBytes(buffer, 0, read);
this.clientStream.Write(data, 0, data.Length); //ClientStream is a NetworkStream
}
}
catch (Exception err)
{
Utilities.ConsoleOut(string.Format("StdOut err for client {0} -- {1}", this.clientID, err));
this.ShutdownClient(true);
}
This code block is run in one Thread which is right now not Background. There is a similar thread for the StandardError stream. I am using this method instead of listening to OutputDataReceived and ErrorDataReceived because there was an issue in Mono that caused these events to not always fire properly and even though it appears to be fixed now I like that this method ensures I'm reading and writing everything sequentially.
ShutdownClient with True simply tries to kill both threads. Unfortunately the only way I have found to make this work is to use an interrupt on the stdErrThread and stdOutThread objects. Ideally peek would not block and I could just use a manual reset event to keep checking for new data on stdOut or stdIn and then just die when the event is flipped.
I doubt this is the best way to do it. Is there a way to execute this without using an Interrupt?
I'd like to change, because I just saw in my logs that I missed a ThreadInterruptException thrown inside Utlities.ConsoleOut. This just does a System.Console.Write if a static variable is true, but I guess this blocks somewhere.
Edits:
These threads are part of a parent Thread that is launched en masse by a server upon a request. Therefore I cannot set the StdOut and StdErr threads to background and kill the application. I could kill the parent thread from the main server, but this again would get sticky with Peek blocking.
Added info about this being a server.
Also I'm starting to realize a better Queuing method for queries might be the ultimate solution.
I can tell this whole mess stems from the fact that Peek blocks. You're really trying to fix something that is fundamentally broken in the framework and that is never easy (i.e. not a dirty hack). Personally, I would fix the root of the problem, which is the blocking Peek. Mono would've followed Microsoft's implementation and thus ends up with the same problem.
While I know exactly how to fix the problem should I be allowed to change the framework source code, the workaround is lengthy and time consuming.
But here goes.
Essentially, what Microsoft needs to do is change Process.StartWithCreateProcess such that standardOutput and standardError are both assigned a specialised type of StreamReader (e.g. PipeStreamReader).
In this PipeStreamReader, they need to override both ReadBuffer overloads (i.e. need to change both overloads to virtual in StreamReader first) such that prior to a read, PeekNamedPipe is called to do the actual peek. As it is at the moment, FileStream.Read() (called by Peek()) will block on pipe reads when no data is available for read. While a FileStream.Read() with 0 bytes works well on files, it doesn't work all that well on pipes. In fact, the .NET team missed an important part of the pipe documentation - PeekNamedPipe WinAPI.
The PeekNamedPipe function is similar to the ReadFile function with the following exceptions:
...
The function always returns immediately in a single-threaded application, even if there is no data in the pipe. The wait mode of a named pipe handle (blocking or nonblocking) has no effect on the function.
The best thing at this moment without this issue solved in the framework would be to roll out your own Process class (a thin wrapper around WinAPI would suffice).
Why dont you just set both Threads to be backround and then kill the app? It would cause an immediate closing of both threads.
You're building a server. You want to avoid blocking. The obvious solution is to use the asynchronous APIs:
var myProcess = Process.GetCurrentProcess();
StreamReader reader = myProcess.StandardOutput;
char[] buffer = new char[4096];
byte[] data;
int read;
while (!myProcess.HasExited)
{
read = await reader.ReadAsync(buffer, 0, 4096);
data = Server.ClientEncoding.GetBytes(buffer, 0, read);
await this.clientStream.WriteAsync(data, 0, data.Length);
}
No need to waste threads doing I/O work :)
Get rid of peek and use the method below to read from the process output streams. ReadLine() returns null when the process ends. To join this thread with your calling thread either wait for the process to end or kill the process yourself. ShutdownClient() should just Kill() the process which will cause the other thread reading the StdOut or StdErr to also exit.
private void ReadToEnd()
{
string nextLine;
while ((nextLine = stream.ReadLine()) != null)
{
output.WriteLine(nextLine);
}
}
Here is the C# code I'm using to launch a subprocess and monitor its output:
using (process = new Process()) {
process.StartInfo.FileName = executable;
process.StartInfo.Arguments = args;
process.StartInfo.UseShellExecute = false;
process.StartInfo.RedirectStandardOutput = true;
process.StartInfo.RedirectStandardInput = true;
process.StartInfo.CreateNoWindow = true;
process.Start();
using (StreamReader sr = process.StandardOutput) {
string line = null;
while ((line = sr.ReadLine()) != null) {
processOutput(line);
}
}
if (process.ExitCode == 0) {
jobStatus.State = ActionState.CompletedNormally;
jobStatus.Progress = 100;
} else {
jobStatus.State = ActionState.CompletedAbnormally;
}
OnStatusUpdated(jobStatus);
}
I am launching multiple subprocesses in separate ThreadPool threads (but no more than four at a time, on a quad-core machine). This all works fine.
The problem I am having is that one of my subprocesses will exit, but the corresponding call to sr.ReadLine() will block until ANOTHER one of my subprocesses exits. I'm not sure what it returns, but this should NOT be happening unless there is something I am missing.
There's nothing about my subprocess that would cause them to be "linked" in any way - they don't communicate with each other. I can even look in Task Manager / Process Explorer when this is happening, and see that my subprocess has actually exited, but the call to ReadLine() on its standard output is still blocking!
I've been able to work around it by spinning the output monitoring code out into a new thread and doing a process.WaitForExit(), but this seems like very odd behavior. Anyone know what's going on here?
The MSDN documents about ProcessStartInfo.RedirectStandardOutput discuss in detail deadlocks that can arise when doing what you are doing here. A solution is provided that uses ReadToEnd but I imagine the same advice and remedy would apply when you use ReadLine.
Synchronous read operations introduce
a dependency between the caller
reading from the StandardOutput stream
and the child process writing to that
stream. These dependencies can cause
deadlock conditions. When the caller
reads from the redirected stream of a
child process, it is dependent on the
child. The caller waits for the read
operation until the child writes to
the stream or closes the stream. When
the child process writes enough data
to fill its redirected stream, it is
dependent on the parent. The child
process waits for the next write
operation until the parent reads from
the full stream or closes the stream.
The deadlock condition results when
the caller and child process wait for
each other to complete an operation,
and neither can continue. You can
avoid deadlocks by evaluating
dependencies between the caller and
child process.
The best solution seems to be async I/O rather than the sync methods:
You can use asynchronous read
operations to avoid these dependencies
and their deadlock potential.
Alternately, you can avoid the
deadlock condition by creating two
threads and reading the output of each
stream on a separate thread.
There is a sample here that ought to be useful to you if you go this route.
I think it's not your code that's the issue. Blocking calls can unblock for a number of reasons, not only because their task was accomplished.
I don't know about Windows, I must admit, but in the Unix world, when a child finishes, a signal is sent to the parent process and this wakes him from any blocking calls. This would unblock a read on whatever input the parent was expecting.
It wouldn't surprise me if Windows worked similarly. In any case, read up on the reasons why a blocking call may unblock.
I want to read from a socket in an asynchronous way.
If I used the synchronous code below
StreamReader aReadStream = new StreamReader(aStream);
String aLine = aReadStream.ReadLine(); // application might hand here
with a valid open connection, but no data available for reading, the application would hang at the ReadLine() function.
If I switch to the code below using Asynchronous Programming Model
NetworkStream aStream = aClient.GetStream();
while(true)
{
IAsyncResult res = aStream.BeginRead(data, 0, data.Length, null, null);
if (res.IsCompleted){
Trace.WriteLine("here");
} else {
Thread.Sleep(100);
}
}
In the same scenario what would happen? If there is no data to read, the variable res would appear as Completed or not?
Most important, when there is no data to read, all of my calls in that while loop to aStream.BeginRead() are they scheduled continuously in the Thread Pool? If this is true am I risking to degrade the application performances because the Thread Pool has increased too much its size?
Thanks for the help
AFG
By writing the code in this way, you've basically made it synchronous.
The proper way to make this code work is to call BeginRead, passing a callback handler which will process the data which has been read when it is ready, then go do other work rather than just entering a loop.
The callback handler you specify will also be triggered when the data stream is terminated (e.g. because the connection was closed) which you can handle appropriately.
See http://msdn.microsoft.com/en-us/library/system.net.sockets.networkstream.beginread.aspx for a sample.
Have a look at this article which I wrote that covers asynchronous sockets here on CodeProject which is what I learnt from the "MSDN, August 2005, 'Winsock - Get closer to the wire with high-performance sockets in .NET', Daryn Kiely, Pg 81."
Hope this helps,
Best regards,
Tom.
I have to be able to save a file, unfortunatly it can potentially be very large so saving it can potentially take minutes. As I need to do this from a GUI thread I don't want to block the GUI from executing. I was thinking about attempting the save operation on a seperate thread to allow the primary GUI thread to continue executing.
Is there a nice (easy) way to spawn a new thread, save the file, and destroy the thread without any nasty side effects?!
It must be said that I have NEVER had to use threads before so I am a complete novice! Any and all help would be greatly appreciated!
BackgroundWorker (as suggested by Frederik) is a good choice, particularly if you want to report progress to the UI while you're saving. A search for BackgroundWorker tutorial gets a lot of hits, so you should be able to follow one of those to get you started.
One thing to be careful of: would there be any way of changing the data structure that you'll be trying to save from the UI thread? If so, you should disable those aspects of the UI while you're saving - it would (probably!) be bad to be half way through saving the data, then allow the user to change some of it. If you can get away with effectively handing off the data to the background thread and then not touching it from the UI thread, that will make your life a lot easier.
You could maybe use the BackGroundWorker component, as it will abstract a bit the Threading part for you.
Your problem might be that there are several nice and easy ways of doing it. If you just want to set off the file save and not worry about knowing when it has completed, then having a method
void SaveMyFile(object state)
{
// SaveTheFile
}
and calling it with
ThreadPool.QueueUserWorkItem( SaveMyFile );
will do what you want.
I would recommend doing Asynchronous I/O. It's a little bit easier to set up and doesn't require you to create new threads yourself.
Asynchronous programming is where you have, for example, a file stream you want to write to but does not want to wait for it to finish. You might want to be notified when it's finished but you don't want to wait.
What you do is using the BeginWrite/BeginRead and EndWrite/EndRead functions that are available on the Stream class.
In your method you start by calling BeginWrite with all the data you want to write and also pass in a callback function. This function will be called when BeginWrite has finished.
Inside the callback function you call EndWrite and clean up the stream and check for errors.
BeginWrite will not block which means that if it's called from within an event handler that thread can finish that handler and continue processing more event (such as other GUI events).
using System;
using System.IO;
using System.Text;
class Program
{
private static FileStream stream;
static void Main(string[] args)
{
stream = new FileStream("foo.txt",
FileMode.Create,
FileAccess.Write);
const string mystring = "Foobarlalala";
ASCIIEncoding encoding = new ASCIIEncoding();
byte[] data = encoding.GetBytes(mystring);
Console.WriteLine("Started writing");
stream.BeginWrite(data, 0, data.Length, callback, null);
Console.WriteLine("Writing dispatched, sleeping 5 secs");
System.Threading.Thread.Sleep(5000);
}
public static void callback(IAsyncResult ia)
{
stream.EndWrite(ia);
Console.WriteLine("Finished writing");
}
}
}
The sleeping is pretty important because the thread that's writing stuff will be killed if the main thread is killed off. This is not an issue in a GUI application, only here in this small example.
MSDN has a pretty good overview on how to write this stuff, and also some good articles on Asynch programming in general in case you go for the backgroundworker or ThreadPool.
or u could use old friends delegates.