I've read about this deadlock condition that I'm pretty certain is affecting my code (below). What I don't understand is: this code worked perfectly well running on Windows Server 2003 (.net 2.0) for the past 12 years. Now we've been trying to move it to Windows Server 2012, where it always deadlocks.
While my DLLs are built for "anyCPU" (still targeting .net 2.0), the executable process being run is absolutely 32-bit, and the move from Server 2003 to Server 2012 goes from a 32-bit to 64-bit OS.
I think I understand what to do to resolve the issue, but does anyone know why this behavior would have changed from Server 2003 to Server 2012?
public string DoMyProcess(string filenameAndPath, string arguments)
{
string stdout="";
int exitCode = 0;
try
{
ProcessStartInfo procStartInfo = new ProcessStartInfo();
procStartInfo.FileName = filenameAndPath;
procStartInfo.CreateNoWindow = true;
procStartInfo.Arguments = arguments;
procStartInfo.RedirectStandardOutput = true;
procStartInfo.UseShellExecute = false;
System.Diagnostics.Process theProcess = null;
try
{
theProcess = Process.Start(procStartInfo);
theProcess.WaitForExit();
exitCode = theProcess.ExitCode;
// moving this ABOVE WaitForExit should eliminate deadlocks
// But why did it always work on Server 2003 but not on Server 2012?
stdout = theProcess.StandardOutput.ReadToEnd();
}
catch (System.Exception e)
{
string errMsg = e.Message;
log_the_error("threw an exception: " + e.Message);
}
}
return stdout;
}
UPDATE:
Mystery deadlock still exists, even after changing the above code as recommended:
try
{
theProcess = Process.Start(procStartInfo);
stdout = theProcess.StandardOutput.ReadToEnd();
}
catch (System.Exception e)
{
string errMsg = e.Message;
log_the_error("threw an exception: " + e.Message);
}
}
What other conditions could cause that deadlock? If I were to examine StandardError, would it reveal anything useful?
UPDATE #2:
FWIW, we have provisioned another Windows Server 2003 (32-bit), which runs IIS 6. That was the original machine configuration this code ran on for 12 years (with only an occasional deadlock). Our same code that deadlocks on Server 2012 IIS 8 DOES NOT DEADLOCK on this Server 2003.
We now have our own minimal and complete code that reproduces the issue. However, the .exe we've licensed that is being executed by the process has confidentiality clauses that prevent us from posting. I realize that doesn't help the experts here.
THE ONE HINT we've encountered is that when run via the Visual Studio 2013 debugger installed on the actual server, the process doesn't deadlock/hang, while invoking the process from a browser OUTSIDE the server does. And oddly -- from a browser ON THE 2012 SERVER we can't connect to that test page -- the browser just says "connecting" and eventually times out (however, other sites hosted by the same server / same IIS 8 CAN BE REACHED from a browser on the server!)
Since the same command line parameters manually run from either an admin command shell or a non-admin command shell works perfectly, it's hard to believe it's a 64-bit / WOW64 problem with this 32-bit executable or it's required DLLs. We continue to search for places our permissions may be causing problems (the process needs to write to a temp folder, which we've placed at c:\temp, for now).
Without a good Minimal, Complete, and Verifiable code example, it's impossible to answer completely.
What I can tell you is that your code was always broken, and always had the potential for deadlock. You fail to read anything from the process until the process exits, but the process may not be able to exit, if it writes so much data to stdout that the buffer fills and blocks the process.
If you haven't recompiled anything, but have found that you see deadlock now when you didn't before, then the most likely explanation is that the process you're starting writes more to stdout than it used to. I.e. all of the output used to fit in the buffer before, but now it doesn't. (I guess it's also possible the buffer size was reduced in the newer OS, but that seems unlikely to me.)
You should go ahead and move the call to ReadToEnd(). In fact, you should do away with the WaitForExit() altogether. If you are calling ReadToEnd(), that won't complete until the process has in fact exited anyway, so calling WaitForExit() afterwards would be pointless.
Related
I have an app that runs DirectX 11 that plays a scene and generates an mp4.
I am trying to launch it through Process.Start so that I can manage the process and force it to timeout if it crashed or doesn't close correctly.
When I test the function on my local Win10 machine it works perfectly, and when I run it through CL or a .BAT file on the WinServ2012R2 machine it works perfectly too.
However when I try to run it through the Process.Start function on the server machine it fails to open DirectX
var startInfo = new System.Diagnostics.ProcessStartInfo($"{AppLocation}", $"{Parameters}")
{
WindowStyle = System.Diagnostics.ProcessWindowStyle.Normal,
UseShellExecute = false,
RedirectStandardOutput = true,
CreateNoWindow = false,
WorkingDirectory = $"{DirectoryName}",
Verb = "runas"
};
var loop = 0;
while (!System.IO.File.Exists($"{FileLocation}"))
{
loop++;
using (var p = System.Diagnostics.Process.Start(startInfo))
{
Logger.Info("Process Running....");
if (!p.WaitForExit(300000))
{
p.Kill();
}
if (loop >= 5)
break;
}
}
Edit: The DirectX error is: DXGI_ERROR_NOT_CURRENTLY_AVAILABLE
0x887A0022
It’s probably something about the environment.
If the parent process is a normal Win32 console or GUI app running inside a desktop of that server, put something like Sleep( 60000 ); in the first line of your main() or WinMain function, and use Process Explorer to find differences between manual launch which works, and programmatic launch which fails. Check the “Image”, “Security” and “Environment” tabs of the processes.
If the parent process is a system service, it’s more complicated. Services generally run under another user account and you gonna need some setup to allow the service, or a child process launched by the service, to access GPU.
Another possible reason is anti-virus or anti-malware breaking things.
P.S. Note you have minor bugs in your code.
One thing, when you detect timeouts, Process.Kill is asynchronous, you need to wait afterwards.
Another one, you specifying RedirectStandardOutput = true but you don't consume that stream. If the child process prints a lot of text it will eventually stall waiting for the parent process to consume the data buffered in that pipe. If you don’t care about output, don’t redirect these streams. If you do care, redirect and consume the data as soon as it printed, either on a separate thread or with async/await.
I'm running a batch from my ASP.NET c# page. I'm trying to use the System.Diagnostics.ProcessStartInfo method described in the link below. When I click my button to run the batch, the page just hangs with "Waiting for ".
The original code I'm using
Here are some similar problems with solutions:
Possible solution 1
Possible solution 2
Unfortunately, I don't understand solution 1 at all and solution 2 does seem to indicate there is a problem with the paths I'm trying to use, but I'm a little unclear on this as well. Any help is greatly appreciated.
My code:
protected void RunPkg_Click(object sender, EventArgs e)
{
// Get the full file path
string strFilePath = "C:\\inetpub\\wwwroot\\DecisionSupport\\CMSBenPerfUpload\\RunPackage.bat";
// Create the ProcessInfo object
System.Diagnostics.ProcessStartInfo psi = new System.Diagnostics.ProcessStartInfo("cmd.exe");
psi.UseShellExecute = false;
psi.RedirectStandardOutput = true;
psi.RedirectStandardInput = true;
psi.RedirectStandardError = true;
psi.WorkingDirectory = "C:\\inetpub\\wwwroot\\DecisionSupport\\CMSBenPerfUpload\\";
// Start the process
System.Diagnostics.Process proc = System.Diagnostics.Process.Start(psi);
// Open the batch file for reading
System.IO.StreamReader strm = System.IO.File.OpenText(strFilePath);
// Attach the output for reading
System.IO.StreamReader sOut = proc.StandardOutput;
// Attach the in for writing
System.IO.StreamWriter sIn = proc.StandardInput;
// Write each line of the batch file to standard input
while(strm.Peek() != -1)
{
sIn.WriteLine(strm.ReadLine());
}
strm.Close();
// Exit CMD.EXE
string stEchoFmt = "# {0} run successfully. Exiting";
sIn.WriteLine(String.Format(stEchoFmt, strFilePath));
sIn.WriteLine("EXIT");
// Close the process
proc.Close();
// Read the sOut to a string.
string results = sOut.ReadToEnd().Trim();
// Close the io Streams;
sIn.Close();
sOut.Close();
// Write out the results.
string fmtStdOut = "<font face=courier size=0>{0}</font>";
this.Response.Write(String.Format(fmtStdOut,results.Replace(System.Environment.NewLine, "<br>")));
UPDATE: I changed the paths to C:\Temp\ just to see if it made a difference, but it didn't. I opened my security wide open on the dirs I'm using, no go.
UPDATE 2: Running this on my dev box is the same, but I actually get feedback shown in the block below. If I manually perform this in cmd, it executes no problem. If I manually run the batch itself, there's no problem.
> Microsoft Windows [Version 6.1.7601]
>Copyright (c) 2009 Microsoft Corporation. All rights reserved.
>
>C:\inetpub\wwwroot\DecisionSupport\CMSBenPerfUpload>dtexec /f "ACO-SHS-PatDB.dtsx"
>Microsoft (R) SQL Server Execute Package Utility
>Version 10.50.1600.1 for 32-bit
>Copyright (C) Microsoft Corporation 2010. All rights reserved.
>
>Option "#" is not valid.
>
>C:\inetpub\wwwroot\DecisionSupport\CMSBenPerfUpload>
Your process identity doesn't have right to the directory.
Another queston where do you plan to publish this. A sane administrator would prvent any website to run executables or batch file on the server
I'm going to put an answer in here because I hate leaving things unresolved. I never could get this to work exactly right in this form. I had a network admin check permissions just to make sure I wasn't overlooking something, and they couldn't find anything that made a difference either. All permissions seemed to be properly granted.
Somewhere between the version differences in my dev PC and the server for all parts, Windows, SQL Server, Visual Studio, somewhere in there the problem kept itself well-hidden. Kudos to Microsoft for making products that fail to work properly from one machine to the next without wasting far too many hours hunting for a resolution.
Either way, I ended up going a different route. I built very simple SSIS packages on my SQL 2012 server that could be executed from Stored Procedures built on the server. This kept permission and version issues at bay. I then used C# scripting to write out my XML results files to the directories needed. The one issue to overcome was that C# executescalar will not handle XML larger than 2033 characters so I had to use ExecuteXmlReader and loop through the results to built my string before writing it to file.
This problem, though ending differently than it began, is resolved.
I am getting an exception on my server side code, which is serving up a silverlight app,
Win32Exception - No such interface supported
Our server side C# code starts up a separate process for a short task because of a third party dll not being thread safe. So the error above occurs in part of the code like this,
Process process = new Process();
ProcessStartInfo processStartInfo = new ProcessStartInfo();
processStartInfo.CreateNoWindow = true;
processStartInfo.WindowStyle = ProcessWindowStyle.Hidden;
processStartInfo.FileName =
Path.Combine(AppDomain.CurrentDomain.BaseDirectory, "bin", "PreviewGenerator.exe");
process.StartInfo = processStartInfo;
process.Start(); // THIS IS WHERE THE EXCEPTION OCCURS
process.WaitForExit();
The PreviewGenerator.exe process does not start when it is not working, the exception occurs where the comment is above.
UPDATE:
I have run process monitor on the IIS server when the issue occurs. This shows that the w3wp process does this,
Thread Create
Access the file PreviewGenerator.exe
Hive unloaded (this is the registry)
Thread Exit
And it does this before calling the other process. If I compare this with a the process monitor log when it is working it does this,
Thread Create
Access the file PreviewGenerator.exe
Process Start
Does heaps of stuff with PreviewGenerator.exe including reading / writing / registry, etc.
Process Exit
Hive unloaded
Thread Exit
But process monitor does not show any information as to why the first case doesn't work.
Is there a way I can see why the thread exits prematurely?
Also I think this problem relates to when my server is being loaded up more, and much more memory is being used. How can I prove this?
I had a similar issue, I used
processStartInfo.UseShellExecute = false;
and that fixed it for me.
http://www.progtown.com/topic31343-process-start-processstartinfo-startinfo.html
I found the best thing to do was to create a separate app pool for my application in IIS and set an upper limit for the amount of RAM it could use. Also I found it useful to turn on the 'Generate Recycle Event Log Entry' items under the app pool settings.
You can then go to the system event log and filter out the items with a source of 'WAS' to understand what is going on in the app pools, when they are restarting and when they stop from being idle etc.
I think the main problem in our case is that the IIS box was running out of memory. Tuning the app pools and adding some extra RAM seems to have solved it.
Why this code runs perfectly on my development computer (win7 32bit) and on target server(2008r2 64bit) as console app. But when I try to run it as a web service on the target server it does nothing. No error, nothing.
If I remove
exitMsg = proc.StandardOutput.ReadToEnd();
then it fail with error:
System.InvalidOperationException:
Process must exit before requested
information can be determined.
[WebMethod]
public string GetRunningProcesses()
{
ProcessStartInfo pInfo = new ProcessStartInfo();
pInfo.FileName = #"E:\bin\PsList.exe";
pInfo.WindowStyle = ProcessWindowStyle.Hidden;
pInfo.CreateNoWindow = true;
pInfo.UseShellExecute = false;
pInfo.RedirectStandardOutput = true;
string exitMsg = "";
int exitCode = 1;
using (Process proc = Process.Start(pInfo))
{
exitMsg = proc.StandardOutput.ReadToEnd();
proc.WaitForExit(1000);
exitCode = proc.ExitCode;
}
return exitMsg;
}
I think there must be something about user under which code runs. As web service this code runs under asp.net user and this might couses the problems.
Please advice me how to fix this. Thank you very much.
RESOLVED
The problem was with EULA dialog, which poped up but it was invisble due to ProcessStartInfo settings.
When I run PsList.exe via CMD under account which is also used for Application pool for this web service, I get prompted for an EULA agreement and after that everthing works fine.
The strange thing is that I have "pInfo.Arguments = "/accepteula";" in my real code. This should prevent my probem, but it didn't and I don't know why.
If any of you knows why, please tell me.
Thank you very much for all the help. You are trully good peoples here.
I think your only problem is with:
proc.WaitForExit(1000);
Which instructs the program to wait for a second for the process to finish. On your machine, the process finishes fine. On another machine, though, it may take longer. Try changing to:
proc.WaitForExit();
Which will wait indefinitely for the program to exit.
You may also want to redirect the output of the Process to see if the programming is hanging or waiting for something else from you (or, in this case, your code).
In addition, the process may be hitting an error and writing a message to StandardError rather than StandardOutput. Try setting pInfo.RedirectStandardError = true; and reading that as well to see if there's anything you're missing.
The problem was with EULA dialog, which poped up but it was invisble due to ProcessStartInfo settings.
When I run PsList.exe via CMD under account which is also used for Application pool for this web service, I get prompted for an EULA agreement and after that everthing works fine.
The strange thing is that I have "pInfo.Arguments = "/accepteula";" in my real code. This should prevent my probem, but it didn't and I don't know why.
If any of you knows why, please tell me.
Thank you very much for all the help. You are trully good peoples here.
Try wrapping your business logic in a try / catch block that catches any exception and either writes it to the output or to a log file.
I am writing a program (Visual Studio 2010 in C# Windows Forms) which keeps track of multiple instances of the Remote Desktop Client (mstsc.exe - tested with Windows 7 version). I have been attempting to launch this program and grab its PID with the following code:
Process mstsc = Process.Start(mstscLocation, mstscConString);
int mstscProcessId = mstsc.Id;
DataRow row = openConn.NewRow();
row["RDP ID"] = mstscID;
openConn.Rows.Add(row);
This starts the client and returns an ID as it should. Problem is, if I try to terminate the PID with the following code, it fails to do so:
int rdpID = Convert.ToInt32(dgvOpenConnections.Rows[selectedIndex].Cells["RDP ID"].Value.ToString());
try
{
// kill off mstsc
Process mstsc = Process.GetProcessById(rdpID);
mstsc.Kill();
}
I have verified that the PID that is recorded from Process.Start is the same as is retrieved from the DataGridView (dgvOpenConnections) and placed into rpdID (try fails and hits catch as the original PID no longer exists). Furthermore, I have issued a "tasklist" at a command prompt after starting one instance of MSTSC.EXE and can verify that it changes PIDs (in this test, C# recorded 4288 but tasklist shows it running as 8172).
I cannot kill all MSTSC processes as I am trying to control more than one. Is there a way to trace down the second PID MSTSC appears to use? My guess is it either starts a second process and gets rid of the first or maybe this is a child process (although the PID that is return no longer exists after start).
How in C# can I ensure I have the right process ID to later monitor or kill a specific instance of the Remote Desktop Client?
This happens if you try to run mstsc from a 32-bit application in 64-bit Windows.
(Source: http://social.msdn.microsoft.com/Forums/en-US/wpf/thread/22c10140-a502-4aa1-98d3-3607b8b573e8/)
There are two versions of mstsc on a 64-bit Windows:
c:\windows\system32\mstsc.exe is a 64-bit version
c:\windows\syswow64\mstsc.exe is more or less a "redirect" that will open c:\windows\system32\mstsc.exe from a different process than your application.
I've had the same issue. My application started mstsc, the process immediately exited, and mstsc reappeared with a different parent process and different PID.
This happens because 64-bit Windows uses file system redirection to redirect calls to the 64-bit c:\windows\system32 executables to c:\windows\syswow64.
There are two solutions:
Recompile your application to 64-bit. Then your application will also use the 64-bit mstsc.
Disable file system redirection (see http://blog.tonycamilli.com/2005/07/disabling-wow64-file-system.html) and access the 64-bit mstsc from your 32-bit application.
I have only tried recompiling, and it worked. :-)
Edit:
If you don't want your users to use the right version (we're using ClickOnce deployment, so we'd rather send one link to everybody), here's a workaround:
If you're using an .RDP file for mstsc, just add a unique token to your file name. mstsc will be started with a command line like mstsc host_user_token.rdp.
Now, after you called Process.Start, do Process.WaitForExit with a short timeout (5s). If the process did not exit, you have the right object.
If the process did exit, do a little polling loop (100ms interval, 5s timeout) that checks for processes with your token:
var timeout = AppSettings.GetIntValue(
Constants.SettingsKeyProcessFinderTimeout, Constants.ProcessFinderTimeoutDefault);
int elapsedTime = 0;
Process process = null;
while (elapsedTime <= timeout)
{
process =
Process.GetProcessesByName("mstsc").FirstOrDefault(p => p.StartInfo.Arguments.Contains(guid));
Logger.TraceVerbose(
string.Format(
"Elapsed time: {0}; Found process with PID {1}", elapsedTime, process == null ? -1 : process.Id));
if (process != null)
{
break;
}
Thread.Sleep(SleepInterval);
elapsedTime += SleepInterval;
}
After that loop, if you still have process == null, there was some error (process was not found or never executed at all). If you have a Process reference, that's the "new" mstsc process.
I tried your example using Process Explorer, and I couldn't see a second or child process being created. From beginning to end the Remote Desktop Process was one and the same, and after being created I was able to kill the process using the same PID I saw in the beginning.