I'm implementing a C# application. I need to execute a program on multiple remote machines at the same time. To do so I'm using PSExec over a CMD with multithreading. Basically, for each machine, I start a thread that starts a CMD process. Depending on the result of the program executed remotely I'd like to either take an action or kill it if it takes more than x minutes (hope that makes sense).
The issue I've got is that I don't really know how to control for how long the process has been running other than using WaitForExit and, that doesn't really let me go multi-threading as it waits till the CMD call has finished.
I'm sure there must be a way of doing this but I cannot really figure it out. Could anyone please help me?
Here is my code (I'm new at c# coding so might not be the best code, feel free to correct any part of it you consider it is not right):
public async void BulkExecution()
{
//Some code
foreach (string machine in Machines)
{
//more code to work out the CMDline and other duties.
var result = Task.Factory.StartNew(r => ExecutePsexec((string)r, RunBeforeKillMsec), CMDLine);
await result;
}
//More Code
}
private static void ExecutePsexec(string CMDline, int RunBeforeKillMsec)
{
Process compiler = new Process();
compiler.StartInfo.FileName = "psexec.exe";
compiler.StartInfo.Arguments = CMDline;
compiler.StartInfo.UseShellExecute = false;
compiler.StartInfo.RedirectStandardOutput = true;
compiler.Start();
if (!compiler.WaitForExit(RunBeforeKillMsec))
{
ExecutePSKill(CMDline);
}
else
{
//Some Actions here
Common.Log(LogFile, CMDline.Split(' ')[0] + " finished successfully");
}
}
ExecutePsexec runs in a separate task. All such tasks are independent. await result; is what sequences them. You need to remove it.
Async void methods should be avoided. You should change the signature of the BulkExecution method to return a Task, so that you can await it and handle any exceptions that may occur. Inside the method create a Task for each machine, and then await all tasks with the Task.WhenAll method:
public async Task BulkExecution()
{
//Some code
Task[] tasks = Machines.Select(machine =>
{
//more code to work out the CMDline and other duties.
return Task.Run(r => ExecutePsexec(CMDLine, ExecutionTimeoutMsec));
}).ToArray();
await Task.WhenAll(tasks);
//More Code
}
Related
How do I make below simple method async so that i can it call it like await DoSomething
public void DoDomething()
{
string d = "doing something";
}
Editing Question for what I am actually doing in my method
public void RunValidationScripts()
{
string scriptDirPath = #"D:\ValidationScripts";
string[] psScriptsPath = Directory.GetFiles(scriptDirPath);
Dictionary<string, Collection<PSObject>> result = new Dictionary<string, Collection<PSObject>>();
foreach (string scriptPath in psScriptsPath)
{
result.Add(scriptPath, ExecutePSScript(scriptPath));
}
}
I am using package Microsoft.PowerShell.SDK
private Collection<PSObject> ExecutePSScript(string scriptFilePath)
{
using (var ps = PowerShell.Create())
{
string fileName = Path.GetFileName(scriptFilePath);
var results = ps.AddScript(File.ReadAllText(scriptFilePath)).Invoke();
return results;
}
}
The question is unclear. It looks like the actual problem is trying to execute PowerShell scripts asynchronously. Nothing will be gained by using Task.Run in this case. PowerShell scripts execute on a different thread already.
Powershell.InvokeAsync can be used to execute a PowerShell script asynchronously. Directory.EnumerateFiles can be used to start enumerating files and processing them without waiting for all files to be retrieved.
Assuming the method executing the scripts looks something like this:
async Task<IEnumerable<PSObject>> ExecutePSScript(string scriptPath)
{
var ps = PowerShell.Create();
var content=await File.ReadAllTextAsync(scriptPath);
ps.AddScript(content);
var results=await ps.InvokeAsync();
return results;
}
The calling method can be :
async Task RunValidationScriptsAsync(string folder)
{
var allResults=new List<PSObject>();
var files=Directory.EnumerateFiles(folder);
foreach(var file in files)
{
var results=await ExecutePSScript(file);
allResults.AddRange(results);
}
//Do something with the results
}
From a developer's perspective, there is one simple pratical rule of thumb that applies to most cases.
Invoking asynchronous APIs
The idea is that if you can choose between a synchronous and asynchronous API in the framework, you should opt for making all your code asynchronous and write your methods asynchronously, so that invocations to the aforementioned async framework methods is awaited.
If you don't utilize async methods, you should write synchronous methods as async does not provide any benefit
Why would you make simple method async, If there is nothing to wait for (IO bound or CPU bound operation). But if there is really a need (fire and forget kind of situation), You could do,
public async Task DoDomething()
{
string d=null;
await Task.Run(()=> {
d = "doing something";
}
}
Having an async is not important if you dont want to wait here, so,
public Task DoDomething()
{
string d=null;
Task.Run(()=> {
d = "doing something";
}
}
will work too.
Note: the question detail has changed significantly, and answer might not be relevant for this.
I am testing running python via Process.Start in parallel
My machine has a 2.8GHz CPU with 4 cores and 8 logical processors
My main console application is as below
static void Main(string[] args) => MainAsync(args).GetAwaiter().GetResult();
static async Task MainAsync(string[] args)
{
var startTime = DateTime.UtcNow;
Console.WriteLine($"Execution started at {DateTime.UtcNow:T}");
await ExecuteInParallelAsync(args).ConfigureAwait(false);
Console.WriteLine($"Executions completed at {DateTime.UtcNow:T}");
var endTime = DateTime.UtcNow;
var duration = (endTime - startTime);
Console.WriteLine($"Execution took {duration.TotalMilliseconds} milliseconds {duration.TotalSeconds} seconds");
Console.WriteLine("Press Any Key to close");
Console.ReadKey();
}
Where ExecuteInParallelAsync is the method that does the work...
private static async Task ExecuteInParallelAsync(string[] args)
{
var executionNumbers = new List<int>();
var executions = 5;
for (var executionNumber = 1; executionNumber <= executions; executionNumber++)
{
executionNumbers.Add(executionNumber);
}
await executionNumbers.ParallelForEachAsync(async executionNumber =>
{
Console.WriteLine($"Execution {executionNumber} of {executions} {DateTime.UtcNow:T}");
ExecuteSampleModel();
Console.WriteLine($"Execution {executionNumber} complete {DateTime.UtcNow:T}");
}).ConfigureAwait(false);
}
ExecuteSampleModel runs the Python model...
IModelResponse GetResponse()
{
_actualResponse = new ModelResponse();
var fileName = $#"main.py";
var p = new Process();
p.StartInfo = new ProcessStartInfo(#"C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\python.exe", fileName)
{
WorkingDirectory = RootFolder,
RedirectStandardOutput = true,
UseShellExecute = false,
CreateNoWindow = true
};
p.Start();
_actualResponse.RawResponseFromModel = p.StandardOutput.ReadToEnd();
p.WaitForExit();
return _actualResponse;
}
As you can see, I am asking this model to be executed 5 times
When I use the debugger it appears as though even though I am using ParalellForEach (introduced by the AsyncEnumerator package) this is not being run in parallel
I thought that each iteration is run on its own thread?
Each Python Model execution takes 5 seconds.
Running in parallel I would expect the whole process to be done in 15 seconds or so but it actually takes 34 seconds
The Console.WriteLines added before and after the call to GetResponse show that the first call is starting, being executed in full, then the second is starting, etc
Is this something to do with me calling Process.Start?
Can anyone see anything wrong with this?
Paul
To make the answer useful here is explanation what happened with async code. Omitting lot of details which aren't so important from standpoint of explanation the code inside ParallelForEachAsync loop looks like as follows:
// some preparations
...
var itemIndex = 0L;
while (await enumerator.MoveNextAsync(cancellationToken).ConfigureAwait(false))
{
...
Task itemActionTask = null;
try
{
itemActionTask = asyncItemAction(enumerator.Current, itemIndex);
}
catch (Exception ex)
{
// some exception handling
}
...
itemIndex++;
}
where asyncItemAction has type Func<T, long, Task> and it's a wrapper around custom asynchronous action with type Func<T, Task> which is passed as parameter to the ParallelForEachAsync call (the wrapper adds indexing functionality). The loop code just calls this action in order to obtain a task which would represent the asynchronous operation promise to wait for its completion. In case of given code example the custom action
async executionNumber =>
{
Console.WriteLine($"Execution {executionNumber} of {executions}{DateTime.UtcNow:T}");
ExecuteSampleModel();
Console.WriteLine($"Execution {executionNumber} complete {DateTime.UtcNow:T}");
}
contains no asynchronous code but prefix async allows compiler to generate state machine with method which returns some Task which makes this code compliant (from the syntax standpoint) with custom action call inside the loop.
The important thing that code inside the loop expects this operation to be asynchronous which implies that the operation implicitly is split into synchronous part which will be executed along with asyncItemAction(enumerator.Current, itemIndex) call and at least one (one or more depending on number of awaits inside) asynchronous parts which can be executed during iterating over other loop items. The following pseudo-code gives an idea of that:
{
... synchronous part
await SomeAsyncOperation();
... asynchronous part
}
In this particular case there is no async part in the custom action at all which consequently means that the call
itemActionTask = asyncItemAction(enumerator.Current, itemIndex);
will be executed synchronously and the next iteration inside the loop won't start until asyncItemAction completes the entire custom action execution.
That's why the switching off of the asynchrony in the code and using simple parallelism helps.
I am using Async await with Task.Factory method.
public async Task<JobDto> ProcessJob(JobDto jobTask)
{
try
{
var T = Task.Factory.StartNew(() =>
{
JobWorker jobWorker = new JobWorker();
jobWorker.Execute(jobTask);
});
await T;
}
This method I am calling inside a loop like this
for(int i=0; i < jobList.Count(); i++)
{
tasks[i] = ProcessJob(jobList[i]);
}
What I notice is that new tasks opens up inside Process explorer and they also start working (based on log file). however out of 10 sometimes 8 or sometimes 7 finishes. Rest of them just never come back.
why would that be happening ?
Are they timing out ? Where can I set timeout for my tasks ?
UPDATE
Basically above, I would like each Task to start running as soon as they are called and wait for the response on AWAIT T keyword. I am assuming here that once they finish each of them will come back at Await T and do the next action. I am alraedy seeing this result for 7 out of 10 tasks but 3 of them are not coming back.
Thanks
It is hard to say what the issues is without the rest if the code, but you code can be simplified by making ProcessJob synchronous and then calling Task.Run with it.
public JobDto ProcessJob(JobDto jobTask)
{
JobWorker jobWorker = new JobWorker();
return jobWorker.Execute(jobTask);
}
Start tasks and wait for all tasks to finish. Prefer using Task.Run rather than Task.Factory.StartNew as it provides more favourable defaults for pushing work to the background. See here.
for(int i=0; i < jobList.Count(); i++)
{
tasks[i] = Task.Run(() => ProcessJob(jobList[i]));
}
try
{
await Task.WhenAll(tasks);
}
catch(Exception ex)
{
// handle exception
}
First, let's make a reproducible version of your code. This is NOT the best way to achieve what you are doing, but to show you what is happening in your code!
I'll keep the code almost same as your code, except I'll use simple int rather than your JobDto and on completion of the job Execute() I'll write in a file that we can verify later. Here's the code
public class SomeMainClass
{
public void StartProcessing()
{
var jobList = Enumerable.Range(1, 10).ToArray();
var tasks = new Task[10];
//[1] start 10 jobs, one-by-one
for (int i = 0; i < jobList.Count(); i++)
{
tasks[i] = ProcessJob(jobList[i]);
}
//[4] here we have 10 awaitable Task in tasks
//[5] do all other unrelated operations
Thread.Sleep(1500); //assume it works for 1.5 sec
// Task.WaitAll(tasks); //[6] wait for tasks to complete
// The PROCESS IS COMPLETE here
}
public async Task ProcessJob(int jobTask)
{
try
{
//[2] start job in a ThreadPool, Background thread
var T = Task.Factory.StartNew(() =>
{
JobWorker jobWorker = new JobWorker();
jobWorker.Execute(jobTask);
});
//[3] await here will keep context of calling thread
await T; //... and release the calling thread
}
catch (Exception) { /*handle*/ }
}
}
public class JobWorker
{
static object locker = new object();
const string _file = #"C:\YourDirectory\out.txt";
public void Execute(int jobTask) //on complete, writes in file
{
Thread.Sleep(500); //let's assume does something for 0.5 sec
lock(locker)
{
File.AppendAllText(_file,
Environment.NewLine + "Writing the value-" + jobTask);
}
}
}
After running just the StartProcessing(), this is what I get in the file
Writing the value-4
Writing the value-2
Writing the value-3
Writing the value-1
Writing the value-6
Writing the value-7
Writing the value-8
Writing the value-5
So, 8/10 jobs has completed. Obviously, every time you run this, the number and order might change. But, the point is, all the jobs did not complete!
Now, if I un-comment the step [6] Task.WaitAll(tasks);, this is what I get in my file
Writing the value-2
Writing the value-3
Writing the value-4
Writing the value-1
Writing the value-5
Writing the value-7
Writing the value-8
Writing the value-6
Writing the value-9
Writing the value-10
So, all my jobs completed here!
Why the code is behaving like this, is already explained in the code-comments. The main thing to note is, your tasks run in ThreadPool based Background threads. So, if you do not wait for them, they will be killed when the MAIN process ends and the main thread exits!!
If you still don't want to await the tasks there, you can return the list of tasks from this first method and await the tasks at the very end of the process, something like this
public Task[] StartProcessing()
{
...
for (int i = 0; i < jobList.Count(); i++)
{
tasks[i] = ProcessJob(jobList[i]);
}
...
return tasks;
}
//in the MAIN METHOD of your application/process
var tasks = new SomeMainClass().StartProcessing();
// do all other stuffs here, and just at the end of process
Task.WaitAll(tasks);
Hope this clears all confusion.
It's possible your code is swallowing exceptions. I would add a ContineWith call to the end of the part of the code that starts the new task. Something like this untested code:
var T = Task.Factory.StartNew(() =>
{
JobWorker jobWorker = new JobWorker();
jobWorker.Execute(jobTask);
}).ContinueWith(tsk =>
{
var flattenedException = tsk.Exception.Flatten();
Console.Log("Exception! " + flattenedException);
return true;
});
},TaskContinuationOptions.OnlyOnFaulted); //Only call if task is faulted
Another possibility is that something in one of the tasks is timing out (like you mentioned) or deadlocking. To track down whether a timeout (or maybe deadlock) is the root cause, you could add some timeout logic (as described in this SO answer):
int timeout = 1000; //set to something much greater than the time it should take your task to complete (at least for testing)
var task = TheMethodWhichWrapsYourAsyncLogic(cancellationToken);
if (await Task.WhenAny(task, Task.Delay(timeout, cancellationToken)) == task)
{
// Task completed within timeout.
// Consider that the task may have faulted or been canceled.
// We re-await the task so that any exceptions/cancellation is rethrown.
await task;
}
else
{
// timeout/cancellation logic
}
Check out the documentation on exception handling in the TPL on MSDN.
I have this method which I would like to run asynchronously so that I can do other things while it runs. It does not rely on any other Async method (it doesn't call out to another resource, download a file or anything). I would like to avoid using new Task(), Task.Factory.StartTask() and Task.Run(), if possible.
Is it possible to run this method asynchronously, with tidy, readable code and without using Task explicitly?
If not, what is the tidiest way of running the method asynchronously?
Note: Please don't be concerned with the silly logic in the method - I have boiled it down to be deliberately slow but not show my actual code.
public static void main(string[] args)
{
RunMySlowLogic();
}
private void RunMySlowLogic()
{
while (true)
for (int i=0; i<100000000;i++)
if (i == new Random().Next(999))
return true;
}
Currently, I believe that I would need to wrap the method in a lambda or Task and mark it async. Where would the await go?
You're confusing two different things. You can run this in the background, and this method can be asynchronous. These are 2 different things and your method can do either, or both.
If you do something asynchronous in that method, like Task.Delay or some non-blocking I/O then call that method, await the returned task and make the method itself async:
async Task RunMySlowLogicAsync()
{
while (true)
{
// ...
await Task.Delay(1000);
}
}
If you don't have such a thing then your method isn't asynchronous, it's synchronous. You can still run it in the background on a different (ThreadPool) thread while you do other things using Task.Run:
var task = Task.Run(() => RunMySlowLogic());
There are multiple ways of executing code asynchronously in the .NET environment. Have a look at the Asynchronous Programming Patterns MSDN article.
Tasks are to make your job easier. I think the only valid reason to avoid using tasks is when you are targeting an older version of .NET.
So without Tasks, you can start a thread yourself, or use a ThreadPool (Tasks do this internally).
public static void main(string[] args)
{
var are = new AutoResetEvent(false);
ThreadPool.QueueUserWorkItem(RunMySlowLogicWrapped, are);
// Do some other work here
are.WaitOne();
}
// you have to match the signature of WaitCallback delegate, we can use it to communicate cross-thread
private void RunMySlowLogicWrapped(Object state) {
AutoResetEvent are = (AutoResetEvent) state;
RunMySlowLogic();
are.Set();
}
private bool RunMySlowLogic()
{
while (true)
for (int i=0; i<100000000;i++)
if (i == new Random().Next(999))
return true;
}
The main idea here is to fetch some data from somewhere, when it's fetched start writing it, and then prepare the next batch of data to be written, while waiting for the previous write to be complete.
I know that a Task cannot be restarted or reused (nor should it be), although I am trying to find a way to do something like this :
//The "WriteTargetData" method should take the "data" variable
//created in the loop below as a parameter
//WriteData basically do a shedload of mongodb upserts in a separate thread,
//it takes approx. 20-30 secs to run
var task = new Task(() => WriteData(somedata));
//GetData also takes some time.
foreach (var data in queries.Select(GetData))
{
if (task.Status != TaskStatus.Running)
{
//start task with "data" as a parameter
//continue the loop to prepare the next batch of data to be written
}
else
{
//wait for task to be completed
//"restart" task
//continue the loop to prepare the next batch of data to be written
}
}
Any suggestion appreciated ! Thanks. I don't necessarily want to use Task, I just think it might be the way to go.
This may be over simplifying your requirements, but would simply "waiting" for the previous task to complete work for you? You can use Task.WaitAny and Task.WaitAll to wait for previous operations to complete.
pseudo code:
// Method that makes calls to fetch and write data.
public async Task DoStuff()
{
Task currTask = null;
object somedata = await FetchData();
while (somedata != null)
{
// Wait for previous task.
if (currTask != null)
Task.WaitAny(currTask);
currTask = WriteData(somedata);
somedata = await FetchData();
}
}
// Whatever method fetches data.
public Task<object> FetchData()
{
var data = new object();
return Task.FromResult(data);
}
// Whatever method writes data.
public Task WriteData(object somedata)
{
return Task.Factory.StartNew(() => { /* write data */});
}
The Task class is not designed to be restarted. so you Need to create a new task and run the body with the same Parameters. Next i do not see where you start the task with the WriteData function in its body. That will property Eliminate the call of if (task.Status != TaskStatus.Running) There are AFAIK only the class Task and Thread where task is only the abstraction of an action that will be scheduled with the TaskScheduler and executed in different threads ( when we talking about the Common task Scheduler, the one you get when you call TaskFactory.Scheduler ) and the Number of the Threads are equal to the number of Processor Cores.
To you Business App. Why do you wait for the execution of WriteData? Would it be not a lot more easy to gater all data and than submit them into one big Write?
something like ?
public void Do()
{
var task = StartTask(500);
var array = new[] {1000, 2000, 3000};
foreach (var data in array)
{
if (task.IsCompleted)
{
task = StartTask(data);
}
else
{
task.Wait();
task = StartTask(data);
}
}
}
private Task StartTask(int data)
{
var task = new Task(DoSmth, data);
task.Start();
return task;
}
private void DoSmth(object time)
{
Thread.Sleep((int) time);
}
You can use a thread and an AutoResetEvent. I have code like this for several different threads in my program:
These are variable declarations that belong to the main program.
public AutoResetEvent StartTask = new AutoResetEvent(false);
public bool IsStopping = false;
public Thread RepeatingTaskThread;
Somewhere in your initialization code:
RepeatingTaskThread = new Thread( new ThreadStart( RepeatingTaskProcessor ) ) { IsBackground = true; };
RepeatingTaskThread.Start();
Then the method that runs the repeating task would look something like this:
private void RepeatingTaskProcessor() {
// Keep looping until the program is going down.
while (!IsStopping) {
// Wait to receive notification that there's something to process.
StartTask.WaitOne();
// Exit if the program is stopping now.
if (IsStopping) return;
// Execute your task
PerformTask();
}
}
If there are several different tasks you want to run, you can add a variable that would indicate which one to process and modify the logic in PerformTask to pick which one to run.
I know that it doesn't use the Task class, but there's more than one way to skin a cat & this will work.