Develop Reference

How to Asynchronously Call Static Methods

I am trying to construct a simple class, which calls a reboot function depending on the machine type to be rebooted. The called methods refer to a library which contains public static methods. I want to asynchronously call these static methods using Task in order to call the reboot methods in parallel. Here is the code so far:
EDIT
Following the community's request, this is now a version of the same question, with the code below compiling. Please not that you need the Renci.SshNet lib, and also need to set references to it in your project.
// libs
using System.IO;
using System.Threading.Tasks;
using System.Collections.Generic;
using Renci.SshNet;
namespace ConsoleApp
{
class Program
{
// Simple Host class
public class CHost
{
public string IP;
public string HostType;
public CHost(string inType, string inIP)
{// constructor
this.IP = inIP;
this.HostType = inType;
}
}
// Call test function
static void Main(string[] args)
{
// Create a set of hosts
var HostList = new List<CHost>();
HostList.Add( new CHost("Machine1", "10.52.0.93"));
HostList.Add( new CHost("Machine1", "10.52.0.30"));
HostList.Add( new CHost("Machine2", "10.52.0.34"));
// Call async host reboot call
RebootMachines(HostList);
}
// Reboot method
public static async void RebootMachines(List<CHost> iHosts)
{
// Locals
var tasks = new List<Task>();
// Build list of Reboot calls - as a List of Tasks
foreach(var host in iHosts)
{
if (host.HostType == "Machine1")
{// machine type 1
var task = CallRestartMachine1(host.IP);
tasks.Add(task); // Add task to task list
}
else if (host.HostType == "Machine2")
{// machine type 2
var task = CallRestartMachine2(host.IP);
tasks.Add(task); // Add task to task list
}
}
// Run all tasks in task list in parallel
await Task.WhenAll(tasks);
}
// ASYNC METHODS until here
private static async Task CallRestartMachine1(string host)
{// helper method: reboot machines of type 1
// The compiler complains here (RebootByWritingAFile is a static method)
// Error: "This methods lacks await operators and will run synchronously..."
RebootByWritingAFile(#"D:\RebootMe.bm","reboot");
}
private static async Task CallRestartMachine2(string host)
{// helper method: reboot machines of type 2
// The compiler warns here (RebootByWritingAFile is a static method)
// Error: "This methods lacks await operators and will run synchronously..."
RebootByNetwork(host,"user","pwd");
}
// STATIC METHODS here, going forward
private static void RebootByWritingAFile(string inPath, string inText)
{// This method does a lot of checks using more static methods, but then only writes a file
try
{
File.WriteAllText(inPath, inText); // static m
}
catch
{
// do nothing for now
}
}
private static void RebootByNetwork(string host, string user, string pass)
{
// Locals
string rawASIC = "";
SshClient SSHclient;
SshCommand SSHcmd;
// Send reboot command to linux machine
try
{
SSHclient = new SshClient(host, 22, user, pass);
SSHclient.Connect();
SSHcmd = SSHclient.RunCommand("exec /sbin/reboot");
rawASIC = SSHcmd.Result.ToString();
SSHclient.Disconnect();
SSHclient.Dispose();
}
catch
{
// do nothing for now
}
}
}
}
My only problem with this setup so far is that the static methods are called immediately (sequentially) and not assigned to a task. For example the line
...
else if (host.HostType == "Machine2")
{// machine type 2
var task = CallRestartMachine2(host.IP);
tasks.Add(task); // Add task to task list
}
...
takes 20 seconds to execute if the host is unreachable. If 10 hosts are unreachable the sequential duration is 20*10 = 200 seconds.
I am aware of some seemingly similar questions such as
c# asynchronously call method
Asynchronous call with a static method in C# .NET 2.0
How to call a method asynchronously
Simple Async Await Example for Asynchronous Programming
However, the cited lambda expressions still leave me with the same compiler error ["This methods lacks await operators..."]. Also, I do not want to spawn explicit threads (new Thread(() => ...)) due to high overhead if restarting a large number of machine in a cluster.
I may need to reboot a large number of machines in a cluster. Hence my question: How can I change my construct in order to be able to call the above static methods in parallel?
EDIT
Thanks to the comments of #JohanP and #MickyD, I would like to elaborate that I have actually tried writing the async version of both static methods. However that sends me down a rabbit hole, where every time a static method is called within the async method I get the compiler warning that the call will be synchronous. Here is an example of how I tried to wrap the call to method as an async task, hoping to call the dependent methods in an async manner.
private static async Task CallRestartMachine1(string host)
{// helper method: reboot machines of type 1
// in this version, compiler underlines '=>' and states that
// method is still called synchronously
var test = await Task.Run(async () =>
{
RebootByWritingAFile(host);
});
}
Is there a way to wrap the static method call such that all static child methods don't all need to rewritten as async?
Thank you all in advance.

Your code has a strange mix of async with continuations and it won't even compile. You need to make it async all the way up. When you call RebootMachines(...) and that call can't be awaited, you can schedule continuations on that i.e. RebootMachines(...).ContinueWith(t=> Console.WriteLine('All Done'))
public static async Task RebootMachines(List<CHost> iHosts)
{
var tasks = new List<Task>();
// Build list of Reboot calls - as a List of Tasks
foreach(var host in iHosts)
{
if (host.HostType == "Machine1")
{// machine type 1
task = CallRestartMachine1(host.IP);
}
else if (host.HostType == "Machine2")
{// machine type 2
task = CallRestartMachine2(host.IP);
}
// Add task to task list - for subsequent parallel processing
tasks.Add(task);
}
// Run all tasks in task list in parallel
await Task.WhenAll(tasks);
}
private static async Task CallRestartMachine1(string host)
{// helper method: reboot machines of type 1
//RebootByWritingAFile is method that returns a Task, you need to await it
// that is why the compiler is warning you
await RebootByWritingAFile(host);
}
private static async Task CallRestartMachine2(string host)
{// helper method: reboot machines of type 2
await RebootByNetwork(host);
}

Everyone, thanks for your input and your help. I have played around with his over the last couple of days, and have come up with the following way to asynchronously call a static method:
// libs
using System.IO;
using System.Threading.Tasks;
using System.Collections.Generic;
using Renci.SshNet;
namespace ConsoleApp
{
class Program
{
// Simple Host class
public class CHost
{
public string IP;
public string HostType;
public CHost(string inType, string inIP)
{// constructor
this.IP = inIP;
this.HostType = inType;
}
}
// Call test function
static void Main(string[] args)
{
// Create a set of hosts
var HostList = new List<CHost>();
HostList.Add( new CHost("Machine1", "10.52.0.93"));
HostList.Add( new CHost("Machine1", "10.52.0.30"));
HostList.Add( new CHost("Machine2", "10.52.0.34"));
// Call async host reboot call
RebootMachines(HostList);
}
// Reboot method
public static async void RebootMachines(List<CHost> iHosts)
{
// Locals
var tasks = new List<Task>();
// Build list of Reboot calls - as a List of Tasks
foreach(var host in iHosts)
{
if (host.HostType == "Machine1")
{// machine type 1
var task = CallRestartMachine1(host.IP);
tasks.Add(task); // Add task to task list
}
else if (host.HostType == "Machine2")
{// machine type 2
var task = CallRestartMachine2(host.IP);
tasks.Add(task); // Add task to task list
}
}
// Run all tasks in task list in parallel
await Task.WhenAll(tasks);
}
// ASYNC METHODS until here
private static async Task CallRestartMachine1(string host)
{// helper method: reboot machines of type 1
await Task.Run(() =>
{
RebootByWritingAFile(#"D:\RebootMe.bm", "reboot");
});
}
private static async Task CallRestartMachine2(string host)
{// helper method: reboot machines of type 2
await Task.Run(() =>
{
RebootByNetwork(host, "user", "pwd");
});
}
// STATIC METHODS here, going forward
private static void RebootByWritingAFile(string inPath, string inText)
{// This method does a lot of checks using more static methods, but then only writes a file
try
{
File.WriteAllText(inPath, inText); // static m
}
catch
{
// do nothing for now
}
}
private static void RebootByNetwork(string host, string user, string pass)
{
// Locals
string rawASIC = "";
SshClient SSHclient;
SshCommand SSHcmd;
// Send reboot command to linux machine
try
{
SSHclient = new SshClient(host, 22, user, pass);
SSHclient.Connect();
SSHcmd = SSHclient.RunCommand("exec /sbin/reboot");
rawASIC = SSHcmd.Result.ToString();
SSHclient.Disconnect();
SSHclient.Dispose();
}
catch
{
// do nothing for now
}
}
}
}
This setup calls my static methods asynchronously. I hope this helps someone who was stuck with a similar problem. Thanks again for all your input.

I think you should reconsider the "high overhead" of threads: This overhead is IMHO negligable when compared to the network roundtrips and waiting times on every RPC call. I am pretty sure, that the resources needed to create N threads is marginal against the resources needed to run N networked requests (be they http[s], RPC or whatever).
My approach would be to queue the tasks into a threadsafe collection (ConcurrentQueue, ConcurrentBag and friends), then spawn a finite number of threads looping through those work items until the collection is empty and just end.
This would not only allow you to run the tasks in parallel, it would allow you to run them in a controlled parallel way.

Keep running specific number of task async

I'm currently working on a concurrent file downloader.
For that reason I want to parametrize the number of concurrent tasks. I don't want to wait for all the tasks to be completed but to keep the same number being runned.
In fact, this thread on star overflow gave me a proper clue, but I'm struggling making it async:
Keep running a specific number of tasks
Here is my code:
public async Task StartAsync()
{
var semaphore = new SemaphoreSlim(1, _concurrentTransfers);
var queueHasMessages = true;
while (queueHasMessages)
{
try {
await Task.Run(async () =>
{
await semaphore.WaitAsync();
await asyncStuff();
});
}
finally {
semaphore.Release();
};
}
}
But the code just get executed one at a time. I think that the await is blocking me for generating the desired amount of tasks, but I don't know how to avoid it while respecting the limit established by the semaphore.
If I add all the tasks to a list and make a whenall, the semaphore throws an exception since it has reached the max count.
Any suggestions?

It was brought to my attention that the struck-through solution will drop any exceptions that occur during execution. That's bad.
Here is a solution that will not drop exceptions:
Task.Run is a Factory Method for creating a Task. You can check yourself with the intellisense return value. You can assign the returned Task anywhere you like.
"await" is an operator that will wait until the task it operates on completes. You are able to use any Task with the await operator.
public static async Task RunTasksConcurrently()
{
IList<Task> tasks = new List<Task>();
for (int i = 1; i < 4; i++)
{
tasks.Add(RunNextTask());
}
foreach (var task in tasks) {
await task;
}
}
public static async Task RunNextTask()
{
while(true) {
await Task.Delay(500);
}
}
By adding the values of the Task we create to a list, we can await them later on in execution.
Previous Answer below
Edit: With the clarification I think I understand better.
Instead of running every task at once, you want to start 3 tasks, and as soon as a task is finished, run the next one.
I believe this can happen using the .ContinueWith(Action<Task>) method.
See if this gets closer to your intended solution.
public void SpawnInitialTasks()
{
for (int i = 0; i < 3; i++)
{
RunNextTask();
}
}
public void RunNextTask()
{
Task.Run(async () => await Task.Delay(500))
.ContinueWith(t => RunNextTask());
// Recurse here to keep running tasks whenever we finish one.
}
The idea is that we spawn 3 tasks right away, then whenever one finishes we spawn the next. If you need to keep data flowing between the tasks, you can use parameters:
RunNextTask(DataObject object)

You can do this easily the old-fashioned way without using await by using Parallel.ForEach(), which lets you specify the maximum number of concurrent threads to use.
For example:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
namespace Demo
{
class Program
{
public static void Main(string[] args)
{
IEnumerable<string> filenames = Enumerable.Range(1, 100).Select(x => x.ToString());
Parallel.ForEach(
filenames,
new ParallelOptions { MaxDegreeOfParallelism = 4},
download
);
}
static void download(string filepath)
{
Console.WriteLine("Downloading " + filepath);
Thread.Sleep(1000); // Simulate downloading time.
Console.WriteLine("Downloaded " + filepath);
}
}
}
If you run this and observe the output, you'll see that the "files" are being "downloaded" in batchs.
A better simulation is the change download() so that it takes a random amount of time to process each "file", like so:
static Random rng = new Random();
static void download(string filepath)
{
Console.WriteLine("Downloading " + filepath);
Thread.Sleep(500 + rng.Next(1000)); // Simulate random downloading time.
Console.WriteLine("Downloaded " + filepath);
}
Try that and see the difference in the output.
However, if you want a more modern way to do this, you could look into the Dataflow part of the TPL (Task Parallel Library) - this works well with async methods.
This is a lot more complicated to get to grips with, but it's a lot more powerful. You could use an ActionBlock to do it, but describing how to do that is a bit beyond the scope of an answer I could give here.
Have a look at this other answer on StackOverflow; it gives a brief example.
Also note that the TPL is not built in to .Net - you have to get it from NuGet.

How to correctly use BlockingCollection.GetConsumingEnumerable?

I'm trying to implement a producer/consumer pattern using BlockingCollection<T> so I've written up a simple console application to test it.
public class Program
{
public static void Main(string[] args)
{
var workQueue = new WorkQueue();
workQueue.StartProducingItems();
workQueue.StartProcessingItems();
while (true)
{
}
}
}
public class WorkQueue
{
private BlockingCollection<int> _queue;
private static Random _random = new Random();
public WorkQueue()
{
_queue = new BlockingCollection<int>();
// Prefill some items.
for (int i = 0; i < 100; i++)
{
//_queue.Add(_random.Next());
}
}
public void StartProducingItems()
{
Task.Run(() =>
{
_queue.Add(_random.Next()); // Should be adding items to the queue constantly, but instead adds one and then nothing else.
});
}
public void StartProcessingItems()
{
Task.Run(() =>
{
foreach (var item in _queue.GetConsumingEnumerable())
{
Console.WriteLine("Worker 1: " + item);
}
});
Task.Run(() =>
{
foreach (var item in _queue.GetConsumingEnumerable())
{
Console.WriteLine("Worker 2: " + item);
}
});
}
}
However there are 3 problems with my design:
I don't know the correct way of blocking/waiting in my Main method. Doing a simple empty while loop seems terribly inefficient and CPU usage wasting simply for the sake of making sure the application doesn't end.
There's also another problem with my design, in this simple application I have a producer that produces items indefinitely, and should never stop. In a real world setup, I'd want it to end eventually (e.g. ran out of files to process). In that case, how should I wait for it to finish in the Main method? Make StartProducingItems async and then await it?
Either the GetConsumingEnumerable or Add is not working as I expected. The producer should constantly adding items, but it adds one item and then never adds anymore. This one item is then processed by one of the consumers. Both consumers then block waiting for items to be added, but none are. I know of the Take method, but again spinning on Take in a while loop seems pretty wasteful and inefficient. There is a CompleteAdding method but that then does not allow anything else to ever be added and throws an exception if you try, so that is not suitable.
I know for sure that both consumers are in fact blocking and waiting for new items, as I can switch between threads during debugging:
EDIT:
I've made the changes suggested in one of the comments, but the Task.WhenAll still returns right away.
public Task StartProcessingItems()
{
var consumers = new List<Task>();
for (int i = 0; i < 2; i++)
{
consumers.Add(Task.Run(() =>
{
foreach (var item in _queue.GetConsumingEnumerable())
{
Console.WriteLine($"Worker {i}: " + item);
}
}));
}
return Task.WhenAll(consumers.ToList());
}

GetConsumingEnumerable() is blocking. If you want to add to the queue constantly, you should put the call to _queue.Add in a loop:
public void StartProducingItems()
{
Task.Run(() =>
{
while (true)
_queue.Add(_random.Next());
});
}
Regarding the Main() method you could call the Console.ReadLine() method to prevent the main thread from finishing before you have pressed a key:
public static void Main(string[] args)
{
var workQueue = new WorkQueue();
workQueue.StartProducingItems();
workQueue.StartProcessingItems();
Console.WriteLine("Press a key to terminate the application...");
Console.ReadLine();
}

c# Rate Limiting foreach iterations

Basically I'm trying to be able to rate limit the execution of iterations of a list.
I really like the idea of using RX as I can build off the top of it, and have a more elegant solution, but it wouldn't have to be done using RX.
I've formulated this with the help of many much smarter than I. My problem is that I'd like to be able to say someCollection.RateLimitedForEach(rate, function), and have it ultimately block until we're done processing... or have it be an async method.
The demo below the function, works in a console app, but if I close after the foreach, it immediately returns.
I'm just kind of at a loss whether this is fixable, or if I should go about it completely different
public static void RateLimitedForEach<T>(this List<T> list, double minumumDelay, Action<T> action)
{
list.ToObservable().Zip(Observable.Interval(TimeSpan.FromSeconds(minumumDelay)), (v, _) => v)
.Do(action).Subscribe();
}
//rate limits iteration of foreach... keep in mind this is not the same thing as just sleeping for a second
//between each iteration, this is saying at the start of the next iteration, if minimum delay time hasnt past, hold until it has
var maxRequestsPerMinute = 60;
requests.RateLimitedForeach(60/maxRequestsPerMinute,(request) => SendRequest(request));

but it wouldn't have to be done using RX
Here is how you can do it synchronously:
public static void RateLimitedForEach<T>(
this List<T> list,
double minumumDelay,
Action<T> action)
{
foreach (var item in list)
{
Stopwatch sw = Stopwatch.StartNew();
action(item);
double left = minumumDelay - sw.Elapsed.TotalSeconds;
if(left > 0)
Thread.Sleep(TimeSpan.FromSeconds(left));
}
}
And here is how you can do it asynchronously (only potential waits are asynchronous):
public static async Task RateLimitedForEachAsync<T>(
this List<T> list,
double minumumDelay,
Action<T> action)
{
foreach (var item in list)
{
Stopwatch sw = Stopwatch.StartNew();
action(item);
double left = minumumDelay - sw.Elapsed.TotalSeconds;
if (left > 0)
await Task.Delay(TimeSpan.FromSeconds(left));
}
}
Please note that you can change the asynchronous version to make the action it self asynchronous like this:
public static async Task RateLimitedForEachAsync<T>(
this List<T> list,
double minumumDelay,
Func<T,Task> async_task_func)
{
foreach (var item in list)
{
Stopwatch sw = Stopwatch.StartNew();
await async_task_func(item);
double left = minumumDelay - sw.Elapsed.TotalSeconds;
if (left > 0)
await Task.Delay(TimeSpan.FromSeconds(left));
}
}
This is helpful if the action you need to run on each item is asynchronous.
The last version can be used like this:
List<string> list = new List<string>();
list.Add("1");
list.Add("2");
var task = list.RateLimitedForEachAsync(1.0, async str =>
{
//Do something asynchronous here, e.g.:
await Task.Delay(500);
Console.WriteLine(DateTime.Now + ": " + str);
});
Now you should wait for task to finish. If this is the Main method, then you need to synchronously wait like this:
task.Wait();
On the other hand, if you are inside an asynchronous method, then you need to asynchronously wait like this:
await task;

Your code was just about perfect.
Try this instead:
public static void RateLimitedForEach<T>(this List<T> list, double minumumDelay, Action<T> action)
{
list
.ToObservable()
.Zip(Observable.Interval(TimeSpan.FromSeconds(minumumDelay)), (v, _) => v)
.Do(action)
.ToArray()
.Wait();
}

The concept that you need to get accross, is that the main thread is not waiting on your RateLimitedForEach call to complete. Also - on your console app - as soon as the main thread ends, the process ends.
What does that mean? It means that the process will end regardless of whatever or not the observer on RateLimitedForEach has finished executing.
Note: The user may still force the execution of you app to finish, and that is a good thing. You may use a form app if you want to be able to wait without hanging the UI, you may use a service if you don't want the user closing windows related to the process.
Using Task is a superios solution to what I present below.
Notice that when using Tasks on the console app, you still need to wait on the task to prevent the main thread to finish before RateLimitedForEach completed its job. Moving away from a console app is still advised.
If you insist in continuing using your code, you can tweak it for it to hang the calling thread until completion:
public static void RateLimitedForEach<T>
(
this List<T> list,
double minumumDelay,
Action<T> action
)
{
using (var waitHandle = new ManualResetEventSlim(false))
{
var mainObservable = list.ToObservable();
var intervalObservable = Observable.Interval(TimeSpan.FromSeconds(minumumDelay));
var zipObservable = mainObservable .Zip(intervalObservable, (v, _) => v);
zipObservable.Subscribe
(
action,
error => GC.KeepAlive(error), // Ingoring them, as you already were
() => waitHandle.Set() // <-- "Done signal"
);
waitHandle.Wait(); // <--- Waiting on the observer to complete
}
}

Does RX Throttle not do what you want?
https://msdn.microsoft.com/en-us/library/hh229400(v=vs.103).aspx

c# do the equivalent of restarting a Task with some parameter

The main idea here is to fetch some data from somewhere, when it's fetched start writing it, and then prepare the next batch of data to be written, while waiting for the previous write to be complete.
I know that a Task cannot be restarted or reused (nor should it be), although I am trying to find a way to do something like this :
//The "WriteTargetData" method should take the "data" variable
//created in the loop below as a parameter
//WriteData basically do a shedload of mongodb upserts in a separate thread,
//it takes approx. 20-30 secs to run
var task = new Task(() => WriteData(somedata));
//GetData also takes some time.
foreach (var data in queries.Select(GetData))
{
if (task.Status != TaskStatus.Running)
{
//start task with "data" as a parameter
//continue the loop to prepare the next batch of data to be written
}
else
{
//wait for task to be completed
//"restart" task
//continue the loop to prepare the next batch of data to be written
}
}
Any suggestion appreciated ! Thanks. I don't necessarily want to use Task, I just think it might be the way to go.

This may be over simplifying your requirements, but would simply "waiting" for the previous task to complete work for you? You can use Task.WaitAny and Task.WaitAll to wait for previous operations to complete.
pseudo code:
// Method that makes calls to fetch and write data.
public async Task DoStuff()
{
Task currTask = null;
object somedata = await FetchData();
while (somedata != null)
{
// Wait for previous task.
if (currTask != null)
Task.WaitAny(currTask);
currTask = WriteData(somedata);
somedata = await FetchData();
}
}
// Whatever method fetches data.
public Task<object> FetchData()
{
var data = new object();
return Task.FromResult(data);
}
// Whatever method writes data.
public Task WriteData(object somedata)
{
return Task.Factory.StartNew(() => { /* write data */});
}

The Task class is not designed to be restarted. so you Need to create a new task and run the body with the same Parameters. Next i do not see where you start the task with the WriteData function in its body. That will property Eliminate the call of if (task.Status != TaskStatus.Running) There are AFAIK only the class Task and Thread where task is only the abstraction of an action that will be scheduled with the TaskScheduler and executed in different threads ( when we talking about the Common task Scheduler, the one you get when you call TaskFactory.Scheduler ) and the Number of the Threads are equal to the number of Processor Cores.
To you Business App. Why do you wait for the execution of WriteData? Would it be not a lot more easy to gater all data and than submit them into one big Write?

something like ?
public void Do()
{
var task = StartTask(500);
var array = new[] {1000, 2000, 3000};
foreach (var data in array)
{
if (task.IsCompleted)
{
task = StartTask(data);
}
else
{
task.Wait();
task = StartTask(data);
}
}
}
private Task StartTask(int data)
{
var task = new Task(DoSmth, data);
task.Start();
return task;
}
private void DoSmth(object time)
{
Thread.Sleep((int) time);
}

You can use a thread and an AutoResetEvent. I have code like this for several different threads in my program:
These are variable declarations that belong to the main program.
public AutoResetEvent StartTask = new AutoResetEvent(false);
public bool IsStopping = false;
public Thread RepeatingTaskThread;
Somewhere in your initialization code:
RepeatingTaskThread = new Thread( new ThreadStart( RepeatingTaskProcessor ) ) { IsBackground = true; };
RepeatingTaskThread.Start();
Then the method that runs the repeating task would look something like this:
private void RepeatingTaskProcessor() {
// Keep looping until the program is going down.
while (!IsStopping) {
// Wait to receive notification that there's something to process.
StartTask.WaitOne();
// Exit if the program is stopping now.
if (IsStopping) return;
// Execute your task
PerformTask();
}
}
If there are several different tasks you want to run, you can add a variable that would indicate which one to process and modify the logic in PerformTask to pick which one to run.
I know that it doesn't use the Task class, but there's more than one way to skin a cat & this will work.