Develop Reference

Asynchronously download and compile list of JsonDocument

I'm a little new (returning after a couple of decades) to C# and to the async/await model of programming. Looking for a little guidance, since I received an understandable warning CS1998 that the asynchronous method lacks await and operators and will run synchronously.
The code below I think is straightforward - the server API returns data in pages of 25 items. I'm using a continuation to add each page of 25 to a List of JsonDocuments. Calling code will handle the parsing as needed. I'm not sure how I could reasonably leverage anything further in this, but looking for any suggestions/guidance.
internal static async Task<List<JsonDocument>> Get_All_Data(HttpClient client, string endpoint)
{
Console.WriteLine("Downloading all data from {0}{1}", client.BaseAddress, endpoint);
var all_pages = new List<JsonDocument>();
// Get first page to determine total number of pages
HttpResponseMessage response = client.GetAsync(endpoint).Result;
Console.WriteLine("Initial download complete - parsing headers to determine total pages");
//int items_per_page;
if (int.TryParse(Get_Header_Value("X-Per-Page", response.Headers), out int items_per_page) == false)
// throw new Exception("Response missing X-Per-Page in header");
items_per_page = 25;
if (int.TryParse(Get_Header_Value("X-Total-Count", response.Headers), out int total_items) == false)
//throw new Exception("Response missing X-Total-Count in header");
total_items = 1;
// Divsion returns number of complete pages, add 1 for partial IF total items_json is not an exact multiple of items_per_page
var total_pages = total_items / items_per_page;
if ((total_items % items_per_page) != 0) total_pages++;
Console.WriteLine("{0} pages to be downloaded", total_pages);
var http_tasks = new Task[total_pages];
for (int i = 1; i <= total_pages; i++)
{
Console.WriteLine("Downloading page {0}", i);
var paged_endpoint = endpoint + "?page=" + i;
response = client.GetAsync(paged_endpoint).Result;
http_tasks[i - 1] = response.Content.ReadAsStringAsync().ContinueWith((_content) => { all_pages.Add(JsonDocument.Parse(_content.Result)); }); ;
//http_tasks[i].ContinueWith((_content) => { all_pages.Add(JsonDocument.Parse_List(_content.Result)); });
}
System.Threading.Tasks.Task.WaitAll(http_tasks); // wait for all of the downloads and parsing to complete
return all_pages;
}
Thanks for your help

My suggestion is to await all asynchronous operations, and use the Parallel.ForEachAsync method to parallelize the downloading of the JSON documents, while maintaining control of the degree of parallelism:
static async Task<JsonDocument[]> GetAllData(HttpClient client, string endpoint)
{
HttpResponseMessage response = await client.GetAsync(endpoint);
response.EnsureSuccessStatusCode();
if (!Int32.TryParse(GetHeaderValue(response, "X-Total-Count"),
out int totalItems) || totalItems < 0)
totalItems = 1;
if (!Int32.TryParse(GetHeaderValue(response, "X-Per-Page"),
out int itemsPerPage) || itemsPerPage < 1)
itemsPerPage = 25;
int totalPages = ((totalItems - 1) / itemsPerPage) + 1;
JsonDocument[] results = new JsonDocument[totalPages];
ParallelOptions options = new() { MaxDegreeOfParallelism = 5 };
await Parallel.ForEachAsync(Enumerable.Range(1, totalPages), options,
async (page, ct) =>
{
string pageEndpoint = endpoint + "?page=" + page;
HttpResponseMessage pageResponse = await client
.GetAsync(pageEndpoint, ct);
pageResponse.EnsureSuccessStatusCode();
string pageContent = await response.Content.ReadAsStringAsync(ct);
JsonDocument result = JsonDocument.Parse(pageContent);
results[page - 1] = result;
});
return results;
}
static string GetHeaderValue(HttpResponseMessage response, string name)
=> response.Headers.TryGetValues(name, out var values) ?
values.FirstOrDefault() : null;
The MaxDegreeOfParallelism is configured to the value 5 for demonstration purposes. You can find the optimal degree of parallelism by experimenting with your API. Setting the value too low might result in mediocre performance. Setting the value too high might overburden the target server, and potentially trigger an anti-DoS-attack mechanism.
If you are not familiar with the Enumerable.Range, it is a LINQ method that returns an incremented numeric sequence of integers that starts from start, and contains count elements.
The GetAllData is an asynchronous method and it is supposed to be awaited. If you are calling it without await, and your application is a UI application like WinForms or WPF, you are at risk of experiencing a deadlock. Don't panic, it happens consistently, and you'll observe it during the testing. One way to prevent it is to append .ConfigureAwait(false) to all awaited operations inside the GetAllData method.

Interrupt calling thread when exception in child task

I have the following scenario:
I have a plugin class (unfortunately synchronous and that's out of my control) which runs automation scripts. At startup, a task is started which runs in the background continuously until the user stops it. As this task runs, it is parsing streaming incoming data from a serial port and continuously updating a list of responses. In parallel, the script will be changing some other values and monitoring how the responses react.
My problem is this -- it is possible that the parser could throw an exception and that renders any of the remaining script invalid. But, I can't find a good way to interrupt the script (i.e. void Main() below) should this parallel task throw. If I wait until my script is done, and then await the task, I do get the exception, but by then there is potentially a lot of time wasted. So I'm looking for better ways to interrupt the script that is running should the parallel task error. I've tried posting on the initial synchronization context but with no luck as well. I've parred down an example below to simulate a similar scenario
public class Test : TestPlugin
{
Task _t;
List<string> data = new List();
public Test(){ }
public override void Cleanup()
{
//_t.Wait();
}
public override void Main()
{
// i want this to be interrupted if the polling task started in startup throws.
//Right here simulates running test steps,
//any of which can access the data that's been updated by the "background" task at any time
try
{
int i = 0;
while (i < 10)
{
Task.Delay(1000).Wait();
Console.WriteLine("step" + i + "Latest: " + data.latest());
i++;
}
}
catch(Exception ex)
{
Console.WriteLine("EXCEPTION");
}
}
public override void Setup()
{
_t = Task.Run(async () =>
{
int i =0;
while (i < 10)
{
await Task.Delay(200);
Console.WriteLine(i);
i++;
//i would parse incoming data and add to array here
data.add(i);
if (i > 3) throw new Exception("UH OH");
}
});
}
}

You could investigate the Task.ContinueWith method on the calling task code.
this ContinueWith allows for a delegate callback that gets triggered on completion and/or exceptions. so perhaps within the ContinueWith delegate, do a check if the task completes successfully, if not then raise a new exception there appending the thrown exception passed to the delegate.
You can use the code below as an example:
Task t1 = factory.StartNew(() =>
{
DoSomething();
}).ContinueWith((t2) =>
{
if (t2.IsCanceled)
DoSomethingWhenCancelled();
else if (t2.IsFaulted)
DoSomethingOnError(t1.Exception);
else
DoSomethingWhenComplete();
});

I think I am going to have to end up with something like the following. Unfortunately this forces developers of these test procedures to need to wrap all test steps that execute in parallel to the serial port reader with a task, but maybe it's possible for me to abstract away that implementation detail into a class with an API that manages this a bit better for them.
public override void Main()
{
Task steps = Task.Run(async () =>
{
int i = 1;
//this represents step 1
await Task.Delay(5000);
i++;
Console.WriteLine("Step " + i);
//this represents step 2
await Task.Delay(5000);
i++;
Console.WriteLine("Step " + i);
//this represents step 3
await Task.Delay(5000);
i++;
Console.WriteLine("Step " + i);
//this represents step 4
await Task.Delay(5000);
i++;
Console.WriteLine("Step " + i);
});
Task.WaitAny(_t, steps);
}

C# Async Tasks wait vs await

There is something that is bugging me for a while, something that happen in my code I can't understand.
I've defined a workflow to extract information from the Facebook API, basically I have 3 different async tasks.
Execute Reports
private static async Task<List<DataTypes.ReportRun>> ExecuteMarketingReports(FacebookClient fb, List<DataTypes.Request.Batch> batch_requests)
{
//Do stuff - Make async POST requests to
//FB to execute reports on FB server side
}
Monitorize Reports
private static async Task<List<DataTypes.AdReportRun>> MonitorizeMarketingReports(FacebookClient fb, List<DataTypes.ReportRun> tListReportRun)
{
//Do stuff -- Check if the reports are ready or not, and return a list with status
}
GetReportData
private static async Task GetReportData(FacebookClient fb, List<DataTypes.AdReportRun> tListReportRun, DateTime since, DateTime until, string breakdown)
{
//Do stuff - Gets report thata once the reports are finish and create the files
}
This is the main Task where all the other methods are called
private static async Task PullInsightsData(FacebookClient fb, List<DataTypes.Request.Batch> batchRequests, DateTime since, DateTime until, string breakdown)
{
var tResult = new List<DataTypes.AdReportRun>();
int retry = 0;
List<DataTypes.AdReportRun> tReportCompleteList = new List<DataTypes.AdReportRun>();
List<DataTypes.AdReportRun> tReportIncompleteList = new List<DataTypes.AdReportRun>();
var report_ids = await ExecuteMarketingReports(fb, batchRequests);
Thread.Sleep(20000); // Waits 20 seconds before get the info.
do
{
/*Start monitorizing the reports*/
var tReport_info = await MonitorizeMarketingReports(fb, report_ids);
/*Get the reports that are complete*/
tReportCompleteList = tReport_info.Where(x => x.async_percent_completion == 100).ToList();
if (tReportCompleteList.Count > 0)
await GetReportData(fb, tReportCompleteList, since, until, breakdown);
tReportIncompleteList = tReport_info.Where(x => x.async_percent_completion < 100).ToList();
report_ids = (from x in tReportIncompleteList
select new DataTypes.ReportRun { report_run_id = x.id }).ToList();
var sleepTime = TimeSpan.FromSeconds(Math.Pow(2, retry + 1));
Thread.Sleep(sleepTime);
retry++;
} while (report_ids.Count > 0 && retry < 8);
}
I call my Main task in this foreach loop, and this is where the problem occurs.
for (int i = 0; i < ActiveAdAccounts.Count; i = i + 50)
{
var AdAccountsSubList = ActiveAdAccounts.Skip(i).Take(50).ToList();
var batchRequests = ....
await PullInsightsData(fb, batchRequests, (DateTime)since, (DateTime)until, breakdown.Replace(",", "_"));
//tTaskList.Add(PullInsightsData(fb, batchRequests, (DateTime)since, (DateTime)until, breakdown.Replace(",", "_")));
}
//Task.WaitAll(tTaskList);
I don't understand why the foreach loop does't continue using the await the console application closes suddenly, shouldn't await "wait" until the task is finish and so then proceed to the next line of code?
I've solved the problem putting all the tasks into a list and waiting for all, but I would like an explanation for this.
[EDITED] Question was edited to create a minimal reproducible example.

How to asynchronously run 3 processes, when one returns desired value stop the other two and continue with program?

I have a method that uses the aforge.net framework to templatematch (check an image against another image for similarity) a number of seperate images against an area of the screen. This task can take a very long time or it can be near instant depending on the number of images, size of the images, and the area being checked.
Only 1 image in the list of images will ever return a match so I would like to test all the images against the screen at the same time and at which point 1 of these images returns true the remaining processes are immediately canceled and my program moves on to its next step.
Now, in the example I give I am grabbing an integer value based upon which match returns true but the concept is always the same.. x number of images tested against a screenshot.. 1 will return true, the rest will not. Sometimes the first returns true and the process is nice and fast other times it's the 30th in the list and synchronously matching the template for 30 images takes a considerable amount of time in comparison to 1.
One caveat to note about my code that follows.. I won't always return an integer, I will normally return a boolean value as to which image was found but this code here was the easiest to detail and the same general principle applies (ie: if I can figure it out one way I'll be able to do it the other).
Currently my (synchronous) code reads as follows... How would I make this an asynchronous call that can do what I've described? If possible please detail your answer as I intend to learn so that I can readily do this type of thing in the future. I understand the concept of async but for some reason cannot wrap my head around exactly how to do it the way I want.
public void Battle()
{
var myGuysTurn = WhosTurn();
// other logic here.
}
private int WhosTurn()
{
var whosTurn = 0;
var whosTurnCheck = _templateMatch.Match(_tabula.BattleHeroTurn1());
if (whosTurnCheck)
{
whosTurn = 1;
return whosTurn;
}
whosTurnCheck = _templateMatch.Match(_tabula.BattleHeroTurn2());
if (whosTurnCheck)
{
whosTurn = 2;
return whosTurn;
}
whosTurnCheck = _templateMatch.Match(_tabula.BattleHeroTurn3());
if (whosTurnCheck)
{
whosTurn = 3;
return whosTurn;
}
return whosTurn;
}

I'd use Task.WaitAny() combined with a CancellationToken. Essentially, start each task in parallel and wait until any of complete. If the completed task was successful, cancel the others. If not, continue to wait for other tasks to complete.
I've replaced _templateMatch.Match(_tabula.BattleHeroTurnX()) with a static method BattleHeroTurnX for brevity:
private int WhosTurn()
{
// Create cancellation token. Will be used to inform other threads that they should immediately cancel processing
CancellationTokenSource cts = new CancellationTokenSource();
// Collection of tasks that run in parallel
List<Task<int>> tasks = new List<Task<int>>()
{
Task.Run<int>(() => {
return BattleHeroTurn1(cts.Token) ? 1 : 0;
}),
Task.Run<int>(() => {
return BattleHeroTurn2(cts.Token) ? 2 : 0;
}),
Task.Run<int>(() => {
return BattleHeroTurn3(cts.Token) ? 3 : 0;
})
};
// Wait for any task to complete and if it is successful, cancel the other tasks and return
while (tasks.Any())
{
// Get the index of the task that completed
int completedTaskIndex = Task.WaitAny(tasks.ToArray());
int turn = tasks[completedTaskIndex].Result;
if(turn > 0)
{
cts.Cancel();
return turn;
}
tasks.RemoveAt(completedTaskIndex);
}
// All tasks have completed but no BattleHeroTurnX returned true
return 0;
}
static bool BattleHeroTurn1(CancellationToken token)
{
// Processing images. After each one is processed, ensure that the token has not been canceled
for(int i = 0; i < 100; i++)
{
Thread.Sleep(50);
if (token.IsCancellationRequested)
{
return false;
}
}
return true;
}
static bool BattleHeroTurn2(CancellationToken token)
{
// Processing images. After each one is processed, ensure that the token has not been canceled
for (int i = 0; i < 10; i++)
{
Thread.Sleep(70);
if (token.IsCancellationRequested)
{
return false;
}
}
return true;
}
static bool BattleHeroTurn3(CancellationToken token)
{
// Processing images. After each one is processed, ensure that the token has not been canceled
for (int i = 0; i < 1000; i++)
{
Thread.Sleep(500);
if (token.IsCancellationRequested)
{
return false;
}
}
return true;
}
See this and this for further information.

ReactiveExtensions Observable FromAsync calling twice Function

Ok, Trying to understand Rx, kinda of lost here.
FromAsyncPattern is now deprecated so I took the example from here (section Light up Task with Rx), and it works, I just made a few changes, not using await just wait the observable and subscribing.....
What I don't understand is Why is called Twice the function SumSquareRoots?
var res = Observable.FromAsync(ct => SumSquareRoots(x, ct))
.Timeout(TimeSpan.FromSeconds(5));
res.Subscribe(y => Console.WriteLine(y));
res.Wait();
class Program
{
static void Main(string[] args)
{
Samples();
}
static void Samples()
{
var x = 100000000;
try
{
var res = Observable.FromAsync(ct => SumSquareRoots(x, ct))
.Timeout(TimeSpan.FromSeconds(5));
res.Subscribe(y => Console.WriteLine(y));
res.Wait();
}
catch (TimeoutException)
{
Console.WriteLine("Timed out :-(");
}
}
static Task<double> SumSquareRoots(long count, CancellationToken ct)
{
return Task.Run(() =>
{
var res = 0.0;
Console.WriteLine("Why I'm called twice");
for (long i = 0; i < count; i++)
{
res += Math.Sqrt(i);
if (i % 10000 == 0 && ct.IsCancellationRequested)
{
Console.WriteLine("Noticed cancellation!");
ct.ThrowIfCancellationRequested();
}
}
return res;
});
}
}

The reason that this is calling SumSquareRoots twice is because you're Subscribing twice:
// Subscribes to res
res.Subscribe(y => Console.WriteLine(y));
// Also Subscribes to res, since it *must* produce a result, even
// if that result is then discarded (i.e. Wait doesn't return IObservable)
res.Wait();
Subscribe is the foreach of Rx - just like if you foreach an IEnumerable twice, you could end up doing 2x the work, multiple Subscribes means multiple the work. To undo this, you could use a blocking call that doesn't discard the result:
Console.WriteLine(res.First());
Or, you could use Publish to "freeze" the result and play it back to > 1 subscriber (kind of like how you'd use ToArray in LINQ):
res = res.Publish();
res.Connect();
// Both subscriptions get the same result, SumSquareRoots is only called once
res.Subscribe(Console.WriteLine);
res.Wait();
The general rule you can follow is, that any Rx method that doesn't return IObservable<T> or Task<T> will result in a Subscription(*)
* - Not technically correct. But your brain will feel better if you think of it this way.