Develop Reference

How to delay multiple messages in parallel?

I am doing some TCP programming and I want to simulate some latency.
Each "message" (a byte[] representing a serialized object) must be delayed by some time t. I thought that I can have one function to collect raw messages:
private Queue<byte[]> rawMessages = new Queue<byte[]>();
private void OnDataReceived(object sender, byte[] data)
{
rawMessages.Enqueue(data);
}
Another method with a while-loop to continuously read from rawMessages and then delay each one:
private Queue<byte[]> delayedRawMessages = new Queue<byte[]>();
delayMessagesInTask = Task.Factory.StartNew(() =>
{
while (true) //TODO: Swap a cancellation token
{
if(rawMessages.Count > 0){
var rawMessage = rawMessages.Dequeue();
//???
//Once the message has been delayed, enqueue it into another buffer.
delayedRawMessages.Enqueue(rawMessage);
}
}
});
I thought that to delay each message, I could have another method to spawn a thread which uses Thread.Sleep(t) to wait for time t and then enqueue delayedRawMessages. I'm sure this will work, but I think there must be a better way. The issue with the Thread.Sleep approach is that message 2 might finish being delayed before message 1... I obviously need the messages to be delayed and done so in the correct order, otherwise I would not be using TCP.
I'm looking for a way to do this that will delay for as close to time t as possible and will not impact the rest of the application by slowing it down.
Does anyone here know of a better way to do this?

I decided to go the producer/multiple consumer approach.
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Threading;
using System.Threading.Tasks;
public class LatencySimulator {
public enum SimulatorType { UP, DOWN };
public SimulatorType type;
public int latency = 0;
public int maxConsumers = 50;
public BlockingCollection<byte[]> inputQueue = new BlockingCollection<byte[]>(new ConcurrentQueue<byte[]>());
public Queue<byte[]> delayedMessagesQueue = new Queue<byte[]>();
void CreateConsumers()
{
for (int i = 0; i < maxConsumers; i++)
{
Task.Factory.StartNew(() => Consumer(),TaskCreationOptions.LongRunning);
}
}
private void Consumer()
{
foreach (var item in inputQueue.GetConsumingEnumerable())
{
Thread.Sleep(latency);
delayedMessagesQueue.Enqueue(item);
}
}
}
To use it, create a new LatencySimulator, set its 'type', max consumers and latency to simulate. Call CreateConsmers() and then populate the inputQueue. When the messages have been delayed, they will appear in the delayedMessagesQueue.
I do not really know if this is an ideal way to achieve my goal, but it works... for now.

HttpClient.PostAsync Hanging Until All System.Timers Are Started

I'm having an odd problem with HttpClient and Timers. I have a large number of objects (up to 10,000) that post to a web service. These objects are on timers and post to the service at some n time after creation. The problem is that the Post stalls, until all of the timers have started. See the code below for an example.
Q: Why does the Post hang until all Timers have started? How can it be fixed so that the post functions correctly while the other Timers start?
public class MyObject
{
public void Run()
{
var result = Post(someData).Result;
DoOtherStuff();
}
}
static async Task<string> Post(string data)
{
using (var client = new HttpClient())
{
//Hangs here until all timers are started
var response = await client.PostAsync(new Uri(url), data).ConfigureAwait(continueOnCapturedContext: false);
var text = await response.Content.ReadAsStringAsync().ConfigureAwait(continueOnCapturedContext: false);
return text;
}
}
static void Main(string[] args)
{
for (int i = 0; i < 1000; i++)
{
TimeSpan delay = TimeSpan.FromSeconds(1);
if (i % 2 == 0) delay = TimeSpan.FromDays(1);
System.Timers.Timer timer = new System.Timers.Timer();
timer.AutoReset = false;
timer.Interval = delay.TotalMilliseconds;
timer.Elapsed += (x, y) =>
{
MyObject o = new MyObject();
o.Run();
};
timer.Start();
}
Console.ReadKey();
}

Because you're using up all the ThreadPool threads.
There's a lot wrong with your sample code. You're killing any chance of having reasonable performance, not to mention that the whole thing is inherently unstable.
You're creating a thousand timers in a loop. You're not keeping a reference to any of them, so they will be collected the next time the GC runs - so I'd expect that in practice, very few of them will actually run, unless there's very little memory allocated until they actually get to run.
The timer's Elapsed event will be invoked on a ThreadPool thread. In that thread, you synchronously wait for a bunch of asynchronous calls to complete. That means you're now wasting a thread pool thread, and completely wasting the underlying asynchronicity of the asynchronous method.
The continuation to the asynchronous I/O will be posted to ThreadPool as well - however, the ThreadPool is full of timer callbacks. It will slowly start creating more and more threads to accomodate the amount of work scheduled, until it finally is able to execute the first callback from the asynchronous I/O and it slowly untangles itself. At this point, you likely have more than 1000 threads, and are showing a complete misunderstanding of how to do asynchronous programming.
One way (still rather bad) to fix both problems is to simply make use of asynchronicity all the time:
public class MyObject
{
public async Task Run()
{
var result = await Post(someData);
DoOtherStuff();
}
}
static async Task<string> Post(string data)
{
using (var client = new HttpClient())
{
//Hangs here until all timers are started
var response = await client.PostAsync(new Uri(url), new StringContent(data)).ConfigureAwait(continueOnCapturedContext: false);
var text = await response.Content.ReadAsStringAsync().ConfigureAwait(continueOnCapturedContext: false);
return text;
}
}
static void Main(string[] args)
{
var tasks = new List<Task>();
for (int i = 0; i < 1000; i++)
{
TimeSpan delay = TimeSpan.FromSeconds(1);
if (i % 2 == 0) delay = TimeSpan.FromDays(1);
tasks.Add(Task.Delay(delay).ContinueWith((_) => new MyObject().Run()));
}
Task.WaitAll(tasks.ToArray());
Console.WriteLine("Work done");
Console.ReadKey();
}
A much better way would be to implement some scheduler that handles dispatching the asynchronous I/O with the throttling you need. You probably want to limit the number of concurrent requests or something like that, rather than running the requests in pre-defined intervals (and ignoring the fact that some requests might take very long, timeout or some such).

As mentioned in another reply, that Result property is the problem. when you are using it, asyc will be come sync. If you want to run async operations in console or windows service applications, try Nito AsyncEx library. It creates an AsyncContext. Now you can change void Run to Task Run which is await-able and doesn't need the blocking Result property and in this case await Post will work in the Run method.
static void Main(string[] args)
{
AsyncContext.Run(async () =>
{
var data = await ...;
});
}

As you're running on a console application, which uses the default ThreadPoolSynchronizationContext, you shouldn't really be experiencing the "hanging" feeling as if you are in a UI application. I assume its because Post is taking longer to return than to allocate 1000 timers.
In order for your method to run async, it has to go "async all the way. Using the Task.Result property, as mentioned before will simply block the asynchronous operation until it completes.
Lets see what we need to do for this to be "async all the way":
First, lets change Run from void to async Task so we can await on the Post method:
public async Task Run()
{
var result = await Post(someData);
DoOtherStuff();
}
Now, since Run became awaitable, as it returns a Task, we can turn Timer.Elapsed to an async event handler and await on Run.
static void Main(string[] args)
{
for (int i = 0; i < 1000; i++)
{
TimeSpan delay = TimeSpan.FromSeconds(1);
if (i % 2 == 0) delay = TimeSpan.FromDays(1);
System.Timers.Timer timer = new System.Timers.Timer();
timer.AutoReset = false;
timer.Interval = delay.TotalMilliseconds;
timer.Elapsed += async (x, y) =>
{
MyObject o = new MyObject();
await o.Run();
};
timer.Start();
}
Console.ReadKey();
}
That's it, now we flow async all the way down to the HTTP request and back.

It's because the timers are set up very quickly so they have all finished setting up by the time PostAsync completes. Try putting a Thread.Sleep(1000) after timer.Start, this will slow down the set up of your timers and you should see some PostAsync executions complete.
By the way, Task.Result is a blocking operation, which can cause a deadlock when run from a GUI application. There is more information in this article.

Why is this TAP async/await code slower than the TPL version?

I had to write a console application that called Microsoft Dynamics CRM web service to perform an action on over eight thousand CRM objects. The details of the web service call are irrelevant and not shown here but I needed a multi-threaded client so that I could make calls in parallel. I wanted to be able to control the number of threads used from a config setting and also for the application to cancel the whole operation if the number of service errors reached a config-defined threshold.
I wrote it using Task Parallel Library Task.Run and ContinueWith, keeping track of how many calls (threads) were in progress, how many errors we'd received, and whether the user had cancelled from the keyboard. Everything worked fine and I had extensive logging to assure myself that threads were finishing cleanly and that everything was tidy at the end of the run. I could see that the program was using the maximum number of threads in parallel and, if our maximum limit was reached, waiting until a running task completed before starting another one.
During my code review, my colleague suggested that it would be better to do it with async/await instead of tasks and continuations, so I created a branch and rewrote it that way. The results were interesting - the async/await version was almost twice as slow, and it never reached the maximum number of allowed parallel operations/threads. The TPL one always got to 10 threads in parallel whereas the async/await version never got beyond 5.
My question is: have I made a mistake in the way I have written the async/await code (or the TPL code even)? If I have not coded it wrong, can you explain why the async/await is less efficient, and does that mean it is better to carry on using TPL for multi-threaded code.
Note that the code I tested with did not actually call CRM - the CrmClient class simply thread-sleeps for a duration specified in the config (five seconds) and then throws an exception. This meant that there were no external variables that could affect the performance.
For the purposes of this question I created a stripped down program that combines both versions; which one is called is determined by a config setting. Each of them starts with a bootstrap runner that sets up the environment, creates the queue class, then uses a TaskCompletionSource to wait for completion. A CancellationTokenSource is used to signal a cancellation from the user. The list of ids to process is read from an embedded file and pushed onto a ConcurrentQueue. They both start off calling StartCrmRequest as many times as max-threads; subsequently, every time a result is processed, the ProcessResult method calls StartCrmRequest again, keeping going until all of our ids are processed.
You can clone/download the complete program from here: https://bitbucket.org/kentrob/pmgfixso/
Here is the relevant configuration:
<appSettings>
<add key="TellUserAfterNCalls" value="5"/>
<add key="CrmErrorsBeforeQuitting" value="20"/>
<add key="MaxThreads" value="10"/>
<add key="CallIntervalMsecs" value="5000"/>
<add key="UseAsyncAwait" value="True" />
</appSettings>
Starting with the TPL version, here is the bootstrap runner that kicks off the queue manager:
public static class TplRunner
{
private static readonly CancellationTokenSource CancellationTokenSource = new CancellationTokenSource();
public static void StartQueue(RuntimeParameters parameters, IEnumerable<string> idList)
{
Console.CancelKeyPress += (s, args) =>
{
CancelCrmClient();
args.Cancel = true;
};
var start = DateTime.Now;
Program.TellUser("Start: " + start);
var taskCompletionSource = new TplQueue(parameters)
.Start(CancellationTokenSource.Token, idList);
while (!taskCompletionSource.Task.IsCompleted)
{
if (Console.KeyAvailable)
{
if (Console.ReadKey().Key != ConsoleKey.Q) continue;
Console.WriteLine("When all threads are complete, press any key to continue.");
CancelCrmClient();
}
}
var end = DateTime.Now;
Program.TellUser("End: {0}. Elapsed = {1} secs.", end, (end - start).TotalSeconds);
}
private static void CancelCrmClient()
{
CancellationTokenSource.Cancel();
Console.WriteLine("Cancelling Crm client. Web service calls in operation will have to run to completion.");
}
}
Here is the TPL queue manager itself:
public class TplQueue
{
private readonly RuntimeParameters parameters;
private readonly object locker = new object();
private ConcurrentQueue<string> idQueue = new ConcurrentQueue<string>();
private readonly CrmClient crmClient;
private readonly TaskCompletionSource<bool> taskCompletionSource = new TaskCompletionSource<bool>();
private int threadCount;
private int crmErrorCount;
private int processedCount;
private CancellationToken cancelToken;
public TplQueue(RuntimeParameters parameters)
{
this.parameters = parameters;
crmClient = new CrmClient();
}
public TaskCompletionSource<bool> Start(CancellationToken cancellationToken, IEnumerable<string> ids)
{
cancelToken = cancellationToken;
foreach (var id in ids)
{
idQueue.Enqueue(id);
}
threadCount = 0;
// Prime our thread pump with max threads.
for (var i = 0; i < parameters.MaxThreads; i++)
{
Task.Run((Action) StartCrmRequest, cancellationToken);
}
return taskCompletionSource;
}
private void StartCrmRequest()
{
if (taskCompletionSource.Task.IsCompleted)
{
return;
}
if (cancelToken.IsCancellationRequested)
{
Program.TellUser("Crm client cancelling...");
ClearQueue();
return;
}
var count = GetThreadCount();
if (count >= parameters.MaxThreads)
{
return;
}
string id;
if (!idQueue.TryDequeue(out id)) return;
IncrementThreadCount();
crmClient.CompleteActivityAsync(new Guid(id), parameters.CallIntervalMsecs).ContinueWith(ProcessResult);
processedCount += 1;
if (parameters.TellUserAfterNCalls > 0 && processedCount%parameters.TellUserAfterNCalls == 0)
{
ShowProgress(processedCount);
}
}
private void ProcessResult(Task<CrmResultMessage> response)
{
if (response.Result.CrmResult == CrmResult.Failed && ++crmErrorCount == parameters.CrmErrorsBeforeQuitting)
{
Program.TellUser(
"Quitting because CRM error count is equal to {0}. Already queued web service calls will have to run to completion.",
crmErrorCount);
ClearQueue();
}
var count = DecrementThreadCount();
if (idQueue.Count == 0 && count == 0)
{
taskCompletionSource.SetResult(true);
}
else
{
StartCrmRequest();
}
}
private int GetThreadCount()
{
lock (locker)
{
return threadCount;
}
}
private void IncrementThreadCount()
{
lock (locker)
{
threadCount = threadCount + 1;
}
}
private int DecrementThreadCount()
{
lock (locker)
{
threadCount = threadCount - 1;
return threadCount;
}
}
private void ClearQueue()
{
idQueue = new ConcurrentQueue<string>();
}
private static void ShowProgress(int processedCount)
{
Program.TellUser("{0} activities processed.", processedCount);
}
}
Note that I am aware that a couple of the counters are not thread safe but they are not critical; the threadCount variable is the only critical one.
Here is the dummy CRM client method:
public Task<CrmResultMessage> CompleteActivityAsync(Guid activityId, int callIntervalMsecs)
{
// Here we would normally call a CRM web service.
return Task.Run(() =>
{
try
{
if (callIntervalMsecs > 0)
{
Thread.Sleep(callIntervalMsecs);
}
throw new ApplicationException("Crm web service not available at the moment.");
}
catch
{
return new CrmResultMessage(activityId, CrmResult.Failed);
}
});
}
And here are the same async/await classes (with common methods removed for the sake of brevity):
public static class AsyncRunner
{
private static readonly CancellationTokenSource CancellationTokenSource = new CancellationTokenSource();
public static void StartQueue(RuntimeParameters parameters, IEnumerable<string> idList)
{
var start = DateTime.Now;
Program.TellUser("Start: " + start);
var taskCompletionSource = new AsyncQueue(parameters)
.StartAsync(CancellationTokenSource.Token, idList).Result;
while (!taskCompletionSource.Task.IsCompleted)
{
...
}
var end = DateTime.Now;
Program.TellUser("End: {0}. Elapsed = {1} secs.", end, (end - start).TotalSeconds);
}
}
The async/await queue manager:
public class AsyncQueue
{
private readonly RuntimeParameters parameters;
private readonly object locker = new object();
private readonly CrmClient crmClient;
private readonly TaskCompletionSource<bool> taskCompletionSource = new TaskCompletionSource<bool>();
private CancellationToken cancelToken;
private ConcurrentQueue<string> idQueue = new ConcurrentQueue<string>();
private int threadCount;
private int crmErrorCount;
private int processedCount;
public AsyncQueue(RuntimeParameters parameters)
{
this.parameters = parameters;
crmClient = new CrmClient();
}
public async Task<TaskCompletionSource<bool>> StartAsync(CancellationToken cancellationToken,
IEnumerable<string> ids)
{
cancelToken = cancellationToken;
foreach (var id in ids)
{
idQueue.Enqueue(id);
}
threadCount = 0;
// Prime our thread pump with max threads.
for (var i = 0; i < parameters.MaxThreads; i++)
{
await StartCrmRequest();
}
return taskCompletionSource;
}
private async Task StartCrmRequest()
{
if (taskCompletionSource.Task.IsCompleted)
{
return;
}
if (cancelToken.IsCancellationRequested)
{
...
return;
}
var count = GetThreadCount();
if (count >= parameters.MaxThreads)
{
return;
}
string id;
if (!idQueue.TryDequeue(out id)) return;
IncrementThreadCount();
var crmMessage = await crmClient.CompleteActivityAsync(new Guid(id), parameters.CallIntervalMsecs);
ProcessResult(crmMessage);
processedCount += 1;
if (parameters.TellUserAfterNCalls > 0 && processedCount%parameters.TellUserAfterNCalls == 0)
{
ShowProgress(processedCount);
}
}
private async void ProcessResult(CrmResultMessage response)
{
if (response.CrmResult == CrmResult.Failed && ++crmErrorCount == parameters.CrmErrorsBeforeQuitting)
{
Program.TellUser(
"Quitting because CRM error count is equal to {0}. Already queued web service calls will have to run to completion.",
crmErrorCount);
ClearQueue();
}
var count = DecrementThreadCount();
if (idQueue.Count == 0 && count == 0)
{
taskCompletionSource.SetResult(true);
}
else
{
await StartCrmRequest();
}
}
}
So, setting MaxThreads to 10 and CrmErrorsBeforeQuitting to 20, the TPL version on my machine completes in 19 seconds and the async/await version takes 35 seconds. Given that I have over 8000 calls to make this is a significant difference. Any ideas?

I think I'm seeing the problem here, or at least a part of it. Look closely at the two bits of code below; they are not equivalent.
// Prime our thread pump with max threads.
for (var i = 0; i < parameters.MaxThreads; i++)
{
Task.Run((Action) StartCrmRequest, cancellationToken);
}
And:
// Prime our thread pump with max threads.
for (var i = 0; i < parameters.MaxThreads; i++)
{
await StartCrmRequest();
}
In the original code (I am taking it as a given that it is functionally sound) there is a single call to ContinueWith. That is exactly how many await statements I would expect to see in a trivial rewrite if it is meant to preserve the original behaviour.
Not a hard and fast rule and only applicable in simple cases, but nevertheless a good thing to keep an eye out for.

I think you over complicated your solution and ended up not getting where you wanted in either implementation.
First of all, connections to any HTTP host are limited by the service point manager. The default limit for client environments is 2, but you can increase it yourself.
No matter how much threads you spawn, there won't be more active requests than those allwed.
Then, as someone pointed out, await logically blocks the execution flow.
And finally, you spent your time creating an AsyncQueue when you should have used TPL data flows.

When implemented with async/await, I would expect the I/O bound algorithm to run on a single thread. Unlike #KirillShlenskiy, I believe that the bit responsible for "bringing back" to caller's context is not responsible for the slow-down. I think you overrun the thread pool by trying to use it for I/O-bound operations. It's designed primarily for compute-bound ops.
Have a look at ForEachAsync. I feel that's what you're looking for (Stephen Toub's discussion, you'll find Wischik's videos meaningful too):
http://blogs.msdn.com/b/pfxteam/archive/2012/03/05/10278165.aspx
(Use degree of concurrency to reduce memory footprint)
http://vimeo.com/43808831
http://vimeo.com/43808833

Run x number of web requests simultaneously

Our company has a web service which I want to send XML files (stored on my drive) via my own HTTPWebRequest client in C#. This already works. The web service supports 5 synchronuous requests at the same time (I get a response from the web service once the processing on the server is completed). Processing takes about 5 minutes for each request.
Throwing too many requests (> 5) results in timeouts for my client. Also, this can lead to errors on the server side and incoherent data. Making changes on the server side is not an option (from different vendor).
Right now, my Webrequest client will send the XML and wait for the response using result.AsyncWaitHandle.WaitOne();
However, this way, only one request can be processed at a time although the web service supports 5. I tried using a Backgroundworker and Threadpool but they create too many requests at same, which make them useless to me. Any suggestion, how one could solve this problem? Create my own Threadpool with exactly 5 threads? Any suggestions, how to implement this?

The easy way is to create 5 threads ( aside: that's an odd number! ) that consume the xml files from a BlockingCollection.
Something like:
var bc = new BlockingCollection<string>();
for ( int i = 0 ; i < 5 ; i++ )
{
new Thread( () =>
{
foreach ( var xml in bc.GetConsumingEnumerable() )
{
// do work
}
}
).Start();
}
bc.Add( xml_1 );
bc.Add( xml_2 );
...
bc.CompleteAdding(); // threads will end when queue is exhausted

If you're on .Net 4, this looks like a perfect fit for Parallel.ForEach(). You can set its MaxDegreeOfParallelism, which means you are guaranteed that no more items are processed at one time.
Parallel.ForEach(items,
new ParallelOptions { MaxDegreeOfParallelism = 5 },
ProcessItem);
Here, ProcessItem is a method that processes one item by accessing your server and blocking until the processing is done. You could use a lambda instead, if you wanted.

Creating your own threadpool of five threads isn't tricky - Just create a concurrent queue of objects describing the request to make, and have five threads that loop through performing the task as needed. Add in an AutoResetEvent and you can make sure they don't spin furiously while there are no requests that need handling.
It can though be tricky to return the response to the correct calling thread. If this is the case for how the rest of your code works, I'd take a different approach and create a limiter that acts a bit like a monitor but allowing 5 simultaneous threads rather than only one:
private static class RequestLimiter
{
private static AutoResetEvent _are = new AutoResetEvent(false);
private static int _reqCnt = 0;
public ResponseObject DoRequest(RequestObject req)
{
for(;;)
{
if(Interlocked.Increment(ref _reqCnt) <= 5)
{
//code to create response object "resp".
Interlocked.Decrement(ref _reqCnt);
_are.Set();
return resp;
}
else
{
if(Interlocked.Decrement(ref _reqCnt) >= 5)//test so we don't end up waiting due to race on decrementing from finished thread.
_are.WaitOne();
}
}
}
}

You could write a little helper method, that would block the current thread until all the threads have finished executing the given action delegate.
static void SpawnThreads(int count, Action action)
{
var countdown = new CountdownEvent(count);
for (int i = 0; i < count; i++)
{
new Thread(() =>
{
action();
countdown.Signal();
}).Start();
}
countdown.Wait();
}
And then use a BlockingCollection<string> (thread-safe collection), to keep track of your xml files. By using the helper method above, you could write something like:
static void Main(string[] args)
{
var xmlFiles = new BlockingCollection<string>();
// Add some xml files....
SpawnThreads(5, () =>
{
using (var web = new WebClient())
{
web.UploadFile(xmlFiles.Take());
}
});
Console.WriteLine("Done");
Console.ReadKey();
}
Update
An even better approach would be to upload the files async, so that you don't waste resources on using threads for an IO task.
Again you could write a helper method:
static void SpawnAsyncs(int count, Action<CountdownEvent> action)
{
var countdown = new CountdownEvent(count);
for (int i = 0; i < count; i++)
{
action(countdown);
}
countdown.Wait();
}
And use it like:
static void Main(string[] args)
{
var urlXML = new BlockingCollection<Tuple<string, string>>();
urlXML.Add(Tuple.Create("http://someurl.com", "filename"));
// Add some more to collection...
SpawnAsyncs(5, c =>
{
using (var web = new WebClient())
{
var current = urlXML.Take();
web.UploadFileCompleted += (s, e) =>
{
// some code to mess with e.Result (response)
c.Signal();
};
web.UploadFileAsyncAsync(new Uri(current.Item1), current.Item2);
}
});
Console.WriteLine("Done");
Console.ReadKey();
}

Multiple async webclient calls in backgroundworker (Silverlight 4)

Requirement:
A Silverlight client that loads a file from the user's computer.
For each row in the file it needs to issue a GET to a url and either approve or reject the row based on the JSON validation message returned.
When all rows have completed, show the user a summary of how many rows passed and how many failed.
I would have preferred doing this "synchronously" in the BackgroundWorker, but that breaks the SL async thinking.
My code currently passes the result from the OpenFileDialog to a BackgroundWorker which reads the file into a list of strongly typed objects (client side validation). I create a WebClient and call DownloadStringAsync multiple times as I loop over the lines. More often than not, the background worker completes and runs the completed event method long before the WebClient is completed. The UI thread can run as async as it wants, but I need to either wait with completing the backgroundworker thread until it is actually finished getting the data, or have some additional event handler that works when all (hundreds) of rows have been validated.
What is best practice when wanting to handle the last Completed event of X total events? Any use of auto reset events here that could help?

You shouldn't need to go to a background thread for that, nor you need to use an AutoResetEvent here. After reading the file, count the number of lines (= number of requests you'll send) and store it in an instance variable; then fire all the WebRequest.DownloadAsync (or some other WebClient async call to download the data). On each callback for the async method, you'd Interlock.Decrement the instance variable, and when it reaches 0, you know that you have all the results and you can display the summary to the client.

You can choose between random response sequence (but parallel execution)
or ordered response but in sequential mode.
There is different cons and pros.
using System;
using System.Net;
using System.Reactive.Linq;//Rx libriary
using System.Threading;
using System.Windows;
using System.Windows.Controls;
public partial class MainPage : UserControl
{
private int count = 0;
private int error = 0;
public MainPage()
{
InitializeComponent();
}
private void Button_Click(object sender, RoutedEventArgs e)
{
ThreadPool.QueueUserWorkItem(StartParallel);
//ThreadPool.QueueUserWorkItem(StartSequential);
}
private void Update(Exception exception)
{
if (exception == null)
Interlocked.Increment(ref count);
else
Interlocked.Increment(ref error);
if ((count%100) == 0)
{
int count1 = count;
Dispatcher.BeginInvoke(() => { textBox.Text = count1.ToString(); });
}
}
private void StartSequential(object o)
{
//single instance of WebClient
WebClient wc = new WebClient();
var observer = Observable.FromEventPattern<DownloadStringCompletedEventArgs>(wc, "DownloadStringCompleted")
.Select(newResult => new {newResult.EventArgs.Error, newResult.EventArgs.Result});
wc.DownloadStringAsync(new Uri("http://localhost:7123/SilverlightApplication2TestPage.aspx"));
int i = 0;
foreach (var nextValue in observer.Next())
{
if (i == 10000) break;
wc.DownloadStringAsync(new Uri("http://localhost:7123/SilverlightApplication2TestPage.aspx"));
Update(nextValue.Error);
}
}
private void StartParallel(object o)
{
for (int i = 0; i < 10000; i++)
{
//multiple instance of WebClient
WebClient t = new WebClient();
t.DownloadStringCompleted +=
(x, nextValue) => Update(nextValue.Error);//order of result sequence is not guaranteed
t.DownloadStringAsync(new Uri("http://localhost:7123/SilverlightApplication2TestPage.aspx"));
}
}
}