Given some code like so
public class CustomCollectionClass : Collection<CustomData> {}
public class CustomData
{
string name;
bool finished;
string result;
}
public async Task DoWorkInParallel(CustomCollectionClass collection)
{
// collection can be retrieved from a DB, may not exist.
if (collection == null)
{
collection = new CustomCollectionClass();
foreach (var data in myData)
{
collection.Add(new CustomData()
{
name = data.Name;
});
}
}
// This part doesn't feel safe. Not sure what to do here.
var processTasks = myData.Select(o =>
this.DoWorkOnItemInCollection(collection.Single(d => d.name = o.Name))).ToArray();
await Task.WhenAll(processTasks);
await SaveModifedCollection(collection);
}
public async Task DoWorkOnItemInCollection(CustomData data)
{
await DoABunchOfWorkElsewhere();
// This doesn't feel safe either. Lock here?
data.finished = true;
data.result = "Parallel";
}
As I noted in a couple comments inline, it doesn't feel safe for me to do the above, but I'm not sure. I do have a collection of elements that I'd like to assign a unique element to each parallel task and have those tasks be able to modify that single element of the collection based on what work is done. End result being, I wanted to save the collection after individual, different elements have been modified in parallel. If this isn't a safe way to do it, how best would I go about this?
Your code is the right way to do this, assuming starting DoABunchOfWorkElsewhere() multiple times is itself safe.
You don't need to worry about your LINQ query, because it doesn't actually run in parallel. All it does is to invoke DoWorkOnItemInCollection() multiple times. Those invocations may work in parallel (or not, depending on your synchronization context and the implementation of DoABunchOfWorkElsewhere()), but the code you showed is safe.
Your above code should work without issue. You are passing off one item to each worker thread. I'm not so sure about the async attribute. You might just return a Task, and then in your method do:
public Task DoWorkOnItemInCollection(CustomData data)
{
return Task.Run(() => {
DoABunchOfWorkElsewhere().Wait();
data.finished = true;
data.result = "Parallel";
});
}
You might want to be careful, with large amount of items, you could overflow your max thread count with background threads. In this case, c# just deletes your threads, which can be difficult to debug later.
I have done this before, It might be easier if instead of handing the whole collection to some magic linq, rather do a classic consumer problem:
class ParallelWorker<T>
{
private Action<T> Action;
private Queue<T> Queue = new Queue<T>();
private object QueueLock = new object();
private void DoWork()
{
while(true)
{
T item;
lock(this.QueueLock)
{
if(this.Queue.Count == 0) return; //exit thread
item = this.Queue.DeQueue();
}
try { this.Action(item); }
catch { /*...*/ }
}
}
public void DoParallelWork(IEnumerable<T> items, int maxDegreesOfParallelism, Action<T> action)
{
this.Action = action;
this.Queue.Clear();
this.Queue.AddRange(items);
List<Thread> threads = new List<Thread>();
for(int i = 0; i < items; i++)
{
ParameterizedThreadStart threadStart = new ParameterizedThreadStart(DoWork);
Thread thread = new Thread(threadStart);
thread.Start();
threads.Add(thread);
}
foreach(Thread thread in threads)
{
thread.Join();
}
}
}
This was done IDE free, so there may be typos.
I'm going to make the suggestion that you use Microsoft's Reactive Framework (NuGet "Rx-Main") to do this task.
Here's the code:
public void DoWorkInParallel(CustomCollectionClass collection)
{
var query =
from x in collection.ToObservable()
from r in Observable.FromAsync(() => DoWorkOnItemInCollection(x))
select x;
query.Subscribe(x => { }, ex => { }, async () =>
{
await SaveModifedCollection(collection);
});
}
Done. That's it. Nothing more.
I have to say though, that when I tried to get your code to run it was full of bugs and issues. I suspect that the code you posted isn't your production code, but an example you wrote specifically for this question. I suggest that you try to make a running compilable example before posting.
Nevertheless, my suggestion should work for you with a little tweaking.
It is multi-threaded and thread-safe. And it does do cleanly save the modified collection when done.
Related
I've a method which could be called by multiple threads, to write data to a database. To reduce database traffic, I cache the data and write it in a bulk.
Now I wanted to know, is there a better (for example lock-free pattern) to use?
Here is a Example how I do it at the moment?
public class WriteToDatabase : IWriter, IDisposable
{
public WriteToDatabase(PLCProtocolServiceConfig currentConfig)
{
writeTimer = new System.Threading.Timer(Writer);
writeTimer.Change((int)currentConfig.WriteToDatabaseTimer.TotalMilliseconds, Timeout.Infinite);
this.currentConfig = currentConfig;
}
private System.Threading.Timer writeTimer;
private List<PlcProtocolDTO> writeChache = new List<PlcProtocolDTO>();
private readonly PLCProtocolServiceConfig currentConfig;
private bool disposed;
public void Write(PlcProtocolDTO row)
{
lock (this)
{
writeChache.Add(row);
}
}
private void Writer(object state)
{
List<PlcProtocolDTO> oldCachce = null;
lock (this)
{
if (writeChache.Count > 0)
{
oldCachce = writeChache;
writeChache = new List<PlcProtocolDTO>();
}
}
if (oldCachce != null)
{
using (var s = VisuDL.CreateSession())
{
s.Insert(oldCachce);
}
}
if (!this.disposed)
writeTimer.Change((int)currentConfig.WriteToDatabaseTimer.TotalMilliseconds, Timeout.Infinite);
}
public void Dispose()
{
this.disposed = true;
writeTimer.Dispose();
Writer(null);
}
}
There are a few issues I can see with the timer based code.
Even in the new version of the code there is still a chance to lose writes on restart or shutdown.
The Dispose method is not waiting for the completion of the last timer callback that may be currently in progress.
Since timer callbacks run on thread pool threads, which are background threads, they will be aborted when the main thread exits.
There is no limit on the size of the batches, this is going to break when you hit a limit of the underlying storage api
(e.g. sql databases have a limit on query length and the number of parameters used).
since you're doing i/o the implementation should probably be async
This will behave poorly under load.
in particular as the load keeps increasing the batches will get bigger and therefore slower to execute,
a slower batch execution in turn will give the next one additional time to accumulate items making them even slower, etc...
ultimately either writing the batch will fail (if you hit a sql limit or the query times out) or the application will just go out of memory.
To handle high load you really have only two choices which are applying backpressure (i.e. slowing down the producers) or dropping writes.
you might want to allow a limited number of concurrent writers if the database can handle it.
There's a race condition on the disposed field which might result in an ObjectDisposedException in writeTimer.Change.
I think a better pattern that addresses the issues above is the consumer-producer pattern, you can implement it in .net
with a ConcurrentQueue or with the new System.Threading.Channels api.
Also keep in mind that if your application crashes for any reason you will lose the records that are still buffered.
This is a sample implementation using channels:
public interface IWriter<in T>
{
ValueTask WriteAsync(IEnumerable<T> items);
}
public sealed record Options(int BatchSize, TimeSpan Interval, int MaxPendingWrites, int Concurrency);
public class BatchWriter<T> : IWriter<T>, IAsyncDisposable
{
readonly IWriter<T> writer;
readonly Options options;
readonly Channel<T> channel;
readonly Task[] consumers;
public BatchWriter(IWriter<T> writer, Options options)
{
this.writer = writer;
this.options = options;
channel = Channel.CreateBounded<T>(new BoundedChannelOptions(options.MaxPendingWrites)
{
// Choose between backpressure (Wait) or
// various ways to drop writes (DropNewest, DropOldest, DropWrite).
FullMode = BoundedChannelFullMode.Wait,
SingleWriter = false,
SingleReader = options.Concurrency == 1
});
consumers = Enumerable.Range(start: 0, options.Concurrency)
.Select(_ => Task.Run(Start))
.ToArray();
}
async Task Start()
{
var batch = new List<T>(options.BatchSize);
var timer = Task.Delay(options.Interval);
var canRead = channel.Reader.WaitToReadAsync().AsTask();
while (true)
{
if (await Task.WhenAny(timer, canRead) == timer)
{
timer = Task.Delay(options.Interval);
await Flush(batch);
}
else if (await canRead)
{
while (channel.Reader.TryRead(out var item))
{
batch.Add(item);
if (batch.Count == options.BatchSize)
{
await Flush(batch);
}
}
canRead = channel.Reader.WaitToReadAsync().AsTask();
}
else
{
await Flush(batch);
return;
}
}
async Task Flush(ICollection<T> items)
{
if (items.Count > 0)
{
await writer.WriteAsync(items);
items.Clear();
}
}
}
public async ValueTask WriteAsync(IEnumerable<T> items)
{
foreach (var item in items)
{
await channel.Writer.WriteAsync(item);
}
}
public async ValueTask DisposeAsync()
{
channel.Writer.Complete();
await Task.WhenAll(consumers);
}
}
Instead of using a mutable List and protecting it using locks, you could use an ImmutableList, and stop worrying about the possibility of the list being mutated by the wrong thread at the wrong time. With immutable collections it is cheap and easy to pass around snapshots of your data, because you don't need to block the writers (and possibly also the readers) while creating copies of the data. An immutable collection is a snapshot by itself.
Although you don't have to worry about the contents of the collection, you still have to worry about its reference. This is because updating an immutable collection means replacing the reference to the old collection with a new collection. You don't want to have multiple threads swapping references in an uncontrollable manner, so you still need some sort of synchronization. You can still use locks, but it is quite easy to avoid locking altogether by using interlocked operations. The example below uses the handy ImmutableInterlocked.Update method, that allows to do an atomic update-and-swap in a single line:
private ImmutableList<PlcProtocolDTO> writeCache
= ImmutableList<PlcProtocolDTO>.Empty;
public void Write(PlcProtocolDTO row)
{
ImmutableInterlocked.Update(ref writeCache, x => x.Add(row));
}
private void Writer(object state)
{
IList<PlcProtocolDTO> oldCache = Interlocked.Exchange(
ref writeCache, ImmutableList<PlcProtocolDTO>.Empty);
using (var s = VisuDL.CreateSession())
s.Insert(oldCache);
}
private void Dump()
{
foreach (var row in Volatile.Read(ref writeCache))
Console.WriteLine(row);
}
Here is the description of the ImmutableInterlocked.Update method:
Mutates a value in-place with optimistic locking transaction semantics via a specified transformation function. The transformation is retried as many times as necessary to win the optimistic locking race.
This method can be used for updating any type of reference-type variables. Its usage may be increased with the advent of the new C# 9 record types, that are immutable by default, and are intended to be used as such.
I'm trying to implement a producer/consumer pattern using BlockingCollection<T> so I've written up a simple console application to test it.
public class Program
{
public static void Main(string[] args)
{
var workQueue = new WorkQueue();
workQueue.StartProducingItems();
workQueue.StartProcessingItems();
while (true)
{
}
}
}
public class WorkQueue
{
private BlockingCollection<int> _queue;
private static Random _random = new Random();
public WorkQueue()
{
_queue = new BlockingCollection<int>();
// Prefill some items.
for (int i = 0; i < 100; i++)
{
//_queue.Add(_random.Next());
}
}
public void StartProducingItems()
{
Task.Run(() =>
{
_queue.Add(_random.Next()); // Should be adding items to the queue constantly, but instead adds one and then nothing else.
});
}
public void StartProcessingItems()
{
Task.Run(() =>
{
foreach (var item in _queue.GetConsumingEnumerable())
{
Console.WriteLine("Worker 1: " + item);
}
});
Task.Run(() =>
{
foreach (var item in _queue.GetConsumingEnumerable())
{
Console.WriteLine("Worker 2: " + item);
}
});
}
}
However there are 3 problems with my design:
I don't know the correct way of blocking/waiting in my Main method. Doing a simple empty while loop seems terribly inefficient and CPU usage wasting simply for the sake of making sure the application doesn't end.
There's also another problem with my design, in this simple application I have a producer that produces items indefinitely, and should never stop. In a real world setup, I'd want it to end eventually (e.g. ran out of files to process). In that case, how should I wait for it to finish in the Main method? Make StartProducingItems async and then await it?
Either the GetConsumingEnumerable or Add is not working as I expected. The producer should constantly adding items, but it adds one item and then never adds anymore. This one item is then processed by one of the consumers. Both consumers then block waiting for items to be added, but none are. I know of the Take method, but again spinning on Take in a while loop seems pretty wasteful and inefficient. There is a CompleteAdding method but that then does not allow anything else to ever be added and throws an exception if you try, so that is not suitable.
I know for sure that both consumers are in fact blocking and waiting for new items, as I can switch between threads during debugging:
EDIT:
I've made the changes suggested in one of the comments, but the Task.WhenAll still returns right away.
public Task StartProcessingItems()
{
var consumers = new List<Task>();
for (int i = 0; i < 2; i++)
{
consumers.Add(Task.Run(() =>
{
foreach (var item in _queue.GetConsumingEnumerable())
{
Console.WriteLine($"Worker {i}: " + item);
}
}));
}
return Task.WhenAll(consumers.ToList());
}
GetConsumingEnumerable() is blocking. If you want to add to the queue constantly, you should put the call to _queue.Add in a loop:
public void StartProducingItems()
{
Task.Run(() =>
{
while (true)
_queue.Add(_random.Next());
});
}
Regarding the Main() method you could call the Console.ReadLine() method to prevent the main thread from finishing before you have pressed a key:
public static void Main(string[] args)
{
var workQueue = new WorkQueue();
workQueue.StartProducingItems();
workQueue.StartProcessingItems();
Console.WriteLine("Press a key to terminate the application...");
Console.ReadLine();
}
I have a C# WinForms (.NET 4.5.2) app utilizing the TPL. The tool has a synchronous function which is passed over to a task factory X amount of times (with different input parameters), where X is a number declared by the user before commencing the process. The tasks are started and stored in a List<Task>.
Assuming the user entered 5, we have this in an async button click handler:
for (int i = 0; i < X; i++)
{
var progress = Progress(); // returns a new IProgress<T>
var task = Task<int>.Factory.StartNew(() => MyFunction(progress), TaskCreationOptions.LongRunning);
TaskList.Add(task);
}
Each progress instance updates the UI.
Now, as soon as a task is finished, I want to fire up a new one. Essentially, the process should run indefinitely, having X tasks running at any given time, unless the user cancels via the UI (I'll use cancellation tokens for this). I try to achieve this using the following:
while (TaskList.Count > 0)
{
var completed = await Task.WhenAny(TaskList.ToArray());
if (completed.Exception == null)
{
// report success
}
else
{
// flatten AggregateException, print out, etc
}
// update some labels/textboxes in the UI, and then:
TaskList.Remove(completed);
var task = Task<int>.Factory.StartNew(() => MyFunction(progress), TaskCreationOptions.LongRunning);
TaskList.Add(task);
}
This is bogging down the UI. Is there a better way of achieving this functionality, while keeping the UI responsive?
A suggestion was made in the comments to use TPL Dataflow but due to time constraints and specs, alternative solutions are welcome
Update
I'm not sure whether the progress reporting might be the problem? Here's what it looks like:
private IProgress<string> Progress()
{
return new Progress<string>(msg =>
{
txtMsg.AppendText(msg);
});
}
Now, as soon as a task is finished, I want to fire up a new one. Essentially, the process should run indefinitely, having X tasks running at any given time
It sounds to me like you want an infinite loop inside your task:
for (int i = 0; i < X; i++)
{
var progress = Progress(); // returns a new IProgress<T>
var task = RunIndefinitelyAsync(progress);
TaskList.Add(task);
}
private async Task RunIndefinitelyAsync(IProgress<T> progress)
{
while (true)
{
try
{
await Task.Run(() => MyFunction(progress));
// handle success
}
catch (Exception ex)
{
// handle exceptions
}
// update some labels/textboxes in the UI
}
}
However, I suspect that the "bogging down the UI" is probably in the // handle success and/or // handle exceptions code. If my suspicion is correct, then push as much of the logic into the Task.Run as possible.
As I understand, you simply need a parallel execution with the defined degree of parallelization. There is a lot of ways to implement what you want. I suggest to use blocking collection and parallel class instead of tasks.
So when user clicks button, you need to create a new blocking collection which will be your data source:
BlockingCollection<IProgress> queue = new BlockingCollection<IProgress>();
CancellationTokenSource source = new CancellationTokenSource();
Now you need a runner that will execute your in parallel:
Task.Factory.StartNew(() =>
Parallel.For(0, X, i =>
{
foreach (IProgress p in queue.GetConsumingEnumerable(source.Token))
{
MyFunction(p);
}
}), source.Token);
Or you can choose more correct way with partitioner. So you'll need a partitioner class:
private class BlockingPartitioner<T> : Partitioner<T>
{
private readonly BlockingCollection<T> _Collection;
private readonly CancellationToken _Token;
public BlockingPartitioner(BlockingCollection<T> collection, CancellationToken token)
{
_Collection = collection;
_Token = token;
}
public override IList<IEnumerator<T>> GetPartitions(int partitionCount)
{
throw new NotImplementedException();
}
public override IEnumerable<T> GetDynamicPartitions()
{
return _Collection.GetConsumingEnumerable(_Token);
}
public override bool SupportsDynamicPartitions
{
get { return true; }
}
}
And runner will looks like this:
ParallelOptions Options = new ParallelOptions();
Options.MaxDegreeOfParallelism = X;
Task.Factory.StartNew(
() => Parallel.ForEach(
new BlockingPartitioner<IProgress>(queue, source.Token),
Options,
p => MyFunction(p)));
So all you need right now is to fill queue with necessary data. You can do it whenever you want.
And final touch, when the user cancels operation, you have two options:
first you can break execution with source.Cancel call,
or you can gracefully stop execution by marking collection complete (queue.CompleteAdding), in that case runner will execute all already queued data and finish.
Of course you need additional code to handle exceptions, progress, state and so on. But main idea is here.
I have a queue of jobs which can be populated by multiple threads (ConcurrentQueue<MyJob>). I need to implement continuous execution of this jobs asynchronously(not by main thread), but only by one thread at the same time. I've tried something like this:
public class ConcurrentLoop {
private static ConcurrentQueue<MyJob> _concurrentQueue = new ConcurrentQueue<MyJob>();
private static Task _currentTask;
private static object _lock = new object();
public static void QueueJob(Job job)
{
_concurrentQueue.Enqueue(job);
checkLoop();
}
private static void checkLoop()
{
if ( _currentTask == null || _currentTask.IsCompleted )
{
lock (_lock)
{
if ( _currentTask == null || _currentTask.IsCompleted )
{
_currentTask = Task.Run(() =>
{
MyJob current;
while( _concurrentQueue.TryDequeue( out current ) )
//Do something
});
}
}
}
}
}
This code in my opinion have a problem: if task finnishing to execute(TryDequeue returns false but task have not been marked as completed yet) and in this moment i get a new job, it will not be executed. Am i right? If so, how to fix this
Your problem statement looks like a producer-consumer problem, with a caveat that you only want a single consumer.
There is no need to reimplement such functionality manually.
Instead, I suggest to use BlockingCollection -- internally it uses ConcurrentQueue and a separate thread for the consumption.
Note, that this may or may not be suitable for your use case.
Something like:
_blockingCollection = new BlockingCollection<your type>(); // you may want to create bounded or unbounded collection
_consumingThread = new Thread(() =>
{
foreach (var workItem in _blockingCollection.GetConsumingEnumerable()) // blocks when there is no more work to do, continues whenever a new item is added.
{
// do work with workItem
}
});
_consumingThread.Start();
Multiple producers (tasks or threads) can add work items to the _blockingCollection no problem, and no need to worry about synchronizing producers/consumer.
When you are done with producing task, call _blockingCollection.CompleteAdding() (this method is not thread safe, so it is advised to stop all producers beforehand).
Probably, you should also do _consumingThread.Join() somewhere to terminate your consuming thread.
I would use Microsoft's Reactive Framework Team's Reactive Extensions (NuGet "System.Reactive") for this. It's a lovely abstraction.
public class ConcurrentLoop
{
private static Subject<MyJob> _jobs = new Subject<MyJob>();
private static IDisposable _subscription =
_jobs
.Synchronize()
.ObserveOn(Scheduler.Default)
.Subscribe(job =>
{
//Do something
});
public static void QueueJob(MyJob job)
{
_jobs.OnNext(job);
}
}
This nicely synchronizes all incoming jobs into a single stream and pushes the execution on to Scheduler.Default (which is basically the thread-pool), but because it has serialized all input only one can happen at a time. The nice thing about this is that it releases the thread if there is a significant gap between the values. It's a very lean solution.
To clean up you just need call either _jobs.OnCompleted(); or _subscription.Dispose();.
Few days ago I tried to perform a fast search on my disks do few things like, Attributes, Extensions, perform change inside files etc ...
The idea was to make it with really few limitation/lock in order to avoid "latency" for big file or directory with a lots of files inside etc ...
I know it's far for "Best Practices", since i'm not using things like "MaxDegreeOfParallelism" or the Pulling loop with "while(true)"
Even though, the code is running quite fast since we have the architecture to support it.
I tried to move to code to a dummy console project if anybody would like to check what's going on.
class Program
{
static ConcurrentQueue<String> dirToCheck;
static ConcurrentQueue<String> fileToCheck;
static int fileCount; //
static void Main(string[] args)
{
Initialize();
Task.Factory.StartNew(() => ScanDirectories(), TaskCreationOptions.LongRunning);
Task.Factory.StartNew(() => ScanFiles(), TaskCreationOptions.LongRunning);
Console.ReadLine();
}
static void Initialize()
{
//Instantiate caches
dirToCheck = new ConcurrentQueue<string>();
fileToCheck = new ConcurrentQueue<string>();
//Enqueue Directory to Scan here
//Avoid to Enqueue Nested/Sub directories, else they are going to be dcan at least twice
dirToCheck.Enqueue(#"C:\");
//Initialize counters
fileCount = 0;
}
static void ScanDirectories()
{
String dirToScan = null;
while (true)
{
if (dirToCheck.TryDequeue(out dirToScan))
{
ExtractDirectories(dirToScan);
ExtractFiles(dirToScan);
}
//Just here as a visual tracker to have some kind an idea about what's going on and where's the load
Console.WriteLine(dirToCheck.Count + "\t\t" + fileToCheck.Count + "\t\t" + fileCount);
}
}
static void ScanFiles()
{
while (true)
{
String fileToScan = null;
if (fileToCheck.TryDequeue(out fileToScan))
{
CheckFileAsync(fileToScan);
}
}
}
private static Task ExtractDirectories(string dirToScan)
{
Task worker = Task.Factory.StartNew(() =>
{
try
{
Parallel.ForEach<String>(Directory.EnumerateDirectories(dirToScan), (dirPath) =>
{
dirToCheck.Enqueue(dirPath);
});
}
catch (UnauthorizedAccessException) { }
}, TaskCreationOptions.AttachedToParent);
return worker;
}
private static Task ExtractFiles(string dirToScan)
{
Task worker = Task.Factory.StartNew(() =>
{
try
{
Parallel.ForEach<String>(Directory.EnumerateFiles(dirToScan), (filePath) =>
{
fileToCheck.Enqueue(filePath);
});
}
catch (UnauthorizedAccessException) { }
}, TaskCreationOptions.AttachedToParent);
return worker;
}
static Task CheckFileAsync(String filePath)
{
Task worker = Task.Factory.StartNew(() =>
{
//Add statement to play along with the file here
Interlocked.Increment(ref fileCount);
//WARNING !!! If your file fullname is too long this code may not be executed or may just crash
//I just put a simple check 'cause i found 2 or 3 different error message between the framework & msdn documentation
//"Full paths must not exceed 260 characters to maintain compatibility with Windows operating systems. For more information about this restriction, see the entry Long Paths in .NET in the BCL Team blog"
if (filePath.Length > 260)
return;
FileInfo fi = new FileInfo(filePath);
//Add statement here to use FileInfo
}, TaskCreationOptions.AttachedToParent);
return worker;
}
}
Problems:
How can I detect that i'm done with ScanDirectory?
Once it's done, I can manage to enqueue a String empty or whatever to the file queue, to exit it.
I know that if I use "AttachedToParent" I can have a Completion state on the parent Task, and then for example do something like "ContinueWith(()=> { /SomeCode to notice the end/})"
But still the parent task is doing Pulling and is stuck in a kind of infinite loop and each sub statement begin new Task.
On the other hand, I cannot simply test "Count" in each Queue 'cause I might have Flush the File List and Directory List but there might be another task that's going to call "EnumerateDirectory()".
I'm trying to find some kind of "reactive" solution and avoid some "if()" inside the loop that would be checked 80% of time for nothing since it's a simple while(true){} with AsyncCall.
PS: I know i could use TPL Dataflow, i'm not because i'm stuck on .net 4.0 for know, anyway, in .net 4.5 without dataflow since there's few improvement in the TPL, i'm still curious about it
Instead of ConcurrentQueue<T>, you could use BlockingCollection<T>.
BlockingCollection<T> is designed specifically for producer/consumer scenarios such as this, and provides a CompleteAdding method so the producer can notify the consumers that it has finished adding work.