Develop Reference

C# Abortable Asynchronous Fifo Queue - leaking massive amounts of memory

I need to process data from a producer in FIFO fashion with the ability to abort processing if the same producer produces a new bit of data.
So I implemented an abortable FIFO queue based on Stephen Cleary's AsyncCollection (called AsyncCollectionAbortableFifoQueuein my sample) and one on TPL's BufferBlock (BufferBlockAbortableAsyncFifoQueue in my sample). Here's the implementation based on AsyncCollection
public class AsyncCollectionAbortableFifoQueue<T> : IExecutableAsyncFifoQueue<T>
{
private AsyncCollection<AsyncWorkItem<T>> taskQueue = new AsyncCollection<AsyncWorkItem<T>>();
private readonly CancellationToken stopProcessingToken;
public AsyncCollectionAbortableFifoQueue(CancellationToken cancelToken)
{
stopProcessingToken = cancelToken;
_ = processQueuedItems();
}
public Task<T> EnqueueTask(Func<Task<T>> action, CancellationToken? cancelToken)
{
var tcs = new TaskCompletionSource<T>();
var item = new AsyncWorkItem<T>(tcs, action, cancelToken);
taskQueue.Add(item);
return tcs.Task;
}
protected virtual async Task processQueuedItems()
{
while (!stopProcessingToken.IsCancellationRequested)
{
try
{
var item = await taskQueue.TakeAsync(stopProcessingToken).ConfigureAwait(false);
if (item.CancelToken.HasValue && item.CancelToken.Value.IsCancellationRequested)
item.TaskSource.SetCanceled();
else
{
try
{
T result = await item.Action().ConfigureAwait(false);
item.TaskSource.SetResult(result); // Indicate completion
}
catch (Exception ex)
{
if (ex is OperationCanceledException && ((OperationCanceledException)ex).CancellationToken == item.CancelToken)
item.TaskSource.SetCanceled();
item.TaskSource.SetException(ex);
}
}
}
catch (Exception) { }
}
}
}
public interface IExecutableAsyncFifoQueue<T>
{
Task<T> EnqueueTask(Func<Task<T>> action, CancellationToken? cancelToken);
}
processQueuedItems is the task that dequeues AsyncWorkItem's from the queue, and executes them unless cancellation has been requested.
The asynchronous action to execute gets wrapped into an AsyncWorkItem which looks like this
internal class AsyncWorkItem<T>
{
public readonly TaskCompletionSource<T> TaskSource;
public readonly Func<Task<T>> Action;
public readonly CancellationToken? CancelToken;
public AsyncWorkItem(TaskCompletionSource<T> taskSource, Func<Task<T>> action, CancellationToken? cancelToken)
{
TaskSource = taskSource;
Action = action;
CancelToken = cancelToken;
}
}
Then there's a task looking and dequeueing items for processing and either processing them, or aborting if the CancellationToken has been triggered.
That all works just fine - data gets processed, and if a new piece of data is received, processing of the old is aborted. My problem now stems from these Queues leaking massive amounts of memory if I crank up the usage (producer producing a lot more than the consumer processes). Given it's abortable, the data that is not processed, should be discarded and eventually disappear from memory.
So let's look at how I'm using these queues. I have a 1:1 match of producer and consumer. Every consumer handles data of a single producer. Whenever I get a new data item, and it doesn't match the previous one, I catch the queue for the given producer (User.UserId) or create a new one (the 'executor' in the code snippet). Then I have a ConcurrentDictionary that holds a CancellationTokenSource per producer/consumer combo. If there's a previous CancellationTokenSource, I call Cancel on it and Dispose it 20 seconds later (immediate disposal would cause exceptions in the queue). I then enqueue processing of the new data. The queue returns me a task that I can await so I know when processing of the data is complete, and I then return the result.
Here's that in code
internal class SimpleLeakyConsumer
{
private ConcurrentDictionary<string, IExecutableAsyncFifoQueue<bool>> groupStateChangeExecutors = new ConcurrentDictionary<string, IExecutableAsyncFifoQueue<bool>>();
private readonly ConcurrentDictionary<string, CancellationTokenSource> userStateChangeAborters = new ConcurrentDictionary<string, CancellationTokenSource>();
protected CancellationTokenSource serverShutDownSource;
private readonly int operationDuration = 1000;
internal SimpleLeakyConsumer(CancellationTokenSource serverShutDownSource, int operationDuration)
{
this.serverShutDownSource = serverShutDownSource;
this.operationDuration = operationDuration * 1000; // convert from seconds to milliseconds
}
internal async Task<bool> ProcessStateChange(string userId)
{
var executor = groupStateChangeExecutors.GetOrAdd(userId, new AsyncCollectionAbortableFifoQueue<bool>(serverShutDownSource.Token));
CancellationTokenSource oldSource = null;
using (var cancelSource = userStateChangeAborters.AddOrUpdate(userId, new CancellationTokenSource(), (key, existingValue) =>
{
oldSource = existingValue;
return new CancellationTokenSource();
}))
{
if (oldSource != null && !oldSource.IsCancellationRequested)
{
oldSource.Cancel();
_ = delayedDispose(oldSource);
}
try
{
var executionTask = executor.EnqueueTask(async () => { await Task.Delay(operationDuration, cancelSource.Token).ConfigureAwait(false); return true; }, cancelSource.Token);
var result = await executionTask.ConfigureAwait(false);
userStateChangeAborters.TryRemove(userId, out var aborter);
return result;
}
catch (Exception e)
{
if (e is TaskCanceledException || e is OperationCanceledException)
return true;
else
{
userStateChangeAborters.TryRemove(userId, out var aborter);
return false;
}
}
}
}
private async Task delayedDispose(CancellationTokenSource src)
{
try
{
await Task.Delay(20 * 1000).ConfigureAwait(false);
}
finally
{
try
{
src.Dispose();
}
catch (ObjectDisposedException) { }
}
}
}
In this sample implementation, all that is being done is wait, then return true.
To test this mechanism, I wrote the following Data producer class:
internal class SimpleProducer
{
//variables defining the test
readonly int nbOfusers = 10;
readonly int minimumDelayBetweenTest = 1; // seconds
readonly int maximumDelayBetweenTests = 6; // seconds
readonly int operationDuration = 3; // number of seconds an operation takes in the tester
private readonly Random rand;
private List<User> users;
private readonly SimpleLeakyConsumer consumer;
protected CancellationTokenSource serverShutDownSource, testAbortSource;
private CancellationToken internalToken = CancellationToken.None;
internal SimpleProducer()
{
rand = new Random();
testAbortSource = new CancellationTokenSource();
serverShutDownSource = new CancellationTokenSource();
generateTestObjects(nbOfusers, 0, false);
consumer = new SimpleLeakyConsumer(serverShutDownSource, operationDuration);
}
internal void StartTests()
{
if (internalToken == CancellationToken.None || internalToken.IsCancellationRequested)
{
internalToken = testAbortSource.Token;
foreach (var user in users)
_ = setNewUserPresence(internalToken, user);
}
}
internal void StopTests()
{
testAbortSource.Cancel();
try
{
testAbortSource.Dispose();
}
catch (ObjectDisposedException) { }
testAbortSource = new CancellationTokenSource();
}
internal void Shutdown()
{
serverShutDownSource.Cancel();
}
private async Task setNewUserPresence(CancellationToken token, User user)
{
while (!token.IsCancellationRequested)
{
var nextInterval = rand.Next(minimumDelayBetweenTest, maximumDelayBetweenTests);
try
{
await Task.Delay(nextInterval * 1000, testAbortSource.Token).ConfigureAwait(false);
}
catch (TaskCanceledException)
{
break;
}
//now randomly generate a new state and submit it to the tester class
UserState? status;
var nbStates = Enum.GetValues(typeof(UserState)).Length;
if (user.CurrentStatus == null)
{
var newInt = rand.Next(nbStates);
status = (UserState)newInt;
}
else
{
do
{
var newInt = rand.Next(nbStates);
status = (UserState)newInt;
}
while (status == user.CurrentStatus);
}
_ = sendUserStatus(user, status.Value);
}
}
private async Task sendUserStatus(User user, UserState status)
{
await consumer.ProcessStateChange(user.UserId).ConfigureAwait(false);
}
private void generateTestObjects(int nbUsers, int nbTeams, bool addAllUsersToTeams = false)
{
users = new List<User>();
for (int i = 0; i < nbUsers; i++)
{
var usr = new User
{
UserId = $"User_{i}",
Groups = new List<Team>()
};
users.Add(usr);
}
}
}
It uses the variables at the beginning of the class to control the test. You can define the number of users (nbOfusers - every user is a producer that produces new data), the minimum (minimumDelayBetweenTest) and maximum (maximumDelayBetweenTests) delay between a user producing the next data and how long it takes the consumer to process the data (operationDuration).
StartTests starts the actual test, and StopTests stops the tests again.
I'm calling these as follows
static void Main(string[] args)
{
var tester = new SimpleProducer();
Console.WriteLine("Test successfully started, type exit to stop");
string str;
do
{
str = Console.ReadLine();
if (str == "start")
tester.StartTests();
else if (str == "stop")
tester.StopTests();
}
while (str != "exit");
tester.Shutdown();
}
So, if I run my tester and type 'start', the Producer class starts producing states that are consumed by Consumer. And memory usage starts to grow and grow and grow. The sample is configured to the extreme, the real-life scenario I'm dealing with is less intensive, but one action of the producer could trigger multiple actions on the consumer side which also have to be executed in the same asynchronous abortable fifo fashion - so worst case, one set of data produced triggers an action for ~10 consumers (that last part I stripped out for brevity).
When I'm having a 100 producers, and each producer produces a new data item every 1-6 seconds (randomly, also the data produces is random). Consuming the data takes 3 seconds.. so there's plenty of cases where there's a new set of data before the old one has been properly processed.
Looking at two consecutive memory dumps, it's obvious where the memory usage is coming from.. it's all fragments that have to do with the queue. Given that I'm disposing every TaskCancellationSource and not keeping any references to the produced data (and the AsyncWorkItem they're put into), I'm at a loss to explain why this keeps eating up my memory and I'm hoping somebody else can show me the errors of my way. You can also abort testing by typing 'stop'.. you'll see that no longer is memory being eaten, but even if you pause and trigger GC, memory is not being freed either.
The source code of the project in runnable form is on Github. After starting it, you have to type start (plus enter) in the console to tell the producer to start producing data. And you can stop producing data by typing stop (plus enter)

Your code has so many issues making it impossible to find a leak through debugging. But here are several things that already are an issue and should be fixed first:
Looks like getQueue creates a new queue for the same user each time processUseStateUpdateAsync gets called and does not reuse existing queues:
var executor = groupStateChangeExecutors.GetOrAdd(user.UserId, getQueue());
CancellationTokenSource is leaking on each call of the code below, as new value created each time the method AddOrUpdate is called, it should not be passed there that way:
userStateChangeAborters.AddOrUpdate(user.UserId, new CancellationTokenSource(), (key, existingValue
Also code below should use the same cts as you pass as new cts, if dictionary has no value for specific user.UserId:
return new CancellationTokenSource();
Also there is a potential leak of cancelSource variable as it gets bound to a delegate which can live for a time longer than you want, it's better to pass concrete CancellationToken there:
executor.EnqueueTask(() => processUserStateUpdateAsync(user, state, previousState,
cancelSource.Token));
By some reason you do not dispose aborter here and in one more place:
userStateChangeAborters.TryRemove(user.UserId, out var aborter);
Creation of Channel can have potential leaks:
taskQueue = Channel.CreateBounded<AsyncWorkItem<T>>(new BoundedChannelOptions(1)
You picked option FullMode = BoundedChannelFullMode.DropOldest which should remove oldest values if there are any, so I assume that that stops queued items from processing as they would not be read. It's a hypotheses, but I assume that if an old item is removed without being handled, then processUserStateUpdateAsync won't get called and all resources won't be freed.
You can start with these found issues and it should be easier to find the real cause after that.

Non-blocking (lock-free) one-time initialization

Original question:
I need to initialize something only once in a multi-threaded application (when first thread enters the block). Subsequent threads should skip the initialization without waiting for it to complete.
I've found this blog entry Lock-free Thread Safe Initialisation in C# but it does not do exactly what I want, as it makes the other threads wait for the initialization to complete (if I understand it correctly).
Here's an example that presents the issue, although it does not work due to lack of synchronization:
using System;
using System.Collections.Concurrent;
using System.Threading.Tasks;
namespace LockFreeInitialization
{
public class Program
{
private readonly ConcurrentQueue<int> _jobsQueue = new ConcurrentQueue<int>();
private volatile bool _initialized;
private async Task EnqueueAndProcessJobsAsync(int taskId, int jobId)
{
Enqueue(taskId, jobId);
/* "Critical section"? Only the first thread to arrive should
* execute OneTimeInitAsync. Subsequent threads should always
* skip this part. This is where things go wrong as all the
* tasks execute this section due to lack of synchronization. */
if (!_initialized)
{
await OneTimeInitAsync(taskId);
}
/* Before and during initialization, all threads should skip
* the ProcessQueueAsync. After initialization is completed,
* it does not matter which thread will execute it (since the
* _jobsQueue is thread-safe). */
if (_initialized)
{
await ProcessQueueAsync(taskId);
}
Console.WriteLine($"Task {taskId} completed.");
}
private void Enqueue(int taskId, int jobId)
{
Console.WriteLine($"Task {taskId} enqueues job {jobId}.");
_jobsQueue.Enqueue(jobId);
}
private async Task OneTimeInitAsync(int taskId)
{
Console.WriteLine($"Task {taskId} is performing initialization");
/* Do some lengthy initialization */
await Task.Delay(TimeSpan.FromSeconds(3));
_initialized = true;
Console.WriteLine($"Task {taskId} completed initialization");
}
private async Task ProcessQueueAsync(int taskId)
{
while (_jobsQueue.TryDequeue(out int jobId))
{
/* Do something lengthy with the jobId */
await Task.Delay(TimeSpan.FromSeconds(1));
Console.WriteLine($"Task {taskId} completed job {jobId}.");
}
}
private static void Main(string[] args)
{
var p = new Program();
var rand = new Random();
/* Start 4 tasks in parallel */
for (var threadId = 1; threadId < 5; threadId++)
{
p.EnqueueAndProcessJobsAsync(threadId, rand.Next(10));
}
/* Give tasks chance to finish */
Console.ReadLine();
}
}
}
Both OneTimeInitAsync and ProcessQueueAsync are lengthy operations that in real-life scenario would communicate with some remote services. Using lock would block the other threads, while I want them to just pile up their work onto _jobsQueue and go their way. I tried using ManualResetEvent to no avail.
Does anyone know how would I make this work? Thanks in advance.
UPDATE (solution)
Based on the discussion below I understood that the scenario presented was not complete to describe my problem. However thanks to the answers and comments I thought about redesigning the solution a bit, to work as I wanted it to.
So imagine two remote services ServiceA (job processor) and ServiceB (job repository) that the client app has to communicate with. We need to establish a connection to ServiceA, meanwhile we get data for multiple jobs from ServiceB. As the jobs data become available, we process jobs (in batches) using ServiceA (real-life example involves Signal-R connection to ServiceA and some job IDs from ServiceB that need to be sent to ServiceA). Here's the code example:
public class StackOverflowSolution
{
private readonly ConcurrentQueue<int> _jobsQueue = new ConcurrentQueue<int>();
/* Just to randomize waiting times */
private readonly Random _random = new Random();
/* Instance-scoped one-time initialization of a remote ServiceA connection */
private async Task<string> InitializeConnectionAsync()
{
Console.WriteLine($"{nameof(InitializeConnectionAsync)} started");
await Task.Delay(TimeSpan.FromSeconds(_random.Next(5) + 1));
Console.WriteLine($"{nameof(InitializeConnectionAsync)} completed");
return "Connection";
}
/* Preparation of a job (assume it requires communication with remote ServiceB) */
private async Task InitializeJobAsync(int id)
{
Console.WriteLine($"{nameof(InitializeJobAsync)}({id}) started");
await Task.Delay(TimeSpan.FromSeconds(_random.Next(10) + 1));
_jobsQueue.Enqueue(id);
Console.WriteLine($"{nameof(InitializeJobAsync)}({id}) completed");
}
/* Does something to the ready jobs in the _jobsQueue using connection to
* ServiceA */
private async Task ProcessQueueAsync(string connection)
{
var sb = new StringBuilder("Processed ");
bool any = false;
while (_jobsQueue.TryDequeue(out int idResult))
{
any = true;
sb.Append($"{idResult}, ");
}
if (any)
{
await Task.Delay(TimeSpan.FromMilliseconds(_random.Next(500)));
Console.WriteLine(sb.ToString());
}
}
/* Orchestrates the processing */
public async Task RunAsync()
{
/* Start initializing the conection to ServiceA */
Task<string> connectionTask = InitializeConnectionAsync();
/* Start initializing jobs */
var jobTasks = new List<Task>();
foreach (int id in new[] {1, 2, 3, 4})
{
jobTasks.Add(InitializeJobAsync(id));
}
/* Wait for initialization to complete */
string connection = await connectionTask;
/* Trigger processing of jobs as they become ready */
var queueProcessingTasks = new List<Task>();
while (jobTasks.Any())
{
jobTasks.Remove(await Task.WhenAny(jobTasks));
queueProcessingTasks.Add(ProcessQueueAsync(connection));
}
await Task.WhenAll(queueProcessingTasks);
}
public static void Main()
{
new StackOverflowSolution().RunAsync().Wait();
}
}
Output example:
InitializeConnectionAsync started
InitializeJobAsync(1) started
InitializeJobAsync(2) started
InitializeJobAsync(3) started
InitializeJobAsync(4) started
InitializeJobAsync(5) started
InitializeJobAsync(3) completed
InitializeJobAsync(2) completed
InitializeConnectionAsync completed
Processed 3, 2,
InitializeJobAsync(1) completed
Processed 1,
InitializeJobAsync(5) completed
Processed 5,
InitializeJobAsync(4) completed
Processed 4,
Thanks for all the feedback!

Honestly the semantics of EnqueueAndProcessJobsAsync for your code is simply not a good idea, given your description of what you're actually doing and what you actually need.
Currently the Task returned from EnqueueAndProcessJobsAsync waits for initialization if initialization wasn't started by someone else, then it completes whenever the queue is empty, or as soon as this logical call context happened to process an item that errored. That...just doesn't make sense.
What you clearly want is for that Task to be completed whenever the job passed in completed (which will of course require initialization to complete), or to be errored if that job errors, and to be unaffected by any other job's errors. Fortunately in addition to being much more useful, it's also far easier to do.
As far as the actual initialization goes, you can just use a Lazy<Task> to ensure proper synchronization of the asynchronous initialization, and to expose the Task to any future calls which can tell them when initialization has finished.
public class MyAsyncQueueRequireingInitialization
{
private readonly Lazy<Task> whenInitialized;
public MyAsyncQueueRequireingInitialization()
{
whenInitialized = new Lazy<Task>(OneTimeInitAsync);
}
//as noted in comments, the taskID isn't actually needed for initialization
private async Task OneTimeInitAsync()
{
Console.WriteLine($"Performing initialization");
/* Do some lengthy initialization */
await Task.Delay(TimeSpan.FromSeconds(3));
Console.WriteLine($"Completed initialization");
}
public async Task ProcessJobAsync(int taskID, int jobId)
{
await whenInitialized.Value;
/* Do something lengthy with the jobId */
await Task.Delay(TimeSpan.FromSeconds(1));
Console.WriteLine($"Completed job {jobId}.");
}
}

As noted in the comments to the OP, a better solution might be to do the initialization in single-threaded mode, and then launch the threads that do the actual work.
If that doesn't work for you, you'll need a lock of some sort - but you can use that lock just for scheduling, to make it less blocking. I would implement something like this:
private bool _initializationIsScheduled = false;
private object _initializationIsScheduledLock = new object();
private bool _isInitialized = false;
private object _isInitializedLock = new object();
private async Task EnqueueAndProcessJobs(int taskId, int jobId)
{
var shouldDoHeavyWork = false;
lock(_initializationIsScheduledLock)
{
if (!_initializationIsScheduled)
{
shouldDoHeavyWork = true;
_initializationIsScheduled= true;
}
}
if (shouldDoHeavyWork)
{
await OneTimeInitAsync(taskId);
lock (_isInitializedLock)
{
_isInitialized = true;
}
}
lock (_isInitializedLock)
{
if (_isInitialized)
{
shouldDoHeavyWork = true;
}
}
if (shouldDoHeavyWork)
{
await ProcessQueueAsync(taskId);
}
Console.WriteLine($"Task {taskId} completed.");
}
Note how the only time the thread locks up the other threads, is when it's about to check or set one of the flags that controls its work. In other words, threads don't have to wait for each other when actually doing heavy work, just when scheduling it (i.e. for a couple of CPU cycles when setting the boolean flags).
The code isn't exactly beautiful, but you should be able to refactor the above sample into something that's reasonably legible... :)

Maybe you can try something like this:
static bool IsInitializing = false;
static int FirstThreadId = -1;
// check if initialised
// return if initialised
// somewhere in init mehtod
lock (lockObject)
{
// first method start initializing
IsInitializing = true;
// set some id to the thread that start initializtion
FirstThreadId = THIS_THREAD_OR_TASK_ID;
}
if (IsInitializing && FristThread != THIS_THREAD_OR_TASK_ID)
return; // skip initializing
lock must work quick, yeap

Ensure a long running task is only fired once and subsequent request are queued but with only one entry in the queue

I have a compute intensive method Calculate that may run for a few seconds, requests come from multiple threads.
Only one Calculate should be executing, a subsequent request should be queued until the initial request completes. If there is already a request queued then the the subsequent request can be discarded (as the queued request will be sufficient)
There seems to be lots of potential solutions but I just need the simplest.
UPDATE: Here's my rudimentaryattempt:
private int _queueStatus;
private readonly object _queueStatusSync = new Object();
public void Calculate()
{
lock(_queueStatusSync)
{
if(_queueStatus == 2) return;
_queueStatus++;
if(_queueStatus == 2) return;
}
for(;;)
{
CalculateImpl();
lock(_queueStatusSync)
if(--_queueStatus == 0) return;
}
}
private void CalculateImpl()
{
// long running process will take a few seconds...
}

The simplest, cleanest solution IMO is using TPL Dataflow (as always) with a BufferBlock acting as the queue. BufferBlock is thread-safe, supports async-await, and more important, has TryReceiveAll to get all the items at once. It also has OutputAvailableAsync so you can wait asynchronously for items to be posted to the buffer. When multiple requests are posted you simply take the last and forget about the rest:
var buffer = new BufferBlock<Request>();
var task = Task.Run(async () =>
{
while (await buffer.OutputAvailableAsync())
{
IList<Request> requests;
buffer.TryReceiveAll(out requests);
Calculate(requests.Last());
}
});
Usage:
buffer.Post(new Request());
buffer.Post(new Request());
Edit: If you don't have any input or output for the Calculate method you can simply use a boolean to act as a switch. If it's true you can turn it off and calculate, if it became true again while Calculate was running then calculate again:
public bool _shouldCalculate;
public void Producer()
{
_shouldCalculate = true;
}
public async Task Consumer()
{
while (true)
{
if (!_shouldCalculate)
{
await Task.Delay(1000);
}
else
{
_shouldCalculate = false;
Calculate();
}
}
}

A BlockingCollection that only takes 1 at a time
The trick is to skip if there are any items in the collection
I would go with the answer from I3aron +1
This is (maybe) a BlockingCollection solution
public static void BC_AddTakeCompleteAdding()
{
using (BlockingCollection<int> bc = new BlockingCollection<int>(1))
{
// Spin up a Task to populate the BlockingCollection
using (Task t1 = Task.Factory.StartNew(() =>
{
for (int i = 0; i < 100; i++)
{
if (bc.TryAdd(i))
{
Debug.WriteLine(" add " + i.ToString());
}
else
{
Debug.WriteLine(" skip " + i.ToString());
}
Thread.Sleep(30);
}
bc.CompleteAdding();
}))
{
// Spin up a Task to consume the BlockingCollection
using (Task t2 = Task.Factory.StartNew(() =>
{
try
{
// Consume consume the BlockingCollection
while (true)
{
Debug.WriteLine("take " + bc.Take());
Thread.Sleep(100);
}
}
catch (InvalidOperationException)
{
// An InvalidOperationException means that Take() was called on a completed collection
Console.WriteLine("That's All!");
}
}))
Task.WaitAll(t1, t2);
}
}
}

It sounds like a classic producer-consumer. I'd recommend looking into BlockingCollection<T>. It is part of the System.Collection.Concurrent namespace. On top of that you can implement your queuing logic.
You may supply to a BlockingCollection any internal structure to hold its data, such as a ConcurrentBag<T>, ConcurrentQueue<T> etc. The latter is the default structure used.

How to limit number of HttpWebRequest per second towards a webserver?

I need to implement a throttling mechanism (requests per second) when using HttpWebRequest for making parallel requests towards one application server. My C# app must issue no more than 80 requests per second to a remote server. The limit is imposed by the remote service admins not as a hard limit but as "SLA" between my platform and theirs.
How can I control the number of requests per second when using HttpWebRequest?

I had the same problem and couldn't find a ready solution so I made one, and here it is. The idea is to use a BlockingCollection<T> to add items that need processing and use Reactive Extensions to subscribe with a rate-limited processor.
Throttle class is the renamed version of this rate limiter
public static class BlockingCollectionExtensions
{
// TODO: devise a way to avoid problems if collection gets too big (produced faster than consumed)
public static IObservable<T> AsRateLimitedObservable<T>(this BlockingCollection<T> sequence, int items, TimeSpan timePeriod, CancellationToken producerToken)
{
Subject<T> subject = new Subject<T>();
// this is a dummyToken just so we can recreate the TokenSource
// which we will pass the proxy class so it can cancel the task
// on disposal
CancellationToken dummyToken = new CancellationToken();
CancellationTokenSource tokenSource = CancellationTokenSource.CreateLinkedTokenSource(producerToken, dummyToken);
var consumingTask = new Task(() =>
{
using (var throttle = new Throttle(items, timePeriod))
{
while (!sequence.IsCompleted)
{
try
{
T item = sequence.Take(producerToken);
throttle.WaitToProceed();
try
{
subject.OnNext(item);
}
catch (Exception ex)
{
subject.OnError(ex);
}
}
catch (OperationCanceledException)
{
break;
}
}
subject.OnCompleted();
}
}, TaskCreationOptions.LongRunning);
return new TaskAwareObservable<T>(subject, consumingTask, tokenSource);
}
private class TaskAwareObservable<T> : IObservable<T>, IDisposable
{
private readonly Task task;
private readonly Subject<T> subject;
private readonly CancellationTokenSource taskCancellationTokenSource;
public TaskAwareObservable(Subject<T> subject, Task task, CancellationTokenSource tokenSource)
{
this.task = task;
this.subject = subject;
this.taskCancellationTokenSource = tokenSource;
}
public IDisposable Subscribe(IObserver<T> observer)
{
var disposable = subject.Subscribe(observer);
if (task.Status == TaskStatus.Created)
task.Start();
return disposable;
}
public void Dispose()
{
// cancel consumption and wait task to finish
taskCancellationTokenSource.Cancel();
task.Wait();
// dispose tokenSource and task
taskCancellationTokenSource.Dispose();
task.Dispose();
// dispose subject
subject.Dispose();
}
}
}
Unit test:
class BlockCollectionExtensionsTest
{
[Fact]
public void AsRateLimitedObservable()
{
const int maxItems = 1; // fix this to 1 to ease testing
TimeSpan during = TimeSpan.FromSeconds(1);
// populate collection
int[] items = new[] { 1, 2, 3, 4 };
BlockingCollection<int> collection = new BlockingCollection<int>();
foreach (var i in items) collection.Add(i);
collection.CompleteAdding();
IObservable<int> observable = collection.AsRateLimitedObservable(maxItems, during, CancellationToken.None);
BlockingCollection<int> processedItems = new BlockingCollection<int>();
ManualResetEvent completed = new ManualResetEvent(false);
DateTime last = DateTime.UtcNow;
observable
// this is so we'll receive exceptions
.ObserveOn(new SynchronizationContext())
.Subscribe(item =>
{
if (item == 1)
last = DateTime.UtcNow;
else
{
TimeSpan diff = (DateTime.UtcNow - last);
last = DateTime.UtcNow;
Assert.InRange(diff.TotalMilliseconds,
during.TotalMilliseconds - 30,
during.TotalMilliseconds + 30);
}
processedItems.Add(item);
},
() => completed.Set()
);
completed.WaitOne();
Assert.Equal(items, processedItems, new CollectionEqualityComparer<int>());
}
}

The Throttle() and Sample() extension methods (On Observable) allow you to regulate a fast sequence of events into a "slower" sequence.
Here is a blog post with an example of Sample(Timespan) that ensures a maxium rate.

My original post discussed how to add a throttling mechanism to WCF via client behavior extensions, but then was pointed out that I misread the question (doh!).
Overall the approach can be to check with a class that determines if we are violating the rate limit or not. There's already been a lot of discussion around how to check for rate violations.
Throttling method calls to M requests in N seconds
If you are violating the rate limit, then sleep for a fix interval and check again. If not, then go ahead and make the HttpWebRequest call.

How should i write producer /consumer code in C#?

I have 1 thread streaming data and a 2nd (the threadpool) processing the data. The data processing takes around 100ms so I use to second thread so not to hold up the 1st thread.
While the 2nd thread is processing the data the 1st thread adds the data to a dictionary cache then when the 2nd thread is finished it processes the cached values.
My questions is this how should be doing producer /consumer code in C#?
public delegate void OnValue(ulong value);
public class Runner
{
public event OnValue OnValueEvent;
private readonly IDictionary<string, ulong> _cache = new Dictionary<string, ulong>(StringComparer.InvariantCultureIgnoreCase);
private readonly AutoResetEvent _cachePublisherWaitHandle = new AutoResetEvent(true);
public void Start()
{
for (ulong i = 0; i < 500; i++)
{
DataStreamHandler(i.ToString(), i);
}
}
private void DataStreamHandler(string id, ulong value)
{
_cache[id] = value;
if (_cachePublisherWaitHandle.WaitOne(1))
{
IList<ulong> tempValues = new List<ulong>(_cache.Values);
_cache.Clear();
_cachePublisherWaitHandle.Reset();
ThreadPool.UnsafeQueueUserWorkItem(delegate
{
try
{
foreach (ulong value1 in tempValues)
if (OnValueEvent != null)
OnValueEvent(value1);
}
finally
{
_cachePublisherWaitHandle.Set();
}
}, null);
}
else
{
Console.WriteLine(string.Format("Buffered value: {0}.", value));
}
}
}
class Program
{
static void Main(string[] args)
{
Stopwatch sw = Stopwatch.StartNew();
Runner r = new Runner();
r.OnValueEvent += delegate(ulong x)
{
Console.WriteLine(string.Format("Processed value: {0}.", x));
Thread.Sleep(100);
if(x == 499)
{
sw.Stop();
Console.WriteLine(string.Format("Time: {0}.", sw.ElapsedMilliseconds));
}
};
r.Start();
Console.WriteLine("Done");
Console.ReadLine();
}
}

The best practice for setting up the producer-consumer pattern is to use the BlockingCollection class which is available in .NET 4.0 or as a separate download of the Reactive Extensions framework. The idea is that the producers will enqueue using the Add method and the consumers will dequeue using the Take method which blocks if the queue is empty. Like SwDevMan81 pointed out the Albahari site has a really good writeup on how to make it work correctly if you want to go the manual route.

There is a good article on MSDN about Synchronizing the Producer and Consumer. There is also a good example on the Albahari site.