Develop Reference

Receive concurrent asynchronous requests and process them one at a time

Background
We have a service operation that can receive concurrent asynchronous requests and must process those requests one at a time.
In the following example, the UploadAndImport(...) method receives concurrent requests on multiple threads, but its calls to the ImportFile(...) method must happen one at a time.
Layperson Description
Imagine a warehouse with many workers (multiple threads). People (clients) can send the warehouse many packages (requests) at the same time (concurrently). When a package comes in a worker takes responsibility for it from start to finish, and the person who dropped off the package can leave (fire-and-forget). The workers' job is to put each package down a small chute, and only one worker can put a package down a chute at a time, otherwise chaos ensues. If the person who dropped off the package checks in later (polling endpoint), the warehouse should be able to report on whether the package went down the chute or not.
Question
The question then is how to write a service operation that...
can receive concurrent client requests,
receives and processes those requests on multiple threads,
processes requests on the same thread that received the request,
processes requests one at a time,
is a one way fire-and-forget operation, and
has a separate polling endpoint that reports on request completion.
We've tried the following and are wondering two things:
Are there any race conditions that we have not considered?
Is there a more canonical way to code this scenario in C#.NET with a service oriented architecture (we happen to be using WCF)?
Example: What We Have Tried?
This is the service code that we have tried. It works though it feels like somewhat of a hack or kludge.
static ImportFileInfo _inProgressRequest = null;
static readonly ConcurrentDictionary<Guid, ImportFileInfo> WaitingRequests =
new ConcurrentDictionary<Guid, ImportFileInfo>();
public void UploadAndImport(ImportFileInfo request)
{
// Receive the incoming request
WaitingRequests.TryAdd(request.OperationId, request);
while (null != Interlocked.CompareExchange(ref _inProgressRequest, request, null))
{
// Wait for any previous processing to complete
Thread.Sleep(500);
}
// Process the incoming request
ImportFile(request);
Interlocked.Exchange(ref _inProgressRequest, null);
WaitingRequests.TryRemove(request.OperationId, out _);
}
public bool UploadAndImportIsComplete(Guid operationId) =>
!WaitingRequests.ContainsKey(operationId);
This is example client code.
private static async Task UploadFile(FileInfo fileInfo, ImportFileInfo importFileInfo)
{
using (var proxy = new Proxy())
using (var stream = new FileStream(fileInfo.FullName, FileMode.Open, FileAccess.Read))
{
importFileInfo.FileByteStream = stream;
proxy.UploadAndImport(importFileInfo);
}
await Task.Run(() => Poller.Poll(timeoutSeconds: 90, intervalSeconds: 1, func: () =>
{
using (var proxy = new Proxy())
{
return proxy.UploadAndImportIsComplete(importFileInfo.OperationId);
}
}));
}
It's hard to write a minimum viable example of this in a Fiddle, but here is a start that give a sense and that compiles.
As before, the above seems like a hack/kludge, and we are asking both about potential pitfalls in its approach and for alternative patterns that are more appropriate/canonical.

Simple solution using Producer-Consumer pattern to pipe requests in case of thread count restrictions.
You still have to implement a simple progress reporter or event. I suggest to replace the expensive polling approach with an asynchronous communication which is offered by Microsoft's SignalR library. It uses WebSocket to enable async behavior. The client and server can register their callbacks on a hub. Using RPC the client can now invoke server side methods and vice versa. You would post progress to the client by using the hub (client side). In my experience SignalR is very simple to use and very good documented. It has a library for all famous server side languages (e.g. Java).
Polling in my understanding is the totally opposite of fire-and-forget. You can't forget, because you have to check something based on an interval. Event based communication, like SignalR, is fire-and-forget since you fire and will get a reminder (cause you forgot). The "event side" will invoke your callback instead of you waiting to do it yourself!
Requirement 5 is ignored since I didn't get any reason. Waiting for a thread to complete would eliminate the fire and forget character.
private BlockingCollection<ImportFileInfo> requestQueue = new BlockingCollection<ImportFileInfo>();
private bool isServiceEnabled;
private readonly int maxNumberOfThreads = 8;
private Semaphore semaphore = new Semaphore(numberOfThreads);
private readonly object syncLock = new object();
public void UploadAndImport(ImportFileInfo request)
{
// Start the request handler background loop
if (!this.isServiceEnabled)
{
this.requestQueue?.Dispose();
this.requestQueue = new BlockingCollection<ImportFileInfo>();
// Fire and forget (requirement 4)
Task.Run(() => HandleRequests());
this.isServiceEnabled = true;
}
// Cache multiple incoming client requests (requirement 1) (and enable throttling)
this.requestQueue.Add(request);
}
private void HandleRequests()
{
while (!this.requestQueue.IsCompleted)
{
// Wait while thread limit is exceeded (some throttling)
this.semaphore.WaitOne();
// Process the incoming requests in a dedicated thread (requirement 2) until the BlockingCollection is marked completed.
Task.Run(() => ProcessRequest());
}
// Reset the request handler after BlockingCollection was marked completed
this.isServiceEnabled = false;
this.requestQueue.Dispose();
}
private void ProcessRequest()
{
ImportFileInfo request = this.requestQueue.Take();
UploadFile(request);
// You updated your question saying the method "ImportFile()" requires synchronization.
// This a bottleneck and will significantly drop performance, when this method is long running.
lock (this.syncLock)
{
ImportFile(request);
}
this.semaphore.Release();
}
Remarks:
BlockingCollection is a IDisposable
TODO: You have to "close" the BlockingCollection by marking it completed:
"BlockingCollection.CompleteAdding()" or it will loop indeterminately waiting for further requests. Maybe you introduce a additional request methods for the client to cancel and/ or to update the process and to mark adding to the BlockingCollection as completed. Or a timer that waits an idle time before marking it as completed. Or make your request handler thread block or spin.
Replace Take() and Add(...) with TryTake(...) and TryAdd(...) if you want cancellation support
Code is not tested
Your "ImportFile()" method is a bottleneck in your multi threading environment. I suggest to make it thread safe. In case of I/O that requires synchronization, I would cache the data in a BlockingCollection and then write them to I/O one by one.

The problem is that your total bandwidth is very small-- only one job can run at a time-- and you want to handle parallel requests. That means that queue time could vary wildly. It may not be the best choice to implement your job queue in-memory, as it would make your system much more brittle, and more difficult to scale out when your business grows.
A traditional, scaleable way to architect this would be:
An HTTP service to accept requests, load balanced/redundant, with no session state.
A SQL Server database to persist the requests in a queue, returning a persistent unique job ID.
A Windows service to process the queue, one job at a time, and mark jobs as complete. The worker process for the service would probably be single-threaded.
This solution requires you to choose a web server. A common choice is IIS running ASP.NET. On that platform, each request is guaranteed to be handled in a single-threaded manner (i.e. you don't need to worry about race conditions too much), but due to a feature called thread agility the request might end with a different thread, but in the original synchronization context, which means you will probably never notice unless you are debugging and inspecting thread IDs.

Given the constraints context of our system, this is the implementation we ended up using:
static ImportFileInfo _importInProgressItem = null;
static readonly ConcurrentQueue<ImportFileInfo> ImportQueue =
new ConcurrentQueue<ImportFileInfo>();
public void UploadAndImport(ImportFileInfo request) {
UploadFile(request);
ImportFileSynchronized(request);
}
// Synchronize the file import,
// because the database allows a user to perform only one write at a time.
private void ImportFileSynchronized(ImportFileInfo request) {
ImportQueue.Enqueue(request);
do {
ImportQueue.TryPeek(out var next);
if (null != Interlocked.CompareExchange(ref _importInProgressItem, next, null)) {
// Queue processing is already under way in another thread.
return;
}
ImportFile(next);
ImportQueue.TryDequeue(out _);
Interlocked.Exchange(ref _importInProgressItem, null);
}
while (ImportQueue.Any());
}
public bool UploadAndImportIsComplete(Guid operationId) =>
ImportQueue.All(waiting => waiting.OperationId != operationId);
This solution works well for the loads we are expecting. That load involves a maximum of about 15-20 concurrent PDF file uploads. The batch of up to 15-20 files tends to arrive all at once and then to go quiet for several hours until the next batch arrives.
Criticism and feedback is most welcome.

Understanding fire and forget when using infinite loops

Can someone tell me what the best practice/proper way of doing this is?
I'm also using WPF, not a console or ASP.NET.
Using Listener to accept clients and spin off a new "thread" for each client that handles all the I/O and Exception catching for that client.
Method 1: Fire and forget, and just throw it into a variable to get rid of the warning.
public static async Task Start(CancellationToken token)
{
m_server = TcpListener.Create(33777);
m_server.Start();
running = true;
clientCount = 0;
// TODO: Add try... catch
while (!token.IsCancellationRequested)
{
var client = await m_server.AcceptTcpClientAsync().ConfigureAwait(false);
Client c = new Client(client);
var _ = HandleClientAsync(c);
}
}
Here's the Client Handler code:
public static async Task HandleClientAsync(Client c)
{
// TODO: add try...catch
while (c.connected)
{
string data = await c.reader.ReadLineAsync();
// Now we will parse the data and update variables accordingly
// Just Regex and some parsing that updates variables
ParseAndUpdate(data);
}
}
Method 2: The same thing... but with Task.Run()
var _ = Task.Run(() => HandleClientAsync());
Method 3: an intermediate non async function (doubt this is good. Should be Async all the way)
But this at least gets rid of the squiggly line without using the variable trick which kinda feels dirty.
while (!token.IsCancellationRequested)
{
var client = await m_server.AcceptTcpClientAsync().ConfigureAwait(false);
Client c = new Client(client);
NonAsync(c);
}
public static void NonAsync(VClient vc)
{
Task.Run(() => HandleClientAsync(vc));
}
Method 4: Make HandleClientAsync an Async void instead of Async Task (really bad)
public static async Task HandleClientAsync(Client c)
// Would change to
public static async Void HandleClientAsync(Client c)
Questions:
Is it any better to use Task.Run() When doing a fire and forget task?
Is it just accepted that you need to use the var _ = FireAndForget() trick to do fire and forget? I could just ignore the warning but something feels wrong about it.
If I wanted to update my UI from a Client, how would I do that? Would I just use a dispatcher?
Thanks guys

I've never been a fan of background workers which you expect to run for a long time, being run in a task. Tasks get scheduled to run on threads drawn from a pool. As you schedule these long running tasks, the thread pool gets smaller and smaller. Eventually all of the threads from the pool are busy running your tasks, and things get really slow and unmanageable.
My recommendation here? Use the Thread class and manage them yourselves. In this way, you keep your thread pool and the overhead for for tasks out of the picture.
Addendum - Producer Consumer Model
Another interesting question to consider: Do you really need a thread for every client? Threads are reasonably costly to create and maintain in terms of memory overhead, and if your client interaction is such that the client threads spend the vast majority of their time waiting around on something to do, then perhaps a producer consumer model is more suited to your use case.
Example:
Client connects on listening thread, gets put in a client queue
Worker thread responsible for checking to see if the clients need anything comes along through that queue every so often and checks - does the client have a new message to service? If so, it services all messages the client has, then moves on
In this way, you limit the number of threads working to just the number needed to manage the message queue. You can even get fancy and add worker threads dynamically based on how long its been since all the clients have been serviced.
If you insist
If you really like what you have going, I suggest refactoring what youre doing a bit so that rather than HandleClientAsync you do something more akin to CreateServiceForClient(c);
This could be a synchronous method that returns something like a ClientService. ClientService could then create the task that does what your HandleClientAsync does now, and store that task as a member. It could also provide methods like
ClientService.WaitUntilEnd()
and
ClientService.Disconnect() (which could set a cancellation token, also stored as a member variable)

Is it correct to delay a Task using TimeSpan.FromTicks?

My program needs to constantly perform many repetitive calculations as fast as possible. There are many tasks running parallelly which cause CPU utilisation is at 100%. To let users slow down processing overload(a little under 100% of CPU, depending on hardware), I added
await Task.Delay(TimeSpan.FromMilliseconds(doubleProcessingCycleIntervalMilliseconds));
to heavy processing methods. This works perfect as far as value of doubleProcessingCycleIntervalMilliseconds is at least 1 ms.
For users who have high-end computers(calculations speed will take less than one millisecond), I wanted to add same option for delay but instead of milliseconds using ticks. So now code looks:
if (ProcessingCycleIntervalOptionsMilliseconds == true)
{
await Task.Delay(TimeSpan.FromMilliseconds(doubleProcessingCycleIntervalMilliseconds));
}
else
{
await Task.Delay(TimeSpan.FromTicks(longProcessingCycleIntervalTicks));
}
When walue of longProcessingCycleIntervalTicks is at least 10000 ticks(=1ms) program works perfect. Unfortunately when values go under 1ms(0 for doubleProcessingCycleIntervalMilliseconds which I can understand) or under 10000(i.e. 9999 for longProcessingCycleIntervalTicks) program becomes not responsive. So literally difference of 1 tick below 1ms hangs the program. I don't use MVVM. (Just in case: I checked Stopwatch.IsHighResolution gives true on the development computer)
Is it possible/correct to use
await Task.Delay(TimeSpan.FromTicks(longProcessingCycleIntervalTicks));
in .NET 4.5.1 ? If yes, then how to determine when user can use it?

Your intention is not to keep CPU utilization below 100%. Your intention is to keep the system responsive. Limiting CPU utilization is a misguided goal.
The way you do this is by using low priority threads. Use a custom task scheduler for your CPU bound tasks.
Timing in Windows has limited accuracy. Thread.Sleep cannot work with fractional milliseconds. .NET rounds them away before handing over to Sleep.

You might be better off looking at the way you are performing the tasks rather than trying to sleep them.
The best way I can think of is by using a task manager to manage each task independently (such as a background worker) and then thread collections of tasks.
This would enable you to manage how many tasks are running instead of trying to 'slow' them down..
i.e
public class Task<returnType>
{
public delegate returnType funcTask(params object[] args);
public delegate void returnCallback(returnType ret);
public funcTask myTask;
public event returnCallback Callback;
public Task(funcTask myTask, returnCallback Callback)
{
this.myTask = myTask;
this.Callback = Callback;
}
public void DoWork(params object[] args)
{
if (this.Callback != null)
{
this.Callback(myTask(args));
}
else
{
throw new Exception("no Callback!");
}
}
}
Then you need a manager that has a Queue in it of the tasks you want to complete, call myQueue.Enqueue to queue, myQueue.Dequeue to run the tasks. Basically you can use the already built-in Queue to do this.
You then can create a Queue of task managers full of tasks and have them all run asychronously, and stack nicely on the CPU as they are event driven and the OS and .NET will sort out the rest.
EDIT:
To continuously run tasks you will need to create a class that inherits the Queue class, then call an event when something is de-queued. The reasoning behind why I say to use events is that they stack on the CPU.
For a neverending stackable 'Loop' something like this would work...
public class TaskManager<T> : Queue<T>
{
public delegate void taskDequeued();
public event taskDequeued OnTaskDequeued;
public override T Dequeue()
{
T ret = base.Dequeue();
if (OnTaskDequeued != null) OnTaskDequeued();
return ret;
}
}
In your function that instantiates the 'loop' you need to do something like...
TaskManager<Task<int>> tasks = new TaskManager<Task<int>>();
Task<int> task = new Task<int>(i => 3 + 4, WriteIntToScreen); // WriteIntToScreen is a fake function to use as the callback
tasks.Enqueue(task);
tasks.OnTaskDequeued += delegate
{
tasks.Enqueue(task);
tasks.Dequeue.Invoke();
};
// start the routine with
tasks.Dequeue.Invoke(); // you call do some async threading here with BeginInvoke or something but I am not gonna write all that out as it will be pages...
To cancel you just empty the queue.

Alternatives to giving a task X amount of time to finish

It's often the case when I need to get a task done within X amount of seconds and if it doesn't complete, I want to keep going, processing the rest of the tasks
I've been defaulting to something like this:
Thread worker = new Thread(() => {
// do some long operation
});
Thread monitor = new Thread(() => {
Thread.Sleep(10000);
if(worker != null && worker.IsAlive) {
worker.Abort();
worker = null;
}
StartNextTask();
});
monitor.Start ();
worker.Start();
This works, but it is cumbersome because it uses two threads (yes, you can use the Task class as well to use threads from the threadpool).
AutoResetEvents and event based models don't quite work because the former blocks the monitor thread until the worker is done, and the event driven approach relies on the client to call the event and notify the monitor.
Are there alternative patterns to this that can follow the same semantics?

If you use the Task class as you say, you can also use Task.Wait() which does exactly what you want. Specify an amount of seconds to wait. The task doesn't get cancelled unless you cancel it by use of a CancellationToken
See: http://msdn.microsoft.com/en-us/library/dd235606.aspx

Task Parallel Library - Custom Task Schedulers

I have a requirement to fire off web service requests to an online api and I thought that Parallel Extensions would be a good fit for my needs.
The web service in question is designed to be called repeatedly, but has a mechanism that charges you if you got over a certain number of calls per second. I obviously want to minimize my charges and so was wondering if anyone has seen a TaskScheduler that can cope with the following requirements:
Limit the number of tasks scheduled per timespan. I guess if the number of requests exceeded this limit then it would need to throw away the task or possibly block? (to stop a back log of tasks)
Detect if the same request is already in the scheduler to be executed but hasn't been yet and if so not queue the second task but return the first instead.
Do people feel that these are the sorts of responsibilities a task scheduler should be dealing with or am i barking up the wrong tree? If you have alternatives I am open to suggestions.

I agree with others that TPL Dataflow sounds like a good solution for this.
To limit the processing, you could create a TransformBlock that doesn't actually transform the data in any way, it just delays it if it arrived too soon after the previous data:
static IPropagatorBlock<T, T> CreateDelayBlock<T>(TimeSpan delay)
{
DateTime lastItem = DateTime.MinValue;
return new TransformBlock<T, T>(
async x =>
{
var waitTime = lastItem + delay - DateTime.UtcNow;
if (waitTime > TimeSpan.Zero)
await Task.Delay(waitTime);
lastItem = DateTime.UtcNow;
return x;
},
new ExecutionDataflowBlockOptions { BoundedCapacity = 1 });
}
Then create a method that produces the data (for example integers starting from 0):
static async Task Producer(ITargetBlock<int> target)
{
int i = 0;
while (await target.SendAsync(i))
i++;
}
It's written asynchronously, so that if the target block isn't able to process the items right now, it will wait.
Then write a consumer method:
static void Consumer(int i)
{
Console.WriteLine(i);
}
And finally, link it all together and start it up:
var delayBlock = CreateDelayBlock<int>(TimeSpan.FromMilliseconds(500));
var consumerBlock = new ActionBlock<int>(
(Action<int>)Consumer,
new ExecutionDataflowBlockOptions { MaxDegreeOfParallelism = DataflowBlockOptions.Unbounded });
delayBlock.LinkTo(consumerBlock, new DataflowLinkOptions { PropagateCompletion = true });
Task.WaitAll(Producer(delayBlock), consumerBlock.Completion);
Here, delayBlock will accept at most one item every 500 ms and the Consumer() method can run multiple times in parallel. To finish processing, call delayBlock.Complete().
If you want to add some caching per your #2, you could create another TransformBlock do the work there and link it to the other blocks.

Honestly I would work at a higher level of abstraction and use the TPL Dataflow API for this. The only catch is you would need to write a custom block that will throttle the requests at the rate at which you need because, by default, blocks are "greedy" and will just process as fast as possible. The implementation would be something like this:
Start with a BufferBlock<T> which is the logical block that you would post to.
Link the BufferBlock<T> to a custom block which has the knowledge of requests/sec and throttling logic.
Link the custom block from 2 to to your ActionBlock<T>.
I don't have the time to write the custom block for #2 right this second, but I will check back later and try to fill in an implementation for you if you haven't already figured it out.

I haven't used RX much, but AFAICT the Observable.Window method would work fine for this.
http://msdn.microsoft.com/en-us/library/system.reactive.linq.observable.window(VS.103).aspx
It would seem to be a better fit than Throttle which seems to throw elements away, which I'm guessing is not what you want

If you need to throttle by time, you should check out Quartz.net. It can facilitate consistent polling. If you care about all requests, you should consider using some sort of queueing mechanism. MSMQ is probably the right solution but there are many specific implementations if you want to go bigger and use an ESB like NServiceBus or RabbitMQ.
Update:
In that case, TPL Dataflow is your preferred solution if you can leverage the CTP. A throttled BufferBlock is the solution.
This example comes from the documentation provided by Microsoft:
// Hand-off through a bounded BufferBlock<T>
private static BufferBlock<int> m_buffer = new BufferBlock<int>(
new DataflowBlockOptions { BoundedCapacity = 10 });
// Producer
private static async void Producer()
{
while(true)
{
await m_buffer.SendAsync(Produce());
}
}
// Consumer
private static async Task Consumer()
{
while(true)
{
Process(await m_buffer.ReceiveAsync());
}
}
// Start the Producer and Consumer
private static async Task Run()
{
await Task.WhenAll(Producer(), Consumer());
}
Update:
Check out RX's Observable.Throttle.