There is a queue. There is a function that processes messages from this queue. This function takes the message from the queue, start new task to process the next message, waiting data from other sources, and then carries out the calculation.
This is example
using System;
using System.Diagnostics;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
namespace TestTaskFactory
{
class Program
{
static int Data = 50;
static int ActiveTasksNumber = 0;
static int MaxActiveTasksNumber = 0;
static Stopwatch clock = new Stopwatch();
static object locker = new object();
static object locker2 = new object();
static void Main(string[] args)
{
clock.Start();
Task.Factory.StartNew(() => DoWork());
while (true)
{
Thread.Sleep(10000);
}
}
public static void DoWork()
{
//imitation of geting message from some queue
int message = GetMessageFromQueue();
lock (locker2)
{
ActiveTasksNumber++;
MaxActiveTasksNumber = Math.Max(MaxActiveTasksNumber,
ActiveTasksNumber);
Console.Write("\r" + message + " ");
}
//Run new task to work with next message
Task.Factory.StartNew(() => DoWork());
//imitation wait some other data
Thread.Sleep(3000);
//imitation of calculations with message
int tmp = 0;
for (int i = 0; i < 30000000; i++)
{
tmp = Math.Max(message, i);
}
lock (locker2)
{
ActiveTasksNumber--;
}
}
public static int GetMessageFromQueue()
{
lock (locker)
{
if (Data == 0)
{
//Queue is empty. All tasks completed except one
//that is waiting for new data
clock.Stop();
Console.WriteLine("\rMax active tasks number = "
+ MaxActiveTasksNumber
+ "\tTime = " + clock.ElapsedMilliseconds + "ms");
Console.Write("Press key to run next iteration");
clock.Reset();
Console.ReadKey();
Console.Write(" ");
//In queue received new data. Processing repeat
clock.Start();
ActiveTasksNumber = 0;
MaxActiveTasksNumber = 0;
Data = 50;
}
Data--;
return Data;
}
}
}
}
My guess, when the queue is empty, all tasks are completed except one task that awaits the new data. When data arrives in the queue the calculations are repeated.
But if you look at the results , every time the number of simultaneously running tasks increases.
Why is this happening?
Test results
Your approach is wrong.
First of all, where is your Queue?
For any jobs you want to queue in a concurrent environment, use the ConcurrentQueue.
The concurrent queue, is used in this fashion, it doesn't need to be locked at any time.
// To create your Queue
ConcurrentQueue<string> queue = new ConcurrentQueue<string>();
// To add objects to your Queue
queue.Enqueue("foo");
// To deque items from your Queue
String bar;
queue.TryDequeue(out bar);
// To loop a process until your Queue is empty
while(!queue.IsEmpty)
{
String bar;
queue.TryDequeue(out bar);
}
Next is how you are incrementing and decrementing your counters, there is a far better way of doing it which is thread safe. Again, the data doesn't need to be locked.
// Change your data type from int to long
static long ActiveTasksNumber = 0;
static long MaxActiveTasksNumber = 0;
// To increment the values in a Thread safe fashion:
Interlocked.Increment(ref ActiveTasksNumber);
// To decrement:
Interlocked.Decrement(ref MaxActiveTasksNumber);
Implement what I've shown you, and it should make your problems disappear
Edit:
Namespaces
using System.Collections.Concurrent;
using System.Threading;
To expand on my comment:
You have, in essence, this:
public static void DoWork()
{
// imitation of geting message from some queue
int message = GetMessageFromQueue();
// Run new task to work with next message
Task.Factory.StartNew(() => DoWork());
// do some work
}
Your code is going to get the first message, start a task to work with the next item, and then do its work. While the first task is working, the second gets an item and spawns yet another task to get an item from the queue. So now you have two threads supposedly doing work and a third that's going to spawn yet another, etc . . .
Nothing in your code stops it from creating a new task for every item in the queue.
If your queue started with 38 things, it's highly likely that you'll end up with 38 concurrent tasks.
You need to limit the number of tasks you're running at the same time. There are many ways to do that. Perhaps the easiest is a simple producer-consumer model using BlockingCollection.
Related
In the system I am working at, I have a REST Api available for multiple parallel requests.
In the background, I have a Thrift connection with another tool, allowing maximal 5 threads.
To protect the access to that connection, I am using a SemaphoreSlim object with also a configurable timeout. This is my code:
private static readonly SemaphoreSlim _lockGuard = new SemaphoreSlim(1, 5);
private static int _waitingThreads = 0;
public async Task<Models.OperationStatus> PushItemWithResultAsync<TPushData>(TPushData data, CancellationToken ct)
where TPushData : Thrift.Protocol.TBase
{
Interlocked.Increment(ref _waitingThreads);
// Use a semaphore to prevent parallel calls.
if (await _lockGuard.WaitAsync(_millisecondsThriftTimeout, ct))
{
_logger.LogDebug($"Push data using {_waitingThreads} threads, but {_lockGuard.CurrentCount} within the semaphore");
Models.OperationStatus result;
try
{
// push the data to the single Thrift connection.
result = await PushItemWithResultAsyncLogic(data, ct).ConfigureAwait(false);
}
finally
{
_lockGuard.Release();
Interlocked.Decrement(ref _waitingThreads);
}
return result;
}
else
{
Interlocked.Decrement(ref _waitingThreads);
return GetFailureOperationStatus();
}
}
I am a bit confused how to use the timeout of the semaphore and the allowed thread.
I first started with new SemaphoreSlim(1, 1); to allow only a single connection.
The code behaves as expected: only one call to PushItemWithResultAsyncLogic runs. If too many calls to the methods are coming, they will timeout.
Now, I want to max 5 running PushItemWithResultAsyncLogic in parallel, thus the max allowed thread in the SemaphoreSlim raised to 5.
I expect to still have possible timeout when a lot of external requests are coming.
If 10 parallel calls are received, the timeout of 500ms is still enough to handle all of them.
The _waitingThreads raises up to 10 but _lockGuard.CurrentCount is always 0.
I was expecting to have something moving between 1 and 5...
And most importantly, I don't see the below component to be called up to 5 times.
What is wrong here?
EDIT: short answer: the initialCount should be set to 5 as explained below.
So with new SemaphoreSlim(5, 5), I do have parallel requests AND the timeout playing its job.
Tried to simulate your scenario with the console app below and added some comments. The Wait method in the semaphore is not used to define a timeout. It is used to block a thread from entering until there is a free slot. Also a distinction needs to be made between allowing a given number of requests to wait and rejecting the others. In the simulation below, 5 is the max number of requests that can be processed in parallel. Others will immediately be rejected until a processing slot is free again.
As mentioned in the comments, this will not throttle. If you want to change the number of max requests processed in parallel, you need to 1. increase the maxCount in Semaphore constructor and 2. Release(maxCount) to reset the Semaphore. Or simply call the constructor with initialCount = maxCount, for ex. SemaphoreSlim(5, 5)
using System;
using System.Threading;
using System.Threading.Tasks;
namespace Semaphore
{
public class Program
{
private static readonly SemaphoreSlim _lockGuard = new SemaphoreSlim(0, 5);
private static int totalProcessed; // Stats only.
public static void Main()
{
// Reset semaphore by allowing max 5 requests to enter the concurrent section.
_lockGuard.Release(5);
var requests = new Task[20];
for (var i = 0; i < 20; i++)
{
Console.WriteLine("Sending request: " + i);
var i1 = i;
requests[i] = Task.Run(() => ProcessRequest(i1));
// This is the interval between each request.
// Bigger value means more requests will be processed.
Thread.Sleep(20);
}
Task.WaitAll(requests);
Console.WriteLine("---");
Console.WriteLine("Processed requests: " + totalProcessed);
Console.WriteLine("Rejected requests: " + (20 - totalProcessed));
Console.ReadLine();
}
public static async Task ProcessRequest(int i)
{
// If the semaphore is already full, reject the request.
// This means requests won't get queued.
if (_lockGuard.CurrentCount == 0)
{
Console.WriteLine("ERROR: CurrentCount == 0 for " + i);
return;
}
try
{
// If this request was allowed to enter the semaphore, wait until a processing slot is free.
// This decrements CurrentCount.
_lockGuard.Wait();
// Once previous request completed (Release()), process the next waiting one.
await Task.Run(() => {
Console.WriteLine("Processing request " + i );
Thread.Sleep(300); // Simulate request processing time.
});
totalProcessed++;
}
finally
{
// Processing completed: release one slot in the semaphore.
_lockGuard.Release();
Console.WriteLine("Releasing " + i);
}
}
}
}
Note
As Kit said in the comments, you don't need a Semaphore to achieve what I understand you're trying to achieve. I assume there are no shared resources in PushItemWithResultAsyncLogic code, so what you need is effectively a counter and/or some request queue management if you want to allow some requests to wait.
EDIT
To process as many requests as possible and have them waiting for a timeout, there is no need to check CurrentCount. Simply Wait(x milliseconds) and either request will be processed with lock aquired or it will be rejected and Wait will return false.
using System;
using System.Threading;
using System.Threading.Tasks;
namespace Semaphore
{
public class Program
{
private static readonly SemaphoreSlim _lockGuard = new SemaphoreSlim(5, 5);
private static readonly Random rnd = new Random((int)DateTimeOffset.Now.Ticks);
private static int totalProcessed; // Stats
public static void Main()
{
var requests = new Task[20];
for (var i = 0; i < 20; i++)
{
var i1 = i;
requests[i] = Task.Run(() => ProcessRequest(i1));
Thread.Sleep(100);
}
Task.WaitAll(requests);
Console.WriteLine("----------------------------");
Console.WriteLine("Processed requests: " + totalProcessed);
Console.WriteLine("Rejected requests: " + (20 - totalProcessed));
Console.WriteLine("----------------------------");
Console.ReadLine();
}
public static async Task ProcessRequest(int i)
{
Console.WriteLine(i + " Wait");
// If at the end of the wait no slot is free, reject.
if (!await _lockGuard.WaitAsync(500))
{
Console.WriteLine(i + " Reject");
return;
}
try
{
// Once previous request completed (Release()), process the next waiting one.
await Task.Run(() =>
{
Console.WriteLine(i + " Process");
Thread.Sleep(rnd.Next(600, 800)); // Simulate random request processing time.
});
totalProcessed++;
}
finally
{
// Processing completed: release one slot in the semaphore.
_lockGuard.Release();
}
}
}
}
Set the initialCount constructor parameter to the same value as maxCount. CurrentCount goes from initialCount towards 0. So, if the initialCount is 1 then the first thread sets the CurrentCount to zero and prevents other threads from running.
I use multithreading to process a list of data.
In this example below, for each element, how to make sure "SecondProcess" always runs after "FirstProcess" finishes? The order of elements in the queue being processed doesn't really matter.
public class Processor
{
public void Process()
{
IList<int> queue = QueueGenerator.GetRandomInt(50); //gets a list of 50 unique random integer
foreach (int eachElement in queue)
{
ThreadPool.QueueUserWorkItem(new WaitCallback(FirstProcess), eachElement);
}
Console.ReadLine();
}
private void FirstProcess(object toProcess)
{
int i = 0;
int.TryParse(toProcess.ToString(), out i);
string odd = "odd";
string even = "even";
string toDisplay = (i%2 == 0)
? string.Format("First step: Processing {0} ({1} number)", i, even)
: string.Format("First step: Processing {0} ({1} number)", i, odd);
Console.WriteLine(toDisplay);
}
private void SecondProcess(object toProcess)
{
int i = 0;
int.TryParse(toProcess.ToString(), out i);
Console.WriteLine("Second step: Processing -> {0}", i);
}
}
any idea please?
Thanks
If, instead of
foreach (int eachElement in queue)
{
ThreadPool.QueueUserWorkItem(new WaitCallback(FirstProcess), eachElement);
}
you did
Parallel.ForEach(queue, eachElement => FirstProcess(eachElement));
this will call the delegate for each item in queue in parallel in the ThreadPool, but block until all elements have been processed.
This means that when the next line of code executes on this calling thread, all the work will be complete.
Now, you just do it again:
Parallel.ForEach(queue, eachElement => SecondProcess(eachElement));
Using the Parallel class will have advantages because it can make use of a partitioner, and so, effectively queue batched operations to the ThreadPool rather than queuing individual items into the ThreadPool queue.
How should the reading of bulk data from a device in C# be handled in .NET 4.0? Specifically I need to read quickly from a USB HID device that emits reports over 26 packets where order must be preserved.
I've tried doing this in a BackgroundWorker thread. It reads one packet from the device at a time, and process it, before reading more. This gives reasonably good response times, but it is liable to lose a packet here and there, and the overhead costs of a single packet read adds up.
while (!( sender as BackgroundWorker ).CancellationPending) {
//read a single packet
//check for header or footer
//process packet data
}
}
What is the best practice in C# for reading a device like this?
Background:
My USB HID device continuously reports a large amount of data. The data is split over 26 packets and I must preserver the order. Unfortunately the device only marks the first the last packets in each report, so I need to be able to catch every other packet in between.
For .Net 4 you can use a BlockingCollection to provide a threadsafe queue that can be used by a producer and a consumer. The BlockingCollection.GetConsumingEnumerable() method provides an enumerator which automatically terminates when the queue has been marked as completed using CompleteAdding() and is empty.
Here's some sample code. The payload is an array of ints in this example, but of course you would use whatever data type you need.
Note that for your specific example, you can use the overload of GetConsumingEnumerable() which accepts an argument of type CancellationToken.
using System;
using System.Collections.Concurrent;
using System.Threading;
using System.Threading.Tasks;
namespace Demo
{
public static class Program
{
private static void Main()
{
var queue = new BlockingCollection<int[]>();
Task.Factory.StartNew(() => produce(queue));
consume(queue);
Console.WriteLine("Finished.");
}
private static void consume(BlockingCollection<int[]> queue)
{
foreach (var item in queue.GetConsumingEnumerable())
{
Console.WriteLine("Consuming " + item[0]);
Thread.Sleep(25);
}
}
private static void produce(BlockingCollection<int[]> queue)
{
for (int i = 0; i < 1000; ++i)
{
Console.WriteLine("Producing " + i);
var payload = new int[100];
payload[0] = i;
queue.Add(payload);
Thread.Sleep(20);
}
queue.CompleteAdding();
}
}
}
For .Net 4.5 and later, you could use the higher-level classes from Microsoft's Task Parallel Library, which has a wealth of functionality (and can be somewhat daunting at first sight).
Here's the same example using TPL DataFlow:
using System;
using System.Threading;
using System.Threading.Tasks;
using System.Threading.Tasks.Dataflow;
namespace Demo
{
public static class Program
{
private static void Main()
{
var queue = new BufferBlock<int[]>();
Task.Factory.StartNew(() => produce(queue));
consume(queue).Wait();
Console.WriteLine("Finished.");
}
private static async Task consume(BufferBlock<int[]> queue)
{
while (await queue.OutputAvailableAsync())
{
var payload = await queue.ReceiveAsync();
Console.WriteLine("Consuming " + payload[0]);
await Task.Delay(25);
}
}
private static void produce(BufferBlock<int[]> queue)
{
for (int i = 0; i < 1000; ++i)
{
Console.WriteLine("Producing " + i);
var payload = new int[100];
payload[0] = i;
queue.Post(payload);
Thread.Sleep(20);
}
queue.Complete();
}
}
}
If missing packets is a concern do not do your processing and your reading on the same thread. Starting with .NET 4.0 they added the System.Collections.Concurrent namespace which makes this very easy to do. All you need is a BlockingCollection which behaves as a queue for your incoming packets.
BlockingCollection<Packet> _queuedPackets = new BlockingCollection<Packet>(new ConcurrentQueue<Packet>());
void readingBackgroundWorker_DoWork(object sender, DoWorkEventArgs e)
{
while (!( sender as BackgroundWorker ).CancellationPending)
{
Packet packet = GetPacket();
_queuedPackets.Add(packet);
}
_queuedPackets.CompleteAdding();
}
void processingBackgroundWorker_DoWork(object sender, DoWorkEventArgs e)
{
List<Packet> report = new List<Packet>();
foreach(var packet in _queuedPackets.GetConsumingEnumerable())
{
report.Add(packet);
if(packet.IsLastPacket)
{
ProcessReport(report);
report = new List<Packet>();
}
}
}
What will happen is while _queuedPackets is empty _queuedPackets.GetConsumingEnumerable() will block the thread not consuming any resources. As soon as a packet arrives it will unblock and do the next iteration of the foreach.
When you call _queuedPackets.CompleteAdding(); the foreach on your processing thread will run till the collection is empty then exit the foreach loop. If you don't want it to "finish up the queue" when you cancel you can easily change it up to quit early. I also am going to switch to using Tasks instead of Background workers because it makes the passing in parameters much easier to do.
void ReadingLoop(BlockingCollection<Packet> queue, CancellationToken token)
{
while (!token.IsCancellationRequested)
{
Packet packet = GetPacket();
queue.Add(packet);
}
queue.CompleteAdding();
}
void ProcessingLoop(BlockingCollection<Packet> queue, CancellationToken token)
{
List<Packet> report = new List<Packet>();
try
{
foreach(var packet in queue.GetConsumingEnumerable(token))
{
report.Add(packet);
if(packet.IsLastPacket)
{
ProcessReport(report);
report = new List<Packet>();
}
}
}
catch(OperationCanceledException)
{
//Do nothing, we don't care that it happened.
}
}
//This would replace your backgroundWorker.RunWorkerAsync() calls;
private void StartUpLoops()
{
var queue = new BlockingCollection<Packet>(new ConcurrentQueue<Packet>());
var cancelRead = new CancellationTokenSource();
var cancelProcess = new CancellationTokenSource();
Task.Factory.StartNew(() => ReadingLoop(queue, cancelRead.Token));
Task.Factory.StartNew(() => ProcessingLoop(queue, cancelProcess.Token));
//You can stop each loop indpendantly by calling cancelRead.Cancel() or cancelProcess.Cancel()
}
I want to have a kind of queue in which a single source inputs data in it and on the other side there will be consumers waiting that when they detect that the queue is not empty will start to execute the data until they are halted. but its important that if the queue is emptied they will still remain watching the queue such that if more data pops in they will be able to consume it. What i found By multiple consumer and multiple producers as the consumers are nested in the producers where in my case i cant do that as i will have a single source and consumers committed to the queue till i stop them. therefore not in series but both the consumer and the producers are executing in parallel.
will be xecutig the consumer and the producers in parallel by
Parallel.Invoke(() => producer(), () => consumers());
the problem as such is how i will execute the content of a queue which is sometimes empty in parallel
You can solve this relatively easily using a BlockingCollection<T>.
You can use one as a queue, and pass a reference to it to the producer() and each of the consumers().
You'll be calling GetConsumingEnumerable() from each consumer thread, and using it with foreach.
The producer thread will add items to the collection, and will call CompleteAdding() when it has finished producing stuff. This will automatically make all the consumer threads exit their foreach loops.
Here's a basic example (with no error handling). The calls to Thread.Sleep() are to simulate load, and should not be used in real code.
using System;
using System.Collections.Concurrent;
using System.Threading;
using System.Threading.Tasks;
namespace Demo
{
internal class Program
{
private static void Main(string[] args)
{
ThreadPool.SetMinThreads(10, 0); // To help the demo; not needed in real code.
var plant = new ProcessingPlant();
plant.Process();
Console.WriteLine("Work complete.");
}
}
public sealed class ProcessingPlant
{
private readonly BlockingCollection<string> _queue = new BlockingCollection<string>();
public void Process()
{
Parallel.Invoke(producer, consumers);
}
private void producer()
{
for (int i = 0; i < 100; ++i)
{
string item = i.ToString();
Console.WriteLine("Producer is queueing {0}", item);
_queue.Add(item); // <- Here's where we add an item to the queue.
Thread.Sleep(0);
}
_queue.CompleteAdding(); // <- Here's where we make all the consumers
} // exit their foreach loops.
private void consumers()
{
Parallel.Invoke(
() => consumer(1),
() => consumer(2),
() => consumer(3),
() => consumer(4),
() => consumer(5)
);
}
private void consumer(int id)
{
Console.WriteLine("Consumer {0} is starting.", id);
foreach (var item in _queue.GetConsumingEnumerable()) // <- Here's where we remove items.
{
Console.WriteLine("Consumer {0} read {1}", id, item);
Thread.Sleep(0);
}
Console.WriteLine("Consumer {0} is stopping.", id);
}
}
}
(I know this is using an extra thread just to start the consumers, but I did it this way to avoid obscuring the real point - which is to demonstrate the use of BlockingCollection.)
i've recently come across a producer/consumer pattern c# implementation. it's very simple and (for me at least) very elegant.
it seems to have been devised around 2006, so i was wondering if this implementation is
- safe
- still applicable
Code is below (original code was referenced at http://bytes.com/topic/net/answers/575276-producer-consumer#post2251375)
using System;
using System.Collections;
using System.Threading;
public class Test
{
static ProducerConsumer queue;
static void Main()
{
queue = new ProducerConsumer();
new Thread(new ThreadStart(ConsumerJob)).Start();
Random rng = new Random(0);
for (int i=0; i < 10; i++)
{
Console.WriteLine ("Producing {0}", i);
queue.Produce(i);
Thread.Sleep(rng.Next(1000));
}
}
static void ConsumerJob()
{
// Make sure we get a different random seed from the
// first thread
Random rng = new Random(1);
// We happen to know we've only got 10
// items to receive
for (int i=0; i < 10; i++)
{
object o = queue.Consume();
Console.WriteLine ("\t\t\t\tConsuming {0}", o);
Thread.Sleep(rng.Next(1000));
}
}
}
public class ProducerConsumer
{
readonly object listLock = new object();
Queue queue = new Queue();
public void Produce(object o)
{
lock (listLock)
{
queue.Enqueue(o);
// We always need to pulse, even if the queue wasn't
// empty before. Otherwise, if we add several items
// in quick succession, we may only pulse once, waking
// a single thread up, even if there are multiple threads
// waiting for items.
Monitor.Pulse(listLock);
}
}
public object Consume()
{
lock (listLock)
{
// If the queue is empty, wait for an item to be added
// Note that this is a while loop, as we may be pulsed
// but not wake up before another thread has come in and
// consumed the newly added object. In that case, we'll
// have to wait for another pulse.
while (queue.Count==0)
{
// This releases listLock, only reacquiring it
// after being woken up by a call to Pulse
Monitor.Wait(listLock);
}
return queue.Dequeue();
}
}
}
The code is older than that - I wrote it some time before .NET 2.0 came out. The concept of a producer/consumer queue is way older than that though :)
Yes, that code is safe as far as I'm aware - but it has some deficiencies:
It's non-generic. A modern version would certainly be generic.
It has no way of stopping the queue. One simple way of stopping the queue (so that all the consumer threads retire) is to have a "stop work" token which can be put into the queue. You then add as many tokens as you have threads. Alternatively, you have a separate flag to indicate that you want to stop. (This allows the other threads to stop before finishing all the current work in the queue.)
If the jobs are very small, consuming a single job at a time may not be the most efficient thing to do.
The ideas behind the code are more important than the code itself, to be honest.
You could do something like the following code snippet. It's generic and has a method for enqueue-ing nulls (or whatever flag you'd like to use) to tell the worker threads to exit.
The code is taken from here: http://www.albahari.com/threading/part4.aspx#_Wait_and_Pulse
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading;
namespace ConsoleApplication1
{
public class TaskQueue<T> : IDisposable where T : class
{
object locker = new object();
Thread[] workers;
Queue<T> taskQ = new Queue<T>();
public TaskQueue(int workerCount)
{
workers = new Thread[workerCount];
// Create and start a separate thread for each worker
for (int i = 0; i < workerCount; i++)
(workers[i] = new Thread(Consume)).Start();
}
public void Dispose()
{
// Enqueue one null task per worker to make each exit.
foreach (Thread worker in workers) EnqueueTask(null);
foreach (Thread worker in workers) worker.Join();
}
public void EnqueueTask(T task)
{
lock (locker)
{
taskQ.Enqueue(task);
Monitor.PulseAll(locker);
}
}
void Consume()
{
while (true)
{
T task;
lock (locker)
{
while (taskQ.Count == 0) Monitor.Wait(locker);
task = taskQ.Dequeue();
}
if (task == null) return; // This signals our exit
Console.Write(task);
Thread.Sleep(1000); // Simulate time-consuming task
}
}
}
}
Back in the day I learned how Monitor.Wait/Pulse works (and a lot about threads in general) from the above piece of code and the article series it is from. So as Jon says, it has a lot of value to it and is indeed safe and applicable.
However, as of .NET 4, there is a producer-consumer queue implementation in the framework. I only just found it myself but up to this point it does everything I need.
These days a more modern option is available using the namespace System.Threading.Tasks.Dataflow. It's async/await friendly and much more versatile.
More info here How to: Implement a producer-consumer dataflow pattern
It's included starting from .Net Core, for older .Nets you may need to install a package with the same name as the namespace.
I know the question is old, but it's the first match in Google for my request, so I decided to update the topic.
A modern and simple way to implement the producer/consumer pattern in C# is to use System.Threading.Channels. It's asynchronous and uses ValueTask's to decrease memory allocations. Here is an example:
public class ProducerConsumer<T>
{
protected readonly Channel<T> JobChannel = Channel.CreateUnbounded<T>();
public IAsyncEnumerable<T> GetAllAsync()
{
return JobChannel.Reader.ReadAllAsync();
}
public async ValueTask AddAsync(T job)
{
await JobChannel.Writer.WriteAsync(job);
}
public async ValueTask AddAsync(IEnumerable<T> jobs)
{
foreach (var job in jobs)
{
await JobChannel.Writer.WriteAsync(job);
}
}
}
Warning: If you read the comments, you'll understand my answer is wrong :)
There's a possible deadlock in your code.
Imagine the following case, for clarity, I used a single-thread approach but should be easy to convert to multi-thread with sleep:
// We create some actions...
object locker = new object();
Action action1 = () => {
lock (locker)
{
System.Threading.Monitor.Wait(locker);
Console.WriteLine("This is action1");
}
};
Action action2 = () => {
lock (locker)
{
System.Threading.Monitor.Wait(locker);
Console.WriteLine("This is action2");
}
};
// ... (stuff happens, etc.)
// Imagine both actions were running
// and there's 0 items in the queue
// And now the producer kicks in...
lock (locker)
{
// This would add a job to the queue
Console.WriteLine("Pulse now!");
System.Threading.Monitor.Pulse(locker);
}
// ... (more stuff)
// and the actions finish now!
Console.WriteLine("Consume action!");
action1(); // Oops... they're locked...
action2();
Please do let me know if this doesn't make any sense.
If this is confirmed, then the answer to your question is, "no, it isn't safe" ;)
I hope this helps.
public class ProducerConsumerProblem
{
private int n;
object obj = new object();
public ProducerConsumerProblem(int n)
{
this.n = n;
}
public void Producer()
{
for (int i = 0; i < n; i++)
{
lock (obj)
{
Console.Write("Producer =>");
System.Threading.Monitor.Pulse(obj);
System.Threading.Thread.Sleep(1);
System.Threading.Monitor.Wait(obj);
}
}
}
public void Consumer()
{
lock (obj)
{
for (int i = 0; i < n; i++)
{
System.Threading.Monitor.Wait(obj, 10);
Console.Write("<= Consumer");
System.Threading.Monitor.Pulse(obj);
Console.WriteLine();
}
}
}
}
public class Program
{
static void Main(string[] args)
{
ProducerConsumerProblem f = new ProducerConsumerProblem(10);
System.Threading.Thread t1 = new System.Threading.Thread(() => f.Producer());
System.Threading.Thread t2 = new System.Threading.Thread(() => f.Consumer());
t1.IsBackground = true;
t2.IsBackground = true;
t1.Start();
t2.Start();
Console.ReadLine();
}
}
output
Producer =><= Consumer
Producer =><= Consumer
Producer =><= Consumer
Producer =><= Consumer
Producer =><= Consumer
Producer =><= Consumer
Producer =><= Consumer
Producer =><= Consumer
Producer =><= Consumer
Producer =><= Consumer