Let me try to explain the design pattern I'm looking to implement. I have a Queue<Item> or some other collection of Items that needs to be processed by some method
static void Process(Item item)
{
// ...
}
They can be processed synchronously or asynchronously.
This queue will be getting items added to it periodically, because I have some method like
static void AddWorkToQueue()
{
// ...
}
that gets run on a timer
timer.Elapsed += AddWorkToQueue();
So what I need is some type "continuous while loop" that stops whenever the queue is empty.
How can I use C#.NET to do this in the best way? Are there any built-in libraries for solving this type of problem?
You could create a custom queue class and change it's listener:
public class CustomQueue : Queue<object>
{
public event EventHandler FirstItemInserted;
protected virtual void OnFirstItemInserted()
{
FirstItemInserted?.Invoke(this, EventArgs.Empty);
}
//Modified Enqueue method.
public new void Enqueue(object obj)
{
//Call the listener every time an item is inserted into the empty queue.
if (Count == 0)
{
base.Enqueue(obj);
OnFirstItemInserted();
}
else
base.Enqueue(obj);
}
}
The "Enqueue" method was changed to call the listener after each first insert.
All you'll need to do is to call a method to consume each item of your queue as long as there are items to dequeue.
class MyProgram
{
private static CustomQueue MyQueue;
public MyProgram()
{
MyQueue = new CustomQueue();
MyQueue.FirstItemInserted += ConsumeQueue;
//Activate timer...
}
private static void ConsumeQueue(object sender, EventArgs e)
{
object item;
while (MyQueue.Count > 0)
{
item = MyQueue.Dequeue();
//Do something...
}
}
}
You can use BufferBlock<Item> from TPL DataFlow (https://msdn.microsoft.com/en-us/library/hh160414(v=vs.110).aspx):
using System.Threading.Tasks.Dataflow;
static void AddWorkToQueue()
{
queue.Send(new Item());
}
static async void MainLoop()
{
while (true) // may be you need some cancellation token to trigger end of processing
{
var item = await queue.ReceiveAsync();
ProcessItem(item);
}
}
Related
I am newbie with Reactive Extensions but What is the best way to implement below scenario using Reactive Extensions:
1- Subscribe an event for every minute in constructor
2- if an operation get called then that subscriber get reset
3- if nothing happened or that operation won't be called for a minute the event in step 1 get fired
Something like this:
public class ImportClient : Carrier<IImportService>, IImportService
{
IObservable<long> proxyCleaner;
void DisposeProxy(long interval)
{
this.Close();
//Dispos proxy
}
public void RunDisposeTimer()
{
proxyCleaner = Observable.Interval(TimeSpan.FromMinutes(1));
proxyCleaner.Subscribe(DisposeProxy);
}
public ImportClient(String endpointConfigurationName) : base(endpointConfigurationName)
{
RunDisposeTimer();
}
public Attach_DTO_OUT AttachImage(AttachImage_DTO_IN source_C)
{
//Reset timer here
//Reset proxyCleaner
using (OperationContextScope scope = new OperationContextScope(this.InnerChannel))
{
AddMessageHeader<Token>(Token);
return base.Channel.AttachImage(source_C);
}
}
}
because my WCF service is Session Full I need to dispose it manually after specific time.
UPDATE:
I think it is possible to do with ObservableCollection but how:
private ObservableCollection<string> collection;
public void RunDisposeTimer()
{
collection = new ObservableCollection<string>();
collection.CollectionChanged += Collection_CollectionChanged;
}
private void Collection_CollectionChanged(object sender, System.Collections.Specialized.NotifyCollectionChangedEventArgs e)
{
//Here reset timer
throw new NotImplementedException();
}
I guess it is sort of a code review, but here is my implementation of the producer / consumer pattern. What I would like to know is would there be a case in which the while loops in the ReceivingThread() or SendingThread() methods might stop executing. Please note that EnqueueSend(DataSendEnqeueInfo info) is called from multiple different threads and I probably can't use tasks here since I definitely have to consume commands in a separate thread.
private Thread mReceivingThread;
private Thread mSendingThread;
private Queue<DataRecievedEnqeueInfo> mReceivingThreadQueue;
private Queue<DataSendEnqeueInfo> mSendingThreadQueue;
private readonly object mReceivingQueueLock = new object();
private readonly object mSendingQueueLock = new object();
private bool mIsRunning;
EventWaitHandle mRcWaitHandle;
EventWaitHandle mSeWaitHandle;
private void ReceivingThread()
{
while (mIsRunning)
{
mRcWaitHandle.WaitOne();
DataRecievedEnqeueInfo item = null;
while (mReceivingThreadQueue.Count > 0)
{
lock (mReceivingQueueLock)
{
item = mReceivingThreadQueue.Dequeue();
}
ProcessReceivingItem(item);
}
mRcWaitHandle.Reset();
}
}
private void SendingThread()
{
while (mIsRunning)
{
mSeWaitHandle.WaitOne();
while (mSendingThreadQueue.Count > 0)
{
DataSendEnqeueInfo item = null;
lock (mSendingQueueLock)
{
item = mSendingThreadQueue.Dequeue();
}
ProcessSendingItem(item);
}
mSeWaitHandle.Reset();
}
}
internal void EnqueueRecevingData(DataRecievedEnqeueInfo info)
{
lock (mReceivingQueueLock)
{
mReceivingThreadQueue.Enqueue(info);
mRcWaitHandle.Set();
}
}
public void EnqueueSend(DataSendEnqeueInfo info)
{
lock (mSendingQueueLock)
{
mSendingThreadQueue.Enqueue(info);
mSeWaitHandle.Set();
}
}
P.S the idea here is that am using WaitHandles to put thread to sleep when the queue is empty, and signal them to start when new items are enqueued.
UPDATE
I am just going to leave this https://blogs.msdn.microsoft.com/benwilli/2015/09/10/tasks-are-still-not-threads-and-async-is-not-parallel/ ,for people who might be trying to implement Producer/Consumer pattern using TPL or tasks.
Use a BlockingCollection instead of Queue, EventWaitHandle and lock objects:
public class DataInfo { }
private Thread mReceivingThread;
private Thread mSendingThread;
private BlockingCollection<DataInfo> queue;
private CancellationTokenSource receivingCts = new CancellationTokenSource();
private void ReceivingThread()
{
try
{
while (!receivingCts.IsCancellationRequested)
{
// This will block until an item is added to the queue or the cancellation token is cancelled
DataInfo item = queue.Take(receivingCts.Token);
ProcessReceivingItem(item);
}
}
catch (OperationCanceledException)
{
}
}
internal void EnqueueRecevingData(DataInfo info)
{
// When a new item is produced, just add it to the queue
queue.Add(info);
}
// To cancel the receiving thread, cancel the token
private void CancelReceivingThread()
{
receivingCts.Cancel();
}
Personally, for simple producer-consumer problems, I would just use BlockingCollection. There would be no need to manually code your own synchronization logic. The consuming threads will also block if there are no items present in the queue.
Here is what your code might look like if you use this class:
private BlockingCollection<DataRecievedEnqeueInfo> mReceivingThreadQueue = new BlockingCollection<DataRecievedEnqeueInfo>();
private BlockingCollection<DataSendEnqeueInfo> mSendingThreadQueue = new BlockingCollection<DataSendEnqeueInfo>();
public void Stop()
{
// No need for mIsRunning. Makes the enumerables in the GetConsumingEnumerable() calls
// below to complete.
mReceivingThreadQueue.CompleteAdding();
mSendingThreadQueue.CompleteAdding();
}
private void ReceivingThread()
{
foreach (DataRecievedEnqeueInfo item in mReceivingThreadQueue.GetConsumingEnumerable())
{
ProcessReceivingItem(item);
}
}
private void SendingThread()
{
foreach (DataSendEnqeueInfo item in mSendingThreadQueue.GetConsumingEnumerable())
{
ProcessSendingItem(item);
}
}
internal void EnqueueRecevingData(DataRecievedEnqeueInfo info)
{
// You can also use TryAdd() if there is a possibility that you
// can add items after you have stopped. Otherwise, this can throw an
// an exception after CompleteAdding() has been called.
mReceivingThreadQueue.Add(info);
}
public void EnqueueSend(DataSendEnqeueInfo info)
{
mSendingThreadQueue.Add(info);
}
As suggested in comments, you also can give a try to the TPL Dataflow blocks.
As far as I can see, you have two similar pipelines, for receive and send, so I assume that your class hierarchy is like this:
class EnqueueInfo { }
class DataRecievedEnqeueInfo : EnqueueInfo { }
class DataSendEnqeueInfo : EnqueueInfo { }
We can assemble an abstract class which will encapsulate the logic for creating the pipeline, and providing the interface for processing the items, like this:
abstract class EnqueueInfoProcessor<T>
where T : EnqueueInfo
{
// here we will store all the messages received before the handling
private readonly BufferBlock<T> _buffer;
// simple action block for actual handling the items
private ActionBlock<T> _action;
// cancellation token to cancel the pipeline
public EnqueueInfoProcessor(CancellationToken token)
{
_buffer = new BufferBlock<T>(new DataflowBlockOptions { CancellationToken = token });
_action = new ActionBlock<T>(item => ProcessItem(item), new ExecutionDataflowBlockOptions
{
MaxDegreeOfParallelism = Environment.ProcessorCount,
CancellationToken = token
});
// we are linking two blocks so all the items from buffer
// will flow down to action block in order they've been received
_buffer.LinkTo(_action, new DataflowLinkOptions { PropagateCompletion = true });
}
public void PostItem(T item)
{
// synchronously wait for posting to complete
_buffer.Post(item);
}
public async Task SendItemAsync(T item)
{
// asynchronously wait for message to be posted
await _buffer.SendAsync(item);
}
// abstract method to implement
protected abstract void ProcessItem(T item);
}
Note that you also can encapsulate the link between two blocks by using the Encapsulate<TInput, TOutput> method, but in that case you have to properly handle the Completion of the buffer block, if you're using it.
After this, we just need to implement two methods for receive and send handle logic:
public class SendEnqueueInfoProcessor : EnqueueInfoProcessor<DataSendEnqeueInfo>
{
SendEnqueueInfoProcessor(CancellationToken token)
: base(token)
{
}
protected override void ProcessItem(DataSendEnqeueInfo item)
{
// send logic here
}
}
public class RecievedEnqueueInfoProcessor : EnqueueInfoProcessor<DataRecievedEnqeueInfo>
{
RecievedEnqueueInfoProcessor(CancellationToken token)
: base(token)
{
}
protected override void ProcessItem(DataRecievedEnqeueInfo item)
{
// recieve logic here
}
}
You also can create more complicated pipeline with TransformBlock<DataRecievedEnqeueInfo, DataSendEnqeueInfo>, if your message flow is about a ReceiveInfo message became SendInfo.
Our existing implementation of domain events limits (by blocking) publishing to one thread at a time to avoid reentrant calls to handlers:
public interface IDomainEvent {} // Marker interface
public class Dispatcher : IDisposable
{
private readonly SemaphoreSlim semaphore = new SemaphoreSlim(1, 1);
// Subscribe code...
public void Publish(IDomainEvent domainEvent)
{
semaphore.Wait();
try
{
// Get event subscriber(s) from concurrent dictionary...
foreach (Action<IDomainEvent> subscriber in eventSubscribers)
{
subscriber(domainEvent);
}
}
finally
{
semaphore.Release();
}
}
// Dispose pattern...
}
If a handler publishes an event, this will deadlock.
How can I rewrite this to serialize calls to Publish? In other words, if subscribing handler A publishes event B, I'll get:
Handler A called
Handler B called
while preserving the condition of no reentrant calls to handlers in a multithreaded environment.
I do not want to change the public method signature; there's no place in the application to call a method to publish a queue, for instance.
We came up with a way to do it synchronously.
public class Dispatcher : IDisposable
{
private readonly ConcurrentQueue<IDomainEvent> queue = new ConcurrentQueue<IDomainEvent>();
private readonly SemaphoreSlim semaphore = new SemaphoreSlim(1, 1);
// Subscribe code...
public void Publish(IDomainEvent domainEvent)
{
queue.Enqueue(domainEvent);
if (IsPublishing)
{
return;
}
PublishQueue();
}
private void PublishQueue()
{
IDomainEvent domainEvent;
while (queue.TryDequeue(out domainEvent))
{
InternalPublish(domainEvent);
}
}
private void InternalPublish(IDomainEvent domainEvent)
{
semaphore.Wait();
try
{
// Get event subscriber(s) from concurrent dictionary...
foreach (Action<IDomainEvent> subscriber in eventSubscribers)
{
subscriber(domainEvent);
}
}
finally
{
semaphore.Release();
}
// Necessary, as calls to Publish during publishing could have queued events and returned.
PublishQueue();
}
private bool IsPublishing
{
get { return semaphore.CurrentCount < 1; }
}
// Dispose pattern for semaphore...
}
}
You will have to make Publish asynchronous to achieve that. Naive implementation would be as simple as:
public class Dispatcher : IDisposable {
private readonly BlockingCollection<IDomainEvent> _queue = new BlockingCollection<IDomainEvent>(new ConcurrentQueue<IDomainEvent>());
private readonly CancellationTokenSource _cts = new CancellationTokenSource();
public Dispatcher() {
new Thread(Consume) {
IsBackground = true
}.Start();
}
private List<Action<IDomainEvent>> _subscribers = new List<Action<IDomainEvent>>();
public void AddSubscriber(Action<IDomainEvent> sub) {
_subscribers.Add(sub);
}
private void Consume() {
try {
foreach (var #event in _queue.GetConsumingEnumerable(_cts.Token)) {
try {
foreach (Action<IDomainEvent> subscriber in _subscribers) {
subscriber(#event);
}
}
catch (Exception ex) {
// log, handle
}
}
}
catch (OperationCanceledException) {
// expected
}
}
public void Publish(IDomainEvent domainEvent) {
_queue.Add(domainEvent);
}
public void Dispose() {
_cts.Cancel();
}
}
It can't be done with that interface. You can process the event subscriptions asynchronously to remove the deadlock while still running them serially, but then you can't guarantee the order you described. Another call to Publish might enqueue something (event C) while the handler for event A is running but before it publishes event B. Then event B ends up behind event C in the queue.
As long as Handler A is on equal footing with other clients when it comes to getting an item in the queue, it either has to wait like everyone else (deadlock) or it has to play fairly (first come, first served). The interface you have there doesn't allow the two to be treated differently.
That's not to say you couldn't get up to some shenanigans in your logic to attempt to differentiate them (e.g. based on thread id or something else identifiable), but anything along those lines would unreliable if you don't control the subscriber code as well.
in my WPF - C# application, I have a time consuming function, which I execute with a BackgroundWorker. The job of this function is to add given data from a file into a database. Now and then, I need some user feedback, for example the data is already in the store and I want to ask the user, whether he wants to merge the data or create a new object or skip the data completely. Much like the dialog windows shows, if I try to copy a file to a location, where a file with the same name already exists.
The problem is, that I cannot call a GUI-window from a non GUI-thread. How could I implement this behavior?
Thanks in advance,
Frank
You could work with EventWaitHandle ou AutoResetEvent, then whenever you want to prompt the user, you could the signal UI, and then wait for the responde. The information about the file could be stored on a variable.
If possible... my suggestion is to architect your long running task into atomic operations. Then you can create a queue of items accessible by both your background thread and UI thread.
public class WorkItem<T>
{
public T Data { get; set; }
public Func<bool> Validate { get; set; }
public Func<T, bool> Action { get; set; }
}
You can use something like this class. It uses a queue to manage the execution of your work items, and an observable collection to signal the UI:
public class TaskRunner<T>
{
private readonly Queue<WorkItem<T>> _queue;
public ObservableCollection<WorkItem<T>> NeedsAttention { get; private set; }
public bool WorkRemaining
{
get { return NeedsAttention.Count > 0 && _queue.Count > 0; }
}
public TaskRunner(IEnumerable<WorkItem<T>> items)
{
_queue = new Queue<WorkItem<T>>(items);
NeedsAttention = new ObservableCollection<WorkItem<T>>();
}
public event EventHandler WorkCompleted;
public void LongRunningTask()
{
while (WorkRemaining)
{
if (_queue.Any())
{
var workItem = _queue.Dequeue();
if (workItem.Validate())
{
workItem.Action(workItem.Data);
}
else
{
NeedsAttention.Add(workItem);
}
}
else
{
Thread.Sleep(500); // check if the queue has items every 500ms
}
}
var completedEvent = WorkCompleted;
if (completedEvent != null)
{
completedEvent(this, EventArgs.Empty);
}
}
public void Queue(WorkItem<T> item)
{
// TODO remove the item from the NeedsAttention collection
_queue.Enqueue(item);
}
}
Your UI codebehind could look something like
public class TaskRunnerPage : Page
{
private TaskRunner<XElement> _taskrunner;
public void DoWork()
{
var work = Enumerable.Empty<WorkItem<XElement>>(); // TODO create your workItems
_taskrunner = new TaskRunner<XElement>(work);
_taskrunner.NeedsAttention.CollectionChanged += OnItemNeedsAttention;
Task.Run(() => _taskrunner.LongRunningTask()); // run this on a non-UI thread
}
private void OnItemNeedsAttention(object sender, NotifyCollectionChangedEventArgs e)
{
// e.NewItems contains items that need attention.
foreach (var item in e.NewItems)
{
var workItem = (WorkItem<XElement>) item;
// do something with workItem
PromptUser();
}
}
/// <summary>
/// TODO Use this callback from your UI
/// </summary>
private void OnUserAction()
{
// TODO create a new workItem with your changed parameters
var workItem = new WorkItem<XElement>();
_taskrunner.Queue(workItem);
}
}
This code is untested! But the basic principle should work for you.
Specifically to your case
private void backgroundWorker1_DoWork(object sender, DoWorkEventArgs e)
{
Thread.Sleep(1000);
var a = Test1("a");
Thread.Sleep(1000);
var b = (string)Invoke(new Func<string>(() => Test2("b")));
MessageBox.Show(a + b);
}
private string Test1(string text)
{
if (this.InvokeRequired)
return (string)this.Invoke(new Func<string>(() => Test1(text)));
else
{
MessageBox.Show(text);
return "test1";
}
}
private string Test2(string text)
{
MessageBox.Show(text);
return "test2";
}
Test2 is a normal method which you have to invoke from background worker. Test1 can be called directly and uses safe pattern to invoke itself.
MessageBox.Show is similar to yourForm.ShowDialog (both are modal), you pass parameters to it (text) and you return value (can be a value of property of yourForm which is set when form is closed). I am using string, but it can be any data type obviously.
From the input of the answers here, I came to the following solution:
(Mis)Using the ReportProgress-method of the Backgroundworker in Combination with a EventWaitHandle. If I want to interact with the user, I call the ReportProgress-method and setting the background process on wait. In the Handler for the ReportProgress event I do the interaction and when finished, I release the EventWaitHandle.
BackgroundWorker bgw;
public MainWindow()
{
InitializeComponent();
bgw = new BackgroundWorker();
bgw.DoWork += new DoWorkEventHandler(bgw_DoWork);
bgw.RunWorkerCompleted += new RunWorkerCompletedEventHandler(bgw_RunWorkerCompleted);
bgw.WorkerReportsProgress = true;
bgw.ProgressChanged += new ProgressChangedEventHandler(bgw_ProgressChanged);
}
// Starting the time consuming operation
private void Button_Click(object sender, RoutedEventArgs e)
{
bgw.RunWorkerAsync();
}
// using the ProgressChanged-Handler to execute the user interaction
void bgw_ProgressChanged(object sender, ProgressChangedEventArgs e)
{
UserStateData usd = e.UserState as UserStateData;
// UserStateData.Message is used to see **who** called the method
if (usd.Message == "X")
{
// do the user interaction here
UserInteraction wnd = new UserInteraction();
wnd.ShowDialog();
// A global variable to carry the information and the EventWaitHandle
Controller.instance.TWS.Message = wnd.TextBox_Message.Text;
Controller.instance.TWS.Background.Set();
}
}
void bgw_RunWorkerCompleted(object sender, RunWorkerCompletedEventArgs e)
{
MessageBox.Show(e.Result.ToString());
}
// our time consuming operation
void bgw_DoWork(object sender, DoWorkEventArgs e)
{
Thread.Sleep(2000);
// need 4 userinteraction: raise the ReportProgress event and Wait
bgw.ReportProgress(0, new UserStateData() { Message = "X", Data = "Test" });
Controller.instance.TWS.Background.WaitOne();
// The WaitHandle was released, the needed information should be written to global variable
string first = Controller.instance.TWS.Message.ToString();
// ... and again
Thread.Sleep(2000);
bgw.ReportProgress(0, new UserStateData() { Message = "X", Data = "Test" });
Controller.instance.TWS.Background.WaitOne();
e.Result = first + Controller.instance.TWS.Message;
}
I hope I did not overlooked some critical issues. I'm not so familar with multithreading - maybe there should be some lock(object) somewhere?
I need to be able to trigger a event whenever an object is added to a Queue<Delegate>.
I created a new class that extends Queue:
public delegate void ChangedEventHandler(object sender, EventArgs e);
public class QueueWithChange<Delegate> : Queue<Delegate>
{
public event ChangedEventHandler Changed;
protected virtual void OnChanged(EventArgs e) {
if (Changed != null)
{
Changed(this, e);
}
}
}
And then attached the event from another class, like such:
QueueWithChange<TimerDelegate> eventQueue = new QueueWithChange<TimerDelegate>();
//
eventQueue.Changed += new ChangedEventHandler(delegate(object s, EventArgs ex) {
//This event is not being triggered, so this code is unreachable atm...and that is my problem
if (eventQueue.Count > 0)
{
eventQueue.Dequeue().Invoke(new DispatcherTimer() { Interval = TimeSpan.FromMilliseconds(5) });
actionTimer.Stop();
}
});
But whenever I enqueue an object (eventQueue.Enqueue(something)), the attached event is not being fired.
What am I missing here?
If you mean the non-generic Queue class, then you can just override Enqueue:
public override void Enqueue(object obj)
{
base.Enqueue(obj);
OnChanged(EventArgs.Empty);
}
However, if you mean the generic Queue<T> class, then note that there is no suitable virtual method to override. You might do better to encapsulate the queue with your own class:
(** important edit: removed base-class!!! **)
class Foo<T>
{
private readonly Queue<T> queue = new Queue<T>();
public event EventHandler Changed;
protected virtual void OnChanged()
{
if (Changed != null) Changed(this, EventArgs.Empty);
}
public virtual void Enqueue(T item)
{
queue.Enqueue(item);
OnChanged();
}
public int Count { get { return queue.Count; } }
public virtual T Dequeue()
{
T item = queue.Dequeue();
OnChanged();
return item;
}
}
However, looking at your code, it seems possible that you are using multiple threads here. If that is the case, consider a threaded queue instead.
I just did write up on what I call a TriggeredQueue. It's inspired the answer by Marc Gravell.
You can find my post here: http://joesauve.com/triggeredqueuet
And the Gist here: http://gist.github.com/jsauve/b2e8496172fdabd370c4
It has four events:
WillEnqueue
WillDequeue
DidEnqueue
DidDequeue
You can hook into any of these like so:
YourQueue.WillEnqueue += (sender, e) => {
// kick off some process
};
YourQueue.DidEnqueue += (sender, e) => {
// kick off some process
// e.Item provides access to the enqueued item, if you like
};
YourQueue.WillDequeue += (sender, e) => {
// kick off some process
};
YourQueue.DidDequeue += (sender, e) => {
// kick off some process
// e.Item provides access to the dequeued item, if you like
};
One neat trick is that you can use the DidDequeue method to kick off some process to ensure that the queue is full by making a web request or loading some data from a filesystem, etc. I use this class in Xamarin mobile apps to ensure that data and images are pre-cached in order to provide a smooth user experience, instead of loading images AFTER they scroll onto the screen (like you might see in Facebook and countless other apps).
try
public new void Enqueue(Delegate d)
{
base.Enqueue(d);
OnChanged(EventArgs.Empty);
}
You have to override Enqueue, to call OnChanged.