Develop Reference

How to avoid deadlock when calling stop in a .net service from a thread?

I have a thread which is used for reading from a TCP socket; I receive a message from the server to perform an automatic update. So, now the read thread has to call servicebase Stop() to trigger the OnStop() function. However the read thread has to be joined to the main thread in order to properly terminate the service. So now the OnStop() function is waiting for the read thread to join but the read thread can't join because it's waiting for the Stop() function to finish running.
So basically it looks like this:
public void Start()
{
OnStart(new string[0]);
}
protected override void OnStart(string[] args)
{
stopEventRecv = new AutoResetEvent(false);
RecvThread = new Thread(RecvLoop);
RecvThread.Start();
}
protected override void OnStop()
{
// Doesn't matter because we are about to deadlock
stopEventRecv.Set();
// Dead lock
RecvThread.Join();
}
private void RecvLoop(object arg)
{
while (true)
{
if (stopEventRecv.WaitOne(5000))
{
return;
}
if (!IsConnected())
continue;
// here we get message from server saying to Stop so
// the message is processed and uses a callback where
// ServiceBase.Stop() is called.
// The reason for the callback isn't relevant to the
// question I don't think but i'm mentioning it in case.
// For the sake of the question I'll just call Stop()
// here to demonstrate the problem.
Stop();
}
}
How can I get around this? Is it necessary to join all threads before exiting the service? Would an Abort be OK here instead of a join?

The Service Control Manager (SCM) is responsible for sending commands to services to tell them to Start, Stop, Pause, as well as any custom commands you choose to define.
You write you OnXxx methods to react to the commands that the SCM sends to your service. In order to properly initiate a Stop, you ought to ask the SCM to stop your service, and you can do that using the ServiceController class's Stop method. This initiates the Stop but does not wait for it to complete.
This ensures that the SCM is aware that your service is deliberately stopping and didn't exit for any other reason. This is important because if any Recovery actions have been configured for your service, you don't want them to occur in this case.

One option to avoid the deadlock is to not join RecvThread if Stop was called from it. If Stop was called from that thread - you know that it's done all useful work and is about to exit, so join is not required anyway:
if (RecvThread.ManagedThreadId != Thread.CurrentThread.ManagedThreadId)
RecvThread.Join();
Also, I'd personally avoid doing Join without timeout. Better do Join(TimeSpan.FromSeconds(X)) and if given thread failed to finish during that timeout - do something meaninful (throw exception, abort that thread and so on). Anything is better than having to deadlock without even realizing it.
But better solution is mentioned by Damian in comments: use ServiceController.Stop method, which will stop all dependent services, if any, and then will send stop signal to your service and return, without waiting for service to actually stop, thus also avoiding a deadlock.

Windows service OnStop wait for finished processing

I actually develop a Windows service in VS 2012 / .NET 4.5.
The service is following the scheme of the code snippet below:
Using a timer
Executes some desired operation every couple of minutes.
The process takes about 10 minutes to complete
I use a single thread in the service
What I am worried about is that if somebody stops the service via the management console, it might be just during the process that the service is doing.
I have done some reading about stopping Windows service with request stop, but am a bit lost. Sometimes WorkerThreads are created, sometimes ManualResetEvents are created, but up to now I couldn't fully grasp the best way forward for my Windows service.
I need to wait until the processing is properly finished in the onStop method before stopping the Windows service.
What is the best way forward then, also considering the code snippet below?
Thanks all!
namespace ImportationCV
{
public partial class ImportationCV : ServiceBase
{
private System.Timers.Timer _oTimer;
public ImportationCV()
{
InitializeComponent();
if (!EventLog.SourceExists(DAL.Utilities.Constants.LOG_JOURNAL))
{
EventLog.CreateEventSource(DAL.Utilities.Constants.LOG_JOURNAL, DAL.Utilities.Constants.SOURCE_JOURNAL);
}
EventLog.Source = DAL.Utilities.Constants.SOURCE_JOURNAL;
EventLog.Log = DAL.Utilities.Constants.LOG_JOURNAL;
}
protected override void OnStart(string[] args)
{
int intDelai = Properties.Settings.Default.WatchDelay * 1000;
_oTimer = new System.Timers.Timer(intDelai);
_oTimer.Elapsed += new ElapsedEventHandler(this.Execute);
_oTimer.Start();
EventLog.WriteEntry(DAL.Utilities.Constants.LOG_JOURNAL, "Service " + DAL.Utilities.Constants.SERVICE_TITLE + " started at " + DateTime.Now.ToString("HH:mm:ss"), EventLogEntryType.Information);
}
protected override void OnStop()
{
if (_oTimer != null && _oTimer.Enabled)
{
_oTimer.Stop();
_oTimer.Dispose();
}
EventLog.WriteEntry(DAL.Utilities.Constants.LOG_JOURNAL, "Service " + DAL.Utilities.Constants.SERVICE_TITLE + " stopped at " + DateTime.Now.ToString("HH:mm:ss"), EventLogEntryType.Information);
}
private void Execute(object source, ElapsedEventArgs e)
{
_oTimer.Stop();
try
{
//Process
}
catch (Exception ex)
{
EventLog.WriteEntry(DAL.Utilities.Constants.LOG_JOURNAL, (ex.StackTrace + ("\r\n" + ex.Message)), EventLogEntryType.Error);
}
_oTimer.Start();
}
}
}

As a test case, I put a call to System.Threading.Thread.Sleep(500000) in the OnStop() callback of my Windows service. I started the service and then stopped it. I got the window with the progress bar indicating that the Service Control Manager (SCM) was attempting to stop the service. After about 2 minutes, I got this response from the SCM:
After I dismissed this window, the status of my service in the SCM changed to Stopping, and I noticed that the service continued to run in Task Manager. After the sleep elapsed (nearly 6 minutes later), the process stopped. Refreshing the SCM window showed the service was no longer running.
I take a couple of things away from this. First, OnStop() should really attempt to stop the service in a timely manner just as part of playing nice with the system. Second, depending on how your OnStop() method is structured, you could force the service to ignore a preemptive request to stop, instead stopping when you say so. This is not recommended, but it appears that you could do this.
As to your particular situation, the thing you have to understand is that the System.Timers.Timer.Elapsed event fires on a ThreadPool thread. By definition, this is a background thread, which means that it will not keep the application running. When the service is told to shut down, the system will stop all background threads and then exit the process. So your concern about keeping the processing going until it is finished despite being told by the SCM to shutdown cannot occur the way you've got things structured currently. To do that, you'd need to create a formal System.Threading.Thread object, set it as a foreground thread, and then use the timer to trigger this thread to execute (as opposed to being done in the Elapsed callback).
All of that said, I still think you'll want to play nicely with the system, which means timely shutdown of the service when requested to do so. What happens if, for example, you need to reboot the machine? I haven't tested it, but if you force your service to continue running until the processing is complete, the system may indeed wait until the process finishes before actually restarting. That's not what I would want from my service.
So I would suggest one of two things. The first option would be to break the processing into distinct chunks that can be done individually. As each chunk is finished, check to see if the service is stopping. If so, exit the thread gracefully. If this cannot be done, then I would introduce something akin to transactions to your processing. Let's say that you're needing to interact with a bunch of database tables and interrupting the flow once it's started becomes problematic because the database may be left in a bad state. If the database system allows transactions, this becomes relatively easy. If not, then do all the processing you can in memory and commit the changes at the last second. That way, you only block shutting down while the changes are being committed as opposed to blocking for the entire duration. And for what it's worth, I do prefer using ManualResetEvent for communicating shutdown commands to threads.
To avoid rambling any further, I'll cut it off here. HTH.
EDIT:
This is off the cuff, so I won't verify its accuracy. I'll fix any problem you (or others) may find.
Define two ManualResetEvent objects, one for shutdown notification and one for processing notification, and the Thread object. Change the OnStart() callback to this:
using System.Threading;
using Timer = System.Timers.Timer; // both Threading and Timers have a timer class
ManualResetEvent _shutdownEvent = new ManualResetEvent(false);
ManualResetEvent _processEvent = new ManualResetEvent(false);
Thread _thread;
Timer _oTimer;
protected override void OnStart(string[] args)
{
// Create the formal, foreground thread.
_thread = new Thread(Execute);
_thread.IsBackground = false; // set to foreground thread
_thread.Start();
// Start the timer. Notice the lambda expression for setting the
// process event when the timer elapses.
int intDelai = Properties.Settings.Default.WatchDelay * 1000;
_oTimer = new Timer(intDelai);
_oTimer.AutoReset = false;
_oTimer.Elapsed += (sender, e) => _processEvent.Set();
_oTimer.Start();
}
Change your Execute() callback to something like this:
private void Execute()
{
var handles = new WaitHandle[] { _shutdownEvent, _processEvent };
while (true)
{
switch (WaitHandle.WaitAny(handles))
{
case 0: // Shutdown Event
return; // end the thread
case 1: // Process Event
Process();
_processEvent.Reset(); // reset for next time
_oTimer.Start(); // trigger timer again
break;
}
}
}
Create the Process() method like this:
private void Process()
{
try
{
// Do your processing here. If this takes a long time, you might
// want to periodically check the shutdown event to see if you need
// exit early.
}
catch (Exception ex)
{
// Do your logging here...
// You *could * also shutdown the thread here, but this will not
// stop the service.
_shutdownEvent.Set();
}
}
Finally, in the OnStop() callback, trigger the thread to shutdown:
protected override void OnStop()
{
_oTimer.Stop(); // no harm in calling it
_oTimer.Dispose();
_shutdownEvent.Set(); // trigger the thread to stop
_thread.Join(); // wait for thread to stop
}

#Matt - thanks for the great code, really helpful.
I found it worked even better if I added another test on _shutdownEvent:
case 1: // Process Event
Process();
if(_shutdownEvent.WaitOne(0)) break; // don't loop again if a shutdown is needed
...

Usage multithreading could lead to excessive memory use

I'm having a windows service project that logs messages to a database (or other place). The frequency of these messages could go up to ten per second. Since sending and processing the messages shouldn't delay the main process of the service I start a new thread for the processing of every message. This means that if the main process needs to send 100 log messages, 100 threads are started that process each message. I learned that when a thread is done, it will be cleaned so I don't have to dispose it. As long as I dispose all used objects in the thread everything should be working fine.
The service could go into a exception that leads to shutting down the service. Before the service shuts down it should wait for all threads that were logging messages. To achieve this it adds the thread to a list every time a thread is started. When the wait-for-threads method is called, all threads in the list are checked if it is still alive and if so, it uses join to wait for it.
The code:
Creating the thread:
/// <summary>
/// Creates a new thread and sends the message
/// </summary>
/// <param name="logMessage"></param>
private static void ThreadSend(IMessage logMessage)
{
ParameterizedThreadStart threadStart = new ParameterizedThreadStart(MessageHandler.HandleMessage);
Thread messageThread = new Thread(threadStart);
messageThread.Name = "LogMessageThread";
messageThread.Start(logMessage);
threads.Add(messageThread);
}
The waiting for threads to end:
/// <summary>
/// Waits for threads that are still being processed
/// </summary>
public static void WaitForThreads()
{
int i = 0;
foreach (Thread thread in threads)
{
i++;
if (thread.IsAlive)
{
Debug.Print("waiting for {0} - {1} to end...", thread.Name, i);
thread.Join();
}
}
}
Now my main concern is if this service runs for a month it will still have all threads (millions) in the list (most of them dead). This will eat memory and I don't know how much. This in whole doesn't seem to be a good practice to me, I want to clean up finished threads but I can't find out how to do it. Does any one have a good or best practice for this?

Remove the threads from the list if they are dead?
/// <summary>
/// Waits for threads that are still being processed
/// </summary>
public static void WaitForThreads()
{
List<Thread> toRemove = new List<int>();
int i = 0;
foreach (Thread thread in threads)
{
i++;
if (thread.IsAlive)
{
Debug.Print("waiting for {0} - {1} to end...", thread.Name, i);
thread.Join();
}
else
{
toRemove.Add(thread);
}
}
threads.RemoveAll(x => toRemove.Contains(x));
}
Have a look at Task Parallelism

First of all: Creating one thread per log message is not a good idea. Either use ThreadPool or create a limited number of worker threads which handle the log items from a common queue (producer/consumer).
Second: Of course you need to also remove the thread references from the list! Either when the thread method ends, it can remove itself, or you can even do it on a regular basis. For example, have a timer run every half and hour that checks the list for dead threads and removes them.

If all you're doing in those threads is logging, you should probably have a single logging thread and a shared queue that the main thread puts messages on. The logging thread can then read the queue and log. This is incredibly easy with the BlockingCollection.
Create the queue in the service's main thread:
BlockingCollection<IMessage> LogMessageQueue = new BlockingCollection<IMessage>();
Your service's main thread creates a Logger (see below) instance, which starts a thread to process log messages. The main thread adds items to the LogMessageQueue. The logger thread reads them from the queue. When the main thread wants to shut down, it calls LogMessageQueue.CompleteAdding. The logger will empty the queue and exit.
Main thread would look like this:
// start the logger
Logger _loggingThread = new Logger(LogMessageQueue);
// to log a message:
LogMessageQueue.Add(logMessage);
// when the program needs to shut down:
LogMessageQueue.CompleteAdding();
And the logger class:
class Logger
{
BlockingCollection<IMessage> _queue;
Thread _loggingThread;
public Logger(BlockingCollection<IMessage> queue)
{
_queue = queue;
_loggingThread = new Thread(LoggingThreadProc);
}
private void LoggingThreadProc(object state)
{
IMessage msg;
while (_queue.TryTake(out msg, TimeSpan.Infinite))
{
// log the item
}
}
}
This way you have just one additional thread, messages are guaranteed to be processed in the order they're sent (not true of your current approach), and you don't have to worry about keeping track of thread shutdown, etc.
Update
If some of your log messages will take time to process (the email you described, for example), you can process them asynchronously. For example:
while (_queue.TryTake(out msg, TimeSpan.Infinite))
{
if (msg.Type == Email)
{
// start asynchronous task to send email
}
else
{
// write to log file
}
}
This way, only those messages that potentially take lots of time will run asynchronously. You can also have a secondary queue there if you want, for the email messages. That way you won't get bogged down with a bunch of email threads. Rather, you limit it to one or two, or perhaps a handful.
Note that you can also have multiple Logger instances if you want, all reading from the same message queue. Just make sure they're each writing to a different log file. The queue itself will support multiple consumers.

I think in general the approach to solve your issue is maybe not the best practice.
I mean, instead of creating 1000s of threads, you just want to store 1000s of messages in a database right? And it seems you want to do this asynchronously.
But creating a thread for each message is not really a good idea and actually does not solve that issue...
Instead I would try to implement something like message queues. You can have multiple queues and each queue has its own thread. If messages are coming in, you send them to one of the queues (alternating)...
The queue either waits for a certain amount of messages, or always waits a certain amount of time (e.g. 1 second, depends of how long it takes to store e.g. 100 messages within the database) until it tries to store the queued messages in the database.
This way you should actually always have a constant number of threads and you shouldn't see any performance issues...
Also it would enable you to batch insert data and not only one by one with the overhead of db connections etc...
Of cause, if your database is slower then the tasks are able to store the messages, more and more messages will be queued... But that's true for your current solution, also.

Since multiple answers and comments led to my solution I will post the complete code here.
I used threadpool to manage the threads and code from this page for the wating function.
Creating the thread:
private static void ThreadSend(IMessage logMessage)
{
ThreadPool.QueueUserWorkItem(MessageHandler.HandleMessage, logMessage);
}
Waiting for the threads to finish:
public static bool WaitForThreads(int maxWaitingTime)
{
int maxThreads = 0;
int placeHolder = 0;
int availableThreads = 0;
while (maxWaitingTime > 0)
{
System.Threading.ThreadPool.GetMaxThreads(out maxThreads, out placeHolder);
System.Threading.ThreadPool.GetAvailableThreads(out availableThreads, out placeHolder);
//Stop if all threads are available
if (availableThreads == maxThreads)
{
return true;
}
System.Threading.Thread.Sleep(TimeSpan.FromMilliseconds(1000));
--maxWaitingTime;
}
return false;
}
Optionally you can add this somewhere outside these methods to limit the amount of threads in the pool.
System.Threading.ThreadPool.SetMaxThreads(MaxWorkerThreads, MaxCompletionPortThreads);

Need ideas on implementing the cancellation on this background worker

I have a button which initiates a connection between the program and a machine known as DDC. To prevent the UI thread from being locked while the connection is being established, I naturally create a backgroundworker to handle the job.
The problem is this: In the event where the client doesn't get any response from the DDC, the client has to wait for 30 seconds before giving a timeout. More precisely speaking, the thread halts executing on the line "GetLogicStatusListResponse result = ddcdao.GetLogicStatusList();".
I'd like the user to be able to cancel the operation if a button has been clicked on the program, but as far as I know there is no way to abort backgroundworker and cancellation wouldn't really help me here since the thread is locked for 30 seconds before the code can even check if cancellationpending is true or not.
I would like to hear some ideas to properly implement the cancellation feature so that the user may start/stop the connection whenever he/she wishes to.
private void bwWorkerConnect_DoWork(object sender, DoWorkEventArgs e)
{
//Branch in only if the DDC returns a ping response
if (DDCGlobal.DDCPingTest(ddc))
{
try
{
//Creates an object of class which communicates with the engine
//Retrieves the object obtained through remoting (Engine)
//and casts it to a ReqDDCLogicListResponse type (See CommonEngine)
DDCDAO ddcdao = new DDCDAO(DDCGlobal.ddcEngineIP, ddc.Ip);
//Request status list from DDC (TIMEOUT = 30 SECONDS)
GetLogicStatusListResponse result = ddcdao.GetLogicStatusList();
...
if (bwWorkerConnect.CancellationPending)
{
e.Cancel = true;
}
public void LogicListLoad()
{
if (!bwWorkerConnect.IsBusy)
bwWorkerConnect.RunWorkerAsync();
else
{
MessageBox.Show(UIConstant.DDC_CONNECT_ALREADY_WARNING);
}
}

Posting my comment as answer for future reference.
You can approach your issue with a second BackgroundWorker running alongside the first, which only polls a flag in the first one to verify its status: since he's independent, it doesn't have to "wait" and is free to kill the first worker in case it gets stuck.
This should be deadlock-free since the second worker does nothing more than a periodic "health check" of the first worker as long as the flag handling instructions are wrapped up in a lock.
Flow would look like this:
When you need a connection, fire up both workers
First worker proceeds to attempt the connection, sets a flag to notify the outer world its "connecting" status
Second worker might check the first worker status every second 30 times: if it never reads anything different from "connecting" it can proceed and kill the first worker. It might also react to user input and kill the first worker before time (ie. any user input cancels the second worker, which in turn can kill the first one before its own demise).
Flow end

How to ensure order of async operation calls?

[This appears to be a loooong question but I have tried to make it as clear as possible. Please have patience and help me...]
I have written a test class which supports an Async operation. That operation does nothing but reports 4 numbers:
class AsyncDemoUsingAsyncOperations
{
AsyncOperation asyncOp;
bool isBusy;
void NotifyStarted () {
isBusy = true;
Started (this, new EventArgs ());
}
void NotifyStopped () {
isBusy = false;
Stopped (this, new EventArgs ());
}
public void Start () {
if (isBusy)
throw new InvalidOperationException ("Already working you moron...");
asyncOp = AsyncOperationManager.CreateOperation (null);
ThreadPool.QueueUserWorkItem (new WaitCallback (StartOperation));
}
public event EventHandler Started = delegate { };
public event EventHandler Stopped = delegate { };
public event EventHandler<NewNumberEventArgs> NewNumber = delegate { };
private void StartOperation (object state) {
asyncOp.Post (args => NotifyStarted (), null);
for (int i = 1; i < 5; i++)
asyncOp.Post (args => NewNumber (this, args as NewNumberEventArgs), new NewNumberEventArgs (i));
asyncOp.Post (args => NotifyStopped (), null);
}
}
class NewNumberEventArgs: EventArgs
{
public int Num { get; private set; }
public NewNumberEventArgs (int num) {
Num = num;
}
}
Then I wrote 2 test programs; one as console app and another as windows form app. Windows form app works as expected when I call Start repeatedly:
But console app has hard time ensuring the order:
Since I am working on class library, I have to ensure that my library works correctly in all app models (Haven't tested in ASP.NET app yet). So I have following questions:
I have tested enough times and it appears to be working but is it OK to assume above code will always work in windows form app?
Whats the reason it (order) doesn't work correctly in console app? How can I fix it?
Not much experienced with ASP.NET. Will the order work in ASP.NET app?
[EDIT: Test stubs can be seen here if that helps.]

Unless I am missing something then given the code above I believe there is no way of guaranteeing the order of execution. I have never used the AsyncOperation and AsyncOperationManager class but I looked in reflector and as could be assumed AsyncOperation.Post uses the thread pool to execute the given code asynchronously.
This means that in the example you have provided 4 tasks will be queued to the thread pool synchronously in very quick succession. The thread pool will then dequeue the tasks in FIFO order (first in first out) but it's entirely possible for one of later threads to be scheduled before an earlier one or one of the later threads to complete before an earlier thread has completed its work.
Therefore given what you have there is no way to control the order in the way you desire. There are ways to do this, a good place to look is this article on MSDN.
http://msdn.microsoft.com/en-us/magazine/dd419664.aspx

I use a Queue you can then Enqueue stuff and Dequeue stuff in the correct order. This solved this problem for me.

The documentation for AsyncOperation.Post states:
Console applications do not synchronize the execution of Post calls. This can cause ProgressChanged events to be raised out of order. If you wish to have serialized execution of Post calls, implement and install a System.Threading.SynchronizationContext class.
I think this is the exact behavior you’re seeing. Basically, if the code that wants to subscribe to notifications from your asynchronous event wants to receive the updates in order, it must ensure that there is a synchronization context installed and that your AsyncOperationManager.CreateOperation() call is run inside of that context. If the code consuming the asynchronous events doesn’t care about receiving them in the correct order, it simply needs to avoid installing a synchronization context which will result in the “default” context being used (which just queues calls directly to the threadpool which may execute them in any order it wants to).
In the GUI version of your application, if you call your API from a UI thread, you will automatically have a synchronization context. This context is wired up to use the UI’s message queueing system which guarantees that posted messages are processed in order and serially (i.e., not concurrently).
In a Console application, unless if you manually install your own synchronization context, you will be using the default, non-synchronizing threadpool version. I am not exactly sure, but I don’t think that .net makes installing a serializing synchronization context very easy to do. I just use Nito.AsyncEx.AsyncContext from the Nito.AsyncEx nuget package to do that for me. Basically, if you call Nito.AsyncEx.AsyncContext.Run(MyMethod), it will capture the current thread and run an event loop with MyMethod as the first “handler” in that event loop. If MyMethod calls something that creates an AsyncOperation, that operation increments an “ongoing operations” counter and that loop will continue until the operation is completed via AsyncOperation.PostOperationCompleted or AsyncOperation.OperationCompleted. Just like the synchronization context provided by a UI thread, AsyncContext will queue posts from AsyncOperation.Post() in the order it receives them and run them serially in its event loop.
Here is an example of how to use AsyncContext with your demo asynchronous operation:
class Program
{
static void Main(string[] args)
{
Console.WriteLine("Starting SynchronizationContext");
Nito.AsyncEx.AsyncContext.Run(Run);
Console.WriteLine("SynchronizationContext finished");
}
// This method is run like it is a UI callback. I.e., it has a
// single-threaded event-loop-based synchronization context which
// processes asynchronous callbacks.
static Task Run()
{
var remainingTasks = new Queue<Action>();
Action startNextTask = () =>
{
if (remainingTasks.Any())
remainingTasks.Dequeue()();
};
foreach (var i in Enumerable.Range(0, 4))
{
remainingTasks.Enqueue(
() =>
{
var demoOperation = new AsyncDemoUsingAsyncOperations();
demoOperation.Started += (sender, e) => Console.WriteLine("Started");
demoOperation.NewNumber += (sender, e) => Console.WriteLine($"Received number {e.Num}");
demoOperation.Stopped += (sender, e) =>
{
// The AsyncDemoUsingAsyncOperation has a bug where it fails to call
// AsyncOperation.OperationCompleted(). Do that for it. If we don’t,
// the AsyncContext will never exit because there are outstanding unfinished
// asynchronous operations.
((AsyncOperation)typeof(AsyncDemoUsingAsyncOperations).GetField("asyncOp", BindingFlags.NonPublic|BindingFlags.Instance).GetValue(demoOperation)).OperationCompleted();
Console.WriteLine("Stopped");
// Start the next task.
startNextTask();
};
demoOperation.Start();
});
}
// Start the first one.
startNextTask();
// AsyncContext requires us to return a Task because that is its
// normal use case.
return Nito.AsyncEx.TaskConstants.Completed;
}
}
With output:
Starting SynchronizationContext
Started
Received number 1
Received number 2
Received number 3
Received number 4
Stopped
Started
Received number 1
Received number 2
Received number 3
Received number 4
Stopped
Started
Received number 1
Received number 2
Received number 3
Received number 4
Stopped
Started
Received number 1
Received number 2
Received number 3
Received number 4
Stopped
SynchronizationContext finished
Note that in my example code I work around a bug in AsyncDemoUsingAsyncOperations which you should probably fix: when your operation stops, you never call AsyncOperation.OperationCompleted or AsyncOperation.PostOperationCompleted. This causes AsyncContext.Run() to hang forever because it is waiting for the outstanding operations to complete. You should make sure that your asynchronous operations complete—even in error cases. Otherwise you might run into similar issues elsewhere.
Also, my demo code, to imitate the output you showed in the winforms and console example, waits for each operation to finish before starting the next one. That kind of defeats the point of asynchronous coding. You can probably tell that my code could be greatly simplified by starting all four tasks at once. Each individual task would receive its callbacks in the correct order, but they would all make progress concurrently.
Recommendation
Because of how AsyncOperation seems to work and how it is intended to be used, it is the responsibility of the caller of an asynchronous API that uses this pattern to decide if it wants to receive events in order or not. If you are going to use AsyncOperation, you should document that the asynchronous events will only be received in order by the caller if the caller has a synchronization context that enforces serialization and suggest that the caller call your API on either a UI thread or in something like AsyncContext.Run(). If you try to use synchronization primitives and whatnot inside of the delegate you call with AsyncOperation.Post(), you could end up putting threadpool threads in a sleeping state which is a bad thing when considering performance and is completely redundant/wasteful when the caller of your API has properly set up a synchronization context already. This also enables the caller to decide that, if it is fine with receiving things out of order, that it is willing to process events concurrently and out of order. That may even enable speedup depending on what you’re doing. Or you might even decide to put something like a sequence number in your NewNumberEventArgs in case the caller wants both concurrency and still needs to be able to assemble the events into order at some point.