Develop Reference

MPI.NET Send/ImmediateProbe not working when using multiple hosts

I developed an MPI test program where the master node distributes work to the worker nodes.
The worker node uses comm.Send() to request work and the master node checks with comm.ImmediateProbe if any of the worker nodes wants to request some work. If a request is available it is read with comm.Receive and the work is sent to the worker for processing.
When I run my test program with mpiexec.exe on a single host, either localhost or also a remote host everything works as expected, but when I run it on two hosts at the same time
the Send on the remote host blocks and the master nodes ImmediateProbe never receives the message sent from the worker on the remote host.
I run the program with mpiexec.exe -wdir \\DESKTOP-58QONBS\MPITest -hosts 2 DESKTOP-58QONBS 2 LAPTOP-L8F7AN5R 1 MPITest.exe
I'm new to MPI, so maybe I am doing something wrong I just could not figure out yet why the behaviour is like this when using two hosts at the same time.
The full code is below:
using System;
using System.Collections.Concurrent;
using System.Linq;
using System.Threading;
namespace MPITest
{
public abstract class MPIMasterWorkerBase<TWork, TResult>
where TWork : class
where TResult : class
{
protected abstract void Initialize(bool isMaster);
protected abstract void Main();
protected abstract void ProcessResult(TResult result);
protected abstract TResult ProcessWork(TWork work);
protected abstract TWork GetWork();
private volatile bool terminate = false;
private Thread thread;
private MPI.Intracommunicator comm;
public void Run(string[] args)
{
MPI.Environment.Run(ref args, comm =>
{
this.comm = comm;
if (comm.Size < 2)
{
Console.WriteLine("At least 2 processes are required.");
return;
}
if (comm.Rank == 0)
{
Initialize(isMaster: true);
thread = new Thread(MasterThread);
thread.Start();
Main();
terminate = true;
thread.Join();
}
else
{
Initialize(isMaster: false);
thread = new Thread(WorkerThread);
thread.Start();
thread.Join();
}
});
}
private void MasterThread()
{
Console.WriteLine($"MasterStart {MPI.Environment.ProcessorName}");
var done = new bool[comm.Size];
done[0] = true;
while (!done.All(x => x == true))
{
for (int i = 1; i < comm.Size; i++)
{
if (comm.ImmediateProbe(i, 0) != null)
{
Console.WriteLine($"Receive: {i}");
comm.Receive<int>(i, 0);
var work = GetWork();
if (work != null)
{
comm.Send(1, i, 0);
comm.Send(work, i, 0);
}
else
{
if (terminate)
{
comm.Send(-1, i, 0);
done[i] = true;
}
else
{
comm.Send(0, i, 0);
}
}
}
if (comm.ImmediateProbe(i, 1) != null)
{
var result = comm.Receive<TResult>(i, 1);
ProcessResult(result);
}
}
Thread.Sleep(1000);
}
Console.WriteLine("MasterStop");
}
private void WorkerThread()
{
Console.WriteLine($"WorkerStart: {comm.Rank} {MPI.Environment.ProcessorName}");
while (!terminate)
{
Thread.Sleep(1000);
Console.WriteLine($"Send: {comm.Rank}");
comm.Send(0, 0, 0);
var flag = comm.Receive<int>(0, 0);
if (flag == -1)
break;
else if (flag == 0)
continue;
var work = comm.Receive<TWork>(0, 0);
var result = ProcessWork(work);
comm.Send(result, 0, 1);
}
Console.WriteLine($"WorkerStop: {comm.Rank}");
}
}
[Serializable]
public class WorkItem
{
public int Id { get; set; }
}
public class MPITest : MPIMasterWorkerBase<WorkItem, WorkItem>
{
private ConcurrentQueue<WorkItem> queue = new();
private int id;
protected override void Initialize(bool isMaster)
{
}
protected override void Main()
{
var startTime = DateTime.UtcNow;
while ((DateTime.UtcNow - startTime).TotalSeconds < 10)
{
for (int i = 0; i < 2; i++)
queue.Enqueue(new WorkItem { Id = id++ });
Thread.Sleep(1000);
}
}
protected override WorkItem GetWork()
{
if (queue.TryDequeue(out var result))
return result;
return null;
}
protected override WorkItem ProcessWork(WorkItem work)
{
Console.WriteLine($"Processing Work {work.Id}");
return work;
}
protected override void ProcessResult(WorkItem result)
{
Console.WriteLine($"Process Result {result.Id}");
}
}
class Program
{
static void Main(string[] args)
{
new MPITest().Run(args);
}
}
}

The comm.Send was blocking, but after some minutes of waiting the program started to work.
The issues were caused by the VirtualBox Host-Only Network Adapter that was also installed on the system. Disabling this adapter in the network settings resolved all the issues.

c# Threading Lock and Monitor

I'm currently running into a problem with multithreading and accessing a static list. A static list holds all items with several properties. The items are identified with a Guid. A main work thread changes some properties for any item in the static list. The child threads all have their own Guid, with this Guid they read their own item in the static list. And after a specific event they remove their assigned element from the static list.
To get to the source I have broken down my code to the essential methods and classes. The work thread has the following simplified code
public void RunWork()
{
Random random = new Random();
Int32 index = -1;
while (!Kill)
{
Thread.Sleep(1);
if (MainWindow.Clients != null)
{
index = random.Next(0, MainWindow.Clients.Count);
MainWindow.Clients[index].State = MainWindow.RandomString(9);
}
}
}
Each child thread has the following simplified code
public void RunChild()
{
Random random = new Random();
while (!Kill)
{
Thread.Sleep(100);
if (MainWindow.Clients.Any(x => x.Id == Id))
{
this.State = MainWindow.Clients.First(x => x.Id == Id).State;
}
Thread.Sleep(random.Next(50));
if (random.Next(100) % 90 == 0)
{
Kill = true;
MainWindow.Clients.RemoveAll(x => x.Id == Id);
}
}
}
If a child removes itself from the MainWindow.Clients list the work thread throws a exception, that the index it is trying to access does not exist.
I have added lock statments around every access of MainWindow.Clients but this does not prevent the work thread from accessing a deleted item. I have also tried Monitor.Enter(MainWindow.Clients) and Monitor.Exit(MainWindow.Clients) but with the same result as with lock.
The static list MainWindow.Clients is created before any thread runs and never gets recreated or disposed.
If the lock statement is set around this block of code in the RunWork() method
lock (MainWindow.Clients)
{
Thread.Sleep(1);
if (MainWindow.Clients != null)
{
index = random.Next(0, MainWindow.Clients.Count);
MainWindow.Clients[index].State = MainWindow.RandomString(9);
}
}
Why does it not block the child threads from changing the list between the lines
where the random index is set and the list gets accessed?
Update 1:
The following code still throws a IndexOutOfRangeException at MainWindow.Clients[index].State = MainWindow.RandomString(9);:
public void RunWork()
{
Random random = new Random();
Int32 index = -1;
while (!Kill)
{
Thread.Sleep(1);
if (MainWindow.Clients != null)
{
lock (MainWindow.Clients)
{
index = random.Next(0, MainWindow.Clients.Count);
MainWindow.Clients[index].State = MainWindow.RandomString(9);
}
}
}
}
public void RunChild()
{
Random random = new Random();
while (!Kill)
{
Thread.Sleep(100);
if (MainWindow.Clients.Any(x => x.Id == Id))
{
this.State = MainWindow.Clients.First(x => x.Id == Id).State;
}
Thread.Sleep(random.Next(50));
if (random.Next(100) % 90 == 0)
{
Kill = true;
lock (MainWindow.Clients)
{
MainWindow.Clients.RemoveAll(x => x.Id == Id);
}
}
}
}
Update 2: Here is the complete code for the quick sample application
Update 3: I have edited my code and wrapped all accesses of MainWindow.Clients with lock statements. But still the threads access the variable while it is locked:

I'm not sure what exactly you are trying to achieve, but I've written something that might help you find the correct solution. Sorry for the lack of correctness - tight schedule ;-)
using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Threading;
namespace ConcurrentCollectionTest
{
internal class Client
{
public string State
{
get; set;
}
public string Name
{
get;
internal set;
}
}
internal class MainWindow
{
private ConcurrentDictionary<int, Client> _dict = new ConcurrentDictionary<int, Client>();
public IDictionary<int, Client> Clients
{
get
{
return _dict;
}
}
}
internal class Program
{
private static bool killAll = false;
private static MainWindow mainWindow = new MainWindow();
private static int id = -100;
private static string state = "Initial";
private static Random random = new Random();
private static object lockObject = new object();
internal static string RandomString(int v)
{
int k = random.Next(0, v);
return k.ToString();
}
public static void RunChild()
{
Debug.WriteLine($"child running {Thread.CurrentThread.Name}");
bool killThis = false;
while (!killThis && !killAll)
{
Thread.Sleep(100);
Client client = null;
if (mainWindow.Clients.TryGetValue(id, out client))
{
state = client.State;
}
Thread.Sleep(random.Next(50));
if (random.Next(100) % 90 == 0)
{
Debug.WriteLine($"killing {Thread.CurrentThread.Name}");
killThis = true;
lock (lockObject)
{
mainWindow.Clients.Remove(id);
}
}
}
}
public static void RunWork()
{
Console.WriteLine("RunWork");
Random random = new Random();
Int32 index = -1;
while (!killAll)
{
if (!mainWindow.Clients.Any())
{
killAll = true;
break;
}
Thread.Sleep(100);
// edit: still need lock here as count can change in between
Client client = null;
lock (lockObject)
{
index = random.Next(0, mainWindow.Clients.Count);
client = mainWindow.Clients[index];
}
Debug.WriteLine($"Changing {client.Name}");
client.State = RandomString(9);
}
Console.WriteLine("Worker killed");
}
private static void Main(string[] args)
{
Console.WriteLine("Starting. Enter id or kill");
for (int i = 0; i < 100; i++)
{
mainWindow.Clients.Add(i, new Client
{
Name = $"Client {i:000}",
State = "Unknown"
});
}
var worker = new Thread(RunWork);
worker.Start();
var threadList = new List<Thread>();
threadList.Add(worker);
for (int i = 0; i < 10; i++)
{
var thread = new Thread(RunChild)
{
Name = $"Child {i:00}"
};
threadList.Add(thread);
thread.Start();
}
while (!killAll)
{
var str = Console.ReadLine();
if (str.Equals("kill", StringComparison.InvariantCultureIgnoreCase))
{
killAll = true;
break;
}
int enteredId = -1;
if (int.TryParse(str, out enteredId))
{
id = enteredId;
}
}
foreach (var thread in threadList)
{
thread.Join();
}
Console.WriteLine("all dead");
}
}
}

How to wait for a method to finish on another thread?

I am new to multi-thread programming in C#. My problem is that I don't know how to wait for a method that is being run on another thread to finish, before it can continue to the next line. For example, something like this
public class A
{
int i;
public A()
{
i = 0;
}
protected void RunLoop()
{
while(i < 100)
{
i++;
}
}
public void Start()
{
TimerResolution.TimeBeginPeriod(1);
runThread = new Thread(new ThreadStart(RunLoop));
running = true;
runThread.Start();
}
}
public class B
{
A classAInstance = new A();
A.Start();
Console.Writeline(i);
}
Right now, it prints 0 on the console, which is not what I want (i.e. i = 100).
What is the best way to do this? BTW, I don't have access to the runThread that is created in class A
Thanks.
EDIT:
It was a bit difficult to solve this problem without modifying a lot codes. Therefore, we ended up with adding a condition in the public void Start() with which it can decide whether to run the RunLoop in a separate thread or not. The condition was defined using an Enum field.
public void Start()
{
TimerResolution.TimeBeginPeriod(1);
running = true;
if (runningMode == RunningMode.Asynchronous)
{
runThread = new Thread(new ThreadStart(RunLoop));
runThread.Start();
}
else
{
RunLoop();
}
}
And
public enum RunningMode { Asynchronous, Synchronous };
Thanks everyone for help.

The preferred method is to use the Task Parallel Library (TPL) and use Task with await.
If you must use Threads, then use a ManualResetEvent or ManualResetEventSlim to signal the end of a method.
void Main()
{
var a = new A();
a.Start();
a.FinishedEvent.WaitOne();
Console.WriteLine(a.Index);
}
// Define other methods and classes here
public class A
{
ManualResetEvent mre = new ManualResetEvent(false);
int i;
public EventWaitHandle FinishedEvent
{
get { return mre; }
}
public int Index
{
get { return i; }
}
public A()
{
i = 0;
}
protected void RunLoop()
{
while (i < 1000)
{
i++;
}
mre.Set();
}
public void Start()
{
var runThread = new Thread(new ThreadStart(RunLoop));
runThread.Start();
}
}

Your life would be so much better with tasks.
Your code could be this simple:
var task = Task.Factory.StartNew(() =>
{
var i = 0;
while (i < 100)
{
i++;
}
return i;
});
Console.WriteLine(task.Result);

I like use Monitor.Wait() and Monitor.Pulse() in conjunction with "lock" operator. It works, but you must be careful, when you use this technique.
I'm added some changes to your code to demonstrate it. Code below are prints i== 100, as you want.
public class A
{
int i;
public object SyncObject
{ get; private set; }
public A()
{
SyncObject = new object();
i = 0;
}
protected void RunLoop()
{
while (i < 100)
{
i++;
}
lock (SyncObject)
{
Monitor.Pulse(SyncObject);
}
}
public void Start()
{
var runThread = new Thread(new ThreadStart(RunLoop));
runThread.Start();
}
public void PrintI()
{
Console.WriteLine("I == " + i);
}
}
public class B
{
public static void Run()
{
A classAInstance = new A();
lock (classAInstance.SyncObject)
{
classAInstance.Start();
Monitor.Wait(classAInstance.SyncObject);
}
classAInstance.PrintI();
}
}

Print Even and Odd numbers in c# using two thread ie even thread and odd thread

/*My requirements is
one threads should print even numbers and the other should print odd numbers.
These threads should print the numbers in order (1, 2, 3, 4, 5...)
I have done this code but when my commenting either method countThreadOdd.Start() or countThreadEven.Start(), it is not printing only even or odd numbers.*/
class Program
{
static Object locker = new Object();
static LinkedList<int> number = new LinkedList<int>();
static int counter = 0;
static void Main(string[] args)
{
Thread countThreadOdd = new Thread(oddThread);
Thread countThreadEven = new Thread(evenThread);
//Thread Start
countThreadOdd.Start();
countThreadEven.Start();
//main thread will untill below thread is in exection mode
countThreadOdd.Join(10);
countThreadEven.Join(10);
Console.ReadLine();
}
//Odd Thread
public static void oddThread()
{
for (; counter < 10; )
{
//Lock the another thread to enter in critial area
lock (locker)
{
if (counter % 2 != 0)
{
Console.WriteLine(counter);
counter++;
}
}
}
}
//Even Thread
public static void evenThread()
{
for (; counter < 10; )
{
//Lock the another thread to enter in critial area
lock (locker)
{
if (counter % 2 == 0)
{
Console.WriteLine(counter);
counter++;
}
}
}
}
}

If you want to alternate between two threads, you can use two AutoResetEvent objects to do so, like so:
public static void oddThread()
{
for (int i = 1; i < 10; i +=2)
{
evenReady.WaitOne();
Console.WriteLine(i);
oddReady.Set();
}
}
public static void evenThread()
{
for (int i = 0; i < 10; i += 2)
{
oddReady.WaitOne();
Console.WriteLine(i);
evenReady.Set();
}
}
If you only want to run one of the threads, you can use a ManualResetEvent instead to effectively remove all locking.
A full example:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading;
using Dmr.Common.Resources;
namespace Demo
{
class Program
{
static EventWaitHandle evenReady;
static EventWaitHandle oddReady;
static void Main(string[] args)
{
bool countOdd = true; // Change these to true/false as wanted.
bool countEven = true;
if (countOdd && countEven)
{
evenReady = new AutoResetEvent(false);
oddReady = new AutoResetEvent(true); // Must be true for the starting thread.
}
else
{
evenReady = new ManualResetEvent(true);
oddReady = new ManualResetEvent(true);
}
Thread countThreadOdd = new Thread(oddThread);
Thread countThreadEven = new Thread(evenThread);
//Thread Start
if (countOdd)
countThreadOdd.Start();
if (countEven)
countThreadEven.Start();
//main thread will untill below thread is in exection mode
if (countOdd)
countThreadOdd.Join();
if (countEven)
countThreadEven.Join();
Console.WriteLine("Done");
Console.ReadLine();
}
public static void oddThread()
{
for (int i = 1; i < 10; i +=2)
{
evenReady.WaitOne();
Console.WriteLine(i);
oddReady.Set();
}
}
public static void evenThread()
{
for (int i = 0; i < 10; i += 2)
{
oddReady.WaitOne();
Console.WriteLine(i);
evenReady.Set();
}
}
}
}

static AutoResetEvent evenReady = new AutoResetEvent(true);
static AutoResetEvent oddReady = new AutoResetEvent(false);
static void Main()
{
Thread countThreadOdd = new Thread(oddThread);
Thread countThreadEven = new Thread(evenThread);
countThreadOdd.Start();
countThreadEven.Start();
Console.WriteLine("Done");
Console.ReadLine();
}
public static void oddThread()
{
for (int i = 1; i < 10; i += 2)
{
oddReady.Set();
evenReady.WaitOne();
Console.WriteLine("Odd Thread: " + i);
//oddReady.Set();
}
}
public static void evenThread()
{
for (int i = 0; i < 10; i += 2)
{
oddReady.WaitOne();
evenReady.Set();
Console.WriteLine("Even Thread: " + i);
}
}

You can actually use Interlocked to communicate between threads. Interlocked allows you to share a variable concurrently across two threads.
using System;
using System.Threading;
using System.Threading.Tasks;
namespace InterlockedTest
{
class Program
{
private static long _counter = 0;
private static void printEvenTask()
{
while (Interlocked.Read(ref _counter) < 100)
{
if (Interlocked.Read(ref _counter) % 2 == 0)
{
Console.WriteLine(Interlocked.Read(ref _counter));
Interlocked.Increment(ref _counter);
}
}
}
private static void printOddTask()
{
while (Interlocked.Read(ref _counter) < 100)
{
if (Interlocked.Read(ref _counter) % 2 == 1)
{
Console.WriteLine(Interlocked.Read(ref _counter));
Interlocked.Increment(ref _counter);
}
}
}
static void Main(string[] args)
{
Task oddTask = Task.Run(() => printOddTask());
Task evenTask = Task.Run(() => printEvenTask());
oddTask.Wait();
evenTask.Wait();
Console.ReadKey();
}
}
}

Try this method. It uses Task Library.
public class OddEvenThread
{
public static async Task printEvenNumber(int n)
{
for (int i = 1; i <= n; i++)
{
if (i % 2 == 0)
Console.WriteLine(i);
}
await Task.Delay(0);
}
private static async Task printOddNumbers(int n)
{
for (int i = 1; i <= n; i++)
{
if (i % 2 == 1)
Console.WriteLine(i);
}
await Task.Delay(0);
}
public async static Task printNumbers(int n)
{
Task evenNumbers = printEvenNumber(n);
Task oddNumbers = printOddNumbers(n);
List<Task> tasks = new List<Task>() { evenNumbers, oddNumbers };
await Task.WhenAll(tasks);
}
}

Using AutoResetEvent, threads can be made wait for each other. Here, two threads write numbers from 1 to 20:
using System;
using System.Threading;
namespace oddeven
{
class Program
{
static void Main(string[] args)
{
C c = new C();
Thread t1 = new Thread(c.PrintOdd);
Thread t2 = new Thread(c.PrintEven);
t1.Start();
t2.Start();
}
}
class C
{
AutoResetEvent e1 = new AutoResetEvent(true);
AutoResetEvent e2 = new AutoResetEvent(true);
int j = 1;
public void PrintOdd()
{
while (j < 20)
{
if (j % 2 != 0)
{
Console.WriteLine(j);
j++;
}
e1.Set();
e2.WaitOne();
}
}
public void PrintEven()
{
while (j <= 20)
{
e1.WaitOne();
if (j % 2 == 0)
{
Console.WriteLine(j);
j++;
}
e2.Set();
}
}
}
}

Randomly Timeout(TimeSpan) doesn't work in Rx

The earlier post seems not very clear, so after some testing, I reopened this post with much more simplified words, hope somebody could help.
My singleton observable was turned from multiple source of I/O events, means they're concurrently raised up in underlying, based on testing (to prove Rx is not thread safe) and RX design guideline, I made it serialized, see that lock(...):
public class EventFireCenter
{
public static event EventHandler<GTCommandTerminalEventArg> OnTerminalEventArrived;
private static object syncObject = new object();
public static void TestFireDummyEventWithId(int id)
{
lock (syncObject)
{
var safe = OnTerminalEventArrived;
if (safe != null)
{
safe(null, new GTCommandTerminalEventArg(id));
}
}
}
}
This is the singleton Observable:
public class UnsolicitedEventCenter
{
private readonly static IObservable<int> publisher;
static UnsolicitedEventCenter()
{
publisher = Observable.FromEventPattern<GTCommandTerminalEventArg>(typeof(EventFireCenter), "OnTerminalEventArrived")
.Select(s => s.EventArgs.Id);
}
private UnsolicitedEventCenter() { }
/// <summary>
/// Gets the Publisher property to start observe an observable sequence.
/// </summary>
public static IObservable<int> Publisher { get { return publisher; } }
}
The scenario of Subscribe(...) can be described by following code, you can see the Subscribe(...) could be called concurrently in different threads:
for (var i = 0; i < concurrentCount; i++)
{
var safe = i;
Scheduler.Default.Schedule(() =>
{
IDisposable dsp = null;
dsp = UnsolicitedEventCenter.Publisher
.Timeout(TimeSpan.FromMilliseconds(8000))
.Where(incomingValue => incomingValue == safe)
.ObserveOn(Scheduler.Default)
//.Take(1)
.Subscribe((incomingEvent) =>
{
Interlocked.Increment(ref onNextCalledTimes);
dsp.Dispose();
}
, ex =>
{
Interlocked.Increment(ref timeoutExceptionOccurredTimes);
lock (timedOutEventIds)
{
// mark this id has been timed out, only for unit testing result check.
timedOutEventIds.Add(safe);
}
dsp.Dispose();
});
Interlocked.Increment(ref threadPoolQueuedTaskCount);
});
}
As pointed out times by experienced people, call Dispose() in OnNext(...) is not recommended, but let's ignore it here since the code was from production.
Now the problem is randomly that .Timeout(TimeSpan.FromMilliseconds(8000)) is not working, the ex was never called, anyone could see any abnormal in the code?
for testing, I setup the stress testing, but so far, I didn't reproduced it, while in production, it appeared several times per day. Just in case, I pasted all the testing code:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Reactive.Concurrency;
using System.Reactive.Linq;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
namespace Rx
{
class Program
{
static void Main(string[] args)
{
// avoid thread creation delay in thread pool.
ThreadPool.SetMinThreads(200, 50);
// let the test run for 100 times
for (int t = 0; t < 100; t++)
{
Console.WriteLine("");
Console.WriteLine("======Current running times: " + t);
// at meantime, 150 XXX.Subscribe(...) will be called.
const int concurrentCount = 150;
// how many fake event will be fire to santisfy that 150 XXX.Subscribe(...).
const int fireFakeEventCount = 40;
int timeoutExceptionOccurredTimes = 0;
var timedOutEventIds = new List<int>();
int onNextCalledTimes = 0;
int threadPoolQueuedTaskCount = 0;
for (var i = 0; i < concurrentCount; i++)
{
var safe = i;
Scheduler.Default.Schedule(() =>
{
IDisposable dsp = null;
dsp = UnsolicitedEventCenter.Publisher
.Timeout(TimeSpan.FromMilliseconds(8000))
.Where(incomingValue => incomingValue == safe)
.ObserveOn(Scheduler.Default)
//.Take(1)
.Subscribe((incomingEvent) =>
{
Interlocked.Increment(ref onNextCalledTimes);
dsp.Dispose();
}
, ex =>
{
Interlocked.Increment(ref timeoutExceptionOccurredTimes);
lock (timedOutEventIds)
{
// mark this id has been timed out, only for unit testing result check.
timedOutEventIds.Add(safe);
}
dsp.Dispose();
});
Interlocked.Increment(ref threadPoolQueuedTaskCount);
});
}
Console.WriteLine("Starting fire event: " + DateTime.Now.ToString("HH:mm:ss.ffff"));
int threadPoolQueuedTaskCount1 = 0;
// simulate a concurrent event fire
for (int i = 0; i < fireFakeEventCount; i++)
{
var safe = i;
Scheduler.Default.Schedule(() =>
{
EventFireCenter.TestFireDummyEventWithId(safe);
Interlocked.Increment(ref threadPoolQueuedTaskCount1);
});
}
// make sure all proceeding task has been done in threadPool.
while (threadPoolQueuedTaskCount < concurrentCount)
{
Thread.Sleep(1000);
}
// make sure all proceeding task has been done in threadPool.
while (threadPoolQueuedTaskCount1 < fireFakeEventCount)
{
Thread.Sleep(100);
}
Console.WriteLine("Finished fire event: " + DateTime.Now.ToString("HH:mm:ss.ffff"));
// sleep a time which >3000ms.
Thread.Sleep(8000);
Console.WriteLine("timeoutExceptionOccurredTimes: " + timeoutExceptionOccurredTimes);
Console.WriteLine("onNextCalledTimes: " + onNextCalledTimes);
if ((concurrentCount - fireFakeEventCount) != timeoutExceptionOccurredTimes)
{
try
{
Console.WriteLine("Non timeout fired for these ids: " +
Enumerable.Range(0, concurrentCount)
.Except(timedOutEventIds).Except(Enumerable.Range(0, fireFakeEventCount)).Select(i => i.ToString())
.Aggregate((acc, n) => acc + "," + n));
}
catch (Exception ex) { Console.WriteLine("faild to output timedout ids..."); }
break;
}
if (fireFakeEventCount != onNextCalledTimes)
{
Console.WriteLine("onNextOccurredTimes assert failed");
break;
}
if ((concurrentCount - fireFakeEventCount) != timeoutExceptionOccurredTimes)
{
Console.WriteLine("timeoutExceptionOccurredTimes assert failed");
break;
}
}
Console.WriteLine("");
Console.WriteLine("");
Console.WriteLine("DONE!");
Console.ReadLine();
}
}
public class EventFireCenter
{
public static event EventHandler<GTCommandTerminalEventArg> OnTerminalEventArrived;
private static object syncObject = new object();
public static void TestFireDummyEventWithId(int id)
{
lock (syncObject)
{
var safe = OnTerminalEventArrived;
if (safe != null)
{
safe(null, new GTCommandTerminalEventArg(id));
}
}
}
}
public class UnsolicitedEventCenter
{
private readonly static IObservable<int> publisher;
static UnsolicitedEventCenter()
{
publisher = Observable.FromEventPattern<GTCommandTerminalEventArg>(typeof(EventFireCenter), "OnTerminalEventArrived")
.Select(s => s.EventArgs.Id);
}
private UnsolicitedEventCenter() { }
/// <summary>
/// Gets the Publisher property to start observe an observable sequence.
/// </summary>
public static IObservable<int> Publisher { get { return publisher; } }
}
public class GTCommandTerminalEventArg : System.EventArgs
{
public GTCommandTerminalEventArg(int id)
{
this.Id = id;
}
public int Id { get; private set; }
}
}

Most likely the Timeout is not triggering because you have it before the Where filter. This means that all events are flowing through and resetting the timer, and then most of the events get filtered by the Where clause. To your subscribing observer, it will seem like it never gets a result and the timeout never triggers. Move the Timeout to be after the Where and you should now have a system that times out individual observers if they do not get their expected event on time.