I'm looking for some advice on writing unit tests for multi-threading in C#. Specifically, I want to check that an object is being locked correctly. However, in order to test this I need to assert against that object, which may have changed before the assert(s) are implemented (with the lock being released, another thread may change the object).
Using AutoResetEvent I have been able to control the flow in the unit test side, allowing me to effectively emulate the lock in the tested object. The issue with this is that I no longer need the lock for the test to pass.
What I'd like is to have a test that passes with the lock in and fails with it out.
Obviously, this is a simplified example. It's also .Net 4, so there is no async and await option (although if that would help, changing could be an option).
Suggestions welcome. Thanks.
Below is example code:
public class BasicClass
{
public int Val
{
get { lock (lockingObject) { return val; } }
private set { lock (lockingObject) { val = value; } }
}
private int val;
public BasicClass(int val = -1)
{
Val = val;
}
public void SetValue(int val)
{
Val = val;
}
private object lockingObject = new object();
}
This is the (NUnit) unit test:
[Test]
public void BasicClassTest()
{
for (int repeat = 0; repeat < 1000; repeat++) // Purely for dev testing and can get away with as no SetUp/TearDown
{
BasicClass b = new BasicClass();
int taskCount = 10;
Task[] tasks = new Task[taskCount];
var taskControl = new AutoResetEvent(false);
var resultControl = new AutoResetEvent(false);
int expected = -1;
for (int i = 0; i < taskCount; i++)
{
int temp = i;
tasks[temp] = new Task(() =>
{
taskControl.WaitOne(); // Hold there here until set
b.SetValue(temp);
expected = temp;
resultControl.Set(); // Allows asserts to be processed.
});
}
// Start each task
foreach (var t in tasks)
t.Start();
// Assert results as tasks finish.
for (int i = 0; i < taskCount; i++)
{
taskControl.Set(); // Unblock, allow one thread to proceed.
resultControl.WaitOne(); // Wait for a task to set a expected value
Assert.That(b.Val, Is.EqualTo(expected));
Console.WriteLine("b.Val = {0}, expected = {1}", b.Val, expected); // Output values to ensure they are changing
}
// Wait for all tasks to finish, but not forever.
Task.WaitAll(tasks, 1000);
}
}
As for other system functions like DateTime.Now, I prefer to abstract threading functions like sleep, mutex, signals and so on (yes, I know there are libraries for DateTime.Now and other system functions, but I think to abstract it is a better way).
So you end up with a kind of IThreadind interface with methods to Sleep and so on. The disadvantage is, that you can't use the handy lock statement in this case. You could have a method Lock(object) that returns you an IDisposable that you can use with the "using" statement, to get nearly the same comfort.
using(threading.Lock(lockObject))
{
...
}
Now you can Create a real implementation with the real functions and a Mock for your unit tests which is injected. So you could for example for your tests shortcut any sleep call to e few ms in order to speed up your tests. And you can verify that all functions where called that you expected.
Sounds like a lot of work? Think over, how many time you will spend to debug some nasty threading issue which from time to time crashes your production system with your customer running amok.
Related
I'm trying to unit test the cancel execution scenario in a class conceptually similar to the following:
public class ContextExecutor
{
public ContextExecutor(IContextRunner runner, IExecutionCanceler canceler)
{
this.runner = runner;
this.canceler = canceler;
}
public void Execute(IEnumerable<IContext> contexts)
{
foreach (var ctx in contexts)
{
if (canceler.IsCanceled)
{
break;
}
runner.Run(ctx);
}
}
readonly IContextRunner runner;
readonly IExecutionCanceler canceler;
}
public interface IContextRunner
{
void Run(IContext context);
}
public interface IExecutionCanceler
{
bool IsCanceled { get; }
}
The test case I was after should go through the following steps:
start ContextExecutor.Execute() asynchronously somehow;
put that method execution on hold until something unlocks it from unit test code;
unlock execution and let it perform 1 (..or 2, or..) loop runs, anyway less than full enumerable length;
invoke canceling by setting canceler.IsCanceled = true;
unlock loop execution free;
wait synchronously for method completion;
assert that loop has been invoked the expected nr of times.
I got tangled up with controlling loop execution locking/unlocking from unit test code. Apart from starting Execute() in a new thread, I avoided using threading synchronization primitives (e.g. semaphores and locks). I also had to discard a Task-based approach, as I could not change signatures to apply async/await constructs. I tried to play with the following, but with no luck:
inject a yield-powered function as IEnumerable<IContext> input parameter to hold loop on foreach() line, to release loop everytime another yield is hit, and try to control that from unit test code.
inject a IContextRunner runner powered by a Reactive Extension Subject to hold loop on runner.Run line, to release loop everytime another Subject.OnNext is hit, and try to control that from unit test code.
For that matters, unit testing framework is NUnit, while NSubstitute is the mocking framework and FluentAssertion is the assertion library of choice. I know how to arrange/act/assert with those.
What is so evident that I missing? Thanks
EDIT
To provide an example of what has been tried, this is a Task-based approach made after posting question and reading #Peter Duniho helpful comment:
// in unit test class
ContextExecutor executor;
IContextRunner runner;
IExecutionCanceler canceler;
IRunnableContext[] runnableContexts;
int totalNrOfContexts;
int nrOfContextToRun; // this will be < totalNrOfContexts
int actualNrOfContextRan;
[SetUp]
public virtual void before_each()
{
// create instance under test, mock dependencies, dummy input data
Initialize();
RunScenarioAsync().Wait();
}
async Task RunScenarioAsync()
{
// prepare mock IContextRunner so that for each input context:
// * there's a related TaskCompletionSource<Object>
// * Run() increments actualNrOfContextRan
// * Run() performs taskSource.Task.Wait();
List<TaskCompletionSource<Object>> runTaskSources = PrepareMockContextRunner();
canceler.IsCanceled.Returns(false); // let execution go initially
// queue up method under test to be processed asynchronously
var executeTask = Task.Run(() =>
{
executor.Execute(runnableContexts);
};
// "unlock" some IContextRunner.Run() invocations,
// for when they will be invoked
for (int i = 0; i < nrOfContextToRun; i++)
{
runTaskSources[i].SetResult(null);
await Task.Delay(0); // tried also with Delay(1) and without this line at all
}
// flag to cancel execution
canceler.IsCanceled.Returns(true);
// unlock all remaining IContextRunner.Run() invocations,
// again for when/if they will be invoked
for (int i = nrOfContextToRun; i < totalNrOfContexts; i++)
{
runTaskSources[i].SetResult(null);
await Task.Delay(0);
}
// wait until method under test completes
await executeTask;
}
[Test]
public void it_should_only_run_until_cancel()
{
int expected = nrOfContextToRun;
int actual = actualNrOfContextRan;
actual.Should().Be(expected);
}
The problem I have here (and similar to other approaches tried) is about giving and regain control to/from the method under test in a predictable way (that is, synchronizing).
Here, if there's no await Task.Delay() or if delay is 0ms, only 1 context is actually ran: the method under test has no chance to run the 2nd and 3rd one, it finds the canceling flag too soon. If delay is 1ms, method executes more context than expected before actually detecting the flag. Also tried with ticks instead of ms, but in my experience playing with delays usually means you're doing something wrong.
I am writing a multithreaded application it is windows service. I have 20 folders. I create 15 threads onstart method. I want to achieve that; 15 threads go to folders 1,2,3,...,15 sequentially. When one thread finished, it creates another thread. This created thread must go 16.th folder. It must not go to working folders. How can I do this? That is, how can I be sure that two threads do not go the same folder?
Could you not just have a static variable that would be a counter for the folder name?
Something like:
private static int _folderNameCounter = 0;
private static readonly object _padlock = new object();
public static int GetFolderCounter()
{
lock(_padlock)
{
_folderNameCounter++;
return _folderNameCounter;
}
}
public static void Main()
{
for(int i = 0; i < 20; i++)
{
Task.Factory.StartNew(() =>
{
var path = #"c:\temp\" + GetFolderCounter();
Directory.CreateDirectory(path);
// add your own code for the thread here
});
}
}
Note: I've used the TPL instead of using Threads directly since I think that the TPL is a better solution. You can of course have specific requirements which can mean that Threads is the better solution for
your case.
Use a BlockingCollection<T> and fill the collection with the folder numbers. Each task handles an item of the collection, and the collection itself handles the multi-threading aspect so that each item is only handled by one consumer.
// Define the blocking collection with a maximum size of 15.
const int maxSize = 15;
var data = new BlockingCollection<int>(maxSize);
// Add the data to the collection.
// Do this in a separate task since BlockingCollection<T>.Add()
// blocks when the specified capacity is reached.
var addingTask = new Task(() => {
for (int i = 1; i <= 20; i++) {
data.Add(i);
}
).Start();
// Define a signal-to-stop bool
var stop = false;
// Create 15 handle tasks.
// You can change this to threads if necessary, but the general idea is that
// each consumer continues to consume until the stop-boolean is set.
// The Take method returns only when an item is/becomes available.
for (int t = 0; t < maxSize; t++) {
new Task(() => {
while (!stop) {
int item = data.Take();
// Note: the Take method will block until an item comes available.
HandleThisItem(item);
}
}).Start();
};
// Wait until you need to stop. When you do, set stop true
stop = true;
I have an event source which fired by a Network I/O very frequently, based on underlying design, of course the event was always on different thread each time, now I wrapped this event via Rx with: Observable.FromEventPattern(...), now I'm using the TakeWhile(predict) to filter some special event data.
At now, I have some concerns on its thread safety, the TakeWhile(predict) works as a hit and mute, but in concurrent situation, can it still be guaranteed? because I guess the underlying implementation could be(I can't read the source code since it's too complicated...):
public static IObservable<TSource> TakeWhile<TSource>(this IObservable<TSource> source, Func<TSource, bool> predict)
{
ISubject<TSource> takeUntilObservable = new TempObservable<TSource>();
IDisposable dps = null;
// 0 for takeUntilObservable still active, 1 for predict failed, diposed and OnCompleted already send.
int state = 0;
dps = source.Subscribe(
(s) =>
{
/* NOTE here the 'hit and mute' still not thread safe, one thread may enter 'else' and under CompareExchange, but meantime another thread may passed the predict(...) and calling OnNext(...)
* so the CompareExchange here mainly for avoid multiple time call OnCompleted() and Dispose();
*/
if (predict(s) && state == 0)
{
takeUntilObservable.OnNext(s);
}
else
{
// !=0 means already disposed and OnCompleted send, avoid multiple times called via parallel threads.
if (0 == Interlocked.CompareExchange(ref state, 1, 0))
{
try
{
takeUntilObservable.OnCompleted();
}
finally
{
dps.Dispose();
}
}
}
},
() =>
{
try
{
takeUntilObservable.OnCompleted();
}
finally { dps.Dispose(); }
},
(ex) => { takeUntilObservable.OnError(ex); });
return takeUntilObservable;
}
That TempObservable is just a simple implementation of ISubject.
If my guess reasonable, then seems the thread safety can't be guaranteed, means some unexpected event data may still incoming to OnNext(...) because that 'mute' is still on going.
Then I write a simple testing to verify, but out of expectation, the results are all positive:
public class MultipleTheadEventSource
{
public event EventHandler OnSthNew;
int cocurrentCount = 1000;
public void Start()
{
for (int i = 0; i < this.cocurrentCount; i++)
{
int j = i;
ThreadPool.QueueUserWorkItem((state) =>
{
var safe = this.OnSthNew;
if (safe != null)
safe(j, null);
});
}
}
}
[TestMethod()]
public void MultipleTheadEventSourceTest()
{
int loopTimes = 10;
int onCompletedCalledTimes = 0;
for (int i = 0; i < loopTimes; i++)
{
MultipleTheadEventSource eventSim = new MultipleTheadEventSource();
var host = Observable.FromEventPattern(eventSim, "OnSthNew");
host.TakeWhile(p => { return int.Parse(p.Sender.ToString()) < 110; }).Subscribe((nxt) =>
{
//try print the unexpected values, BUT I Never saw it happened!!!
if (int.Parse(nxt.Sender.ToString()) >= 110)
{
this.testContextInstance.WriteLine(nxt.Sender.ToString());
}
}, () => { Interlocked.Increment(ref onCompletedCalledTimes); });
eventSim.Start();
}
// simply wait everything done.
Thread.Sleep(60000);
this.testContextInstance.WriteLine("onCompletedCalledTimes: " + onCompletedCalledTimes);
}
before I do the testing, some friends here suggest me try to use Synchronize<TSource> or ObserveOn to make it thread safe, so any idea on my proceeding thoughts and why the issue not reproduced?
As per your other question, the answer still remains the same: In Rx you should assume that Observers are called in a serialized fashion.
To provider a better answer; Originally the Rx team ensured that the Observable sequences were thread safe, however the performance penalty for well behaved/designed applications was unnecessary. So a decision was taken to remove the thread safety to remove the performance cost. To allow you to opt back into to thread safety you could apply the Synchronize() method which would serialize all method calls OnNext/OnError/OnCompleted. This doesn't mean they will get called on the same thread, but you wont get your OnNext method called while another one is being processed.
The bad news, from memory this happened in Rx 2.0, and you are specifically asking about Rx 1.0. (I am not sure Synchonize() even exists in 1.xx?)
So if you are in Rx v1, then you have this blurry certainty of what is thread safe and what isn't. I am pretty sure the Subjects are safe, but I can't be sure about the factory methods like FromEventPattern.
My recommendation is: if you need to ensure thread safety, Serialize your data pipeline. The easiest way to do this is to use a single threaded IScheduler implementation i.e. DispatcherScheduler or a EventLoopScheduler instance.
Some good news is that when I wrote the book on Rx it did target v1, so this section is very relevant for you http://introtorx.com/Content/v1.0.10621.0/15_SchedulingAndThreading.html
So if your query right now looked like this:
Observable.FromEventPatter(....)
.TakeWhile(x=>x>5)
.Subscribe(....);
To ensure that the pipeline is serialized you can create an EventLoopScheduler (at the cost of dedicating a thread to this):
var scheduler = new EventLoopScheduler();
Observable.FromEventPatter(....)
.ObserveOn(scheduler)
.TakeWhile(x=>x>5)
.Subscribe(....);
Here the scenario :
A method is called each minute by a timer. This method could be call through UI (a button). I want that if my method is "in process", and is called, it does not execute the method twice.
In my method I use a simple boolean :
private bool _isProcessing;
public void JustDoIt(Action a, int interval, int times)
{
if (!_isProcessing)
{
_isProcessing = true;
for (int i = 0; i < times; i++)
{
a();
Thread.Sleep(interval);
}
}
_isProcessing = false;
}
It works fine. I test this functionality with this test :
[Test]
public void Should_Output_A_String_Only_3_Times()
{
var consoleMock = new Mock<IConsole>();
IConsole console = consoleMock.Object;
var doer = new Doer { Console = console };
Action a = new Action(() => console.Writeline("TASK DONE !"));
// Simulate a call by Timer
var taskA = Task.Factory.StartNew(() => doer.JustDoIt(a, 1000, 3));
// Simulate a call by UI
var taskB = Task.Factory.StartNew(() => doer.JustDoIt(a));
taskA.Wait();
consoleMock.Verify(c => c.Writeline("TASK DONE !"), Times.Exactly(3));
}
A developer reviews my code and says : "I replaced your boolean by a lock keyword. It's more Thread Safe. Frankly I'm not masterize multithreading so I answered him "OK Guy !"
few days later (today to be more precise), I want to test what if the difference between using lock or a simple boolean. So I was surprised to constate when I replace a boolean by the lock keyword like this :
private object _locker = new Object();
public void JustDoIt(Action a, int interval, int times)
{
lock (_locker)
{
//_isProcessing = true;
for (int i = 0; i < times; i++)
{
a();
Thread.Sleep(interval);
}
}
//_isProcessing = false;
}
The precedent test don't pass :
Message : Moq.MockException : Expected invocation on the mock exactly 3 times, but was 4 times: c=>c.Writeline("TASK DONE !")
So, do I use the lock keyword badly ? Should it be 'Static' ?
Thank you
Make _isProcessing volatile. And then do this:
public void JustDoIt(Action a, int interval, int times)
{
if (_isProcessing) return
_isProcessing = true;
for (int i = 0; i < times; i++)
{
a();
Thread.Sleep(interval);
}
_isProcessing = false;
}
This has a minor race condition, but since your code isn't synchronized to anything anyway, I don't believe it can possibly matter.
You just lock it, that means others thread which would like to enter the critical section wait for the lock and they will enter the lock if the current thread/task releases it.
Eg.: TaskA aquires the lock, it is now in the Critical Section and executes the method a() 3 times. When TaskA has finished the executions, it releases the lock and maybe there is context switch, so TaskB runs the method a() (the 4th time).
After TaskB returns the main-threads says.. "hey, TaskA has finished, so i verify my results"
Addiontional to that, i don't know if TaskA has to run before TaskB. So, i don't know if the Task-Scheduler is FIFO.
I am trying to bypass the the wait64 handle limit that .net 3.5 imposes
I have seen this thread : Workaround for the WaitHandle.WaitAll 64 handle limit?
So I understand the general idea but I am having difficulty because I am not using a delegate but rather
I am basically working of this example :
http://msdn.microsoft.com/en-us/library/3dasc8as%28VS.80%29.aspx
This link http://www.switchonthecode.com/tutorials/csharp-tutorial-using-the-threadpool
is similar but again the int variable keeping track of the tasks is a member variable.
Where in the above example would I pass the threadCount integer?
Do I pass it in the callback method as an object? I think I am having trouble with the callback method and passing by reference.
Thanks Stephen,
That link is not entirely clear to me.
Let me post my code to help myself clarify:
for (int flows = 0; flows < NumFlows; flows++)
{
ResetEvents[flows] = new ManualResetEvent(false);
ICalculator calculator = new NewtonRaphson(Perturbations);
Calculators[flows] = calculator;
ThreadPool.QueueUserWorkItem(calculator.ThreadPoolCallback, flows);
}
resetEvent.WaitOne();
Where would I pass in my threadCount variable. I assume it needs to be decremented in calculator.ThreadPoolCallback?
You should not be using multiple wait handles to wait for the completion of multiple work items in the ThreadPool. Not only is it not scalable you will eventually bump into the 64 handle limit imposed by the WaitHandle.WaitAll method (as you have done already). The correct pattern to use in this situation is a counting wait handle. There is one available in the Reactive Extensions download for .NET 3.5 via the CountdownEvent class.
var finished = new CountdownEvent(1);
for (int flows = 0; flows < NumFlows; flows++)
{
finished.AddCount();
ICalculator calculator = new NewtonRaphson(Perturbations);
Calculators[flows] = calculator;
ThreadPool.QueueUserWorkItem(
(state) =>
{
try
{
calculator.ThreadPoolCallback(state);
}
finally
{
finished.Signal();
}
}, flows);
}
finished.Signal();
finished.Wait();
An anonymous method might be easiest:
int threadCount = 0;
for (int flows = 0; flows < NumFlows; flows++)
{
ICalculator calculator = new NewtonRaphson(Perturbations);
Calculators[flows] = calculator;
// We're about to queue a new piece of work:
// make a note of the fact a new work item is starting
Interlocked.Increment(ref threadCount);
ThreadPool.QueueUserWorkItem(
delegate
{
calculator.ThreadPoolCallback(flows);
// We've finished this piece of work...
if (Interlocked.Decrement(ref threadCount) == 0)
{
// ...and we're the last one.
// Signal back to the main thread.
resetEvent.Set();
}
}, null);
}
resetEvent.WaitOne();