I have the following tasks, they share the sum variable and at the end the sum should be 9, but I get 3. Can you please help me how to fix it. Many thanks.
int sum = 0;
Task t1 = Task.Factory.StartNew(() =>
{
sum = sum + Computation();
});
Task t2 = Task.Factory.StartNew(() =>
{
sum = sum + Computation();
});
Task t3 = Task.Factory.StartNew(() =>
{
sum = sum + Computation();
});
Task.WaitAll(t1, t2, t3);
Console.WriteLine($"The sum is {sum}");
private static int Computation()
{
return 3;
}
It's because you're writing the same field from multiple threads at the same time.
Use Interlocked.Add from System.Threading, which will prevent each thread from writing the variable at the same exact moment.
int sum = 0;
Task t1 = Task.Factory.StartNew(() =>
{
Interlocked.Add(ref sum,Computation());
});
Task t2 = Task.Factory.StartNew(() =>
{
Interlocked.Add(ref sum,Computation());
});
Task t3 = Task.Factory.StartNew(() =>
{
Interlocked.Add(ref sum,Computation());
});
Task.WaitAll(t1, t2, t3);
Console.WriteLine($"The sum is {sum}");
You never tell your code to wait until task 't1' is finished until you start 't2', etc, so everything executes in parallel. Each thread reads the value in "sum" (initially 0) and adds 3. So 0+3 = 3. After that it then writes back the 3. So the code does exactly you programmed it to do.
Galister explained how you could add locks (one side note on this comments: operations in a computer almost never happen at exactly the same moment ;) )
Interlocked class are great when atomic operations are required but if you care about performance consider combine it with Thread Local Storage (TLS).
The Parallel.For has a unique overloads for them, documantation.
Example:
int sum = 0;
Parallel.For(1, 3,
() => 0, //The type of the thread-local data.
(x, state, tls) => // The delegate that is invoked once per iteration.
{
tls += x;
return tls;
},
partial => //The delegate that performs a final action on the local state of each task.
{
Interlocked.Add(ref sum, partial);
});
Do consider that for a small loops, it does not matter, and there will be no actual difference between using the Thread Local Storage and `Interlocked. For big loops, it will make a difference, using lock in big loops can cause serious overhead (blog):
This will potentially add a huge amount of overhead to our
calculation. Since we can potentially block while waiting on the lock
for every single iteration, we will most likely slow this down to
where it is actually quite a bit slower than our serial
implementation. The problem is the lock statement – any time you use
lock(object), you’re almost assuring reduced performance in a parallel
situation. When parallelizing a routine, try to avoid locks.
The idea is to reduce the acquire a lock on the sum variable, note that every task is trying to acquire a lock it in every single point of time. Using the Thread Local Storage making the sum variable to be locked as much only as the number of threads.
instead of synchronizing once per element (potentially millions of
times), you’ll only have to synchronize once per thread
Related
This is my first attempt at parallel programming.
I'm writing a test console app before using this in my real app and I can't seem to get it right. When I run this, the parallel search is always faster than the sequential one, but the parallel search never finds the correct value. What am I doing wrong?
I tried it without using a partitioner (just Parallel.For); it was slower than the sequential loop and gave the wrong number. I saw a Microsoft doc that said for simple computations, using Partitioner.Create can speed things up. So I tried that but still got the wrong values. Then I saw Interlocked, but I think I'm using it wrong.
Any help would be greatly appreciated
Random r = new Random();
Stopwatch timer = new Stopwatch();
do {
// Make and populate a list
List<short> test = new List<short>();
for (int x = 0; x <= 10000000; x++)
{
test.Add((short)(r.Next(short.MaxValue) * r.NextDouble()));
}
// Initialize result variables
short rMin = short.MaxValue;
short rMax = 0;
// Do min/max normal search
timer.Start();
foreach (var amp in test)
{
rMin = Math.Min(rMin, amp);
rMax = Math.Max(rMax, amp);
}
timer.Stop();
// Display results
Console.WriteLine($"rMin: {rMin} rMax: {rMax} Time: {timer.ElapsedMilliseconds}");
// Initialize parallel result variables
short pMin = short.MaxValue;
short pMax = 0;
// Create list partioner
var rangePortioner = Partitioner.Create(0, test.Count);
// Do min/max parallel search
timer.Restart();
Parallel.ForEach(rangePortioner, (range, loop) =>
{
short min = short.MaxValue;
short max = 0;
for (int i = range.Item1; i < range.Item2; i++)
{
min = Math.Min(min, test[i]);
max = Math.Max(max, test[i]);
}
_ = Interlocked.Exchange(ref Unsafe.As<short, int>(ref pMin), Math.Min(pMin, min));
_ = Interlocked.Exchange(ref Unsafe.As<short, int>(ref pMax), Math.Max(pMax, max));
});
timer.Stop();
// Display results
Console.WriteLine($"pMin: {pMin} pMax: {pMax} Time: {timer.ElapsedMilliseconds}");
Console.WriteLine("Press enter to run again; any other key to quit");
} while (Console.ReadKey().Key == ConsoleKey.Enter);
Sample output:
rMin: 0 rMax: 32746 Time: 106
pMin: 0 pMax: 32679 Time: 66
Press enter to run again; any other key to quit
The correct way to do a parallel search like this is to compute local values for each thread used, and then merge the values at the end. This ensures that synchronization is only needed at the final phase:
var items = Enumerable.Range(0, 10000).ToList();
int globalMin = int.MaxValue;
int globalMax = int.MinValue;
Parallel.ForEach<int, (int Min, int Max)>(
items,
() => (int.MaxValue, int.MinValue), // Create new min/max values for each thread used
(item, state, localMinMax) =>
{
var localMin = Math.Min(item, localMinMax.Min);
var localMax = Math.Max(item, localMinMax.Max);
return (localMin, localMax); // return the new min/max values for this thread
},
localMinMax => // called one last time for each thread used
{
lock(items) // Since this may run concurrently, synchronization is needed
{
globalMin = Math.Min(globalMin, localMinMax.Min);
globalMax = Math.Max(globalMax, localMinMax.Max);
}
});
As you can see this is quite a bit more complex than a regular loop, and this is not even doing anything fancy like partitioning. An optimized solution would work over larger blocks to reduce overhead, but this is omitted for simplicity, and it looks like the OP is aware such issues already.
Be aware that multi threaded programming is difficult. While it is a great idea to try out such techniques in a playground rather than a real program, I would still suggest that you should start by studying the potential dangers of thread safety, there is fairly easy to find good resources about this.
Not all problems will be as obviously wrong like this, and it is quite easy to cause issues that breaks once in a million, or only when the cpu load is high, or only on single CPU systems, or issues that are only detected long after the code is put into production. It is a good practice to be paranoid whenever multiple threads may read and write the same memory concurrently.
I would also recommend learning about immutable data types, and pure functions, since these are much safer and easier to reason about once multiple threads are involved.
Interlocked.Exchange is thread safe only for Exchange, every Math.Min and Math.Max can be with race condition. You should compute min/max for every batch separately and then join results.
Using low-lock techniques like the Interlocked class is tricky and advanced. Taking into consideration that your experience in multithreading is not excessive, I would say go with a simple and trusty lock:
object locker = new object();
//...
lock (locker)
{
pMin = Math.Min(pMin, min);
pMax = Math.Max(pMax, max);
}
I am trying to find out why parallel foreach does not give the expected speedup on a machine with 32 physical cores and 64 logical cores with a simple test computation.
...
var parameters = new List<string>();
for (int i = 1; i <= 9; i++) {
parameters.Add(i.ToString());
if (Scenario.UsesParallelForEach)
{
Parallel.ForEach(parameters, parameter => {
FireOnParameterComputed(this, parameter, Thread.CurrentThread.ManagedThreadId, "started");
var lc = new LongComputation();
lc.Compute();
FireOnParameterComputed(this, parameter, Thread.CurrentThread.ManagedThreadId, "stopped");
});
}
else
{
foreach (var parameter in parameters)
{
FireOnParameterComputed(this, parameter, Thread.CurrentThread.ManagedThreadId, "started");
var lc = new LongComputation();
lc.Compute();
FireOnParameterComputed(this, parameter, Thread.CurrentThread.ManagedThreadId, "stopped");
}
}
}
...
class LongComputation
{
public void Compute()
{
var s = "";
for (int i = 0; i <= 40000; i++)
{
s = s + i.ToString() + "\n";
}
}
}
The Compute function takes about 5 seconds to complete. My assumption was, that with the parallel foreach loop each additional iteration creates a parallel thread running on one of the cores and taking as much as it would take to compute the Compute function only once. So, if I run the loop twice, then with the sequential foreach, it would take 10 seconds, with the parallel foreach only 5 seconds (assuming 2 cores are available). The speedup would be 2. If I run the loop three times, then with the sequential foreach, it would take 15 seconds, but again with the parallel foreach only 5 seconds. The speedup would be 3, then 4, 5, 6, 7, 8, and 9. However, what I observe is a constant speedup of 1.3.
Sequential vs parallel foreach. X-axis: number of sequential/parallel execution of the computation. Y-axis: time in seconds
Speedup, time of the sequential foreach divided by parallel foreach
The event fired in FireOnParameterComputed is intended to be used in a GUI progress bar to show the progress. In the progress bar it can be clearly see, that for each iteration, a new thread is created.
My question is, why don't I see the expected speedup or at least close to the expected speedup?
Tasks aren't threads.
Sometimes starting a task will cause a thread to be created, but not always. Creating and managing threads consumes time and system resources. When a task only takes a short amount of time, even though it's counter-intuitive, the single-threaded model is often faster.
The CLR knows this and tries to make its best judgment on how to execute the task based on a number of factors including any hints that you've passed to it.
For Parallel.ForEach, if you're certain that you want multiple threads to be spawned, try passing in ParallelOptions.
Parallel.ForEach(parameters, new ParallelOptions { MaxDegreeOfParallelism = 100 }, parameter => {});
MSDN
My question is: The third parameter in the parallel.for, what does it do?
When I change it to ()=> 1d, it doubles my result, set to two it triples, but it ignores the decimals.
Why does it ignore the decimals, if it was some sort of doubling? What is really happening there?
I've now tried adding locks. And it does not just initialize the interimresult to the value specified.
Here is the code Im using:
static void RunParallelForCorrectedAdam()
{
object _lock = new object();
double result = 0d;
// Here we call same method several times.
// for (int i = 0; i < 32; i++)
Parallel.For(0, 32,
// Func<TLocal> localInit,
() => 3d,
// Func<int, ParallelLoopState, TLocal, TLocal> body,
(i, state, interimResult) =>
{
lock (_lock)
{
return interimResult + 1;
}
},
//Final step after the calculations
//we add the result to the final result
// Action<TLocal> localFinally
(lastInterimResult) =>
{
lock (_lock)
{
result += lastInterimResult;
}
}
);
// Print the result
Console.WriteLine("The result is {0}", result);
}
With () => 3d, result will be 32 + 3 * t, where t is the number of threads that were used. 3d is passed as interimResult to the first call to body within each thread.
The whole purpose of Parallel.For is to distribute the work on several threads. So interimResult + 1 is executed exactly 32 times (possibly on different threads). But each thread has to have some initial value for interimResult. That's the value that is returned by localInit.
So if the work is distributed on e.g. two threds, each one does + 1 16 times and thus calculates 3 + 16. At the end, the partial results are summed yielding 6 + 32.
In short, in this example, it doesn't make much sense for localInit to return somthing different than 0d.
My question is: The third parameter in the parallel.for, what does it do?
It's a Func that gets executed once per thread. If your loop requires thread-local variable, this is where you initialize it.
EDIT:
Step by step:
(i, state, interimResult) => interimResult + 1,
Do you understand that interimResult is your local variable, the same one you initialized as 0d?
In My Parallel.ForEach Loop the localFinally delegate does get called on all the threads.
I have found this to happen as my Parallel Loop stalls.
In my Parallel Loop I have about three condition check stages that return before completion of the Loop. And it seems that it is when the Threads are returned from these stages and not the execution of the entire body that it does not execute the localFinally delegate.
The Loop structure is as follows:
var startingThread = Thread.CurrentThread;
Parallel.ForEach(fullList, opt,
()=> new MultipleValues(),
(item, loopState, index, loop) =>
{
if (cond 1)
return loop;
if (cond 2)
{
process(item);
return loop;
}
if (cond 3)
return loop;
Do Work(item);
return loop;
},
partial =>
{
Log State of startingThread and threads
} );
I have run the loop on a small data set and logged in detail and found that while the Parallel.ForEach completes all the iterations and the Log at the last thread of localFinally is --
Calling Thread State is WaitSleepJoin for Thread 6 Loop Indx 16
the Loop still does not complete gracefully and remains stalled... any clues why the stalls ?
Cheers!
Just did a quick test run after seeing the definition of localFinally (executed after each thread finished), which had me suspecting that that could mean there would be far less threads created by parallelism than loops executed. e.g.
var test = new List<List<string>> ();
for (int i = 0; i < 1000; i++)
{
test.Add(null);
}
int finalcount = 0;
int itemcount = 0;
int loopcount = 0;
Parallel.ForEach(test, () => new List<string>(),
(item, loopState, index, loop) =>
{
Interlocked.Increment(ref loopcount);
loop.Add("a");
//Thread.Sleep(100);
return loop;
},
l =>
{
Interlocked.Add(ref itemcount, l.Count);
Interlocked.Increment(ref finalcount);
});
at the end of this loop, itemcount and loopcount were 1000 as expected, and (on my machine) finalcount 1 or 2 depending on the speed of execution. In the situation with the conditions: when returned directly the execution is probably much faster and no extra threads are needed. only when the dowork is executed more threads are needed. However the parameter (l in my case) contains the combined list of all executions.
Could this be the cause of the logging difference?
I think you just misunderstood what localFinally means. It's not called for each item, it's called for each thread that is used by Parallel.ForEach(). And many items can share the same thread.
The reason why it exists is that you can perform some aggregation independently on each thread, and join them together only in the end. This way, you have to deal with synchronization (and have it impact your performance) only in a very small piece of code.
For example, if you want to compute the sum of score for a collection of items, you could do it like this:
int totalSum = 0;
Parallel.ForEach(
collection, item => Interlocked.Add(ref totalSum, ComputeScore(item)));
But here, you call Interlocked.Add() for every item, which can be slow. Using localInit and localFinally, you can rewrite the code like this:
int totalSum = 0;
Parallel.ForEach(
collection,
() => 0,
(item, state, localSum) => localSum + ComputeScore(item),
localSum => Interlocked.Add(ref totalSum, localSum));
Notice that the code uses Interlocked.Add() only in the localFinally and does access the global state in body. This way, the cost of synchronization is paid only a few times, once for each thread used.
Note: I used Interlocked in this example, because it is very simple and quite obviously correct. If the code was more complicated, I would use lock first, and try to use Interlocked only when it was necessary for good performance.
My goal is the following:
There is a certain range of integers, and I have to test every integer in that range for something random. I want to use multiple threads for this, and divide the work equally among the threads using a shared counter. I set the counter at the beginning value, and let every thread take a number, increase it, do some calculations, and return a result. This shared counter has to be incremented with locks, because otherwise there will be gaps / overlaps in the range of integers to test.
I have no idea where to start. Let's say I want 12 threads to do the work, I do:
for (int t = 0; t < threads; t++)
{
Thread thr = new Thread(new ThreadStart(startThread));
}
startThread() isthe method I use for the calculations.
Can you help me on my way? I know I have to use the Interlocked class, but that's all….
Say you have an int field somewhere (initialized to -1 initially) then:
int newVal = Interlocked.Increment(ref theField);
is a thread-safe increment; assuming you don't mind the (very small) risk of overflowing the upper int limit, then:
int next;
while((next = Interlocked.Increment(ref theField)) <= upperInclusive) {
// do item with index "next"
}
However, Parallel.For will do all of this a lot more conveniently:
Parallel.For(lowerInclusive, upperExclusive, i => DoWork(i));
or (to constrain to 12 threads):
var options = new ParallelOptions { MaxDegreeOfParallelism = 12 };
Parallel.For(lowerInclusive, upperExclusive, options, i => DoWork(i));