Related
I am trying to find out why parallel foreach does not give the expected speedup on a machine with 32 physical cores and 64 logical cores with a simple test computation.
...
var parameters = new List<string>();
for (int i = 1; i <= 9; i++) {
parameters.Add(i.ToString());
if (Scenario.UsesParallelForEach)
{
Parallel.ForEach(parameters, parameter => {
FireOnParameterComputed(this, parameter, Thread.CurrentThread.ManagedThreadId, "started");
var lc = new LongComputation();
lc.Compute();
FireOnParameterComputed(this, parameter, Thread.CurrentThread.ManagedThreadId, "stopped");
});
}
else
{
foreach (var parameter in parameters)
{
FireOnParameterComputed(this, parameter, Thread.CurrentThread.ManagedThreadId, "started");
var lc = new LongComputation();
lc.Compute();
FireOnParameterComputed(this, parameter, Thread.CurrentThread.ManagedThreadId, "stopped");
}
}
}
...
class LongComputation
{
public void Compute()
{
var s = "";
for (int i = 0; i <= 40000; i++)
{
s = s + i.ToString() + "\n";
}
}
}
The Compute function takes about 5 seconds to complete. My assumption was, that with the parallel foreach loop each additional iteration creates a parallel thread running on one of the cores and taking as much as it would take to compute the Compute function only once. So, if I run the loop twice, then with the sequential foreach, it would take 10 seconds, with the parallel foreach only 5 seconds (assuming 2 cores are available). The speedup would be 2. If I run the loop three times, then with the sequential foreach, it would take 15 seconds, but again with the parallel foreach only 5 seconds. The speedup would be 3, then 4, 5, 6, 7, 8, and 9. However, what I observe is a constant speedup of 1.3.
Sequential vs parallel foreach. X-axis: number of sequential/parallel execution of the computation. Y-axis: time in seconds
Speedup, time of the sequential foreach divided by parallel foreach
The event fired in FireOnParameterComputed is intended to be used in a GUI progress bar to show the progress. In the progress bar it can be clearly see, that for each iteration, a new thread is created.
My question is, why don't I see the expected speedup or at least close to the expected speedup?
Tasks aren't threads.
Sometimes starting a task will cause a thread to be created, but not always. Creating and managing threads consumes time and system resources. When a task only takes a short amount of time, even though it's counter-intuitive, the single-threaded model is often faster.
The CLR knows this and tries to make its best judgment on how to execute the task based on a number of factors including any hints that you've passed to it.
For Parallel.ForEach, if you're certain that you want multiple threads to be spawned, try passing in ParallelOptions.
Parallel.ForEach(parameters, new ParallelOptions { MaxDegreeOfParallelism = 100 }, parameter => {});
The title may seem a little odd, because I have no idea how to describe this in one sentence.
For the course Algorithms we have to micro-optimize some stuff, one is finding out how deleting from an array works. The assignment is delete something from an array and re-align the contents so that there are no gaps, I think it is quite similar to how std::vector::erase works from c++.
Because I like the idea of understanding everything low-level, I went a little further and tried to bench my solutions. This presented some weird results.
At first, here is a little code that I used:
class Test {
Stopwatch sw;
Obj[] objs;
public Test() {
this.sw = new Stopwatch();
this.objs = new Obj[1000000];
// Fill objs
for (int i = 0; i < objs.Length; i++) {
objs[i] = new Obj(i);
}
}
public void test() {
// Time deletion
sw.Restart();
deleteValue(400000, objs);
sw.Stop();
// Show timings
Console.WriteLine(sw.Elapsed);
}
// Delete function
// value is the to-search-for item in the list of objects
private static void deleteValue(int value, Obj[] list) {
for (int i = 0; i < list.Length; i++) {
if (list[i].Value == value) {
for (int j = i; j < list.Length - 1; j++) {
list[j] = list[j + 1];
//if (list[j + 1] == null) {
// break;
//}
}
list[list.Length - 1] = null;
break;
}
}
}
}
I would just create this class and call the test() method. I did this in a loop for 25 times.
My findings:
The first round it takes a lot longer than the other 24, I think this is because of caching, but I am not sure.
When I use a value that is in the start of the list, it has to move more items in memory than when I use a value at the end, though it still seems to take less time.
Benchtimes differ quite a bit.
When I enable the commented if, performance goes up (10-20%) even if the value I search for is almost at the end of the list (which means the if goes off a lot of times without actually being useful).
I have no idea why these things happen, is there someone who can explain (some of) them? And maybe if someone sees this who is a pro at this, where can I find more info to do this the most efficient way?
Edit after testing:
I did some testing and found some interesting results. I run the test on an array with a size of a million items, filled with a million objects. I run that 25 times and report the cumulative time in milliseconds. I do that 10 times and take the average of that as a final value.
When I run the test with my function described just above here I get a score of:
362,1
When I run it with the answer of dbc I get a score of:
846,4
So mine was faster, but then I started to experiment with a half empty empty array and things started to get weird. To get rid of the inevitable nullPointerExceptions I added an extra check to the if (thinking it would ruin a bit more of the performance) like so:
if (fromItem != null && fromItem.Value != value)
list[to++] = fromItem;
This seemed to not only work, but improve performance dramatically! Now I get a score of:
247,9
The weird thing is, the scores seem to low to be true, but sometimes spike, this is the set I took the avg from:
94, 26, 966, 36, 632, 95, 47, 35, 109, 439
So the extra evaluation seems to improve my performance, despite of doing an extra check. How is this possible?
You are using Stopwatch to time your method. This calculates the total clock time taken during your method call, which could include the time required for .Net to initially JIT your method, interruptions for garbage collection, or slowdowns caused by system loads from other processes. Noise from these sources will likely dominate noise due to cache misses.
This answer gives some suggestions as to how you can minimize some of the noise from garbage collection or other processes. To eliminate JIT noise, you should call your method once without timing it -- or show the time taken by the first call in a separate column in your results table since it will be so different. You might also consider using a proper profiler which will report exactly how much time your code used exclusive of "noise" from other threads or processes.
Finally, I'll note that your algorithm to remove matching items from an array and shift everything else down uses a nested loop, which is not necessary and will access items in the array after the matching index twice. The standard algorithm looks like this:
public static void RemoveFromArray(this Obj[] array, int value)
{
int to = 0;
for (int from = 0; from < array.Length; from++)
{
var fromItem = array[from];
if (fromItem.Value != value)
array[to++] = fromItem;
}
for (; to < array.Length; to++)
{
array[to] = default(Obj);
}
}
However, instead of using the standard algorithm you might experiment by using Array.RemoveAt() with your version, since (I believe) internally it does the removal in unmanaged code.
I'm trying to test the C# parallel methods and this is my test program:
class Program
{
static int counter;
static void Main(string[] args)
{
counter = 0;
Parallel.Invoke(
() => func(1),
() => func(2),
() => func(3)
);
Console.Read();
}
static void func(int num)
{
for (int i = 0; i < 5;i++ )
{
Console.WriteLine(string.Format("This is function #{0} loop. counter - {1}", num, counter));
counter++;
}
}
}
What I tried to do is to have 1 static shared variable and each function instance will increase it by 1.
I expected that counter will be printed in order (1,2,3,...)
But the output is surprising:
This is function #1 loop. counter - 0
This is function #1 loop. counter - 1
This is function #1 loop. counter - 2
This is function #1 loop. counter - 3
This is function #1 loop. counter - 4
This is function #3 loop. counter - 5
This is function #2 loop. counter - 1
This is function #3 loop. counter - 6
This is function #3 loop. counter - 8
This is function #3 loop. counter - 9
This is function #3 loop. counter - 10
This is function #2 loop. counter - 7
This is function #2 loop. counter - 12
This is function #2 loop. counter - 13
This is function #2 loop. counter - 14
Can anyone explain to me why this is happening?
The problem is that your code is not thread-safe. For example, what can happen is this:
function #2 gets the value of counter to use it in Console.WriteLine()
function #1 gets the value of counter, calls Console.WriteLine(), increments counter
function #1 gets the value of counter, calls Console.WriteLine(), increments counter
function #2 finally calls Console.WriteLine() with the old value
Also, ++ by itself is not thread-safe, so the final value may not be 15.
To fix both of these issues, you can use Interlocked.Increment():
for (int i = 0; i < 5; i++)
{
int incrementedCounter = Interlocked.Increment(ref counter);
Console.WriteLine("This is function #{0} loop. counter - {1}", num, incrementedCounter);
}
This way, you will get the number after increment, not before, as in your original code. Also, this code still won't print the numbers in the correct order, but you can be sure that each number will be printed exactly once.
If you do want to have the numbers in the correct order, you will need to use lock:
private static readonly object lockObject = new object();
…
for (int i = 0; i < 5; i++)
{
lock (lockObject)
{
Console.WriteLine("This is function #{0} loop. counter - {1}", num, counter);
counter++;
}
}
Of course, if you do this, you won't actually get any parallelism, but I assume this is not your real code.
Actually what happens - Invoke just queues up these tasks, and runtime assigns threads for these tasks, what gives alot of random element to it (which one is going to get picked up first etc).
Even msdn article states this:
This method can be used to execute a set of operations, potentially in parallel.
No guarantees are made about the order in which the operations execute or whether they execute in parallel. This method does not return until each of the provided operations has completed, regardless of whether completion occurs due to normal or exceptional termination.
This problem looks like that many threads access the same variable. This is a problem of concurrency.
You can try it:
static object syncObj = new object();
static void func(int num)
{
for (int i = 0; i < 5; i++)
{
lock (syncObj)
{
Console.WriteLine(string.Format("This is function #{0} loop. counter - {1}", num, counter));
counter++;
}
}
}
I wrote a class which uses Stopwatch to profile methods and for/foreach loops. With for and foreach loops it tests a standard loop against a Parallel.For or Parallel.ForEach implementation.
You would write performance tests like so:
Method:
PerformanceResult result = Profiler.Execute(() => { FooBar(); });
For loop:
SerialParallelPerformanceResult result = Profiler.For(0, 100, x => { FooBar(x); });
ForEach loop:
SerialParallelPerformanceResult result = Profiler.ForEach(list, item => { FooBar(item); });
Whenever I run the tests (one of .Execute, .For or .ForEach) I put them in a loop so I can see how the performance changes over time.
Example of performance might be:
Method execution 1 = 200ms
Method execution 2 = 12ms
Method execution 3 = 0ms
For execution 1 = 300ms (Serial), 100ms (Parallel)
For execution 2 = 20ms (Serial), 75ms (Parallel)
For execution 3 = 2ms (Serial), 50ms (Parallel)
ForEach execution 1 = 350ms (Serial), 300ms (Parallel)
ForEach execution 2 = 24ms (Serial), 89ms (Parallel)
ForEach execution 3 = 1ms (Serial), 21ms (Parallel)
My questions are:
Why does performance change over time, what is .NET doing in the background to facilitate this?
How/why is a serial operation faster than a parallel one? I have made sure that I make the operations complex to see the difference properly...in most cases serial operations seem faster!?
NOTE: For parallel processing I am testing on an 8 core machine.
After some more exploration into performance profiling, I have discovered that using a Stopwatch is not an accurate way to measure the performance of a particular task
(Thanks hatchet and Loren for your comments on this!)
Reasons a stopwatch are not accurate:
Measurements are calculated in elapsed time in milliseconds, not CPU time.
Measurements can be influenced by background "noise" and thread intensive processes.
Measurements do not take into account JIT compilation and overhead.
That being said, using a stopwatch is OK for casual exploration of performance. With that in mind, I have improved my profiling algorithm somewhat.
Where before it simply executed the expression that was passed to it, it now has the facility to iterate over the expression several times, building an average execution time. The first run can be omitted since this is where JIT kicks in, and some major overhead may occur. Understandably, this will never be as sophisticated as using a professional profiling tool like Redgate's ANTS profiler, but it's OK for simpler tasks!
As per my comment above: I did some simple tests on my own and found no difference over time. Can you share your code? I'll put mine in an answer as it doesn't fit here.
This is my sample code.
(I also tried with both static and instance methods with no difference)
class Program
{
static void Main(string[] args)
{
int to = 50000000;
OtherStuff os = new OtherStuff();
Console.WriteLine(Profile(() => os.CountTo(to)));
Console.WriteLine(Profile(() => os.CountTo(to)));
Console.WriteLine(Profile(() => os.CountTo(to)));
}
static long Profile(Action method)
{
Stopwatch st = Stopwatch.StartNew();
method();
st.Stop();
return st.ElapsedMilliseconds;
}
}
class OtherStuff
{
public void CountTo(int to)
{
for (int i = 0; i < to; i++)
{
// some work...
i++;
i--;
}
}
}
A sample output would be:
331
331
334
Consider executing this method instead:
class OtherStuff
{
public string CountTo(Guid id)
{
using(SHA256 sha = SHA256.Create())
{
int x = default(int);
for (int index = 0; index < 16; index++)
{
x = id.ToByteArray()[index] >> 32 << 16;
}
RNGCryptoServiceProvider rng = new RNGCryptoServiceProvider();
byte[] y = new byte[1024];
rng.GetBytes(y);
y = y.Concat(BitConverter.GetBytes(x)).ToArray();
return BitConverter.ToString(sha.ComputeHash(BitConverter.GetBytes(x).Where(o => o >> 2 < 0).ToArray()));
}
}
}
Sample output:
11
0
0
In a C# (feel free to answer for other languages) loop, what's the difference between break and continue as a means to leave the structure of the loop, and go to the next iteration?
Example:
foreach (DataRow row in myTable.Rows)
{
if (someConditionEvalsToTrue)
{
break; //what's the difference between this and continue ?
//continue;
}
}
break will exit the loop completely, continue will just skip the current iteration.
For example:
for (int i = 0; i < 10; i++) {
if (i == 0) {
break;
}
DoSomeThingWith(i);
}
The break will cause the loop to exit on the first iteration - DoSomeThingWith will never be executed. This here:
for (int i = 0; i < 10; i++) {
if(i == 0) {
continue;
}
DoSomeThingWith(i);
}
Will not execute DoSomeThingWith for i = 0, but the loop will continue and DoSomeThingWith will be executed for i = 1 to i = 9.
A really easy way to understand this is to place the word "loop" after each of the keywords. The terms now make sense if they are just read like everyday phrases.
break loop - looping is broken and stops.
continue loop - loop continues to execute with the next iteration.
break causes the program counter to jump out of the scope of the innermost loop
for(i = 0; i < 10; i++)
{
if(i == 2)
break;
}
Works like this
for(i = 0; i < 10; i++)
{
if(i == 2)
goto BREAK;
}
BREAK:;
continue jumps to the end of the loop. In a for loop, continue jumps to the increment expression.
for(i = 0; i < 10; i++)
{
if(i == 2)
continue;
printf("%d", i);
}
Works like this
for(i = 0; i < 10; i++)
{
if(i == 2)
goto CONTINUE;
printf("%d", i);
CONTINUE:;
}
When to use break vs continue?
Break - We're leaving the loop forever and breaking up forever. Good bye.
Continue - means that you're gonna give today a rest and sort it all out tomorrow (i.e. skip the current iteration)!
(Corny stories ¯¯\(ツ)/¯¯ and pics but hopefully helps you remember.
Grip Alert: No idea why those words are being used. If you want to skip the iteration, why not use the word skip instead of continue? This entire Stack overflow question and 1000s of developers would not be confused if the proper name was given.)
break would stop the foreach loop completely, continue would skip to the next DataRow.
There are more than a few people who don't like break and continue. The latest complaint I saw about them was in JavaScript: The Good Parts by Douglas Crockford. But I find that sometimes using one of them really simplifies things, especially if your language doesn't include a do-while or do-until style of loop.
I tend to use break in loops that are searching a list for something. Once found, there's no point in continuing, so you might as well quit.
I use continue when doing something with most elements of a list, but still want to skip over a few.
The break statement also comes in handy when polling for a valid response from somebody or something. Instead of:
Ask a question
While the answer is invalid:
Ask the question
You could eliminate some duplication and use:
While True:
Ask a question
If the answer is valid:
break
The do-until loop that I mentioned before is the more elegant solution for that particular problem:
Do:
Ask a question
Until the answer is valid
No duplication, and no break needed either.
All have given a very good explanation. I am still posting my answer just to give an example if that can help.
// break statement
for (int i = 0; i < 5; i++) {
if (i == 3) {
break; // It will force to come out from the loop
}
lblDisplay.Text = lblDisplay.Text + i + "[Printed] ";
}
Here is the output:
0[Printed] 1[Printed] 2[Printed]
So 3[Printed] & 4[Printed] will not be displayed as there is break when i == 3
//continue statement
for (int i = 0; i < 5; i++) {
if (i == 3) {
continue; // It will take the control to start point of loop
}
lblDisplay.Text = lblDisplay.Text + i + "[Printed] ";
}
Here is the output:
0[Printed] 1[Printed] 2[Printed] 4[Printed]
So 3[Printed] will not be displayed as there is continue when i == 3
Break
Break forces a loop to exit immediately.
Continue
This does the opposite of break. Instead of terminating the loop, it immediately loops again, skipping the rest of the code.
Simple answer:
Break exits the loop immediately.
Continue starts processing the next item. (If there are any, by jumping to the evaluating line of the for/while)
By example
foreach(var i in Enumerable.Range(1,3))
{
Console.WriteLine(i);
}
Prints 1, 2, 3 (on separate lines).
Add a break condition at i = 2
foreach(var i in Enumerable.Range(1,3))
{
if (i == 2)
break;
Console.WriteLine(i);
}
Now the loop prints 1 and stops.
Replace the break with a continue.
foreach(var i in Enumerable.Range(1,3))
{
if (i == 2)
continue;
Console.WriteLine(i);
}
Now to loop prints 1 and 3 (skipping 2).
Thus, break stops the loop, whereas continue skips to the next iteration.
Ruby unfortunately is a bit different.
PS: My memory is a bit hazy on this so apologies if I'm wrong
instead of break/continue, it has break/next, which behave the same in terms of loops
Loops (like everything else) are expressions, and "return" the last thing that they did. Most of the time, getting the return value from a loop is pointless, so everyone just does this
a = 5
while a < 10
a + 1
end
You can however do this
a = 5
b = while a < 10
a + 1
end # b is now 10
HOWEVER, a lot of ruby code 'emulates' a loop by using a block.
The canonical example is
10.times do |x|
puts x
end
As it is much more common for people to want to do things with the result of a block, this is where it gets messy.
break/next mean different things in the context of a block.
break will jump out of the code that called the block
next will skip the rest of the code in the block, and 'return' what you specify to the caller of the block. This doesn't make any sense without examples.
def timesten
10.times{ |t| puts yield t }
end
timesten do |x|
x * 2
end
# will print
2
4
6
8 ... and so on
timesten do |x|
break
x * 2
end
# won't print anything. The break jumps out of the timesten function entirely, and the call to `puts` inside it gets skipped
timesten do |x|
break 5
x * 2
end
# This is the same as above. it's "returning" 5, but nobody is catching it. If you did a = timesten... then a would get assigned to 5
timesten do |x|
next 5
x * 2
end
# this would print
5
5
5 ... and so on, because 'next 5' skips the 'x * 2' and 'returns' 5.
So yeah. Ruby is awesome, but it has some awful corner-cases. This is the second worst one I've seen in my years of using it :-)
Please let me state the obvious: note that adding neither break nor continue, will resume your program; i.e. I trapped for a certain error, then after logging it, I wanted to resume processing, and there were more code tasks in between the next row, so I just let it fall through.
To break completely out of a foreach loop, break is used;
To go to the next iteration in the loop, continue is used;
Break is useful if you’re looping through a collection of Objects (like Rows in a Datatable) and you are searching for a particular match, when you find that match, there’s no need to continue through the remaining rows, so you want to break out.
Continue is useful when you have accomplished what you need to in side a loop iteration. You’ll normally have continue after an if.
if you don't want to use break you just increase value of I in such a way that it make iteration condition false and loop will not execute on next iteration.
for(int i = 0; i < list.Count; i++){
if(i == 5)
i = list.Count; //it will make "i<list.Count" false and loop will exit
}
Since the example written here are pretty simple for understanding the concept I think it's also a good idea to look at the more practical version of the continue statement being used.
For example:
we ask the user to enter 5 unique numbers if the number is already entered we give them an error and we continue our program.
static void Main(string[] args)
{
var numbers = new List<int>();
while (numbers.Count < 5)
{
Console.WriteLine("Enter 5 uniqe numbers:");
var number = Convert.ToInt32(Console.ReadLine());
if (numbers.Contains(number))
{
Console.WriteLine("You have already entered" + number);
continue;
}
numbers.Add(number);
}
numbers.Sort();
foreach(var number in numbers)
{
Console.WriteLine(number);
}
}
lets say the users input were 1,2,2,2,3,4,5.the result printed would be:
1,2,3,4,5
Why? because every time user entered a number that was already on the list, our program ignored it and didn't add what's already on the list to it.
Now if we try the same code but without continue statement and let's say with the same input from the user which was 1,2,2,2,3,4,5.
the output would be :
1,2,2,2,3,4
Why? because there was no continue statement to let our program know it should ignore the already entered number.
Now for the Break statement, again I think its the best to show by example. For example:
Here we want our program to continuously ask the user to enter a number. We want the loop to terminate when the user types “ok" and at the end Calculate the sum of all the previously entered numbers and display it on the console.
This is how the break statement is used in this example:
{
var sum = 0;
while (true)
{
Console.Write("Enter a number (or 'ok' to exit): ");
var input = Console.ReadLine();
if (input.ToLower() == "ok")
break;
sum += Convert.ToInt32(input);
}
Console.WriteLine("Sum of all numbers is: " + sum);
}
The program will ask the user to enter a number till the user types "OK" and only after that, the result would be shown. Why?
because break statement finished or stops the ongoing process when it has reached the condition needed.
if there was no break statement there, the program would keep running and nothing would happen when the user typed "ok".
I recommend copying this code and trying to remove or add these statements and see the changes yourself.
As for other languages:
'VB
For i=0 To 10
If i=5 then Exit For '= break in C#;
'Do Something for i<5
next
For i=0 To 10
If i=5 then Continue For '= continue in C#
'Do Something for i<>5...
Next