Develop Reference

How to speed up nested loops in C#

This is a piece of my code, which calculate the differentiate. It works correctly but it takes a lot (because of height and width).
"Data" is a grey image bitmap.
"Filter" is [3,3] matrix.
"fh" and "fw" maximum values are 3.
I am looking to speed up this code.
I also tried with using parallel, for but it didn't work correct (error with out of bounds).
private float[,] Differentiate(int[,] Data, int[,] Filter)
{
int i, j, k, l, Fh, Fw;
Fw = Filter.GetLength(0);
Fh = Filter.GetLength(1);
float sum = 0;
float[,] Output = new float[Width, Height];
for (i = Fw / 2; i <= (Width - Fw / 2) - 1; i++)
{
for (j = Fh / 2; j <= (Height - Fh / 2) - 1; j++)
{
sum=0;
for(k = -Fw/2; k <= Fw/2; k++)
{
for(l = -Fh/2; l <= Fh/2; l++)
{
sum = sum + Data[i+k, j+l] * Filter[Fw/2+k, Fh/2+l];
}
}
Output[i,j] = sum;
}
}
return Output;
}

For parallel execution you need to drop c language like variable declaration at the beginning of method and declare them in actual scope that they are used so they are not shared between threads. Making it parallel should provide some benefit for performance, but making them all ParallerFors is not a good idea as there is a limit for threads amount that actually can run in parallel. I would try to make it with top level loop only:
private static float[,] Differentiate(int[,] Data, int[,] Filter)
{
var Fw = Filter.GetLength(0);
var Fh = Filter.GetLength(1);
float[,] Output = new float[Width, Height];
Parallel.For(Fw / 2, Width - Fw / 2 - 1, (i, state) =>
{
for (var j = Fh / 2; j <= (Height - Fh / 2) - 1; j++)
{
var sum = 0;
for (var k = -Fw / 2; k <= Fw / 2; k++)
{
for (var l = -Fh / 2; l <= Fh / 2; l++)
{
sum = sum + Data[i + k, j + l] * Filter[Fw / 2 + k, Fh / 2 + l];
}
}
Output[i, j] = sum;
}
});
return Output;
}

This is a perfect example of a task where using the GPU is better than using the CPU. A GPU is able to perform trillions of floating point operations per second (TFlops), while CPU performance is still measured in GFlops. The catch is that it's only any good if you use SIMD instructions (Single Instruction Multiple Data). The GPU excels at data-parallel tasks. If different data needs different instructions, using the GPU has no advantage.
In your program, the elements of your bitmap go through the same calculations: the same computations just with slightly different data (SIMD!). So using the GPU is a great option. This won't be too complex because with your calculations threads on the GPU would not need to exchange information, nor would they be dependent on results of previous iterations (Each element would be processed by a different thread on the GPU).
You can use, for example, OpenCL to easily access the GPU. More on OpenCL and using the GPU here: https://www.codeproject.com/Articles/502829/GPGPU-image-processing-basics-using-OpenCL-NET

Find the difference between two 1d arrays to form a 2d array of those differences in the shortest time

I am trying to find the absolute difference between every element of one array and every element of another to form a matrix.
I have achieved this using for loops but it is slow and I need it to be faster. I can do it faster in R for example by using the dist method but I am struggling to make it fast in C#.
double[] array1 = new double [] { 1.1, 2.0, 3.0, 4.0, 5.0 };
double[] array2 = new double[] { 6.1, 7.0, 8.0};
double[,] final_array = new double[5, 3];
for (int i = 0; i < 5; i++)
{
for (j = 0; j < 3; j++)
{
final_array[i,j] = Math.Abs(array1[i] - array2[j])
}
}
# expected result of final_array
5 4.1 3.1 2.1 1.1
5.9 5 4 3 2
6.9 6 5 4 3
Although this result is the correct answer I want to do this faster as I will need to do this calculation for arrays of up to 15,000 in size.

You can use vectors in the System.Numerics namespace. The caveat is that it will only work with float, not with double. That shouldn't be a problem for subtraction though:
float[] array1 = new float[] { 1.1F, 2.0F, 3.0F, 4.0F, 5.0F };
float[] array2 = new float[] { 6.1F, 7.0F, 8.0F };
float[,] final_array = new float[array1.Length, array2.Length];
int vectorCount = array2.Length / 4;
Vector4[] array2Vectors = new Vector4[vectorCount];
Parallel.For(0, vectorCount, i =>
{
int offset = i * 4;
array2Vectors[i] = new Vector4(array2[offset], array2[offset + 1],
array2[offset + 2], array2[offset + 3]);
});
Parallel.For(0, array1.Length, i =>
{
Vector4 v1 = new Vector4(array1[i], array1[i], array1[i], array1[i]);
for (int j = 0; j < array2Vectors.Length; j++)
{
Vector4 result = Vector4.Abs(Vector4.Subtract(v1, array2Vectors[j]));
int offset = j * 4;
final_array[i, offset] = result.X;
final_array[i, offset + 1] = result.Y;
final_array[i, offset + 2] = result.Z;
final_array[i, offset + 3] = result.W;
}
for (int j = vectorCount * 4; j < array2.Length; j++)
{
final_array[i,j] = Math.Abs(array1[i] - array2[j]);
}
});
Since you are using vectors now, you will make use of the CPU's SIMD instructions, which should speed up your task.
Additional performance gains come from parallel execution with Parallel.For, which makes use of all available CPU cores.
You can try it out here.

There is no way to do this faster in terms of algorithmic complexity. It requires exactly O(n * m) operations to calculate this result, at least because you have the resulting array of that size.
There are some ways to slightly improve the performance of the code itself.
The easiest one is to switch to jagged arrays, as already suggested in the comments:
double[] array1 = new double [] { 1.1, 2.0, 3.0, 4.0, 5.0 };
double[] array2 = new double[] { 6.1, 7.0, 8.0};
double[][] final_array = new double[5][];
for (int i = 0; i < 5; i++)
{
final_array[i] = new double[3];
for (int j = 0; j < 3; j++)
{
final_array[i][j] = Math.Abs(array1[i] - array2[j]);
}
}
You can read more about multidimensional array vs jagged arrays and their performance here:
What are the differences between a multidimensional array and an array of arrays in C#?
You could also go further and increase performance by using unsafe pointers to access multidimensional array or by utilizing advanced processor instructions (intrinsics), but... the question is: is this really something you need to think of? Is it an only bottle neck in an extremely high-load system? If it is not, then just leave your code as it is, in a clearly readable and understandable form. Saying about performance, O(n * m) asymptotic complexity is perfectly fine for arrays of size 15000.

Alternative to array pointers in safe C#?

I'm trying to translate a bit of C++ into C# and I can't seem to do it without losing a ton of performance due to the loss of speed in looking up and accessing array elements. I'm using 3d jagged arrays because that was the most intuitive to me at the time but I'm very open to alternate suggestions. And so my question, is there a way to access some kind of collection in the same way, or a similar way, as array pointers do? Here's a bit of the C++ I was converting:
void Upsample(float* from, float* to, int n, int stride)
{
float* p, pCoeffs[4] = { 0.25, 0.75, 0.75, 0.25 };
p = &pCoeffs[2];
for (int i = 0; i < n; i++)
{
to[i * stride] = 0;
for (int k = i / 2; k <= i / 2 + 1; k++)
to[i * stride] += p[i - 2 * k] * from[Mod(k, n / 2) * stride];
}
}
////
for (iy=0; iy<n; iy++) for (iz=0; iz<n; iz++) {
Upsample( &noise[i], &temp1[i], n, 1 );
}

finding maximum in a 3d matrix

I have matrix with 3 dimension (n*m*k). I am trying to fined the maximum number for each n and m by searching in k dimension.((I try to find the maximum number in k dimension for each n and m)) and at last i have a 2d matrix (n*m). i have the following code but it is so slow. Is there any new code or any changes to the current code that do this more quickly.
thanks.
my c# code: note: li is the 3 dimension matrix and they are grater or equal to zero.
int[,] array2 = new int[n, m];
int[,] array = new int[n, m];
List<Array> li = new List<Array>();
for(int k = 0; k <'not a specific value, change each time' ; k++)
{
li.Add(array);
%% changing array
} %% so li will became a (n*m*k) matrix
for (int i = 0; i < n; i++)
for (int j = 0; j < m; j++)
{
int ma = -2;
int d = 0;
while (d <= k)
{
ma = Math.Max(ma, Convert.ToInt32(li[d].GetValue(i, j)));
d++;
}
array2[i, j] = ma;
}

The biggest performance issue is that you use Array objects as elements of your list. This makes it so that every element access using GetValue boxes the value, i.e. allocates a new tiny object to hold the element value.
Your code will run a lot faster if you replace
List<Array> li = new List<Array>();
with
List<int[,]> li = new List<int[,]>();
and
ma = Math.Max(ma, Convert.ToInt32(li[d].GetValue(i, j)));
with
ma = Math.Max(ma, li[d][i, j];
Since you don't know the 3rd dimension in advance, it is harder to use 3D arrays.
An entirely different approach would be to compute the maximum as you're building the list li. This will help in two ways: 1. You avoid indexing into the list of arrays and 2. as long as m and n aren't too large, you improve locality. That is: the values you're working with are closer together in memory, and more likely to be in the processor cache.

This should do the trick (even though it could be kinda slower than your approach):
// put this at the top of your source file
using System.Linq;
// put this where you calculate the maxima
for(int i = 0; i < array2.GetLength(0); ++i)
for(int j = 0; j < array2.GetLength(1); ++j)
{
array2[i, j] = Convert.ToInt32(li.Max(x => x.GetValue(i, j)));
}

You could use a three-dimensional array like this:
int xRange = 10;
int yRange = 10;
int zRange = 10;
int[, ,] matrix = new int[xRange, yRange, zRange];
// set up some dummy values
for (int x = 0; x < xRange; x++)
for (int y = 0; y < yRange; y++)
for (int z = 0; z < zRange; z++)
matrix[x, y, z] = x * y * z;
// calculate maximum values
int[,] maxValues = new int[xRange, yRange];
/* LINQ version of maximum calculation
for (int x = 0; x < xRange; x++)
for (int y = 0; y < yRange; y++)
maxValues[x, y] = Enumerable.Range(0, zRange).Select(z => matrix[x, y, z]).Max();
*/
// longhand version of maximum calculation
for (int x = 0; x < xRange; x++)
for (int y = 0; y < yRange; y++)
for (int z = 0; z < zRange; z++)
maxValues[x, y] = Math.Max(maxValues[x, y], matrix[x, y, z]);
// display results
for (int x = 0; x < xRange; x++)
{
for (int y = 0; y < yRange; y++)
Console.Write("{0}\t", maxValues[x, y]);
Console.WriteLine();
}

From an algorithm perspective, there's no more efficient way to evaluate the maximum value for fixed n,m for all k. It will take O(n*m*k) operations.
Now, the only way to improve performance is to find improvements in your implementation, particularly in your storage of the 3D matrix.
Using List<Array> is a major area for improvement. You are prone to boxing issues (conversion of primitive types to objects) and making more function calls than are necessary.
Reduce your 3D matrix to an array of primitives:
int[] my3DArray = new int[n * m * l]; // Note I'm using l where you use k
Now index into your array at [i, j, k] using the following offset:
int elementAtIJK = my3DArray[i + (n * j) + (m * n * k)];
If you just use arrays of primitives you should see a definite improvement.
Edit:
In fact, in C# (and several other languages) it's very easy to implement 3D arrays directly, e.g.:
int[,,] my3DArray = new int[n,m,l];
int elementAtIJK = my3DArray[i,j,k];
Which is much simpler than I first described (but at the end of the day is internally translated in the 1D form).
What to do if the 3rd dimension varies in size...
Now, it gets more interesting if the size of the 3rd dimension varies significantly. If it has a known maximum and isn't too big, you can simply set it to the maximum and fill the empty values with zeroes. This is simple and may meet your needs.
However, if the 3rd dimension can be very big, all these extra stored zeroes could waste a lot of valuable space and what you need is a Sparse Matrix representation.
There are different storage mechanisms for sparse matrices. For your purposes, you could consider your 3D array to be a 2D matrix, with (n*m) rows and max(k) columns. As the 3rd dimension varies in length, there are lots of empty spaces in your columns. This is called a sparse row and the standard data storage for this is "Compressed Sparse Row". Again, for performance this can be represented just by three primitive arrays, a data array, a row index array and a column pointer array. There are resources elsewhere on the web that describe the CSR implementation better than I can, but hopefully this will point you in the right direction.

Counting sort - implementation differences

I heard about Counting Sort and wrote my version of it based on what I understood.
public void my_counting_sort(int[] arr)
{
int range = 100;
int[] count = new int[range];
for (int i = 0; i < arr.Length; i++) count[arr[i]]++;
int index = 0;
for (int i = 0; i < count.Length; i++)
{
while (count[i] != 0)
{
arr[index++] = i;
count[i]--;
}
}
}
The above code works perfectly.
However, the algorithm given in CLRS is different. Below is my implementation
public int[] counting_sort(int[] arr)
{
int k = 100;
int[] count = new int[k + 1];
for (int i = 0; i < arr.Length; i++)
count[arr[i]]++;
for (int i = 1; i <= k; i++)
count[i] = count[i] + count[i - 1];
int[] b = new int[arr.Length];
for (int i = arr.Length - 1; i >= 0; i--)
{
b[count[arr[i]]] = arr[i];
count[arr[i]]--;
}
return b;
}
I've directly translated this from pseudocode to C#. The code doesn't work and I get an IndexOutOfRange Exception.
So my questions are:
What's wrong with the second piece of code ?
What's the difference algorithm wise between my naive implementation and the one given in the book ?

The problem with your version is that it won't work if the elements have satellite data.
CLRS version would work and it's stable.
EDIT:
Here's an implementation of the CLRS version in Python, which sorts pairs (key, value) by key:
def sort(a):
B = 101
count = [0] * B
for (k, v) in a:
count[k] += 1
for i in range(1, B):
count[i] += count[i-1]
b = [None] * len(a)
for i in range(len(a) - 1, -1, -1):
(k, v) = a[i]
count[k] -= 1
b[count[k]] = a[i]
return b
>>> print sort([(3,'b'),(2,'a'),(3,'l'),(1,'s'),(1,'t'),(3,'e')])
[(1, 's'), (1, 't'), (2, 'a'), (3, 'b'), (3, 'l'), (3, 'e')]

It should be
b[count[arr[i]]-1] = arr[i];
I'll leave it to you to track down why ;-).
I don't think they perform any differently. The second just pushes the correlation of counts out of the loop so that it's simplified a bit within the final loop. That's not necessary as far as I'm concerned. Your way is just as straightforward and probably more readable. In fact (I don't know about C# since I'm a Java guy) I would expect that you could replace that inner while-loop with a library array fill; something like this:
for (int i = 0; i < count.Length; i++)
{
arrayFill(arr, index, count[i], i);
index += count[i];
}
In Java the method is java.util.Arrays.fill(...).

The problem is that you have hard-coded the length of the array that you are using to 100. The length of the array should be m + 1 where m is the maximum element on the original array. This is the first reason that you would think using counting-sort, if you have information about the elements of the array are all minor that some constant and it would work great.