Develop Reference

Expensive C# operation, looking to improve performance [closed]

Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 3 years ago.
Improve this question
This will be a long question, sorry in advance. I don't expect a full code solution, I am looking for some input from people with a different perspective and some more experience than me.
My company is developing software for a product that does some rather expensive calculations using a film from an IR camera where every pixel contains a temperature value. The most costly of those methods is called Thermal Signal Reconstruction (if you are interested, you can read about it here https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4321698/ ). It basically performs a polynomial fit for each pixel over time (the number of frames). My C# implementation looks something like this:
public static double[,,] ThermalSignalReconstruction(List<Frame<double>> thermalFilm, byte polyOrder)
{
Resolution filmResolution = thermalFilm[0].Resolution;
uint width = filmResolution.Width;
uint height = filmResolution.Height;
int frames = thermalFilm.Count;
double[,,] result = new double[polyOrder + 1, height, width];
// using frame indexes as x-values for poly fit
List<double> frameIndexes = new List<double>(frames);
for (var frame = 0U; frame < frames; ++frame)
frameIndexes.Add(frame);
// make local copy of thermal film and fill with difference images
List<Frame<double>> localThermalFilm = new List<Frame<double>>(frames);
for (var frame = 0U; frame < frames; ++frame)
localThermalFilm.Add(new Frame<double>(filmResolution));
Parallel.For(0U, frames, frame =>
{
for (var row = 0U; row < height; ++row)
for (var col = 0U; col < width; ++col)
localThermalFilm[(int)frame].Data[row, col] = thermalFilm[(int)frame].Data[row, col] - thermalFilm[0].Data[row, col];
});
// determine flashpoint by finding the frame with the maximum average pixel value
double maxAverage = double.MinValue;
uint maxIndex = 0U;
Parallel.For(0U, frames, frame =>
{
double average = Math.MatrixMean(localThermalFilm[(int)frame].Data);
if (average > maxAverage)
{
maxAverage = average;
maxIndex = (uint)frame;
}
});
// remove frames preceeding flashpoint, including itself, from film
localThermalFilm.RemoveRange(0, (int)maxIndex + 1);
frameIndexes.RemoveRange(0, (int)maxIndex + 1);
frames -= (int)maxIndex + 1;
// calculate base 10 logarithm of all pixels and frame indexes
Parallel.For(0U, frames, frame =>
{
for (var row = 0U; row < height; ++row)
for (var col = 0U; col < width; ++col)
localThermalFilm[(int)frame].Data[row, col] = System.Math.Log10(localThermalFilm[(int)frame].Data[row, col]);
frameIndexes[(int)frame] = System.Math.Log10(frameIndexes[(int)frame]);
});
// perform polynomial fit for each pixel
Parallel.For(0U, height, row =>
{
for (var col = 0U; col < width; ++col)
{
// extract poly fit input y-values for current pixel
double[] pixelValues = new double[frames];
for (var frame = 0U; frame < frames; ++frame)
pixelValues[frame] = localThermalFilm[(int)frame].Data[row, col];
// (...) do some value validations
// poly fit for current pixel - this is the longest step
double[] coefficients = Math.PolynomialRegression(frameIndexesValidated.ToArray(), pixelValuesValidated.ToArray(), polyOrder);
// insert into coefficient images result array
for (var coefficient = 0U; coefficient < result.GetLength(0); ++coefficient)
result[coefficient, row, col] = coefficients[coefficient];
}
});
return result;
}
As you can see, several parallelized loops performing several operations on the frames are executed in sequence with the polynomial fit (Math.PolynomialRegression) being the last and most expensive one. This is a function containing a polynomial fit algorithm I pieced together myself since it doesn't exist in the standard System.Math library and the only other one I tried from the Math.NET library actually runs slower than the one I wrote. My code is based on examples given on Rosetta Code: https://rosettacode.org/wiki/Polynomial_regression
My point is, that I wrote this entire algorithm before in unmanaged C++, our company decided to move away from that because of some licensing issues with the GUI framework we were using back then and to instead now use C#/.NET. A direct comparison of the old unmanaged C++ code with the one I posted above in managed C# recently showed me that the C# code takes about 48% (!!!) longer to execute than the C++ code even though the algorithm is identical. I am aware that C# is a higher level, managed language and therefore has a greater translation distance than C++ so I fully expected it to run slower, but I didn't expect it to be this bad. 48% is quite a big deal which leads me to believe that I may be doing something wrong. On the other hand, I don't have that much experience yet so if I'm perfectly honest I also don't really know what to expect in a situation like this.
What I have tried so far:
switching between running the various individual loops sequentially and parallelized, it's fastest with all of them parallelized like above
adjusting the variables that are accessed by the individual parallelized loop instances (e.g. not accessing the same resolution object each time but declaring separate variables for width and height before starting the loop) which already improved performance quite a bit, but the 48% are still left after that
trying a Parallel.ForEach(Partitioner.Create(0, frames) ... ) approach, i.e. partitioning the chunks of data more coarsely with the Partitioner class, it didn't help, made the code run slower
optimizing other functions that are called as well as code on the side of this one's caller as best I can
To home in to the question: is it even possible to make a C# code like this run with comparable performance than the same code in C++, if so how? Or is what I observed perfectly normal and I have to deal with it?
EDIT: added the first three loop bodies in TSR per request and my polynomial regression implementation looks like this:
public static double[] PolynomialRegression(in double[] xValues, in double[] yValues, byte order)
{
Debug.Assert(xValues != null && yValues != null);
Debug.Assert(xValues.Length == yValues.Length);
Debug.Assert(xValues.Length != 0 || yValues.Length != 0);
int dataSamples = xValues.Length;
double[] result = new double[order + 1];
// array containing N,sigma(xi),sigma(xi^2),sigma(xi^3)...sigma(xi^2*poly_order), where N=number of samples
double[] sigmaX = new double[2 * order + 1];
for (var index = 0U; index < sigmaX.Length; ++index)
{
sigmaX[index] = 0.0;
for (var dataPoint = 0U; dataPoint < dataSamples; ++dataPoint)
sigmaX[index] += System.Math.Pow(xValues[(int)dataPoint], index);
}
// array containing sigma(yi),sigma(xi*yi),sigma(xi^2*yi)...sigma(xi^poly_order*yi)
double[] sigmaY = new double[order + 1];
for (var pOrder = 0U; pOrder < sigmaY.Length; ++pOrder)
{
sigmaY[pOrder] = 0.0;
for (var dataPoint = 0U; dataPoint < dataSamples; ++dataPoint)
sigmaY[pOrder] += System.Math.Pow(xValues[(int)dataPoint], pOrder) * yValues[(int)dataPoint];
}
// equation system's augmented normal matrix
int matrixRows = order + 1;
int matrixCols = order + 2;
double[,] matrix = new double[matrixRows, matrixCols];
for (var row = 0U; row < matrixRows; ++row)
for (var col = 0U; col < matrixCols - 1; ++col)
matrix[row, col] = sigmaX[row + col];
for (var row = 0U; row < matrixRows; ++row)
matrix[row, order + 1] = sigmaY[row];
// pivotisation of matrix
for (var pivotRow = 0U; pivotRow < matrixRows; ++pivotRow)
for (var lowerRow = pivotRow + 1U; lowerRow < matrixRows; ++lowerRow)
if (matrix[pivotRow, pivotRow] < matrix[lowerRow, pivotRow])
for (var col = 0U; col < matrixCols; ++col)
{
double temp = matrix[pivotRow, col];
matrix[pivotRow, col] = matrix[lowerRow, col];
matrix[lowerRow, col] = temp;
}
// Gaussian elimination
for (var pivotRow = 0U; pivotRow < matrixRows; ++pivotRow)
for (var lowerRow = pivotRow + 1U; lowerRow < matrixRows; ++lowerRow)
{
double ratio = matrix[lowerRow, pivotRow] / matrix[pivotRow, pivotRow];
for (var col = 0U; col < matrixCols; ++col)
matrix[lowerRow, col] -= ratio * matrix[pivotRow, col];
}
// back-substitution
for (var row = (short)order; row >= 0; --row)
{
result[row] = matrix[row, order + 1];
for (var col = 0U; col < matrixCols - 1; ++col)
if (col != row)
result[row] -= matrix[row, col] * result[col];
result[row] /= matrix[row, row];
}
return result;
}

Thank you to everyone who commented. I have tried some of the suggestions: rewriting the PolynomialRegression method using fixed pointers, this did not have any effect. I also combined some of the loops in the TSR method, I now just have two parallel loops running in sequence (need the first one definitely to find the flash point), this helped but only little bit (something like 1m21s instead of 1m26s).
Then I analyzed the code a little more with the VS CPU profiler. 85% of all CPU cycles within the TSR method were done in the PolynomialRegression method, as expected this one does the bulk of the work. Within polynomial regression however was what surprised me: the System.Math.Pow method is actually a huge bottleneck.
the first call: sigmaX[index] += System.Math.Pow(xValues[(int)dataPoint], index); within the first loop owned about 55% of the CPU cycles
the second call in the second loop: sigmaY[pOrder] += System.Math.Pow(xValues[(int)dataPoint], pOrder) * yValues[(int)dataPoint]; owned about 26%
the other steps, even the big matrix pivotisation and Gaussian elimination steps were almost negligible by comparison.
I gathered this is because System.Math.Pow is the general implementation of the exponentiation problem, thus including all sorts of checking for negative and fractional exponents. Since in my current problem I only ever had positive integer exponents I wrote my own specialized method instead:
public static double UIntPow(double #base, uint power)
{
if (power == 0)
return 1.0;
else if (power == 1)
return #base;
else
return #base * UIntPow(#base, power - 1);
}
While this recursive method runs extremely slowly in debug mode (about twice as slowly as System.Math.Pow), it actually is very fast in the release build where code gets optimized. The execution of TSR now actually runs faster than the C++ equivalent, although I assume I could have gotten the same performance improvement there if I had also used my own UIntPow method.
Again thanks to everyone who took the time to look at my problem and maybe this solution helps someone in the future.
EDIT: Thanks again for the input! This algorithm runs even faster than my recursive attempt:
public static double UIntPow(double #base, uint power)
{
double result = 1.0;
while (power != 0)
{
if ((power & 1) == 1)
result *= #base;
#base *= #base;
power >>= 1;
}
return result;
}

Is it possible to have smoother contour lines with the ContourSeries in OxyPlot?

I used OxyPlot for drawing contours, but they seem a little rough at some places, so I would need to smooth them. So, I was wondering if it was possible to smooth the contour with OxyPlot or if I will need something else?
I know that you can smooth the LineSeries, so by looking inside the code for the LineSeries I have tried to put the code for smoothing in the ContourSeries, but I am not quite able to figure out how everything works. I tried to find smoothing algorithm on the internet, but I wasn't able to find many and those that I tried didn't work at all.
Also, I have tried to look for another library for drawing contours, but OxyPlot seems to be the best for free for WPF, but if you have other suggestions that can give me better results I would appreciate it.

After some more searching I found an algorithm that did a pretty good job, but some lines that needed to be closed were not, but the performance were really good by comparing to other algorithms I tried.
Here's the algorithm
private void SmoothContour(List<DataPoint> points, int severity = 1)
{
for (int i = 1; i < points.Count; i++)
{
var start = (i - severity > 0 ? i - severity + 1 : 0);
var end = (i + severity < points.Count ? i + severity + 1 : points.Count);
double sumX = 0, sumY = 0;
for (int j = start; j < end; j++)
{
sumX += points[j].X;
sumY += points[j].Y;
}
DataPoint sum = new DataPoint(sumX, sumY);
DataPoint avg = new DataPoint(sum.X / (end - start), sum.Y / (end - start));
points[i] = avg;
}
}
It's based on this method: https://stackoverflow.com/a/18830268/11788646
After that I put this method in the ContourSeries class like this :
List<Contour> smoothedContours = new List<Contour>();
for (int i = 0; i < contours.Count; i++)
{
List<DataPoint> smoothedPoints = new List<DataPoint>(contours[i].Points);
for (int j = 0; j < SmoothLevel; j++)
{
SmoothContour(smoothedPoints);
}
Contour contour = new Contour(smoothedPoints, contours[i].ContourLevel);
smoothedContours.Add(contour);
}
contours = smoothedContours;
It's located in the CalculateContours() method just after the call of the JoinContourSegments() method. I also added a SmoothLevel property to add more smoothing to the lines. The higher it is the smoother the lines are, but when it is set too high, it doesn't do too well. So I keep at 10 and it's good.

Optimizing an adjacency matrix creation from a 2D array of nodes

I am attempting to create an adjacency matrix from a 2D array of nodes. The adjacency matrix will be passed to a program that will cluster the nodes either through
Spectral clustering algorithm
Kmeans clustering algorithm
**Node class **
public class Node{
public int _id;
public bool _isWalkable;
public int _positionX;
public int _positionY;
public Vector3 _worldPosition;
}
Grid Class
public class Grid : MonoBehaviour
{
void CreateGrid()
{
grid = new Node[_gridSizeX, _gridSizeY];
Vector3 worldBottomLeft = transform.position -
Vector3.right * worldSize.x / 2 - Vector3.forward * worldSize.y / 2;
//set the grid
int id = 0;
for (int x = 0; x < _gridSizeX; x++)
{
for (int y = 0; y < _gridSizeY; y++)
{
Vector3 worldPosition = worldBottomLeft + Vector3.right *
(x * _nodeDiameter + _nodeRadius) +
Vector3.forward * (y * _nodeDiameter + _nodeRadius);
//check to see if current position is walkable
bool isWalkable =
!Physics.CheckSphere(worldPosition, _nodeRadius, UnwalkableMask);
grid[x, y] = new Node(isWalkable, worldPosition, x, y);
grid[x, y].Id = id ++;
}
}
totalNodes = id;
}
}
Nodes are stored inside a 2D array called grid and represent a walkable path for a character to move on. I have succesfully implemented an A* algorithm with a euclidean distance heuristic. What I would like to do is cluster these nodes using the aforementioned clustering algorithms, but first I need to create an adjacency algorithm for them. This is the best pseudocode I could come up with
int[][] _adjacencyMatrix = new int[gridSizeX*gridSizeY][gridSizeX*gridSizeY];
for(int x = 0; x < gridSize;x< XgridSize; i++)
{
for(int y = 0; y < gridSize;y< YgridSize; i++)
{
if( !Grid[x][y]._isWalkable)
continue;
Node n = Grid[x][y];
List<Node> neighbors = GetNeighbors(n);
for(int k; k<neighbors.Count(); k++)
{
_adjacencyMatrix[n._id][neighbors[k]._id]=1;
}
}
}
public List<Node> GetNeighbours(Node n)
{
//where is this node in the grid?
List<Node> neighbours = new List<Node>();
//this will search in a 3X3 block
for (int x = -1; x <= 1; x++)
{
for (int y = -1; y <= 1; y++)
{
if (x == 0 && y == 0)
continue; //we're at the current node
int checkX = n._positionX + x;
int checkY = n._positionY + y;
if (checkX >= 0 && checkX < _gridSizeX && checkY >= 0
&& checkY < _gridSizeY)
{
if(grid[checkX, checkY]._isWalkable)
neighbours.Add(grid[checkX, checkY]);
else
continue;
}
}
}
return neighbours;
}
My main concern
My main concern with this is the total complexity of the above algorithm. It feels like it's going to be heavy and I have a total of (75^2 = 5625) nodes in a adjacency matrix that will be 5625X5625 in size! There must be a better way to find the neighbors than this, is there?

The matrix is symmetric, so you only need to save half of it, see (How to store a symmetric matrix?) for an example. The matrix values are binary, so saving them as booleans or in a bit vector will cut down memory by a factor of 4 or 32, respectively.
Alternatively, since the check for two adjacent nodes takes constant time (abs(n1.x - n2.x) <= 1 && abs(n1.y - n1.y) <= 1 && grid[n1.x, n2.x].isWalkable() && grid[n2.x, n2.y]), you could just pass the clustering algorithm a function which checks for adjacency on-the-fly.

5k by 5k is not very large. 100 MB is something you can keep in memory. If you want to avoid this cost, do not use algorithms based on distance matrixes!
However, since your similarity appears to be
d(x,y) = 1 if adjacent and both nodes walkable else 0
your results will degenerate. If you are lucky, you get something like connected components (which you could have gotten much easier).
Pairwise shortest paths would be more useful, but also more expensive to build. Maybe consider solving this first, though. Having a full adjacency matrix is a good starting point I guess.
k-means cannot work with pairwise distances at all. It needs distances point-to-mean only, for arbitrary means.
I suggest to look at graph algorithms, and spend some more time understanding your objective, before trying to squeeze the data into clustering algorithms that may be solving a different problem.

Neural Net backpropagation doesn't work properly

Lately I've implemented my own neural network (using different guides, but mainly from here), for future use (I intend to use it for an OCR program i'l develop). currently I'm testing it, and I'm having this weird problem.
Whenever I give my network a training example, the algorithm changes the weights in a way that leads to the desired output. However, after a few training examples, the weights get messed up- making the network work well for some outputs, and making it wrong for other outputs (even if I enter the input of the training examples, exactly as it was).
I would appreciate if someone directed me towards the problem, should they see it.
Here are the methods for calculating the error of the neurons and the weight adjusting-
private static void UpdateOutputLayerDelta(NeuralNetwork Network, List<double> ExpectedOutputs)
{
for (int i = 0; i < Network.OutputLayer.Neurons.Count; i++)
{
double NeuronOutput = Network.OutputLayer.Neurons[i].Output;
Network.OutputLayer.Neurons[i].ErrorFactor = ExpectedOutputs[i]-NeuronOutput; //calculating the error factor
Network.OutputLayer.Neurons[i].Delta = NeuronOutput * (1 - NeuronOutput) * Network.OutputLayer.Neurons[i].ErrorFactor; //calculating the neuron's delta
}
}
//step 3 method
private static void UpdateNetworkDelta(NeuralNetwork Network)
{
NeuronLayer UpperLayer = Network.OutputLayer;
for (int i = Network.HiddenLayers.Count - 1; i >= 0; i--)
{
foreach (Neuron LowerLayerNeuron in Network.HiddenLayers[i].Neurons)
{
for (int j = 0; j < UpperLayer.Neurons.Count; j++)
{
Neuron UpperLayerNeuron = UpperLayer.Neurons[j];
LowerLayerNeuron.ErrorFactor += UpperLayerNeuron.Delta * UpperLayerNeuron.Weights[j + 1]/*+1 because of bias*/;
}
LowerLayerNeuron.Delta = LowerLayerNeuron.Output * (1 - LowerLayerNeuron.Output) * LowerLayerNeuron.ErrorFactor;
}
UpperLayer = Network.HiddenLayers[i];
}
}
//step 4 method
private static void AdjustWeights(NeuralNetwork Network, List<double> NetworkInputs)
{
//Adjusting the weights of the hidden layers
List<double> LowerLayerOutputs = new List<double>(NetworkInputs);
for (int i = 0; i < Network.HiddenLayers.Count; i++)
{
foreach (Neuron UpperLayerNeuron in Network.HiddenLayers[i].Neurons)
{
UpperLayerNeuron.Weights[0] += -LearningRate * UpperLayerNeuron.Delta;
for (int j = 1; j < UpperLayerNeuron.Weights.Count; j++)
UpperLayerNeuron.Weights[j] += -LearningRate * UpperLayerNeuron.Delta * LowerLayerOutputs[j - 1] /*-1 because of bias*/;
}
LowerLayerOutputs = Network.HiddenLayers[i].GetLayerOutputs();
}
//Adjusting the weight of the output layer
foreach (Neuron OutputNeuron in Network.OutputLayer.Neurons)
{
OutputNeuron.Weights[0] += -LearningRate * OutputNeuron.Delta * 1; //updating the bias - TODO: change this if the bias is also changed throughout the program
for (int j = 1; j < OutputNeuron.Weights.Count; j++)
OutputNeuron.Weights[j] += -LearningRate * OutputNeuron.Delta * LowerLayerOutputs[j - 1];
}
}
The learning rate is 0.5, and the neurons' activation function is a sigmoid function.
EDIT: I've noticed I never implemented the function to calculate the overall error: E=0.5 * Sum(t-y) for each training example. could that be the problem? and if so, how should I fix it?

The learning rate 0.5 seems a bit too large. Usually values closer to 0.01 or 0.1 are used. Also, it usually helps in convergence if training patterns are presented in random order. More useful hints can be found here: Neural Network FAQ (comp.ai.neural archive).

Estimate cubic polynomial that maps set x to set y

I have a (sampled) set of uncalibrated values (x) coming from a device and a set of what they should be (y). I'm looking to find/estimate the cubic polynomial y=ax^3 + bx^2 + cx + d that maps any x to y.
So I think what I need to do is Polynomial Regression first and then find its inverse, but I'm not so sure; and I wonder whether there is a better solution like least squares.
I would appreciate a nudge in the right direction and/or any links to a math library that would be of use.

Have you checked Langrange Interpolation?
It is about the polynomial approximation of a given function.
You can stop the approximation on a given degree of the polynomial (let's say the 3rd degree) on a proper range of the indipendent variable.
Refs:
http://en.wikipedia.org/wiki/Lagrange_polynomial
https://math.stackexchange.com/a/108623

Looks like its just Polynomial Regression; I just need to feed in the raw (x) values and the expected values (y).
Code from Rosetta Code, that uses Math.Net Numerics
using MathNet.Numerics.LinearAlgebra.Double;
using MathNet.Numerics.LinearAlgebra.Double.Factorization;
public static class PolyRegression
{
public static double[] Polyfit(double[] x, double[] y, int degree)
{
// Vandermonde matrix
var v = new DenseMatrix(x.Length, degree + 1);
for (int i = 0; i < v.RowCount; i++)
for (int j = 0; j <= degree; j++) v[i, j] = Math.Pow(x[i], j);
var yv = new DenseVector(y).ToColumnMatrix();
QR qr = v.QR();
// Math.Net doesn't have an "economy" QR, so:
// cut R short to square upper triangle, then recompute Q
var r = qr.R.SubMatrix(0, degree + 1, 0, degree + 1);
var q = v.Multiply(r.Inverse());
var p = r.Inverse().Multiply(q.TransposeThisAndMultiply(yv));
return p.Column(0).ToArray();
}
}