Develop Reference

Resilient Propagation (RProp) training slowly

I am creating a library for Neural Networks; I have successfully implemented the Back Propagation algorithm, but I'm having trouble with Resilient Propagation.
I have been using the XOR scenario to test my implementation; the error seems to decrease after a few epochs, then it increases and eventually stops changing.
The network I am testing on has 3 layers - 2 input, 2 hidden and 1 output neuron/s. The synapse/dendrite weights are on the following layer's neurons, for example, the input-hidden weights would be stored on the hidden neurons and the hidden-output weights would be stored on the output neurons.
This is my gradient calculation (EDIT 4):
private void CalculateGradient()
{
for (int t = 0; t < TrainingInputs.Count; ++t) //loop through training data
{
Network.Run(TrainingInputs[t]);
for (int l = Network.Layers.Count - 1; l > 0; l--)
{
for (int i = 0; i < Network.Layers[l].Neurons.Count; i++)
{
Network.Layers[l].Neurons[i].Delta = l < Network.Layers.Count - 1
? CalculateNonLastGradient(l + 1, i, Network.Layers[l].Neurons[i].Value)
: CalculateLastGradient(TrainingOutputs[t][i], Network.Layers[l].Neurons[i].Value);
}
}
for (int l = Network.Layers.Count - 1; l > 0; l--)
{
for (int j = 0; j < Network.Layers[l].Neurons.Count; ++j)
{
double grad = Network.Layers[l].Neurons[j].Delta;
biasGrads[l][j] += grad;
for (int i = 0; i < Network.Layers[l - 1].Neurons.Count; ++i)
{
grad = Network.Layers[l].Neurons[j].Delta * Network.Layers[l - 1].Neurons[i].Value;
grads[l][j][i] += grad;
}
}
}
int o = 0;
Layer layer = Network.Layers[Network.Layers.Count - 1];
errors.Add(layer.Neurons.Sum(n => Math.Abs(TrainingOutputs[t][o++] - n.Value)));
}
Error = errors.Average();
}
private double CalculateLastGradient(double ideal, double nValue)
{
return Network.Activation.Derivitive(nValue) * (ideal - nValue);
}
private double CalculateNonLastGradient(int nextLayer, int j, double nValue)
{
double sum = 0.0;
for (int i = 0; i < Network.Layers[nextLayer].Neurons.Count; i++)
{
sum += Network.Layers[nextLayer].Neurons[i].Delta * Network.Layers[nextLayer].Neurons[i].Dendrites[j].Weight;
}
return Network.Activation.Derivitive(nValue) * sum;
}
My RProp implementation (EDIT 4):
public bool Algorithm()
{
//initialise matrices
if (!initializedMatrices)
{
InitializeMatrices();
}
ZeroOut();
//calculate gradients
CalculateGradient();
for (int l = 1; l < Network.Layers.Count; l++) //layers
{
for (int i = 0; i < Network.Layers[l - 1].Neurons.Count; ++i) //prev layer neurons
{
for (int j = 0; j < Network.Layers[l].Neurons.Count; ++j) //current layer neurons
{
double delta = prevDeltas[l][j][i];
int change = Math.Sign(prevGrads[l][j][i] * grads[l][j][i]);
if (change > 0)
{
delta = Math.Min(delta * etaPlus, deltaMax);
double deltaWeight = -Math.Sign(grads[l][j][i]) * delta;
Network.Layers[l].Neurons[j].Dendrites[i].Weight += deltaWeight;
}
else if (change < 0)
{
delta = Math.Max(delta * etaMinus, deltaMin);
Network.Layers[l].Neurons[j].Dendrites[i].Weight -= prevDeltas[l][j][i];
prevGrads[l][j][i] = 0;
}
else if (change == 0)
{
double deltaWeight = -Math.Sign(grads[l][j][i]) * delta;
Network.Layers[l].Neurons[j].Dendrites[i].Weight += deltaWeight;
}
prevGrads[l][j][i] = grads[l][j][i];
prevDeltas[l][j][i] = delta;
} //j
} //i
for (int i = 1; i < Network.Layers[l].Neurons.Count; ++i)
{
double delta = prevBiasDeltas[l][i];
int change = Math.Sign(prevBiasGrads[l][i] * biasGrads[l][i]);
if (change > 0)
{
delta = Math.Min(prevBiasDeltas[l][i] * etaPlus, deltaMax);
double biasDeltaWeight = -Math.Sign(biasGrads[l][i]) * delta;
Network.Layers[l].Neurons[i].Bias += biasDeltaWeight;
}
else if (change < 0)
{
delta = Math.Max(prevBiasDeltas[l][i] * etaMinus, deltaMin);
Network.Layers[l].Neurons[i].Bias -= prevBiasDeltas[l][i];
prevBiasGrads[l][i] = 0;
}
else if (change == 0)
{
double biasDeltaWeight = -Math.Sign(biasGrads[l][i]) * delta;
Network.Layers[l].Neurons[i].Bias += biasDeltaWeight;
}
prevBiasGrads[l][i] = biasGrads[l][i];
prevBiasDeltas[l][i] = delta;
}
}
return true;
}
Could anyone point me as to what could be going wrong?
EDIT:
The issue seems to be that the delta is not changing.
EDIT 2:
I have fixed the delta not changing by initializing the first previous deltas to 0.01 and Zero'ing the gradients before each call. The error increases very fast, then decreases very slowly with TANH; with Sigmoid it does the opposite.
EDIT 3:
The loop for the bias' was starting at 0, when it should have been 1. Fixing this fixed the original issue, but has brought up at new one - the error stops decreasing after a certain point.
EDIT 4: I realised that I had not accumulated the gradients for each neuron over the training set. The error now decreases, but very slowly.

3X3 Median Filtering in c# but not working?

Really I'm trying to apply 3X3 Median Filtering by C# and depending on my understanding the concepts of Median Filtering I wrote the following code but when I'm running it the Form hangs. I think have some problem in the last nested for loop but i don't know where is the error or the wrong in applying the Median concepts!
public static Bitmap MedianFiltering(Bitmap bm)
{
List<int> termsList = new List<int>();
Bitmap res, temp;
Color c;
int counter = 0;
//Convert to Grayscale
for (int i = 0; i < bm.Width; i++)
{
for (int j = 0; j < bm.Height; j++)
{
c = bm.GetPixel(i, j);
byte gray = (byte)(.333 * c.R + .333 * c.G + .333 * c.B);
bm.SetPixel(i, j, Color.FromArgb(gray, gray, gray));
}
}
temp = bm;
//applying Median Filtering
for (int i = 0; i <= temp.Width - 3; i++)
for (int j = 0; j <= temp.Height - 3; j++)
{
for (int x = i; x <= i + 2; x++)
for (int y = j; y <= j + 2; y++)
{
c = temp.GetPixel(x, y);
termsList.Add(c.R);
counter++;
}
int[] terms = termsList.ToArray();
Array.Sort<int>(terms);
Array.Reverse(terms);
int color = terms[4];
temp.SetPixel(i + 1, j + 1, Color.FromArgb(color, color, color));
counter = 0;
}
res = temp;
return res;
}
Thanks.

You are not clearing the termsList after each pixel processing. This is causing the list to keep growing. Sorting and reversing the list will keep taking longer and longer times. This will also cause incorrect results since you only want to get the median of the 9 pixels related to the current pixel.
Simply clear the list like this:
...
int[] terms = termsList.ToArray();
termsList.Clear();
...
UPDATE:
I did more optimization for the code:
public static void MedianFiltering(Bitmap bm)
{
List<byte> termsList = new List<byte>();
byte[,] image = new byte[bm.Width,bm.Height];
//Convert to Grayscale
for (int i = 0; i < bm.Width; i++)
{
for (int j = 0; j < bm.Height; j++)
{
var c = bm.GetPixel(i, j);
byte gray = (byte)(.333 * c.R + .333 * c.G + .333 * c.B);
image[i, j] = gray;
}
}
//applying Median Filtering
for (int i = 0; i <= bm.Width - 3; i++)
for (int j = 0; j <= bm.Height - 3; j++)
{
for (int x = i; x <= i + 2; x++)
for (int y = j; y <= j + 2; y++)
{
termsList.Add(image[x, y]);
}
byte[] terms = termsList.ToArray();
termsList.Clear();
Array.Sort<byte>(terms);
Array.Reverse(terms);
byte color = terms[4];
bm.SetPixel(i + 1, j + 1, Color.FromArgb(color, color, color));
}
}
Please note that in your original method, you returned a Bitmap. I removed this.
Please note that temp = bm; does not create a copy of the Bitmap. It is just pointing the temp variable to the same object (that is pointed by bm). So in your original method, you returned the exact object that is passed in the method parameter. To use the new method pass the Bitmap and then the bitmap it self will be modified (this is also true for your method).
This enhanced performance 4 times on my machine.
What I did is mainly read the bitmap data into a byte array instead of using the Bitmap it self to read/write data multiple times.
If you need to further enhance the performance, take a look at this question.

Neural network [ocr]

I come looking for general tips about the program I'm writing now.
The goal is:
Use neural network program to recognize 3 letters [D,O,M] (or display "nothing is recognized" if i input anything other than those 3).
Here's what I have so far:
A class for my single neuron
public class neuron
{
double[] weights;
public neuron()
{
weights = null;
}
public neuron(int size)
{
weights = new double[size + 1];
Random r = new Random();
for (int i = 0; i <= size; i++)
{
weights[i] = r.NextDouble() / 5 - 0.1;
}
}
public double output(double[] wej)
{
double s = 0.0;
for (int i = 0; i < weights.Length; i++) s += weights[i] * wej[i];
s = 1 / (1 + Math.Exp(s));
return s;
}
}
A class for a layer:
public class layer
{
neuron[] tab;
public layer()
{
tab = null;
}
public layer(int numNeurons, int numInputs)
{
tab = new neuron[numNeurons];
for (int i = 0; i < numNeurons; i++)
{
tab[i] = new neuron(numInputs);
}
}
public double[] compute(double[] wejscia)
{
double[] output = new double[tab.Length + 1];
output[0] = 1;
for (int i = 1; i <= tab.Length; i++)
{
output[i] = tab[i - 1].output(wejscia);
}
return output;
}
}
And finally a class for a network
public class network
{
layer[] layers = null;
public network(int numLayers, int numInputs, int[] npl)
{
layers = new layer[numLayers];
for (int i = 0; i < numLayers; i++)
{
layers[i] = new layer(npl[i], (i == 0) ? numInputs : (npl[i - 1]));
}
}
double[] compute(double[] inputs)
{
double[] output = layers[0].compute(inputs);
for (int i = 1; i < layers.Length; i++)
{
output = layers[i].compute(output);
}
return output;
}
}
Now for the algorythm I chose:
I have a picture box, size 200x200, where you can draw a letter (or read one from jpg file).
I then convert it to my first array(get the whole picture) and 2nd one(cut the non relevant background around it) like so:
Bitmap bmp2 = new Bitmap(this.pictureBox1.Image);
int[,] binaryfrom = new int[bmp2.Width, bmp2.Height];
int minrow=0, maxrow=0, mincol=0, maxcol=0;
for (int i = 0; i < bmp2.Height; i++)
{
for (int j = 0; j < bmp2.Width; j++)
{
if (bmp2.GetPixel(j, i).R == 0)
{
binaryfrom[i, j] = 1;
if (minrow == 0) minrow = i;
if (maxrow < i) maxrow = i;
if (mincol == 0) mincol = j;
else if (mincol > j) mincol = j;
if (maxcol < j) maxcol = j;
}
else
{
binaryfrom[i, j] = 0;
}
}
}
int[,] boundaries = new int[binaryfrom.GetLength(0)-minrow-(binaryfrom.GetLength(0)-(maxrow+1)),binaryfrom.GetLength(1)-mincol-(binaryfrom.GetLength(1)-(maxcol+1))];
for(int i = 0; i < boundaries.GetLength(0); i++)
{
for(int j = 0; j < boundaries.GetLength(1); j++)
{
boundaries[i, j] = binaryfrom[i + minrow, j + mincol];
}
}
And convert it to my final array of 12x8 like so (i know I could shorten this a fair bit, but wanted to have every step in different loop so I can see what went wrong easier[if anything did]):
int[,] finalnet = new int[12, 8];
int k = 1;
int l = 1;
for (int i = 0; i < finalnet.GetLength(0); i++)
{
for (int j = 0; j < finalnet.GetLength(1); j++)
{
finalnet[i, j] = 0;
}
}
while (k <= finalnet.GetLength(0))
{
while (l <= finalnet.GetLength(1))
{
for (int i = (int)(boundaries.GetLength(0) / finalnet.GetLength(0)) * (k - 1); i < (int)(boundaries.GetLength(0) / finalnet.GetLength(0)) * k; i++)
{
for (int j = (int)(boundaries.GetLength(1) / finalnet.GetLength(1)) * (l - 1); j < (int)(boundaries.GetLength(1) / finalnet.GetLength(1)) * l; j++)
{
if (boundaries[i, j] == 1) finalnet[k-1, l-1] = 1;
}
}
l++;
}
l = 1;
k++;
}
int a = boundaries.GetLength(0);
int b = finalnet.GetLength(1);
if((a%b) != 0){
k = 1;
while (k <= finalnet.GetLength(1))
{
for (int i = (int)(boundaries.GetLength(0) / finalnet.GetLength(0)) * finalnet.GetLength(0); i < boundaries.GetLength(0); i++)
{
for (int j = (int)(boundaries.GetLength(1) / finalnet.GetLength(1)) * (k - 1); j < (int)(boundaries.GetLength(1) / finalnet.GetLength(1)) * k; j++)
{
if (boundaries[i, j] == 1) finalnet[finalnet.GetLength(0) - 1, k - 1] = 1;
}
}
k++;
}
}
if (boundaries.GetLength(1) % finalnet.GetLength(1) != 0)
{
k = 1;
while (k <= finalnet.GetLength(0))
{
for (int i = (int)(boundaries.GetLength(0) / finalnet.GetLength(0)) * (k - 1); i < (int)(boundaries.GetLength(0) / finalnet.GetLength(0)) * k; i++)
{
for (int j = (int)(boundaries.GetLength(1) / finalnet.GetLength(1)) * finalnet.GetLength(1); j < boundaries.GetLength(1); j++)
{
if (boundaries[i, j] == 1) finalnet[k - 1, finalnet.GetLength(1) - 1] = 1;
}
}
k++;
}
for (int i = (int)(boundaries.GetLength(0) / finalnet.GetLength(0)) * finalnet.GetLength(0); i < boundaries.GetLength(0); i++)
{
for (int j = (int)(boundaries.GetLength(1) / finalnet.GetLength(1)) * finalnet.GetLength(1); j < boundaries.GetLength(1); j++)
{
if (boundaries[i, j] == 1) finalnet[finalnet.GetLength(0) - 1, finalnet.GetLength(1) - 1] = 1;
}
}
}
The result is a 12x8 (I can change it in the code to get it from some form controls) array of 0 and 1, where 1 form the rough shape of a letter you drawn.
Now my questions are:
Is this a correct algorythm?
Is my function
1/(1+Math.Exp(x))
good one to use here?
What should be the topology? 2 or 3 layers, and if 3, how many neurons in hidden layer? I have 96 inputs (every field of the finalnet array), so should I also take 96 neurons in the first layer? Should I have 3 neurons in the final layer or 4(to take into account the "not recognized" case), or is it not necessary?
Thank you for your help.
EDIT: Oh, and I forgot to add, I'm gonna train my network using Backpropagation algorythm.

You may need 4 layers at least to get accurate results using back propagation method. 1 input, 2 middle layers, and an output layer.
12 * 8 matrix is too small(and you may end up in data loss which will result in total failure) - try some thing 16 * 16. If you want to reduce the size then you have to peel out the outer layers of black pixels further.
Think about training the network with your reference characters.
Remember that you have to feed back the output back to the input layer again and iterate it multiple times.

A while back I created a neural net to recognize digits 0-9 (python, sorry), so based on my (short) experience, 3 layers are ok and 96/50/3 topology will probably do a good job. As for the output layer, it's your choice; you can either backpropagate all 0s when the input image is not a D, O or M or use the fourth output neuron to indicate that the letter was not recognized. I think that the first option would be the best one because it's simpler (shorter training time, less problems debugging the net...), you just need to apply a threshold under which you classify the image as 'not recognized'.
I also used the sigmoid as activation function, I didn't try others but it worked :)

index outside bounds of the array adaptive median

I'm trying to output a new image after applying the adaptive median filter but it only works if the maximum window size is 3*3, but it should work for all odd window sizes, and it does so if the image is so small for example 10*10 pixels, but if the image is for example 440*445 pixels it only works if the max window size is 3*3 otherwise it says index outside bounds of the array, here is my code using counting sort to sort the pixels:
using System;
using System.Collections.Generic;
using System.Text;
using System.Linq;
namespace ImageFilters
{
public class Class2
{
public byte[,] median(byte[,] array, int width, int height, int msize)
{
byte[,] marr = new byte[height + 1, width + 1];
int size = 3;
for (int i = 0; i <height+1; i++)
{
marr[i, 0] = 255;
}
for (int j = 0; j < width+1; j++)
{
marr[0, j] = 255;
}
int n=0;
int z=0;
for (int k = 0; k < (height) * width; k++)
{
repeat:
byte[] farr = new byte[size * size];
int I=0;
for (int i = n; i < n+size; i++)
{
for (int j = z; j < z+size; j++)
{
if (j < width && i < height)
{
farr[I] = array[i, j];
I++;
}
else
{
farr[I] = 0;
I++;
}
}
}
int zxy = farr[(size*size)/2];
byte[] check = new byte[size * size];
check=counting(farr,size);
int median = 0;
int lennn;
lennn = check.Length;
int min = 99999;
int maxi = -1;
int a1, a2;
min = check.Min();
maxi = check.Max();
median = check[lennn / 2];
a1 = median - min;
a2 = maxi - median;
int b1, b2;
b1 = zxy - min;
b2 = maxi - zxy;
if (a1 > 0 && a2 > 0)
{
if (b1 > 0 && b2 > 0)
{
marr[n + 1, z + 1] = Convert.ToByte(zxy);
z++;
if (z + (size - 1) > (width + 1))
{
n++;
z = 0;
}
}
else
{
marr[n + 1, z + 1] = Convert.ToByte(median);
z++;
if (z + (size - 1) > (width + 1))
{
n++;
z = 0;
}
}
}
else
{
size = size + 2;
if (size <= msize)
goto repeat;
else
{
marr[n +1, z +1] = Convert.ToByte(median);
z++;
if (size > 3)
size = 3;
if (z + (size - 1) > (width + 1))
{
n++;
z = 0;
}
}
}
}
return marr;
}
public static byte[] counting(byte[] array1D, int siz)
{
int max = -10000000;
byte[] SortedArray = new byte[siz * siz];
max = array1D.Max();
byte[] Array2 = new byte[max + 1];
//for (int i = 0; i < Array2.Length; i++)
// Array2[i] = 0; // create new array ( Array2) with zeros in every index .
for (int i = 0; i < (siz*siz); i++)
{
Array2[array1D[i]] += 1; //take the element in the index(i) of(array1d) AS the index of (Array2)
// and increment the element in this index by 1 .
}
for (int i = 1; i <= max; i++)
{
Array2[i] += Array2[i - 1]; // Count every element in (array1d) and put it in (Array2).
}
for (int i = (siz*siz) - 1; i >= 0; i--)
{
SortedArray[Array2[array1D[i]] - 1] = array1D[i]; // transfer the element in index (i) of (array1d) to (SortedArray
Array2[array1D[i]]--;
}
return SortedArray;
}
}
}
Any help will be appreciated, thank you.

1D Fast Convolution without FFT

I need an 1D Convolution against 2 big arrays. I'm using this code in C# but it takes a loooong time to run.
I know, i know! FFT convolutions is very fast. But in this project i CAN'T use it.
It is a constraint of the project to not use FFT (please don't ask why :/).
This is my code in C# (ported from matlab, by the way):
var result = new double[input.Length + filter.Length - 1];
for (var i = 0; i < input.Length; i++)
{
for (var j = 0; j < filter.Length; j++)
{
result[i + j] += input[i] * filter[j];
}
}
So, anyone knows any fast convolution algorithm widthout FFT?

Convolution is numerically the same as a polynomial multiplication with an extra wrap-around step. Therefore, all the polynomial and large integer multiplication algorithms can be used to perform convolution.
FFT is the only way to get the fast O(n log(n)) run-time. But you can still get sub-quadratic run-time using the divide-and-conquer approaches like Karatsuba's algorithm.
Karatsuba's algorithm is fairly easy to implement once you understand how it works. It runs in O(n^1.585), and will probably be faster than trying to super-optimize the classic O(n^2) approach.

You could reduce the number of indexed accesses to result, as well as the Length properties:
int inputLength = filter.Length;
int filterLength = filter.Length;
var result = new double[inputLength + filterLength - 1];
for (int i = resultLength; i >= 0; i--)
{
double sum = 0;
// max(i - input.Length + 1,0)
int n1 = i < inputLength ? 0 : i - inputLength + 1;
// min(i, filter.Length - 1)
int n2 = i < filterLength ? i : filterLength - 1;
for (int j = n1; j <= n2; j++)
{
sum += input[i - j] * filter[j];
}
result[i] = sum;
}
If you further split the outer loop, you can get rid of some repeating conditionals. (This assumes 0 < filterLength &leq; inputLength &leq; resultLength)
int inputLength = filter.Length;
int filterLength = filter.Length;
int resultLength = inputLength + filterLength - 1;
var result = new double[resultLength];
for (int i = 0; i < filterLength; i++)
{
double sum = 0;
for (int j = i; j >= 0; j--)
{
sum += input[i - j] * filter[j];
}
result[i] = sum;
}
for (int i = filterLength; i < inputLength; i++)
{
double sum = 0;
for (int j = filterLength - 1; j >= 0; j--)
{
sum += input[i - j] * filter[j];
}
result[i] = sum;
}
for (int i = inputLength; i < resultLength; i++)
{
double sum = 0;
for (int j = i - inputLength + 1; j < filterLength; j++)
{
sum += input[i - j] * filter[j];
}
result[i] = sum;
}

You can use a special IIR filter.
Then process that as like:
y(n)= a1*y(n-1)+b1*y(n-2)...+a2*x(n-1)+b2*x(n-2)......
I think it's faster.

Here are two possibilities that may give minor speedups, but you'd need to test to be sure.
Unroll the inner loop to remove some tests. This will be easier if you know the filter length will always be a multiple if some N.
Reverse the order of the loops. Do filter.length passes over the whole array. This does less dereferencing in the inner loop but may have worse caching behavior.