I'm taking the Coursera machine learning course right now and I cant get my gradient descent linear regression function to minimize. I use: one dependent variable, an intercept, and four values of x and y, therefore the equations are fairly simple. The final value of the Gradient Decent equation varies wildly depending on the initial values of alpha and beta and I cant figure out why.
I've only been coding for about two weeks, so my knowledge is limited to say the least, please keep this in mind if you take the time to help.
using System;
namespace LinearRegression
{
class Program
{
static void Main(string[] args)
{
Random rnd = new Random();
const int N = 4;
//We randomize the inital values of alpha and beta
double theta1 = rnd.Next(0, 100);
double theta2 = rnd.Next(0, 100);
//Values of x, i.e the independent variable
double[] x = new double[N] { 1, 2, 3, 4 };
//VAlues of y, i.e the dependent variable
double[] y = new double[N] { 5, 7, 9, 12 };
double sumOfSquares1;
double sumOfSquares2;
double temp1;
double temp2;
double sum;
double learningRate = 0.001;
int count = 0;
do
{
//We reset the Generalized cost function, called sum of squares
//since I originally used SS to
//determine if the function was minimized
sumOfSquares1 = 0;
sumOfSquares2 = 0;
//Adding 1 to counter for each iteration to keep track of how
//many iterations are completed thus far
count += 1;
//First we calculate the Generalized cost function, which is
//to be minimized
sum = 0;
for (int i = 0; i < (N - 1); i++)
{
sum += Math.Pow((theta1 + theta2 * x[i] - y[i]), 2);
}
//Since we have 4 values of x and y we have 1/(2*N) = 1 /8 = 0.125
sumOfSquares1 = 0.125 * sum;
//Then we calcualte the new alpha value, using the derivative of
//the cost function.
sum = 0;
for (int i = 0; i < (N - 1); i++)
{
sum += theta1 + theta2 * x[i] - y[i];
}
//Since we have 4 values of x and y we have 1/(N) = 1 /4 = 0.25
temp1 = theta1 - learningRate * 0.25 * sum;
//Same for the beta value, it has a different derivative
sum = 0;
for (int i = 0; i < (N - 1); i++)
{
sum += (theta1 + theta2 * x[i]) * x[i] - y[i];
}
temp2 = theta2 - learningRate * 0.25 * sum;
//WE change the values of alpha an beta at the same time, otherwise the
//function wont work
theta1 = temp1;
theta2 = temp2;
//We then calculate the cost function again, with new alpha and beta values
sum = 0;
for (int i = 0; i < (N - 1); i++)
{
sum += Math.Pow((theta1 + theta2 * x[i] - y[i]), 2);
}
sumOfSquares2 = 0.125 * sum;
Console.WriteLine("Alpha: {0:N}", theta1);
Console.WriteLine("Beta: {0:N}", theta2);
Console.WriteLine("GCF Before: {0:N}", sumOfSquares1);
Console.WriteLine("GCF After: {0:N}", sumOfSquares2);
Console.WriteLine("Iterations: {0}", count);
Console.WriteLine(" ");
} while (sumOfSquares2 <= sumOfSquares1 && count < 5000);
//we end the iteration cycle once the generalized cost function
//cannot be reduced any further or after 5000 iterations
Console.ReadLine();
}
}
}
There are two bugs in the code.
First, I assume that you would like to iterate through all the element in the array. So rework the for loop like this: for (int i = 0; i < N; i++)
Second, when updating the theta2 value the summation is not calculated well. According to the update function it should be look like this: sum += (theta1 + theta2 * x[i] - y[i]) * x[i];
Why the final values depend on the initial values?
Because the gradient descent update step is calculated from these values. If the initial values (Starting Point) are too big or too small, then it will be too far away from the final values (Final Value). You could solve this problem by:
Increasing the iteration steps (e.g. 5000 to 50000): gradient descent algorithm has more time to converge.
Decreasing the learning rate (e.g. 0.001 to 0.01): gradient descent update steps are bigger, therefore it converges faster. Note: if the learning rate is too small, then it is possible to step through the global minimum.
The slope (theta2) is around 2.5 and the intercept (theta1) is around 2.3 for the given data. I have created a github project to fix your code and i have also added a shorter solution using LINQ. It is 5 line of codes. If you are curious check it out here.
Related
All Decimal numbers are rounded to 2 digits when saved into application. I'm given a number totalAmount and asked to divide it into n equal parts(or close to equal).
Example :
Given : totalAmount = 421.9720; count = 2 (totalAmount saved into application is 421.97)
Expected : 210.99, 210.98 => sum = 421.97
Actual(with plain divide) : 210.9860 (210.99), 210.9860 (210.99) => sum = 412.98
My approach :
var totalAmount = 421.972m;
var count = 2;
var individualCharge = Math.Floor(totalAmount / count);
var leftOverAmount = totalAmount - (individualCharge * count);
for(var i = 0;i < count; i++) {
Console.WriteLine(individualCharge + leftOverAmount);
leftOverAmount = 0;
}
This gives (-211.97, -210)
public IEnumerable<decimal> GetDividedAmounts(decimal amount, int count)
{
var pennies = (int)(amount * 100) % count;
var baseAmount = Math.Floor((amount / count) * 100) / 100;
foreach (var _ in Enumerable.Range(1, count))
{
var offset = pennies-- > 0 ? 0.01m : 0m;
yield return baseAmount + offset;
}
}
Feel free to alter this if you want to get an array or an IEnumerable which is not deferred. I updated it to get the baseAmount to be the floor value so it isn't recalculated within the loop.
Basically you need to find the base amount and a total of all the leftover pennies. Then, simply add the pennies back one by one until you run out. Because the pennies are based on the modulus operator, they'll always be in the range of [0, count - 1], so you'll never have a final leftover penny.
You're introducing a few rounding errors here, then compounding them. This is a common problem with financial data, especially when you have to constrain your algorithm to only produce outputs with 2 decimal places. It's worse when dealing with actual money in countries where 1 cent/penny/whatever coins are no longer legal tender. At least when working with electronic money the rounding isn't as big an issue.
The naive approach of dividing the total by the count and rounding the results is, as you've already discovered, not going to work. What you need is some way to spread out the errors while varying the output amounts by no more than $0.01. No output value can be more than $0.01 from any other output value, and the total must be the truncated total value.
What you need is a way to distribute the error across the output values, with the smallest possible variation between the values in the result. The trick is to track your error and adjust the output down once the error is high enough. (This is basically how the Bresenham line-drawing algorithm figures out when to increase the y value, if that helps.)
Here's the generalized form, which is pretty quick:
public IEnumerable<decimal> RoundedDivide(decimal amount, int count)
{
int totalCents = (int)Math.Floor(100 * amount);
// work out the true division, integer portion and error values
float div = totalCents / (float)count;
int portion = (int)Math.Floor(div);
float stepError = div - portion;
float error = 0;
for (int i = 0; i < count; i++)
{
int value = portion;
// add in the step error and see if we need to add 1 to the output
error += stepError;
if (error > 0.5)
{
value++;
error -= 1;
}
// convert back to dollars and cents for outputput
yield return value / 100M;
}
}
I've tested it with count values from 1 through 100, all outputs sum to match the (floored) input value exactly.
Try to break it down to steps:
int decimals = 2;
int factor = (int)Math.Pow(10, decimals);
int count = 2;
decimal totalAmount = 421.97232m;
totalAmount = Math.Floor(totalAmount * factor) / factor; // 421.97, you may want round here, depends on your requirement.
int baseAmount = (int)(totalAmount * factor / count); // 42197 / 2 = 21098
int left = (int)(totalAmount * factor) % count; // 1
// Adding back the left for Mod operation
for (int i = 0; i < left; i++)
{
Console.WriteLine((decimal)(baseAmount + 1) / factor); // 21098 + 1 / 100 = 210.99
}
// The reset that does not needs adjust
for (int i = 0; i < count - left; i++)
{
Console.WriteLine((decimal)baseAmount / factor); // 21098 / 100 = 210.98
}
I want to implement the Hilbert Transform in C#. From this article I saw that the fastest FFT open source implementation seems to be the FFTW, so I downloaded that example and used it to learn how to use the fftw wrapper for C#.
I have a current signal of 200.000 points which I'm using for testing. Getting the Hilbert transform through the fft is relatively simple:
Compute the fft.
Multiply by 2 all positive frequencies except for the DC and Nyquist components (0 and n/2 + 1, if the sample size is even).
Multiply by 0 all the negative frequencies ([n/2 + 1, n]).
Compute the inverse fft.
This far, I've done all of it. The only problem is the inverse fft. I'm not able to get the same results with fftw than with the ifft from Matlab.
My code
RealArray _input;
ComplexArray _fft;
void ComputeFFT()
{
_fft = new ComplexArray(_length / 2 + 1);
_input.Set(Data);
_plan = Plan.Create1(_length, _input, _fft, Options.Estimate);
_plan.Execute();
}
This far, I've a fft with only the positive frequencies. So I don't need to multiply by zero the negative frequencies: they don't even exist. With the following code, I can get my original signal back:
double[] ComputeIFFT(ComplexArray input)
{
double[] temp = new double[_length];
RealArray output = new RealArray(_length);
_plan = Plan.Create1(_length, input, output, Options.Estimate);
_plan.Execute();
temp = output.ToArray();
for (int i = 0; i < _length; ++i)
{
temp[i] /= _length;
}
return temp;
}
The problem comes when I try to get a complex inverse from the signal.
void ComputeHilbert()
{
double[] fft = FFT.ToArray();
double[] h = new double[_length / 2 + 1];
double[] temp = new double[_length * 2];
bool fftLengthIsOdd = (_length | 1) == 1;
h[0] = 1;
for (int i = 1; i < _length / 2; i++) h[i] = 2;
if (!fftLengthIsOdd) h[(_length / 2)] = 1;
for (int i = 0; i <= _length / 2; i++)
{
temp[2 * i] = fft[2*i] * h[i];
temp[2 * i + 1] = fft[2*i + 1] * h[i];
}
ComplexArray _tempHilbert = new ComplexArray(_length);
_tempHilbert.Set(temp);
_hilbert = ComputeIFFT(_tempHilbert);
_hilbertComputed = true;
}
It's important to note that, when I do apply the ToArray() method on a ComplexArray object, I get as result a double[] with twice as length as the original array, having the real and imaginary parts consecutive. That's it, for a ComplexArray object containing "3 + 1i", I would get a double vector with [3, 1].
So, at this moment, what I have is something like:
[DC Frequency, 2*positive frequencies, Nyquist Frequency, zeros]
If I export this data to Matlab and compute the IFFT, I get the same result as its hilbert(signal).
However, if I try to apply the IFFT provided by fftw, I get weird values from Nyquist Frequency to the end (that is to say, the zeros mess with fftw).
This is the ifft I'm using to do this:
double[] ComputeIFFT(ComplexArray input)
{
double[] temp;
ComplexArray output = new ComplexArray(_length);
_plan = Plan.Create1(_length, input, output, Direction.Backward, Options.Estimate);
_plan.Execute();
temp = output.ToArray();
for (int i = 0; i < _length; ++i)
{
temp[i] /= _length;
}
return temp;
}
So, just to sum it up, my problem is the way I'm using to calculate the ifft. It doesn't seems to work well with zeros. Or maybe Matlab is capable to understand that it has to apply some different approach and I should do it manually, but I don't know how.
Thank you very much for your help in advance, much appreciated!
So the problem was the ComputeIFFT function. In the for loop, I was doing i < _length, but the length of temp array is 2 * _length, because it holds both real and imaginary values.
That's why I only got half of the values right.
The correct code for it is:
double[] ComputeIFFT(ComplexArray input)
{
double[] temp;
ComplexArray output = new ComplexArray(_length);
_plan = Plan.Create1(_length, input, output, Direction.Backward, Options.Estimate);
_plan.Execute();
temp = output.ToArray();
for (int i = 0; i < temp.Length; ++i)
{
temp[i] /= _length;
}
return temp;
}
I hope this will be useful for anyone trying to implement the Hilbert Transform through FFTW in C#.
When I using Math.Exp() in C# I have some questions?This code is about Kernel density estimation, and I don't have any knowledge about kernel density estimation. So I look up some wiki and some paper.
I try to write it by C#. The problem is when "distance" is getting higher the result is become 0. It's confuse me and I cannot find any other way to get the right result.
disExp = Math.Pow(Math.E, -(distance / 2 * Math.Pow(h, 2)));
So, can any one help me to get the solution? Or give me some idea about Kernel density estimation on C#. Sorry for poor English.
Try this
public static double[,] KernelDensityEstimation(double[] data, double sigma, int nsteps)
{
// probability density function (PDF) signal analysis
// Works like ksdensity in mathlab.
// KDE performs kernel density estimation (KDE)on one - dimensional data
// http://en.wikipedia.org/wiki/Kernel_density_estimation
// Input: -data: input data, one-dimensional
// -sigma: bandwidth(sometimes called "h")
// -nsteps: optional number of abscis points.If nsteps is an
// array, the abscis points will be taken directly from it. (default 100)
// Output: -x: equispaced abscis points
// -y: estimates of p(x)
// This function is part of the Kernel Methods Toolbox(KMBOX) for MATLAB.
// http://sourceforge.net/p/kmbox
// Converted to C# code by ksandric
double[,] result = new double[nsteps, 2];
double[] x = new double[nsteps], y = new double[nsteps];
double MAX = Double.MinValue, MIN = Double.MaxValue;
int N = data.Length; // number of data points
// Find MIN MAX values in data
for (int i = 0; i < N; i++)
{
if (MAX < data[i])
{
MAX = data[i];
}
if (MIN > data[i])
{
MIN = data[i];
}
}
// Like MATLAB linspace(MIN, MAX, nsteps);
x[0] = MIN;
for (int i = 1; i < nsteps; i++)
{
x[i] = x[i - 1] + ((MAX - MIN) / nsteps);
}
// kernel density estimation
double c = 1.0 / (Math.Sqrt(2 * Math.PI * sigma * sigma));
for (int i = 0; i < N; i++)
{
for (int j = 0; j < nsteps; j++)
{
y[j] = y[j] + 1.0 / N * c * Math.Exp(-(data[i] - x[j]) * (data[i] - x[j]) / (2 * sigma * sigma));
}
}
// compilation of the X,Y to result. Good for creating plot(x, y)
for (int i = 0; i < nsteps; i++)
{
result[i, 0] = x[i];
result[i, 1] = y[i];
}
return result;
}
kernel density estimation C#
plot
I have this function which is SMA (Simple Moving Average). The result in the array I display as graph in ZedGraph and now it will start from 0 to 1956. I want the graph to start from frameSize / 2 in this case example it will be 300 / 2 so 150 so the graph should start from 150 to 2016.
I don't want to make the graph to grow I mean the array should stay length as 1956 I just want it to be pushed by 150 indexes from the beginning so it will start from index 150 instead of 0.
So this is the SMA function:
private static double[] smaDoubles(int frameSize, int[] data)
{
int padding = frameSize / 2;
double sum = 0;
double[] avgPoints = new double[(padding + data.Length) - frameSize + 1];
for (int counter = padding; counter <= data.Length - frameSize; counter++)
{
int innerLoopCounter = 0;
int index = counter;
while (innerLoopCounter < frameSize)
{
sum = sum + data[index];
innerLoopCounter += 1;
index += 1;
}
avgPoints[counter] = sum / frameSize;
sum = 0;
}
return avgPoints;
}
In the for loop counter = padding before it was counter = 0 so the result of that is in the image here.
The green one is the SMA from this function. And the green start from 150 but ends at 1956 and it should end at 2106. When I moved it to start from 150 I want the whole graph to move as one unit by 150 so it will start from 150 and end in 2106. The red graph should stay the same all
How can I do it?
Now as it is in the image the graph end by 300 from the right edge.
This is the function as it is now i changed the line: double[] avgPoints = new double[data.Length - frameSize + 1]; this is how it was original so i changed it to this one now.
And the function get frameSize as 3 and data as [10] and im getting the same exception:
private static double[] smaDoubles(int frameSize, int[] data)
{
int padding = frameSize / 2;
double sum = 0;
double[] avgPoints = new double[data.Length - frameSize + 1];
for (int counter = padding; counter <= data.Length - padding; counter++)//for (int counter = padding; counter <= data.Length - frameSize; counter++)
{
int innerLoopCounter = 0;
int index = counter;
while (innerLoopCounter < frameSize)
{
// if (index < data.Length)
sum = sum + data[index];
innerLoopCounter += 1;
index += 1;
}
avgPoints[counter] = sum / frameSize;
sum = 0;
}
return avgPoints;
}
Not possible* Some languages, like pascal allow this, but not c#.
Why not just subtract the offset 150:
sum += data[index - 150];
*although not possible with an array, you can achieve the effect with a custom object that implements an indexed property.
private int[] _array;
public int this[int index]
{
get{ return _array[index - 150]; }
}
Are you sure:
for (int counter = padding; counter <= data.Length - frameSize; counter++)
shouldn't be:
for (int counter = padding; counter <= data.Length - padding; counter++)
And then your function that computes the moving average should go from counter - padding to counter + padding instead of from counter to counter + frameSize.
To debug this type of problem, it's often helpful to try it which a much smaller data set where you can compute the expected result by hand and see if your algorithm matches your expectations. I don't believe that your algorithm is necessarily calculating what you think it's calculating here. Try it with 10 data elements and a window size of 3 to see if you're getting the results that you expect.
Note, the first code line actually contains 2 logic errors, one of which is not necessarily apparent until you try the code 2nd line. The error is preserved for illustrative purposes
I am trying to write a frequency program that will represent a bar diagram (in console code). The problem is I have no idea how exactly to calculate this frequency or how do I exactly then give the bars different heights according to their frequency (trough calculation).
The frequency height is capped at 21, meaning the bars go from 1 to 21, so the max bar height would be for example 21 stars (* as display sign for the bar itself).
A calculation I have so far (although not sure if correct) for frequency is the following, where this array takes the random values generated:
for (int j = 0; j < T.Length; j++)
{
T[j] = (MaxHeight* T[j]) / Ber.GreatestElement(T);
Console.Write("{0,7}", T[j]);
}
This results in values between 0 and 21. Based on the values my bars should give a certain height compared to all the other frequency values. For example, 8000 could have 21 in height where 39 could have 1).
To represent this diagram I used 2 for loops to display height and width (keep in mind I only wish to use using System; to keep it to the "basics").
for (int height= 1; height<= 21; height++)
{
for (int width= 0; width<= 10; width++)
{
if(...??)
{
Console.Write("{0,7}", bar); // string bar= ("*");
}
else
{
Console.Write("{0,7}", empty);
}
}
Console.WriteLine();
}
So far I have an entire field filled with * and the random values generated along with their frequency value (although I have no idea if the freq value is properly calculated).
I assume I need an if (...) in the second for but I cannot seem to get further than this.
There are some bits of your code that aren't really defined for us to analyze, but you could try a basic linear interpolation function to achieve interpolated values along a range (e.g. mapping 0->8000 to 0->21).
public static float MapToRange(float valueMeasured, float minMeasured, float maxMeasured, float minMapped, float maxMapped)
{
float mappedValue = minMapped + ((valueMeasured - minMeasured)/(maxMeasured - minMeasured)) * (maxMapped - minMapped);
return mappedValue;
}
So let's say you measured a minimum frequency of 450, a maximum of 8000, and you want to map all values to a range of 0 to 21. You could call it along the lines of this (assuming your current measurement is, say, 2700):
float mappedValue = MapToRange(2700, 450, 8000, 0, 21);
This would yield: 0 + ((2700 - 450)/(8000 - 450)) * (21 - 0) = 6.25827815
So cast this value as an int and draw 6 stars.
EDIT:
sorry I wrote in a hurry and my solution was wrong, bufferz wrote the correct one.
i.e. (in a less generic way)
int starsNum = (int)((currentValue - lowestValue)/(highestValue - lowestValue) * 21);
So, if you start with code like this, where T is the array of frequencies:
for (int j = 0; j < T.Length; j++)
{
T[j] = (MaxHeight* T[j]) / Ber.GreatestElement(T);
Console.Write("{0,7}", T[j]);
}
You would want to be able to take this code and represent it ina graph, correct?
To do so, you'll want to capture the largest T value, do a bit of math, then write the appropriate number of stars to the screen:
double max = 0.0;
for (int j = 0; j < T.Length; j++)
{
T[j] = (MaxHeight* T[j]) / Ber.GreatestElement(T);
if (T[j] > max) max = T[j];
}
Now that you have the max value, you can determine the number of stars through a SECOND for loop:
for (int j = 0; j < T.Length; j++)
{
int numStars = Convert.ToInt32((max / 21) * T[j]);
Console.Write("{0,7}", T[j]);
Console.WriteLine("".PadLeft(numStars, '*');
}
Hope that is what you're looking for.