Develop Reference

NAudio FFT returns small and equal magnitude values for all frequencies

I'm working on a project with NAudio 1.9 and I want to compute an fft for an entire song, i.e split the song in chunks of equal size and compute fft for each chunk. The problem is that NAudio FFT function returns really small and equal values for any freq in the freq spectrum.
I searched for previous related posts but none seemed to help me.
The code that computes FFT using NAudio:
public IList<FrequencySpectrum> Fft(uint windowSize) {
IList<Complex[]> timeDomainChunks = this.SplitInChunks(this.audioContent, windowSize);
return timeDomainChunks.Select(this.ToFrequencySpectrum).ToList();
}
private IList<Complex[]> SplitInChunks(float[] audioContent, uint chunkSize) {
IList<Complex[]> splittedContent = new List<Complex[]>();
for (uint k = 0; k < audioContent.Length; k += chunkSize) {
long size = k + chunkSize < audioContent.Length ? chunkSize : audioContent.Length - k;
Complex[] chunk = new Complex[size];
for (int i = 0; i < chunk.Length; i++) {
//i've tried windowing here but didn't seem to help me
chunk[i].X = audioContent[k + i];
chunk[i].Y = 0;
}
splittedContent.Add(chunk);
}
return splittedContent;
}
private FrequencySpectrum ToFrequencySpectrum(Complex[] timeDomain) {
int m = (int) Math.Log(timeDomain.Length, 2);
//true = forward fft
FastFourierTransform.FFT(true, m, timeDomain);
return new FrequencySpectrum(timeDomain, 44100);
}
The FrequencySpectrum:
public struct FrequencySpectrum {
private readonly Complex[] frequencyDomain;
private readonly uint samplingFrequency;
public FrequencySpectrum(Complex[] frequencyDomain, uint samplingFrequency) {
if (frequencyDomain.Length == 0) {
throw new ArgumentException("Argument value must be greater than 0", nameof(frequencyDomain));
}
if (samplingFrequency == 0) {
throw new ArgumentException("Argument value must be greater than 0", nameof(samplingFrequency));
}
this.frequencyDomain = frequencyDomain;
this.samplingFrequency = samplingFrequency;
}
//returns magnitude for freq
public float this[uint freq] {
get {
if (freq >= this.samplingFrequency) {
throw new IndexOutOfRangeException();
}
//find corresponding bin
float k = freq / ((float) this.samplingFrequency / this.FftWindowSize);
Complex c = this.frequencyDomain[checked((uint) k)];
return (float) Math.Sqrt(c.X * c.X + c.Y * c.Y);
}
}
}
for a file that contains a sine wave of 440Hz
expected output: values like 0.5 for freq=440 and 0 for the others
actual output: values like 0.000168153987f for any freq in the spectrum

It seems that I made 4 mistakes:
1) Here I'm asumming that sampling freq is 44100. This was not the reason my code wasn't working, though
return new FrequencySpectrum(timeDomain, 44100);
2) Always make a visual representation of your output data! I must learn this lesson... It seems that for a file containing a 440Hz sine wave I'm getting the right result but...
3) The frequency spectrum is a little shifted from what I was expecting because of this:
int m = (int) Math.Log(timeDomain.Length, 2);
FastFourierTransform.FFT(true, m, timeDomain);
timeDomain is an array of size 44100 becaused that's the value of windowSize (I called the method with windowSize = 44100), but FFT method expects a window size with a value power of 2. I'm saying "Here, NAudio, compute me the fft of this array that has 44100 elements, but take into account only the first 32768". I didn't realize that this was going to have serious implications on the result:
float k = freq / ((float) this.samplingFrequency / this.FftWindowSize);
Here this.FftWindowSize is a property based on the size of the array, not on m. So, after visualizing the result I found out that magnitude of 440Hz freq was actually corresponding to the call:
spectrum[371]
instead of
spectrum[440]
So, my mistake was that the window size of fft (m) was not corresponding to the actual length of the array (FrequencySpectrum.FftWindowSize).
4) The small values that I was receiving for the magnitudes came from the fact that the audio file on which I was testing my code wasn't recorded with enough gain.

Writing and Reading a big file for analytical purposes

I'm trying to make a DNA Analytical tool, but I'm facing a big problem here.
Here's a screenshot on how the application looks like.
The problem I'm facing is handling large data. I've used streams and memory mapped files, but I'm not really sure if I'm heading in the right direction.
What I'm trying to achieve is to be able to write a text file with 3 billion random letters, and then use that text file for later purposes.
Currently i'm at 3000 letters, but generating more then that takes ages. How would you tackle this? Storing the full text file into a string seems like overload to me. Any ideas?
private void WriteDNASequence(string dnaFile)
{
Dictionary<int, char> neucleotides = new Dictionary<int, char>();
neucleotides.Add(0, 'A');
neucleotides.Add(1, 'T');
neucleotides.Add(2, 'C');
neucleotides.Add(3, 'G');
int BasePairs = 3000;
using (StreamWriter sw = new StreamWriter(filepath + #"\" + dnaFile))
{
for (int i = 0; i < (BasePairs / 2); i++)
{
int neucleotide = RandomNumber(0, 4);
sw.Write(neucleotides[neucleotide]);
}
}
}
private string ReadDNASequence(string dnaFile)
{
_DNAData = "";
using (StreamReader file = new StreamReader(filepath + #"\" + dnaFile))
{
_DNAData = file.ReadToEnd();
}
return _DNAData;
}
//Function to get a random number
private static readonly Random random = new Random();
private static readonly object syncLock = new object();
public static int RandomNumber(int min, int max)
{
lock (syncLock)
{ // synchronize
return random.Next(min, max);
}
}

When working with such big amount of data - every bit matters and you have to pack data as dense as possible.
As of now, each nucleotide is represented by one char, and one char in encoding you use (that's UTF-8 by default) takes 1 byte (for those 4 chars you use).
But since you have just 4 different characters - each character holds only 2 bits of information, so we can represent them as:
00 - A
01 - T
10 - C
11 - G
That means we can pack 4 nucleotides in one byte, making output file size 4 times smaller.
Assuming you have such map:
static readonly Dictionary<char, byte> _neucleotides = new Dictionary<char, byte> {
{ 'A', 0},
{ 'T', 1},
{ 'C', 2},
{ 'G', 3}
};
static readonly Dictionary<int, char> _reverseNucleotides = new Dictionary<int, char> {
{0, 'A'},
{1, 'T'},
{2, 'C'},
{3, 'G'}
};
You can pack 4 nucleotides like in one byte like this:
string toPack = "ATCG";
byte packed = 0;
for (int i = 0; i < 4; i++) {
packed = (byte) (packed | _neucleotides[toPack[i]] << (i * 2));
}
And unpack back like this:
string unpacked = new string(new[] {
_reverseNucleotides[packed & 0b11],
_reverseNucleotides[(packed & 0b1100) >> 2],
_reverseNucleotides[(packed & 0b110000) >> 4],
_reverseNucleotides[(packed & 0b11000000) >> 6],
});
As for writing bytes to file, I think that's easy enough. If you need some random data in this case, use:
int chunkSize = 1024 * 1024; // 8 million pairs at once (since each byte is 4 nucleotides)
byte[] chunk = new byte[chunkSize];
random.NextBytes(chunk);
// fileStream is instance of `FileStream`, no need for `StreamWriter`
fileStream.Write(chunk, 0, chunk.Length);
There are some caveats (like last byte in a file might store not 4 nucleotides but less), but I hope you'll figure that out yourself.
With that approach (packing in binary, generating big random chunk at once, writing big chunk to file) - generating 3 billion pairs took 8 seconds on my very old (7 years) HDD, and output size is 350MB. You can even read all that 350MB into memory at once if necessary.

Loading multi-dimensional array dynamically

I have the following code. It's roughly analogous in concept to the python reshape function. It successfully loads 1-dimensional data into a multi-dimensional array, the dimensions of which are not known until runtime. For example {209,64,64,3}. I have to iterate over the 1-dimensional data and create the correct indexes for each dimension of the array.
private void InitializeData()
{
var imageData = ImageData.Load(txtFileName.Text); // one dimensional array
var dimensions = txtDimensions.Text.Split(',').Select(d => int.Parse(d)).ToArray(); // e.g., {-1,64,64,3}
int elements = 1;
foreach (var dim in dimensions.Skip(1))
{
elements *= dim;
}
dimensions[0] = imageData.Length / elements; // {209,64,64,3}
// create multipliers
var multipliers = new int[dimensions.Length - 1];
for (var dimension = 1; dimension < dimensions.Length; dimension++)
{
var multiplier = 1;
for (var followingdimension = dimension; followingdimension < dimensions.Length; followingdimension++)
{
multiplier *= dimensions[followingdimension];
}
multipliers[dimension - 1] = multiplier;
}
// load data
var dataArray = Array.CreateInstance(typeof(int), dimensions);
var indexes = new int[dimensions.Length];
for (var imageDataIndex = 0; imageDataIndex < imageData.Length; imageDataIndex++)
{
indexes[0] = imageDataIndex / multipliers[0];
indexes[dimensions.Length - 1] = imageDataIndex % multipliers[multipliers.Length - 1];
for (var multiplier = 1; multiplier < dimensions.Length - 1; multiplier++)
indexes[multiplier] = (imageDataIndex / multipliers[multiplier]) % dimensions[multiplier];
dataArray.SetValue(imageData[imageDataIndex], indexes);
}
}
Is there a faster or more elegant way of doing this? I do realize those are two different things. I'll do bench-marking on the elegant suggestions, but I'd still like to see them. Because this is just too ugly to look at and was too painful to write to be the best way.
Note (Please)
The data may not always be image data, so I am not looking for bitmap operations. That just happens here but it's not necessarily a typical case. And, my goal is not to get a bitmap, but an array.

I have a partial answer thanks to How to reshape an Array in c#
The code can be replaced with just this:
var imageData = ImageData.Load(txtFileName.Text); // one dimensional array
// e.g., {209,64,64,3}
var dimensions = txtDimensions.Text.Split(',').Select(d => int.Parse(d)).ToArray();
int elements = 1;
foreach (var dim in dimensions.Skip(1))
{
elements *= dim;
}
dimensions[0] = imageData.Length / elements;
// load data
var dataArray = Array.CreateInstance(typeof(int), dimensions);
Buffer.BlockCopy(imageData, 0, dataArray, 0, imageData.Length * sizeof(int));
I would be surprised if there's a faster way to do the actual load then Buffer.BlockCopy, or a simpler one. It turns out whatever dimensional form your original data is in, BlockCopy handles it as long as you can specify your target dimensions as part of a target array.
I'll keep looking for ways to further refine the rest of the original code.

Optimization - Encode a string and get hexadecimal representation of 3 bytes

I am currently working in an environment where performance is critical and this is what I am doing :
var iso_8859_5 = System.Text.Encoding.GetEncoding("iso-8859-5");
var dataToSend = iso_8859_5.GetBytes(message);
The I need to group the bytes by 3 so I have a for loop that does this (i being the iterator of the loop):
byte[] dataByteArray = { dataToSend[i], dataToSend[i + 1], dataToSend[i + 2], 0 };
I then get an integer out of these 4 bytes
BitConverter.ToUInt32(dataByteArray, 0)
and finally the integer is converted to a hexadecimal string that I can place in a network packet.
The last two lines repeat about 150 times
I am currently hitting 50 milliseconds of execution times and ideally I would want to reach 0... Is there a faster way to do this that I am not aware of?
UPDATE
Just tried
string hex = BitConverter.ToString(dataByteArray);
hex.Replace("-", "")
to get the hex string directly but it is 3 times slower
Ricardo Silva's answer adapted
public byte[][] GetArrays(byte[] fullMessage, int size)
{
var returnArrays = new byte[(fullMessage.Length / size)+1][];
int i, j;
for (i = 0, j = 0; i < (fullMessage.Length - 2); i += size, j++)
{
returnArrays[j] = new byte[size + 1];
Buffer.BlockCopy(
src: fullMessage,
srcOffset: i,
dst: returnArrays[j],
dstOffset: 0,
count: size);
returnArrays[j][returnArrays[j].Length - 1] = 0x00;
}
switch ((fullMessage.Length % i))
{
case 0: {
returnArrays[j] = new byte[] { 0, 0, EOT, 0 };
} break;
case 1: {
returnArrays[j] = new byte[] { fullMessage[i], 0, EOT, 0 };
} break;
case 2: {
returnArrays[j] = new byte[] { fullMessage[i], fullMessage[i + 1], EOT, 0 };
} break;
}
return returnArrays;
}

After the line below you will get the total byte array.
var dataToSend = iso_8859_5.GetBytes(message);
My sugestion is work with Buffer.BlockCopy and test to see if this will be faster than your current method.
Try the code below and tell us if is faster than your current code:
public byte[][] GetArrays(byte[] fullMessage, int size)
{
var returnArrays = new byte[fullMessage.Length/size][];
for(int i = 0, j = 0; i < fullMessage.Length; i += size, j++)
{
returnArrays[j] = new byte[size + 1];
Buffer.BlockCopy(
src: fullMessage,
srcOffset: i,
dst: returnArrays[j],
dstOffset: 0,
count: size);
returnArrays[j][returnArrays[j].Length - 1] = 0x00;
}
return returnArrays;
}
EDIT1: I run the test below and the output was 245900ns (or 0,2459ms).
[TestClass()]
public class Form1Tests
{
[TestMethod()]
public void GetArraysTest()
{
var expected = new byte[] { 0x30, 0x31, 0x32, 0x00 };
var size = 3;
var stopWatch = new Stopwatch();
stopWatch.Start();
var iso_8859_5 = System.Text.Encoding.GetEncoding("iso-8859-5");
var target = iso_8859_5.GetBytes("012");
var arrays = Form1.GetArrays(target, size);
BitConverter.ToUInt32(arrays[0], 0);
stopWatch.Stop();
foreach(var array in arrays)
{
for(int i = 0; i < expected.Count(); i++)
{
Assert.AreEqual(expected[i], array[i]);
}
}
Console.WriteLine(string.Format("{0}ns", stopWatch.Elapsed.TotalMilliseconds * 1000000));
}
}
EDIT 2
I looked to your code and I have only one suggestion. I understood that you need to add EOF message and the length of input array will not be Always multiple of size that you want to break.
BUT, now the code below has TWO responsabilities, that break the S of SOLID concept.
The S talk about Single Responsability - Each method has ONE, and only ONE responsability.
The code you posted has TWO responsabilities (break input array into N smaller arrays and add EOF). Try think a way to create two totally independente methods (one to break an array into N other arrays, and other to put EOF in any array that you pass). This will allow you to create unit tests for each method (and guarantee that they Works and will never be breaked for any changed), and call the two methods from your class that make the system integration.

Predicting data based on averages

I have GPS data entering a serial port on my PC every second. I have successfully processed the GPS data and the latitude and longitude are stored in separate arrays as floating point numbers.
double[] dlat = new double[100000]; //contains the latitude data
double[] dlon = new double[100000]; //contains the longitude data
Most of the time the latitude and longitude numbers remain the same as the GPS position only changes every 5 meters. When the either the latitude or longitude value in the arrays changes I want my program to predict based on averages the latitude or longitude for the the data points stored in between the changes. For example:
Let's say this is the contents of the latitude array:
2,2,2,2,2,17
I would want my program to change what is in the array to:
2,5,8,11,14,17
I've tried tackling the problem but my method doesn't work :-/ I am new to C#; there must be a better way of doing this. Here is the snippet of my code that attempts to do the prediction (the bit after ---GPS coordinate prediction--- is the bit that doesn't work):
string RxString;// where the raw serial data is stored
string mag;
double[] dmag = new double[100000];//magnetic data stored here
string lat;
double[] dlat = new double[100000];//latitude data stored here
string lon;
double[] dlon = new double[100000];//longitude data stored here
double average;//average step between change in latiude
int i; //pointer double array data;
int count;//counter for prediction code
private void serialPort1_DataReceived(object sender, System.IO.Ports.SerialDataReceivedEventArgs e)//activates when port is open and data in buffer
{
RxString = serialPort1.ReadTo("\r\n");//read raw data from serial port into string
this.Invoke(new EventHandler(DisplayText));//invoke allows it to call function diplay text*/
if(RxString.StartsWith("G"))
{
lat = RxString.Split(',')[0].Substring(4);// extract latitude
this.Invoke(new EventHandler(DisplayText1));//invoke allows it to call function diplay text
dlat[i] = Convert.ToDouble(lat);//convert and store in double array
this.Invoke(new EventHandler(Form1_Load));//invoke allows it to call function
lon = RxString.Split(',')[2];// extract longitude
this.Invoke(new EventHandler(DisplayText2));//invoke allows it to call function diplay text
dlon[i] = Convert.ToDouble(lon);//covert and store in double array
this.Invoke(new EventHandler(Form1_Load));//invoke allows it to call function
mag = RxString.Split(',')[3].Substring(6).Trim();// extract magnetic data
this.Invoke(new EventHandler(DisplayText3));//invoke allows it to call function diplay text
dmag[i] = Convert.ToDouble(mag);//convert and store in double array
this.Invoke(new EventHandler(Form1_Load));//invoke allows it to call function
i++;
RxString = null;
/* -------------------------GPS coordinate prediction--------------------------------------------- */
if (i > 0)
{
if (dlat[i] == dlat[i - 1])
{
count++;
}
if (dlat[i] != dlat[i - 1])
{
double average = (dlat[i] - dlat[i - 1]) / (count);//average data step beween changed values
int firstAv = i - (count - 1);//position of first average
int lastAv = i - 1;//position of last average
for (int j = firstAv; j <= lastAv; i++)
{
dlat[j] = dlat[j - 1] + average;
}
count = 0;
}
}
if (i==0) count = 1;
}

The following works:
using System;
using System.Text;
namespace Practice
{
public class Hello
{
static double[] ldat = {2.0,2.0,2.00,2.0,2.0,17.0};
static double[] ldat2 = {2.0,3.0,4.00,4.0,7.0,19.0};
static double[] ldat3 = {0.0, 0.0, -5.0, -5.0, -11.0, -11.0, -20};
public static void Main(string[] args)
{
test(ldat);
test(ldat2);
test(ldat3);
}
public static void test(double[] array){
//Use Code from here.....
int firstEqualIndex = -1;
for(int i = 1; i < array.Length ; i ++)
{
if (i > 0)
{
if(array[i] == array[i - 1])
{
if(firstEqualIndex == -1)
{
firstEqualIndex = i - 1;
}
}
else //They are not equal
{
//Figure out the average.
if(firstEqualIndex >= 0)
{
double average = (array[i] - array[firstEqualIndex]) / (Double)((i - firstEqualIndex));
int k = 0;
for(int j = firstEqualIndex; j < i; j++)
{
array[j] += average * k;
k++;
}
firstEqualIndex = -1;
}
}
}
}
//..... to here.
StringBuilder builder = new StringBuilder();
foreach (double entry in array)
{
// Append each int to the StringBuilder overload.
builder.Append(entry).Append(", ");
}
string result = builder.ToString();
Console.WriteLine(result);
}
}
}
This test outputs
2, 5, 8, 11, 14, 17,
2, 3, 4, 5.5, 7, 19,
0, -2.5, -5, -8, -11, -15.5, -20,
Sorry about all the edits I am trying to make sure that the method works with additional test cases.
EDIT: Added a test for negative case.

I would formulate this problem in terms of signal processing. So if you have a signal f(t) (which could be your discretized latitude array for example), you want to create a new signal g(t) defined by
g(t) = E[f(z) | t-0.5*w <= z <= t+0.5*w]
where E is denoting the expected value (or average) and w is the width of your filter.
One of the benefits of modeling the problem this way is that you have a very concrete way of specifying your movement model. That is, how are you going to transform the data [0, 0, 0, 0, 1, 1, 1, 1]?
Should it be [0, 0, 0, 1/3, 2/3, 1, 1, 1]?
Or should it be [0, 1/7, 2/7, 3/7, 4/7, 5/7, 6/7, 1]?
Given that you know how much time passes between your samples, you can choose a w duration that specifies the model you want.
Another benefit is that, if you want a nonlinear movement model, you can easily extend to that too. In the example I gave above, I used a box filter to do the smoothing, but you could use something else to take into account the physical limitations on the acceleration/deceleration of whatever it is you're tracking. A filter shaped more like a Gaussian curve could accomplish that.