Develop Reference

How to add method in unmanaged code in a FaceRecognition project

I'm implementing a little FaceRecognition program using Emgu as a wrapper of OpenCV libraries. It seems to work fine, but I need a function that returns all the distances between the image sample and the faces in the database (the FaceRecognizer.Predict method implemented only returns the smallest distance and label).
So I built Emgu from Git, in order to adapt functions in the unmanaged code (cvextern.dll) to my needs.
Here's the original in face_c.cpp
void cveFaceRecognizerPredict(cv::face::FaceRecognizer* recognizer, cv::_InputArray* image, int* label, double* dist)
{
int l = -1;
double d = -1;
recognizer->predict(*image, l, d);
*label = l;
*dist = d;
}
that stores minimum distance and corresponding label in l and d, thanks to predict.
The method I wrote, following the summary in opencv face.hpp:
void cveFaceRecognizerPredictCollector(cv::face::FaceRecognizer * recognizer, cv::_InputArray * image, std::vector<int>* labels, std::vector<double>* distances)
{
std::map<int, double> result_map = std::map<int, double>();
cv::Ptr<cv::face::StandardCollector> collector = cv::face::StandardCollector::create();
recognizer->predict(*image, collector);
result_map = collector->getResultsMap();
for (std::map<int, double>::iterator it = result_map.begin(); it != result_map.end(); ++it) {
distances->push_back(it->second);
labels->push_back(it->first);
}
}
And the caller in c#
using (Emgu.CV.Util.VectorOfInt labels = new Emgu.CV.Util.VectorOfInt())
using (Emgu.CV.Util.VectorOfDouble distances = new Emgu.CV.Util.VectorOfDouble())
using (InputArray iaImage = image.GetInputArray())
{
FaceInvoke.cveFaceRecognizerPredictCollector(_ptr, iaImage, labels, distances);
}
[DllImport(CvInvoke.ExternLibrary, CallingConvention = CvInvoke.CvCallingConvention)]
internal extern static void cveFaceRecognizerPredictCollector(IntPtr recognizer, IntPtr image, IntPtr labels, IntPtr distances);
The application works in real-time, so the function in c# is called continuously. I have only two faces and one label (same person) stored in my database, so the first call returns correctly the only possible label and stores it in labels. Keeping the application running, returned labels and the size of labels vector keep growing, filled with unregistered labels that I don't know where he takes. It seems to me like the collector in c++ is not well referenced, so that every time the function is called it keeps storing data without releasing the previous ones, overwriting them. But it's only my guess, I'm not very good with c++.
What else could possily be wrong?
Hope you can help

Display an image in real time as it is being created in a loop

In my code, the output is an image each pixel of which is determined using nested loops.
1) How can I force a window to open and show the output image as it is being constructed in the loop? (The window shows up when everything is finished. I don't want this.)
2) How can I have the output be displayed line by line (or even pixel by pixel) as the loop goes on. User must have the sense of getting the output in real-time.
outImage = new Image<Hsv, Byte>(numberOfColumns, numberOfRows);
byte[,,] pixelValue = outImage.Data;
for (int i = 0; i < numberOfRows - 1; i++)
{
for (int j = 0; j < numberOfColumns - 1; j++)
{
//pixelValue[i, j, k] is determined here using some other functions
imageBox1.Image = outImage; //too slow and impossible
}
}

You can display an image pixel by pixel in real time by putting it on a separate thread and using GetPixel and SetPixel. Keep in mind though that these methods are slow and if you are displaying high resolution pictures, it will take a while.
What you'll want to do is create a form with a picture box control on it. Next, create a Bitmap object containing the image you'll want to display. You can do this using a line like this:
Bitmap _bmp = new Bitmap(Image.FromFile(#"C:\MyImage.jpg"));
Next, in your form's shown event, spin off a new task to do the work, so the GUI doesn't lock up:
private void Form1_Shown(object sender, EventArgs e)
{
Task.Factory.StartNew(ShowImage);
}
This line will spin off a new thread every time the form is displayed. The thread will fork off and call ShowImage(), which should look like this:
private void ShowImage()
{
Graphics g = pbImage.CreateGraphics();
for (int x = 0; x < _bmp.Width; x++)
{
for (int y = 0; y < _bmp.Height; y++)
{
Color c = _bmp.GetPixel(x, y);
if (pbImage.InvokeRequired)
{
var x1 = x;
var y1 = y;
pbImage.BeginInvoke((Action) (() =>
{
g.FillRectangle(new SolidBrush(c), x1, y1, 1, 1);
}));
}
else
{
g.FillRectangle(new SolidBrush(c), x, y, 1, 1);
}
System.Threading.Thread.Sleep(1);
}
}
}
If you wanted to speed this up a bit, you could spin up two or more tasks, each task working in parallel (e.g. one thread starts at the beginning, another at the end, another in the middle maybe, etc). Just make sure your threads don't "overlap".
Another way to speed this up is to use pointers instead of GetPixel() and SetPixel(). You'd have to mark your code as unsafe though.

put your code in background Worker => Do Work
A separate thread would be initiated

I am not a WinForms expert I am more of a WPF type. But I have an application running a solid 30fps and that is faster than humans can detect. I really do not quite understand what you want to do here. You have to blit each pixel individually but have display in real time? An ImageBox derives from the Windows Forms PictureBox, that won't work I do not think.
You could try moving to WPF, and use a WriteableBitmap for a ImageSource for an Image object or the background of a Canvas Object. A WriteableBitmap will let you access each pixel, but the refresh rate is controlled by WPF and the monitor refresh rate is controlled by the AC current frequency.
Doug

NAudio fft result gives intensity on all frequencies C#

I have a working implementation of NAudio's wasapi loopback recording and the FFT of the data.
Most of the data I get is just as it should be but every once in a while (10 sec to minutes intervals) it shows amplitude on almost all frequencies.
Basicly the picture is rolling from right to left with time and frequencies going on logarithmic scale from lowest frequencies on the bottom. The lines are the errors. As far as i can tell those are not supposed to be there.
I get the audio buffer and send the samples to an aggregator (applies Hamming window) which implements the NAudio FFT. I have checked the data (FFT result) before I modify it in any way (the picture is not from the raw FFT output, but desibel scaled) confirming the FFT result is giving those lines. I could also point out the picture is modified with LockBits so I thought I had something wrong with the logic there, but that's why I checked the FFT output data which shows the same problem.
Well I could be wrong and the problem might be somewhere I said it isn't but it really seems it originates from the FFT OR the buffer data (data itself or the aggregation of samples). Somehow I doubt the buffer itself is corrupted like this.
If anyone has any idea what could cause this I would greatly appreciate it!
UPDATE
So I decided to draw the whole FFT result range rather than half of it. It showed something strange. I'm not sure of FFT but I thought Fourier transformation should give a result that is mirrored around the middle. This certainly is not the case here.
The picture is in linear scale so the exact middle of the picture is the middle point of the FFT result. Bottom is the first and top is the last.
I was playing a 10kHz sine wave which gives the two horizontal lines there but the top part is beyond me. It also seems like the lines are mirrored around the bottom quarter of the picture so that seems strange to me as well.
UPDATE 2
So I increased the FFT size from 4096 to 8192 and tried again. This is the output with me messing with the sine frequency.
It would seem the result is mirrored twice. Once in the middle and then again on the top and bottom halves. And the huge lines are now gone.. And it would seem like the lines only appear on the bottom half now.
After some further testing with different FFT lengths it seems the lines are completely random in that account.
UPDATE 3
I have done some testing with many things. The latest thing I added was overlapping of samples so that I reuse the last half of the sample array in the beginning of the next FFT. On Hamming and Hann windows it gives me massive intensities (quite like in the second picture I posted) but not with BlackmannHarris. Disabling overlapping removes the biggest errors on every window function. The smaller errors like in the top picture still remain even with BH window. I still have no idea why those lines appear.
My current form allows control over which window function to use (of the three previously mentioned), overlapping (on/off) and multiple different drawing options. This allows me to compare all the affecting parties effects when changed.
I shall investigate further (I am quite sure I have made a mistake at some point) but good suggestions are more than welcome!

The problem was in the way I handled the data arrays. Working like a charm now.
Code (removed excess and might have added mistakes):
// Other inputs are also usable. Just look through the NAudio library.
private IWaveIn waveIn;
private static int fftLength = 8192; // NAudio fft wants powers of two!
// There might be a sample aggregator in NAudio somewhere but I made a variation for my needs
private SampleAggregator sampleAggregator = new SampleAggregator(fftLength);
public Main()
{
sampleAggregator.FftCalculated += new EventHandler<FftEventArgs>(FftCalculated);
sampleAggregator.PerformFFT = true;
// Here you decide what you want to use as the waveIn.
// There are many options in NAudio and you can use other streams/files.
// Note that the code varies for each different source.
waveIn = new WasapiLoopbackCapture();
waveIn.DataAvailable += OnDataAvailable;
waveIn.StartRecording();
}
void OnDataAvailable(object sender, WaveInEventArgs e)
{
if (this.InvokeRequired)
{
this.BeginInvoke(new EventHandler<WaveInEventArgs>(OnDataAvailable), sender, e);
}
else
{
byte[] buffer = e.Buffer;
int bytesRecorded = e.BytesRecorded;
int bufferIncrement = waveIn.WaveFormat.BlockAlign;
for (int index = 0; index < bytesRecorded; index += bufferIncrement)
{
float sample32 = BitConverter.ToSingle(buffer, index);
sampleAggregator.Add(sample32);
}
}
}
void FftCalculated(object sender, FftEventArgs e)
{
// Do something with e.result!
}
And the Sample Aggregator class:
using NAudio.Dsp; // The Complex and FFT are here!
class SampleAggregator
{
// FFT
public event EventHandler<FftEventArgs> FftCalculated;
public bool PerformFFT { get; set; }
// This Complex is NAudio's own!
private Complex[] fftBuffer;
private FftEventArgs fftArgs;
private int fftPos;
private int fftLength;
private int m;
public SampleAggregator(int fftLength)
{
if (!IsPowerOfTwo(fftLength))
{
throw new ArgumentException("FFT Length must be a power of two");
}
this.m = (int)Math.Log(fftLength, 2.0);
this.fftLength = fftLength;
this.fftBuffer = new Complex[fftLength];
this.fftArgs = new FftEventArgs(fftBuffer);
}
bool IsPowerOfTwo(int x)
{
return (x & (x - 1)) == 0;
}
public void Add(float value)
{
if (PerformFFT && FftCalculated != null)
{
// Remember the window function! There are many others as well.
fftBuffer[fftPos].X = (float)(value * FastFourierTransform.HammingWindow(fftPos, fftLength));
fftBuffer[fftPos].Y = 0; // This is always zero with audio.
fftPos++;
if (fftPos >= fftLength)
{
fftPos = 0;
FastFourierTransform.FFT(true, m, fftBuffer);
FftCalculated(this, fftArgs);
}
}
}
}
public class FftEventArgs : EventArgs
{
[DebuggerStepThrough]
public FftEventArgs(Complex[] result)
{
this.Result = result;
}
public Complex[] Result { get; private set; }
}
And that is it I think. I might have missed something though.
Hope this helps!

More Precise timer than Stopwatch?

I'm trying to have a stopwatch start and stop when recording positions for the Kinect:
//process x and y coordinates
public void calculateJoints(Skeleton skeleton)
{
Joint rightHand = skeleton.Joints[JointType.HandRight];
Joint leftHand = skeleton.Joints[JointType.HandRight];
rightX = rightHand.Position.X;
rightY = rightHand.Position.Y;
}
//start the stopwatch (tried to use a greater time between positions 1 and 5 vs 1 and 2
public void processJointsAndRepeat(Skeleton skeleton)
{
startTime();
while (numPositions < 5)
{
calculateJoints(skeleton);
numPositions++;
}
stopTime();
double tempTime = calculateTimeElapsed();
}
//calculate time in milliseconds
private double calculateTimeElapsed()
{
long milliseconds = stopWatch.ElapsedMilliseconds;
return (double)milliseconds;
}
But whenever I try to put in the x, y, and time values with time as the key, it throws an error for duplicate keys. When I retrieved the value of tempTime, it only showed 0.
Is this a problem with my code, or do I need a more precise stopwatch?
I realize that getting a time for something that is 30 fps is difficult, so if you have any other suggestions, that'd be great! I'm basically just trying to calculate the average velocities between points to adjust the playback speed of an audio file. Thanks!

Stopwatch is wrapper around timer with higerst resolution on regular Windows box. You can use less fancy functions to get higer than MS resolution by using Stopwatch.ElapsedTicks and Stopwatch.Frequency.
Note that your problem is probably not related to timers but rather some other code you did not show...

Thread for my user control

I have a user control that shows a speed in a dial format (An image).
It has a single method: SetSpeed(int speed);
It then makes the dial move to the desired speed, from the last set speed. It does then in incriments. That is, it moves to the new speed.
Problem is, if the car is going 10km/h, and then goes (very quickly) to 100km/h, it takes the dial maybe 2 seconds to reach that speed... (It's not 100% real time - which is realistic .. it moves to the new speed).
But, if, before the dial gets to 100km/h, the person slows to 50km/h, I need to interupt the movement to 100, and start moving to 50. Regardless of the current position (not speed) of the dial, I need to change the 'target speed'.
My control is using a BackgroundWorker to handle the refreshing, and not lock the UI.
So, my call is very basic from the UI:
dial1.SetSpeed(int.Parse(value));
And then the user control does this:
BackgroundWorker bw = new BackgroundWorker();
public void SetSpeed(int speed)
{
while(bw.IsBusy)
{
Thread.Sleep(50);
}
bw.RunWorkerAsync(speed);
}
And the method that does the work:
private void UpdateSpeed(object sender, DoWorkEventArgs e)
{
var oldSpeed = airSpeed;
var newSpeed = (int) e.Argument;
if(oldSpeed <= newSpeed)
{
for (int i = oldSpeed; i < newSpeed; i++)
{
airSpeed++;
this.Invoke(new MethodInvoker(Refresh));
}
}
else
{
for (int i = oldSpeed; i > newSpeed; i--)
{
airSpeed--;
this.Invoke(new MethodInvoker(Refresh));
}
}
airSpeed = newSpeed;
}
It locks when I send it two values in quick succession...
How do I interrupt the thread, if it's running, and change the value?
(Also, I think my code to change the speed is bad - can that be made neater?)

You do not. You handle it in a proper way, with locks, and checking whether the value needs changing. You do NOT interrupt it.
Basically, you need a lock area between SetSpeed and the Refresh method, so that one blocks the other. Then, when you set speed and the thread is currently in a critical area, it simply waits until the update is finished.
And your UpdateSpeed makes no sense - the change (airspeed-- and airspeed++) should be timer driven... you currently change them in a "arbitrary" speed, depending on processor speed. No timing.