Detecting open fist and close fist - c#

I tried following the algorithm but it does not work. I can't figure what is the problem.
Can somebody help me?
Where can I learn/find examples of gesture recognitions streamed from Kinect, using OpenCV?
Image<Gray, Byte> dest = new Image<Gray, Byte>(this.bitmap.Width, this.bitmap.Height);
CvInvoke.cvThreshold(src, dest, 220, 300, Emgu.CV.CvEnum.THRESH.CV_THRESH_BINARY);
Bitmap nem1 = new Bitmap(dest.Bitmap);
this.bitmap = nem1;
Graphics g = Graphics.FromImage(this.bitmap);
using (MemStorage storage = new MemStorage()) //allocate storage for contour approximation
{
for (Contour<Point> contours = dest.FindContours();
contours != null;
contours = contours.HNext)
{
g.DrawRectangle(new Pen(new SolidBrush(Color.Green)),contours.BoundingRectangle);
IntPtr seq = CvInvoke.cvConvexHull2(contours,storage.Ptr, Emgu.CV.CvEnum.ORIENTATION.CV_CLOCKWISE, 0);
IntPtr defects = CvInvoke.cvConvexityDefects(contours, seq, storage);
Seq<Point> tr= contours.GetConvexHull(Emgu.CV.CvEnum.ORIENTATION.CV_CLOCKWISE);
Seq<Emgu.CV.Structure.MCvConvexityDefect> te = contours.GetConvexityDefacts(storage, Emgu.CV.CvEnum.ORIENTATION.CV_CLOCKWISE);
g.DrawRectangle(new Pen(new SolidBrush(Color.Green)), tr.BoundingRectangle);
}
}

Without having some graphical data it's hard to help (I'm also without proper hardware). Anyway, I suggest you two things:
since it's a graphical procedure, debug everything saving or showing any intermediate step (threshold, contours, convexhull)
change to a simpler approach. For example:
apply a threshold (resulting in a 0/1 map of your hands)
for each row, count 0/1 transitions
test the maximum number of transitions: i.e., if it's above 7, hands are open
Let me know if it works :-)

Related

C# .Net Emgu.CV is inconsistently transferring bytes to itself

I'm using Emgu.CV to templateMatch and to save Images.
Unfortunetly I have ran into an issue that I have no been able to solve for a weeks.
Problem is that i serialize byte array and size from original Image to json file, and whenever i try to convert it back sometimes the image is distorted.
I have already tried skipping over serializing procces and it still became distorted.
Here is code of converting procces:
Image<Bgr565, byte> screenCrop = SnipMaker.takeSnip();//method creates screenshot at this point when i display the images they are 100% correct
byte[] data = screenCrop.Bytes;//I would get normaly all this from json file(in this case im skipping over it)
Mat mat = new Mat(screenCrop.Rows, screenCrop.Cols, screenCrop.Mat.Depth, asset.NumberOfChannels);
Marshal.Copy(data, 0, mat.DataPointer, screenCrop.asset.Cols * screenCrop.asset.Rows * asset.NumberOfChannels);
Image<Bgr565, byte> img = mat.ToImage<Bgr565, byte>();//This image is suddenly distorted
Problem is that this results depending on "I'm not sure what" is either prefecly good image or skwed one:
normal result
same code different result
Its almost like its sometimes 1 pixel behind but only thing that is changing is size and dimentions of screen shots.
I have tried dirrect ways like
Image<Bgr, byte> img = new Image<Bgr, byte>(width, height);
img.Bytes = data;//data is byte array that i got from file
This also gives sometimes correct picture but other times it throws an exeption (out of range exception in marshal.cs when trying to copy bytes from data to img)
only thing that i suspect at this point is that im doing something wrong whenever im taking screenshot but im not sure what:
public static Image<Bgr565, byte> Snip()
{
int screenWidth = (int)System.Windows.SystemParameters.PrimaryScreenWidth;
int screenHeight = (int)System.Windows.SystemParameters.PrimaryScreenHeight;
using (Bitmap bmp = new Bitmap(screenWidth, screenHeight))
{
using (Graphics gr = Graphics.FromImage(bmp))
gr.CopyFromScreen(0, 0, 0, 0, bmp.Size);
using (var snipper = new SnippingTool(bmp))
{
if (snipper.ShowDialog() == true)
{
Bitmap bitmapImage = new Bitmap(snipper.Image);
Rectangle rectangle = new Rectangle(0, 0, bitmapImage.Width, bitmapImage.Height);//System.Drawing
BitmapData bmpData = bitmapImage.LockBits(rectangle, ImageLockMode.ReadWrite, System.Drawing.Imaging.PixelFormat.Format24bppRgb);//System.Drawing.Imaging
Image<Bgr565, byte> outputImage = new Image<Bgr565, byte>(bitmapImage.Width, bitmapImage.Height, bmpData.Stride, bmpData.Scan0);
bitmapImage.Dispose();
snipper.Close();
return outputImage;
}
}
return null;
}
}
So far I have not been able to solve this and knowing my luck noone will proppably anwser me here. But please could someone help me with this?
Thank you in advance
So thank you to everyones help.
The issue was indeed in the screenshot script. I've used incorrect combination of
pixel formats which resulted in inconsistent bit transfer.
But because the step property in Image<bgr,byte>.Mat was calculated based on the width of the image (Emgucv SC):
step = sizeof(byte) * s.Width * channels;
It caused that some of the images looked normal and other didn't.(speculation based on observation)
Fix:
change all Image<Bgr, byte> to Image<Bgra, byte>
to make it 32bit and then change:
BitmapData bmpData = bitmapImage.LockBits(rectangle, ImageLockMode.ReadWrite, System.Drawing.Imaging.PixelFormat.Format24bppRgb);
to:
BitmapData bmpData = bitmapImage.LockBits(rectangle, ImageLockMode.ReadWrite, System.Drawing.Imaging.PixelFormat.Format32bppRgb);
Hope this will help someone in the future. : )

Shift Emgu.CV.Image to the right using WarpAffine()

my goal is, to shift an Image an x-amount of pixels to the right. I guess this can be achieved by using WarpAffine. This is at least what my reseach tells me. I used quite a variety of different approaches, like:
CvInvoke.WarpAffine(realImage, transformedImage, transformationMatrix, new Size(realImage.Size);
//or while m is the Mat used to create realImage
transImage.WarpAffine(m,Emgu.CV.CvEnum.Inter.Area,Emgu.CV.CvEnum.Warp.Default,Emgu.CV.CvEnum.BorderType.Default,new Gray());
I get the Exeption:
Exception thrown: 'Emgu.CV.Util.CvException' in Emgu.CV.Platform.NetStandard.dll
[ WARN:0] global E:\bb\cv_x86\build\opencv\modules\videoio\src\cap_msmf.cpp (436) `anonymous-namespace'::SourceReaderCB::~SourceReaderCB terminating async callback
I guess I am using it the wrong way, but there is no suiting example online for me to learn from.
Does anyone has a clean way to explain it to me?
Thank you in advance!
If you are shifting the pixels x amount to the right, I assume that there would be black empty pixels on the left side? If so, you could create an ROI and cut off some pixels on the right, since you are shifting all pixels to the right, and copy the image onto another image.
//The image that you want to shift pixels with
Image<Bgr, byte> inputImage = new Image<Bgr, byte>(1000, 1000);
//The output image
Image<Bgr, byte> image = new Image<Bgr, byte>(990, 1000);
//Create the roi, with 10 pixels cut off from the right side because of the shift
inputImage.ROI = new Rectangle(0, 0, inputImage.Width - 10, inputImage.Height);
inputImage.CopyTo(image);
CvInvoke.ImShow("The Output", image);
CvInvoke.WaitKey(0);
EDIT
Now lets say you want to keep that black stripe on the left side of the image as well. Doing this is very similar to the code above, but only with a few modifications.
//The image that you want to shift pixels with
Image<Bgr, byte> inputImage = new Image<Bgr, byte>(1000, 1000);
//The output image
Image<Bgr, byte> image = new Image<Bgr, byte>(1000, 1000);
//Create the roi, with 10 pixels cut off from the right side because of the shift
inputImage.ROI = new Rectangle(0, 0, inputImage.Width - 10, inputImage.Height);
//The image we want to shift, the same one we created a ROI with,
//has the dimensions 990 X 1000 and the output image has
//dimensions of 1000 x 1000. Unfortunately, in order to paste
//an image onto another image, they need to be the same dimensions.
//How do we do this? We must create an ROI with the output image
//that has the same dimensions as the input image.
image.ROI = new Rectangle(10, 0, image.Width, image.Height);
//Now we can past the image onto the output because the dimensions match
inputImage.CopyTo(image);
//Inorder to make our output seem normal, we must empty the ROI of the output image
image.ROI = Rectangle.Empty;
CvInvoke.ImShow("The Output", image);
CvInvoke.WaitKey(0);

How to detect number of objects from image?

I have a windows forms application and i want to count number of objects from a medical images. For instance
I used an algorithm which can take the contours of every object from the image.
private void findContoursToolStripMenuItem_Click(object sender, EventArgs e)
{
Image<Gray, byte> imgOutput = imgInput.Convert<Gray, byte>().ThresholdBinary(new Gray(100), new Gray(255));
Emgu.CV.Util.VectorOfVectorOfPoint contours = new Emgu.CV.Util.VectorOfVectorOfPoint();
Mat hier = new Mat();
Image<Gray, byte> imgout = new Image<Gray, byte>(imgInput.Width, imgInput.Height, new Gray(100));
CvInvoke.FindContours(imgOutput, contours, hier, Emgu.CV.CvEnum.RetrType.External, Emgu.CV.CvEnum.ChainApproxMethod.ChainApproxSimple);
CvInvoke.DrawContours(imgout, contours, -1, new MCvScalar(123, 0, 0));
pictureBox2.Image = imgout.Bitmap;
}
But I can't find out the number of cells from the image. Any advice or algorithm I have to use ?
I search within EMGU documentation but I don't find any function which does somethink like I want.
Any advice or answer will be rewarded.
If you consider that is too broad, I don't want the implemented algorithm. I just need some ideas or a suggestion of algorithm i have to use.
It probably a bit basic and brute force, but how about selecting a random point on the image that is close to the green colour, then effectively search for 'matching' colours (with a tolerance for the same colour. As you visit each pixel, colour it black so you don't look at it again and count how many pixels you have coloured in. Each time you select a pixel, make sure it's not black. Once you can't find any more points, if the number of black pixels is greater than a tolerance (so you only find 'big' polygons), then count it in the number of cells.

How to convert Bitmap to Mat structur in EmguCV & How to detect two images shift

Hello Dear Forum Members !
I am working on a project to detect change view from security camera. I mean, when someone try to move camera (some kind of sabotage...) I have to notice this. My idea is:
capture images from camera every 10 sec and compare this two pictures ( Old and actual picture).
There are almost 70 cameras which I need to control, so I can't use live streaming because it could occupy my internet connection. I use Emgu CV library to make this task, but during my work I stuck with some problem.. Here im piece of my code what I prepared:
public class EmguCV
{
static public Model Test(string BaseImagePath, string ActualImagePath)
{
double noise = 0;
Mat curr64f = new Mat();
Mat prev64f = new Mat();
Mat hann = new Mat();
Mat src1 = CvInvoke.Imread(BaseImagePath, 0);
Mat src2 = CvInvoke.Imread(ActualImagePath, 0);
Size size = new Size(50, 50);
src1.ConvertTo(prev64f, Emgu.CV.CvEnum.DepthType.Cv64F);
src2.ConvertTo(curr64f, Emgu.CV.CvEnum.DepthType.Cv64F);
CvInvoke.CreateHanningWindow(hann, src1.Size, Emgu.CV.CvEnum.DepthType.Cv64F);
MCvPoint2D64f shift = CvInvoke.PhaseCorrelate(curr64f, prev64f, hann, out noise );
double value = noise ;
double radius = Math.Sqrt(shift.X * shift.X + shift.Y * shift.Y);
Model Test = new Model() { osX = shift.X, osY = shift.Y, noise = value };
return Test;
}
}
Therefore, I have two questions:
How to convert Bitmap to Mat structure.
At the moment I read my images to compare from disc according to file path. But I would like to send to compare collection of bitmaps without saving on my hard drive.
Do you know any another way to detect shift between two pictures ?. I would be really grateful for any other suggestion in this area.
Regards,
Mariusz
I know its very late to answer this but today I was looking for this problem in the internet and I found something like this:
Bitmap bitmap; //This is your bitmap
Image<Bgr, byte> imageCV = new Image<Bgr, byte>(bitmap); //Image Class from Emgu.CV
Mat mat = imageCV.Mat; //This is your Image converted to Mat

Emgucv Convert<Hsv, Byte>() image

I am having a problem with EmguCV. I used a demo application, and edited it to my needs.
It involves the following function:
public override Image<Gray, byte> DetectSkin(Image<Bgr, byte> Img, IColor min, IColor max)
{
Image<Hsv, Byte> currentHsvFrame = Img.Convert<Hsv, Byte>();
Image<Gray, byte> skin = new Image<Gray, byte>(Img.Width, Img.Height);
skin = currentHsvFrame.InRange((Hsv)min,(Hsv)max);
return skin;
}
In the demo application, the Image comes from a video. The frame is capured from the video like this:
Image<Bgr, Byte> currentFrame;
grabber = new Emgu.CV.Capture(#".\..\..\..\M2U00253.MPG");
grabber.QueryFrame();
currentFrame = grabber.QueryFrame();
In my application, the Image comes from a microsoft kinect stream.
I use the following function:
private void SensorColorFrameReady(object sender, ColorImageFrameReadyEventArgs e)
{
using (ColorImageFrame colorFrame = e.OpenColorImageFrame())
{
if (colorFrame != null)
{
// Copy the pixel data from the image to a temporary array
colorFrame.CopyPixelDataTo(this.colorPixels);
// Write the pixel data into our bitmap
this.colorBitmap.WritePixels(
new Int32Rect(0, 0, this.colorBitmap.PixelWidth, this.colorBitmap.PixelHeight),
this.colorPixels,
this.colorBitmap.PixelWidth * sizeof(int),
0);
Bitmap b = BitmapFromWriteableBitmap(this.colorBitmap);
currentFrame = new Image<Bgr, byte>(b);
currentFrameCopy = currentFrame.Copy();
skinDetector = new YCrCbSkinDetector();
Image<Gray, Byte> skin = skinDetector.DetectSkin(currentFrame, YCrCb_min, YCrCb_max);
}
}
}
private static System.Drawing.Bitmap BitmapFromWriteableBitmap(WriteableBitmap writeBmp)
{
System.Drawing.Bitmap bmp;
using (System.IO.MemoryStream outStream = new System.IO.MemoryStream())
{
BitmapEncoder enc = new BmpBitmapEncoder();
enc.Frames.Add(BitmapFrame.Create((BitmapSource)writeBmp));
enc.Save(outStream);
bmp = new System.Drawing.Bitmap(outStream);
}
return bmp;
}
Now, the demo application works, and mine doesn't. Mine gives the following exception:
And, the image here, contains the following:
I really don't understand this exception. And, now, when I run the demo, working aplication, the image, contains:
Which is, in my eyes, exactly the same. I really don't understand this. Help is very welcome!
To make things easier I've uploaded a working WPF solution for you to the code reference sourceforge page I've been building:
http://sourceforge.net/projects/emguexample/files/Capture/Kinect_SkinDetector_WPF.zip/download
https://sourceforge.net/projects/emguexample/files/Capture/
This was designed and tested using EMGU x64 2.42 so in the Lib folder of the project you will find the referenced dlls. If you are using a different version you will need to delete the current references and replace them with the version you're using.
Secondly the project is design like all projects from the code reference library to be built from the Emgu.CV.Example folder into the ..\EMGU 2.X.X.X\bin.. global bin directory where the opencv compiled libraries are within a folder either x86 or x64.
If you struggle to get the code working I can provide all components but I hate redistributing all the opencv files that you already have so let me know if you want this.
You will need to resize the Mainwindow manually to display both images as I didn't spend to much time playing with layout.
So the code...
In the form initialisation method I check for the kinect sensor and set up the eventhandlers for the frames ready. I have left the original threshold values and skinDetector type although I don't use the EMGU version I just forgot to remove it. You will need to play with the threshold values and so on.
//// Look through all sensors and start the first connected one.
//// This requires that a Kinect is connected at the time of app startup.
//// To make your app robust against plug/unplug,
//// it is recommended to use KinectSensorChooser provided in Microsoft.Kinect.Toolkit (See components in Toolkit Browser).
foreach (var potentialSensor in KinectSensor.KinectSensors)
{
if (potentialSensor.Status == KinectStatus.Connected)
{
this.KS = potentialSensor;
break;
}
}
//If we have a Kinect Sensor we will set it up
if (null != KS)
{
// Turn on the color stream to receive color frames
KS.ColorStream.Enable(ColorImageFormat.RgbResolution640x480Fps30);
//Turn on the depth stream to recieve depth frames
KS.DepthStream.Enable(DepthImageFormat.Resolution640x480Fps30);
//Start the Streaming process
KS.Start();
//Create a link to a callback to deal with the frames
KS.AllFramesReady += new EventHandler<AllFramesReadyEventArgs>(KS_AllFramesReady);
//We set up a thread to process the image/disparty map from the kinect
//Why? The kinect AllFramesReady has a timeout if it has not finished the streams will simply stop
KinectBuffer = new Thread(ProcessBuffer);
hsv_min = new Hsv(0, 45, 0);
hsv_max = new Hsv(20, 255, 255);
YCrCb_min = new Ycc(0, 131, 80);
YCrCb_max = new Ycc(255, 185, 135);
detector = new AdaptiveSkinDetector(1, AdaptiveSkinDetector.MorphingMethod.NONE);
skinDetector = new YCrCbSkinDetector();
}
I always play with the kinect data in a new thread for speed but you may want to advanced this to a Background worker if you plan to do any more heavy processing so it is better managed.
The thread calls the ProcessBuffer() method you can ignore all the commented code as this is the remanence of the code used to display the depth image. Again I'm using the Marshall copy method to keep things fast but the thing to look for is the Dispatcher.BeginInvoke in WPF that allows the images to be displayed from the Kinect thread. This is required as I'm not processing on the main thread.
//This takes the byte[] array from the kinect and makes a bitmap from the colour data for us
byte[] pixeldata = new byte[CF.PixelDataLength];
CF.CopyPixelDataTo(pixeldata);
System.Drawing.Bitmap bmap = new System.Drawing.Bitmap(CF.Width, CF.Height, System.Drawing.Imaging.PixelFormat.Format32bppRgb);
BitmapData bmapdata = bmap.LockBits(new System.Drawing.Rectangle(0, 0, CF.Width, CF.Height), ImageLockMode.WriteOnly, bmap.PixelFormat);
IntPtr ptr = bmapdata.Scan0;
Marshal.Copy(pixeldata, 0, ptr, CF.PixelDataLength);
bmap.UnlockBits(bmapdata);
//display our colour frame
currentFrame = new Image<Bgr, Byte>(bmap);
Image<Gray, Byte> skin2 = skinDetector.DetectSkin(currentFrame, YCrCb_min, YCrCb_max);
ExtractContourAndHull(skin2);
DrawAndComputeFingersNum();
//Display our images using WPF Dispatcher Invoke as this is a sub thread.
Dispatcher.BeginInvoke((Action)(() =>
{
ColorImage.Source = BitmapSourceConvert.ToBitmapSource(currentFrame);
}), System.Windows.Threading.DispatcherPriority.Render, null);
Dispatcher.BeginInvoke((Action)(() =>
{
SkinImage.Source = BitmapSourceConvert.ToBitmapSource(skin2);
}), System.Windows.Threading.DispatcherPriority.Render, null);
I hope this helps I will at some point neaten up the code I uploaded,
Cheers

Categories

Resources