Develop Reference

OpenCV FindContours does not find the correct rectangle

Using CvInvoke.Canny and CvInvoke.FindContours I'm trying to find the rectangle containing the item name (Schematic: Maple Desk). This rectangle is shown in the image below:
I'm able to find a lot of rectangles but I'm not able to get this one. Tried a lot of different thresholds for Canny but to no effect. The following image shows all rectangles I currently get:
Any ideas how to tackle this? Do I need to use other thresholds or another approach? I already experimented using grayscale and blurring but that didn't give better result. I added my current source below and the original image I'm using is this:
public Mat ProcessImage(Mat img)
{
UMat filter = new UMat();
UMat cannyEdges = new UMat();
Mat rectangleImage = new Mat(img.Size, DepthType.Cv8U, 3);
//Convert the image to grayscale and filter out the noise
//CvInvoke.CvtColor(img, filter, ColorConversion.Bgr2Gray);
//Remove noise
//CvInvoke.GaussianBlur(filter, filter, new System.Drawing.Size(3, 3), 1);
// Canny and edge detection
double cannyThreshold = 1.0; //180.0
double cannyThresholdLinking = 1.0; //120.0
//CvInvoke.Canny(filter, cannyEdges, cannyThreshold, cannyThresholdLinking);
CvInvoke.Canny(img, cannyEdges, cannyThreshold, cannyThresholdLinking);
// Find rectangles
List<RotatedRect> rectangleList = new List<RotatedRect>();
using (VectorOfVectorOfPoint contours = new VectorOfVectorOfPoint())
{
CvInvoke.FindContours(cannyEdges, contours, null, RetrType.List, ChainApproxMethod.ChainApproxSimple);
int count = contours.Size;
for (int i = 0; i < count; i++)
{
using (VectorOfPoint contour = contours[i])
using (VectorOfPoint approxContour = new VectorOfPoint())
{
CvInvoke.ApproxPolyDP(contour, approxContour, CvInvoke.ArcLength(contour, true) * 0.05, true);
// Only consider contours with area greater than 250
if (CvInvoke.ContourArea(approxContour, false) > 250)
{
// The contour has 4 vertices.
if (approxContour.Size == 4)
{
// Determine if all the angles in the contour are within [80, 100] degree
bool isRectangle = true;
System.Drawing.Point[] pts = approxContour.ToArray();
LineSegment2D[] edges = Emgu.CV.PointCollection.PolyLine(pts, true);
for (int j = 0; j < edges.Length; j++)
{
double angle = Math.Abs(edges[(j + 1) % edges.Length].GetExteriorAngleDegree(edges[j]));
if (angle < 80 || angle > 100)
{
isRectangle = false;
break;
}
}
if (isRectangle) rectangleList.Add(CvInvoke.MinAreaRect(approxContour));
}
}
}
}
}
// Draw rectangles
foreach (RotatedRect rectangle in rectangleList)
{
CvInvoke.Polylines(rectangleImage, Array.ConvertAll(rectangle.GetVertices(), System.Drawing.Point.Round), true,
new Bgr(Color.DarkOrange).MCvScalar, 2);
}
//Drawing a light gray frame around the image
CvInvoke.Rectangle(rectangleImage,
new Rectangle(System.Drawing.Point.Empty,
new System.Drawing.Size(rectangleImage.Width - 1, rectangleImage.Height - 1)),
new MCvScalar(120, 120, 120));
//Draw the labels
CvInvoke.PutText(rectangleImage, "Rectangles", new System.Drawing.Point(20, 20),
FontFace.HersheyDuplex, 0.5, new MCvScalar(120, 120, 120));
Mat result = new Mat();
CvInvoke.VConcat(new Mat[] { img, rectangleImage }, result);
return result;
}
Edit 1:
After some more fine tuning with the following thresholds for Canny
cannyThreshold 100
cannyThresholdLinking 400
And using a minimum size for all ContourAreas of 10000 I can get the following result:
Edit 2:
For those interested, solved using the current detection, no changes were needed. Used the 2 detected rectangles in the screenshot above to get the location of the missing rectangle containing the item name.
Result can be found here:
https://github.com/josdemmers/NewWorldCompanion

For those interested, solved using the current detection, no changes were needed. Used the 2 detected rectangles in the screenshot above to get the location of the missing rectangle containing the item name.
Result can be found here: https://github.com/josdemmers/NewWorldCompanion

EmguCv EigenObjectRecognizer creating instance

My application gets closed when it reaches at creating the recognizer object of EigenObjectRecognizer class without giving any error or warning , am I passing wrong parameters or there is some other problem ? Here is my code
string[] allFaces = Directory.GetFiles(savepath);
if (allFaces != null)
{
Image<Gray, Byte>[] trainingImages = new Image<Gray, Byte>[allFaces.Length];
string[] labels = new String[allFaces.Length];
for (int i = 0; i < allFaces.Length; i++)
{
trainingImages[i] = new Image<Gray, byte>(new Bitmap(allFaces[i]));
labels[i] = allFaces[i].Substring(allFaces[i].LastIndexOf("\\")+1);
}
MCvTermCriteria termCrit = new MCvTermCriteria(allFaces.Length, 0.001);
EigenObjectRecognizer recognizer = new EigenObjectRecognizer(
trainingImages,
labels,
1000,
ref termCrit);
Image<Gray, Byte> testImage = new Image<Gray, Byte>(#"C:\..test\1");
string label = recognizer.Recognize(testImage).Label;
MessageBox.Show(label);
}

Solved the problem by getting an error from raw run of the compiled exe which gave an error of input of not same size from opencv instead of emgucv . When I looked at my training images they were of different size. Hope it helps others

Additional information: OpenCV: Different sizes of objects using c#

Currently, I am facing problem with my EmguCV c# code. I am trying to recognize my images from the database, but its not working. Once my face is detected, its crashing and then this error shows up
Additional information: OpenCV: Different sizes of objects.
I tried searching for this error but I am clueless.
This is my code:
//Action for each element detected
foreach (MCvAvgComp f in facesDetected[0])
{
t = t + 1;
result = currentFrame.Copy(f.rect).Convert<Gray, byte>().Resize(100, 100, Emgu.CV.CvEnum.INTER.CV_INTER_CUBIC);
//draw the face detected in the 0th (gray) channel with blue color
currentFrame.Draw(f.rect, new Bgr(Color.Green), 2);
//Database select the image row and pass to the eigenobjectrecognizer
//ConnectToDatabase();
if (Connection.State.Equals(ConnectionState.Open))
{
Connection.Close();
TSTable.Clear();
ConnectToDatabase();
}
//Connection.Open();
OleDbCommand OledbSelect = new OleDbCommand("Select FaceName, FaceImage From TrainingSet1",Connection);
OleDbDataReader reader = OledbSelect.ExecuteReader();
while (reader.Read())
{
labels.Add(reader.GetValue(1).ToString());
trainingImages.Add(gray);
}
if (TSTable.Rows.Count != 0)
{
//TermCriteria for face recognition with numbers of trained images like maxIteration
MCvTermCriteria termCrit = new MCvTermCriteria(ContTrain, 0.001);
//Eigen face recognizer
EigenObjectRecognizer recognizer = new EigenObjectRecognizer(
trainingImages.ToArray(), //database faceimage list
labels.ToArray(), //facename list
3000,
ref termCrit);
name = recognizer.Recognize(result);
//Draw the label for each face detected and recognized
currentFrame.Draw(name, ref font, new Point(f.rect.X - 2, f.rect.Y - 2), new Bgr(Color.LightGreen));
}
I currently still new to EmguCV and C#. So some of the exceptions I don't understand. Can anyone help me with this.
And once the code breaks, it goes to the EigenObjectRecognizer.cs. This is the code where it breaks:
public static float[] EigenDecomposite(Image<Gray, Byte> src, Image<Gray, Single>[] eigenImages, Image<Gray, Single> avg)
{
return CvInvoke.cvEigenDecomposite(
src.Ptr,
Array.ConvertAll<Image<Gray, Single>, IntPtr>(eigenImages, delegate(Image<Gray, Single> img) { return img.Ptr; }),
avg.Ptr);
}

I guess, problem is in trainingImages. The width and height of each eigenImages must be the same as the width and height of the input image.
Your result image has size 100x100. So, all your trainingImage should have same size. Resize them, before adding to your list.
I review your code. Trouble still in sizes.
Line 305 replace to :
result = currentFrame.Copy(f.rect).Convert<Gray, byte>().Resize(gray.Width, gray.Height, Emgu.CV.CvEnum.INTER.CV_INTER_CUBIC);

Image doesn't update when written to.. weird goings on

I have a Kinect WPF Application that takes images from the Kinect, does some feature detection using EmguCV (A C# opencv wrapper) and displays the output on the using a WPF image.
I have had this working before, but the application now refuses to update the screen image when the imagesource is written to, but I have not changed the way it works.
the Image(called video) is written to as such:
video.Source = bitmapsource;
in the colorframeready event handler.
This works fine until I introduce some opencv code before the imagesource is written to. It does not matter what source is used, so I don't think it is a conflict there. I have narrowed down the offending EmguCV code to this line:
RecentKeyPoints = surfCPU.DetectKeyPointsRaw(ImageRecent, null);
which jumps straight into the opencv code. It is worth noting that:
ImageRecent has completely different origins to the bitmapsource updating the screen.
Reading video.Source returns the bitmapsource, so it seems to be writing correctly, just not updating the screen.
Let me know if you want any more information...
void nui_ColorFrameReady(object sender, ColorImageFrameReadyEventArgs e)
{
// Checks for a recent Depth Image
if (!TrackingReady) return;
// Stores image
using (ColorImageFrame colorImageFrame = e.OpenColorImageFrame())
{
if (colorImageFrame != null)
{
if (FeatureTracker.ColourImageRecent == null)
//allocate the first time
FeatureTracker.ColourImageRecent = new byte[colorImageFrame.PixelDataLength];
colorImageFrame.CopyPixelDataTo(FeatureTracker.ColourImageRecent);
}
else return;
}
FeatureTracker.FeatureDetect(nui);
//video.Source = FeatureTracker.ColourImageRecent.ToBitmapSource();
video.Source = ((Bitmap)Bitmap.FromFile("test1.png")).ToBitmapSource();
TrackingReady = false;
}
public Bitmap FeatureDetect(KinectSensor nui)
{
byte[] ColourClone = new byte[ColourImageRecent.Length];
Array.Copy(ColourImageRecent, ColourClone, ColourImageRecent.Length);
Bitmap test = (Bitmap)Bitmap.FromFile("test1.png");
test.RotateFlip(RotateFlipType.RotateNoneFlipY);
Image<Gray, Byte> ImageRecent = new Image<Gray, byte>(test);
SURFDetector surfCPU = new SURFDetector(2000, false);
VectorOfKeyPoint RecentKeyPoints;
Matrix<int> indices;
Matrix<float> dist;
Matrix<byte> mask;
bool MatchFailed = false;
// extract SURF features from the object image
RecentKeyPoints = surfCPU.DetectKeyPointsRaw(ImageRecent, null);
//Matrix<float> RecentDescriptors = surfCPU.ComputeDescriptorsRaw(ImageRecent, null, RecentKeyPoints);
//MKeyPoint[] RecentPoints = RecentKeyPoints.ToArray();
// don't feature detect on first attempt, just store image details for next attempt
#region
/*
if (KeyPointsOld == null)
{
KeyPointsOld = RecentKeyPoints;
PointsOld = RecentPoints;
DescriptorsOld = RecentDescriptors;
return ImageRecent.ToBitmap();
}
*/
#endregion
// Attempt to match points to their nearest neighbour
#region
/*
BruteForceMatcher SURFmatcher = new BruteForceMatcher(BruteForceMatcher.DistanceType.L2F32);
SURFmatcher.Add(RecentDescriptors);
int k = 5;
indices = new Matrix<int>(DescriptorsOld.Rows, k);
dist = new Matrix<float>(DescriptorsOld.Rows, k);
*/
// Match features, provide the top k matches
//SURFmatcher.KnnMatch(DescriptorsOld, indices, dist, k, null);
// Create mask and set to allow all features
//mask = new Matrix<byte>(dist.Rows, 1);
//mask.SetValue(255);
#endregion
//Features2DTracker.VoteForUniqueness(dist, 0.8, mask);
// Check number of good maches and for error and end matching if true
#region
//int nonZeroCount = CvInvoke.cvCountNonZero(mask);
//if (nonZeroCount < 5) MatchFailed = true;
/*
try
{
nonZeroCount = Features2DTracker.VoteForSizeAndOrientation(RecentKeyPoints, KeyPointsOld, indices, mask, 1.5, 20);
}
catch (SystemException)
{
MatchFailed = true;
}
if (nonZeroCount < 5) MatchFailed = true;
if (MatchFailed)
{
return ImageRecent.ToBitmap();
}
*/
#endregion
//DepthMapColourCoordsRecent = CreateDepthMap(nui, DepthImageRecent);
//PointDist[] FeatureDistances = DistanceToFeature(indices, mask, RecentPoints);
//Image<Rgb,Byte> rgbimage = ImageRecent.Convert<Rgb, Byte>();
//rgbimage = DrawPoints(FeatureDistances, rgbimage);
// Store recent image data for next feature detect.
//KeyPointsOld = RecentKeyPoints;
//PointsOld = RecentPoints;
//DescriptorsOld = RecentDescriptors;
//CreateDepthMap(nui, iva);
//rgbimage = CreateDepthImage(DepthMapColourCoordsRecent, rgbimage);
// Convert image back to a bitmap
count++;
//Bitmap bitmap3 = rgbimage.ToBitmap();
//bitmapstore = bitmap3;
//bitmap3.Save("test" + count.ToString() + ".png");
return null;
}

This is a little late, but I had a similar problem and thought I'd share my solution.
In my case I was processing the depth stream. The default resolution was 640x480, and Emgu just wasn't able to process the image fast enough to keep up with the frameready handler. As soon as I reduced the depth stream resolution to 320x240 the problem went away.
I also went a bit further and moved my image processing to a different thread which sped it up even more (do a search for ComponentDispatcher.ThreadIdle). I'm still not able to do 640x480 at a reasonable frame rate, but at least the image renders so I can see what's going on.

Fill the holes in emgu cv

How can I fill the holes in binary image in emgu cv?
In Aforge.net it's easy, use Fillholes class.

Thought the question is a little bit old, I'd like to contribute an alternative solution to the problem.
You can obtain the same result as Chris' without memory problem if you use the following:
private Image<Gray,byte> FillHoles(Image<Gray,byte> image)
{
var resultImage = image.CopyBlank();
Gray gray = new Gray(255);
using (var mem = new MemStorage())
{
for (var contour = image.FindContours(
CHAIN_APPROX_METHOD.CV_CHAIN_APPROX_SIMPLE,
RETR_TYPE.CV_RETR_CCOMP,
mem); contour!= null; contour = contour.HNext)
{
resultImage.Draw(contour, gray, -1);
}
}
return resultImage;
}
The good thing about the method above is that you can selectively fill holes that meets your criteria. For example, you may want to fill holes whose pixel count (count of black pixels inside the blob) is below 50, etc.
private Image<Gray,byte> FillHoles(Image<Gray,byte> image, int minArea, int maxArea)
{
var resultImage = image.CopyBlank();
Gray gray = new Gray(255);
using (var mem = new MemStorage())
{
for (var contour = image.FindContours(
CHAIN_APPROX_METHOD.CV_CHAIN_APPROX_SIMPLE,
RETR_TYPE.CV_RETR_CCOMP,
mem); contour!= null; contour = contour.HNext)
{
if ( (contour.Area < maxArea) && (contour.Area > minArea) )
resultImage.Draw(contour, gray, -1);
}
}
return resultImage;
}

Yes there is a method but it's a bit messy as its based on cvFloodFill operation. Now all this algorithm is designed to do is fill an area with a colour until it reaches an edge similar to a region growing algorithm. To use this effectively you need to use a little inventive coding but I warn you this code is only to get you started it may require re-factoring to speed things up . As it stands the loop goes through each of your pixels that are less then 255 applies cvFloodFill checks what size the area is and then if it is under a certain area fill it in.
It is important to note that a copy of the image is made of the original image to be supplied to the cvFloodFill operation as a pointer is used. If the direct image is supplied then you will end up with a white image.
OpenFileDialog OpenFile = new OpenFileDialog();
if (OpenFileDialog.ShowDialog() == DialogResult.OK)
{
Image<Bgr, byte> image = new Image<Bgr, byte>(OpenFile.FileName);
for (int i = 0; i < image.Width; i++)
{
for (int j = 0; j < image.Height; j++)
{
if (image.Data[j, i, 0] != 255)
{
Image<Bgr, byte> image_copy = image.Copy();
Image<Gray, byte> mask = new Image<Gray, byte>(image.Width + 2, image.Height + 2);
MCvConnectedComp comp = new MCvConnectedComp();
Point point1 = new Point(i, j);
//CvInvoke.cvFloodFill(
CvInvoke.cvFloodFill(image_copy.Ptr, point1, new MCvScalar(255, 255, 255, 255),
new MCvScalar(0, 0, 0),
new MCvScalar(0, 0, 0), out comp,
Emgu.CV.CvEnum.CONNECTIVITY.EIGHT_CONNECTED,
Emgu.CV.CvEnum.FLOODFILL_FLAG.DEFAULT, mask.Ptr);
if (comp.area < 10000)
{
image = image_copy.Copy();
}
}
}
}
}
The "new MCvScalar(0, 0, 0), new MCvScalar(0, 0, 0)," are not really important in this case as you are only filling in results of a binary image. YOu could play around with other settings to see what results you can achieve. "if (comp.area < 10000)" is the key constant to change is you want to change what size hole the method will fill.
These are the results that you can expect:
Original
Results
The problem with this method is it's extremely memory intensive and it managed to eat up 6GB of ram on a 200x200 image and when I tried 200x300 it ate all 8GB of my RAM and brought everything to a crashing halt. Unless a majority of your image is white and you want to fill in tiny gaps or you can minimise where you apply the method I would avoid it. I would suggest writing you own class to examine each pixel that is not 255 and add the number of pixels surrounding it. You can then record the position of each pixel that was not 255 (in a simple list) and if your count was bellow a threshold set these positions to 255 in your images (by iterating though the list).
I would stick with the Aforge FillHoles class if you do not wish to write your own as it is designed for this purpose.
Cheers
Chris

you can use FillConvexPoly
image.FillConvexPoly(externalContours.ToArray(), new Gray(255));