Develop Reference

C# Emgucv CvInvoke.DrawContours run very slow

I am using DrawContours method in emgucv package, but it's working very slow, it takes about "12" second
I do the same thing in python, it only takes maybe "2" second to do this.
The target is "Find out all external contour areas that is bigger than 800 pixels and Fill it up on a new canvas" (The number of contours here is 74 )
than I will use this image as a mask
Please tell me how can I improve the time consuming problem
Here's the C# code :
VectorOfVectorOfPoint contours = new VectorOfVectorOfPoint();
CvInvoke.FindContours(img, contours, null, RetrType.External, ChainApproxMethod.ChainApproxSimple);
UMat mask = new UMat(img.Size, DepthType.Cv8U, 1);
int minArea = 800;
for (int i = 0; i < contours.Size; i++)
{
int area = CvInvoke.ContourArea(contours[i]);
if (area > minArea)
{
CvInvoke.DrawContours(mask, contours, i, new MCvScalar(255), -1);
}
else
continue;
}
also show my python code :
contours,_ = cv2.findContours(img, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)
C = len(contours)
for i in range(0, C):
if cv2.contourArea(contours[i]) > 800 :
cv2.drawContours(mask, contours, i, (255,255,255), -1)
else:
continue
Thank you

OpenCV FindContours does not find the correct rectangle

Using CvInvoke.Canny and CvInvoke.FindContours I'm trying to find the rectangle containing the item name (Schematic: Maple Desk). This rectangle is shown in the image below:
I'm able to find a lot of rectangles but I'm not able to get this one. Tried a lot of different thresholds for Canny but to no effect. The following image shows all rectangles I currently get:
Any ideas how to tackle this? Do I need to use other thresholds or another approach? I already experimented using grayscale and blurring but that didn't give better result. I added my current source below and the original image I'm using is this:
public Mat ProcessImage(Mat img)
{
UMat filter = new UMat();
UMat cannyEdges = new UMat();
Mat rectangleImage = new Mat(img.Size, DepthType.Cv8U, 3);
//Convert the image to grayscale and filter out the noise
//CvInvoke.CvtColor(img, filter, ColorConversion.Bgr2Gray);
//Remove noise
//CvInvoke.GaussianBlur(filter, filter, new System.Drawing.Size(3, 3), 1);
// Canny and edge detection
double cannyThreshold = 1.0; //180.0
double cannyThresholdLinking = 1.0; //120.0
//CvInvoke.Canny(filter, cannyEdges, cannyThreshold, cannyThresholdLinking);
CvInvoke.Canny(img, cannyEdges, cannyThreshold, cannyThresholdLinking);
// Find rectangles
List<RotatedRect> rectangleList = new List<RotatedRect>();
using (VectorOfVectorOfPoint contours = new VectorOfVectorOfPoint())
{
CvInvoke.FindContours(cannyEdges, contours, null, RetrType.List, ChainApproxMethod.ChainApproxSimple);
int count = contours.Size;
for (int i = 0; i < count; i++)
{
using (VectorOfPoint contour = contours[i])
using (VectorOfPoint approxContour = new VectorOfPoint())
{
CvInvoke.ApproxPolyDP(contour, approxContour, CvInvoke.ArcLength(contour, true) * 0.05, true);
// Only consider contours with area greater than 250
if (CvInvoke.ContourArea(approxContour, false) > 250)
{
// The contour has 4 vertices.
if (approxContour.Size == 4)
{
// Determine if all the angles in the contour are within [80, 100] degree
bool isRectangle = true;
System.Drawing.Point[] pts = approxContour.ToArray();
LineSegment2D[] edges = Emgu.CV.PointCollection.PolyLine(pts, true);
for (int j = 0; j < edges.Length; j++)
{
double angle = Math.Abs(edges[(j + 1) % edges.Length].GetExteriorAngleDegree(edges[j]));
if (angle < 80 || angle > 100)
{
isRectangle = false;
break;
}
}
if (isRectangle) rectangleList.Add(CvInvoke.MinAreaRect(approxContour));
}
}
}
}
}
// Draw rectangles
foreach (RotatedRect rectangle in rectangleList)
{
CvInvoke.Polylines(rectangleImage, Array.ConvertAll(rectangle.GetVertices(), System.Drawing.Point.Round), true,
new Bgr(Color.DarkOrange).MCvScalar, 2);
}
//Drawing a light gray frame around the image
CvInvoke.Rectangle(rectangleImage,
new Rectangle(System.Drawing.Point.Empty,
new System.Drawing.Size(rectangleImage.Width - 1, rectangleImage.Height - 1)),
new MCvScalar(120, 120, 120));
//Draw the labels
CvInvoke.PutText(rectangleImage, "Rectangles", new System.Drawing.Point(20, 20),
FontFace.HersheyDuplex, 0.5, new MCvScalar(120, 120, 120));
Mat result = new Mat();
CvInvoke.VConcat(new Mat[] { img, rectangleImage }, result);
return result;
}
Edit 1:
After some more fine tuning with the following thresholds for Canny
cannyThreshold 100
cannyThresholdLinking 400
And using a minimum size for all ContourAreas of 10000 I can get the following result:
Edit 2:
For those interested, solved using the current detection, no changes were needed. Used the 2 detected rectangles in the screenshot above to get the location of the missing rectangle containing the item name.
Result can be found here:
https://github.com/josdemmers/NewWorldCompanion

For those interested, solved using the current detection, no changes were needed. Used the 2 detected rectangles in the screenshot above to get the location of the missing rectangle containing the item name.
Result can be found here: https://github.com/josdemmers/NewWorldCompanion

Kinect depth detection

I know how to do it in WPF but I have problem for capturing depth in winforms application.
I found some code as below:
private void Kinect_DepthFrameReady(object sender, DepthImageFrameReadyEventArgs e)
{
using (DepthImageFrame depthFrame = e.OpenDepthImageFrame())
{
if (depthFrame != null)
{
Bitmap DepthBitmap = new Bitmap(depthFrame.Width, depthFrame.Height, PixelFormat.Format32bppRgb);
if (_depthPixels.Length != depthFrame.PixelDataLength)
{
_depthPixels = new DepthImagePixel[depthFrame.PixelDataLength];
_mappedDepthLocations = new ColorImagePoint[depthFrame.PixelDataLength];
}
//Copy the depth frame data onto the bitmap
var _pixelData = new short[depthFrame.PixelDataLength];
depthFrame.CopyPixelDataTo(_pixelData);
BitmapData bmapdata = DepthBitmap.LockBits(new Rectangle(0, 0, depthFrame.Width,
depthFrame.Height), ImageLockMode.WriteOnly, DepthBitmap.PixelFormat);
IntPtr ptr = bmapdata.Scan0;
Marshal.Copy(_pixelData, 0, ptr, depthFrame.Width * depthFrame.Height);
DepthBitmap.UnlockBits(bmapdata);
pictureBox2.Image = DepthBitmap;
}
}
}
but this is not giving me the greyScale depth and it's purple. Any improvement or help?

I found the solution myself, by a function to convert the depth frame:
void Kinect_DepthFrameReady(object sender, DepthImageFrameReadyEventArgs e)
{
using (DepthImageFrame depthFrame = e.OpenDepthImageFrame())
{
if (depthFrame != null)
{
this.depthFrame32 = new byte[depthFrame.Width * depthFrame.Height * 4];
//Update the image to the new format
this.depthPixelData = new short[depthFrame.PixelDataLength];
depthFrame.CopyPixelDataTo(this.depthPixelData);
byte[] convertedDepthBits = this.ConvertDepthFrame(this.depthPixelData, ((KinectSensor)sender).DepthStream);
Bitmap bmap = new Bitmap(depthFrame.Width, depthFrame.Height, PixelFormat.Format32bppRgb);
BitmapData bmapdata = bmap.LockBits(new Rectangle(0, 0, depthFrame.Width, depthFrame.Height), ImageLockMode.WriteOnly, bmap.PixelFormat);
IntPtr ptr = bmapdata.Scan0;
Marshal.Copy(convertedDepthBits, 0, ptr, 4 * depthFrame.PixelDataLength);
bmap.UnlockBits(bmapdata);
pictureBox2.Image = bmap;
}
}
}
private byte[] ConvertDepthFrame(short[] depthFrame, DepthImageStream depthStream)
{
//Run through the depth frame making the correlation between the two arrays
for (int i16 = 0, i32 = 0; i16 < depthFrame.Length && i32 < this.depthFrame32.Length; i16++, i32 += 4)
{
// Console.WriteLine(i16 + "," + i32);
//We don’t care about player’s information here, so we are just going to rule it out by shifting the value.
int realDepth = depthFrame[i16] >> DepthImageFrame.PlayerIndexBitmaskWidth;
//We are left with 13 bits of depth information that we need to convert into an 8 bit number for each pixel.
//There are hundreds of ways to do this. This is just the simplest one.
//Lets create a byte variable called Distance.
//We will assign this variable a number that will come from the conversion of those 13 bits.
byte Distance = 0;
//XBox Kinects (default) are limited between 800mm and 4096mm.
int MinimumDistance = 800;
int MaximumDistance = 4096;
//XBox Kinects (default) are not reliable closer to 800mm, so let’s take those useless measurements out.
//If the distance on this pixel is bigger than 800mm, we will paint it in its equivalent gray
if (realDepth > MinimumDistance)
{
//Convert the realDepth into the 0 to 255 range for our actual distance.
//Use only one of the following Distance assignments
//White = Far
//Black = Close
//Distance = (byte)(((realDepth – MinimumDistance) * 255 / (MaximumDistance-MinimumDistance)));
//White = Close
//Black = Far
Distance = (byte)(255 - ((realDepth - MinimumDistance) * 255 / (MaximumDistance - MinimumDistance)));
//Use the distance to paint each layer (R G & of the current pixel.
//Painting R, G and B with the same color will make it go from black to gray
this.depthFrame32[i32 + RedIndex] = (byte)(Distance);
this.depthFrame32[i32 + GreenIndex] = (byte)(Distance);
this.depthFrame32[i32 + BlueIndex] = (byte)(Distance);
}
//If we are closer than 800mm, the just paint it red so we know this pixel is not giving a good value
else
{
this.depthFrame32[i32 + RedIndex] = 0;
this.depthFrame32[i32 + GreenIndex] = 0;
this.depthFrame32[i32 + BlueIndex] = 0;
}
}

so i presume that rgb frame is working out for you in that case:
first to enable depth camera you need to call:
sensor->NuiInitialize(NUI_INITIALIZE_FLAG_USES_DEPTH|all stuff you use also);
second to start streaming you need to call:
if (int(streams&_Kinect_zed)) ret=sensor->NuiImageStreamOpen(
NUI_IMAGE_TYPE_DEPTH, // Depth camera or rgb camera?
NUI_IMAGE_RESOLUTION_640x480, // Image resolution
NUI_IMAGE_STREAM_FLAG_DISTINCT_OVERFLOW_DEPTH_VALUES, // Image stream flags // NUI_IMAGE_STREAM_FLAG_ENABLE_NEAR_MODE nefunguje !!!
2, // Number of frames to buffer
NULL, // Event handle
&stream_hzed); else stream_hzed=NULL;
beware not all resolution/flags combinations work on all models of kinect !!!
this one above is safe even for the older models like mine
this is how i capture frame (called repeatedly from timer or thread loop)
ret=sensor->NuiImageStreamGetNextFrame(stream_hzed,0,&imageFrame); if (ret>=0)
{
// copy data from frame
imageFrame.pFrameTexture->LockRect(0, &LockedRect, NULL, 0);
if (LockedRect.Pitch!=0)
{
const BYTE* curr = (const BYTE*) LockedRect.pBits;
union _col { BYTE u8[2]; WORD u16; } col;
col.u16=0;
pnt3d p;
long ax,ay;
float mxs=float(xs)/(62.0*deg),mys=float(ys)/(48.6*deg);
for(int x=0,y=0;;)
{
col.u8[0]=*curr; curr++;
col.u8[1]=*curr; curr++;
p.raw=col.u16;
p.rgb=&rgb_default;
if (p.raw==0x0000) p.z=0.0; // p.z je kolma vzdialenost od senzora (kinect to correctuje sam)
else if (p.raw>=0x8000) p.z=4.0;
else p.z=0.8+(float(p.raw-6576)*0.00012115165336374002280501710376283);
// depth FOV correction
p.x=zx[x]*p.z;
p.y=zy[y]*p.z;
// color FOV correction zed 58.5° x 45.6° | rgb 62.0° x 48.6° | 25mm distance
if (p.z>0.0)
{
ax=(((x+10-xs2)*241)>>8)+xs2; // cameras x-offset and different FOV
ay=(((y+30-ys2)*240)>>8)+ys2; // cameras y-offset??? and different FOV
if ((ax>=0)&&(ax<xs))
if ((ay>=0)&&(ay<ys)) p.rgb=&rgb[ay][ax];
}
xyz[y][x]=p;
x++; if (x>=xs) { x=0; y++; if (y>=ys) break; }
}
}
// release frame
imageFrame.pFrameTexture->UnlockRect(0);
ret=sensor->NuiImageStreamReleaseFrame(stream_hzed, &imageFrame);
stream_changed|=_Kinect_zed;
}
Sorry for incomplete source code ...
- all is copy pasted from my kinect class (BDS2006 Turbo C++)
- so you need to check your code if you do not forget something
- and if yes then transform my code to C# (i am not C# user)
- most likely you forget to NUIinitialize with depth flag
- or set invalid resolution/flags/ precision or framerate for your HW
if nothing work at all then you need to initialize the sensor in the first place
int sensors;
INuiSensor *sensor;
if ((NUIGetSensorCount(&sensors)<0)||(sensors<1)) return false;
if (NUICreateSensorByIndex(0,&sensor)<0) return false;
if you link to dll on your own then link only these functions:
typedef HRESULT(__stdcall *_NuiGetSensorCount )(int * pCount); _NuiGetSensorCount NUIGetSensorCount =NULL;
typedef HRESULT(__stdcall *_NuiCreateSensorByIndex)(int index,INuiSensor **ppNuiSensor); _NuiCreateSensorByIndex NUICreateSensorByIndex=NULL;
Every other function (must) is obtained via COM inside SDK headers !!!
if you link and use them on your own then you will not be connected to your physical Kinect !!!

Basically kinect sdk is developed for WPf application. In windows form you have convert the short array of the depth data to the BItmap to display it on picturebox. And based on my expriment WPF is better for programming with kinect.
Below is the function that I used to convert depth frame to Bitmap for showing in picture box.
private Bitmap ImageToBitmap(DepthImageFrame Image)
{
short[] pixeldata = new short[Image.PixelDataLength];
int stride = Image.Width * 2;
Image.CopyPixelDataTo(pixeldata);
Bitmap bmap = new Bitmap(Image.Width, Image.Height, PixelFormat.Format16bppRgb555);
BitmapData bmapdata = bmap.LockBits(new Rectangle(0, 0, Image.Width, Image.Height), ImageLockMode.WriteOnly, bmap.PixelFormat);
IntPtr ptr = bmapdata.Scan0;
Marshal.Copy(pixeldata, 0, ptr, Image.PixelDataLength);
bmap.UnlockBits(bmapdata);
return bmap;
}
You may call it like this:
DepthImageFrame VFrame = e.OpenDepthImageFrame();
if (VFrame == null) return;
short[] pixelS = new short[VFrame.PixelDataLength];
Bitmap bmap = ImageToBitmap(VFrame);

Threshold value when converting image to 1bpp?

I don't know how to tag this question, please edit if possible.
The job: Create an application which can auto-crop black borders in images in batch runs. Images vary in quality from 100-300dpi, 1bpp-24bpp and a batch can vary from 10 - 10 000 images.
The plan: Convert image to 1bpp (bitonal, black/white, if it isn't already) and after "cleaning up" white spots/dirt/noise find where the black ends and the white begins, these are the new coords for the image crop, apply them to a clone of the original image. Delete old image, save new one.
The progress: All of the above is done, and works, but...
The problem: When converting to 1bpp I have no control of a "threshold" value. I need this. A lot of dark images get cropped too much.
The tries: I've tried
Bitmap imgBitonal = imgOriginal.Clone(new Rectangle(0, 0, b.Width, b.Height), PixelFormat.Format1bppIndexed)
And also this. Both of which work, but none seem to give me the possibility to manually set a threshold value. I need for the user to be able to set this value, amongst others, and use my "preview" function before running the batch so as to see if the settings are any good.
The cry: I'm at a loss here. I don't now what to do or how to do it. Please help a fellow coder out. Point me in a direction, show me where in the code found in the link a threshold value is found (I haven't found one, or don't know where to look) or just give me some code that works. Any help is appreciated.

Try this, from very fast 1bpp convert:
Duplicate from here Convert 24bpp Bitmap to 1bpp
private static unsafe void Convert(Bitmap src, Bitmap conv)
{
// Lock source and destination in memory for unsafe access
var bmbo = src.LockBits(new Rectangle(0, 0, src.Width, src.Height), ImageLockMode.ReadOnly,
src.PixelFormat);
var bmdn = conv.LockBits(new Rectangle(0, 0, conv.Width, conv.Height), ImageLockMode.ReadWrite,
conv.PixelFormat);
var srcScan0 = bmbo.Scan0;
var convScan0 = bmdn.Scan0;
var srcStride = bmbo.Stride;
var convStride = bmdn.Stride;
byte* sourcePixels = (byte*)(void*)srcScan0;
byte* destPixels = (byte*)(void*)convScan0;
var srcLineIdx = 0;
var convLineIdx = 0;
var hmax = src.Height-1;
var wmax = src.Width-1;
for (int y = 0; y < hmax; y++)
{
// find indexes for source/destination lines
// use addition, not multiplication?
srcLineIdx += srcStride;
convLineIdx += convStride;
var srcIdx = srcLineIdx;
for (int x = 0; x < wmax; x++)
{
// index for source pixel (32bbp, rgba format)
srcIdx += 4;
//var r = pixel[2];
//var g = pixel[1];
//var b = pixel[0];
// could just check directly?
//if (Color.FromArgb(r,g,b).GetBrightness() > 0.01f)
if (!(sourcePixels[srcIdx] == 0 && sourcePixels[srcIdx + 1] == 0 && sourcePixels[srcIdx + 2] == 0))
{
// destination byte for pixel (1bpp, ie 8pixels per byte)
var idx = convLineIdx + (x >> 3);
// mask out pixel bit in destination byte
destPixels[idx] |= (byte)(0x80 >> (x & 0x7));
}
}
}
src.UnlockBits(bmbo);
conv.UnlockBits(bmdn);
}

Fill the holes in emgu cv

How can I fill the holes in binary image in emgu cv?
In Aforge.net it's easy, use Fillholes class.

Thought the question is a little bit old, I'd like to contribute an alternative solution to the problem.
You can obtain the same result as Chris' without memory problem if you use the following:
private Image<Gray,byte> FillHoles(Image<Gray,byte> image)
{
var resultImage = image.CopyBlank();
Gray gray = new Gray(255);
using (var mem = new MemStorage())
{
for (var contour = image.FindContours(
CHAIN_APPROX_METHOD.CV_CHAIN_APPROX_SIMPLE,
RETR_TYPE.CV_RETR_CCOMP,
mem); contour!= null; contour = contour.HNext)
{
resultImage.Draw(contour, gray, -1);
}
}
return resultImage;
}
The good thing about the method above is that you can selectively fill holes that meets your criteria. For example, you may want to fill holes whose pixel count (count of black pixels inside the blob) is below 50, etc.
private Image<Gray,byte> FillHoles(Image<Gray,byte> image, int minArea, int maxArea)
{
var resultImage = image.CopyBlank();
Gray gray = new Gray(255);
using (var mem = new MemStorage())
{
for (var contour = image.FindContours(
CHAIN_APPROX_METHOD.CV_CHAIN_APPROX_SIMPLE,
RETR_TYPE.CV_RETR_CCOMP,
mem); contour!= null; contour = contour.HNext)
{
if ( (contour.Area < maxArea) && (contour.Area > minArea) )
resultImage.Draw(contour, gray, -1);
}
}
return resultImage;
}

Yes there is a method but it's a bit messy as its based on cvFloodFill operation. Now all this algorithm is designed to do is fill an area with a colour until it reaches an edge similar to a region growing algorithm. To use this effectively you need to use a little inventive coding but I warn you this code is only to get you started it may require re-factoring to speed things up . As it stands the loop goes through each of your pixels that are less then 255 applies cvFloodFill checks what size the area is and then if it is under a certain area fill it in.
It is important to note that a copy of the image is made of the original image to be supplied to the cvFloodFill operation as a pointer is used. If the direct image is supplied then you will end up with a white image.
OpenFileDialog OpenFile = new OpenFileDialog();
if (OpenFileDialog.ShowDialog() == DialogResult.OK)
{
Image<Bgr, byte> image = new Image<Bgr, byte>(OpenFile.FileName);
for (int i = 0; i < image.Width; i++)
{
for (int j = 0; j < image.Height; j++)
{
if (image.Data[j, i, 0] != 255)
{
Image<Bgr, byte> image_copy = image.Copy();
Image<Gray, byte> mask = new Image<Gray, byte>(image.Width + 2, image.Height + 2);
MCvConnectedComp comp = new MCvConnectedComp();
Point point1 = new Point(i, j);
//CvInvoke.cvFloodFill(
CvInvoke.cvFloodFill(image_copy.Ptr, point1, new MCvScalar(255, 255, 255, 255),
new MCvScalar(0, 0, 0),
new MCvScalar(0, 0, 0), out comp,
Emgu.CV.CvEnum.CONNECTIVITY.EIGHT_CONNECTED,
Emgu.CV.CvEnum.FLOODFILL_FLAG.DEFAULT, mask.Ptr);
if (comp.area < 10000)
{
image = image_copy.Copy();
}
}
}
}
}
The "new MCvScalar(0, 0, 0), new MCvScalar(0, 0, 0)," are not really important in this case as you are only filling in results of a binary image. YOu could play around with other settings to see what results you can achieve. "if (comp.area < 10000)" is the key constant to change is you want to change what size hole the method will fill.
These are the results that you can expect:
Original
Results
The problem with this method is it's extremely memory intensive and it managed to eat up 6GB of ram on a 200x200 image and when I tried 200x300 it ate all 8GB of my RAM and brought everything to a crashing halt. Unless a majority of your image is white and you want to fill in tiny gaps or you can minimise where you apply the method I would avoid it. I would suggest writing you own class to examine each pixel that is not 255 and add the number of pixels surrounding it. You can then record the position of each pixel that was not 255 (in a simple list) and if your count was bellow a threshold set these positions to 255 in your images (by iterating though the list).
I would stick with the Aforge FillHoles class if you do not wish to write your own as it is designed for this purpose.
Cheers
Chris

you can use FillConvexPoly
image.FillConvexPoly(externalContours.ToArray(), new Gray(255));