Develop Reference

EmguCV C# : FindContours() to detect different shapes

I have this image :
What I try to do is detecting the contours of it. So with looking to the documentation and some code on the web I made this :
Image<Gray, byte> image = receivedImage.Convert<Gray, byte>().ThresholdBinary(new Gray(80), new Gray(255));
Emgu.CV.Util.VectorOfVectorOfPoint contours = new Emgu.CV.Util.VectorOfVectorOfPoint();
Mat hier = new Mat();
CvInvoke.FindContours(image, contours, hier, Emgu.CV.CvEnum.RetrType.External, Emgu.CV.CvEnum.ChainApproxMethod.ChainApproxSimple);
CvInvoke.DrawContours(receivedImage, contours, 0, new MCvScalar(255, 0, 0), 2);
Then it detects this contour in blue :
Now I would like to detect both rectangles in differents contours. So the result would be this :
(made with paint) So now I would like to detect the two rectangles separetely, (the blue and red rectangles would be two different contours). But I have no idea about how to do that !
Thanks in advance for your help ! ;)

The problem comes from the process of ThresholdBinary. As I assume you understand, this method will return a binary image, whereby all pixels above or equal to the threshold parameter, will be pulled up to maxValue parameter, and all those below, pulled down to 0. The produced image will consist therefore of only two values (binary), 0 or maxValue. If we follow your example with some assumed gray values:
After Image<Gray, byte> image = receivedImage.Convert<Gray, byte>().ThresholdBinary(new Gray(80), new Gray(255));, you will produce:
This is in fact the image that you are passing to CvInvoke.FindContours() and subsequently finding only the out most contour.
What you need, if indeed you want to continue with FindContours, is an algorithm that will "bin", or "bandpass" your image to first produce the following segments, each to be converted to binary, and contour detected independently.
I feel that you currently example is probably and over simplification of the problem to offer you a solution on how you might achieve that here. However please do ask another question with more realistic data, and I will be happy to provide some suggestions.
Alternatively look towards more sophisticated edge detection methods such and Canny or Sobel. This video may be a good starting point: Edge Detection

OpenCV MatchTemplate in C# is too slow compared to Python

I've programmed a solution in Python which worked great, but required several libraries to install and a lot of burocratic setup to work. I've decided to build it with a GUI in C# on Visual Studio Community 2017 but in the first successful function the result was way slower than in Python. Which IMO it should actually be faster.
The code essentially is just doing a needle in a haystack image search, by getting all images from a folder and testing each needle (total 60 images) in a haystack, in python I return the string, but in C# I'm only printing.
My code in Python is the following:
def getImages(tela):
retorno = []
folder = 'Images'
img_rgb = cv2.imread(tela)
for filename in os.listdir(folder):
template = cv2.imread(os.path.join(folder,filename))
w, h = template.shape[:-1]
res = cv2.matchTemplate(img_rgb, template, cv2.TM_CCOEFF_NORMED)
threshold = .96
loc = np.where(res >= threshold)
if loc[0]>0:
retorno.append(filename[0]+filename[1].lower())
if len(retorno)> 1:
return retorno
and in C#:
Debug.WriteLine(ofd.FileName);
Image<Bgr, byte> source = new Image<Bgr, byte>(ofd.FileName);
string filepath = Directory.GetCurrentDirectory().ToString()+"\\Images";
DirectoryInfo d = new DirectoryInfo(filepath);
var files = d.GetFiles();
foreach (var fname in files){
Image<Bgr, byte> template = new Image<Bgr, byte>(fname.FullName);
Image<Gray, float> result = source.MatchTemplate(template, Emgu.CV.CvEnum.TemplateMatchingType.CcoeffNormed);
double[] minValues, maxValues;
Point[] minLocations, maxLocations;
result.MinMax(out minValues, out maxValues, out minLocations, out maxLocations);
if (maxValues[0] > 0.96) {
Debug.WriteLine(fname);
}
}
I didn't measure the time elapsed between each one, but I can say the result in C# takes about 3 seconds and in Python about 100ms.
There is room for optimization, if anyone would like to suggest any improvements, they are welcome.

The issue is that in Python code you finish the iteration when at least one match is added to retorno:
if len(retorno)> 1:
return retorno
In C# sample you continue iteration until all files are looped through.

I've combined the solutions proposed by denfromufa and HouseCat in the source code below, and did some overall cleanup, so you can see how your code could be. You will also notice minor readability improvements, since I wrote the refactored code using C# 7.0 / .NET 4.7.
Real Algorithm Optimization
Although denfromula correctly pointed out that implementation issue, and HouseCat mentioned using more CPU resources, the true gain relies on reducing the number of operations executed during your image search algorithm.
TURBO STAGE 1 - Suppose the MinMax() function goes through all your image's pixels to collect all those statistics, but you are only interested in using maxValue[0]. An extreme fine tuning would be to write a specific function which stops iterating through all your image's pixels when maxValue[0] goes below your minimum threshold. Apparently, that's all you need in your function. Remember: never burn all your processors computing lots of unused image statistics.
TURBO STAGE 2 - It looks like you are trying to recognize whether any image of your set of images matches your input screenshot (tela). If there are not too many images to be matched, and if you are constantly checking your screen for new matches, it is highly recommended to pre-load all those image match objects, and reuse them among your function calls. Constant disk IO operations and instantiating bitmap classes (for every single screenshot) leads to strong performance hit.
TURBO STAGE 3 - Just in case you are taking several screenshots per second, then try to reuse the screenshot's buffer. Constantly reallocating the whole screenshot's buffer when its dimensions simply did not change also causes performance loss.
TURBO STAGE 4 - This is hard to get, and depends on how much you want to invest on this. Think about your image recognition system as a big pipeline. The bitmaps as containers of data flowing among your stages (image matching stage, OCR stage, mouse position painting stage, video recording stage, etc). The idea is to create a fixed number of containers and reuse them, avoiding their creation and their destruction. The amount of containers is like the "buffer size" for your pipeline system. When the several stages of your pipeline finished using these containers, they are returned to the start of your pipeline, to a kind of container pool.
This last optimization this is really hard to achieve using these external libraries, because in most cases their API require some internal bitmap instantiation, and the fine tuning would also cause extreme software coupling between your library and the external one. So you will have to dig into these nice libraries to understand how they actually work, and build your own custom Framework. I can say it's a nice learning experience.
Those libraries are really cool for many purposes; they provide a generic API for improved functionality re-usability. This also means they address much more stuff than you actually need in a single API call. When it comes to high performance algorithms, you should always re-think what is the essential functionality you need from those libraries to achieve your goal, and if they are your bottleneck, do it by yourself.
I can say that a good fine-tuned image recognition algorithm doesn't take more than a few milliseconds to do what you want. I've experienced image recognition applications which do it almost instantaneously for larger screenshots (e.g. Eggplant Functional).
Now back to your code...
Your refactored code should look like below. I did not include all those fine-tuned algorithms I've mentioned - you should better ask separate questions for them in SO.
Image<Bgr, byte> source = new Image<Bgr, byte>(ofd.FileName);
// Preferably use Path.Combine here:
string dir = Path.Combine(Directory.GetCurrentDirectory(), "Images");
// Check whether directory exists:
if (!Directory.Exists(dir))
throw new Exception($"Directory was not found: '{dir}'");
// It looks like you just need filenames here...
// Simple parallel foreach suggested by HouseCat (in 2.):
Parallel.ForEach(Directory.GetFiles(dir), (fname) =>
{
Image<Gray, float> result = source.MatchTemplate(
new Image<Bgr, byte>(fname.FullName),
Emgu.CV.CvEnum.TemplateMatchingType.CcoeffNormed);
// By using C# 7.0, we can do inline out declarations here:
result.MinMax(
out double[] minValues,
out double[] maxValues,
out Point[] minLocations,
out Point[] maxLocations);
if (maxValues[0] > 0.96)
{
// ...
var result = ...
return result; // <<< As suggested by: denfromufa
}
// ...
});
Happy Tuning ;-)

This (denfromufa's answer) indeed explains your issue but to piggy back and add a few suggestions/optimizations as well:
1.) Your GetFiles can be replaced with a Parallel file enumerator, that is also recursive with children directories. I have shamelessly written a few on GitHub.
2.) You can parellelize the foreach loop into a Parallel.ForEach(files, fname () => { Code(); }); Again, my FileSearchBenchmark Repository on GitHub has plenty of File code execution in Parallel to provide examples.

Some Background remain after Background Subtraction EMGU CV

I am a newbie at EmguCV image processing and trying different methods of background subtraction. I came across at the method absdiff and gave it a try but after a bunch of processing, some part of the object seems to be transparent and the background behind it can be seen,Background subtraction sample
here is the part of my code that processes the image
img = _capture.QueryFrame().ToImage<Bgr, Byte>();
Mat smoothedFrame = new Mat();
CvInvoke.GaussianBlur(img, smoothedFrame, new Size(3, 3), 1);
img3 = img2gray.AbsDiff(smoothedFrame.ToImage<Gray, Byte>());//.Convert<Gray, Byte>());
img3 = img3.ThresholdBinary(new Gray(60), new Gray(255));
IbOriginal.Image = img;
IbProcessed.Image = img3;
How can i remove those "blank or hollow" space in the image above. Any help would be much appreciated

I'm guessing you want to create a mask with pixels of the truck only. You may have taken away pixels in the hollow spaces with
ThresholdBinary(new Gray(60), new Gray(255));
Decreasing the lower threshold might be what you need but it might include some background noise too. You can always identify the location of the truck first with a higher threshold (what you've done here) then ThresholdBinary on a previous image in the ROI identified.
ThresholdBinary(new Gray(10), new Gray(255));
Or you can try CvInvoke.FloodFill

Template matching - how to ignore pixels

I'm trying to find a digit within an image. To test my code I took an image of the digit and then used AForge's Exhaustive Template Matching algorithm to search for it in another image. But I think there is a problem in that the digit is obviously not rectangular whereas the image that contains it is. That means that there are a lot of pixels participating in the comparison which shouldn't be. Is there any way to make this comparison while ignoring those pixels? If not in AForge then maybe EMGU/OpenCV or Octave?
Here's my code:
Grayscale gray = new GrayscaleRMY();
Bitmap template = (Bitmap)Bitmap.FromFile(#"5template.png");
template = gray.Apply(template);
Bitmap image = (Bitmap)Bitmap.FromFile(filePath);
Bitmap sourceImage = gray.Apply(image);
ExhaustiveTemplateMatching tm = new ExhaustiveTemplateMatching(0.7f);
TemplateMatch[] matchings = tm.ProcessImage(sourceImage, template);

As mentioned above in the comment, you should preprocess your data to improve matching.
The first thing that comes to mind is morphological opening (erode then dilate) to reduce the background noise
Read in your image and invert it so that your character vectors are white:
Apply opening with smallest possible structuring element/window (3x3):
You could try slightly larger structuring elements (5x5):
Invert it back to the original:
See if that helps!

Perspective Image Distortion

The application I am working on currently requires functionality for Perspective Image Distortion. Basically what I want to do is to allow users to load an image into the application and adjust its perspective view properties based on 4 corner points that they can specify.
I had a look at ImageMagic. It has some distort functions with perpective adjustment but is very slow and some certain inputs are giving incorrect outputs.
Any of you guys used any other library or algorithm. I am coding in C#.
Any pointers would be much appreciated.
Thanks

This seems to be exactly what you (and I) were looking for:
http://www.codeproject.com/KB/graphics/YLScsFreeTransform.aspx
It will take an image and distort it using 4 X/Y coordinates you provide.
Fast, free, simple code. Tested and it works beautifully. Simply download the code from the link, then use FreeTransform.cs like this:
using (System.Drawing.Bitmap sourceImg = new System.Drawing.Bitmap(#"c:\image.jpg"))
{
YLScsDrawing.Imaging.Filters.FreeTransform filter = new YLScsDrawing.Imaging.Filters.FreeTransform();
filter.Bitmap = sourceImg;
// assign FourCorners (the four X/Y coords) of the new perspective shape
filter.FourCorners = new System.Drawing.PointF[] { new System.Drawing.PointF(0, 0), new System.Drawing.PointF(300, 50), new System.Drawing.PointF(300, 411), new System.Drawing.PointF(0, 461)};
filter.IsBilinearInterpolation = true; // optional for higher quality
using (System.Drawing.Bitmap perspectiveImg = filter.Bitmap)
{
// perspectiveImg contains your completed image. save the image or do whatever.
}
}

Paint .NET can do this and there are also custom implementations of the effect. You could ask for the source code or use Reflector to read it and get an idea of how to code it.

If it is a perspective transform, you should be able to specify a 4x4 transformation matrix that matches the four corners.
Calculate that matrix, then apply each pixel on the resulting image on the matrix, resulting in the "mapped" pixel. Notice that this "mapped" pixel is very likely going to lie between two or even four pixels. In this case, use your favorite interpolation algorithm (e.g. bilinear, bicubic) to get the interpolated color.
This really is the only way for it to be done and cannot be done faster. If this feature is crucial and you absolutely need it to be fast, then you'll need to offload the task to a GPU. For example, you can call upon the DirectX library to apply a perspective transformation on a texture. That can make it extremely fast, even when there is no GPU because the DirectX library uses SIMD instructions to accelerate matrix calculations and color interpolations.

Had the same problem. Here is the demo code with sources ported from gimp.

YLScsFreeTransform doesn't work as expected. Way better solution is ImageMagic
Here is how you use it in c#:
using(MagickImage image = new MagickImage("test.jpg"))
{
image.Distort(DistortMethod.Perspective, new double[] { x0,y0, newX0,newY0, x1,y1,newX1,newY1, x2,y2,newX2,newY2, x3,y3,newX3,newY3 });
control.Image = image.ToBitmap();
}