Develop Reference

OpenCV MatchTemplate in C# is too slow compared to Python

I've programmed a solution in Python which worked great, but required several libraries to install and a lot of burocratic setup to work. I've decided to build it with a GUI in C# on Visual Studio Community 2017 but in the first successful function the result was way slower than in Python. Which IMO it should actually be faster.
The code essentially is just doing a needle in a haystack image search, by getting all images from a folder and testing each needle (total 60 images) in a haystack, in python I return the string, but in C# I'm only printing.
My code in Python is the following:
def getImages(tela):
retorno = []
folder = 'Images'
img_rgb = cv2.imread(tela)
for filename in os.listdir(folder):
template = cv2.imread(os.path.join(folder,filename))
w, h = template.shape[:-1]
res = cv2.matchTemplate(img_rgb, template, cv2.TM_CCOEFF_NORMED)
threshold = .96
loc = np.where(res >= threshold)
if loc[0]>0:
retorno.append(filename[0]+filename[1].lower())
if len(retorno)> 1:
return retorno
and in C#:
Debug.WriteLine(ofd.FileName);
Image<Bgr, byte> source = new Image<Bgr, byte>(ofd.FileName);
string filepath = Directory.GetCurrentDirectory().ToString()+"\\Images";
DirectoryInfo d = new DirectoryInfo(filepath);
var files = d.GetFiles();
foreach (var fname in files){
Image<Bgr, byte> template = new Image<Bgr, byte>(fname.FullName);
Image<Gray, float> result = source.MatchTemplate(template, Emgu.CV.CvEnum.TemplateMatchingType.CcoeffNormed);
double[] minValues, maxValues;
Point[] minLocations, maxLocations;
result.MinMax(out minValues, out maxValues, out minLocations, out maxLocations);
if (maxValues[0] > 0.96) {
Debug.WriteLine(fname);
}
}
I didn't measure the time elapsed between each one, but I can say the result in C# takes about 3 seconds and in Python about 100ms.
There is room for optimization, if anyone would like to suggest any improvements, they are welcome.

The issue is that in Python code you finish the iteration when at least one match is added to retorno:
if len(retorno)> 1:
return retorno
In C# sample you continue iteration until all files are looped through.

I've combined the solutions proposed by denfromufa and HouseCat in the source code below, and did some overall cleanup, so you can see how your code could be. You will also notice minor readability improvements, since I wrote the refactored code using C# 7.0 / .NET 4.7.
Real Algorithm Optimization
Although denfromula correctly pointed out that implementation issue, and HouseCat mentioned using more CPU resources, the true gain relies on reducing the number of operations executed during your image search algorithm.
TURBO STAGE 1 - Suppose the MinMax() function goes through all your image's pixels to collect all those statistics, but you are only interested in using maxValue[0]. An extreme fine tuning would be to write a specific function which stops iterating through all your image's pixels when maxValue[0] goes below your minimum threshold. Apparently, that's all you need in your function. Remember: never burn all your processors computing lots of unused image statistics.
TURBO STAGE 2 - It looks like you are trying to recognize whether any image of your set of images matches your input screenshot (tela). If there are not too many images to be matched, and if you are constantly checking your screen for new matches, it is highly recommended to pre-load all those image match objects, and reuse them among your function calls. Constant disk IO operations and instantiating bitmap classes (for every single screenshot) leads to strong performance hit.
TURBO STAGE 3 - Just in case you are taking several screenshots per second, then try to reuse the screenshot's buffer. Constantly reallocating the whole screenshot's buffer when its dimensions simply did not change also causes performance loss.
TURBO STAGE 4 - This is hard to get, and depends on how much you want to invest on this. Think about your image recognition system as a big pipeline. The bitmaps as containers of data flowing among your stages (image matching stage, OCR stage, mouse position painting stage, video recording stage, etc). The idea is to create a fixed number of containers and reuse them, avoiding their creation and their destruction. The amount of containers is like the "buffer size" for your pipeline system. When the several stages of your pipeline finished using these containers, they are returned to the start of your pipeline, to a kind of container pool.
This last optimization this is really hard to achieve using these external libraries, because in most cases their API require some internal bitmap instantiation, and the fine tuning would also cause extreme software coupling between your library and the external one. So you will have to dig into these nice libraries to understand how they actually work, and build your own custom Framework. I can say it's a nice learning experience.
Those libraries are really cool for many purposes; they provide a generic API for improved functionality re-usability. This also means they address much more stuff than you actually need in a single API call. When it comes to high performance algorithms, you should always re-think what is the essential functionality you need from those libraries to achieve your goal, and if they are your bottleneck, do it by yourself.
I can say that a good fine-tuned image recognition algorithm doesn't take more than a few milliseconds to do what you want. I've experienced image recognition applications which do it almost instantaneously for larger screenshots (e.g. Eggplant Functional).
Now back to your code...
Your refactored code should look like below. I did not include all those fine-tuned algorithms I've mentioned - you should better ask separate questions for them in SO.
Image<Bgr, byte> source = new Image<Bgr, byte>(ofd.FileName);
// Preferably use Path.Combine here:
string dir = Path.Combine(Directory.GetCurrentDirectory(), "Images");
// Check whether directory exists:
if (!Directory.Exists(dir))
throw new Exception($"Directory was not found: '{dir}'");
// It looks like you just need filenames here...
// Simple parallel foreach suggested by HouseCat (in 2.):
Parallel.ForEach(Directory.GetFiles(dir), (fname) =>
{
Image<Gray, float> result = source.MatchTemplate(
new Image<Bgr, byte>(fname.FullName),
Emgu.CV.CvEnum.TemplateMatchingType.CcoeffNormed);
// By using C# 7.0, we can do inline out declarations here:
result.MinMax(
out double[] minValues,
out double[] maxValues,
out Point[] minLocations,
out Point[] maxLocations);
if (maxValues[0] > 0.96)
{
// ...
var result = ...
return result; // <<< As suggested by: denfromufa
}
// ...
});
Happy Tuning ;-)

This (denfromufa's answer) indeed explains your issue but to piggy back and add a few suggestions/optimizations as well:
1.) Your GetFiles can be replaced with a Parallel file enumerator, that is also recursive with children directories. I have shamelessly written a few on GitHub.
2.) You can parellelize the foreach loop into a Parallel.ForEach(files, fname () => { Code(); }); Again, my FileSearchBenchmark Repository on GitHub has plenty of File code execution in Parallel to provide examples.

Find blobs in bitmap

I use AForge.Net for find blobs in bitmap, my bitmap is as follows:
My problem is that AForge.Net detects only one blob when in fact there are two connected blobs on a thin line.
My question is there an algorithm that identifies that there are two large blobs with thin connection between them? And how I implement this algorithm in C# or VB?
Image for samples:

As others suggested, I would use OpenCv instead of AForge (it seems AForge has not been updated for a while plus OpenCv has lots of samples available).
With C#, I suggest the OpenCvSharp nuget package. It's easy to use because the code really looks like C++ or python code, like most samples.
So, OpenCv has a blob detector, but it detects blob centers, so in your case, it seems you're more after contours than blobs (which is often the case).
Luckily, with OpenCv and your sample image, it just works w/o doing anything fancy (we don't even have to erode the image first), we can just use findContours, filter some glitches, and get the convexHull. Here is a sample code that demonstrates that:
using (var src = new Mat(filePath))
using (var gray = new Mat())
{
using (var bw = src.CvtColor(ColorConversionCodes.BGR2GRAY)) // convert to grayscale
{
// invert b&w (specific to your white on black image)
Cv2.BitwiseNot(bw, gray);
}
// find all contours
var contours = gray.FindContoursAsArray(RetrievalModes.List, ContourApproximationModes.ApproxSimple);
using (var dst = src.Clone())
{
foreach (var contour in contours)
{
// filter small contours by their area
var area = Cv2.ContourArea(contour);
if (area < 15 * 15) // a rect of 15x15, or whatever you see fit
continue;
// also filter the whole image contour (by 1% close to the real area), there may be smarter ways...
if (Math.Abs((area - (src.Width * src.Height)) / area) < 0.01f)
continue;
var hull = Cv2.ConvexHull(contour);
Cv2.Polylines(dst, new[] { hull }, true, Scalar.Red, 2);
}
using (new Window("src image", src))
using (new Window("dst image", dst))
{
Cv2.WaitKey();
}
}
}

One quick solution would be to apply the opening operator
http://www.aforgenet.com/framework/features/morphology_filters.html
If the maximum thickness of the line is known in advance, one could apply the erosion operator multiple times and then apply the dilation operator the same number of times, effectively removing the thin line. This will change the shape of the 2 blobs, however.
If something more sophisticated is required, you might want to follow the approach in this, which combines the distance transform with the watershed algorithm:
https://docs.opencv.org/3.1.0/d3/db4/tutorial_py_watershed.html

Try Erosion Class , it can clear up thin line in the center.
http://www.aforgenet.com/framework/docs/html/90a69d73-0e5a-3e27-cc52-5864f542b53e.htm
Call Dilatation Class , get original size,
http://www.aforgenet.com/framework/docs/html/88f713d4-a469-30d2-dc57-5ceb33210723.htm
and find blobs again , you will get it.

Maybe you want to use OpenCV for your project. It's more easier and faster.
Nuget:
https://www.nuget.org/packages/OpenCvSharp3-AnyCPU/3.3.1.20171117
Mat im = Cv2.ImRead("blob.jpg", ImreadModes.GrayScale);
SimpleBlobDetector detector = SimpleBlobDetector.Create();
KeyPoint[] points = detector.Detect(im);
Mat result = new Mat();
Cv2.DrawKeypoints(im, points, result, Scalar.Red);

(EMGU) How do I split and merge an Image?

I am working in C# on Visual Studio with Emgu.
I am doing a several image manipulations on a large image. I had the idea of splitting the image in half, doing the manipulations in parallel, them merging the image.
In pursuit of this goal, I have found a number of questions regarding the acquisition of rectangular parts of images for processing as well as splitting an image into channels (RGB, HSV, etc). I have not found a question that addresses the task of taking an image, and making it into two images. I have also not found a question that addresses taking two images and tacking them together.
The following code is what I would like to do, where split and merge are imaginary methods to accomplish it.
Image<Bgr,Byte> ogImage = new Image<Bgr, byte>(request.image);
Image<Bgr,Byte> topHalf = new Image<Bgr, byte>();
Image<Bgr,Byte> bottomHalf = new Image<Bgr, byte>();
ogImage.splitHorizonally(topHalf,bottomHalf);
//operations
ogImage = topHalf.merge(bottomHalf);
This is the type of question I hate asking, because it is simple and you would think it has a simple, easily available solution, but I have not found it, or I have found it and not understood it.

There are a number of ways to solve this but here is what I did. I took the easiest way out ;-)
Mat lena = new Mat(#"D:\OpenCV\opencv-3.2.0\samples\data\Lena.jpg",
ImreadModes.Unchanged);
CvInvoke.Imshow("Lena", lena);
System.Drawing.Rectangle topRect = new Rectangle(0,
0,
lena.Width,
(lena.Height / 2));
System.Drawing.Rectangle bottomRect = new Rectangle(0,
(lena.Height / 2),
lena.Width,
(lena.Height / 2));
Mat lenaTop = new Mat(lena, topRect);
CvInvoke.Imshow("Lena Top", lenaTop);
Mat lenaBottom = new Mat(lena, bottomRect);
CvInvoke.Imshow("Lena Bottom", lenaBottom);
Mat newLena = new Mat();
CvInvoke.VConcat(lenaBottom, lenaTop, newLena);
CvInvoke.Imshow("New Lena", newLena);
CvInvoke.WaitKey(0);
Original Lena
Lena Top Half
Lena Bottom Half
The New Lena Rearranged

Your goal isn't splitting an image. Your goal is to parallelize some operation on the image.
You did not disclose the specific operations you need to perform. That is important to know however, if you want to parallelize those operations.
You need to learn about strategies for parallelization in general. Commonly, a "kernel" is executed on several partitions of the data in parallel.
One practical approach is called OpenMP. You apply "pragmas" to your own loops and OpenMP spreads those loop iterations across different threads.

Emgucv People Detection

I'm having problems with my person detection code. I'm using the Emgucv library, and it ends up picking up objects that have nothing to do with people ... Is something missing? How can I make it better?
Rectangle[] regions;
int indexx = 0;
using (HOGDescriptor des = new HOGDescriptor())
{
des.SetSVMDetector(HOGDescriptor.GetDefaultPeopleDetector());
regions = des.DetectMultiScale(currentFrame.Copy());
}
foreach (Rectangle rect in regions)
{
indexx++;
currentFrame.Draw(rect, new Bgr(Color.Red), 2);
}
if (regions.Count() > 0)
currentFrame.Save("pedestre\\pedestre0" + indexx + ".jpg");

The people detection results may vary, depending on your testing/application environment. Your best bet may be training the algorithm to work best on your specific conditions.
If you want the algorithm to work on very specific conditions, like a specific, closed and controlled environment, your best bet is training your algorithm to work on this environment.
If you want your detection to work on more varied conditions, you may have to search for articles about people detection or image filtering on the conditions you will meet on your application real world uses. You can try to do some image pre-processing before you use the classifier - sometimes a simple histogram normalizatior, for example, improves your results enough to be good enough for your application.

Perspective Image Distortion

The application I am working on currently requires functionality for Perspective Image Distortion. Basically what I want to do is to allow users to load an image into the application and adjust its perspective view properties based on 4 corner points that they can specify.
I had a look at ImageMagic. It has some distort functions with perpective adjustment but is very slow and some certain inputs are giving incorrect outputs.
Any of you guys used any other library or algorithm. I am coding in C#.
Any pointers would be much appreciated.
Thanks

This seems to be exactly what you (and I) were looking for:
http://www.codeproject.com/KB/graphics/YLScsFreeTransform.aspx
It will take an image and distort it using 4 X/Y coordinates you provide.
Fast, free, simple code. Tested and it works beautifully. Simply download the code from the link, then use FreeTransform.cs like this:
using (System.Drawing.Bitmap sourceImg = new System.Drawing.Bitmap(#"c:\image.jpg"))
{
YLScsDrawing.Imaging.Filters.FreeTransform filter = new YLScsDrawing.Imaging.Filters.FreeTransform();
filter.Bitmap = sourceImg;
// assign FourCorners (the four X/Y coords) of the new perspective shape
filter.FourCorners = new System.Drawing.PointF[] { new System.Drawing.PointF(0, 0), new System.Drawing.PointF(300, 50), new System.Drawing.PointF(300, 411), new System.Drawing.PointF(0, 461)};
filter.IsBilinearInterpolation = true; // optional for higher quality
using (System.Drawing.Bitmap perspectiveImg = filter.Bitmap)
{
// perspectiveImg contains your completed image. save the image or do whatever.
}
}

Paint .NET can do this and there are also custom implementations of the effect. You could ask for the source code or use Reflector to read it and get an idea of how to code it.

If it is a perspective transform, you should be able to specify a 4x4 transformation matrix that matches the four corners.
Calculate that matrix, then apply each pixel on the resulting image on the matrix, resulting in the "mapped" pixel. Notice that this "mapped" pixel is very likely going to lie between two or even four pixels. In this case, use your favorite interpolation algorithm (e.g. bilinear, bicubic) to get the interpolated color.
This really is the only way for it to be done and cannot be done faster. If this feature is crucial and you absolutely need it to be fast, then you'll need to offload the task to a GPU. For example, you can call upon the DirectX library to apply a perspective transformation on a texture. That can make it extremely fast, even when there is no GPU because the DirectX library uses SIMD instructions to accelerate matrix calculations and color interpolations.

Had the same problem. Here is the demo code with sources ported from gimp.

YLScsFreeTransform doesn't work as expected. Way better solution is ImageMagic
Here is how you use it in c#:
using(MagickImage image = new MagickImage("test.jpg"))
{
image.Distort(DistortMethod.Perspective, new double[] { x0,y0, newX0,newY0, x1,y1,newX1,newY1, x2,y2,newX2,newY2, x3,y3,newX3,newY3 });
control.Image = image.ToBitmap();
}