I develop a screen sharing app and i would like to make it as efficient as posibble so im trying to send only the differences between the screen shots.
So, suppose we have this image for example:its a 32bpprgba image with transpert parts around.
I would like to store each one of the blocks here as a rectangle in a List and get them bounds. It may sounds very complex but actually it just requires a little logic.
This is my code so far:
private unsafe List<Rectangle> CodeImage(Bitmap bmp)
{
List<Rectangle> rec = new List<Rectangle>();
Bitmap bmpRes = new Bitmap(bmp.Width, bmp.Height);
BitmapData bmData = bmp.LockBits(new Rectangle(0, 0, bmp.Width, bmp.Height), System.Drawing.Imaging.ImageLockMode.ReadOnly, bmp.PixelFormat);
IntPtr scan0 = bmData.Scan0;
int stride = bmData.Stride;
int nWidth = bmp.Width;
int nHeight = bmp.Height;
int minX = int.MaxValue; ;
int minY = int.MaxValue;
int maxX = 0;
bool found = false;
for (int y = 0; y < bmp.Height; y++)
{
byte* p = (byte*)scan0.ToPointer();
p += y * stride;
for (int x = 0; x < bmp.Width; x++)
{
if (p[3] != 0) //Check if pixel is not transparent;
{
found = true;
if (x < minX)
minX = x;
if (x > maxX)
maxX = x;
if (y < minY)
minY = y;
}
else
{
if (found)
{
int height = getBlockHeight(stride, scan0, maxX, minY);
found = false;
Rectangle temp = new Rectangle(minX, minY, maxX - minX, height);
rec.Add(temp);
y += minY;
break;
}
}
p += 4;//add to the pointer 4 bytes;
}
}
return rec;
}
as you see im trying to scan the image using the height and width, and when i found a pixel i send it to GetBlockHeight function to get it's height:
public unsafe int getBlockHeight(int stride, IntPtr scan, int x, int y1)
{
int height = 0; ;
for (int y = y1; y < 1080; y++)
{
byte* p = (byte*)scan.ToPointer();
p += (y * stride) + (x * 4);
if (p[3] != 0) //Check if pixel is not transparent;
{
height++;
}
}
return height;
}
But im just not getting the result... i think there's somthing with the logic here... can anyone light my eyes? i know it requires a bit time and thinking but i would very very appreciate anyone who could help a little.
In your current algorithm, after successfully matching a rectangle, you increase y with its height and break out of the inner loop. This means you can only detect data for one rectangle per horizontal line.
If I were you I'd think about the following things, before jumping back into the code:
Save the complete image as a PNG file, and look at its size. Is further processing really required?
Are these rectangles accurate? Will there be scenario's in which you would be constantly sending the contents of the entire screen anyway?
If you're developing for Windows, you might be able to hook into the procedure that invalidates areas on the screen, in which case you wouldn't have to determine these rectangles yourself. I don't know about other OSes
Also I personally wouldn't try to solve the rectangle-detection algorithm in a "nesty" for-loop, but go with something like this:
public void FindRectangles(Bitmap bitmap, Rectangle searchArea, List<Rectangle> results)
{
// Find the first non-transparent pixel within the search area.
// Ensure that it is the pixel with the lowest y-value possible
Point p;
if (!TryFindFirstNonTransparent(bitmap, searchArea, out p))
{
// No non-transparent pixels in this area
return;
}
// Find its rectangle within the area
Rectangle r = GetRectangle(bitmap, p, searchArea);
results.Add(r);
// No need to search above the rectangle we just found
Rectangle left = new Rectangle(searchArea.Left, r.Top, r.Left - searchArea.Left, searchArea.Bottom - r.Top);
Rectangle right = new Rectangle(r.Right, r.Top, searchArea.Right - r.Right, searchArea.Bottom - r.Top);
Rectangle bottom = new Rectangle(r.Left, r.Bottom, r.Width, searchArea.Bottom - r.Bottom);
FindRectangles(bitmap, left, results);
FindRectangles(bitmap, right, results);
FindRectangles(bitmap, bottom, results);
}
public Rectangle GetRectangle(Bitmap bitmap, Point p, Rectangle searchArea)
{
int right = searchArea.Right;
for (int x = p.X; x < searchArea.Right; x++)
{
if (IsTransparent(x, p.Y))
{
right = x - 1;
break;
}
}
int bottom = searchArea.Bottom;
for (int y = p.Y; y < searchArea.Bottom; y++)
{
if (IsTransparent(p.X, y))
{
bottom = y - 1;
break;
}
}
return new Rectangle(p.X, p.Y, right - p.X, bottom - p.Y);
}
This approach, when fully implemented, should give you a list of rectangles (although it will occasionally split a rectangle in two).
(Of course instead of providing the bitmap, you'd pass the pointer to the pixel data with some metadata instead)
I'm trying to scan 2 images (32bppArgb format), identify when there is a difference and store the difference block's bounds in a list of rectangles.
Suppose these are the images:
second:
I want to get the different rectangle bounds (the opened directory window in our case).
This is what I've done:
private unsafe List<Rectangle> CodeImage(Bitmap bmp, Bitmap bmp2)
{
List<Rectangle> rec = new List<Rectangle>();
bmData = bmp.LockBits(new System.Drawing.Rectangle(0, 0, 1920, 1080), System.Drawing.Imaging.ImageLockMode.ReadOnly, bmp.PixelFormat);
bmData2 = bmp2.LockBits(new System.Drawing.Rectangle(0, 0, 1920, 1080), System.Drawing.Imaging.ImageLockMode.ReadOnly, bmp2.PixelFormat);
IntPtr scan0 = bmData.Scan0;
IntPtr scan02 = bmData2.Scan0;
int stride = bmData.Stride;
int stride2 = bmData2.Stride;
int nWidth = bmp.Width;
int nHeight = bmp.Height;
int minX = int.MaxValue;;
int minY = int.MaxValue;
int maxX = 0;
bool found = false;
for (int y = 0; y < nHeight; y++)
{
byte* p = (byte*)scan0.ToPointer();
p += y * stride;
byte* p2 = (byte*)scan02.ToPointer();
p2 += y * stride2;
for (int x = 0; x < nWidth; x++)
{
if (p[0] != p2[0] || p[1] != p2[1] || p[2] != p2[2] || p[3] != p2[3]) //found differences-began to store positions.
{
found = true;
if (x < minX)
minX = x;
if (x > maxX)
maxX = x;
if (y < minY)
minY = y;
}
else
{
if (found)
{
int height = getBlockHeight(stride, scan0, maxX, minY, scan02, stride2);
found = false;
Rectangle temp = new Rectangle(minX, minY, maxX - minX, height);
rec.Add(temp);
//x += minX;
y += height;
minX = int.MaxValue;
minY = int.MaxValue;
maxX = 0;
}
}
p += 4;
p2 += 4;
}
}
return rec;
}
public unsafe int getBlockHeight(int stride, IntPtr scan, int x, int y1, IntPtr scan02, int stride2) //a function to get an existing block height.
{
int height = 0;;
for (int y = y1; y < 1080; y++) //only for example- in our case its 1080 height.
{
byte* p = (byte*)scan.ToPointer();
p += (y * stride) + (x * 4); //set the pointer to a specific potential point.
byte* p2 = (byte*)scan02.ToPointer();
p2 += (y * stride2) + (x * 4); //set the pointer to a specific potential point.
if (p[0] != p2[0] || p[1] != p2[1] || p[2] != p2[2] || p[3] != p2[3]) //still change on the height in the increasing **y** of the block.
height++;
}
return height;
}
This is actually how I call the method:
Bitmap a = Image.FromFile(#"C:\Users\itapi\Desktop\1.png") as Bitmap;//generates a 32bppRgba bitmap;
Bitmap b = Image.FromFile(#"C:\Users\itapi\Desktop\2.png") as Bitmap;//
List<Rectangle> l1 = CodeImage(a, b);
int i = 0;
foreach (Rectangle rec in l1)
{
i++;
Bitmap tmp = b.Clone(rec, a.PixelFormat);
tmp.Save(i.ToString() + ".png");
}
But I'm not getting the exact rectangle.. I'm getting only half of that and sometimes even worse. I think something in the code's logic is wrong.
Code for #nico
private unsafe List<Rectangle> CodeImage(Bitmap bmp, Bitmap bmp2)
{
List<Rectangle> rec = new List<Rectangle>();
var bmData1 = bmp.LockBits(new System.Drawing.Rectangle(0, 0, bmp.Width, bmp.Height), System.Drawing.Imaging.ImageLockMode.ReadOnly, bmp.PixelFormat);
var bmData2 = bmp2.LockBits(new System.Drawing.Rectangle(0, 0, bmp.Width, bmp.Height), System.Drawing.Imaging.ImageLockMode.ReadOnly, bmp2.PixelFormat);
int bytesPerPixel = 3;
IntPtr scan01 = bmData1.Scan0;
IntPtr scan02 = bmData2.Scan0;
int stride1 = bmData1.Stride;
int stride2 = bmData2.Stride;
int nWidth = bmp.Width;
int nHeight = bmp.Height;
bool[] visited = new bool[nWidth * nHeight];
byte* base1 = (byte*)scan01.ToPointer();
byte* base2 = (byte*)scan02.ToPointer();
for (int y = 0; y < nHeight; y += 5)
{
byte* p1 = base1;
byte* p2 = base2;
for (int x = 0; x < nWidth; x += 5)
{
if (!ArePixelsEqual(p1, p2, bytesPerPixel) && !(visited[x + nWidth * y]))
{
// fill the different area
int minX = x;
int maxX = x;
int minY = y;
int maxY = y;
var pt = new Point(x, y);
Stack<Point> toBeProcessed = new Stack<Point> ();
visited[x + nWidth * y] = true;
toBeProcessed.Push(pt);
while (toBeProcessed.Count > 0)
{
var process = toBeProcessed.Pop();
var ptr1 = (byte*)scan01.ToPointer() + process.Y * stride1 + process.X * bytesPerPixel;
var ptr2 = (byte*) scan02.ToPointer() + process.Y * stride2 + process.X * bytesPerPixel;
//Check pixel equality
if (ArePixelsEqual(ptr1, ptr2, bytesPerPixel))
continue;
//This pixel is different
//Update the rectangle
if (process.X < minX) minX = process.X;
if (process.X > maxX) maxX = process.X;
if (process.Y < minY) minY = process.Y;
if (process.Y > maxY) maxY = process.Y;
Point n;
int idx;
//Put neighbors in stack
if (process.X - 1 >= 0)
{
n = new Point(process.X - 1, process.Y);
idx = n.X + nWidth * n.Y;
if (!visited[idx])
{
visited[idx] = true;
toBeProcessed.Push(n);
}
}
if (process.X + 1 < nWidth)
{
n = new Point(process.X + 1, process.Y);
idx = n.X + nWidth * n.Y;
if (!visited[idx])
{
visited[idx] = true;
toBeProcessed.Push(n);
}
}
if (process.Y - 1 >= 0)
{
n = new Point(process.X, process.Y - 1);
idx = n.X + nWidth * n.Y;
if (!visited[idx])
{
visited[idx] = true;
toBeProcessed.Push(n);
}
}
if (process.Y + 1 < nHeight)
{
n = new Point(process.X, process.Y + 1);
idx = n.X + nWidth * n.Y;
if (!visited[idx])
{
visited[idx] = true;
toBeProcessed.Push(n);
}
}
}
if (((maxX - minX + 1) > 5) & ((maxY - minY + 1) > 5))
rec.Add(new Rectangle(minX, minY, maxX - minX + 1, maxY - minY + 1));
}
p1 += 5 * bytesPerPixel;
p2 += 5 * bytesPerPixel;
}
base1 += 5 * stride1;
base2 += 5 * stride2;
}
bmp.UnlockBits(bmData1);
bmp2.UnlockBits(bmData2);
return rec;
}
I see a couple of problems with your code. If I understand it correctly, you
find a pixel that's different between the two images.
then you continue to scan from there to the right, until you find a position where both images are identical again.
then you scan from the last "different" pixel to the bottom, until you find a position where both images are identical again.
then you store that rectangle and start at the next line below it
Am I right so far?
Two obvious things can go wrong here:
If two rectangles have overlapping y-ranges, you're in trouble: You'll find the first rectangle fine, then skip to the bottom Y-coordinate, ignoring all the pixels left or right of the rectangle you just found.
Even if there is only one rectangle, you assume that every pixel on the rectangle's border is different, and all the other pixels are identical. If that assumption isn't valid, you'll stop searching too early, and only find parts of rectangles.
If your images come from a scanner or digital camera, or if they contain lossy compression (jpeg) artifacts, the second assumption will almost certainly be wrong. To illustrate this, here's what I get when I mark every identical pixel the two jpg images you linked black, and every different pixel white:
What you see is not a rectangle. Instead, a lot of pixels around the rectangles you're looking for are different:
That's because of jpeg compression artifacts. But even if you used lossless source images, pixels at the borders might not form perfect rectangles, because of antialiasing or because the background just happens to have a similar color in that region.
You could try to improve your algorithm, but if you look at that border, you will find all kinds of ugly counterexamples to any geometric assumptions you'll make.
It would probably be better to implement this "the right way". Meaning:
Either implement a flood fill algorithm that erases different pixels (e.g. by setting them to identical or by storing a flag in a separate mask), then recursively checks if the 4 neighbor pixels.
Or implement a connected component labeling algorithm, that marks each different pixel with a temporary integer label, using clever data structures to keep track which temporary labels are connected. If you're only interested in a bounding box, you don't even have to merge the temporary labels, just merge the bounding boxes of adjacent labeled areas.
Connected component labeling is in general a bit faster, but is a bit trickier to get right than flood fill.
One last advice: I would rethink your "no 3rd party libraries" policy if I were you. Even if your final product will contain no 3rd party libraries, development might by a lot faster if you used well-documented, well-tested, useful building blocks from a library, then replaced them one by one with your own code. (And who knows, you might even find an open source library with a suitable license that's so much faster than your own code that you'll stick with it in the end...)
ADD: In case you want to rethink your "no libraries" position: Here's a quick and simple implementation using AForge (which has a more permissive library than emgucv):
private static void ProcessImages()
{
(* load images *)
var img1 = AForge.Imaging.Image.FromFile(#"compare1.jpg");
var img2 = AForge.Imaging.Image.FromFile(#"compare2.jpg");
(* calculate absolute difference *)
var difference = new AForge.Imaging.Filters.ThresholdedDifference(15)
{OverlayImage = img1}
.Apply(img2);
(* create and initialize the blob counter *)
var bc = new AForge.Imaging.BlobCounter();
bc.FilterBlobs = true;
bc.MinWidth = 5;
bc.MinHeight = 5;
(* find blobs *)
bc.ProcessImage(difference);
(* draw result *)
BitmapData data = img2.LockBits(
new Rectangle(0, 0, img2.Width, img2.Height),
ImageLockMode.ReadWrite, img2.PixelFormat);
foreach (var rc in bc.GetObjectsRectangles())
AForge.Imaging.Drawing.FillRectangle(data, rc, Color.FromArgb(128,Color.Red));
img2.UnlockBits(data);
img2.Save(#"compareResult.jpg");
}
The actual difference + blob detection part (without loading and result display) takes about 43ms, for the second run (this first time takes longer of course, due to JITting, cache, etc.)
Result (the rectangle is larger due to jpeg artifacts):
Here is a flood-fill based version of your code. It checks every pixel for difference. If it finds a different pixel, it runs an exploration to find the entire different area.
The code is only meant as an illustration. There are certainly some points that could be improved.
unsafe bool ArePixelsEqual(byte* p1, byte* p2, int bytesPerPixel)
{
for (int i = 0; i < bytesPerPixel; ++i)
if (p1[i] != p2[i])
return false;
return true;
}
private static unsafe List<Rectangle> CodeImage(Bitmap bmp, Bitmap bmp2)
{
if (bmp.PixelFormat != bmp2.PixelFormat || bmp.Width != bmp2.Width || bmp.Height != bmp2.Height)
throw new ArgumentException();
List<Rectangle> rec = new List<Rectangle>();
var bmData1 = bmp.LockBits(new System.Drawing.Rectangle(0, 0, bmp.Width, bmp.Height), System.Drawing.Imaging.ImageLockMode.ReadOnly, bmp.PixelFormat);
var bmData2 = bmp2.LockBits(new System.Drawing.Rectangle(0, 0, bmp.Width, bmp.Height), System.Drawing.Imaging.ImageLockMode.ReadOnly, bmp2.PixelFormat);
int bytesPerPixel = Image.GetPixelFormatSize(bmp.PixelFormat) / 8;
IntPtr scan01 = bmData1.Scan0;
IntPtr scan02 = bmData2.Scan0;
int stride1 = bmData1.Stride;
int stride2 = bmData2.Stride;
int nWidth = bmp.Width;
int nHeight = bmp.Height;
bool[] visited = new bool[nWidth * nHeight];
byte* base1 = (byte*)scan01.ToPointer();
byte* base2 = (byte*)scan02.ToPointer();
for (int y = 0; y < nHeight; y++)
{
byte* p1 = base1;
byte* p2 = base2;
for (int x = 0; x < nWidth; ++x)
{
if (!ArePixelsEqual(p1, p2, bytesPerPixel) && !(visited[x + nWidth * y]))
{
// fill the different area
int minX = x;
int maxX = x;
int minY = y;
int maxY = y;
var pt = new Point(x, y);
Stack<Point> toBeProcessed = new Stack<Point>();
visited[x + nWidth * y] = true;
toBeProcessed.Push(pt);
while (toBeProcessed.Count > 0)
{
var process = toBeProcessed.Pop();
var ptr1 = (byte*)scan01.ToPointer() + process.Y * stride1 + process.X * bytesPerPixel;
var ptr2 = (byte*)scan02.ToPointer() + process.Y * stride2 + process.X * bytesPerPixel;
//Check pixel equality
if (ArePixelsEqual(ptr1, ptr2, bytesPerPixel))
continue;
//This pixel is different
//Update the rectangle
if (process.X < minX) minX = process.X;
if (process.X > maxX) maxX = process.X;
if (process.Y < minY) minY = process.Y;
if (process.Y > maxY) maxY = process.Y;
Point n; int idx;
//Put neighbors in stack
if (process.X - 1 >= 0)
{
n = new Point(process.X - 1, process.Y); idx = n.X + nWidth * n.Y;
if (!visited[idx]) { visited[idx] = true; toBeProcessed.Push(n); }
}
if (process.X + 1 < nWidth)
{
n = new Point(process.X + 1, process.Y); idx = n.X + nWidth * n.Y;
if (!visited[idx]) { visited[idx] = true; toBeProcessed.Push(n); }
}
if (process.Y - 1 >= 0)
{
n = new Point(process.X, process.Y - 1); idx = n.X + nWidth * n.Y;
if (!visited[idx]) { visited[idx] = true; toBeProcessed.Push(n); }
}
if (process.Y + 1 < nHeight)
{
n = new Point(process.X, process.Y + 1); idx = n.X + nWidth * n.Y;
if (!visited[idx]) { visited[idx] = true; toBeProcessed.Push(n); }
}
}
rec.Add(new Rectangle(minX, minY, maxX - minX + 1, maxY - minY + 1));
}
p1 += bytesPerPixel;
p2 += bytesPerPixel;
}
base1 += stride1;
base2 += stride2;
}
bmp.UnlockBits(bmData1);
bmp2.UnlockBits(bmData2);
return rec;
}
You can achieve this easily using a flood fill segmentation algorithm.
First an utility class to make fast bitmap access easier. This will help to encapsulate the complex pointer-logic and make the code more readable:
class BitmapWithAccess
{
public Bitmap Bitmap { get; private set; }
public System.Drawing.Imaging.BitmapData BitmapData { get; private set; }
public BitmapWithAccess(Bitmap bitmap, System.Drawing.Imaging.ImageLockMode lockMode)
{
Bitmap = bitmap;
BitmapData = bitmap.LockBits(new Rectangle(Point.Empty, bitmap.Size), lockMode, System.Drawing.Imaging.PixelFormat.Format32bppArgb);
}
public Color GetPixel(int x, int y)
{
unsafe
{
byte* dataPointer = MovePointer((byte*)BitmapData.Scan0, x, y);
return Color.FromArgb(dataPointer[3], dataPointer[2], dataPointer[1], dataPointer[0]);
}
}
public void SetPixel(int x, int y, Color color)
{
unsafe
{
byte* dataPointer = MovePointer((byte*)BitmapData.Scan0, x, y);
dataPointer[3] = color.A;
dataPointer[2] = color.R;
dataPointer[1] = color.G;
dataPointer[0] = color.B;
}
}
public void Release()
{
Bitmap.UnlockBits(BitmapData);
BitmapData = null;
}
private unsafe byte* MovePointer(byte* pointer, int x, int y)
{
return pointer + x * 4 + y * BitmapData.Stride;
}
}
Then a class representing a rectangle containing different pixels, to mark them in the resulting image. In general this class can also contain a list of Point instances (or a byte[,] map) to make indicating individual pixels in the resulting image possible:
class Segment
{
public int Left { get; set; }
public int Top { get; set; }
public int Right { get; set; }
public int Bottom { get; set; }
public Bitmap Bitmap { get; set; }
public Segment()
{
Left = int.MaxValue;
Right = int.MinValue;
Top = int.MaxValue;
Bottom = int.MinValue;
}
};
Then the steps of a simple algorithm are as follows:
find different pixels
use a flood-fill algorithm to find segments on the difference image
draw bounding rectangles for the segments found
The first step is the easiest one:
static Bitmap FindDifferentPixels(Bitmap i1, Bitmap i2)
{
var result = new Bitmap(i1.Width, i2.Height, System.Drawing.Imaging.PixelFormat.Format32bppArgb);
var ia1 = new BitmapWithAccess(i1, System.Drawing.Imaging.ImageLockMode.ReadOnly);
var ia2 = new BitmapWithAccess(i2, System.Drawing.Imaging.ImageLockMode.ReadOnly);
var ra = new BitmapWithAccess(result, System.Drawing.Imaging.ImageLockMode.ReadWrite);
for (int x = 0; x < i1.Width; ++x)
for (int y = 0; y < i1.Height; ++y)
{
var different = ia1.GetPixel(x, y) != ia2.GetPixel(x, y);
ra.SetPixel(x, y, different ? Color.White : Color.FromArgb(0, 0, 0, 0));
}
ia1.Release();
ia2.Release();
ra.Release();
return result;
}
And the second and the third steps are covered with the following three functions:
static List<Segment> Segmentize(Bitmap blackAndWhite)
{
var bawa = new BitmapWithAccess(blackAndWhite, System.Drawing.Imaging.ImageLockMode.ReadOnly);
var result = new List<Segment>();
HashSet<Point> queue = new HashSet<Point>();
bool[,] visitedPoints = new bool[blackAndWhite.Width, blackAndWhite.Height];
for (int x = 0;x < blackAndWhite.Width;++x)
for (int y = 0;y < blackAndWhite.Height;++y)
{
if (bawa.GetPixel(x, y).A != 0
&& !visitedPoints[x, y])
{
result.Add(BuildSegment(new Point(x, y), bawa, visitedPoints));
}
}
bawa.Release();
return result;
}
static Segment BuildSegment(Point startingPoint, BitmapWithAccess bawa, bool[,] visitedPoints)
{
var result = new Segment();
List<Point> toProcess = new List<Point>();
toProcess.Add(startingPoint);
while (toProcess.Count > 0)
{
Point p = toProcess.First();
toProcess.RemoveAt(0);
ProcessPoint(result, p, bawa, toProcess, visitedPoints);
}
return result;
}
static void ProcessPoint(Segment segment, Point point, BitmapWithAccess bawa, List<Point> toProcess, bool[,] visitedPoints)
{
for (int i = -1; i <= 1; ++i)
{
for (int j = -1; j <= 1; ++j)
{
int x = point.X + i;
int y = point.Y + j;
if (x < 0 || y < 0 || x >= bawa.Bitmap.Width || y >= bawa.Bitmap.Height)
continue;
if (bawa.GetPixel(x, y).A != 0 && !visitedPoints[x, y])
{
segment.Left = Math.Min(segment.Left, x);
segment.Right = Math.Max(segment.Right, x);
segment.Top = Math.Min(segment.Top, y);
segment.Bottom = Math.Max(segment.Bottom, y);
toProcess.Add(new Point(x, y));
visitedPoints[x, y] = true;
}
}
}
}
And the following program given your two images as arguments:
static void Main(string[] args)
{
Image ai1 = Image.FromFile(args[0]);
Image ai2 = Image.FromFile(args[1]);
Bitmap i1 = new Bitmap(ai1.Width, ai1.Height, System.Drawing.Imaging.PixelFormat.Format32bppArgb);
Bitmap i2 = new Bitmap(ai2.Width, ai2.Height, System.Drawing.Imaging.PixelFormat.Format32bppArgb);
using (var g1 = Graphics.FromImage(i1))
using (var g2 = Graphics.FromImage(i2))
{
g1.DrawImage(ai1, Point.Empty);
g2.DrawImage(ai2, Point.Empty);
}
var difference = FindDifferentPixels(i1, i2);
var segments = Segmentize(difference);
using (var g1 = Graphics.FromImage(i1))
{
foreach (var segment in segments)
{
g1.DrawRectangle(Pens.Red, new Rectangle(segment.Left, segment.Top, segment.Right - segment.Left, segment.Bottom - segment.Top));
}
}
i1.Save("result.png");
Console.WriteLine("Done.");
Console.ReadKey();
}
produces the following result:
As you can see there are more differences between the given images. You can filter the resulting segments with regard to their size for example to drop the small artefacts. Also there is of course much work to do in terms of error checking, design and performance.
One idea is to proceed as follows:
1) Rescale images to a smaller size (downsample)
2) Run the above algorithm on smaller images
3) Run the above algorithm on original images, but restricting yourself only to rectangles found in step 2)
This can be of course extended to a multi-level hierarchical approach (using more different image sizes, increasing accuracy with each step).
Ah an algorithm challenge. Like! :-)
There are other answers here using f.ex. floodfill that will work just fine. I just noticed that you wanted something fast, so let me propose a different idea. Unlike the other people, I haven't tested it; it shouldn't be too hard and should be quite fast, but I simply don't have the time at the moment to test it myself. If you do, please share the results. Also, note that it's not a standard algorithm, so there are probably some bugs here and there in my explanation (and no patents).
My idea is derived from the idea of mean adaptive thresholding but with a lot of important differences. I cannot find the link from wikipedia anymore or my code, so I'll do this from the top of my mind. Basically you create a new (64-bit) buffer for both images and fill it with:
f(x,y) = colorvalue + f(x-1, y) + f(x, y-1) - f(x-1, y-1)
f(x,0) = colorvalue + f(x-1, 0)
f(0,y) = colorvalue + f(0, y-1)
The main trick is that you can calculate the sum value of a portion of the image fast, namely by:
g(x1,y1,x2,y2) = f(x2,y2)-f(x1-1,y2)-f(x2,y1-1)+f(x1-1,y1-1)
In other words, this will give the same result as:
result = 0;
for (x=x1; x<=x2; ++x)
for (y=y1; y<=y2; ++y)
result += f(x,y)
In our case this means that with only 4 integer operations this will get you some unique number of the block in question. I'd say that's pretty awesome.
Now, in our case, we don't really care about the average value; we just care about some sort-of unique number. If the image changes, it should change - simple as that. As for colorvalue, usually some gray scale number is used for thresholding - instead, we'll be using the complete 24-bit RGB value. Because there are only so few compares, we can simply scan until we find a block that doesn't match.
The basic algorithm that I propose works as follows:
for (y=0; y<height;++y)
for (x=0; x<width; ++x)
if (src[x,y] != dst[x,y])
if (!IntersectsWith(x, y, foundBlocks))
FindBlock(foundBlocks);
Now, IntersectsWith can be something like a quad tree of if there are only a few blocks, you can simply iterate through the blocks and check if they are within the bounds of the block. You can also update the x variable accordingly (I would). You can even balance things by re-building the buffer for f(x,y) if you have too many blocks (more precise: merge found blocks back from dst into src, then rebuild the buffer).
FindBlocks is where it gets interesting. Using the formula for g that's now pretty easy:
int x1 = x-1; int y1 = y-1; int x2 = x; int y2 = y;
while (changes)
{
while (g(srcimage,x1-1,y1,x1,y2) == g(dstimage,x1-1,y1,x1,y2)) { --x1; }
while (g(srcimage,x1,y1-1,x1,y2) == g(dstimage,x1,y1-1,x1,y2)) { --y1; }
while (g(srcimage,x1,y1,x1+1,y2) == g(dstimage,x1,y1,x1+1,y2)) { ++x1; }
while (g(srcimage,x1,y1,x1,y2+1) == g(dstimage,x1,y1,x1,y2+1)) { ++y1; }
}
That's it. Note that the complexity of the FindBlocks algorithm is O(x + y), which is pretty awesome for finding a 2D block IMO. :-)
As I said, let me know how it turns out.
This is what i did in form1 constructor:
Bitmap bmp2 = new Bitmap(#"e:\result1001.jpg");
CropImageWhiteAreas.ImageTrim(bmp2);
bmp2.Save(#"e:\result1002.jpg");
bmp2.Dispose();
The class CropImageWhiteAreas:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Drawing;
using System.Drawing.Imaging;
using System.Runtime.InteropServices;
namespace Test
{
class CropImageWhiteAreas
{
public static Bitmap ImageTrim(Bitmap img)
{
//get image data
BitmapData bd = img.LockBits(new Rectangle(Point.Empty, img.Size),
ImageLockMode.ReadOnly, PixelFormat.Format32bppArgb);
int[] rgbValues = new int[img.Height * img.Width];
Marshal.Copy(bd.Scan0, rgbValues, 0, rgbValues.Length);
img.UnlockBits(bd);
#region determine bounds
int left = bd.Width;
int top = bd.Height;
int right = 0;
int bottom = 0;
//determine top
for (int i = 0; i < rgbValues.Length; i++)
{
int color = rgbValues[i] & 0xffffff;
if (color != 0xffffff)
{
int r = i / bd.Width;
int c = i % bd.Width;
if (left > c)
{
left = c;
}
if (right < c)
{
right = c;
}
bottom = r;
top = r;
break;
}
}
//determine bottom
for (int i = rgbValues.Length - 1; i >= 0; i--)
{
int color = rgbValues[i] & 0xffffff;
if (color != 0xffffff)
{
int r = i / bd.Width;
int c = i % bd.Width;
if (left > c)
{
left = c;
}
if (right < c)
{
right = c;
}
bottom = r;
break;
}
}
if (bottom > top)
{
for (int r = top + 1; r < bottom; r++)
{
//determine left
for (int c = 0; c < left; c++)
{
int color = rgbValues[r * bd.Width + c] & 0xffffff;
if (color != 0xffffff)
{
if (left > c)
{
left = c;
break;
}
}
}
//determine right
for (int c = bd.Width - 1; c > right; c--)
{
int color = rgbValues[r * bd.Width + c] & 0xffffff;
if (color != 0xffffff)
{
if (right < c)
{
right = c;
break;
}
}
}
}
}
int width = right - left + 1;
int height = bottom - top + 1;
#endregion
//copy image data
int[] imgData = new int[width * height];
for (int r = top; r <= bottom; r++)
{
Array.Copy(rgbValues, r * bd.Width + left, imgData, (r - top) * width, width);
}
//create new image
Bitmap newImage = new Bitmap(width, height, PixelFormat.Format32bppArgb);
BitmapData nbd
= newImage.LockBits(new Rectangle(0, 0, width, height),
ImageLockMode.WriteOnly, PixelFormat.Format32bppArgb);
Marshal.Copy(imgData, 0, nbd.Scan0, imgData.Length);
newImage.UnlockBits(nbd);
return newImage;
}
}
}
I also tried before it Peter solution.
In both the result is(This is a screenshot of my facebook after uploaded the image) still the white areas around:
You can the rectangle around the image i just uploaded and see what i mean by white area around.
If I understand correctly, you have found a sample code snippet that uses LockBits(), but you are not sure how it works or how to modify it to suit your specific need. So I will try to answer from that perspective.
First, a wild guess (since you didn't include the implementation of the LockBitmap class you're using in the first example): the LockBitmap class is some kind of helper class that is supposed to encapsulate the work of calling LockBits() and using the result, including providing versions of GetPixel() and SetPixel() which are presumably much faster than calling those methods on a Bitmap object directly (i.e. access the buffer obtained by calling LockBits()).
If that's the case, then modifying the first example to suit your need is probably best:
public void Change(Bitmap bmp)
{
Bitmap newBitmap = new Bitmap(bmp.Width, bmp.Height, bmp.PixelFormat);
LockBitmap source = new LockBitmap(bmp),
target = new LockBitmap(newBitmap);
source.LockBits();
target.LockBits();
Color white = Color.FromArgb(255, 255, 255, 255);
for (int y = 0; y < source.Height; y++)
{
for (int x = 0; x < source.Width; x++)
{
Color old = source.GetPixel(x, y);
if (old != white)
{
target.SetPixel(x, y, old);
}
}
}
source.UnlockBits();
target.UnlockBits();
newBitmap.Save("d:\\result.png");
}
In short: copy the current pixel value to a local variable, compare that value to the white color value, and if it is not the same, go ahead and copy the pixel value to the new bitmap.
Some variation on the second code example should work as well. The second code example does explicitly what is (I've assumed) encapsulated inside the LockBitmap class that the first code example uses. If for some reason, the first approach isn't suitable for your needs, you can follow the second example.
In that code example you provide, most of the method there is just handling the "grunt work" of locking the bitmap so that the raw data can be accessed, and then iterating through that raw data.
It computes the oRow and nRow array offsets (named for "old row" and "new row", I presume) based on the outer y loop, and then accesses individual pixel data by computing the offset within a given row based on the inner x loop.
Since you want to do essentially the same thing, but instead of converting the image to grayscale, you just want to selectively copy all non-white pixels to the new bitmap, you can (should be able to) simply modify the body of the inner x loop. For example:
byte red = oRow[x * pixelSize + 2],
green = oRow[x * pixelSize + 1],
blue = oRow[x * pixelSize];
if (red != 255 || green != 255 || blue != 255)
{
nRow[x * pixelSize + 2] = red;
nRow[x * pixelSize + 1] = green;
nRow[x * pixelSize] = blue;
}
The above would entirely replace the body of the inner x loop.
One caveat: do note that when using the LockBits() approach, knowing the pixel format of the bitmap is crucial. The example you've shown assumes the bitmaps are in 24 bpp format. If your own bitmaps are in this format, then you don't need to change anything. But if they are in a different format, you'll need to adjust the code to suit that. For example, if your bitmap is in 32 bpp format, you need to pass the correct PixelFormat value to the LockBits() method calls, and then set pixelSize to 4 instead of 3 as the code does now.
Edit:
You've indicated that you would like to crop the new image so that it is the minimize size required to contain all of the non-white pixels. Here is a version of the first example above that should accomplish that:
public void Change(Bitmap bmp)
{
LockBitmap source = new LockBitmap(bmp);
source.LockBits();
Color white = Color.FromArgb(255, 255, 255, 255);
int minX = int.MaxValue, maxX = int.MinValue,
minY = int.MaxValue, maxY = int.MinValue;
// Brute-force scan of the bitmap to find image boundary
for (int y = 0; y < source.Height; y++)
{
for (int x = 0; x < source.Width; x++)
{
if (source.GetPixel(x, y) != white)
{
if (x < minX) minX = x;
if (x > maxX) maxX = x;
if (y < minY) minY = y;
if (y > maxY) maxY = y;
}
}
}
Bitmap newBitmap = new Bitmap(maxX - minx + 1, maxY - minY + 1, bmp.PixelFormat);
LockBitmap target = new LockBitmap(newBitmap);
target.LockBits();
for (int y = 0; y < target.Height; y++)
{
for (int x = 0; x < target.Width; x++)
{
target.SetPixel(x, y, source.GetPixel(x + minX, y + minY));
}
}
source.UnlockBits();
target.UnlockBits();
newBitmap.Save("d:\\result.png");
}
This example includes an initial scan of the original bitmap, after locking it, to find the minimum and maximum coordinate values for any non-white pixel. Having done that, it uses the results of that scan to determine the dimensions of the new bitmap. When copying the pixels, it restricts the x and y loops to the dimensions of the new bitmap, adjusting the x and y values to map from the location in the new bitmap to the given pixel's original location in the old one.
Note that since the initial scan determines where the non-white pixels are, there's no need to check again when actually copying the pixels.
There are more efficient ways to scan the bitmap than the above. This version simply looks at every single pixel in the original bitmap, keeping track of the min and max values for each coordinate. I'm guessing this will be fast enough for your purposes, but if you want something faster, you can change the scan so that it scans for each min and max in sequence:
Scan each row from y of 0 to determine the first row with a non-white pixel. This is the min y value.
Scan each row from y of source.Height - 1 backwards, to find the max y value.
Having found the min and max y values, now scan the columns from x of 0 to find the min x and from source.Width - 1 backwards to find the max x.
Doing it that way involves a lot more code and is probably harder to read and understand, but would involve inspecting many fewer pixels in most cases.
Edit #2:
Here is a sample of the output of the second code example:
Note that all of the white border of the original bitmap (shown on the left side) has been cropped out, leaving only the smallest subset of the original bitmap that can contain all of the non-white pixels (shown on the right side).
lets say that i have this image:
http://srv2.jpg.co.il/9/51c614f7c280e.png
i want to get the white ball position(x,y),
this is a verey big image, then i cut the image by rectangle.(because when the image is smaller everything is faster),
the results:
http://srv2.jpg.co.il/1/51c616787a3fa.png
now i want to track the white ball position by his color(white=rbg(255,255,255)),
my code:
Public Function GetBallPosition(ByRef HaxScreenOnly As Bitmap) As Point
For y = 0 To HaxScreenOnly.Height - 1
For x = 0 To HaxScreenOnly.Width - 1
If HaxScreenOnly.GetPixel(x, y) = Color.FromArgb(0, 0, 0) Then
If HaxScreenOnly.GetPixel(x + 8, y) = Color.FromArgb(0, 0, 0) And HaxScreenOnly.GetPixel(x + 8, y + 3) = Color.FromArgb(255, 255, 255) Then
Return New Point(x, y)
End If
End If
Next
Next
Return New Point(0, 0)
End Function
If the color of the current pixel is black and the color of the current pixel(x+8,y+3) is white then this is the ball
it's working...but its verey slow, something like 200 miliseconds to track the ball position.
this is not fast enough.
there is faster way to track the white ball(C# or VB.net)?
Finally, I have you a solution for you. Calling GetPixel is a costly process, but you use Bitmap.LockBits and manipulate / access the image data from a pointer. I took the LockBitmap class from this article.
I checked on the performance from what I was getting previously, which was exactly like you mentioned, around 200ms~.
Here is a picture of the result using LockBitmap, rescanning the image continuously with the optimized code!
static void Main(string[] args)
{
byte[] data = new WebClient().DownloadData("http://srv2.jpg.co.il/1/51c616787a3fa.png");
Image image = Image.FromStream(new MemoryStream(data));
LockBitmap bitmap = new LockBitmap(new Bitmap(image));
// this essentially copies the data into memory and copies from a pointer to an array
bitmap.LockBits();
Color black = Color.FromArgb(0, 0, 0);
Color white = Color.FromArgb(255, 255, 255);
Stopwatch stopwatch = Stopwatch.StartNew();
for (int y = 0; y < bitmap.Height; y++)
{
for (int x = 0; x < bitmap.Width; x++)
{
// GetPixel is a nice abstraction the author in the Article created so we don't have to do any of the gritty stuff.
if (bitmap.GetPixel(x, y) == black)
{
if (bitmap.GetPixel(x + 8, y) == black && bitmap.GetPixel(x + 8, y + 3) == white)
{
Console.WriteLine("White Ball Found in {0}", stopwatch.Elapsed.ToString());
break;
}
}
}
}
bitmap.UnlockBits(); // copies the data from the array back to the original pointer
Console.Read();
}
Hope this helps, it was certainly an interesting read for me.
Update:
As mentioned by King, I was able to further reduce the timing for you based on the algorithm improvement. So we've gone from O(n) to O(n log) time complexity (I think).
for (int y = 0; y < bitmap.Height; y += 3) // As we know the radius of the ball
{
for (int x = 0; x < bitmap.Width; x += 3) // We can increase this
{
if (bitmap.GetPixel(x, y) == black && bitmap.GetPixel(x, y + 3) == white)
{
Console.WriteLine("White Ball Found ({0},{1}) in {2}", x, y, stopwatch.Elapsed.ToString());
break;
}
}
}