Develop Reference

OpenCV C# Frame Differencing

I'm new to computer vision and currently playing around with static frame differencing to try and determine whether there is motion in video.
My variables:
public Mat currentFrame = new Mat();
public Mat prevFrame = new Mat();
public Mat result = new Mat();
bool motion = false;
Simple differencing function (being called every frame):
public Mat getDifference(Mat videoFrame)
{
currentFrame = videoFrame.Clone();
Cv2.Absdiff(currentFrame, prevFrame, result);
prevFrame = currentFrame.Clone();
return result;
}
When motion exists the result matrix looks like this:
When motion doesn't exist the result matrix looks like this (empty):
My original idea was that if the result matrix is effectively empty (all black), then I could say motion = false. However, this is proving to be more difficult that anticipated since it is technically never empty, so I can't say:
if(!result.Empty())
{
motion = true;
}
Without the need of for loops and pixel by pixel analysis, is there a simple/clean 'if' statement I can use that simply says (if the matrix contains anything that isn't black pixels, motion = true). Or... is this too simplistic? I'm open to hearing better ways of doing this, I had a look around on the web but there aren't many solid examples for C#. My video is playing within a WPF application in real-time so nested for loops are to be avoided.
Thanks for your time!

You could for example convert the matrix to an image. That should give you access to all the image manipulation functions. For example ThresholdBinary to make pixels either zero or a given value, and CountNonZero. That should give you some tools balance how much things need to change, and how large area need to change.

Found a simple way to do it, may not be the best but it does work.
public bool motion = false;
public Mat currentFrame = new Mat();
public Mat prevFrame = new Mat();
public Mat absDiffImage = new Mat();
public Mat grayImage = new Mat();
public Point[][] frameContours;
public HierarchyIndex[] external;
public Mat frameThresh = new Mat();
Cv2.CvtColor(currentFrame, currentFrame, ColorConversionCodes.BGRA2GRAY);
Cv2.Absdiff(currentFrame, prevFrame, absDiffImage);
Cv2.Threshold(absDiffImage, frameThresh, 80, 255, ThresholdTypes.Binary);
Cv2.FindContours(frameThresh, out frameContours, out external, RetrievalModes.List, ContourApproximationModes.ApproxSimple);
if (frameContours.Length > 20)
{
motion = true;
}
else
{
motion = false;
}

Ghost-like artifact when passing image to blur() or Canny() in openCV

So I have some C++ opencv code that I'm calling from C# inside unity to process the input from the webcam.
I compile my C++ code to a DLL that I then import into C# with a DLLImport. I pass a pinned GCHandle reference to the C++ code so that I can manipulate the image array from C++. This all works. I can pass each frame to my C++ dll and have it grayscale it. It works wonderfully.
The problem arises when I try to do things other than just making the frame grayscale. I tried to do a simple blur() and the output comes out with a weird ghosted image to the left and right. I'm not sure what could be going wrong. It also happens when I do GaussianBlur() or Canny().
On the left is when I cover the camera, you can see the weird artifact more clearly. In the middle is the artifact itself after passing through GaussianBlur(). It seems like it creates copies of the image and overlays them with itself. And on the right is when it's just grayscaled to show that THAT works properly. So I figure it's not something that's happening between C# and C++, it's something that happens only when I pass the frame through opencv's blur or gaussianblur or canny.
Here is the C# code in unity
using UnityEngine;
using System.Collections;
using System;
using System.Runtime.InteropServices;
public class camera : MonoBehaviour {
[DllImport("tee")]
public static extern void bw([MarshalAs(UnmanagedType.LPStruct)]
IntPtr data,
int width,
int height);
WebCamTexture back;
Color32[] data;
Byte[] byteData;
Renderer rend;
String test;
Texture2D tex;
GCHandle dataHandle;
// Use this for initialization
void Start () {
back = new WebCamTexture();
back.Play();
rend = GetComponent<Renderer>();
tex = new Texture2D(back.width, back.height, TextureFormat.ARGB32, false);
data = back.GetPixels32();
dataHandle = GCHandle.Alloc(data, GCHandleType.Pinned);
}
void OnDisable()
{
dataHandle.Free();
}
// Update is called once per frame
void Update () {
back.GetPixels32(data);
bw(dataHandle.AddrOfPinnedObject(), back.width, back.height);
tex.SetPixels32(data);
tex.Apply();
rend.material.mainTexture = tex;
}
}
and here is the C++ code that gets compiled into a DLL
#include <opencv2\core\core.hpp>
#include <opencv2\imgproc\imgproc.hpp>
using namespace std;
using namespace cv;
extern "C"
{
__declspec(dllexport) void bw(int data, int width, int height) {
unsigned char * buffer = reinterpret_cast<unsigned char *>(data);
Mat mat = Mat(width, height, CV_8UC4, buffer).clone();
Mat gray;
cvtColor(mat, gray, CV_RGBA2GRAY);
Mat blurred;
GaussianBlur(gray, blurred, Size(3, 3), 2, 2);
if (blurred.isContinuous()) {
for (int i = 0; i < (width * height); i++) {
unsigned char * pxl = buffer + 4 * i;
pxl[0] = blurred.data[i]; //red channel
pxl[1] = blurred.data[i]; //green channel
pxl[2] = blurred.data[i]; //blue channel
pxl[3] = (unsigned char)255; // alpha channel
}
}
}
}

According to OpenCV's documentation, the Mat constructor takes rows and cols as parameters, so you should switch the width and height parameters. See here http://docs.opencv.org/2.4/modules/core/doc/basic_structures.html#mat-mat
Another thing, do you know how the images are stored in C#? Do they have any kind of data alignment(i.e. The rows aren't continuous)? Because that could also be an issue when you create the containing Mat.
I'm from my phone currently, I'll try to reformat my answer ASAP
EDIT: thinking about it, the thing about switching width and height when constructing makes sense only if both OpenCV and texture2D store images in row major order. I've checked here (http://rbwhitaker.wikidot.com/extracting-texture-data) and it seems it's like that.

I think your problem is the way you are accessing blurred pixel value. You should access the channels values using the following instead
for (int i = 0; i < (width * height); i++) {
unsigned char * pxl = buffer + 4 * i;
pxl[0] = blurred.ptr<uchar>(i); //red channel
pxl[1] = blurred.ptr<uchar>(i); //green channel
pxl[2] = blurred.ptr<uchar>(i); //blue channel
pxl[3] = (unsigned char)255; // alpha channel
}
One other thing you can look into is the way opencv stores the pixel values versus pointer access of the data buffer (. You can test this easily by rotating the blurred image before accessing it and see if this gives you the correct output or creating Mat mat = Mat(height,width, CV_8UC4, buffer).clone(); instead.
And you are right about the blurred type, it should be one channel as the gray image.
Try the current code for another way of accessing the values in the blurred image

XNA and HLSL: Sample entire texture for luminosity average

I am trying to make an HDR rendering pass in my 3D app.
I understand that in order to get the average light on a scene, you'd need to downsample the render output down to 1x1 texture. This is where I'm struggling with at the moment.
I have set up a 1x1 resolution render target to which I'm going to draw the previous render output. I have tried drawing the output to that render target using a simple SpriteBatch Draw call and using target rectangle. However it was too much to hope for I'd discover something nobody else thought of, the result of this was not actually entire scene downsampled to 1x1 texture, it appears as if only the top left pixel was being drawn, no matter how much I played with target rectangles or Scale overloads.
Now, I'm trying to do another screen-quad render pass, using a shader technique to sample out the scene, and render a single pixel into the render target. So to be fair, title is a bit misleading, what I'm trying to do is sample a grid of pixels spread evenly across the surface, and average those out. But this is where I'm stumped.
I have come across this tutorial:
http://www.xnainfo.com/content.php?content=28
In the file that can be downloaded, there are several examples of downsampling and one that I like most is using loop that goes through 16 pixels, averages them out and returns that.
Nothing I did so far managed to produce viable output. The downsampled texture is rendered to the corner of my screen for debugging purposes.
I have modified the HLSL code to look like this:
pixelShaderStruct vertShader(vertexShaderStruct input)
{
pixelShaderStruct output;
output.position = float4(input.pos, 1);
output.texCoord = input.texCoord + 0.5f;
return output;
};
float4 PixelShaderFunction(pixelShaderStruct input) : COLOR0
{
float4 color = 0;
float2 position = input.texCoord;
for(int x = 0; x < 4; x++)
{
for (int y = 0; y < 4; y++)
{
color += tex2D(getscene, position + float2(offsets[x], offsets[y]));
}
}
color /= 16;
return color;
}
This particular line here is where I believe I'm making the error:
color += tex2D(getscene, position + float2(offsets[x], offsets[y]));
I have never properly understood how the texCoord values, used in tex2D sampling, work.
When making motion blur effect, I had to forward such infinitismal values that I was afraid it'd get rounded off to zero, in order to produce a normal looking effect, while other times, forwarding large values like 30 and 50 was neccessary to produce effects that occupy maybe one third of the screen.
So anyway, my question is:
Given a screen quad (so, flat surface), how do I increment or modify the texCoord values to have a grid of pixels, evenly spread out, sampled across it?
I have tried using:
color += tex2D(getscene, position + float2(offsets[x] * (1/maxX), offsets[y] * (1/maxY)));
Where maxX and maxY are screen resolution,
color += tex2D(getscene, position + float2(offsets[x] * x, offsets[y] * y));
...and other "shots in the dark", and all results have ended up being the same: the final resulting pixel appears to be identical to the one in the exact middle of my screen, as if that's the only one being sampled.
How to solve that?
Also, how do those texture coordinates work? Where is (0,0)? What's the maximum?
Thanks all in advance.

I have solved the problem.
I believed that using a dozen renderTargets which half the resolution each step would be expensive to do, but I was wrong.
On a middle-end GPU in 2013, nVidia GTX 560, the cost of rerendering to 10 render targets was not noticeable, specific numbers: from 230 FPS the performance dropped to some 220 FPS.
The solution follows. It is implied you already have your entire scene processed and rendered to a renderTarget, which in my case is "renderOutput".
First, I declare a renderTarget array:
public RenderTarget2D[] HDRsampling;
Next, I calculate how many targets I will need in my Load() method, which is called between the Menu update and Game update loops (transition state for loading game assets not required in menu), and initialize them properly:
int counter = 0;
int downX = Game1.maxX;
int downY = Game1.maxY;
do
{
downX /= 2;
downY /= 2;
counter++;
} while (downX > 1 && downY > 1);
HDRsampling = new RenderTarget2D[counter];
downX = Game1.maxX / 2;
downY = Game1.maxY / 2;
for (int i = 0; i < counter; i++)
{
HDRsampling[i] = new RenderTarget2D(Game1.graphics.GraphicsDevice, downX, downY);
downX /= 2;
downY /= 2;
}
And finally, C# rendering code is as follows:
if (settings.HDRpass)
{ //HDR Rendering passes
//Uses Hardware bilinear downsampling method to obtain 1x1 texture as scene average
Game1.graphics.GraphicsDevice.SetRenderTarget(HDRsampling[0]);
Game1.graphics.GraphicsDevice.Clear(ClearOptions.Target, Color.Black, 0, 0);
downsampler.Parameters["maxX"].SetValue(HDRsampling[0].Width);
downsampler.Parameters["maxY"].SetValue(HDRsampling[0].Height);
downsampler.Parameters["scene"].SetValue(renderOutput);
downsampler.CurrentTechnique.Passes[0].Apply();
quad.Render();
for (int i = 1; i < HDRsampling.Length; i++)
{ //Downsample the scene texture repeadetly until last HDRSampling target, which should be 1x1 pixel
Game1.graphics.GraphicsDevice.SetRenderTarget(HDRsampling[i]);
Game1.graphics.GraphicsDevice.Clear(ClearOptions.Target, Color.Black, 0, 0);
downsampler.Parameters["maxX"].SetValue(HDRsampling[i].Width);
downsampler.Parameters["maxY"].SetValue(HDRsampling[i].Height);
downsampler.Parameters["scene"].SetValue(HDRsampling[i-1]);
downsampler.CurrentTechnique.Passes[0].Apply();
quad.Render();
}
//assign the 1x1 pixel
downsample1x1 = HDRsampling[HDRsampling.Length - 1];
Game1.graphics.GraphicsDevice.SetRenderTarget(extract);
//switch out rendertarget so we can send the 1x1 sample to the shader.
bloom.Parameters["downSample1x1"].SetValue(downsample1x1);
}
This obtains the downSample1x1 texture, which is later used in the final pass of the final shader.
The shader code for actual downsampling is barebones simple:
texture2D scene;
sampler getscene = sampler_state
{
texture = <scene>;
MinFilter = linear;
MagFilter = linear;
MipFilter = point;
MaxAnisotropy = 1;
AddressU = CLAMP;
AddressV = CLAMP;
};
float maxX, maxY;
struct vertexShaderStruct
{
float3 pos : POSITION0;
float2 texCoord : TEXCOORD0;
};
struct pixelShaderStruct
{
float4 position : POSITION0;
float2 texCoord : TEXCOORD0;
};
pixelShaderStruct vertShader(vertexShaderStruct input)
{
pixelShaderStruct output;
float2 offset = float2 (0.5 / maxX, 0.5/maxY);
output.position = float4(input.pos, 1);
output.texCoord = input.texCoord + offset;
return output;
};
float4 PixelShaderFunction(pixelShaderStruct input) : COLOR0
{
return tex2D(getscene, input.texCoord);
}
technique Sample
{
pass P1
{
VertexShader = compile vs_2_0 vertShader();
PixelShader = compile ps_2_0 PixelShaderFunction();
}
}
How you implement your scene average luminosity is up to you, I'm still experimenting with all that, but I hope this helps somebody out there!

The tutorial you linked is a good demonstration of how to do Tonemapping, but it sounds to me like you haven't performed the repeated downsampling that's required to get to a 1x1 image.
You can't simply take a high resolution image (1920x1080, say) take 16 arbitrary pixels, add them up and divide by 16 and call that your luminance.
You need to repeatedly downsample the source image to smaller and smaller textures, usually by half in each dimension at every stage. Each pixel of the resulting downsample is an average of a 2x2 grid of pixels on the previous texture (this is handled by bilinear sampling). Eventually you'll end up with a 1x1 image that is the average colour value for the entire original 1920x1080 image and from that you can calculate the average luminance of the source image.
Without repeated downsampling, your luminance calculation is going to be a very noisy value since its input is a mere 16 of the ~2M pixels in the original image. To get a correct and smooth luminance every pixel in the original image needs to contribute.

Getting a portion of a larger Texture2d with XNA

So I'm making an asteroids like game for practice in XNA. I have a Large texture I'm using for a texture atlas as described in R.B.Whitaker's wiki here: http://rbwhitaker.wikidot.com/texture-atlases-1
I've now branched off from the wiki and am attempting to make collision detection for my ship and the alien. The problem is I need the current sprite from the atlas as a separate texture2d to where I can do accurate collision detection. I've read examples using the texture2d.GetData method, but I can't seem to get a working implementation. Any detailed implementations of that method or other options would be greatly appreciated.

Use Texture.GetData<Color>() on your atlas texture to get the array of colors representing individual pixels:
Color[] imageData = new Color[image.Width * image.Height];
image.GetData<Color>(imageData);
When you need a rectangle of data from the texture, use a method as such:
Color[] GetImageData(Color[] colorData, int width, Rectangle rectangle)
{
Color[] color = new Color[rectangle.Width * rectangle.Height];
for (int x = 0; x < rectangle.Width; x++)
for (int y = 0; y < rectangle.Height; y++)
color[x + y * rectangle.Width] = colorData[x + rectangle.X + (y + rectangle.Y) * width];
return color;
}
The method above throws index out of range exception if the rectangle is out of bounds of the original texture. width parameter is the width of the texture.
Usage with sourceRectangle of a specific sprite:
Color[] imagePiece = GetImageData(imageData, image.Width, sourceRectangle);
If you really need another texture (though I imagine you actually need color data for per pixel collision), you could do this:
Texture2D subtexture = new Texture2D(GraphicsDevice, sourceRectangle.Width, sourceRectangle.Height);
subtexture.SetData<Color>(imagePiece);
Side note - If you need to use color data from textures constantly for collision tests, save it as an array of colors, not as textures. Getting data from textures uses the GPU while your CPU waits, which degrades performance.

How do you do a 3D transform (perspective) in C# or VB.Net?

What I am looking to do sounds really simple, but no where on the Internet so far have I found a way to do this in DotNet nor found a 3rd party component that does this either (without spending thousands on completely unnecessary features).
Here goes:
I have a jpeg of a floor tile (actual photo) that I create a checkerboard pattern with.
In dotnet, it is easy to rotate and stitch photos together and save the final image as a jpeg.
Next, I want to take that final picture and make it appear as if the "tiles" are laying on a floor for a generic "room scene". Basically adding a 3D perspective to make it appear as if it is actually in the room scene.
Heres a website that is doing something similar with carpeting, however I need to do this in a WinForms application:
Flor Website
Basically, I need to create a 3D perspective of a jpeg, then save it as a new jpeg (then I can put an overlay of the generic room scene).
Anyone have any idea on where to get a 3rd party DotNet image processing module that can do this seemingly simple task?

It is not so simple because you need a 3D transformation, which is more complicated and computationally expensive than a simple 2D transformation such as rotation, scaling or shearing. For you to have an idea of the difference in the math, 2D transformations require 2 by 2 matrices, whereas a projection transformation (which is more complicated than other 3D transforms) requires a 4 by 4 matrix...
What you need is some 3D rendering engine in which you can draw polygons (in a perspective view) and them cover them with a texture (like a carpet). For .Net 2.0, I'd recommend using SlimDX which is a port of DirectX that would allow you to render polygons, but there is some learning curve. If you are using WPF (.Net 3.0 and up), there is a built in 3D canvas that allows you to draw textured polygons in perspective. That might be easier/better to learn than SlimDX for your purposes. I'm sure that there is a way to redirect the output of the 3D canvas towards a jpeg...
You might simplify the problem a lot if you don't require great performance and if you restrict the orientation of the texture (eg. always a horizontal floor or always a vertical wall). If so, you could probably render it yourself with a simple drawing loop in .Net 2.0.

If you just want a plain floor, your code would look like this. WARNING: Obtaining your desired results will take some significant time and refinement, specially if you don't know the math very well. But on the other hand, it is always fun to play with code of this type... (:
Find some sample images below.
using System;
using System.Collections.Generic;
using System.Drawing;
using System.Windows.Forms;
namespace floorDrawer
{
public partial class Form1 : Form
{
public Form1()
{
InitializeComponent();
ResizeRedraw = DoubleBuffered = true;
Width = 800;
Height = 600;
Paint += new PaintEventHandler(Form1_Paint);
}
void Form1_Paint(object sender, PaintEventArgs e)
{
// a few parameters that control the projection transform
// these are the parameters that you can modify to change
// the output
double cz = 10; // distortion
double m = 1000; // magnification, usually around 1000 (the pixel width of the monitor)
double y0 = -100; // floor height
string texturePath = #"c:\pj\Hydrangeas.jpg";//#"c:\pj\Chrysanthemum.jpg";
// screen size
int height = ClientSize.Height;
int width = ClientSize.Width;
// center of screen
double cx = width / 2;
double cy = height / 2;
// render destination
var dst = new Bitmap(width, height);
// source texture
var src = Bitmap.FromFile(texturePath) as Bitmap;
// texture dimensions
int tw = src.Width;
int th = src.Height;
for (int y = 0; y < height; y++)
for (int x = 0; x < width; x++)
{
double v = m * y0 / (y - cy) - cz;
double u = (x - cx) * (v + cz) / m;
int uu = ((int)u % tw + tw) % tw;
int vv = ((int)v % th + th) % th;
// The following .SetPixel() and .GetPixel() are painfully slow
// You can replace this whole loop with an equivalent implementation
// using pointers inside unsafe{} code to make it much faster.
// Note that by casting u and v into integers, we are performing
// a nearest pixel interpolation... It's sloppy but effective.
dst.SetPixel(x, y, src.GetPixel(uu, vv));
}
// draw result on the form
e.Graphics.DrawImage(dst, 0, 0);
}
}
}