Develop Reference

Convert Texture2D to OpenCV Mat?

There is a post about converting OpenCV cv::Mat to Texture2D in Unity and I provided an answer which works well. Now, I am trying to do the opposite but have been stuck on this for few hours now.
I want to convert Unity's Texture2D to OpenCV cv::Mat so that I can process the Texture on the C++ side.
Here is the original Texture2D in my Unity project that I want to convert to cv:Mat:
Here is what it looks like after converting it into cv:Mat:
It looks so washed out. I am not worried about the rotation of the image. I can fix that. Just wondering why it looks so washed out. Also used cv::imwrite to save the image for testing purposes but the issue in also in the saved image.
C# code:
[DllImport("TextureConverter")]
private static extern float TextureToCVMat(IntPtr texData, int width, int height);
unsafe void TextureToCVMat(Texture2D texData)
{
Color32[] texDataColor = texData.GetPixels32();
//Pin Memory
fixed (Color32* p = texDataColor)
{
TextureToCVMat((IntPtr)p, texData.width, texData.height);
}
}
public Texture2D tex;
void Start()
{
TextureToCVMat(tex);
}
C++ code:
DLLExport void TextureToCVMat(unsigned char* texData, int width, int height)
{
Mat texture(height, width, CV_8UC4, texData);
cvNamedWindow("Unity Texture", CV_WINDOW_NORMAL);
//cvResizeWindow("Unity Texture", 200, 200);
cv::imshow("Unity Texture", texture);
cv::imwrite("Inno Image.jpg", texture);
}
I also tried creating a struct on the C++ side to hold the pixel information instead of using unsigned char* but the result is still the-same:
struct Color32
{
uchar r;
uchar g;
uchar b;
uchar a;
};
DLLExport void TextureToCVMat(Color32* texData, int width, int height)
{
Mat texture(height, width, CV_8UC4, texData);
cvNamedWindow("Unity Texture", CV_WINDOW_NORMAL);
cvResizeWindow("Unity Texture", 200, 200);
cv::imshow("Unity Texture", texture);
}
Why does the image look so so washed out and how do you fix this?

OpenCV creates images as BGR by default, whereas Color32 stores pixels as RGBA. However since OP mentioned in the comments that the Texture2D.format gives texture format as RGB24, we can ignore the alpha channel altogether.
DLLExport void TextureToCVMat(unsigned char* texData, int width, int height)
{
Mat texture(height, width, CV_8UC4, texData);
cv::cvtColor(texture,texture,cv::COLOR_BGRA2RGB);
cv::imshow("Unity Texture", texture);
cv::waitKey(0);
cv::destroyAllWindows();
}

Calling an opencv Mat from a dll to windows forms, image is glitchy

I have an openCv based dll, that connects to a camera. I then call a cv::mat object into a C# application, and display the image as a Bitmap in a picturebox object.
This works, but the image occasionally 'glitches', showing flashes of lines, static and pops, every few seconds.
Is there a way I can check if the bitmap is valid before displaying it?
When i show the image in the dll, using cv::imshow, it looks fine.
The code I have is:
in the c++ dll:
__declspec(dllexport) uchar* getArucoFrame(void)
{
cv::Mat OriginalImg = returnLeftFrame(); // calls the frame from where the camera thread stores it.
cv::Mat tmp;
cv::cvtColor(OriginalImg, tmp, CV_BGRA2BGR);
//if I cv::imshow the Mat here, it looks good.
return tmp.data;
}
in the C# side:
//on a button
threadImageShow = new Thread(imageShow);
threadImageShow.Start();
//show image frame in box
private void imageShow()
{
while(true)
{
IntPtr ptr = getArucoFrame();
if (pictureBoxFrame.Image != null)
{
pictureBoxFrame.Image.Dispose();
}
Bitmap a = new Bitmap(640, 360, 3 * 640, PixelFormat.Format24bppRgb, ptr);
pictureBoxFrame.Image = a;
Thread.Sleep(20);
}
}
//the dll call
[DllImport("Vector.dll", CallingConvention = CallingConvention.Cdecl)]
public static extern IntPtr getArucoFrame();
As the image looks good in the dll, and glitchy in the picturebox, i am having trouble debugging this. Any help much appreciated. thank you.

The problem you have here is that you pass pointer to data of temporary image cv::Mat tmp; into C#, but it gets freed on exit of getArucoFrame(void), thus it is dangling pointer. It may work, but seems it sometimes gets overwritten by new data. An easiest, but not the most optimal fix would be declaring it static static cv::Mat tmp; so it gets freed on DLL unload.

Ghost-like artifact when passing image to blur() or Canny() in openCV

So I have some C++ opencv code that I'm calling from C# inside unity to process the input from the webcam.
I compile my C++ code to a DLL that I then import into C# with a DLLImport. I pass a pinned GCHandle reference to the C++ code so that I can manipulate the image array from C++. This all works. I can pass each frame to my C++ dll and have it grayscale it. It works wonderfully.
The problem arises when I try to do things other than just making the frame grayscale. I tried to do a simple blur() and the output comes out with a weird ghosted image to the left and right. I'm not sure what could be going wrong. It also happens when I do GaussianBlur() or Canny().
On the left is when I cover the camera, you can see the weird artifact more clearly. In the middle is the artifact itself after passing through GaussianBlur(). It seems like it creates copies of the image and overlays them with itself. And on the right is when it's just grayscaled to show that THAT works properly. So I figure it's not something that's happening between C# and C++, it's something that happens only when I pass the frame through opencv's blur or gaussianblur or canny.
Here is the C# code in unity
using UnityEngine;
using System.Collections;
using System;
using System.Runtime.InteropServices;
public class camera : MonoBehaviour {
[DllImport("tee")]
public static extern void bw([MarshalAs(UnmanagedType.LPStruct)]
IntPtr data,
int width,
int height);
WebCamTexture back;
Color32[] data;
Byte[] byteData;
Renderer rend;
String test;
Texture2D tex;
GCHandle dataHandle;
// Use this for initialization
void Start () {
back = new WebCamTexture();
back.Play();
rend = GetComponent<Renderer>();
tex = new Texture2D(back.width, back.height, TextureFormat.ARGB32, false);
data = back.GetPixels32();
dataHandle = GCHandle.Alloc(data, GCHandleType.Pinned);
}
void OnDisable()
{
dataHandle.Free();
}
// Update is called once per frame
void Update () {
back.GetPixels32(data);
bw(dataHandle.AddrOfPinnedObject(), back.width, back.height);
tex.SetPixels32(data);
tex.Apply();
rend.material.mainTexture = tex;
}
}
and here is the C++ code that gets compiled into a DLL
#include <opencv2\core\core.hpp>
#include <opencv2\imgproc\imgproc.hpp>
using namespace std;
using namespace cv;
extern "C"
{
__declspec(dllexport) void bw(int data, int width, int height) {
unsigned char * buffer = reinterpret_cast<unsigned char *>(data);
Mat mat = Mat(width, height, CV_8UC4, buffer).clone();
Mat gray;
cvtColor(mat, gray, CV_RGBA2GRAY);
Mat blurred;
GaussianBlur(gray, blurred, Size(3, 3), 2, 2);
if (blurred.isContinuous()) {
for (int i = 0; i < (width * height); i++) {
unsigned char * pxl = buffer + 4 * i;
pxl[0] = blurred.data[i]; //red channel
pxl[1] = blurred.data[i]; //green channel
pxl[2] = blurred.data[i]; //blue channel
pxl[3] = (unsigned char)255; // alpha channel
}
}
}
}

According to OpenCV's documentation, the Mat constructor takes rows and cols as parameters, so you should switch the width and height parameters. See here http://docs.opencv.org/2.4/modules/core/doc/basic_structures.html#mat-mat
Another thing, do you know how the images are stored in C#? Do they have any kind of data alignment(i.e. The rows aren't continuous)? Because that could also be an issue when you create the containing Mat.
I'm from my phone currently, I'll try to reformat my answer ASAP
EDIT: thinking about it, the thing about switching width and height when constructing makes sense only if both OpenCV and texture2D store images in row major order. I've checked here (http://rbwhitaker.wikidot.com/extracting-texture-data) and it seems it's like that.

I think your problem is the way you are accessing blurred pixel value. You should access the channels values using the following instead
for (int i = 0; i < (width * height); i++) {
unsigned char * pxl = buffer + 4 * i;
pxl[0] = blurred.ptr<uchar>(i); //red channel
pxl[1] = blurred.ptr<uchar>(i); //green channel
pxl[2] = blurred.ptr<uchar>(i); //blue channel
pxl[3] = (unsigned char)255; // alpha channel
}
One other thing you can look into is the way opencv stores the pixel values versus pointer access of the data buffer (. You can test this easily by rotating the blurred image before accessing it and see if this gives you the correct output or creating Mat mat = Mat(height,width, CV_8UC4, buffer).clone(); instead.
And you are right about the blurred type, it should be one channel as the gray image.
Try the current code for another way of accessing the values in the blurred image

Marshalling struct containing int and int[] from C# to C++

I have a C++ DLL with unmanaged code and a C# UI. There's a function imported from C++ DLL that takes a written-by-me struct as parameter.
After marshalling the written-by-me struct (MyImage) from C# to C++ I can access the content of the int[] array inside of it, but the content is different. I do not know what I am missing here as I spent quite some time and tried a few tricks to resolve this (obviously not enough).
MyImage struct in C#:
[StructLayout(LayoutKind.Sequential)]
struct MyImage
{
public int width;
public int height;
public int[] bits; //these represent colors of image - 4 bytes for each pixel
}
MyImage struct in C++:
struct MyImage
{
int width;
int height;
Color* bits; //typedef unsigned int Color;
MyImage(int w, int h)
{
bits = new Color[w*h];
}
Color GetPixel(int x, int y)
{
if (x or y out of image bounds) return UNDEFINED_COLOR;
return bits[y*width+x];
}
}
C# function declaration with MyImage as parameter:
[DLLImport("G_DLL.dll")]
public static extern void DisplayImageInPolygon(Point[] p, int n, MyImage texture,
int tex_x0, int tex_y0);
C++ implementation
DLLEXPORT void __stdcall DisplayImageInPolygon(Point *p, int n, MyImage img,
int imgx0, int imgy0)
{
//And below they have improper values (i don't know where they come from)
Color test1 = img.GetPixel(0,0);
Color test2 = img.GetPixel(1,0);
}
So when debugging the problem I noticed that the MyImage.bits array in c++ struct holds different data.
How can I fix it?

Since the bits field is a pointer to memory allocated in the native code, you are going to need to declare it as IntPtr in the C# code.
struct MyImage
{
public int width;
public int height;
public IntPtr bits;
}
If you want to access individual pixels in the C# code you'll need to write a GetPixel method, just as you did in the C++ code.
Note that since the bits field is a pointer to memory allocated in the native code, I'd expect the actual code to have a destructor for the struct that calls delete[] bits. Otherwise your code will leak.
This also means that you are going to need to create and destroy instances in the native code, and never do so in the managed code. Is this the policy you currently follow? I suspect not based on the code that I can see here.
You also need to reconsider passing the struct by value. Do you really want to take a copy of it when you call that function? Doing so means you've got two instances of the struct whose bits fields both point to the same memory. But, which one owns that memory? This structure really needs to be passed by reference.
I think you've got some problems in your design, but I can't see enough of the code, or know enough about your problem to be able to give you concrete advice.
In comments you state that your main goal is to transfer these bits from your C# code to the C++ code. I suggest you do it like this:
MyImage* NewImage(int w, int h, Color* bits)
{
MyImage* img = new MyImage;
img->w = w;
img->h = h;
img->bits = new Color[w*h];
for (int i=0; i<w*h; i++)
img->bits[i] = bits[i];
return img;
}
void DeleteImage(MyImage* img)
{
delete[] img->bits;
delete img;
}
void DoSomethingWithImage(MyImage* img)
{
// do whatever it is you need to do
}
On the C# side you can declare it like this:
[DllImport(#"dllname.dll", CallingConvention=CallingConvention.Cdecl)]
static extern IntPtr NewImage(int w, int h, int[] bits);
[DllImport(#"dllname.dll", CallingConvention=CallingConvention.Cdecl)]
static extern void DeleteImage(ImtPtr img);
[DllImport(#"dllname.dll", CallingConvention=CallingConvention.Cdecl)]
static extern void DoSomethingWithImage(ImtPtr img);

The first thing you should try is declaring your C# code with unsigned int types as well. It is possible that one bit is being interpreted as a sign for your int.
So in C# something like this (just note the bits is now uint[]):
[StructLayout(LayoutKind.Sequential)]
struct MyImage
{
public int width;
public int height;
public uint[] bits; //these represent colors of image - 4 bytes for each pixel
}

You can use the PInvoke Interop Assistant. You simply paste your struct and function declaration and it will generate the C# code for you. It has helped me a lot quite a few times.

Use native HBitmap in C# while preserving alpha channel/transparency

Let's say I get a HBITMAP object/handle from a native Windows function. I can convert it to a managed bitmap using Bitmap.FromHbitmap(nativeHBitmap), but if the native image has transparency information (alpha channel), it is lost by this conversion.
There are a few questions on Stack Overflow regarding this issue. Using information from the first answer of this question (How to draw ARGB bitmap using GDI+?), I wrote a piece of code that I've tried and it works.
It basically gets the native HBitmap width, height and the pointer to the location of the pixel data using GetObject and the BITMAP structure, and then calls the managed Bitmap constructor:
Bitmap managedBitmap = new Bitmap(bitmapStruct.bmWidth, bitmapStruct.bmHeight,
bitmapStruct.bmWidth * 4, PixelFormat.Format32bppArgb, bitmapStruct.bmBits);
As I understand (please correct me if I'm wrong), this does not copy the actual pixel data from the native HBitmap to the managed bitmap, it simply points the managed bitmap to the pixel data from the native HBitmap.
And I don't draw the bitmap here on another Graphics (DC) or on another bitmap, to avoid unnecessary memory copying, especially for large bitmaps.
I can simply assign this bitmap to a PictureBox control or the the Form BackgroundImage property. And it works, the bitmap is displayed correctly, using transparency.
When I no longer use the bitmap, I make sure the BackgroundImage property is no longer pointing to the bitmap, and I dispose both the managed bitmap and the native HBitmap.
The Question: Can you tell me if this reasoning and code seems correct. I hope I will not get some unexpected behaviors or errors. And I hope I'm freeing all the memory and objects correctly.
private void Example()
{
IntPtr nativeHBitmap = IntPtr.Zero;
/* Get the native HBitmap object from a Windows function here */
// Create the BITMAP structure and get info from our nativeHBitmap
NativeMethods.BITMAP bitmapStruct = new NativeMethods.BITMAP();
NativeMethods.GetObjectBitmap(nativeHBitmap, Marshal.SizeOf(bitmapStruct), ref bitmapStruct);
// Create the managed bitmap using the pointer to the pixel data of the native HBitmap
Bitmap managedBitmap = new Bitmap(
bitmapStruct.bmWidth, bitmapStruct.bmHeight, bitmapStruct.bmWidth * 4, PixelFormat.Format32bppArgb, bitmapStruct.bmBits);
// Show the bitmap
this.BackgroundImage = managedBitmap;
/* Run the program, use the image */
MessageBox.Show("running...");
// When the image is no longer needed, dispose both the managed Bitmap object and the native HBitmap
this.BackgroundImage = null;
managedBitmap.Dispose();
NativeMethods.DeleteObject(nativeHBitmap);
}
internal static class NativeMethods
{
[StructLayout(LayoutKind.Sequential)]
public struct BITMAP
{
public int bmType;
public int bmWidth;
public int bmHeight;
public int bmWidthBytes;
public ushort bmPlanes;
public ushort bmBitsPixel;
public IntPtr bmBits;
}
[DllImport("gdi32", CharSet = CharSet.Auto, EntryPoint = "GetObject")]
public static extern int GetObjectBitmap(IntPtr hObject, int nCount, ref BITMAP lpObject);
[DllImport("gdi32.dll")]
internal static extern bool DeleteObject(IntPtr hObject);
}

The following code worked for me even if the HBITMAP is an icon or bmp, it doesn't flip the image when it's an icon, and also works with bitmaps that don't contain Alpha channel:
private static Bitmap GetBitmapFromHBitmap(IntPtr nativeHBitmap)
{
Bitmap bmp = Bitmap.FromHbitmap(nativeHBitmap);
if (Bitmap.GetPixelFormatSize(bmp.PixelFormat) < 32)
return bmp;
BitmapData bmpData;
if (IsAlphaBitmap(bmp, out bmpData))
return GetlAlphaBitmapFromBitmapData(bmpData);
return bmp;
}
private static Bitmap GetlAlphaBitmapFromBitmapData(BitmapData bmpData)
{
return new Bitmap(
bmpData.Width,
bmpData.Height,
bmpData.Stride,
PixelFormat.Format32bppArgb,
bmpData.Scan0);
}
private static bool IsAlphaBitmap(Bitmap bmp, out BitmapData bmpData)
{
Rectangle bmBounds = new Rectangle(0, 0, bmp.Width, bmp.Height);
bmpData = bmp.LockBits(bmBounds, ImageLockMode.ReadOnly, bmp.PixelFormat);
try
{
for (int y = 0; y <= bmpData.Height - 1; y++)
{
for (int x = 0; x <= bmpData.Width - 1; x++)
{
Color pixelColor = Color.FromArgb(
Marshal.ReadInt32(bmpData.Scan0, (bmpData.Stride * y) + (4 * x)));
if (pixelColor.A > 0 & pixelColor.A < 255)
{
return true;
}
}
}
}
finally
{
bmp.UnlockBits(bmpData);
}
return false;
}

Right, no copy is made. Which is why the Remarks section of the MSDN Library says:
The caller is responsible for
allocating and freeing the block of
memory specified by the scan0
parameter, however, the memory should
not be released until the related
Bitmap is released.
This wouldn't be a problem if the pixel data was copied. Incidentally, this is normally a difficult problem to deal with. You can't tell when the client code called Dispose(), there's no way to intercept that call. Which makes it impossible to make such a bitmap behave like a replacement for Bitmap. The client code has to be aware that additional work is needed.

After reading the good points made by Hans Passant in his answer, I changed the method to immediately copy the pixel data into the managed bitmap, and free the native bitmap.
I'm creating two managed bitmap objects (but only one allocates memory for the actual pixel data), and use graphics.DrawImage to copy the image. Is there a better way to accomplish this? Or is this good/fast enough?
public static Bitmap CopyHBitmapToBitmap(IntPtr nativeHBitmap)
{
// Get width, height and the address of the pixel data for the native HBitmap
NativeMethods.BITMAP bitmapStruct = new NativeMethods.BITMAP();
NativeMethods.GetObjectBitmap(nativeHBitmap, Marshal.SizeOf(bitmapStruct), ref bitmapStruct);
// Create a managed bitmap that has its pixel data pointing to the pixel data of the native HBitmap
// No memory is allocated for its pixel data
Bitmap managedBitmapPointer = new Bitmap(
bitmapStruct.bmWidth, bitmapStruct.bmHeight, bitmapStruct.bmWidth * 4, PixelFormat.Format32bppArgb, bitmapStruct.bmBits);
// Create a managed bitmap and allocate memory for pixel data
Bitmap managedBitmapReal = new Bitmap(bitmapStruct.bmWidth, bitmapStruct.bmHeight, PixelFormat.Format32bppArgb);
// Copy the pixels of the native HBitmap into the canvas of the managed bitmap
Graphics graphics = Graphics.FromImage(managedBitmapReal);
graphics.DrawImage(managedBitmapPointer, 0, 0);
// Delete the native HBitmap object and free memory
NativeMethods.DeleteObject(nativeHBitmap);
// Return the managed bitmap, clone of the native HBitmap, with correct transparency
return managedBitmapReal;
}