Develop Reference

Get correct cursor image from Windows API

I'm trying to get cursor data (mask and color, if available) as a byte array using GetDIBits(), with the correct size and correct animation frame but it's proving to be unreliable.
Also, I would like to get or be able to simulate the multiple frames available in a cursor, such as what is possible when using DrawIconEx(), passing the cursorStep parameter.
Basically what I'm trying to emulate is part of the behavior of IDXGIOutputDuplication::GetFramePointerShape(), to get a buffer (byte[]) with the contents of the current cursor, just without having to rely on the DirectX library.
So, I want:
Get the cursor data as a byte[], even with the mask working (SOLVED).
Get the cursor data with the correct size, not a fixed 32x32px cursor.
Get the actual animation frame of the cursor.
Just as the DXGI method provides if I'm using OutputDuplication, but in my case, I'm not.
1) Get the cursor mask correctly (solved):
TL;DR: Wrong data interpretation on my part.
I'm calling GetDIBits() once, passing null as the buffer parameter to get the details of the color/mask images and a second time to get the image.
It works all right with the color image but returns a wrong data size and wrong image content for the mask image.
Here's a comparison and the code sample.
Files ending with "2" were created by using System.Drawing.Bitmap.FromHBitmap() while the others were obtained from GetDiBits().
For some reason, GetDIBits() returns this data (BitmapInfoHeader) for the mask. Also, only returns 128 bytes (256 if the mask has two frames, such as for the I-Beam/Text cursor), which is way too little for 32x32px or 32x64px masks.
Here's my code so far (it's C#):
WindowDeviceContext = User32.GetWindowDC(IntPtr.Zero);
//The parameter passed in the structs is just to be able to calculate the size of it.
var cursorInfo = new CursorInfo(false);
if (!User32.GetCursorInfo(out cursorInfo))
return;
if (cursorInfo.Flags != Native.Constants.CursorShowing)
{
Gdi32.DeleteObject(cursorInfo.CursorHandle);
return;
}
var iconHandle = User32.CopyIcon(cursorInfo.CursorHandle);
if (iconHandle == IntPtr.Zero)
{
User32.DestroyIcon(iconHandle);
Gdi32.DeleteObject(cursorInfo.CursorHandle);
return;
}
if (!User32.GetIconInfo(iconHandle, out var iconInfo))
{
Gdi32.DeleteObject(iconInfo.Color);
Gdi32.DeleteObject(iconInfo.Mask);
User32.DestroyIcon(iconHandle);
Gdi32.DeleteObject(cursorInfo.CursorHandle);
return;
}
//Color.
var colorHeader = new BitmapInfoHeader(false);
//Gets the color image details.
Gdi32.GetDIBits(WindowDeviceContext, iconInfo.Color, 0, 0, null, ref colorHeader, DibColorModes.RgbColors);
if (colorHeader.Height != 0)
{
colorHeader.Height *= -1;
var colorBuffer = new byte[colorHeader.SizeImage];
Gdi32.GetDIBits(WindowDeviceContext, iconInfo.Color, 0, (uint)(colorHeader.Height * -1), colorBuffer, ref colorHeader, DibColorModes.RgbColors);
//Converts a byte array to image (System.Windows.Media.Imaging.BitmapSource) just for testing.
var image = ImageUtil.ImageMethods.FromArray(colorBuffer, colorHeader.Width, colorHeader.Height * -1, 4); //4 channels
using var fileStream = new FileStream(#"Color.png", FileMode.Create);
var bmpEncoder = new PngBitmapEncoder();
bmpEncoder.Frames.Add(BitmapFrame.Create(image));
bmpEncoder.Save(fileStream);
//Converts the HBitmap to Bitmap, just for testing too.
var image2 = System.Drawing.Image.FromHbitmap(iconInfo.Color);
image2.Save(#"Color2.png");
}
//Mask.
var maskHeader = new BitmapInfoHeader(false);
//Gets the mask image details.
Gdi32.GetDIBits(WindowDeviceContext, iconInfo.Mask, 0, 0, null, ref maskHeader, DibColorModes.RgbColors);
if (maskHeader.Height != 0)
{
maskHeader.Height *= -1;
var maskBuffer = new byte[maskHeader.SizeImage];
Gdi32.GetDIBits(WindowDeviceContext, iconInfo.Mask, 0, (uint) maskHeader.Height, maskBuffer, ref maskHeader, DibColorModes.RgbColors);
//Converts a byte array to image (System.Windows.Media.Imaging.BitmapSource) just for testing.
var image = ImageUtil.ImageMethods.FromArray(maskBuffer.ToList(), maskHeader.Width, maskHeader.Height * -1, 1, 1);
using var fileStream = new FileStream(#"Mask.png", FileMode.Create);
var bmpEncoder = new PngBitmapEncoder();
bmpEncoder.Frames.Add(BitmapFrame.Create(image));
bmpEncoder.Save(fileStream);
//Converts the HBitmap to Bitmap, just for testing too.
var image2 = System.Drawing.Image.FromHbitmap(iconInfo.Mask);
image2.Save(#"Mask2.png");
}
Gdi32.DeleteObject(iconInfo.Color);
Gdi32.DeleteObject(iconInfo.Mask);
User32.DestroyIcon(iconHandle);
Gdi32.DeleteObject(cursorInfo.CursorHandle);
Solved:
With the proper interpretation of the bits per pixel of the mask image, this issue was solved.
public static BitmapSource FromArray(byte[] data, int width, int height, int channels, int bitsPerPixel = 32)
{
var format = PixelFormats.Default;
var stride = channels * ((bitsPerPixel * width + 31) / 32);
//Abridged channel and bits by pixel to format converter.
if (channels == 1)
{
format = PixelFormats.BlackWhite;
stride = width / 8;
}
else if (channels == 3)
format = PixelFormats.Bgr24; //RGB.
else if (channels == 4)
format = PixelFormats.Bgr32; //RGB + alpha.
var wbm = new WriteableBitmap(width, height, 96, 96, format, null);
wbm.WritePixels(new Int32Rect(0, 0, width, height), data, stride, 0);
return wbm;
}
2) Get the correct size of the cursor:
As noted above, I'm only getting cursor images of size 32x32 or 32x64 (for masks).
I've tried to increase the size being passed to GetDIBits(), but the image being returned is always the same size.
Even if the size in Windows 10/11 settings is set to a bigger cursor size.
The IDXGIOutputDuplication::GetFramePointerShape() method returns a correct size, so it looks like it's possible. What does this method do in order to get the correct image size?
3) Get the actual cursor frame being displayed (for animated cursors):
As for the animation frame (steps) of the cursor, I don't see a way of getting multiple frames based on an index the same way that I would be able to draw with DrawIconEx():
//If the cursor rate needs to be precisely captured, I could use this undocumented API:
//https://source.winehq.org/source/dlls/user32/cursoricon.c#2325
//int rate = 0, num = 0;
//var ok1 = User32.GetCursorFrameInfo(cursorInfo.hCursor, IntPtr.Zero, 17, ref rate, ref num);
//CursorStep
var ok = User32.DrawIconEx(CompatibleDeviceContext, frame.CursorX - iconInfo.XHotspot, frame.CursorY - iconInfo.YHotspot, cursorInfo.CursorHandle, 0, 0, CursorStep, IntPtr.Zero, 0x0003);
if (!ok)
{
CursorStep = 0;
User32.DrawIconEx(CompatibleDeviceContext, frame.CursorX - iconInfo.XHotspot, frame.CursorY - iconInfo.YHotspot, cursorInfo.CursorHandle, 0, 0, CursorStep, IntPtr.Zero, 0x0003);
}
else
CursorStep++;
Maybe I could draw the cursor color and masks separately using DrawIconEx() into two bitmaps (using the flags DI_IMAGE and DI_MASK in each call, passing the cursor step), and then getting the pixels using GetDIBits()?
But how to do that?

Getting the screen pixels as byte array

I need to change my screen capture code to get a pixel array instead of a Bitmap.
I change the code to this:
BitBlt > Image.FromHbitmap(pointer) > LockBits > pixel array
But, I'm checking if it's possible to cut some middle man, and have something like this:
BitBlt > Marshal.Copy > pixel array
Or even:
WinApi method that gets the screen region as a pixel array
So far, I tried to use this code, without success:
public static byte[] CaptureAsArray(Size size, int positionX, int positionY)
{
var hDesk = GetDesktopWindow();
var hSrce = GetWindowDC(hDesk);
var hDest = CreateCompatibleDC(hSrce);
var hBmp = CreateCompatibleBitmap(hSrce, (int)size.Width, (int)size.Height);
var hOldBmp = SelectObject(hDest, hBmp);
try
{
new System.Security.Permissions.UIPermission(System.Security.Permissions.UIPermissionWindow.AllWindows).Demand();
var b = BitBlt(hDest, 0, 0, (int)size.Width, (int)size.Height, hSrce, positionX, positionY, CopyPixelOperation.SourceCopy | CopyPixelOperation.CaptureBlt);
var length = 4 * (int)size.Width * (int)size.Height;
var bytes = new byte[length];
Marshal.Copy(hBmp, bytes, 0, length);
//return b ? Image.FromHbitmap(hBmp) : null;
return bytes;
}
finally
{
SelectObject(hDest, hOldBmp);
DeleteObject(hBmp);
DeleteDC(hDest);
ReleaseDC(hDesk, hSrce);
}
return null;
}
This code gives me an System.AccessViolationException while stepping on Marshal.Copy.
Is there any more efficient way of getting screen pixels as a byte array while using BitBlt or similar screen capture methods?
EDIT:
As found in here and as suggested by CodyGray, I should use
var b = Native.BitBlt(_compatibleDeviceContext, 0, 0, Width, Height, _windowDeviceContext, Left, Top, Native.CopyPixelOperation.SourceCopy | Native.CopyPixelOperation.CaptureBlt);
var bi = new Native.BITMAPINFOHEADER();
bi.biSize = (uint)Marshal.SizeOf(bi);
bi.biBitCount = 32;
bi.biClrUsed = 0;
bi.biClrImportant = 0;
bi.biCompression = 0;
bi.biHeight = Height;
bi.biWidth = Width;
bi.biPlanes = 1;
var data = new byte[4 * Width * Height];
Native.GetDIBits(_windowDeviceContext, _compatibleBitmap, 0, (uint)Height, data, ref bi, Native.DIB_Color_Mode.DIB_RGB_COLORS);
My data array has all the pixels of the screenshot.
Now, I'm going to test if there's any performance improvements or not.

Yeah, you can't just start accessing the raw bits of a BITMAP object through an HBITMAP (as returned by CreateCompatibleBitmap). HBITMAP is just a handle, as the name suggests. It's not a pointer in the classic "C" sense that it points to the beginning of the array. Handles are like indirect pointers.
GetDIBits is the appropriate solution to get the raw, device-independent pixel array from a bitmap that you can iterate through. But you'll still need to use the code you have to get the screen bitmap in the first place. Essentially, you want something like this. Of course, you'll need to translate it into C#, but that shouldn't be difficult, since you already know how to call WinAPI functions.
Note that you do not need to call GetDesktopWindow or GetWindowDC. Just pass NULL as the handle to GetDC; it has the same effect of returning a screen DC, which you can then use to create a compatible bitmap. In general, you should almost never call GetDesktopWindow.

Copying From and To Clipboard loses image transparency

I've been trying to copy a transparent PNG image to clipboard and preserve its transparency to paste it into a specific program that supports it.
I tried many solutions already but the background always ended up gray in one way or another.
So I tried copying the same image using Chrome and pasting it into the program and it worked. It preserved transparency. So then I tried Getting the image from the Clipboard that I had copied using Chrome and Set the image again, expecting the transparency to still be there - but no, transparency was not preserved even though I just took the image from the clipboard and set it again.
var img = Clipboard.GetImage(); // copied using Chrome and transparency is preserved
Clipboard.SetImage(img); // transparency lost
Same issue even if I use the System.Windows.Forms.Clipboard or try getting and setting the DataObject instead of the Image.

The Windows clipboard, by default, does not support transparency, but you can put content on the clipboard in many types together to make sure most applications find some type in it that they can use. Sadly, the most common type, DeviceIndependentBitmap (which Windows itself seems to use) is a really dirty and unreliable one. I wrote a big rant explanation about that here.
I'll assume you have read through that before continuing with my answer here, because it contains the background information required for the next part.
Now, the cleanest way of putting an image on the clipboard with transparency support is a PNG stream, but it won't guarantee that all applications can paste it. Gimp supports PNG paste, and apparently so do the newer MS Office programs, but Google Chrome, for example, doesn't, and will only accept the messy DIB type detailed in the answer I linked to. On the other hand, Gimp will not accept DIB as having transparency, because its creators actually followed the format's specifications, and realized that the format was unreliable (as clearly demonstrated by that question I linked).
Because of the DIB mess, sadly, the best thing to do is simply to put it in there in as many generally-supported types as you can, including PNG, DIB and the normal Image.
PNG and DIB are both put on the clipboard in the same way: by putting them in the DataObject as MemoryStream, and then giving the clipboard the "copy" instruction when actually putting it on.
Most of this is straightforward, but the DIB one is a bit more complex. Note that the following part contains a couple of references to my own toolsets. The GetImageData one can be found in this answer, the BuildImage one can be found here, and the ArrayUtils ones are given below.
These toolsets all use System.Drawing, though. You'll have to figure out for yourself exactly how to do the same things in WPF.
/// <summary>
/// Copies the given image to the clipboard as PNG, DIB and standard Bitmap format.
/// </summary>
/// <param name="image">Image to put on the clipboard.</param>
/// <param name="imageNoTr">Optional specifically nontransparent version of the image to put on the clipboard.</param>
/// <param name="data">Clipboard data object to put the image into. Might already contain other stuff. Leave null to create a new one.</param>
public static void SetClipboardImage(Bitmap image, Bitmap imageNoTr, DataObject data)
{
Clipboard.Clear();
if (data == null)
data = new DataObject();
if (imageNoTr == null)
imageNoTr = image;
using (MemoryStream pngMemStream = new MemoryStream())
using (MemoryStream dibMemStream = new MemoryStream())
{
// As standard bitmap, without transparency support
data.SetData(DataFormats.Bitmap, true, imageNoTr);
// As PNG. Gimp will prefer this over the other two.
image.Save(pngMemStream, ImageFormat.Png);
data.SetData("PNG", false, pngMemStream);
// As DIB. This is (wrongly) accepted as ARGB by many applications.
Byte[] dibData = ConvertToDib(image);
dibMemStream.Write(dibData, 0, dibData.Length);
data.SetData(DataFormats.Dib, false, dibMemStream);
// The 'copy=true' argument means the MemoryStreams can be safely disposed after the operation.
Clipboard.SetDataObject(data, true);
}
}
/// <summary>
/// Converts the image to Device Independent Bitmap format of type BITFIELDS.
/// This is (wrongly) accepted by many applications as containing transparency,
/// so I'm abusing it for that.
/// </summary>
/// <param name="image">Image to convert to DIB</param>
/// <returns>The image converted to DIB, in bytes.</returns>
public static Byte[] ConvertToDib(Image image)
{
Byte[] bm32bData;
Int32 width = image.Width;
Int32 height = image.Height;
// Ensure image is 32bppARGB by painting it on a new 32bppARGB image.
using (Bitmap bm32b = new Bitmap(image.Width, image.Height, PixelFormat.Format32bppArgb))
{
using (Graphics gr = Graphics.FromImage(bm32b))
gr.DrawImage(image, new Rectangle(0, 0, bm32b.Width, bm32b.Height));
// Bitmap format has its lines reversed.
bm32b.RotateFlip(RotateFlipType.Rotate180FlipX);
Int32 stride;
bm32bData = ImageUtils.GetImageData(bm32b, out stride);
}
// BITMAPINFOHEADER struct for DIB.
Int32 hdrSize = 0x28;
Byte[] fullImage = new Byte[hdrSize + 12 + bm32bData.Length];
//Int32 biSize;
ArrayUtils.WriteIntToByteArray(fullImage, 0x00, 4, true, (UInt32)hdrSize);
//Int32 biWidth;
ArrayUtils.WriteIntToByteArray(fullImage, 0x04, 4, true, (UInt32)width);
//Int32 biHeight;
ArrayUtils.WriteIntToByteArray(fullImage, 0x08, 4, true, (UInt32)height);
//Int16 biPlanes;
ArrayUtils.WriteIntToByteArray(fullImage, 0x0C, 2, true, 1);
//Int16 biBitCount;
ArrayUtils.WriteIntToByteArray(fullImage, 0x0E, 2, true, 32);
//BITMAPCOMPRESSION biCompression = BITMAPCOMPRESSION.BITFIELDS;
ArrayUtils.WriteIntToByteArray(fullImage, 0x10, 4, true, 3);
//Int32 biSizeImage;
ArrayUtils.WriteIntToByteArray(fullImage, 0x14, 4, true, (UInt32)bm32bData.Length);
// These are all 0. Since .net clears new arrays, don't bother writing them.
//Int32 biXPelsPerMeter = 0;
//Int32 biYPelsPerMeter = 0;
//Int32 biClrUsed = 0;
//Int32 biClrImportant = 0;
// The aforementioned "BITFIELDS": colour masks applied to the Int32 pixel value to get the R, G and B values.
ArrayUtils.WriteIntToByteArray(fullImage, hdrSize + 0, 4, true, 0x00FF0000);
ArrayUtils.WriteIntToByteArray(fullImage, hdrSize + 4, 4, true, 0x0000FF00);
ArrayUtils.WriteIntToByteArray(fullImage, hdrSize + 8, 4, true, 0x000000FF);
Array.Copy(bm32bData, 0, fullImage, hdrSize + 12, bm32bData.Length);
return fullImage;
}
Now, as for getting an image off the clipboard, I noticed there is apparently a difference in behaviour between .Net 3.5 and the later ones, which seem to actually use that DIB. Given that difference, and given how unreliable the DIB format is, you'll want to actually check manually for all types, preferably starting with the completely reliable PNG format.
You can get the DataObject from the clipboard with this code:
DataObject retrievedData = Clipboard.GetDataObject() as DataObject;
The CloneImage function used here is basically just the combination of my GetImageData and BuildImage toolsets, ensuring that a new image is created without any backing resources that might mess up; image objects are known to cause crashes when they're based on a Stream that then gets disposed. A compacted and optimised version of it was posted here, in a question well worth reading on the subject of why this cloning is so important.
/// <summary>
/// Retrieves an image from the given clipboard data object, in the order PNG, DIB, Bitmap, Image object.
/// </summary>
/// <param name="retrievedData">The clipboard data.</param>
/// <returns>The extracted image, or null if no supported image type was found.</returns>
public static Bitmap GetClipboardImage(DataObject retrievedData)
{
Bitmap clipboardimage = null;
// Order: try PNG, move on to try 32-bit ARGB DIB, then try the normal Bitmap and Image types.
if (retrievedData.GetDataPresent("PNG", false))
{
MemoryStream png_stream = retrievedData.GetData("PNG", false) as MemoryStream;
if (png_stream != null)
using (Bitmap bm = new Bitmap(png_stream))
clipboardimage = ImageUtils.CloneImage(bm);
}
if (clipboardimage == null && retrievedData.GetDataPresent(DataFormats.Dib, false))
{
MemoryStream dib = retrievedData.GetData(DataFormats.Dib, false) as MemoryStream;
if (dib != null)
clipboardimage = ImageFromClipboardDib(dib.ToArray());
}
if (clipboardimage == null && retrievedData.GetDataPresent(DataFormats.Bitmap))
clipboardimage = new Bitmap(retrievedData.GetData(DataFormats.Bitmap) as Image);
if (clipboardimage == null && retrievedData.GetDataPresent(typeof(Image)))
clipboardimage = new Bitmap(retrievedData.GetData(typeof(Image)) as Image);
return clipboardimage;
}
public static Bitmap ImageFromClipboardDib(Byte[] dibBytes)
{
if (dibBytes == null || dibBytes.Length < 4)
return null;
try
{
Int32 headerSize = (Int32)ArrayUtils.ReadIntFromByteArray(dibBytes, 0, 4, true);
// Only supporting 40-byte DIB from clipboard
if (headerSize != 40)
return null;
Byte[] header = new Byte[40];
Array.Copy(dibBytes, header, 40);
Int32 imageIndex = headerSize;
Int32 width = (Int32)ArrayUtils.ReadIntFromByteArray(header, 0x04, 4, true);
Int32 height = (Int32)ArrayUtils.ReadIntFromByteArray(header, 0x08, 4, true);
Int16 planes = (Int16)ArrayUtils.ReadIntFromByteArray(header, 0x0C, 2, true);
Int16 bitCount = (Int16)ArrayUtils.ReadIntFromByteArray(header, 0x0E, 2, true);
//Compression: 0 = RGB; 3 = BITFIELDS.
Int32 compression = (Int32)ArrayUtils.ReadIntFromByteArray(header, 0x10, 4, true);
// Not dealing with non-standard formats.
if (planes != 1 || (compression != 0 && compression != 3))
return null;
PixelFormat fmt;
switch (bitCount)
{
case 32:
fmt = PixelFormat.Format32bppRgb;
break;
case 24:
fmt = PixelFormat.Format24bppRgb;
break;
case 16:
fmt = PixelFormat.Format16bppRgb555;
break;
default:
return null;
}
if (compression == 3)
imageIndex += 12;
if (dibBytes.Length < imageIndex)
return null;
Byte[] image = new Byte[dibBytes.Length - imageIndex];
Array.Copy(dibBytes, imageIndex, image, 0, image.Length);
// Classic stride: fit within blocks of 4 bytes.
Int32 stride = (((((bitCount * width) + 7) / 8) + 3) / 4) * 4;
if (compression == 3)
{
UInt32 redMask = ArrayUtils.ReadIntFromByteArray(dibBytes, headerSize + 0, 4, true);
UInt32 greenMask = ArrayUtils.ReadIntFromByteArray(dibBytes, headerSize + 4, 4, true);
UInt32 blueMask = ArrayUtils.ReadIntFromByteArray(dibBytes, headerSize + 8, 4, true);
// Fix for the undocumented use of 32bppARGB disguised as BITFIELDS. Despite lacking an alpha bit field,
// the alpha bytes are still filled in, without any header indication of alpha usage.
// Pure 32-bit RGB: check if a switch to ARGB can be made by checking for non-zero alpha.
// Admitted, this may give a mess if the alpha bits simply aren't cleared, but why the hell wouldn't it use 24bpp then?
if (bitCount == 32 && redMask == 0xFF0000 && greenMask == 0x00FF00 && blueMask == 0x0000FF)
{
// Stride is always a multiple of 4; no need to take it into account for 32bpp.
for (Int32 pix = 3; pix < image.Length; pix += 4)
{
// 0 can mean transparent, but can also mean the alpha isn't filled in, so only check for non-zero alpha,
// which would indicate there is actual data in the alpha bytes.
if (image[pix] == 0)
continue;
fmt = PixelFormat.Format32bppPArgb;
break;
}
}
else
// Could be supported with a system that parses the colour masks,
// but I don't think the clipboard ever uses these anyway.
return null;
}
Bitmap bitmap = ImageUtils.BuildImage(image, width, height, stride, fmt, null, null);
// This is bmp; reverse image lines.
bitmap.RotateFlip(RotateFlipType.Rotate180FlipX);
return bitmap;
}
catch
{
return null;
}
}
Because BitConverter always requires that dumb check on system endianness, I got my own ReadIntFromByteArray and WriteIntToByteArray in an ArrayUtils class:
public static void WriteIntToByteArray(Byte[] data, Int32 startIndex, Int32 bytes, Boolean littleEndian, UInt32 value)
{
Int32 lastByte = bytes - 1;
if (data.Length < startIndex + bytes)
throw new ArgumentOutOfRangeException("startIndex", "Data array is too small to write a " + bytes + "-byte value at offset " + startIndex + ".");
for (Int32 index = 0; index < bytes; index++)
{
Int32 offs = startIndex + (littleEndian ? index : lastByte - index);
data[offs] = (Byte)(value >> (8 * index) & 0xFF);
}
}
public static UInt32 ReadIntFromByteArray(Byte[] data, Int32 startIndex, Int32 bytes, Boolean littleEndian)
{
Int32 lastByte = bytes - 1;
if (data.Length < startIndex + bytes)
throw new ArgumentOutOfRangeException("startIndex", "Data array is too small to read a " + bytes + "-byte value at offset " + startIndex + ".");
UInt32 value = 0;
for (Int32 index = 0; index < bytes; index++)
{
Int32 offs = startIndex + (littleEndian ? index : lastByte - index);
value += (UInt32)(data[offs] << (8 * index));
}
return value;
}

Kinect depth detection

I know how to do it in WPF but I have problem for capturing depth in winforms application.
I found some code as below:
private void Kinect_DepthFrameReady(object sender, DepthImageFrameReadyEventArgs e)
{
using (DepthImageFrame depthFrame = e.OpenDepthImageFrame())
{
if (depthFrame != null)
{
Bitmap DepthBitmap = new Bitmap(depthFrame.Width, depthFrame.Height, PixelFormat.Format32bppRgb);
if (_depthPixels.Length != depthFrame.PixelDataLength)
{
_depthPixels = new DepthImagePixel[depthFrame.PixelDataLength];
_mappedDepthLocations = new ColorImagePoint[depthFrame.PixelDataLength];
}
//Copy the depth frame data onto the bitmap
var _pixelData = new short[depthFrame.PixelDataLength];
depthFrame.CopyPixelDataTo(_pixelData);
BitmapData bmapdata = DepthBitmap.LockBits(new Rectangle(0, 0, depthFrame.Width,
depthFrame.Height), ImageLockMode.WriteOnly, DepthBitmap.PixelFormat);
IntPtr ptr = bmapdata.Scan0;
Marshal.Copy(_pixelData, 0, ptr, depthFrame.Width * depthFrame.Height);
DepthBitmap.UnlockBits(bmapdata);
pictureBox2.Image = DepthBitmap;
}
}
}
but this is not giving me the greyScale depth and it's purple. Any improvement or help?

I found the solution myself, by a function to convert the depth frame:
void Kinect_DepthFrameReady(object sender, DepthImageFrameReadyEventArgs e)
{
using (DepthImageFrame depthFrame = e.OpenDepthImageFrame())
{
if (depthFrame != null)
{
this.depthFrame32 = new byte[depthFrame.Width * depthFrame.Height * 4];
//Update the image to the new format
this.depthPixelData = new short[depthFrame.PixelDataLength];
depthFrame.CopyPixelDataTo(this.depthPixelData);
byte[] convertedDepthBits = this.ConvertDepthFrame(this.depthPixelData, ((KinectSensor)sender).DepthStream);
Bitmap bmap = new Bitmap(depthFrame.Width, depthFrame.Height, PixelFormat.Format32bppRgb);
BitmapData bmapdata = bmap.LockBits(new Rectangle(0, 0, depthFrame.Width, depthFrame.Height), ImageLockMode.WriteOnly, bmap.PixelFormat);
IntPtr ptr = bmapdata.Scan0;
Marshal.Copy(convertedDepthBits, 0, ptr, 4 * depthFrame.PixelDataLength);
bmap.UnlockBits(bmapdata);
pictureBox2.Image = bmap;
}
}
}
private byte[] ConvertDepthFrame(short[] depthFrame, DepthImageStream depthStream)
{
//Run through the depth frame making the correlation between the two arrays
for (int i16 = 0, i32 = 0; i16 < depthFrame.Length && i32 < this.depthFrame32.Length; i16++, i32 += 4)
{
// Console.WriteLine(i16 + "," + i32);
//We don’t care about player’s information here, so we are just going to rule it out by shifting the value.
int realDepth = depthFrame[i16] >> DepthImageFrame.PlayerIndexBitmaskWidth;
//We are left with 13 bits of depth information that we need to convert into an 8 bit number for each pixel.
//There are hundreds of ways to do this. This is just the simplest one.
//Lets create a byte variable called Distance.
//We will assign this variable a number that will come from the conversion of those 13 bits.
byte Distance = 0;
//XBox Kinects (default) are limited between 800mm and 4096mm.
int MinimumDistance = 800;
int MaximumDistance = 4096;
//XBox Kinects (default) are not reliable closer to 800mm, so let’s take those useless measurements out.
//If the distance on this pixel is bigger than 800mm, we will paint it in its equivalent gray
if (realDepth > MinimumDistance)
{
//Convert the realDepth into the 0 to 255 range for our actual distance.
//Use only one of the following Distance assignments
//White = Far
//Black = Close
//Distance = (byte)(((realDepth – MinimumDistance) * 255 / (MaximumDistance-MinimumDistance)));
//White = Close
//Black = Far
Distance = (byte)(255 - ((realDepth - MinimumDistance) * 255 / (MaximumDistance - MinimumDistance)));
//Use the distance to paint each layer (R G & of the current pixel.
//Painting R, G and B with the same color will make it go from black to gray
this.depthFrame32[i32 + RedIndex] = (byte)(Distance);
this.depthFrame32[i32 + GreenIndex] = (byte)(Distance);
this.depthFrame32[i32 + BlueIndex] = (byte)(Distance);
}
//If we are closer than 800mm, the just paint it red so we know this pixel is not giving a good value
else
{
this.depthFrame32[i32 + RedIndex] = 0;
this.depthFrame32[i32 + GreenIndex] = 0;
this.depthFrame32[i32 + BlueIndex] = 0;
}
}

so i presume that rgb frame is working out for you in that case:
first to enable depth camera you need to call:
sensor->NuiInitialize(NUI_INITIALIZE_FLAG_USES_DEPTH|all stuff you use also);
second to start streaming you need to call:
if (int(streams&_Kinect_zed)) ret=sensor->NuiImageStreamOpen(
NUI_IMAGE_TYPE_DEPTH, // Depth camera or rgb camera?
NUI_IMAGE_RESOLUTION_640x480, // Image resolution
NUI_IMAGE_STREAM_FLAG_DISTINCT_OVERFLOW_DEPTH_VALUES, // Image stream flags // NUI_IMAGE_STREAM_FLAG_ENABLE_NEAR_MODE nefunguje !!!
2, // Number of frames to buffer
NULL, // Event handle
&stream_hzed); else stream_hzed=NULL;
beware not all resolution/flags combinations work on all models of kinect !!!
this one above is safe even for the older models like mine
this is how i capture frame (called repeatedly from timer or thread loop)
ret=sensor->NuiImageStreamGetNextFrame(stream_hzed,0,&imageFrame); if (ret>=0)
{
// copy data from frame
imageFrame.pFrameTexture->LockRect(0, &LockedRect, NULL, 0);
if (LockedRect.Pitch!=0)
{
const BYTE* curr = (const BYTE*) LockedRect.pBits;
union _col { BYTE u8[2]; WORD u16; } col;
col.u16=0;
pnt3d p;
long ax,ay;
float mxs=float(xs)/(62.0*deg),mys=float(ys)/(48.6*deg);
for(int x=0,y=0;;)
{
col.u8[0]=*curr; curr++;
col.u8[1]=*curr; curr++;
p.raw=col.u16;
p.rgb=&rgb_default;
if (p.raw==0x0000) p.z=0.0; // p.z je kolma vzdialenost od senzora (kinect to correctuje sam)
else if (p.raw>=0x8000) p.z=4.0;
else p.z=0.8+(float(p.raw-6576)*0.00012115165336374002280501710376283);
// depth FOV correction
p.x=zx[x]*p.z;
p.y=zy[y]*p.z;
// color FOV correction zed 58.5° x 45.6° | rgb 62.0° x 48.6° | 25mm distance
if (p.z>0.0)
{
ax=(((x+10-xs2)*241)>>8)+xs2; // cameras x-offset and different FOV
ay=(((y+30-ys2)*240)>>8)+ys2; // cameras y-offset??? and different FOV
if ((ax>=0)&&(ax<xs))
if ((ay>=0)&&(ay<ys)) p.rgb=&rgb[ay][ax];
}
xyz[y][x]=p;
x++; if (x>=xs) { x=0; y++; if (y>=ys) break; }
}
}
// release frame
imageFrame.pFrameTexture->UnlockRect(0);
ret=sensor->NuiImageStreamReleaseFrame(stream_hzed, &imageFrame);
stream_changed|=_Kinect_zed;
}
Sorry for incomplete source code ...
- all is copy pasted from my kinect class (BDS2006 Turbo C++)
- so you need to check your code if you do not forget something
- and if yes then transform my code to C# (i am not C# user)
- most likely you forget to NUIinitialize with depth flag
- or set invalid resolution/flags/ precision or framerate for your HW
if nothing work at all then you need to initialize the sensor in the first place
int sensors;
INuiSensor *sensor;
if ((NUIGetSensorCount(&sensors)<0)||(sensors<1)) return false;
if (NUICreateSensorByIndex(0,&sensor)<0) return false;
if you link to dll on your own then link only these functions:
typedef HRESULT(__stdcall *_NuiGetSensorCount )(int * pCount); _NuiGetSensorCount NUIGetSensorCount =NULL;
typedef HRESULT(__stdcall *_NuiCreateSensorByIndex)(int index,INuiSensor **ppNuiSensor); _NuiCreateSensorByIndex NUICreateSensorByIndex=NULL;
Every other function (must) is obtained via COM inside SDK headers !!!
if you link and use them on your own then you will not be connected to your physical Kinect !!!

Basically kinect sdk is developed for WPf application. In windows form you have convert the short array of the depth data to the BItmap to display it on picturebox. And based on my expriment WPF is better for programming with kinect.
Below is the function that I used to convert depth frame to Bitmap for showing in picture box.
private Bitmap ImageToBitmap(DepthImageFrame Image)
{
short[] pixeldata = new short[Image.PixelDataLength];
int stride = Image.Width * 2;
Image.CopyPixelDataTo(pixeldata);
Bitmap bmap = new Bitmap(Image.Width, Image.Height, PixelFormat.Format16bppRgb555);
BitmapData bmapdata = bmap.LockBits(new Rectangle(0, 0, Image.Width, Image.Height), ImageLockMode.WriteOnly, bmap.PixelFormat);
IntPtr ptr = bmapdata.Scan0;
Marshal.Copy(pixeldata, 0, ptr, Image.PixelDataLength);
bmap.UnlockBits(bmapdata);
return bmap;
}
You may call it like this:
DepthImageFrame VFrame = e.OpenDepthImageFrame();
if (VFrame == null) return;
short[] pixelS = new short[VFrame.PixelDataLength];
Bitmap bmap = ImageToBitmap(VFrame);

Copy a channel from one image to another using safe code

I'm trying to refactor this unsafe code to copy a single ARGB channel from one image to another using System.Runtime.InteropServices.Marshal.Copy as per this example on MSDN but I'm totally lost.
Could anyone walk me through how I would go about it?
public enum ChannelARGB
{
Blue = 0,
Green = 1,
Red = 2,
Alpha = 3
}
public static void transferOneARGBChannelFromOneBitmapToAnother(
Bitmap source,
Bitmap dest,
ChannelARGB sourceChannel,
ChannelARGB destChannel )
{
if ( source.Size!=dest.Size )
throw new ArgumentException();
Rectangle r = new Rectangle( Point.Empty, source.Size );
BitmapData bdSrc = source.LockBits( r,
ImageLockMode.ReadOnly,
PixelFormat.Format32bppArgb );
BitmapData bdDst = dest.LockBits( r,
ImageLockMode.ReadWrite,
PixelFormat.Format32bppArgb );
unsafe
{
byte* bpSrc = (byte*)bdSrc.Scan0.ToPointer();
byte* bpDst = (byte*)bdDst.Scan0.ToPointer();
bpSrc += (int)sourceChannel;
bpDst += (int)destChannel;
for ( int i = r.Height * r.Width; i > 0; i-- )
{
*bpDst = *bpSrc;
bpSrc += 4;
bpDst += 4;
}
}
source.UnlockBits( bdSrc );
dest.UnlockBits( bdDst );
}
Edit
In an attempt to work through #Ben Voigt walk though I have come up with this so far. Unfortunately I am now getting the following error:
Attempted to read or write protected memory. This is often an
indication that other memory is corrupt.
private static void TransferOneArgbChannelFromOneBitmapToAnother(
Bitmap source,
Bitmap destination,
ChannelARGB sourceChannel,
ChannelARGB destinationChannel)
{
if (source.Size != destination.Size)
{
throw new ArgumentException();
}
Rectangle rectangle = new Rectangle(Point.Empty, source.Size);
// Lockbits the source.
BitmapData bitmapDataSource = source.LockBits(rectangle,
ImageLockMode.ReadWrite,
PixelFormat.Format32bppArgb);
// Declare an array to hold the bytes of the bitmap.
int bytes = bitmapDataSource.Stride * bitmapDataSource.Height;
// Allocate a buffer for the source image
byte[] sourceRgbValues = new byte[bytes];
// Get the address of the first line.
IntPtr ptrSource = bitmapDataSource.Scan0;
// Copy the RGB values into the array.
System.Runtime.InteropServices.Marshal.Copy(ptrSource,
sourceRgbValues,
0,
bytes);
// Unlockbits the source.
source.UnlockBits(bitmapDataSource);
// Lockbits the destination.
BitmapData bitmapDataDestination = destination.LockBits(rectangle,
ImageLockMode.ReadWrite,
PixelFormat.Format32bppArgb);
// Allocate a buffer for image
byte[] destinationRgbValues = new byte[bytes];
IntPtr ptrDestination = bitmapDataDestination.Scan0;
// Copy the RGB values into the array.
System.Runtime.InteropServices.Marshal.Copy(ptrDestination,
destinationRgbValues,
0,
bytes);
ptrSource += (int)sourceChannel;
ptrDestination += (int)destinationChannel;
for (int i = rectangle.Height * rectangle.Width; i > 0; i--)
{
destinationRgbValues[i] = sourceRgbValues[i];
ptrSource += 4;
ptrDestination += 4;
}
// Copy the RGB values back to the bitmap
// ******This is where I am getting the exception*******.
System.Runtime.InteropServices.Marshal.Copy(destinationRgbValues,
0,
ptrDestination,
bytes);
// Unlock bits the destination.
destination.UnlockBits(bitmapDataDestination);
}
Can anyone see what I have done wrong? This is all a bit over my head to be honest. I think I should buy some books.

LockBits the source.
Marshal.Copy the source BitmapData to a byte[] buffer.
UnlockBits the source.
LockBits the destination.
Marshal.Copy the destination BitmapData to a byte[] buffer.
Loop through and copy that channel from the source byte[] to the destination byte[] (note, use arithmetic on indexes instead of on pointers)
Marshal.Copy the destination byte[] back to the BitmapData.
UnlockBits the destination.
I'm not sure what the point is, though. Code that uses Marshal.Copy is just as dangerous as code that uses the unsafe keyword, and should require similar code security permission.
A potentially more efficient way would be to use ImageAttributes.SetColorMatrix to remove the desired channel from the destination image, remove all other channels from the source image, and then blend. See the example for ColorMatrix
Or use DirectX (or OpenGL) and a shader that just transfers the one channel.

You could use my simple LINQ based image processing framework from Nuget or Codeplex and write a simple query that swaps the channels around.
You could also use a ColorMatrix to perform the channel swap like in this code.

Unfortunately, a ColorMatrix won't work if you want to combine channels from two separate images. You would need an additive (or bitwise or) blending method, and the only blending provided by GDI+ is Over and Copy. It also looks to me like any methods that would allow you to access the bits directly, including LockBits, are locked down.
I think the only option is to use GetPixel and SetPixel on each pixel, something like this:
Color dstColor = bpDst.GetPixel(x, y);
Color srcColor = bpSrc.GetPixel(x, y);
int srcValue = (srcColor.ToArgb() >> (sourceChannel * 8)) & 0xff;
int dstArgb = (dstColor.ToArgb() & ~(0xff << (destChannel * 8))) | (srcValue << (destChannel * 8));
bpDst.SetPixel(x, y, Color.FromArgb(dstArgb));