Develop Reference

Efficient Bitmap to OnnxRuntime Tensor in C#

I am using Microsoft OnnxRuntime to detect and classify objects in images and I want to apply it to real-time video. To do that, I have to convert each frame into an OnnxRuntime Tensor. Right now I have implemented a method that takes around 300ms:
public Tensor<float> ConvertImageToFloatTensor(Bitmap image)
{
// Create the Tensor with the appropiate dimensions for the NN
Tensor<float> data = new DenseTensor<float>(new[] { 1, image.Width, image.Height, 3 });
// Iterate over the bitmap width and height and copy each pixel
for (int x = 0; x < image.Width; x++)
{
for (int y = 0; y < image.Height; y++)
{
Color color = image.GetPixel(x, y);
data[0, y, x, 0] = color.R / (float)255.0;
data[0, y, x, 1] = color.G / (float)255.0;
data[0, y, x, 2] = color.B / (float)255.0;
}
}
return data;
}
I need this code to run as fast as possible since I am representing the output bounding boxes of the detector as a layer on top of the video. Does anyone know a faster way of doing this conversión?

based in the answers by davidtbernal (Fast work with Bitmaps in C#) and FelipeDurar (Grayscale image from binary data) you should be able to access pixels faster using LockBits and a bit of "unsafe" code
public Tensor<float> ConvertImageToFloatTensorUnsafe(Bitmap image)
{
// Create the Tensor with the appropiate dimensions for the NN
Tensor<float> data = new DenseTensor<float>(new[] { 1, image.Width, image.Height, 3 });
BitmapData bmd = image.LockBits(new Rectangle(0, 0, image.Width, image.Height), System.Drawing.Imaging.ImageLockMode.ReadOnly, image.PixelFormat);
int PixelSize = 3;
unsafe
{
for (int y = 0; y < bmd.Height; y++)
{
// row is a pointer to a full row of data with each of its colors
byte* row = (byte*)bmd.Scan0 + (y * bmd.Stride);
for (int x = 0; x < bmd.Width; x++)
{
// note the order of colors is BGR
data[0, y, x, 0] = row[x*PixelSize + 2] / (float)255.0;
data[0, y, x, 1] = row[x*PixelSize + 1] / (float)255.0;
data[0, y, x, 2] = row[x*PixelSize + 0] / (float)255.0;
}
}
image.UnlockBits(bmd);
}
return data;
}
I've compared this piece of code averaging over 1000 runs and got about 3x performance improvement against your original code but results may vary.
Also note I've used 3 channels per pixel as your original answer uses those values only, if you use a 32bpp bitmap, you may change PixelSize to 4 and the last channel should be alpha channel (row[x*PixelSize + 3])

RenderTargetBitmap with format rgba32

I am not very familiar with bitmaps and I need to save a FrameworkElement (specificaly Grid) as bitmap and copy it to buffer. The problem is I need to save it in Rgba format, not Pgrba, which isn't supported in RenderTargetBitmap. Relevant code is here:
_targetBitmap = new RenderTargetBitmap(xres, yres, 96, 96, PixelFormats.Pbgra32);
_targetBitmap.Clear();
// Child is grid
_targetBitmap.Render(Child);
// copy the pixels into the buffer
_targetBitmap.CopyPixels(new Int32Rect(0, 0, xres, yres), bufferPtr, _bufferSize, _stride);
I tried using WriteableBitmap, but I didn't how to render the Child. Any suggestions?

The CopyPixels function is already giving you direct access to the pixel data, so all you need to do is convert between formats. In this case, you need to swap the channel orders around and undo the premultiplication of alpha values.
NOTE: This code assumes your bufferPtr is a byte array or a byte pointer.
for (int y = 0; y < yres; y++)
{
for (int x = 0; x < xres; x++)
{
// Calculate array offset for this pixel
int offset = y * _stride + x * 4;
// Extract individual color channels from pixel value
int pb = bufferPtr[offset];
int pg = bufferPtr[offset + 1];
int pr = bufferPtr[offset + 2];
int alpha = bufferPtr[offset + 3];
// Remove premultiplication
int r = 0, g = 0, b = 0;
if (alpha > 0)
{
r = pr * 255 / alpha;
g = pg * 255 / alpha;
b = pb * 255 / alpha;
}
// Write color channels in desired order
bufferPtr[offset] = (byte)r;
bufferPtr[offset + 1] = (byte)g;
bufferPtr[offset + 2] = (byte)b;
bufferPtr[offset + 3] = (byte)alpha;
}
}

How to properly address 16bpp with pointers in C#

I am trying to copy camera metadata into a Bitmap, and seing as each value in the metadata is a 16bit (or ushort) I thought it would be sensible to display it in a 16bpp garyscale Bitmap. The code I wrote is as follows:
// Getting the metadata from the device
metaData = new DepthMetaData();
dataSource.GetMetaData(metaData);
// Setting up bitmap, rect and data to use pointer
Bitmap bitmap = new Bitmap(metaData.XRes, metaData.YRes, PixelFormat.Format16bppGrayScale);
Rectangle rect = new Rectangle(0, 0, bitmap.Width, bitmap.Height);
BitmapData data = bitmap.LockBits(rect, ImageLockMode.WriteOnly, PixelFormat.Format16bppGrayScale);
// Pointer pointing to metadata
ushort* ptrMetaData = (ushort*)dataSource.DepthMapPtr.ToPointer();
lock(this)
{
// Runs through the whole bitmap and assigns the entry in the metadata
// to a pixel
for (int y = 0; y < bitmap.Height; ++y)
{
ushort* ptrDestination = (ushort*)data.Scan0.ToPointer() + y * data.Stride;
for (int x = 0; x < bitmap.Width; ++x, ++ptrMetaData)
{
ptrDestination[x] = (ushort)*ptrMetaData;
}
}
}
// Once done unlock the bitmap so that it can be read again
bitmap.UnlockBits(data);
When running the Metadata's XRes = 640 and YRes = 480. The code throws a memory access exception in the for-loops on "ptrDestination[x] = (ushort)*ptrMetaData;" after only running though 240, half the total, lines.
I used this with 8bpp where I reduced the resolution and it worked nicely, so I don't see why it should not here. Maybe someone finds the problem.
Thanks already

ushort* ptrDestination = (ushort*)data.Scan0.ToPointer() + y * data.Stride;
The data.Stride value is expressed in bytes, not ushorts. So the pointer is off by a factor of 2 so it bombs at bitmap.Height/2. Your for loops are broken, swap bitmap.Width and bitmap.Height. The lock keyword doesn't make much sense here, you are accessing thread-local data, other than dataSource. Fix:
for (int y = 0; y < bitmap.Height; ++y)
{
ushort* ptrDestination = (ushort*)data.Scan0.ToPointer() + y * data.Stride / 2;
for (int x = 0; x < bitmap.Width; ++x, ++ptrMetaData)
{
ptrDestination[x] = (ushort)*ptrMetaData;
}
}

Determine image overall lightness

I need to overlay some texts on an image; this text should be lighter or darker based on the overall image lightness.
How to compute the overall (perceived) lightness of an image?
Found something interesting for single pixel:
Formula to determine brightness of RGB color

Solved by me:
public static double CalculateAverageLightness(Bitmap bm)
{
double lum = 0;
var tmpBmp = new Bitmap(bm);
var width = bm.Width;
var height = bm.Height;
var bppModifier = bm.PixelFormat == PixelFormat.Format24bppRgb ? 3 : 4;
var srcData = tmpBmp.LockBits(new Rectangle(0, 0, bm.Width, bm.Height), ImageLockMode.ReadOnly, bm.PixelFormat);
var stride = srcData.Stride;
var scan0 = srcData.Scan0;
//Luminance (standard, objective): (0.2126*R) + (0.7152*G) + (0.0722*B)
//Luminance (perceived option 1): (0.299*R + 0.587*G + 0.114*B)
//Luminance (perceived option 2, slower to calculate): sqrt( 0.299*R^2 + 0.587*G^2 + 0.114*B^2 )
unsafe
{
byte* p = (byte*)(void*)scan0;
for (int y = 0; y < height; y++)
{
for (int x = 0; x < width; x++)
{
int idx = (y * stride) + x * bppModifier;
lum += (0.299*p[idx + 2] + 0.587*p[idx + 1] + 0.114*p[idx]);
}
}
}
tmpBmp.UnlockBits(srcData);
tmpBmp.Dispose();
var avgLum = lum / (width * height);
return avgLum/255.0;
}

I think all you can do is measure every pixel in the image and take an average. If thats too slow for your purposes then I would suggest taking an evenly distributed sample of pixels and using that to calculate an average. You could also limit the pixels to the area where you need to draw the text.
You can load the image as a Bitmap (http://msdn.microsoft.com/en-us/library/system.drawing.bitmap.aspx) and use the GetPixel method to actually get the colour values.
How you assess the brightness is entirely up to you. I would suggest a simpler approach (say just taking the highest colour value) may actually be better as some users will perceive colour differently to the human norm (colour-blindness etc).

Help with Bitmap decoder

I've been working on a bitmap decoder, but my algorithm for processing the pixel data doesn't seem to be quite right:
public IntPtr ReadPixels(Stream fs, int offset, int width, int height, int bpp)
{
IntPtr bBits;
int pixelCount = bpp * width * height;
int Row = 0;
decimal value = ((bpp*width)/32)/4;
int RowSize = (int)Math.Ceiling(value);
int ArraySize = RowSize * Math.Abs(height);
int Col = 0;
Byte[] BMPData = new Byte[ArraySize];
BinaryReader r = new BinaryReader(fs);
r.BaseStream.Seek(offset, SeekOrigin.Begin);
while (Row < height)
{
Byte ReadByte;
if (!(Col >= RowSize))
{
ReadByte = r.ReadByte();
BMPData[(Row * RowSize) + Col] = ReadByte;
Col += 1;
}
if (Col >= RowSize)
{
Col = 0;
Row += 1;
}
}
bBits = System.Runtime.InteropServices.Marshal.AllocHGlobal(BMPData.Length);
System.Runtime.InteropServices.Marshal.Copy(BMPData, 0, bBits, BMPData.Length);
return bBits;
}
I can process only monochrome bitmaps and on some, parts of the bitmap is processed fine. None are compressed and they are rendered upside down and flipped around. I really could do with some help on this one.

decimal value = ((bpp*width)/32)/4;
int RowSize = (int)Math.Ceiling(value);
That isn't correct. Your RowSize variable is actually called "stride". You compute it like this:
int bytes = (width * bitsPerPixel + 7) / 8;
int stride = 4 * ((bytes + 3) / 4);

You are ignoring the stride.
Image rows can be padded to the left with additional Bytes to make their size divide by a number such as (1 = no padding, 2, 4, 8 = default for many images, 16, ...).
Also, images can be a rectangle region within a larger image, making the "padding" between lines in the smaller image even larger (since the stride is the larger image's stride). - In this case the image can also have an offset for its start point within the buffer.
Better practice is:
// Overload this method 3 time for different bit per SUB-pixel values (8, 16, or 32)
// = (byte, int, float)
// SUB-pixel != pixel (= 1 3 or 4 sub-pixels (grey or RGB or BGR or BGRA or RGBA or ARGB or ABGR)
unsafe
{
byte[] buffer = image.Buffer;
int stride = image.buffer.Length / image.PixelHeight;
// or int stride = image.LineSize; (or something like that)
fixed (float* regionStart = (float*)(void*)buffer) // or byte* or int* depending on datatype
{
for (int y = 0; y < height; y++) // height in pixels
{
// float* and float or byte* and byte or int* and int
float* currentPos
= regionStart + offset / SizeOf(float) + stride / SizeOf(float) * y;
for (int x = 0; x < width; x++) // width in pixels
{
for (int chan = 0; chan < channel; chan++) // 1, 3 or 4 channels
{
// DO NOT USE DECIMAL - you want accurate image values
// with best performance - primative types
// not a .NET complex type used for nice looking values for users e.g. 12.34
// instead use actual sub pixel type (float/int/byte) or double instead!
var currentValue = value;
currentPos++;
}
}
}
}
}

I find something I don't understand:
decimal value = ((bpp*width)/32)/4;
int RowSize = (int)Math.Ceiling(value);
RowSize, in my opinion, should be (bpp*width) / 8 + (bpp%8==0?0:1)