Develop Reference

Efficient Bitmap to OnnxRuntime Tensor in C#

I am using Microsoft OnnxRuntime to detect and classify objects in images and I want to apply it to real-time video. To do that, I have to convert each frame into an OnnxRuntime Tensor. Right now I have implemented a method that takes around 300ms:
public Tensor<float> ConvertImageToFloatTensor(Bitmap image)
{
// Create the Tensor with the appropiate dimensions for the NN
Tensor<float> data = new DenseTensor<float>(new[] { 1, image.Width, image.Height, 3 });
// Iterate over the bitmap width and height and copy each pixel
for (int x = 0; x < image.Width; x++)
{
for (int y = 0; y < image.Height; y++)
{
Color color = image.GetPixel(x, y);
data[0, y, x, 0] = color.R / (float)255.0;
data[0, y, x, 1] = color.G / (float)255.0;
data[0, y, x, 2] = color.B / (float)255.0;
}
}
return data;
}
I need this code to run as fast as possible since I am representing the output bounding boxes of the detector as a layer on top of the video. Does anyone know a faster way of doing this conversión?

based in the answers by davidtbernal (Fast work with Bitmaps in C#) and FelipeDurar (Grayscale image from binary data) you should be able to access pixels faster using LockBits and a bit of "unsafe" code
public Tensor<float> ConvertImageToFloatTensorUnsafe(Bitmap image)
{
// Create the Tensor with the appropiate dimensions for the NN
Tensor<float> data = new DenseTensor<float>(new[] { 1, image.Width, image.Height, 3 });
BitmapData bmd = image.LockBits(new Rectangle(0, 0, image.Width, image.Height), System.Drawing.Imaging.ImageLockMode.ReadOnly, image.PixelFormat);
int PixelSize = 3;
unsafe
{
for (int y = 0; y < bmd.Height; y++)
{
// row is a pointer to a full row of data with each of its colors
byte* row = (byte*)bmd.Scan0 + (y * bmd.Stride);
for (int x = 0; x < bmd.Width; x++)
{
// note the order of colors is BGR
data[0, y, x, 0] = row[x*PixelSize + 2] / (float)255.0;
data[0, y, x, 1] = row[x*PixelSize + 1] / (float)255.0;
data[0, y, x, 2] = row[x*PixelSize + 0] / (float)255.0;
}
}
image.UnlockBits(bmd);
}
return data;
}
I've compared this piece of code averaging over 1000 runs and got about 3x performance improvement against your original code but results may vary.
Also note I've used 3 channels per pixel as your original answer uses those values only, if you use a 32bpp bitmap, you may change PixelSize to 4 and the last channel should be alpha channel (row[x*PixelSize + 3])

How to detect custom object using EmguCV

I'm working on some object detection code. So, I've made a training and got .pb and graph.pbtxt files from tensorflow. The next thing that I've done, was python code, which performs my object detection based on these 2 files, using opencv for Python.
Here is my python script, which works well:
import cv2 as cv
cvNet = cv.dnn.readNetFromTensorflow('frozen_inference_graph.pb', 'graph.pbtxt')
img = cv.imread('75.png')
rows = img.shape[0]
cols = img.shape[1]
cvNet.setInput(cv.dnn.blobFromImage(img, size=(300, 300), swapRB=True, crop=False))
cvOut = cvNet.forward()
print(rows)
print(cols)
for detection in cvOut[0,0,:,:]:
print(type(cvOut[0,0,:,:]))
score = float(detection[2])
if score > 0.1:
left = detection[3] * cols
top = detection[4] * rows
right = detection[5] * cols
bottom = detection[6] * rows
cv.rectangle(img, (int(left), int(top)), (int(right), int(bottom)), (0, 0, 255), thickness=2)
print('true')
print(score)
cv.imshow('img', cv.resize(img, None, fx=0.3, fy=0.3))
cv.waitKey()
However, I need the same code, done with .NET (C#), using EmguCV library, which is a wrapper of conventional OpenCV.
Here is a part of code, that I've managed to write:
private bool RecognizeCO(string fileName)
{
Image<Bgr, byte> img = new Image<Bgr, byte>(fileName);
int cols = img.Width;
int rows = img.Height;
imageBox2.Image = img;
Net netcfg = DnnInvoke.ReadNetFromTensorflow("CO.pb", "graph.pbtxt");
netcfg.SetInput(DnnInvoke.BlobFromImage(img));
Mat mat = netcfg.Forward();
return false;
}
Unfortunately, I have no idea what to do after that.... Actually, I nedd the same result in this C# code, like in Python one. I know, that I can just to call python scripts from C#, but I really need this code to be done in C# with EmguCV.
Please, help me!
Thank you in advance for your help!

So, finally I've managed to end up that code...
the solution was quite easy:
after getting mat variable, we can just get Data from Mat as an float[,,,] array: float[,,,] flt = (float[,,,])mat.GetData();
or just use one-dimensional array: float[] flt = (float[])mat.GetData(jagged:false) (but I prefer the previous one)
Than, just make a loop threw that array:
for (int x = 0; x < flt.GetLength(2); x++)
{
if (flt[0, 0, x, 2] > 0.1)
{
int left = Convert.ToInt32(flt[0, 0, x, 3] * cols);
int top = Convert.ToInt32(flt[0, 0, x, 4] * rows);
int right = Convert.ToInt32(flt[0, 0, x, 5] * cols);
int bottom = Convert.ToInt32(flt[0, 0, x, 6] * rows);
image1.Draw(new Rectangle(left, top, right - left, bottom - top), new Bgr(0, 0, 255), 2);
}
}
And finally, we can save that image:
image1.Save("testing-1.png");
So, the result code wil look like:
using (Image<Bgr, byte> image1 = new Image<Bgr, byte>("testing.png"))
{
int interception = 0;
int cols = image1.Width;
int rows = image1.Height;
Net netcfg = DnnInvoke.ReadNetFromTensorflow(Directory.GetCurrentDirectory() + #"\fldr\CO.pb", Directory.GetCurrentDirectory() + #"\fldr\graph.pbtxt");
netcfg.SetInput(DnnInvoke.BlobFromImage(image1.Mat, 1, new System.Drawing.Size(300, 300), default(MCvScalar), true, false));
Mat mat = netcfg.Forward();
float[,,,] flt = (float[,,,])mat.GetData();
for (int x = 0; x < flt.GetLength(2); x++)
{
if (flt[0, 0, x, 2] > 0.1)
{
int left = Convert.ToInt32(flt[0, 0, x, 3] * cols);
int top = Convert.ToInt32(flt[0, 0, x, 4] * rows);
int right = Convert.ToInt32(flt[0, 0, x, 5] * cols);
int bottom = Convert.ToInt32(flt[0, 0, x, 6] * rows);
image1.Draw(new Rectangle(left, top, right - left, bottom - top), new Bgr(0, 0, 255), 2);
}
}
image1.Save("testing-1.png");
}

How to draw an audio waveform to a bitmap

I am attempting to extract the audio content of a wav file and export the resultant waveform as an image (bmp/jpg/png).
So I have found the following code which draws a sine wave and works as expected:
string filename = #"C:\0\test.bmp";
int width = 640;
int height = 480;
Bitmap b = new Bitmap(width, height);
for (int i = 0; i < width; i++)
{
int y = (int)((Math.Sin((double)i * 2.0 * Math.PI / width) + 1.0) * (height - 1) / 2.0);
b.SetPixel(i, y, Color.Black);
}
b.Save(filename);
This works completely as expected, what I would like to do is replace
int y = (int)((Math.Sin((double)i * 2.0 * Math.PI / width) + 1.0) * (height - 1) / 2.0);
with something like
int y = converted and scaled float from monoWaveFileFloatValues
So how would I best go about doing this in the simplest manner possible?
I have 2 basic issues I need to deal with (i think)
convert float to int in a way which does not loose information, this is due to SetPixel(i, y, Color.Black); where x & y are both int
sample skipping on the x axis so the waveform fits into the defined space audio length / image width give the number of samples to average out intensity over which would be represented by a single pixel
The other options is find another method of plotting the waveform which does not rely on the method noted above. Using a chart might be a good method, but I would like to be able to render the image directly if possible
This is all to be run from a console application and I have the audio data (minus the header) already in a float array.
UPDATE 1
The following code enabled me to draw the required output using System.Windows.Forms.DataVisualization.Charting but it took about 30 seconds to process 27776 samples and whilst it does do what I need, it is far too slow to be useful. So I am still looking towards a solution which will draw the bitmap directly.
System.Windows.Forms.DataVisualization.Charting.Chart chart = new System.Windows.Forms.DataVisualization.Charting.Chart();
chart.Size = new System.Drawing.Size(640, 320);
chart.ChartAreas.Add("ChartArea1");
chart.Legends.Add("legend1");
// Plot {sin(x), 0, 2pi}
chart.Series.Add("sin");
chart.Series["sin"].LegendText = args[0];
chart.Series["sin"].ChartType = System.Windows.Forms.DataVisualization.Charting.SeriesChartType.Spline;
//for (double x = 0; x < 2 * Math.PI; x += 0.01)
for (int x = 0; x < audioDataLength; x ++)
{
//chart.Series["sin"].Points.AddXY(x, Math.Sin(x));
chart.Series["sin"].Points.AddXY(x, leftChannel[x]);
}
// Save sin_0_2pi.png image file
chart.SaveImage(#"c:\tmp\example.png", System.Drawing.Imaging.ImageFormat.Png);
Output shown below:

So I managed to figure it out using a code sample found here, though I made some minor changes to the way I interact with it.
public static Bitmap DrawNormalizedAudio(List<float> data, Color foreColor, Color backColor, Size imageSize, string imageFilename)
{
Bitmap bmp = new Bitmap(imageSize.Width, imageSize.Height);
int BORDER_WIDTH = 0;
float width = bmp.Width - (2 * BORDER_WIDTH);
float height = bmp.Height - (2 * BORDER_WIDTH);
using (Graphics g = Graphics.FromImage(bmp))
{
g.Clear(backColor);
Pen pen = new Pen(foreColor);
float size = data.Count;
for (float iPixel = 0; iPixel < width; iPixel += 1)
{
// determine start and end points within WAV
int start = (int)(iPixel * (size / width));
int end = (int)((iPixel + 1) * (size / width));
if (end > data.Count)
end = data.Count;
float posAvg, negAvg;
averages(data, start, end, out posAvg, out negAvg);
float yMax = BORDER_WIDTH + height - ((posAvg + 1) * .5f * height);
float yMin = BORDER_WIDTH + height - ((negAvg + 1) * .5f * height);
g.DrawLine(pen, iPixel + BORDER_WIDTH, yMax, iPixel + BORDER_WIDTH, yMin);
}
}
bmp.Save(imageFilename);
bmp.Dispose();
return null;
}
private static void averages(List<float> data, int startIndex, int endIndex, out float posAvg, out float negAvg)
{
posAvg = 0.0f;
negAvg = 0.0f;
int posCount = 0, negCount = 0;
for (int i = startIndex; i < endIndex; i++)
{
if (data[i] > 0)
{
posCount++;
posAvg += data[i];
}
else
{
negCount++;
negAvg += data[i];
}
}
if (posCount > 0)
posAvg /= posCount;
if (negCount > 0)
negAvg /= negCount;
}
In order to get it working I had to do a couple of things prior to calling the method DrawNormalizedAudio you can see below what I needed to do:
Size imageSize = new Size();
imageSize.Width = 1000;
imageSize.Height = 500;
List<float> lst = leftChannel.OfType<float>().ToList(); //change float array to float list - see link below
DrawNormalizedAudio(lst, Color.Red, Color.White, imageSize, #"c:\tmp\example2.png");
* change float array to float list
The result of this is as follows, a waveform representation of a hand clap wav sample:
I am quite sure there needs to be some updates/revisions to the code, but it's a start and hopefully this will assist someone else who is trying to do the same thing I was.
If you can see any improvements that can be made, let me know.
UPDATES
NaN issue mentioned in the comments now resolved and code above updated.
Waveform Image updated to represent output fixed by removal of NaN values as noted in point 1.
UPDATE 1
Average level (not RMS) was determined by summing the max level for each sample point and dividing by the total number of samples. Examples of this can be seen below:
Silent Wav File:
Hand Clap Wav File:
Brownian, Pink & White Noise Wav File:

Here is a variation you may want to study. It scales the Graphics object so it can use the float data directly.
Note how I translate (i.e. move) the drawing area twice so I can do the drawing more conveniently!
It also uses the DrawLines method for drawing. The benefit in addition to speed is that the lines may be semi-transparent or thicker than one pixel without getting artifacts at the joints. You can see the center line shine through.
To do this I convert the float data to a List<PointF> using a little Linq magick.
I also make sure to put all GDI+ objects I create in using clause so they will get disposed of properly.
...
using System.Windows.Forms;
using System.IO;
using System.Drawing;
using System.Drawing.Imaging;
using System.Drawing.Drawing2D;
..
..
class Program
{
static void Main(string[] args)
{
float[] data = initData(10000);
Size imgSize = new Size(1000, 400);
Bitmap bmp = drawGraph(data, imgSize , Color.Green, Color.Black);
bmp.Save("D:\\wave.png", ImageFormat.Png);
}
static float[] initData(int count)
{
float[] data = new float[count];
for (int i = 0; i < count; i++ )
{
data[i] = (float) ((Math.Sin(i / 12f) * 880 + Math.Sin(i / 15f) * 440
+ Math.Sin(i / 66) * 110) / Math.Pow( (i+1), 0.33f));
}
return data;
}
static Bitmap drawGraph(float[] data, Size size, Color ForeColor, Color BackColor)
{
Bitmap bmp = new System.Drawing.Bitmap(size.Width, size.Height,
PixelFormat.Format32bppArgb);
Padding borders = new Padding(20, 20, 10, 50);
Rectangle plotArea = new Rectangle(borders.Left, borders.Top,
size.Width - borders.Left - borders.Right,
size.Height - borders.Top - borders.Bottom);
using (Graphics g = Graphics.FromImage(bmp))
using (Pen pen = new Pen(Color.FromArgb(224, ForeColor),1.75f))
{
g.SmoothingMode = SmoothingMode.AntiAlias;
g.Clear(Color.Silver);
using (SolidBrush brush = new SolidBrush(BackColor))
g.FillRectangle(brush, plotArea);
g.DrawRectangle(Pens.LightGoldenrodYellow, plotArea);
g.TranslateTransform(plotArea.Left, plotArea.Top);
g.DrawLine(Pens.White, 0, plotArea.Height / 2,
plotArea.Width, plotArea.Height / 2);
float dataHeight = Math.Max( data.Max(), - data.Min()) * 2;
float yScale = 1f * plotArea.Height / dataHeight;
float xScale = 1f * plotArea.Width / data.Length;
g.ScaleTransform(xScale, yScale);
g.TranslateTransform(0, dataHeight / 2);
var points = data.ToList().Select((y, x) => new { x, y })
.Select(p => new PointF(p.x, p.y)).ToList();
g.DrawLines(pen, points.ToArray());
g.ResetTransform();
g.DrawString(data.Length.ToString("###,###,###,##0") + " points plotted.",
new Font("Consolas", 14f), Brushes.Black,
plotArea.Left, plotArea.Bottom + 2f);
}
return bmp;
}
}

SoftwareBitmapEditor setPixel or getPixel switches colors

Hi i was making some image processing for UWP using the SoftwareBitmapEditor class and after using it a bit with B&W images, but when I had switched to color images i realized that getPixel or setPixel aren't working properly. Am I doing something wrong or there is a big bug in the SoftwareBitmapEditor class? Why is the following code changing the colors in the image?
for (uint y = 0; y < _editor.height; y++)
{
for (uint x = 0; x < _editor.width; x++)
{
var pixel = _editor.getPixel(x, y);
_editor.setPixel(x, y, pixel.r, pixel.g, pixel.b);
}
}
And this not
for (uint y = 0; y < _editor.height; y++)
{
for (uint x = 0; x < _editor.width; x++)
{
var pixel = _editor.getPixel(x, y);
_editor.setPixel(x, y, pixel.r, pixel.b, pixel.g);
}
}
Even though this is the signature for setPixel
public void setPixel(uint posX, uint posY, byte r, byte g, byte b);

Am I doing something wrong or there is a big bug in the SoftwareBitmapEditor class?
I think this is a bug of this class.
Why is the following code changing the colors in the image?
Before we use setPixel method, we need to get the pixel data of the SoftwareBitmap first using getPixel method. And you can refer to the source code of this class, the getPixel method is like this:
public SoftwareBitmapPixel getPixel(uint posX, uint posY)
{
var inputCurrPixel = desc.StartIndex + desc.Stride * posY + BYTES_PER_PIXEL * posX;
SoftwareBitmapPixel ret = new SoftwareBitmapPixel();
unsafe
{
ret.r = pixels[inputCurrPixel + 2];
ret.b = pixels[inputCurrPixel + 1];
ret.g = pixels[inputCurrPixel + 0];
}
return ret;
}
As you can see, the rgb order in the getPixel is different as it in the setPixel:
public void setPixel(uint posX, uint posY, Byte r, Byte g, Byte b)
{
unsafe
{
var inputCurrPixel = desc.StartIndex + desc.Stride * posY + BYTES_PER_PIXEL * posX;
pixels[inputCurrPixel + 0] = b; // Blue
pixels[inputCurrPixel + 1] = g; // Green
pixels[inputCurrPixel + 2] = r; // Red
}
}
So here comes to your problem. One solution is as you did:
editor.setPixel(x, y, pixel.r, pixel.b, pixel.g);
Another method is:
editor.setPixel(x, y, pixel.r, b: pixel.g, g: pixel.b);
You can submit an issue on Github.

Performance issue while converting Rgb image to grayscale C# Code

I am writing a .Net wrapper for Tesseract Ocr and if I use a grayscale image instead of rgb image as an input file to it then results are pretty good.
So I was searching the web for C# solution to convert a Rgb image to grayscale image and I found this code.
This performs 3 operations to increase the accuracy of tesseract.
Resize the image
then convert into grayscale image and remove noise from image
Now this converted image gives almost 90% accurate results.
//Resize
public Bitmap Resize(Bitmap bmp, int newWidth, int newHeight)
{
Bitmap temp = (Bitmap)bmp;
Bitmap bmap = new Bitmap(newWidth, newHeight, temp.PixelFormat);
double nWidthFactor = (double)temp.Width / (double)newWidth;
double nHeightFactor = (double)temp.Height / (double)newHeight;
double fx, fy, nx, ny;
int cx, cy, fr_x, fr_y;
Color color1 = new Color();
Color color2 = new Color();
Color color3 = new Color();
Color color4 = new Color();
byte nRed, nGreen, nBlue;
byte bp1, bp2;
for (int x = 0; x < bmap.Width; ++x)
{
for (int y = 0; y < bmap.Height; ++y)
{
fr_x = (int)Math.Floor(x * nWidthFactor);
fr_y = (int)Math.Floor(y * nHeightFactor);
cx = fr_x + 1;
if (cx >= temp.Width)
cx = fr_x;
cy = fr_y + 1;
if (cy >= temp.Height)
cy = fr_y;
fx = x * nWidthFactor - fr_x;
fy = y * nHeightFactor - fr_y;
nx = 1.0 - fx;
ny = 1.0 - fy;
color1 = temp.GetPixel(fr_x, fr_y);
color2 = temp.GetPixel(cx, fr_y);
color3 = temp.GetPixel(fr_x, cy);
color4 = temp.GetPixel(cx, cy);
// Blue
bp1 = (byte)(nx * color1.B + fx * color2.B);
bp2 = (byte)(nx * color3.B + fx * color4.B);
nBlue = (byte)(ny * (double)(bp1) + fy * (double)(bp2));
// Green
bp1 = (byte)(nx * color1.G + fx * color2.G);
bp2 = (byte)(nx * color3.G + fx * color4.G);
nGreen = (byte)(ny * (double)(bp1) + fy * (double)(bp2));
// Red
bp1 = (byte)(nx * color1.R + fx * color2.R);
bp2 = (byte)(nx * color3.R + fx * color4.R);
nRed = (byte)(ny * (double)(bp1) + fy * (double)(bp2));
bmap.SetPixel(x, y, System.Drawing.Color.FromArgb(255, nRed, nGreen, nBlue));
}
}
//here i included the below to functions logic without the for loop to remove repetitive use of for loop but it did not work and taking the same time.
bmap = SetGrayscale(bmap);
bmap = RemoveNoise(bmap);
return bmap;
}
//SetGrayscale
public Bitmap SetGrayscale(Bitmap img)
{
Bitmap temp = (Bitmap)img;
Bitmap bmap = (Bitmap)temp.Clone();
Color c;
for (int i = 0; i < bmap.Width; i++)
{
for (int j = 0; j < bmap.Height; j++)
{
c = bmap.GetPixel(i, j);
byte gray = (byte)(.299 * c.R + .587 * c.G + .114 * c.B);
bmap.SetPixel(i, j, Color.FromArgb(gray, gray, gray));
}
}
return (Bitmap)bmap.Clone();
}
//RemoveNoise
public Bitmap RemoveNoise(Bitmap bmap)
{
for (var x = 0; x < bmap.Width; x++)
{
for (var y = 0; y < bmap.Height; y++)
{
var pixel = bmap.GetPixel(x, y);
if (pixel.R < 162 && pixel.G < 162 && pixel.B < 162)
bmap.SetPixel(x, y, Color.Black);
}
}
for (var x = 0; x < bmap.Width; x++)
{
for (var y = 0; y < bmap.Height; y++)
{
var pixel = bmap.GetPixel(x, y);
if (pixel.R > 162 && pixel.G > 162 && pixel.B > 162)
bmap.SetPixel(x, y, Color.White);
}
}
return bmap;
}
But the problem is it takes lot of time to convert it
So I included SetGrayscale(Bitmap bmap)
RemoveNoise(Bitmap bmap) function logic inside the Resize() method to remove repetitive use of for loop
but it did not solve my problem.

The Bitmap class's GetPixel() and SetPixel() methods are notoriously slow for multiple read/writes. A much faster way to access and set individual pixels in a bitmap is to lock it first.
There's a good example here on how to do that, with a nice class LockedBitmap to wrap around the stranger Marshaling code.
Essentially what it does is use the LockBits() method in the Bitmap class, passing a rectangle for the region of the bitmap you want to lock, and then copy those pixels from its unmanaged memory location to a managed one for easier access.
Here's an example on how you would use that example class with your SetGrayscale() method:
public Bitmap SetGrayscale(Bitmap img)
{
LockedBitmap lockedBmp = new LockedBitmap(img.Clone());
lockedBmp.LockBits(); // lock the bits for faster access
Color c;
for (int i = 0; i < lockedBmp.Width; i++)
{
for (int j = 0; j < lockedBmp.Height; j++)
{
c = lockedBmp.GetPixel(i, j);
byte gray = (byte)(.299 * c.R + .587 * c.G + .114 * c.B);
lockedBmp.SetPixel(i, j, Color.FromArgb(gray, gray, gray));
}
}
lockedBmp.UnlockBits(); // remember to release resources
return lockedBmp.Bitmap; // return the bitmap (you don't need to clone it again, that's already been done).
}
This wrapper class has saved me a ridiculous amount of time in bitmap processing. Once you've implemented this in all your methods, preferably only calling LockBits() once, then I'm sure your application's performance will improve tremendously.
I also see that you're cloning the images a lot. This probably doesn't take up as much time as the SetPixel()/GetPixel() thing, but its time can still be significant especially with larger images.

The easiest way would be to redraw the image onto itself using DrawImage and passing a suitable ColorMatrix. Google for ColorMatrix and gray scale and you'll find a ton of examples, this one for example: http://www.codeproject.com/Articles/3772/ColorMatrix-Basics-Simple-Image-Color-Adjustment