Kinect V2 Color Stream Byte Order - c#

I'm working on an application which will stream the color, depth, and IR video data from the Kinect V2 sensor. Right now I'm just putting together the color video part of the app. I've read through some tutorials and actually got some video data coming into my app - the problem seems to be that the byte order seems to be in the wrong order which gives me an oddly discolored image (see below).
So, let me explain how I got here. In my code, I first open the sensor and also instantiate a new multi source frame reader. After I've created the reader, I create an event handler called Reader_MultiSourceFrameArrived:
void Reader_MultiSourceFrameArrived(object sender, MultiSourceFrameArrivedEventArgs e)
{
if (proccessing || gotframe) return;
// Get a reference to the multi-frame
var reference = e.FrameReference.AcquireFrame();
// Open color frame
using (ColorFrame frame = reference.ColorFrameReference.AcquireFrame())
{
if (frame != null)
{
proccessing = true;
var description = frame.ColorFrameSource.FrameDescription;
bw2 = description.Width / 2;
bh2 = description.Height / 2;
bpp = (int)description.BytesPerPixel;
if (imgBuffer == null)
{
imgBuffer = new byte[description.BytesPerPixel * description.Width * description.Height];
}
frame.CopyRawFrameDataToArray(imgBuffer);
gotframe = true;
proccessing = false;
}
}
}
Now, every time a frame is received (and not processing) it should copy the frame data into an array called imgBuffer. When my application is ready I then call this routine to convert the array into a Windows Bitmap that I can display on my screen.
if (gotframe)
{
if (theBitmap.Rx != bw2 || theBitmap.Ry != bh2) theBitmap.SetSize(bw2, bh2);
int kk = 0;
for (int j = 0; j < bh2; ++j)
{
for (int i = 0; i < bw2; ++i)
{
kk = (j * bw2 * 2 + i) * 2 * bpp;
theBitmap.pixels[i, bh2 - j - 1].B = imgBuffer[kk];
theBitmap.pixels[i, bh2 - j - 1].G = imgBuffer[kk + 1];
theBitmap.pixels[i, bh2 - j - 1].R = imgBuffer[kk + 2];
theBitmap.pixels[i, bh2 - j - 1].A = 255;
}
}
theBitmap.needupdate = true;
gotframe = false;
}
}
So, after this runs theBitmap now contains the image information needed to draw the image on the screen... however, as seen in the image above - it looks quite strange. The most obvious change is to simply change the order of the pixel B,G,R values when they get assigned to the bitmap in the double for loop (which I tried)... however, this simply results in other strange color images and none provide an accurate color image. Any thoughts where I might be going wrong?

Is this Bgra?
The normal "RGB" in Kinect v2 for C# is BGRA.
Using the Kinect SDK 2.0, you don't need all of those "for" cycles.
The function used to allocate the pixels in the bitmap is this one:
colorFrame.CopyConvertedFrameDataToIntPtr(
this.colorBitmap.BackBuffer,
(uint)(colorFrameDescription.Width * colorFrameDescription.Height * 4),
ColorImageFormat.Bgra);
1) Get the Frame From the kinect, using Reader_ColorFrameArrived (go see ColorSamples - WPF);
2) Create the colorFrameDescription from the ColorFrameSource using Bgra format;
3) Create the bitmap to display;
If you have any problems, please say. But if you follow the sample it's actually pretty clean there how to do it.

I was stuck on this problem forever. Problem is, that all almost all examples you find, are WPF examples. But for Windows Forms its a different story.
frame.CopyRawFrameDataToArray(imgBuffer);
gets you the rawdata whitch is
ColorImageFormat.Yuy2
By converting it to RGB you should be able to fix your color problem. The transformation from YUY2 to RGB is very expensive, you might want to use a Parallel foreach loop to maintain your framerate

Related

Working on each frame from a Vimba camera

I am working with an Allied Vision Vimba camera to monitor an experiment involving a laser.
I have written a code that allows me to take one frame from the camera live imaging (displayed in pictureBoxLiveCamera, in black and white), put it in another PictureBox (pictureBoxFixe1) and work on it. The code works fine for the most part, even though it is not finished.
My problem is that, eventually, I will need to work continuously, meaning on every frame from the camera (about 15-20 per second), and I am starting to feel like this is going to be complicated.
I am not asking you to help me through the whole process, but I have a precise exemple of something that is not working and I would like to understand. I am working with the Vimba .NET API, so some of the functions called might not be familiar to most of you (and they are not much more to me, really), but I'll try my best to explain them in the code (the API manual is kinda cryptic).
My code has a part that, when I isolate one frame from the camera to work on it (via a button-clickevent, draws an histogram that graphs the number of pixels on the vertical axis and digital number value on the horizontal axis. This way I know if some of the pixels are saturated (at 255), and I can lower the exposure time. This works fine, here is the code for the button that isolate one frame and draws the histogram :
private void fixButton_Click(object sender, EventArgs e)
{
if (m_Acquiring == true) //mm_Acquiring being true indicates that the camera is sending images continuously, and that they are toggled in pictureBoxLiveCamera
{
pictureBoxFixe1.Image = SaveBitmap; //SaveBitmap is the bitmap isolated from the live camera in an earlier code portion
Array.Clear(PixelColorCount, 0, 256);
foreach (var Series in chartHist.Series)
{
Series.Points.Clear();
}
//Creating a bitmap from pictureBoxFixe1 to draw the histogram from
Bitmap Couleur = new Bitmap(pictureBoxFixe1.Image);
//Collecting data from pictureBoxFixe1
for (int i = 0; i < 1023; i++)
{
for (int j = 157; j < 288; j++)
{
PixelColorCount[Couleur.GetPixel(i + 1, j).B] += 1;
}
for (int j = 484; j < 615; j++)
{
PixelColorCount[Couleur.GetPixel(i + 1, j).B] += 1;
}
}
//Plotting the pixel counter, to detect saturation
for (int i = 0; i < 256; i++)
{
chartHist.Series["Pixel count"].Points.AddXY(i, PixelColorCount[i]);
}
//If there are saturated pixels : toggle a title on chart 1 to warn the user
if (PixelColorCount[255] > 1)
{
chartHist.Titles["Title1"].Visible = false;
chartHist.Titles["Title2"].Visible = true;
}
else
{
chartHist.Titles["Title1"].Visible = true;
chartHist.Titles["Title2"].Visible = false;
}
}
else
{
MessageBox.Show("Acquisition is not running.", "Something went wrong !", MessageBoxButtons.OK, MessageBoxIcon.Information);
}
}
Now what I want is to make this work live, with every frame incoming from the camera. But it doesn't work and I can't find why. Here is what I tried :
private void BtAcquisitionLiveCam_Click(object sender, EventArgs e)
{
//Here is the part of the code that starts the live acquisition when I click the button, I don't copy it completely because I don't think this is relevant.
this.m_Acquiring = true;
mycamera.OnFrameReceived += new Camera.OnFrameReceivedHandler(this.OnFrameReceived); //This is the event handler for the reception of a frame for the camera, it is part of the Vimba API
mycamera.StartContinuousImageAcquisition(1); //According to the manual, this "Starts streaming and allocates the needed frames", with the argument being "count of Frame(s) which should be used for this method"
}
private void OnFrameReceived(Frame frame) //Frame is the class in which the code puts the data from the camera, frame is the last pack of data received from the camera
{
Bitmap myBitmap = null;
if (true == m_Acquiring)
{
try
{
mycamera.QueueFrame(frame);
}
catch (Exception)
{
MessageBox.Show("Frame queuing failed.", "Something went wrong !", MessageBoxButtons.OK, MessageBoxIcon.Warning);
}
}
else
{
MessageBox.Show("Acquisition is not running.", "Something went wrong !", MessageBoxButtons.OK, MessageBoxIcon.Warning);
}
featureExposureTime.FloatValue = (double)hScrollBarLiveExposureTime.Value;
frame.Fill(ref myBitmap);
pictureBoxLiveCamera.Image = myBitmap;
SaveBitmap = myBitmap; //Up until here the code works perfectly, because I can use SaveBitmap without problem later in the code (not shown here)
Array.Clear(PixelColorCount, 0, 256);
foreach (var Series in chartHist.Series)
{
Series.Points.Clear();
}
//Collecting data from pictureBoxFixe1
for (int i = 1; i < 1023; i++)
{
for (int j = 1; j < 767; j++)
{
PixelColorCount[SaveBitmap.GetPixel(i, j).B] += 1;
}
}
//Plotting the pixel counter, to detect saturation
for (int i = 0; i < 256; i++)
{
chartHist.Series["Pixel count"].Points.AddXY(i, PixelColorCount[i]);
}
//If there are saturated pixels : toggle a title on chart 1 to warn the user
if (PixelColorCount[255] > 1)
{
chartInit.Titles["Title1"].Visible = true;
}
else { }
}
Sorry for the amount of code, but I thought I couldn't do otherwise. My problem is that the histogram doesn't show at all (well it is set as Visible, I see the title, but nothing gets plotted in it). Does anyone have any idea ?
The thing that is driving me crazy is that the VimbaViewer program that's provided with the camera does exactly what I want in a flawless way, but I don't have access to it's source code...
Thanks !
If I follow correctly, your OnFrameReceived event is firing correctly, but the WinForm application UI is not updating after each frame is processed.
It appears that "chartInit" is being referenced in your OnFrameReceived instead of "chartHist", is that supposed to be the case?
Whatever your chart control is, try "chart.Refresh()" or "chart.Update()". It might not be updating the UI thread from the event loop.
Another thing you'll need to think about is how fast these images are coming in. Will it draw and redraw the chart 15-20 times per second? Is that reasonable? Might need some sort of queue/averaging mechanism to only update the chart every x seconds, or perhaps if some threshold value is exceeded with respect to change in the images. Just a thought.

How to create a Bitmap and draw it in UWP?

I have a quite simple question today: how to create a bitmap and "draw" it, changing every single pixel in it, in a UWP app.
I have read many things here on StackOverflow, but now I am a little bit confused, because there are so many different types (WritableBitmap, SoftwareBitmap, BitmapImage, BitmapSource... now, in FCU, they added even BitmapIconSource...) and so many ways... but they mostly starts with a given image file or source, and it's not my case.
Let's say, e.g., that I want to create a 20x20 Bitmap and want to assign to every pixel a different argb value... and then assign it to a BitmapSource property.
What would be the best and efficient way, in a UWP?
Thank you for your patience and your attention.
Best regards
You can use WriteableBitmap and modify it's PixelBuffer directly:
var wb = new WriteableBitmap(100, 100);
byte[] imageArray = new byte[100 * 100 * 4];
for (int i = 0; i < imageArray.Length; i += 4)
{
//BGRA format
imageArray[i] = 0; // Blue
imageArray[i + 1] = 0; // Green
imageArray[i + 2] = 255; // Red
imageArray[i + 3] = 255; // Alpha
}
using (Stream stream = wb.PixelBuffer.AsStream())
{
//write to bitmap
await stream.WriteAsync(imageArray, 0, imageArray.Length);
}
TargetImage.Source = wb;
If you want more abstraction, look into WriteableBitmapEx which adds very useful and easy to use extension methods and helpers that make working with WriteableBitmap a breeze.

Display an image in real time as it is being created in a loop

In my code, the output is an image each pixel of which is determined using nested loops.
1) How can I force a window to open and show the output image as it is being constructed in the loop? (The window shows up when everything is finished. I don't want this.)
2) How can I have the output be displayed line by line (or even pixel by pixel) as the loop goes on. User must have the sense of getting the output in real-time.
outImage = new Image<Hsv, Byte>(numberOfColumns, numberOfRows);
byte[,,] pixelValue = outImage.Data;
for (int i = 0; i < numberOfRows - 1; i++)
{
for (int j = 0; j < numberOfColumns - 1; j++)
{
//pixelValue[i, j, k] is determined here using some other functions
imageBox1.Image = outImage; //too slow and impossible
}
}
You can display an image pixel by pixel in real time by putting it on a separate thread and using GetPixel and SetPixel. Keep in mind though that these methods are slow and if you are displaying high resolution pictures, it will take a while.
What you'll want to do is create a form with a picture box control on it. Next, create a Bitmap object containing the image you'll want to display. You can do this using a line like this:
Bitmap _bmp = new Bitmap(Image.FromFile(#"C:\MyImage.jpg"));
Next, in your form's shown event, spin off a new task to do the work, so the GUI doesn't lock up:
private void Form1_Shown(object sender, EventArgs e)
{
Task.Factory.StartNew(ShowImage);
}
This line will spin off a new thread every time the form is displayed. The thread will fork off and call ShowImage(), which should look like this:
private void ShowImage()
{
Graphics g = pbImage.CreateGraphics();
for (int x = 0; x < _bmp.Width; x++)
{
for (int y = 0; y < _bmp.Height; y++)
{
Color c = _bmp.GetPixel(x, y);
if (pbImage.InvokeRequired)
{
var x1 = x;
var y1 = y;
pbImage.BeginInvoke((Action) (() =>
{
g.FillRectangle(new SolidBrush(c), x1, y1, 1, 1);
}));
}
else
{
g.FillRectangle(new SolidBrush(c), x, y, 1, 1);
}
System.Threading.Thread.Sleep(1);
}
}
}
If you wanted to speed this up a bit, you could spin up two or more tasks, each task working in parallel (e.g. one thread starts at the beginning, another at the end, another in the middle maybe, etc). Just make sure your threads don't "overlap".
Another way to speed this up is to use pointers instead of GetPixel() and SetPixel(). You'd have to mark your code as unsafe though.
put your code in background Worker => Do Work
A separate thread would be initiated
I am not a WinForms expert I am more of a WPF type. But I have an application running a solid 30fps and that is faster than humans can detect. I really do not quite understand what you want to do here. You have to blit each pixel individually but have display in real time? An ImageBox derives from the Windows Forms PictureBox, that won't work I do not think.
You could try moving to WPF, and use a WriteableBitmap for a ImageSource for an Image object or the background of a Canvas Object. A WriteableBitmap will let you access each pixel, but the refresh rate is controlled by WPF and the monitor refresh rate is controlled by the AC current frequency.
Doug

Access preview frame from MediaCapture

I would like to grab the preview frames that are displayed inside my CaptureElement xaml element. The source of my CaptureElement is set to a MediaCapture object and I use the StartPreview() method to start displaying the camera. I would like to access the frames that are being shown without saving them to an img or video file. The goal is to capture 10 fps from the preview and send each frame to another class that accepts byte[].
I tried using the CapturePhotoToStorageFileAsync method however this is not a feasible option as I do not want to take 10 actual images / second. I also do not want to use ScreenCapture as it stores what is captured into a video file. Ideally I do not want to store any media files temporarily on the phone. After looking at the msdn for MediaCapture, I noticed there's a method called GetPreviewFrameAsync() however this method does not exist inside Windows Phone 8.1. I also stumbled on this example however I do not completely understand how it works.
Any suggestions on how to approach this is greatly appreciated.
There is a sample on the Microsoft github page that is relevant, although they target Windows 10. You may be interested in migrating your project to get this functionality.
GetPreviewFrame: This sample will capture preview frames as opposed to full-blown photos. Once it has a preview frame, it can edit the pixels on it.
Here is the relevant part:
private async Task GetPreviewFrameAsSoftwareBitmapAsync()
{
// Get information about the preview
var previewProperties = _mediaCapture.VideoDeviceController.GetMediaStreamProperties(MediaStreamType.VideoPreview) as VideoEncodingProperties;
// Create the video frame to request a SoftwareBitmap preview frame
var videoFrame = new VideoFrame(BitmapPixelFormat.Bgra8, (int)previewProperties.Width, (int)previewProperties.Height);
// Capture the preview frame
using (var currentFrame = await _mediaCapture.GetPreviewFrameAsync(videoFrame))
{
// Collect the resulting frame
SoftwareBitmap previewFrame = currentFrame.SoftwareBitmap;
// Add a simple green filter effect to the SoftwareBitmap
EditPixels(previewFrame);
}
}
private unsafe void EditPixels(SoftwareBitmap bitmap)
{
// Effect is hard-coded to operate on BGRA8 format only
if (bitmap.BitmapPixelFormat == BitmapPixelFormat.Bgra8)
{
// In BGRA8 format, each pixel is defined by 4 bytes
const int BYTES_PER_PIXEL = 4;
using (var buffer = bitmap.LockBuffer(BitmapBufferAccessMode.ReadWrite))
using (var reference = buffer.CreateReference())
{
// Get a pointer to the pixel buffer
byte* data;
uint capacity;
((IMemoryBufferByteAccess)reference).GetBuffer(out data, out capacity);
// Get information about the BitmapBuffer
var desc = buffer.GetPlaneDescription(0);
// Iterate over all pixels
for (uint row = 0; row < desc.Height; row++)
{
for (uint col = 0; col < desc.Width; col++)
{
// Index of the current pixel in the buffer (defined by the next 4 bytes, BGRA8)
var currPixel = desc.StartIndex + desc.Stride * row + BYTES_PER_PIXEL * col;
// Read the current pixel information into b,g,r channels (leave out alpha channel)
var b = data[currPixel + 0]; // Blue
var g = data[currPixel + 1]; // Green
var r = data[currPixel + 2]; // Red
// Boost the green channel, leave the other two untouched
data[currPixel + 0] = b;
data[currPixel + 1] = (byte)Math.Min(g + 80, 255);
data[currPixel + 2] = r;
}
}
}
}
}
And declare this outside your class:
[ComImport]
[Guid("5b0d3235-4dba-4d44-865e-8f1d0e4fd04d")]
[InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
unsafe interface IMemoryBufferByteAccess
{
void GetBuffer(out byte* buffer, out uint capacity);
}
And of course, your project will have to allow unsafe code for all of this to work.
Have a closer look at the sample to see how to get all the details. Or, to have a walkthrough, you can watch the camera session from the recent //build/ conference, which includes a little bit of a walkthrough through some camera samples.

Does XNA have a BGRA/BGRX surface format?

I have a video stream coming in from the Kinect. It's data is packed in a 32bit BGRX format. I would like to move this data directly into a Texture2d but I cant find a matching SurfaceFormat. The closest thing I can find is SurfaceFormat.Color which looks to be 32bit RGBX.
Assuming that there is not a compatible format. What is the quickest way to convert and push the data to a texture 2d
I was thinking something like this would be good, but it seems to slow down the framerate:
Edit: I changed the algorithm a bit and it seems to be running decently now.
void nui_VideoFrameReady(object sender, ImageFrameReadyEventArgs e)
{
byte x;
for (int i = 0; i < e.ImageFrame.Image.Bits.Length; i += 4)
{
x = e.ImageFrame.Image.Bits[i];
e.ImageFrame.Image.Bits[i] = e.ImageFrame.Image.Bits[i + 2];
e.ImageFrame.Image.Bits[i + 2] = x;
}
canvas.SetData(e.ImageFrame.Image.Bits);
}
This is the fastest thing I have been able to come up with:
void nui_VideoFrameReady(object sender, ImageFrameReadyEventArgs e)
{
byte[] bits = e.ImageFrame.Image.Bits;
for (int i = 0; i < bits.Length; i += 4)
{
frameBuffer[i] = bits[i + 2];
frameBuffer[i + 1] = bits[i + 1];
frameBuffer[i + 2] = bits[i];
}
canvas.SetData(frameBuffer);
}
Without the kinect involved at all I get a framerate of 4200
Moving the bytes directly to the texture2d without correcting the format difference yield: 3100
If you give your self a local variable with e.ImageFrame.Image.Bits and copy the bytes into a preallocated buffer fixing the bytes as you go I get: 2700
The algorithm i have listed in my original question where you swap the bits in place and then send to texture2d yield about 750
So to recap changing to this algorithm bumped the framerate to 2700 up from 750

Categories

Resources