I have c# TensorFlow.NET working in Unity. But it using an image from the file system. I want to be able to use an image from memory (Texture2D).
I tried to follow some examples of people using TensorFlowSharp. But that didn't work.
What am I doing wrong?
Note: with both functions, I am using the same image. The image is 512x512. but the result of both pictures is different.
// Doesn't work
private NDArray FromTextureToNDArray(Texture2D texture) {
Color32[] pixels = texture.GetPixels32();
byte[] floatValues = new byte[(texture.width * texture.height) * 3];
for (int i = 0; i < pixels.Length; i++) {
var color = pixels[i];
floatValues[i * 3] = color.r;
floatValues[i * 3 + 1] = color.g;
floatValues[i * 3 + 2] = color.b;
}
Shape shape = new Shape(1, texture.width, texture.height, 3);
NDArray image = new NDArray(floatValues, shape);
return image;
}
// Works
private NDArray ReadFromFile(string fileName) {
var graph = new Graph().as_default();
// Change image
var file_reader = tf.read_file(fileName, "file_reader");
var decodeJpeg = tf.image.decode_jpeg(file_reader, channels: 3, name: "DecodeJpeg");
var casted = tf.cast(decodeJpeg, TF_DataType.TF_UINT8);
var dims_expander = tf.expand_dims(casted, 0);
using (var sess = tf.Session(graph)) {
return sess.run(dims_expander);
}
}
I ended up using this code from Shaqian: https://github.com/shaqian/TF-Unity/blob/master/TensorFlow/Utils.cs
Add this script to your project and then you could use it like this:
// Get image
byte[] imageData = Utils.DecodeTexture(texture, texture.width, texture.height, 0, Flip.VERTICAL);
Shape shape = new Shape(1, texture.width, texture.height, 3);
NDArray image = new NDArray(imageData, shape);
Use Barracuda as step in between.
var encoder = new Unity.Barracuda.TextureAsTensorData(your_2d_texture);
Related
I am trying to create movie from images.
I am following following links :
https://www.leadtools.com/support/forum/posts/t11084- // Here I am trying option 2 mentioned & https://social.msdn.microsoft.com/Forums/sqlserver/en-US/b61726a4-4b87-49c7-b4fc-8949cd1366ac/visual-c-visual-studio-2017-how-do-you-convert-jpg-images-to-video-in-visual-c?forum=csharpgeneral
void convert()
{
bmp = new Bitmap(320, 240, System.Drawing.Imaging.PixelFormat.Format24bppRgb);
// create sample source object
SampleSource smpsrc = new SampleSource();
ConvertCtrl convertCtrl = new ConvertCtrl();
// create a new media type wrapper
MediaType mt = new MediaType();
double AvgTimePerFrame = (10000000 / 15);
// set the type to 24-bit RGB video
mt.Type = Constants.MEDIATYPE_Video;
mt.SubType = Constants.MEDIASUBTYPE_RGB24;
// set the format
mt.FormatType = Constants.FORMAT_VideoInfo;
VideoInfoHeader vih = new VideoInfoHeader();
int bmpSize = GetBitmapSize(bmp);
// setup the video info header
vih.bmiHeader.biCompression = 0; // BI_RGB
vih.bmiHeader.biBitCount = 24;
vih.bmiHeader.biWidth = bmp.Width;
vih.bmiHeader.biHeight = bmp.Height;
vih.bmiHeader.biPlanes = 1;
vih.bmiHeader.biSizeImage = bmpSize;
vih.bmiHeader.biClrImportant = 0;
vih.AvgTimePerFrame.lowpart = (int)AvgTimePerFrame;
vih.dwBitRate = bmpSize * 8 * 15;
mt.SetVideoFormatData(vih, null, 0);
// set fixed size samples matching the bitmap size
mt.SampleSize = bmpSize;
mt.FixedSizeSamples = true;
// assign the source media type
smpsrc.SetMediaType(mt);
// select the LEAD compressor
convertCtrl.VideoCompressors.MCmpMJpeg.Selected = true;
convertCtrl.SourceObject = smpsrc;
convertCtrl.TargetFile = #"D:\Projects\LEADTool_Movie_fromImage\ImageToVideo_LeadTool\ImageToVideo_LeadTool\Images\Out\aa.avi";
//convertCtrl.TargetFile = "C:\\Users\\vipul.langalia\\Documents\\count.avi";
convertCtrl.TargetFormat = TargetFormatType.WMVMux;
convertCtrl.StartConvert();
BitmapData bmpData;
int i = 1;
byte[] a = new byte[bmpSize];
System.Drawing.Rectangle rect = new System.Drawing.Rectangle(0, 0, bmp.Width, bmp.Height);
var imgs = GetAllFiles();
foreach (var item in imgs)
{
bmpSize = GetBitmapSize(item);
MediaSample ms = smpsrc.GetSampleBuffer(30000);
ms.SyncPoint = true;
bmpData = item.LockBits(rect, ImageLockMode.ReadWrite, item.PixelFormat);
Marshal.Copy(bmpData.Scan0, a, 0, bmpSize);
item.UnlockBits(bmpData);
ms.SetData(bmpSize, a);
SetSampleTime(ms, i, AvgTimePerFrame);
smpsrc.DeliverSample(1000, ms);
i++;
}
smpsrc.DeliverEndOfStream(1000);
}
byte[] GetByteArrayFroMWritableBitmap(WriteableBitmap bitmapSource)
{
var width = bitmapSource.PixelWidth;
var height = bitmapSource.PixelHeight;
var stride = width * ((bitmapSource.Format.BitsPerPixel + 7) / 8);
var bitmapData = new byte[height * stride];
bitmapSource.CopyPixels(bitmapData, stride, 0);
return bitmapData;
}
private int GetBitmapSize(WriteableBitmap bmp)
{
int BytesPerLine = (((int)bmp.Width * 24 + 31) & ~31) / 8;
return BytesPerLine * (int)bmp.Height;
}
private int GetBitmapSize(Bitmap bmp)
{
int BytesPerLine = ((bmp.Width * 24 + 31) & ~31) / 8;
return BytesPerLine * bmp.Height;
}
It is throwing out of memory exception when execute ms.SetData(bmpSize, a); statement. Plus If I directly pass byte[] by var a = System.IO.File.ReadAllBytes(imagePath); in ms.SetData(bmpSize, a); statement then it will not throw error but video file is not properly created.
Can anybody please help me?
There are a couple of problems with your code:
Are all your images 320x240 pixels? If not, you should resize them to these exact dimensions before delivering them as video samples to the Convert control.
If you want to use a different size, you can, but it should be the same size for all images, and you should modify the code accordingly.
You are setting the TargetFormat property to WMVMux, but the name of the output file has “.avi” extension. If you want to save AVI files, set TargetFormat = TargetFormatType.AVI.
If you still face problems after this, feel free to contact support#leadtools.com and provide full details about what you tried and what errors you’re getting. Email support is free for LEADTOOLS SDK owners and also for free evaluation users.
I'm working on pedestrians detection algorithm and got stuck on using ConnectedComponentsWithStats() method. I can't get to the values of stats. Here is what I wrote so far:
var labels = new Mat();
var stats = new Mat();
var centroids = new Mat();
var nLabels = CvInvoke.ConnectedComponentsWithStats(greyImage, labels, stats, centroids);
var centroidPoints = new MCvPoint2D64f[nLabels];
centroids.CopyTo(centroidPoints);
foreach (MCvPoint2D64f point in centroidPoints)
{
var x = point.X;
var y = point.Y;
}
The stats are stored as int32's in opencv c++ and have 5 fields x,y,width,height and area.
You can see the opencv implementation of ConnectedComponentsWithStats here.
Also you can see the struct CCStatsOp opencv returns the stats as here.
Sadly EMGUCV has not created the data structure CCStatsOp so you will need to just store the raw integer data like such.
int[] statsData = new int[stats.Rows * stats.Cols];
stats.CopyTo(statsData);
for (int i = 0; i < stats.Rows; i++)
{
var x = statsData[i * stats.Cols + 0];
var y = statsData[i * stats.Cols + 1];
var width = statsData[i * stats.Cols + 2];
var height = statsData[i * stats.Cols + 3];
var area = statsData[i * stats.Cols + 4];
}
You can also roll your own CCStats op with this code here.
Struct
public struct CCStatsOp
{
public Rectangle Rectangle;
public int Area;
}
Code
var labels = new Mat();
var stats = new Mat();
var centroids = new Mat();
var nLabels = CvInvoke.ConnectedComponentsWithStats(greyImage, labels, stats, centroids);
var centroidPoints = new MCvPoint2D64f[nLabels];
centroids.CopyTo(centroidPoints);
CCStatsOp[] statsOp = new CCStatsOp[stats.Rows];
stats.CopyTo(statsOp);
foreach (var statop in statsOp)
{
Console.WriteLine($"Rectangle: {statop.Rectangle} Area: {statop.Area}");
}
I am following this tutorial on image hashing.
So far I have achieved the following:
Code:
private async Task<ImageSource> ProcessImageAsync(StorageFile ImageFile)
{
if (ImageFile == null)
throw new ArgumentNullException("ImageFile cannot be null.");
//The new size of processed image.
const int side = 300; //300 is for clarity. Should be 8 or 16 px.
//Initialize bitmap transformations to be applied to the image.
var transform = new BitmapTransform() { ScaledWidth = side, ScaledHeight = side, InterpolationMode = BitmapInterpolationMode.Cubic };
//Get image pixels.
var stream = await ImageFile.OpenStreamForReadAsync();
var decoder = await BitmapDecoder.CreateAsync(stream.AsRandomAccessStream());
var pixelData = await decoder.GetPixelDataAsync(BitmapPixelFormat.Bgra8, BitmapAlphaMode.Premultiplied, transform, ExifOrientationMode.RespectExifOrientation, ColorManagementMode.ColorManageToSRgb);
var pixels = pixelData.DetachPixelData();
//Initialize writable bitmap.
var wBitmap = new WriteableBitmap((int)decoder.PixelWidth, (int)decoder.PixelHeight);
await wBitmap.SetSourceAsync(stream.AsRandomAccessStream());
//Make it gray
var grayBitmapBuffer = await ConvertToGrayAsync(wBitmap);
//Create a software bitmap from the writable bitmap's pixel buffer.
var sBitmap = SoftwareBitmap.CreateCopyFromBuffer(wBitmap.PixelBuffer, BitmapPixelFormat.Bgra8, side, side, BitmapAlphaMode.Premultiplied);
//Create software bitmap source.
var sBitmapSource = new SoftwareBitmapSource();
await sBitmapSource.SetBitmapAsync(sBitmap);
return sBitmapSource;
}
private async Task<IBuffer> ConvertToGrayAsync(WriteableBitmap srcBitmap)
{
// Get the source bitmap pixels
byte[] srcPixels = new byte[4 * srcBitmap.PixelWidth * srcBitmap.PixelHeight];
using (Stream pixelStream = srcBitmap.PixelBuffer.AsStream())
{
await pixelStream.ReadAsync(srcPixels, 0, srcPixels.Length);
}
// Create a destination bitmap and pixels array
WriteableBitmap dstBitmap = new WriteableBitmap(srcBitmap.PixelWidth, srcBitmap.PixelHeight);
byte[] dstPixels = new byte[4 * dstBitmap.PixelWidth * dstBitmap.PixelHeight];
for (int i = 0; i < srcPixels.Length; i += 4)
{
double b = (double)srcPixels[i] / 255.0;
double g = (double)srcPixels[i + 1] / 255.0;
double r = (double)srcPixels[i + 2] / 255.0;
byte a = srcPixels[i + 3];
double e = (0.21 * r + 0.71 * g + 0.07 * b) * 255;
byte f = Convert.ToByte(e);
dstPixels[i] = f;
dstPixels[i + 1] = f;
dstPixels[i + 2] = f;
dstPixels[i + 3] = a;
}
// Move the pixels into the destination bitmap
using (Stream pixelStream = dstBitmap.PixelBuffer.AsStream())
{
await pixelStream.WriteAsync(dstPixels, 0, dstPixels.Length);
}
dstBitmap.Invalidate();
// Display the new bitmap
return dstBitmap.PixelBuffer;
}
This is the original image which user selects:
And this is the output produced by my code:
It is getting scaled and grayed all-right. But I don't understand why is the image repeating horizontally??
Here is the XAML code that renders the image:
<Image x:Name="myImage" Stretch="None" />
And the C# code that sets the image source:
StorageFile userPickedFile = ...; //code ignored.
myImage.Source = await ProcessImageAsync(userPickedFile);
What am I doing wrong? Something looks fishy about the processed image.. I am lost from here... How should I move ahead with Hashing? Any help?
Your ConvertToGrayAsync method should simply return the converted WriteableBitmap. The pixel conversion code inside the loop could also be simplified, and the method does not need to operate on a source and destination buffer. It could as well manipulate the pixel values in place.
private async Task<WriteableBitmap> ConvertToGrayAsync(WriteableBitmap srcBitmap)
{
var pixels = srcBitmap.PixelBuffer.ToArray();
for (int i = 0; i < pixels.Length; i += 4)
{
var b = pixels[i];
var g = pixels[i + 1];
var r = pixels[i + 2];
var f = (byte)(0.21 * r + 0.71 * g + 0.07 * b);
pixels[i] = f;
pixels[i + 1] = f;
pixels[i + 2] = f;
}
var dstBitmap = new WriteableBitmap(srcBitmap.PixelWidth, srcBitmap.PixelHeight);
using (var pixelStream = dstBitmap.PixelBuffer.AsStream())
{
await pixelStream.WriteAsync(pixels, 0, pixels.Length);
}
return dstBitmap;
}
The following method loads a WriteableBitmap with a predefined size:
private async Task<WriteableBitmap> LoadWriteableBitmapAsync(
StorageFile file, int width, int height)
{
using (var fileStream = await file.OpenReadAsync())
using (var memoryStream = new InMemoryRandomAccessStream())
{
var decoder = await BitmapDecoder.CreateAsync(fileStream);
var transform = new BitmapTransform
{
ScaledWidth = (uint)width,
ScaledHeight = (uint)height,
InterpolationMode = BitmapInterpolationMode.Cubic
};
var pixelData = await decoder.GetPixelDataAsync(
BitmapPixelFormat.Bgra8, BitmapAlphaMode.Straight, transform,
ExifOrientationMode.RespectExifOrientation,
ColorManagementMode.ColorManageToSRgb);
var pixels = pixelData.DetachPixelData();
var encoder = await BitmapEncoder.CreateAsync(
BitmapEncoder.PngEncoderId, memoryStream);
encoder.SetPixelData(
BitmapPixelFormat.Bgra8, BitmapAlphaMode.Straight,
(uint)width, (uint)height, 96, 96, pixels);
await encoder.FlushAsync();
memoryStream.Seek(0);
var bitmap = new WriteableBitmap(width, height);
await bitmap.SetSourceAsync(memoryStream);
return bitmap;
}
}
Your ProcessImageAsync method would now look like this:
private async Task<WriteableBitmap> ProcessImageAsync(
StorageFile file, int width, int height)
{
return await ConvertToGrayAsync(
await LoadWriteableBitmapAsync(file, width, height));
}
I followed this solution for my project : How to create bitmap from Surface (SharpDX)
I don't have enough reputation to comment so I'm opening a new question here.
My project is basically in Direct 2D, I have a Surface buffer, a swapchain. I want to put my buffer into a datastream and reads it's value to put it into a bitmap and save it on disk ( like a screen capture), but my code won't work since all the bytes values are 0 (which is black) and this doesn't make sense since my image is fully white with a bit of blue.
Here is my code :
SwapChainDescription description = new SwapChainDescription()
{
ModeDescription = new ModeDescription(this.Width, this.Height, new Rational(60, 1), Format.B8G8R8A8_UNorm),
SampleDescription = new SampleDescription(1, 0),
Usage = Usage.RenderTargetOutput,
BufferCount = 1,
SwapEffect = SwapEffect.Discard,
IsWindowed = true,
OutputHandle = this.Handle
};
Device.CreateWithSwapChain(DriverType.Hardware, DeviceCreationFlags.Debug | DeviceCreationFlags.BgraSupport, description, out device, out swapChain);
SharpDX.DXGI.Device dxgiDevice = device.QueryInterface<SharpDX.DXGI.Device>();
SharpDX.DXGI.Adapter dxgiAdapter = dxgiDevice.Adapter;
SharpDX.Direct2D1.Device d2dDevice = new SharpDX.Direct2D1.Device(dxgiDevice);
d2dContext = new SharpDX.Direct2D1.DeviceContext(d2dDevice, SharpDX.Direct2D1.DeviceContextOptions.None);
SharpDX.Direct3D11.DeviceContext d3DeviceContext = new SharpDX.Direct3D11.DeviceContext(device);
properties = new BitmapProperties(new SharpDX.Direct2D1.PixelFormat(SharpDX.DXGI.Format.B8G8R8A8_UNorm, SharpDX.Direct2D1.AlphaMode.Premultiplied),
96, 96);
Surface backBuffer = swapChain.GetBackBuffer<Surface>(0);
d2dTarget = new SharpDX.Direct2D1.Bitmap(d2dContext, backBuffer, properties);
d2dContext.Target = d2dTarget;
playerBitmap = this.LoadBitmapFromContentFile(#"C:\Users\ndesjardins\Desktop\wave.png");
//System.Drawing.Bitmap bitmapCanva = new System.Drawing.Bitmap(1254, 735);
d2dContext.BeginDraw();
d2dContext.Clear(SharpDX.Color.White);
d2dContext.DrawBitmap(playerBitmap, new SharpDX.RectangleF(0, 0, playerBitmap.Size.Width, playerBitmap.Size.Height), 1f, SharpDX.Direct2D1.BitmapInterpolationMode.NearestNeighbor);
SharpDX.Direct2D1.SolidColorBrush brush = new SharpDX.Direct2D1.SolidColorBrush(d2dContext, SharpDX.Color.Green);
d2dContext.DrawRectangle(new SharpDX.RectangleF(200, 200, 100, 100), brush);
d2dContext.EndDraw();
swapChain.Present(1, PresentFlags.None);
Texture2D backBuffer3D = backBuffer.QueryInterface<SharpDX.Direct3D11.Texture2D>();
Texture2DDescription desc = backBuffer3D.Description;
desc.CpuAccessFlags = CpuAccessFlags.Read;
desc.Usage = ResourceUsage.Staging;
desc.OptionFlags = ResourceOptionFlags.None;
desc.BindFlags = BindFlags.None;
var texture = new Texture2D(device, desc);
d3DeviceContext.CopyResource(backBuffer3D, texture);
byte[] data = null;
using (Surface surface = texture.QueryInterface<Surface>())
{
DataStream dataStream;
var map = surface.Map(SharpDX.DXGI.MapFlags.Read, out dataStream);
int lines = (int)(dataStream.Length / map.Pitch);
data = new byte[surface.Description.Width * surface.Description.Height * 4];
dataStream.Position = 0;
int dataCounter = 0;
// width of the surface - 4 bytes per pixel.
int actualWidth = surface.Description.Width * 4;
for (int y = 0; y < lines; y++)
{
for (int x = 0; x < map.Pitch; x++)
{
if (x < actualWidth)
{
data[dataCounter++] = dataStream.Read<byte>();
}
else
{
dataStream.Read<byte>();
}
}
}
dataStream.Dispose();
surface.Unmap();
int width = surface.Description.Width;
int height = surface.Description.Height;
byte[] bytewidth = BitConverter.GetBytes(width);
byte[] byteheight = BitConverter.GetBytes(height);
Array.Copy(bytewidth, 0, data, 0, 4);
Array.Copy(byteheight, 0, data, 4, 4);
}
Do you guys have any idea why the byte array that is returned at the end is full of 0 since it should be mostly 255? All I did in my backbuffer was to draw a bitmap image and a rectangle form. Array.Copy is to add the width and height header to the byte array, therefore I could create a bitmap out of it.
I answered in a comment but formatting is horrible so apologies!
https://gamedev.stackexchange.com/a/112978/29920 This looks promising but as you said in reply to mine, this was some time ago and I'm pulling this out of thin air, if it doesn't work either someone with more current knowledge will have to answer or I'll have to grab some source code and try myself.
SharpDX.Direct2D1.Bitmap dxbmp = new SharpDX.Direct2D1.Bitmap(renderTarget,
new SharpDX.Size2(bmpWidth, bmpHeight), new
BitmapProperties(renderTarget.PixelFormat));
dxbmp.CopyFromMemory(bmpBits, bmpWidth * 4);
This looks kind of like what you need. I'm assuming bmpBits in here is either a byte array or a memory stream either of which could then be saved off or at least give you something to look at to see if you're actually getting pixel data
Okay, I'm trying to do some work analyzing heuristics of an image in a WPF app. When the user chooses the image file location, I want to open the image in the codebehind, check the colors of the image, and output them in an application. However to properly check the pixels of the image I need to acquire the dimensions to load it to a WriteableBitmap that can get me the pixel colors.
When I open a stream to the file a second time the app just hangs. Here's the code:
private static async Task<Dictionary<Color, int>> GetColorCountsAsync(string path)
{
var colorCounts = new Dictionary<Color, int>();
var absolutePath = string.Format(#"{0}\{1}",Directory.GetCurrentDirectory(),path);
var dimension = await GetBitmapDimensions(absolutePath); //Method below - opens via StorageFile
var file = await StorageFile.GetFileFromPathAsync(absolutePath); //Hangs forever
using (var stream = await file.OpenStreamForReadAsync().ConfigureAwait(false))
{
var pixelWidth = dimension.Width;
var pixelHeight = dimension.Height;
var bitmap = new WriteableBitmap(pixelWidth, pixelHeight);
bitmap.SetSource(stream.AsRandomAccessStream());
using (var buffer = bitmap.PixelBuffer.AsStream())
{
var pixels = new byte[4 * pixelWidth * pixelHeight];
buffer.Read(pixels, 0, pixels.Length);
for (var y = 0; y < pixelHeight; y++)
{
for (var x = 0; x < pixelWidth; x++)
{
var index = ((y * pixelWidth) + x) * 4;
var alpha = pixels[index + 4];
var red = pixels[index + 2];
var green = pixels[index + 1];
var blue = pixels[index + 0];
var color = Color.FromArgb(alpha, red, green, blue);
if (!colorCounts.ContainsKey(color))
{
colorCounts.Add(color, 0);
}
colorCounts[color] = colorCounts[color] + 1;
}
}
}
}
return colorCounts;
}
private static async Task<Dimension> GetBitmapDimensions(string absolutePath)
{
var file = await StorageFile.GetFileFromPathAsync(absolutePath);
var bitmapImage = new BitmapImage();
using (var fileStream = await file.OpenAsync(FileAccessMode.Read))
{
bitmapImage.SetSource(fileStream);
}
return new Dimension { Height = bitmapImage.PixelHeight, Width = bitmapImage.PixelWidth };
}
I can't close the bitmap image nor dispose it - what's causing the app to freeze?
You don't need the bitmap in the correct dimensions, as SetStream() is an extension method. It just needs an object first, call it like this:
var bitmap = new WriteableBitmap(1,1).SetSource(stream.AsRandomAccessStream());