Develop Reference

How to resize VideoFrame or ImageFeatureValue to specific size to match input shape requirements?

My goal is to use tinyYolov3 model to perform object detection in real-time through the HoloLens. I want to incorporate the model as an ONNX file directly in the project and to compute the predictions inside the HoloLens itself. In order to do so, I am planning to use Windows.media and Windows.AI.MachineLearning libraries as a pipeline between the camera and my predictions.
Following this tutorial, I am able to capture the frames as VideoFrame and I can convert them in ImageFeatureValue to match my input type requirement. My issue now is about the shape requirement. Yolo models need a 3x416x416 frame as input and I can't find any docs online about resizing VideoFrame or ImageFeatureValue.
Thank you very much for your help.
using (var frameReference = CameraFrameReader.TryAcquireLatestFrame())
using (var videoFrame = frameReference?.VideoMediaFrame?.GetVideoFrame())
await ModelHelper.EvaluateVideoFrameAsync(videoFrame).ConfigureAwait(false);
public async Task EvaluateVideoFrameAsync(VideoFrame frame)
{
if (frame != null)
{
try
{
ModelInput inputData = new ModelInput();
inputData.image = ImageFeatureValue.CreateFromVideoFrame(frame);
//TODO: CHANGE SIZE FRAME
var output = await Model.EvaluateAsync(inputData).ConfigureAwait(false);
}
}
}

I have no experience using the Windows Machine Learning API and ImageFeatureValue class. But when I tried to resize frames from the HoloLens, I had to use the SoftwareBitmap instead of VideoFrame. Then, I use BitmapEncoder to resize them, and convert back to VideoFrame:
private async Task<SoftwareBitmap> ResizeBitmap(SoftwareBitmap softwareBitmap, uint width, uint height)
{
using (InMemoryRandomAccessStream stream = new InMemoryRandomAccessStream())
{
BitmapEncoder encoder = await BitmapEncoder.CreateAsync(BitmapEncoder.BmpEncoderId, stream);
encoder.SetSoftwareBitmap(softwareBitmap);
encoder.BitmapTransform.ScaledWidth = width;
encoder.BitmapTransform.ScaledHeight = height;
encoder.BitmapTransform.InterpolationMode = BitmapInterpolationMode.NearestNeighbor;
await encoder.FlushAsync();
BitmapDecoder decoder = await BitmapDecoder.CreateAsync(stream);
return await decoder.GetSoftwareBitmapAsync(softwareBitmap.BitmapPixelFormat, softwareBitmap.BitmapAlphaMode);
}
}
var inputBitmap = frameReference.VideoMediaFrame.SoftwareBitmap;
var outputBitmap = ResizeBitmap(inputBitmap, your_width, your_height);
var outputVideoFrame = VideoFrame.CreateWithSoftwareBitmap(SoftwareBitmap);

Send InkCanvas strokes directly to IRandomnaccessStream

I'm trying to get a byte array from the InkCanvas control, but the method i've come up with so far seems a bit long winded.
Currently I use the following:
StorageFolder folder = await Windows.Storage.ApplicationData.Current.LocalFolder.GetFolderAsync("Temp");
StorageFile file = await folder.CreateFileAsync(GenerateString(5)+".zzx", Windows.Storage.CreationCollisionOption.GenerateUniqueName);
using (IRandomAccessStream stream = await file.OpenAsync(FileAccessMode.ReadWrite))
{
await SignatureCanvas.InkPresenter.StrokeContainer.SaveAsync(stream);
var array = await IRandomAccessStreamToByteArray(stream);
}
The custom stream reader is as follows.
private async Task<byte[]> IRandomAccessStreamToByteArray(IRandomAccessStream stream)
{
var reader = new DataReader(stream.GetInputStreamAt(0));
var bytes = new byte[stream.Size];
await reader.LoadAsync((uint)stream.Size);
reader.ReadBytes(bytes);
return bytes;
}
This works, and gives me the byte array that i need, but also leaves me with unwanted images. Was having some access issues due to files still being written when another call wanted to replace the file so decided to go down the multiple images route. Is there a way to skip the image file entirely? It's not too much of an issue to clear out a temp folder, but if it can be avoided that would be preferable.
I had read somewhere already that InkCanvas doesn't support direct to array dumps, so any suggestions would be appreciated!

If you want to save your InkCanvas strokes to bytes Array without create the file, you should be able to use the Win2D.uwp.
To install Win2D.uwp, run the "Install-Package Win2D.uwp" command in the Package Manager Console.
There is a CanvasDrawingSession.DrawInk method to draw a collection of ink strokes. That we should be able to use CanvasBitmap.GetPixelBytes method to get an array of raw byte data for the entire bitmap.
For example:
private byte[] ConvertInkCanvasToByteArray()
{
var canvasStrokes = SignatureCanvas.InkPresenter.StrokeContainer.GetStrokes();
if (canvasStrokes.Count > 0)
{
var width = (int)SignatureCanvas.ActualWidth;
var height = (int)SignatureCanvas.ActualHeight;
var device = CanvasDevice.GetSharedDevice();
device.DeviceLost += DeviceOnDeviceLost;
var renderTarget = new CanvasRenderTarget(device, width, height, 96);
using (var ds = renderTarget.CreateDrawingSession())
{
ds.Clear(Windows.UI.Colors.White);
ds.DrawInk(SignatureCanvas.InkPresenter.StrokeContainer.GetStrokes());
}
return renderTarget.GetPixelBytes();
}
else
{
return null;
}
}
private void DeviceOnDeviceLost(CanvasDevice sender, object args)
{
Debug.WriteLine("DeviceOnDeviceLost");
}
Also if we want to convert the bytes array to image, we should be able to use following code:
WriteableBitmap bitmap = new WriteableBitmap((int)SignatureCanvas.ActualWidth, (int)SignatureCanvas.ActualHeight);
await bitmap.PixelBuffer.AsStream().WriteAsync(mybytes, 0, mybytes.Length);

How do i take a photo with the correct rotation, aspect ratio in Windows Phone 8.1? (using MediaCapture)

Can any of you provide an actual working sample of how to take and save a photo using the MediaCapture element. I've tried looking for an actual solution in MSDN but none of those explanations or code actually describe the process in a simple way.
I need to take a picture and save it to my library (i need to show the correct preview for this), however right now it is rotated 90 degrees and i can't adjust it. I've tried setting the rotation of the video preview and it works for the preview however when i do this the aspect ratio its all wrong and the saved image its not correct.
The examples from channel 9 kind of suck too. I just need a simple implementation...
Im using a Runtime app NOT a silverlight app for Windows Phone 8.1.

I have had the same issue, SetRecordRotation doesn't work for me. I found workaround - take photo and rotate an image, it works great. I use method like that:
private async void CapturePhoto()
{
string photoPath = string.Empty;
ImageEncodingProperties format = ImageEncodingProperties.CreateJpeg();
using (var imageStream = new InMemoryRandomAccessStream())
{
await MediaCapture.CapturePhotoToStreamAsync(format, imageStream);
BitmapDecoder dec = await BitmapDecoder.CreateAsync(imageStream);
BitmapEncoder enc = await BitmapEncoder.CreateForTranscodingAsync(imageStream, dec);
enc.BitmapTransform.Rotation = BitmapRotation.Clockwise90Degrees;
await enc.FlushAsync();
StorageFolder folder = ApplicationData.Current.LocalFolder;
StorageFile capturefile = await folder.CreateFileAsync("photo.jpg", CreationCollisionOption.GenerateUniqueName);
photoPath = capturefile.Name;
using (var fileStream = await capturefile.OpenAsync(FileAccessMode.ReadWrite))
{
try
{
await RandomAccessStream.CopyAsync(imageStream, fileStream);
}
catch {}
}
}
}
I modified sample of code from article How to capture a photo in your Windows Phone 8.1 Runtime app by Marco Siccardi
http://dotnet.dzone.com/articles/how-capture-photo-your-windows-0

There are two samples posted on the Microsoft github page that are relevant, although they target Windows 10. Still, the APIs should work on 8/8.1.
GetPreviewFrame: This sample will not lock the page rotation, and apply a corrective rotation to the preview stream. It does not use SetPreviewRotation, as that method is more resource-heavy than using the metadata approach. This sample doesn't capture photos (just preview frames)
UniversalCameraSample: This one does capture photos, and supports portrait and landscape orientations. Here is the relevant part:
var stream = new InMemoryRandomAccessStream();
try
{
await _mediaCapture.CapturePhotoToStreamAsync(ImageEncodingProperties.CreateJpeg(), stream);
var photoOrientation = ConvertOrientationToPhotoOrientation(GetCameraOrientation());
await ReencodeAndSavePhotoAsync(stream, photoOrientation);
}
catch (Exception ex)
{
Debug.WriteLine("Exception when taking a photo: {0}", ex.ToString());
}
With:
private static async Task ReencodeAndSavePhotoAsync(IRandomAccessStream stream, PhotoOrientation photoOrientation)
{
using (var inputStream = stream)
{
var decoder = await BitmapDecoder.CreateAsync(inputStream);
var file = await KnownFolders.PicturesLibrary.CreateFileAsync("SimplePhoto.jpeg", CreationCollisionOption.GenerateUniqueName);
using (var outputStream = await file.OpenAsync(FileAccessMode.ReadWrite))
{
var encoder = await BitmapEncoder.CreateForTranscodingAsync(outputStream, decoder);
var properties = new BitmapPropertySet { { "System.Photo.Orientation", new BitmapTypedValue(photoOrientation, PropertyType.UInt16) } };
await encoder.BitmapProperties.SetPropertiesAsync(properties);
await encoder.FlushAsync();
}
}
}
Have a closer look at the sample to see how to get the orientation of the camera in the first place (a call to it is being made in the first snippet I posted).
Or, if you prefer a video, you can watch the camera session from the recent //build/ conference, which includes a little bit of a walkthrough through some camera samples.

you can change the aspect ratio for your video preview & captured photo by setting in the MediaCapture.VideoDeviceController.
Also, you can set your video preview upright by using the following code.
MediaCapture.SetPreviewRotation(VideoRotation.Clockwise90Degrees);
I have answered a similar questions in the another post in the link below. Hope it helps.
https://stackoverflow.com/a/29875992/4672579

The best way load files from Isolated Storage

In my Windows Store App I save/use files (almost images) in Isolated Storage. When I need to present image i use following:
var file = await folder.GetFileAsync(fileName);
using (var stream = await file.OpenAsync(FileAccessMode.Read))
{
obj.Image = new BitmapImage();
await obj.Image.SetSourceAsync(stream);
}
But when I use 3+ images in same page I have lags. I'm looking for faster solution to access Isolated Storage files.

You can try to start with this article Optimize media resources (Windows Store apps using C#/VB/C++ and XAML), and couple more things:
Make sure that if you show them in the ListView / GridView - you have enabled virtualization (you use right ItemsPanel which supports virtualization).
If you just need to load images from local storage - set the binding from Image.Source to to the right URI (ms-appx:/// or ms-appdata:///local/), and Image control will do everything for you.

I don't know how you're opening multiple images, but since all the methods are asynchronous you shouldn't iterate through your files sequentially, but open all of them in parallel.
So instead of doing this (where you're waiting for the previous image to load before starting to load the next one):
foreach (var fileName in fileNames)
{
var file = await folder.GetFileAsync(fileName);
using (var stream = await file.OpenAsync(FileAccessMode.Read))
{
obj.Image = new BitmapImage();
await obj.Image.SetSourceAsync(stream);
}
}
You should approach it like this:
// not sure about the type of obj
public async Task<Image> LoadImage(string fileName, dynamic obj)
{
var file = await folder.GetFileAsync(fileName);
using (var stream = await file.OpenAsync(FileAccessMode.Read))
{
obj.Image = new BitmapImage();
await obj.Image.SetSourceAsync(stream);
}
}
var tasks = fileNames.Select(f => LoadImage(f, obj)).ToArray();
await Task.WhenAll(tasks);
This will initialize an array of awaitable tasks loading the images and then await all of them at the same time so that they will execute in parallel.

WinRT Convert Jpg or PNG to GIF

I'm working on a WinRT application that will do some image processing and one of the things I want to do is convert some jpgs or pngs to gif. I have something that sort of works. For some of my test jpgs it works others it's a scrambled image that get's output. Just wondering if there was something I was missing. Here is what I have so far
public async static void ConvertToGif(IRandomAccessStream stream)
{
var decoder = await BitmapDecoder.CreateAsync(stream);
var pixels = await decoder.GetPixelDataAsync();
var file = await KnownFolders.PicturesLibrary.CreateFileAsync("test.gif", CreationCollisionOption.ReplaceExisting);
var outStream = await file.OpenAsync(FileAccessMode.ReadWrite);
var encoder = await BitmapEncoder.CreateAsync(BitmapEncoder.GifEncoderId, outStream);
encoder.SetPixelData(decoder.BitmapPixelFormat, BitmapAlphaMode.Ignore,
decoder.PixelWidth, decoder.PixelHeight,
decoder.DpiX, decoder.DpiY,
pixels.DetachPixelData());
await encoder.FlushAsync();
outStream.Dispose();
}
Smaller jpgs seem to work, but larger ones come out scrambled. Is there another way to achieve this?

Duh, the problem was that I was using PixelWidth/Height and I sholud have been using OrientedPixelWidth/Height.
That seems to have resolved my issue for this.