I coded example from https://learn.microsoft.com/en-us/windows/uwp/audio-video-camera/custom-video-effects
Part of the code:
public void ProcessFrame(ProcessVideoFrameContext context)
{
using (CanvasBitmap inputBitmap = CanvasBitmap.CreateFromDirect3D11Surface(canvasDevice, context.InputFrame.Direct3DSurface))
using (CanvasRenderTarget renderTarget = CanvasRenderTarget.CreateFromDirect3D11Surface(canvasDevice, context.OutputFrame.Direct3DSurface))
using (CanvasDrawingSession ds = renderTarget.CreateDrawingSession())
{
var gaussianBlurEffect = new GaussianBlurEffect
{
Source = inputBitmap,
BlurAmount = (float)BlurAmount,
Optimization = EffectOptimization.Speed
};
ds.DrawImage(gaussianBlurEffect);
}
}
The problem is: i want to draw points (bitmaps) on frames but i have no idea how to pass specific coord to ProcessFrame function. On input i have x and y coords for every frame where to draw point and on the output i want to have video with added points for every frame.
Thanks for help.
EDIT:
The code below is not suitable solution as the ProcessFrame(ProcessVideoFrameContext context) is part of an interface implementation.
My next solution proposal is to create a custom effect, similar to the GaussianBlusEffect and many more. An example here:
https://github.com/Microsoft/Win2D-Samples/blob/master/ExampleGallery
~~~
Below the original answer for reference.
You can pass in the X and Y parameters and access the pixels of the image.
public void ProcessFrame(ProcessVideoFrameContext context, int X, int Y)
{
using (CanvasBitmap inputBitmap = CanvasBitmap.CreateFromDirect3D11Surface(canvasDevice, context.InputFrame.Direct3DSurface))
using (CanvasRenderTarget renderTarget = CanvasRenderTarget.CreateFromDirect3D11Surface(canvasDevice, context.OutputFrame.Direct3DSurface))
using (CanvasDrawingSession ds = renderTarget.CreateDrawingSession())
{
Color[] Pixels = inputBitmap.GetPixelColors();
// Manipulate the array using X and Y with the Width parameter of the bitmap
var gaussianBlurEffect = new GaussianBlurEffect
{
Source = inputBitmap,
BlurAmount = (float)BlurAmount,
Optimization = EffectOptimization.Speed
};
ds.DrawImage(gaussianBlurEffect);
}
}
More info: https://microsoft.github.io/Win2D/html/M_Microsoft_Graphics_Canvas_CanvasBitmap_GetPixelColors.htm
I did not check if the Color[] is a pointer to the live buffer or a copy. If it is a copy, then you have to write back the buffer with SetPixelColors.
Related
I'm new to computer vision and currently playing around with static frame differencing to try and determine whether there is motion in video.
My variables:
public Mat currentFrame = new Mat();
public Mat prevFrame = new Mat();
public Mat result = new Mat();
bool motion = false;
Simple differencing function (being called every frame):
public Mat getDifference(Mat videoFrame)
{
currentFrame = videoFrame.Clone();
Cv2.Absdiff(currentFrame, prevFrame, result);
prevFrame = currentFrame.Clone();
return result;
}
When motion exists the result matrix looks like this:
When motion doesn't exist the result matrix looks like this (empty):
My original idea was that if the result matrix is effectively empty (all black), then I could say motion = false. However, this is proving to be more difficult that anticipated since it is technically never empty, so I can't say:
if(!result.Empty())
{
motion = true;
}
Without the need of for loops and pixel by pixel analysis, is there a simple/clean 'if' statement I can use that simply says (if the matrix contains anything that isn't black pixels, motion = true). Or... is this too simplistic? I'm open to hearing better ways of doing this, I had a look around on the web but there aren't many solid examples for C#. My video is playing within a WPF application in real-time so nested for loops are to be avoided.
Thanks for your time!
You could for example convert the matrix to an image. That should give you access to all the image manipulation functions. For example ThresholdBinary to make pixels either zero or a given value, and CountNonZero. That should give you some tools balance how much things need to change, and how large area need to change.
Found a simple way to do it, may not be the best but it does work.
public bool motion = false;
public Mat currentFrame = new Mat();
public Mat prevFrame = new Mat();
public Mat absDiffImage = new Mat();
public Mat grayImage = new Mat();
public Point[][] frameContours;
public HierarchyIndex[] external;
public Mat frameThresh = new Mat();
Cv2.CvtColor(currentFrame, currentFrame, ColorConversionCodes.BGRA2GRAY);
Cv2.Absdiff(currentFrame, prevFrame, absDiffImage);
Cv2.Threshold(absDiffImage, frameThresh, 80, 255, ThresholdTypes.Binary);
Cv2.FindContours(frameThresh, out frameContours, out external, RetrievalModes.List, ContourApproximationModes.ApproxSimple);
if (frameContours.Length > 20)
{
motion = true;
}
else
{
motion = false;
}
Okay, I have a media composition and have added to it 3 media clips...
MediaComposition composition = new MediaComposition();
MediaClip clip1 = await MediaClip.CreateFromFileAsync(file);
MediaClip clip2 = await MediaClip.CreateFromFileAsync(file);
MediaClip clip3 = await MediaClip.CreateFromFileAsync(file);
composition.Clips.Add(clip1);
composition.Clips.Add(clip2;
composition.Clips.Add(clip3);
Now I want each clip to display a distinct caption and to accomplish this I have written a custom video effect that uses the DrawTextLayout method of the Win2D API. Let me be clear, I don't want to do this with overlays! But since video effects are built using the factory design pattern, how would I go about loading different data on each instance of the video effect?
Here is the core of MyVideoEffect:
public void ProcessFrame(ProcessVideoFrameContext context)
{
using (CanvasBitmap inputBitmap = CanvasBitmap.CreateFromDirect3D11Surface(canvasDevice, context.InputFrame.Direct3DSurface))
using (CanvasRenderTarget renderTarget = CanvasRenderTarget.CreateFromDirect3D11Surface(canvasDevice, context.OutputFrame.Direct3DSurface))
using (CanvasDrawingSession ds = renderTarget.CreateDrawingSession())
{
CanvasTextFormat textFormat = new CanvasTextFormat();
CanvasTextLayout textLayout = new CanvasTextLayout(canvasDevice, text, textFormat, width, height);
ds.DrawTextLayout(someTextLayout, x, y, Colors.Yellow);
}
}
Here, the "text" parameter should have a different value each time.
How can I specify the unique value when adding the video effect?
composition.Clips[0].VideoEffectDefinitions.Add(new VideoEffectDefinition(typeof(MyVideoEffect).FullName));
composition.Clips[1].VideoEffectDefinitions.Add(new VideoEffectDefinition(typeof(MyVideoEffect).FullName));
composition.Clips[2].VideoEffectDefinitions.Add(new VideoEffectDefinition(typeof(MyVideoEffect).FullName));
Is there an elegant way of doing this hopefully without having to resort to timers?
Looking for example code to understand how to implement EmguCV tracker. I tried few things like in this poorly written code:
class ObjectTracker
{
Emgu.CV.Tracking.TrackerCSRT tracker = new Emgu.CV.Tracking.TrackerCSRT();
bool isActive = false;
public bool trackerActive = false;
public void Track(Bitmap pic, Rectangle selection,out Rectangle bound)
{
Rectangle result = new Rectangle();
Bitmap bitmap=pic; //This is your bitmap
Emgu.CV.Image<Bgr, Byte> imageCV = new Emgu.CV.Image<Bgr, byte>(bitmap); //Image Class from Emgu.CV
Emgu.CV.Mat mat = imageCV.Mat; //This is your Image converted to Mat
if (tracker.Init(mat,selection))
{
while (tracker.Update(mat, out bound))
{
result = bound;
}
}
bound = result;
}
I'm aware of there is few logic flaws, but still I couldn't menage to get any result in different attempts.
Thanks!
Turns out, it is really simple.
First define a Tracker in a type you desire.
Example:
Emgu.CV.Tracking.TrackerCSRT Tracker= new Emgu.CV.Tracking.TrackerCSRT();
Then before scaning the selected area(RoI) you need to initialize tracker
Example:
Tracker.Initimage, roi);
Important Note: image must be in type of Emgu.CV.Mat , roi Drawing.Rectangle.
To convert your Bitmap to Mat you can use following method:
Example:
public Emgu.CV.Mat toMat(Bitmap pic)
{
Bitmap bitmap=pic;
Emgu.CV.Image<Bgr, Byte> imageCV = new Emgu.CV.Image<Bgr, byte>(bitmap);
Emgu.CV.Mat mat = imageCV.Mat;
return mat;
}
Note: This code belongs to someone else from stackoverflow which I forget the source. Thanks to him/her.
Finally using following statement will returns Rectangle that envelopes tracked object. Trick is object doesn't return Rectangle as method output, it returns it via out statement.
tracker.Update(image, out roi);
Last Note: If anyone knows how to improve performance or multithread this method, please leave a comment.
Have a nice day!
I have been following this tutorial on how to make a custom marker for Google maps, and I have got this part to work. However, I had to change some code in order to resize the image I am using on the marker.
This is the method I am using to do resize:
private void UpdateMarkers(float zoom)
{
// Max zoom out => zoom = 3
// Max zoom in => zoom = 21
int dimension = (int)zoom * 10;
if (dimension == currentDimension)
return;
currentDimension = dimension;
map.Clear();
foreach (var pin in customPins)
{
var immutableBitmap = BitmapFactory.DecodeResource(Context.Resources, Resource.Drawable.icon);
var mutableBitmap = immutableBitmap.Copy(Bitmap.Config.Argb8888, true);
mutableBitmap.Height = dimension;
mutableBitmap.Width = dimension;
BitmapDescriptorFactory.FromBitmap(mutableBitmap);
var img = BitmapDescriptorFactory.FromBitmap(mutableBitmap);
var marker = new MarkerOptions();
marker.SetPosition(new LatLng(pin.Pin.Position.Latitude, pin.Pin.Position.Longitude));
marker.SetTitle(pin.Pin.Label);
marker.SetSnippet(pin.Pin.Address);
marker.SetIcon(img);
map.AddMarker(marker);
}
}
And here is a picture on how it looks:
In the original code, I would do something like this
var img = BitmapDescriptorFactory.FromResource(Resource.Drawable.icon);
And this works. But I want to resize the image, so I found a way to do it, sort of.
Any idea what's going on?
Once you have a Bitmap, you can scale it via Bitmap.CreateScaledBitmap:
var bitmap = BitmapFactory.DecodeResource(Context.Resources, Resource.Drawable.icon);
var scaledBitmap = Bitmap.CreateScaledBitmap(bitmap, dimension, dimension, false);
var img = BitmapDescriptorFactory.FromBitmap(scaledBitmap);
Note: Make sure you Recycle and Dispose your Bitmaps to avoid leaking memory.
Note BitmapFactory's DecodeResource and CreateScaledBitmap are heavy API calls, you might want to cache the results versus calling them over and over in your foreach (var pin in customPins) loop.
I'm writing a 2D game in the latest XNA (C#). I'm doing a bit of an overhaul of old code and have just put back in buttons to change resolution. However, after either fullscreen mode being toggled or the window size/resolution being changed the screen is just black (the color of my clear screen call) forever, although the game still runs, as I can quit it with a keypress.
This didn't happen in my previous version - it all worked fine - but there's no difference in the relevant code I can find. However I did redo my graphics loading to use my own Texture loader instead of the Content Loader - could this be the issue?
If no other option, is there a simple way to make the game restart, as the graphics are ok and in the selected format after a restart?
My code is:
public override void Click()
{
base.Click();
Settings.Default.screenWidth = resolution[0];
Settings.Default.screenHeight = resolution[1];
Settings.Default.Save();
Variables.screenWidth = Settings.Default.screenWidth;
Variables.screenHeight = Settings.Default.screenHeight;
Game1.graphics.PreferredBackBufferWidth = Variables.screenWidth;
Game1.graphics.PreferredBackBufferHeight = Variables.screenHeight;
Game1.graphics.ApplyChanges();
}
Thanks!
Edit: My texture loadign code - loads all files with a name included in an enum as Texture2Ds..
public static void LoadAll(string subFolder)
{
List<string> s = Directory.GetFiles(path + "\\" + subFolder, "*.png", SearchOption.AllDirectories).ToList<string>();
foreach (string S in s)
{
if (Enum.IsDefined(typeof(T), Path.GetFileNameWithoutExtension(S)))
{
FileStream stream = new FileStream(S, FileMode.Open);
Texture2D t = Texture2D.FromStream(Game1.graphics.GraphicsDevice, stream);
RenderTarget2D result = null;
//Setup a render target to hold our final texture which will have premulitplied alpha values
result = new RenderTarget2D(Game1.graphics.GraphicsDevice, t.Width, t.Height);
Game1.graphics.GraphicsDevice.SetRenderTarget(result);
Game1.graphics.GraphicsDevice.Clear(Microsoft.Xna.Framework.Color.Black);
//Multiply each color by the source alpha, and write in just the color values into the final texture
BlendState blendColor = new BlendState();
blendColor.ColorWriteChannels = ColorWriteChannels.Red | ColorWriteChannels.Green | ColorWriteChannels.Blue;
blendColor.AlphaDestinationBlend = Blend.Zero;
blendColor.ColorDestinationBlend = Blend.Zero;
blendColor.AlphaSourceBlend = Blend.SourceAlpha;
blendColor.ColorSourceBlend = Blend.SourceAlpha;
Game1.spriteBatch.Begin(SpriteSortMode.Immediate, blendColor);
Game1.spriteBatch.Draw(t, t.Bounds, Microsoft.Xna.Framework.Color.White);
Game1.spriteBatch.End();
//Now copy over the alpha values from the PNG source texture to the final one, without multiplying them
BlendState blendAlpha = new BlendState();
blendAlpha.ColorWriteChannels = ColorWriteChannels.Alpha;
blendAlpha.AlphaDestinationBlend = Blend.Zero;
blendAlpha.ColorDestinationBlend = Blend.Zero;
blendAlpha.AlphaSourceBlend = Blend.One;
blendAlpha.ColorSourceBlend = Blend.One;
Game1.spriteBatch.Begin(SpriteSortMode.Immediate, blendAlpha);
Game1.spriteBatch.Draw(t, t.Bounds, Microsoft.Xna.Framework.Color.White);
Game1.spriteBatch.End();
//Release the GPU back to drawing to the screen
Game1.graphics.GraphicsDevice.SetRenderTarget(null);
t = result;
textureDictionary.Add(Path.GetFileNameWithoutExtension(S), t);
}
// else
// Console.WriteLine("Did not load -- " + Path.GetFileNameWithoutExtension(S) + " -- (add to enum to enable loading)");
}
}
Edit: The working code from following below advice - probably not the most efficient but it works!
public static void LoadAll(string subFolder)
{
List<string> s = Directory.GetFiles(path + "\\" + subFolder, "*.png", SearchOption.AllDirectories).ToList<string>();
foreach (string S in s)
{
if (Enum.IsDefined(typeof(T), Path.GetFileNameWithoutExtension(S)))
{
FileStream stream = new FileStream(S, FileMode.Open);
Texture2D t = Texture2D.FromStream(Game1.graphics.GraphicsDevice, stream);
RenderTarget2D result = null;
Texture2D resultTexture;
//Setup a render target to hold our final texture which will have premulitplied alpha values
result = new RenderTarget2D(Game1.graphics.GraphicsDevice, t.Width, t.Height);
Game1.graphics.GraphicsDevice.SetRenderTarget(result);
Game1.graphics.GraphicsDevice.Clear(Microsoft.Xna.Framework.Color.Black);
//Multiply each color by the source alpha, and write in just the color values into the final texture
BlendState blendColor = new BlendState();
blendColor.ColorWriteChannels = ColorWriteChannels.Red | ColorWriteChannels.Green | ColorWriteChannels.Blue;
blendColor.AlphaDestinationBlend = Blend.Zero;
blendColor.ColorDestinationBlend = Blend.Zero;
blendColor.AlphaSourceBlend = Blend.SourceAlpha;
blendColor.ColorSourceBlend = Blend.SourceAlpha;
Game1.spriteBatch.Begin(SpriteSortMode.Immediate, blendColor);
Game1.spriteBatch.Draw(t, t.Bounds, Microsoft.Xna.Framework.Color.White);
Game1.spriteBatch.End();
//Now copy over the alpha values from the PNG source texture to the final one, without multiplying them
BlendState blendAlpha = new BlendState();
blendAlpha.ColorWriteChannels = ColorWriteChannels.Alpha;
blendAlpha.AlphaDestinationBlend = Blend.Zero;
blendAlpha.ColorDestinationBlend = Blend.Zero;
blendAlpha.AlphaSourceBlend = Blend.One;
blendAlpha.ColorSourceBlend = Blend.One;
Game1.spriteBatch.Begin(SpriteSortMode.Immediate, blendAlpha);
Game1.spriteBatch.Draw(t, t.Bounds, Microsoft.Xna.Framework.Color.White);
Game1.spriteBatch.End();
//Release the GPU back to drawing to the screen
Game1.graphics.GraphicsDevice.SetRenderTarget(null);
resultTexture = new Texture2D(Game1.graphics.GraphicsDevice, result.Width, result.Height);
Color[] data = new Color[result.Height * result.Width];
Color[] textureColor = new Color[result.Height * result.Width];
result.GetData<Color>(textureColor);
resultTexture.SetData(textureColor);
textureDictionary.Add(Path.GetFileNameWithoutExtension(S), resultTexture);
}
// else
// Console.WriteLine("Did not load -- " + Path.GetFileNameWithoutExtension(S) + " -- (add to enum to enable loading)");
}
}
As per your updated code, you're copying textures into a render target (RenderTarget2D).
Unlike regular textures, these are not backed by CPU-side memory. Regular textures will automatically re-set their data on the GPU when the graphics device is lost. A render target, however, will raise its ContentLost event and set IsContentLost - you are then expected to reset its content yourself!
There are a few solutions: You could simply respond to ContentLost and refresh the data.
I prefer to use GetData and SetData at load time to copy the contents of the render target into a regular Texture2D, because this "just works". Although for simple cases like switching a texture to use premultiplied alpha, I prefer to just keep everything on the CPU.
I've got a very detailed answer about RenderTarget2D usage for fixed textures over here. It includes code to do premultiplication directly on the CPU.
Although in your case, I'd try and use the content manager. It is possible to have it process all files in a directory and then load them dynamically.