I want to use an invisible MediaPlayer to generate thumbnails at specific time positions in a video
I've tried waiting on the player's dispatcher thread with _player.Dispatcher.BeginInvoke(DispatcherPriority.SystemIdle, new Action(() => { })).Wait();
but that didn't seem to make any difference. It works with a long enough wait on the thread, but I don't like using an arbitrary number.
private MediaPlayer _player;
private void Player_MediaOpened(object sender, EventArgs e)
{
DrawingVisual dv = new DrawingVisual();
TimeSpan duration = _player.NaturalDuration.TimeSpan;
TimeSpan time = new TimeSpan();
TimeSpan interval = new TimeSpan(0, 0, 1); // One Second
RenderTargetBitmap bmp = new RenderTargetBitmap(160, 90, 96, 96, PixelFormats.Pbgra32);
while (time < duration)
{
_player.Position = time;
Thread.Sleep(10); // This is the line I'm unhappy about!
Application.Current?.Dispatcher.BeginInvoke(DispatcherPriority.ApplicationIdle, new Action(() => { })).Wait(); // Wait for the UI to catch up so you see all the images as they are being rendered
using (DrawingContext dc = dv.RenderOpen())
{
dc.DrawVideo(_player, new Rect(0, 0, 160, 90));
}
bmp.Render(dv);
Image thumbImage = new Image()
{
Width = 160,
Height = 90
};
thumbImage.Source = bmp.GetAsFrozen() as ImageSource;
thumbnailsPanel.Children.Add(thumbImage);
time = time.Add(interval);
}
_player.Close();
}
With the Thead.sleep(10); it works correctly but I'm uncomfortable with an arbirary pause. I'd like to wait until the MediaPlayer has completed _player.Position = time;. If Thead.sleep(10); is reduced to, say, 1ms, then the images produced are incorrect - they show earlier frames instead of the actual ones.
Does anyone know how to wait until the MediaPlayer has finished this asynchronous operation?
Related
I am using Intel Realsense RGB camera to show live stream on WPF window as well as save the stream into a file. What I am showing on window has correct colors but when I save it, the video colors are off (more purple). Please see screenshot: https://ibb.co/txy9Sgd
I am using a EmguCV video writer to save the video. I don't have much knowledge about formats. I am guessing I am doing something wrong with Format24bppRgb format?
private Pipeline pipeline = new Pipeline(); // Create and config the pipeline to strem color and depth frames.
private CancellationTokenSource tokenSource;
private VideoWriter writer = null;
public StartTests()
{
InitializeComponent();
}
private void Window_Loaded(object sender, RoutedEventArgs e)
{
tokenSource = new CancellationTokenSource();
int fcc = VideoWriter.Fourcc('M', 'P', '4', 'V'); //'M', 'J', 'P', 'G'
float fps = 15F;
writer = new VideoWriter("testttt.mp4", fcc, fps, new System.Drawing.Size(640, 480), true);
Config cfg = new Config();
cfg.EnableStream(Stream.Color, 640, 480, format: Format.Rgb8);
PipelineProfile pp = pipeline.Start(cfg);
StartRenderFrames(pp);
}
private void StartRenderFrames(PipelineProfile pp)
{
// Allocate bitmaps for rendring. Since the sample aligns the depth frames to the color frames, both of the images will have the color resolution
using (VideoStreamProfile p = pp.GetStream(Stream.Color) as VideoStreamProfile)
{
imgColor.Source = new WriteableBitmap(p.Width, p.Height, 96d, 96d, PixelFormats.Rgb24, null);
}
Action<VideoFrame> updateColor = UpdateImage(imgColor);
Task.Factory.StartNew(() =>
{
while (!tokenSource.Token.IsCancellationRequested)
{
using (FrameSet frames = pipeline.WaitForFrames()) // Wait for the next available FrameSet
{
VideoFrame colorFrame = frames.ColorFrame.DisposeWith(frames);
// Save frames to file here...
System.Drawing.Bitmap ColorImg = new System.Drawing.Bitmap(colorFrame.Width, colorFrame.Height, colorFrame.Stride, System.Drawing.Imaging.PixelFormat.Format24bppRgb, colorFrame.Data);
Image<Bgr, Byte> imageCV = new Image<Bgr, byte>(ColorImg); //Image Class from Emgu.CV
Mat matFrame = imageCV.Mat;
writer.Write(matFrame);
// Render to WPF window here...
Dispatcher.Invoke(DispatcherPriority.Render, updateColor, colorFrame);
}
}
}, tokenSource.Token);
}
static Action<VideoFrame> UpdateImage(Image img)
{
WriteableBitmap wbmp = img.Source as WriteableBitmap;
return new Action<VideoFrame>(frame =>
{
using (frame)
{
var rect = new Int32Rect(0, 0, frame.Width, frame.Height);
wbmp.WritePixels(rect, frame.Data, frame.Stride * frame.Height, frame.Stride);
}
});
}
private void Window_Closing(object sender, System.ComponentModel.CancelEventArgs e)
{
tokenSource.Cancel();
tokenSource.Dispose();
pipeline.Stop();
writer.Dispose();
tokenSource = new CancellationTokenSource();
}
I want to see consistent colors in the saved .mp4 file but I am seeing weird colors.
Note - My code is based on this example: https://github.com/IntelRealSense/librealsense/blob/master/wrappers/csharp/cs-tutorial-2-capture/Window.xaml.cs
OpenCV assumes BGR colour format, so if you change the RealSense stream format to Format.Bgra8 and the WriteableBitmap format to PixelFormats.Bgr24, you should be alright.
So you should have:
cfg.EnableStream(Stream.Color, 640, 480, format: Format.Bgr8);
and
new WriteableBitmap(p.Width, p.Height, 96d, 96d, PixelFormats.Bgr24, null);
I don't think you'll need to change the System.Drawing.Bitmap pixelformat as you're only using it to feed the OpenCV mat.
I am working in WPF and creating a gallery folder where I want to show the video thumbnails of the videos in a folder present in the system. I wrote the following code in C# to get the video thumbnails.
private BitmapSource RenderThumb(Uri uri)
{
var player = new MediaPlayer { Volume = 0, ScrubbingEnabled = true };
player.Open(uri);
Thread.Sleep(3000);
player.Pause();
player.Position = player.NaturalDuration.HasTimeSpan
? TimeSpan.FromSeconds(player.NaturalDuration.TimeSpan.TotalSeconds / 2)
: TimeSpan.FromSeconds(2);
int width = player.NaturalVideoWidth;
int height = player.NaturalVideoHeight;
if (width == 0 || height == 0)
{
throw new InvalidOperationException("Width or Height cannot be 0");
}
var rtb = new RenderTargetBitmap(width, height, 96, 96, PixelFormats.Pbgra32);
var dv = new DrawingVisual();
using (DrawingContext dc = dv.RenderOpen())
{
dc.DrawVideo(player, new Rect(0, 0, width, height));
}
player.Close();
rtb.Render(dv);
Freezable frame = BitmapFrame.Create(rtb).GetCurrentValueAsFrozen();
Freezable smallerFrame =
BitmapFrame.Create(new TransformedBitmap(source: frame as BitmapSource, newTransform: new ScaleTransform(0.5, 0.5))).
GetCurrentValueAsFrozen();
return smallerFrame as BitmapSource;
}
however if I try to load thumbnails of 10 videos for example, then I get blank/black thumbnails for 2 3 videos everytime and all the others load fine. This behavior is random and sometimes I get blank thumbnails for last 3 videos and sometimes for the first 3 videos so I am not sure where to look for a solution to my issue. I have tried looking at the properties of BitmapSource but the properties of all 10 videos have the same info in the debugger.
Moving player.Close(); at the bottom of the function after rendering solved the problem for me.
I am receiving video from kinect device. Server is sending video frame by frame and on the client side it receives frame but starts flickering on image control if I use BitmapSource. Create function which is responsible of increasing CPU usage after that i use WriteableBitmap class but I'm stuck into a new problem, it is giving me error "the calling thread cannot access the object but different thread own it", i use dispather.invoke to solve the problem but it is giving me the same error.
public partial class MainWindow : Window
{
TcpClient client;
NetworkStream ns;
Thread vedioframe;
WriteableBitmap vediofram = null;
public MainWindow()
{
InitializeComponent();
client = new TcpClient();
client.Connect("127.0.0.1", 9000);
vedioframe = new Thread(Display_Frame);
vedioframe.Start();
}
public void Display_Frame()
{
ns = client.GetStream();
while (true)
{
byte[] vedio = new byte[1228800];
ns.Read(vedio, 0, vedio.Length);
try
{
if (vediofram == null)
{
vediofram = new WriteableBitmap(640, 480, 96, 96, PixelFormats.Bgr32, null);
}
else
{
vediofram.WritePixels(new Int32Rect(0, 0, 640, 480), vedio, 640 * 4, 0);
}
Update_Frame(vediofram);
}
catch (Exception e)
{
MessageBox.Show(e.Message);
}
// Dispatcher.Invoke(new Action(() => { BitmapSource s = BitmapSource.Create(640, 480, 96, 96, PixelFormats.Bgr32, null, vedio, 640 * 4);
// Vedio.Source = s;
/// }));
}
}
void Update_Frame(WriteableBitmap src)
{
Dispatcher.Invoke(new Action(() => { Vedio.Source = src; }));
}
}
The problem is that you're creating the WriteableBitmap in your background thread. It needs to be created on the UI thread, and you would want to pass the data to the UI thread to update the bitmap.
The first answer to Asynchronous operations on WriteableBitmap elaborates further.
I have a UISlider. It is used to navigate quickly through a PDF. Whenever the threshold for the next page is reached, I display a UIView next to the slider's knob that contains a small preview of the target page.
The slider code looks is below this (some parts stripped). If the next page is reached, a new preview is generated, Otherwise, the existing one is moved along the slider.
I get various effects:
If previewing many many pages, the app crashes at
MonoTouch.CoreGraphics.CGContext.Dispose (bool) <0x00047>
Oct 11 17:21:13 unknown UIKitApplication:com.brainloop.brainloopbrowser[0x1a2d][2951] : at MonoTouch.CoreGraphics.CGContext.Finalize () <0x0002f>
or if I remove the calls to Dispose() in the last method: [NSAutoreleasePool release]: This pool has already been released, do not drain it (double release).
From looking at the code, does somebody have an idea what's wrong? Or is the whole approach of using a thread wrong?
this.oScrollSlider = new UISlider ();
this.oScrollSlider.TouchDragInside += delegate( object oSender, EventArgs oArgs )
{
this.iCurrentPage = (int)Math.Round (oScrollSlider.Value);
if (this.iCurrentPage != this.iLastScrollSliderPage)
{
this.iLastScrollSliderPage = this.iCurrentPage;
this.RenderScrollPreviewImage(this.iCurrentPage);
}
};
this.oScrollSlider.ValueChanged += delegate
{
if (this.oScrollSliderPreview != null)
{
this.oScrollSliderPreview.RemoveFromSuperview ();
this.oScrollSliderPreview.Dispose();
this.oScrollSliderPreview = null;
}
// Go to the selected page.
};
The method creating the preview is spinning off a new thread. If the user changes pages whil the thread is still going, it is aborted and the next page is previewed:
private void RenderScrollPreviewImage (int iPage)
{
// Create a new preview view if not there.
if(this.oScrollSliderPreview == null)
{
SizeF oSize = new SizeF(150, 200);
RectangleF oFrame = new RectangleF(new PointF (this.View.Bounds.Width - oSize.Width - 50, this.GetScrollSliderOffset (oSize)), oSize);
this.oScrollSliderPreview = new UIView(oFrame);
this.oScrollSliderPreview.BackgroundColor = UIColor.White;
this.View.AddSubview(this.oScrollSliderPreview);
UIActivityIndicatorView oIndicator = new UIActivityIndicatorView(UIActivityIndicatorViewStyle.Gray);
oIndicator.Center = new PointF(this.oScrollSliderPreview.Bounds.Width/2, this.oScrollSliderPreview.Bounds.Height/2);
this.oScrollSliderPreview.AddSubview(oIndicator);
oIndicator.StartAnimating();
}
// Remove all subviews, except the activity indicator.
if(this.oScrollSliderPreview.Subviews.Length > 0)
{
foreach(UIView oSubview in this.oScrollSliderPreview.Subviews)
{
if(!(oSubview is UIActivityIndicatorView))
{
oSubview.RemoveFromSuperview();
}
}
}
// Kill the currently running thread that renders a preview.
if(this.oRenderScrollPreviewImagesThread != null)
{
try
{
this.oRenderScrollPreviewImagesThread.Abort();
}
catch(ThreadAbortException)
{
// Expected.
}
}
// Start a new rendering thread.
this.oRenderScrollPreviewImagesThread = new Thread (delegate()
{
using (var oPool = new NSAutoreleasePool())
{
try
{
// Create a quick preview.
UIImageView oImgView = PdfViewerHelpers.GetLowResPagePreview (this.oPdfDoc.GetPage (iPage), new RectangleF (0, 0, 150, 200));
this.InvokeOnMainThread(delegate
{
if(this.oScrollSliderPreview != null)
{
oImgView.Center = new PointF(this.oScrollSliderPreview.Bounds.Width/2, this.oScrollSliderPreview.Bounds.Height/2);
// Add the PDF image to the preview view.
this.oScrollSliderPreview.AddSubview(oImgView);
}
});
}
catch (Exception)
{
}
}
});
// Start the thread.
this.oRenderScrollPreviewImagesThread.Start ();
}
To render the PDF image, I use this:
internal static UIImageView GetLowResPagePreview (CGPDFPage oPdfPage, RectangleF oTargetRect)
{
RectangleF oPdfPageRect = oPdfPage.GetBoxRect (CGPDFBox.Media);
// If preview is requested for the PDF index view, render a smaller version.
float fAspectScale = 1.0f;
if (!oTargetRect.IsEmpty)
{
fAspectScale = GetAspectZoomFactor (oTargetRect.Size, oPdfPageRect.Size, false);
// Resize the PDF page so that it fits the target rectangle.
oPdfPageRect = new RectangleF (new PointF (0, 0), GetFittingBox (oTargetRect.Size, oPdfPageRect.Size));
}
// Create a low res image representation of the PDF page to display before the TiledPDFView
// renders its content.
int iWidth = Convert.ToInt32 ( oPdfPageRect.Size.Width );
int iHeight = Convert.ToInt32 ( oPdfPageRect.Size.Height );
CGColorSpace oColorSpace = CGColorSpace.CreateDeviceRGB();
CGBitmapContext oContext = new CGBitmapContext(null, iWidth, iHeight, 8, iWidth * 4, oColorSpace, CGImageAlphaInfo.PremultipliedLast);
// First fill the background with white.
oContext.SetFillColor (1.0f, 1.0f, 1.0f, 1.0f);
oContext.FillRect (oPdfPageRect);
// Scale the context so that the PDF page is rendered
// at the correct size for the zoom level.
oContext.ScaleCTM (fAspectScale, fAspectScale);
oContext.DrawPDFPage (oPdfPage);
CGImage oImage = oContext.ToImage();
UIImage oBackgroundImage = UIImage.FromImage( oImage);
oContext.Dispose();
oImage.Dispose ();
oColorSpace.Dispose ();
UIImageView oBackgroundImageView = new UIImageView (oBackgroundImage);
oBackgroundImageView.Frame = new RectangleF (new PointF (0, 0), oPdfPageRect.Size);
oBackgroundImageView.ContentMode = UIViewContentMode.ScaleToFill;
oBackgroundImageView.UserInteractionEnabled = false;
oBackgroundImageView.AutoresizingMask = UIViewAutoresizing.None;
return oBackgroundImageView;
}
Avoid Thread.Abort().
Yeah, here are some links talking about it:
http://www.interact-sw.co.uk/iangblog/2004/11/12/cancellation
http://haacked.com/archive/2004/11/12/how-to-stop-a-thread.aspx
If you can use the .Net 4.0 features, go with them instead. Using a Task<T> is probably easier to work with in your case.
Also, I think it would be helpful to create some throttling and only start your new thread after 25-100 milliseconds of inactivity from the user.
Get rid of Thread.Abort() as Jonathan suggests.
Don't start a new thread for each image and instead have one background thread with a work queue. Then simply put the most important page to the front of the queue so it renders as soon as possible. You can also optionally limit the size of the queue or remove unneeded items from it.
I've got a background thread that is creating grayscale thumbnails of images in a given folder. The problem I'm seeing is that the Graphics.DrawImage() call in the background thread seems to be somehow blocking Graphics operations on the main UI thread.
I may be misinterpreting what I'm seeing here, and won't have a chance to do any in-depth profiling until later tonight, though I don't expect to be able to find much.
I've tried to come up with as small a repro case as possible. If you replace the form in a default project with the form below (and have some images in a folder to test with), you'll notice that the animating label will stutter as it bounces back and forth across the window. Yet if you uncomment the #define at the top so that a child control animates rather than redrawing the window contents, it runs perfectly smoothly.
Can anyone see what I'm doing wrong here or help me figure out how to avoid this stutter during the update loop?
//#define USE_LABEL_CONTROL
using System;
using System.Collections.Generic;
using System.Drawing;
using System.Drawing.Imaging;
using System.IO;
using System.Threading;
using System.Windows.Forms;
using Timer = System.Windows.Forms.Timer;
namespace ThreadTest
{
public partial class Form1 : Form
{
private const string ImageFolder = "c:\\pics";
private const string ImageType = "*.jpg";
public Form1()
{
InitializeComponent();
}
protected override void OnLoad(EventArgs e)
{
this.Size = new Size(300, 300);
string[] ImageFiles = Directory.GetFiles(ImageFolder,
ImageType,
SearchOption.AllDirectories);
// kick off a thread to create grayscale thumbnails of all images
this.thumbnailThread = new Thread(this.thumbnailThreadFunc);
this.thumbnailThread.Priority = ThreadPriority.Lowest;
this.thumbnailThread.Start(ImageFiles);
// set a timer to start us off...
this.startTimer = new Timer();
this.startTimer.Interval = 500;
this.startTimer.Tick += this.startTimer_Tick;
this.startTimer.Start();
#if USE_LABEL_CONTROL
this.label.Location = this.labelRect.Location;
this.label.Size = this.labelRect.Size;
this.label.Text = "Loaded: 0";
this.label.BorderStyle = BorderStyle.FixedSingle;
this.Controls.Add(this.label);
#endif
base.OnLoad(e);
}
void startTimer_Tick(object sender, EventArgs e)
{
// kill the timer
this.startTimer.Stop();
// update ourself in a loop
while (this.IsHandleCreated)
{
int NextTick = Environment.TickCount + 50;
// update the label position
this.labelRect.Offset(this.currentLabelDirection, 0);
if (this.labelRect.Right == this.ClientRectangle.Right ||
this.labelRect.Left == 0)
{
this.currentLabelDirection = -this.currentLabelDirection;
}
// update the display
#if USE_LABEL_CONTROL
this.label.Text = "Loaded: " + this.thumbs.Count;
this.label.Location = this.labelRect.Location;
#else
using (Graphics Dest = this.CreateGraphics())
{
this.redrawControl(Dest, this.ClientRectangle);
}
#endif
Application.DoEvents();
Thread.Sleep(Math.Max(0, NextTick - Environment.TickCount));
}
}
private void thumbnailThreadFunc(object ThreadData)
{
string[] ImageFiles = (string[]) ThreadData;
foreach (string ImageFile in ImageFiles)
{
if (!this.IsHandleCreated)
{
return;
}
using (Image SrcImg = Image.FromFile(ImageFile))
{
Rectangle SrcRect = new Rectangle(Point.Empty, SrcImg.Size);
Rectangle DstRect = new Rectangle(Point.Empty, new Size(300, 200));
Bitmap DstImg = new Bitmap(DstRect.Width, DstRect.Height);
using (Graphics Dst = Graphics.FromImage(DstImg))
{
using (ImageAttributes Attrib = new ImageAttributes())
{
Attrib.SetColorMatrix(this.grayScaleMatrix);
Dst.DrawImage(SrcImg,
DstRect,
0, 0, SrcRect.Width, SrcRect.Height,
GraphicsUnit.Pixel,
Attrib);
}
}
lock (this.thumbs)
{
this.thumbs.Add(DstImg);
}
}
}
}
#if !USE_LABEL_CONTROL
private void redrawControl (Graphics Dest, Rectangle UpdateRect)
{
Bitmap OffscreenImg = new Bitmap(this.ClientRectangle.Width,
this.ClientRectangle.Height);
using (Graphics Offscreen = Graphics.FromImage(OffscreenImg))
{
Offscreen.FillRectangle(Brushes.White, this.ClientRectangle);
Offscreen.DrawRectangle(Pens.Black, this.labelRect);
Offscreen.DrawString("Loaded: " + this.thumbs.Count,
SystemFonts.MenuFont,
Brushes.Black,
this.labelRect);
}
Dest.DrawImageUnscaled(OffscreenImg, 0, 0);
OffscreenImg.Dispose();
}
protected override void OnPaintBackground(PaintEventArgs e)
{
return;
}
protected override void OnPaint(PaintEventArgs e)
{
this.redrawControl(e.Graphics, e.ClipRectangle);
}
#endif
private ColorMatrix grayScaleMatrix = new ColorMatrix(new float[][]
{
new float[] {.3f, .3f, .3f, 0, 0},
new float[] {.59f, .59f, .59f, 0, 0},
new float[] {.11f, .11f, .11f, 0, 0},
new float[] {0, 0, 0, 1, 0},
new float[] {0, 0, 0, 0, 1}
});
private Thread thumbnailThread;
private Timer startTimer;
private List<Bitmap> thumbs = new List<Bitmap>();
private Label label = new Label();
private int currentLabelDirection = 1;
private Rectangle labelRect = new Rectangle(0, 125, 75, 20);
}
}
It turns out that the answer is to use multiple processes to handle background GDI+ tasks. If you run the above code under the concurrency profiler in VS2010, you'll see the foreground thread blocking on a critical section secured by the DrawImage() call in the background thread.
This thread also discusses this issue and points out that since it's using a critical section, the locks will be per-process and the background tasks can be parallelized using multiple processes instead of threads:
Parallelizing GDI+ Image Resizing .net
Any time you have performance problems, you need to consider as many factors as possible. In your case, there's a rather large difference between the label and the redraw implementations.
The label version is redrawing only a small (or two small) sections of the screen.
The redraw version is doing much more work and redrawing a larger area.
This in itself could be the problem, depending upon the capabilities of the computer you're using.