I want to create a simple video renderer to play around, and do stuff like creating what would be a mobile OS just for fun. My father told me that in the very first computers, you would edit a specific memory address and the screen would update. I would like to simulate this inside a window in Windows. Is there any way I can do this with C#?
This used to be done because you could get direct access to the video buffer. This is typically not available with today's systems, as the video memory is managed by the video driver and OS. Further, there really isn't a 1:1 mapping of video memory buffer and what is displayed anymore. With so much memory available, it became possible to have multiple buffers and switch between them. The currently displayed buffer is called the "front buffer" and other, non-displayed buffers are called "back buffers" (for more, see https://en.wikipedia.org/wiki/Multiple_buffering). We typically write to back buffers and then have the video system update the front buffer for us. This provides smooth updates, as the video driver synchronizes the update with the scan rate of the monitor.
To write to back buffers using C#, my favorite technique is to use the WPF WritableBitmap. I've also used the System.Drawing.Bitmap to update the screen by writing pixels to it via LockBits.
It's a full featured topic that's outside the scope (it won't fit, not that i won't ramble about it for hours :-) of this answer..but this should get you started with drawing in C#
http://www.geekpedia.com/tutorial50_Drawing-with-Csharp.html
Things have come a bit from the old days of direct memory manipulation..although everything is still tied to pixels.
Edit: Oh, and if you run into flickering problems and get stuck, drop me a line and i'll send you a DoubleBuffered panel to paint with.
Related
I need a very simple video player in my C# app. It only has to loop a video from file and nothing more. Since I'm developing a WPF application, I've tried to use System.Windows.Controls.MediaElement. It has all the functions I need, but works quite poor: I've played some full HD videos on it, and it's always lagging and spiking.
To make sure, it's not my app problem, I've created 2 test applications. The first in a WinForms borderless 1920x1080 window with only AxWMPLib.AxWindowsMediaPlayer control. And the second in a borderless WPF window of the same size with System.Windows.Controls.MediaElement.
Then I run 2 videos on both of players. Here are their specs:
1: 1920x1080, 12000kb/s, 25 FPS, wmv
2: 1920x1080, 5730kb/s, 25 FPS, mp4
On AxWindowsMediaPlayer everything looks fine. But MediaElement seems to drop some frames and ignore vertical sync (it's possible to see parts of one frame on another during fast scene changes). So, it's completely unsuitable and shouldn't be like that, but I've found nothing about the problem in Microsoft official docs (they only suggest to use MediaElement instead of AxWindowsMediaPlayer in WPF apps). Is it possible to make it work more smoothly or using an additional WinForms Form with AxWindowsMediaPlayer is the only solution?
It was written over five years ago (look up James Dailey messages in the thread), there were possibly some improvements but overall I suppose the statements are still in good standing. I will pick up some relevant quotes:
As you know the WPF environment is constructed from the ground up to offer developers a very rich “graphics first” environment. The MediaElement in particular was designed to allow you to mix video with various other UI components seamlessly. This solution will give you the flicker free, “draw over video” solution that you are looking for. The best part is you can do all of this in C#. The bad part of this solution is that the MediaElement is not designed for displaying time sensitive media content. In other words, the MediaElement is notorious for dropping and delaying the display of video frames. There are ways to minimize this such as using SD rather than HD content, use a video accelerated codec, etc.
also:
Unfortunately you can’t really tell the WPF MediaElement to never drop frames. The term we use for this class of issues is “disparate clocks”. In this case WPF is updating the screen at a certain rate (clock 1). The MediaElement (based on WMP) is cranking out video frames at a slightly different rate (clock 2). Given the underlying technologies there is currently no way to synchronize the two clocks and force them to “tick” at the same rate. Since the display will only be updated according to the WPF clock, multiple frames of video may be sent from the MediaElement to WPF between clock ticks. Because of this the MediaElement may appear to drop frames. This is a very common problem in multimedia development and there is no simple solution.
Windows Media Player uses Media Foundation and DirectShow APIs which power media playback with high quality video experience.
We have a c# application that performs processing on video streams. This is a low-level application that receives each frame in Bitmap format, so basically we need 25 images each second. This application is already working for some of our media sources, but we now need to add a webcam as an input device.
So we basically need to capture bitmap images from a webcam continuously so that we can pass all these frames as a "stream" to our application.
What is the best and simplest way to access the webcam and read the actual frames directly from the webcam as individual images? I am still in the starting blocks.
There are a multitude of libraries out there that allows one to access the webcam, preview the content of the webcam on a windows panel and then use screen capturing to capture this image again. This, unfortunately, will not give us the necessary performance when capturing 25 frames per second. IVMRWindowlessControl9::GetCurrentImage has been mentioned as another alternative, but this again seems to be aimed at an infrequent snapshot rather than a constant stream of images. Directshow.Net is mentioned by many as a good candidate, but it is unclear how to simply grab the images from the webcam. Also, many sources state a concern about Microsoft no longer supporting Directshow. Also, implementations I've seen of this requires ImageGrabber which is apparently also no longer supported. The newer alternative from MS seems to be Media Foundation, but my research hasn't turned up any working examples of how this can be implemented (and I'm not sure if this will run on older versions of windows such as XP). DirectX.Capture is an awesome library (see a nice implementation) but seems to lack the filters and methods to get the video images directly. I have also started looking at Filters and Filter Graphs but this seems awfully complex and does feel a bit like "reinventing the wheel".
Overall, all the solutions briefly mentioned above seem to rather old. Can someone please point me in the direction of a step-by-step guide for getting a webcam working in C# and grabbing several images per second from it? (We will also have to do audio at some point, so a solution that does not exclude video would be most helpful).
I use AForge.Video (find it here: code.google.com/p/aforge/) because it's a very fast c# implementation. i am very pleased with the performance and it effortlessly captures from two HD webcams at 30fps on an 8 year old PC. the data is supplied as a native IntPtr so it's ideal for further processing using native code or opencv.
opencv wrappers emgu and opencvsharp both implement a rudimentary video capture functionality which might be sufficient for your purposes. clearly if you are going perform image processing / computer vision you might want to use those anyway.
As dr.mo suggests, Aforge was the answer.
I used the tutorial from here: http://en.code-bude.net/2013/01/02/how-to-easily-record-from-a-webcam-in-c/
In the tutorial, they have an event handler fire each time a frame is received from the webcam. In the original tutorial, this bitmap is used to write the image to a PictureBox. I have simply modified it to save the bitmap image to a file rather than to a picturebox. So I have replaced the following code:
pictureBoxVideo.BackgroundImage = (Bitmap)eventArgs.Frame.Clone();
with the following code:
Bitmap myImage = (Bitmap)eventArgs.Frame.Clone();
string strGrabFileName = String.Format("C:\\My_folder\\Snapshot_{0:yyyyMMdd_hhmmss.fff}.bmp", DateTime.Now);
myImage.Save(strGrabFileName, System.Drawing.Imaging.ImageFormat.Bmp);
and it works like a charm!
As you might surmise from the question title, we are required to decode and display multiple (e.g., eight) H.264 encoded videos at the same time (and keep them all time synchronized, but that's another question for another time). The videos are usually at at 25 FPS with a resolution of 640x480.
I'm going to provide a bit of background before I get to the crux of the problem.
The feature needs to be baked into a fairly large C# 3.5 (WinForms) application. The videos will occupy rectangles in the application - the managed code needs to be able to specify where each video is drawn as well as it's size.
We get the H264 packets in C# and fire them into a native H264 decoder to get YUV12 image data.
An early attempt consisted of converting the YUV12 images to RGB24 and BitBlt'ing them to a HWND passed into the native code from C#. While functional, all BitBlt'ing had to happen on the UI thread which caused it to bog down when displaying more than a couple videos (on a 2.6 GHZ core 2 duo).
The current attempt spins up one-thread-per-cpu-core on startup and load balances the decoding/displaying of videos across these threads. The performance of this is mind-blasting (I find watching task manager much more interesting than the videos being displayed). UI-wise, it leaves a lot to be desired.
The millisecond we started drawing to an HWND created on the UI thread (e.g., a panel docked in a WinForms control) from a non-UI thread, we started getting all sorts of funky behavior due to the un-thread-safeness of WinForms. This led us to create the HWND's in native code and draw to those, with C# providing the rectangles they should be drawn to in screen coordinates.
Gah! CanOfWorms.Open().
Problem: When the C# application receives focus, it jumps to the front of the Z-Order and hides the video windows.
Solution: Place the video windows Always On Top.
Problem: When the user switches to another application, the video windows are still on top.
Solution: Detect activation and deactivation of the C# application and show/hide the video windows accordingly.
Problem: User says, "I want my videos playing on one monitor while I edit a Word document in the other!"
Solution: Tell user to shut up and that Word sucks anyways.
Problem: I get fired.
etc. etc.
I guess the crux of the problem is that we have HWND's created on a non-UI thread and we want to 'simulate' those being embedded in the C# application.
Any thoughts/suggestions? Am I completely out to lunch here?
I'm more than open to taking a completely different approach if one exists (This project required a lot of learning - winning the lottery would have a greater likelihood than me having picked the best approach at every step along the road).
Forget about BitBlt-ing and do this:
for each window you want your video to be played, create one DirectShow graph and attach the renderer of the graph to that window
before renderer in the graph put the samplegrabber filter. It will allow you to have callback in which you'll be able to fill the buffer
instead of blitting, decode to the buffer provided in samplegrabber.
In addition, I guess that you'll be able to put raw YUV12 into the buffer, as VMRenderer is able to display them directly.
Use DirectShowNet library.
EDIT:
And yes, BTW, if the videos are on the same 'canvas', you can use same technique with renderer and create only one large window, then shift decoded video rectangles 'by hand' and put them into the framebuffers buffer.
YET ANOTHER EDIT:
BitBlts are ALWAYS serialized, i.e. they can't run in parallel.
The millisecond we started drawing to an HWND created on the UI thread (e.g., a panel docked in a WinForms control) from a non-UI thread, we started getting all sorts of funky behavior due to the un-thread-safeness of WinForms. This led us to create the HWND's in native code and draw to those, with C# providing the rectangles they should be drawn to in screen coordinates.
What kind of funky behavior?
If you mean flickering or drawing delay, have you tried to lock() the panel or any other class for thread/drawing synchronisation?
Again: Whats the exact problem when you send the data to the decoder, receive a image, convert it and then draw it with an OnPaint handler. (Setup a different thread that loops at 25fps, call panel1.Invalidate() then)
I guess the crux of the problem is that we have HWND's created on a non-UI thread and we want to 'simulate' those being embedded in the C# application.
Don't do that. Try to draw the received data in your c# application.
In general, I wouldn't reccomend mixing native code and c#. Having the h264 decoder in native code is the only exception here.
Use your threads to decode the video packets (as you already do) then have one thread that loops and calls Invalidate(as said above). Then have an OnPaint handler for each panel you are displaying a video in. In this handler get the most recent video picture and draw it (e.Graphics).
I hope this helps, but also need more information about the problem...
I like the DirectShow answer posted earlier, but I wanted to include an additional option that might be easier for you to implement, based on this excerpt from your question:
While functional, all BitBlt'ing had to happen on the UI thread which caused it to bog down when displaying more than a couple videos
My idea is to start from that code, and use the Async CTP for Visual Studio 2010 that is currently available and includes a go-live license. From there it should be a relatively simple to modify this existing code to be more responsive: just add await and async keywords in a few places and the rest of the code should be largely unchanged.
How is scrolling typically handled in a Windows application that has computationally expensive graphics to render? For example, if I am rendering a waveform graph of a sound, after processing the wave form from a peakfile, should I:
Render the entire graphical representation to an in-memory GDI surface, and then simply have a scrollable control change the start/end of the render area?
Render the visible portion of the wave only. In a separate thread, process any new chunks of the graphc that come into view.
Render the visible portion of the wave, plus a buffer. This way, there's less of a chance of the user seeing "blank" or "currently rendering" portions of the waveform. Still, if a user quickly scrolls to a distant area, the whole section will be blank until rendering is complete.
The problem is, many applications handle this in different ways.
For example:
Adobe Acrobat - renders blank pages during scroll unless page is in cache. Any pages that would be visible within the document render area are rendered in a separate thread and are presented opon completion.
Microsoft Word - Essentially, the same as above. Documents are separated into distinct pages, so each page is processed/rendered on an as-needed basis and added to a cache.
Internet Explorer - Unknown. It appears that an entire "webpage" is rendered in graphic memory, no matter how many "screens" worth of graphic data it consumes. Theoretically, with a web page that scrolls for 10 or 15 screen lengths, this could mean 50-60MB worth of graphic memory consumption. Could anyone with experience with WebKit or FireFox explain whether or not the rendering engine favors consuming a ton of memory, or tries to render peices of the page "on the fly" to conserve memory?
If it helps, my application is based on C#, .NET 3.5, and WinForms.
This is a complexity vs. user experience trade-off. Your third option will give you the best user experience (they can start to see things right away and start to work). It is also the most complex to code (will take the longest to develop, with the most amount of bugs to kill).
The "correct" solution depends on how "expensive" expensive is, and on the demands of your user base. I would select the option with the least complexity that will provide a user experience that will satisfy the bulk of the customers:
Make it as complex as it needs to be, but no more complex than that.
I think this is actually a memory-usage versus a processor-usage tradeoff. Your first option (rendering the entire wave on an appropriately-sized canvas, and then moving that canvas around with only a visible window portion showing) might be the best approach, assuming you have enough memory for it. After an initial rendering delay, the user experience will be smooth and seamless.
If you don't have enough memory for this, then you have to render the visible portion on the fly. I've written this application (WAV data viewer) many times, and usually GDI+ is more than fast enough to render portions (even large portions) of WAV data in realtime (with a high framerate above 30 fps, which produces perfectly smooth animation). The key to this, however, is not to render each sample value as a separate point - this would be dog slow. What you want to do is for each pixel on your X axis, scan the corresponding chunk of WAV samples to get the minimimum and maximum sample value, and then render a single line between these values.
It's hard to put this into the title, so let me explain.
I have an application that uses Direct3D to display some mesh and directshow(vmr9 + allocator) to play some video, and then send the video frame as texture to the Direct3D portion to be applied onto the mesh. The application needs to run 24/7. At least it's allowed to be restarted every 24hours but not more frequent than that.
Now the problem is that directshow seems to be giving problem after a few hours of playback, either due to the codec, video driver or video file itself. At which point the application simply refuse playing anymore video. But the Direct3D portion is still running fine, mesh still displayed. Once the application is restarted, everything back to normal.
So, I'm thinking of splitting the 2 parts into 2 different process. So that when ever the video process failed to play video, at least I could restart it immediately, without loosing the Direct3D portion.
So here comes the actual question, whether it's possible to pass the texture from the video player to the direct3d process by passing the pointer, aka retrieve the texture of another process from pointer? My initial guess is not possible due to protected memory addressing.
I have TCP communication setup on both process, and let's not worry about communicating the pointer at this point.
This might be a crazy idea, but it will work wonder of it's ever possible
Yes you can do this with Direct3D 9Ex. This only works with Vista and you must use a Direct3DDevice9Ex. You can read about sharing resources here.
Now the problem is that directshow seems to be giving problem after a few hours of playback, either due to the codec, video driver or video file itself. At which point the application simply refuse playing anymore video.
Why not just fix this bug instead?
If you separate it out as a separate process then I suspect this would not be possible, but if it were a child thread then they would have shared memory addressing I believe.
Passing textures doesn't work.
I'd do it using the following methods:
Replace the VMR with a custom renderer+allocator that places the picture into memory
You allocate memory for pictures from a shared memory pool
Once you receive another picture you signal an event
The Direct3D process waits for this event and updates the mesh with the new texture
Note you'll need to transfer the picture data to the graphics card. The big difference is that this transfer now happens in the Direct3D app and not in the DirectShow app.
You could also try to use the VMR for this. I'm not sure if the custom allocator/renderer parts will allow you to render into shared memory.
Maybe you could use the Sample Grabber in your DirectShow host process to get the image as a system memory buffer. Then you could use WriteProcessMemory to write the data into a pre-agreed address (which you setup over TCP or something) in your Direct3D app.