It's hard to put this into the title, so let me explain.
I have an application that uses Direct3D to display some mesh and directshow(vmr9 + allocator) to play some video, and then send the video frame as texture to the Direct3D portion to be applied onto the mesh. The application needs to run 24/7. At least it's allowed to be restarted every 24hours but not more frequent than that.
Now the problem is that directshow seems to be giving problem after a few hours of playback, either due to the codec, video driver or video file itself. At which point the application simply refuse playing anymore video. But the Direct3D portion is still running fine, mesh still displayed. Once the application is restarted, everything back to normal.
So, I'm thinking of splitting the 2 parts into 2 different process. So that when ever the video process failed to play video, at least I could restart it immediately, without loosing the Direct3D portion.
So here comes the actual question, whether it's possible to pass the texture from the video player to the direct3d process by passing the pointer, aka retrieve the texture of another process from pointer? My initial guess is not possible due to protected memory addressing.
I have TCP communication setup on both process, and let's not worry about communicating the pointer at this point.
This might be a crazy idea, but it will work wonder of it's ever possible
Yes you can do this with Direct3D 9Ex. This only works with Vista and you must use a Direct3DDevice9Ex. You can read about sharing resources here.
Now the problem is that directshow seems to be giving problem after a few hours of playback, either due to the codec, video driver or video file itself. At which point the application simply refuse playing anymore video.
Why not just fix this bug instead?
If you separate it out as a separate process then I suspect this would not be possible, but if it were a child thread then they would have shared memory addressing I believe.
Passing textures doesn't work.
I'd do it using the following methods:
Replace the VMR with a custom renderer+allocator that places the picture into memory
You allocate memory for pictures from a shared memory pool
Once you receive another picture you signal an event
The Direct3D process waits for this event and updates the mesh with the new texture
Note you'll need to transfer the picture data to the graphics card. The big difference is that this transfer now happens in the Direct3D app and not in the DirectShow app.
You could also try to use the VMR for this. I'm not sure if the custom allocator/renderer parts will allow you to render into shared memory.
Maybe you could use the Sample Grabber in your DirectShow host process to get the image as a system memory buffer. Then you could use WriteProcessMemory to write the data into a pre-agreed address (which you setup over TCP or something) in your Direct3D app.
Related
Situation
I have a video stream coming from a native library in iOS. I'm trying to display the image in an iPhoneOSGameView using glTexImage2D and glTexSubImage2D for updates. I can update subregions of the image, I receive a structure that tells me which rectangle has to be updated on the gpu.
The issue
Framerate is quite low. After much profiling both in OpenGLES and the application code, I have concluded that the application usually is waiting on the texture upload. The slow function is glClear, but I suspect there's an intrinsic glFlush in there.
My question
I've seen some people talking about glMapBuffer that could allow me to stream the video directly to the texture in user-space. I've looked at pixel buffer objects, but they require OpenGLES 3.0 or an extension in 2.0. Is there an efficient way (for mobile) to stream a texture with minimal memory copying OR a way to transfer the texture from different thread?
Additional information
I'm working in C# Xamarin and I'm testing on different devices such as an iPod Touch Gen3, a iPad Air 2 and a iPad Pro 12".
there an efficient way (for mobile) to stream a texture with minimal memory copying
Most operating systems have some media framework which allows import of images directly using the EGL_image_external extension avoiding the need to upload. I'm not sure how it works on iOS, but I strongly suspect it should be possible. It's OS-specific unfortunately, so no standard way of doing it.
OR a way to transfer the texture from different thread?
Just create two GL Contexts in the same share group.
https://developer.apple.com/library/ios/documentation/3DDrawing/Conceptual/OpenGLES_ProgrammingGuide/ConcurrencyandOpenGLES/ConcurrencyandOpenGLES.html
I need to be able to record video from an external camera in a C# application.
Unfortunately a webcam is pretty much out of the question as the application will record outside and during the evening/night. That is why I was thinking of a camcorder since it also has manual control over exposure and focus, lower noise and better sensor.
So far I would use the AV/S-Video output from the camcorder and send the signal to a USB capture card (the computer is a laptop so no PCI-E cards).
How would I be able to access the video stream from the C# application, now that it comes from the capture card ?
Does my proposed system seem feasible (achievable, good video quality, good fps)? Does anybody have another working solution?
Thanks
This Code Project Article could be of a good starting point.
The Author mentions :
The main goal of the application was to make it flexible and
extensible. The application itself can communicate with any video
source – it may be an IP video camera or a server, it may be a local
camera attached to USB, it may be an MMS stream from a remote server,
or it may be any other video source. And more of it, the application
can work with all these video sources simultaneously, displaying them
all on a single screen.
The solution I used in the end was Microsoft Expression Encoder.
I'm working on a quite big application that is in charge of doing real-time motion tracking and camera movement controlling. Its tasks are:
Motion tracking (done by a native module, which decodes a video stream from a network camera and supplies both image buffers as big as 1280x720 pixels and the tracking results to the managed application via callbacks)
Receiving positioning feedback data from and sending movement data to a pan/tilt hardware about 20 times a second as well as zoom commands from/to the camera
Displaying the image data including live visualizations
Encoding and writing of the image and session data
Automatic postprocessing of the video is done by another process
The application uses .NET 4.0 and has a WPF user interface.
Managed threads freezing
From the beginning we had to face managed threads that freeze for between 500 to 1500ms, which is really much for a real-time application like this.
To find out when these hangs occur I created a thread whose only task is to do a sleep for 100ms all the time. I then calculated how long the sleep really took and got exactly the times when the camera movement stopped. It works very reliably, the threads all hang at the same time!
Unmanaged threads don't freeze
While all the managed threads freeze the unmanaged threads work without any problem. We check that by logs that are written independent from the managed part of the application.
Analysis
I tried to figure out with phenomenons could maybe cause this behaviour:
The whole machine slows down when we encounter these problems: Windows is responding very slowly (e.g. directory listings hang for half a minute in both Windows Explorer and my application, or launching applications takes incredibly long)
We read and write thousands of files at the same time (the tracking and postprocessing application), maybe this overcharges windows
The response of the GUI becomes very slow
The app uses about 1.3GB of virtual memory (according to Process.VirtualMemory64) / 500MB of working set memory (Process.WorkingSet64) - could it be that some of that is swapped onto the harddisk? (How can that be checked or solved?)
Of course if we kill the process Windows is responding fast again, but it takes Windows a while to be responding normally again
Hints on how to investigate into this would be highly appreciated. Thank you very much!
Maybe GC is working?
See the great article: http://samsaffron.com/archive/2011/10/28/in-managed-code-we-trust-our-recent-battles-with-the-net-garbage-collector
I would suggest taking a process dump as you are encountering the performance problem. You can do this several ways (taskmgr.exe or procdump.exe from SysInternals). Take a full memory dump.
Once you have the .dmp file, you can analyze it with windbg (or Visual Studio 2010). For managed processes you need to load the sos.dll extensions.
There are a lot of good windbg resources out there, but here are a few that have helped me:
1) Tess Fernandez video (ASP.NET process, but the techniques are the same)
2) WinDbg cheatsheet
The memory analysis will be able to give you the stack (!clrstack) while you are encountering the problem and tell you the exact culprit.
As you might surmise from the question title, we are required to decode and display multiple (e.g., eight) H.264 encoded videos at the same time (and keep them all time synchronized, but that's another question for another time). The videos are usually at at 25 FPS with a resolution of 640x480.
I'm going to provide a bit of background before I get to the crux of the problem.
The feature needs to be baked into a fairly large C# 3.5 (WinForms) application. The videos will occupy rectangles in the application - the managed code needs to be able to specify where each video is drawn as well as it's size.
We get the H264 packets in C# and fire them into a native H264 decoder to get YUV12 image data.
An early attempt consisted of converting the YUV12 images to RGB24 and BitBlt'ing them to a HWND passed into the native code from C#. While functional, all BitBlt'ing had to happen on the UI thread which caused it to bog down when displaying more than a couple videos (on a 2.6 GHZ core 2 duo).
The current attempt spins up one-thread-per-cpu-core on startup and load balances the decoding/displaying of videos across these threads. The performance of this is mind-blasting (I find watching task manager much more interesting than the videos being displayed). UI-wise, it leaves a lot to be desired.
The millisecond we started drawing to an HWND created on the UI thread (e.g., a panel docked in a WinForms control) from a non-UI thread, we started getting all sorts of funky behavior due to the un-thread-safeness of WinForms. This led us to create the HWND's in native code and draw to those, with C# providing the rectangles they should be drawn to in screen coordinates.
Gah! CanOfWorms.Open().
Problem: When the C# application receives focus, it jumps to the front of the Z-Order and hides the video windows.
Solution: Place the video windows Always On Top.
Problem: When the user switches to another application, the video windows are still on top.
Solution: Detect activation and deactivation of the C# application and show/hide the video windows accordingly.
Problem: User says, "I want my videos playing on one monitor while I edit a Word document in the other!"
Solution: Tell user to shut up and that Word sucks anyways.
Problem: I get fired.
etc. etc.
I guess the crux of the problem is that we have HWND's created on a non-UI thread and we want to 'simulate' those being embedded in the C# application.
Any thoughts/suggestions? Am I completely out to lunch here?
I'm more than open to taking a completely different approach if one exists (This project required a lot of learning - winning the lottery would have a greater likelihood than me having picked the best approach at every step along the road).
Forget about BitBlt-ing and do this:
for each window you want your video to be played, create one DirectShow graph and attach the renderer of the graph to that window
before renderer in the graph put the samplegrabber filter. It will allow you to have callback in which you'll be able to fill the buffer
instead of blitting, decode to the buffer provided in samplegrabber.
In addition, I guess that you'll be able to put raw YUV12 into the buffer, as VMRenderer is able to display them directly.
Use DirectShowNet library.
EDIT:
And yes, BTW, if the videos are on the same 'canvas', you can use same technique with renderer and create only one large window, then shift decoded video rectangles 'by hand' and put them into the framebuffers buffer.
YET ANOTHER EDIT:
BitBlts are ALWAYS serialized, i.e. they can't run in parallel.
The millisecond we started drawing to an HWND created on the UI thread (e.g., a panel docked in a WinForms control) from a non-UI thread, we started getting all sorts of funky behavior due to the un-thread-safeness of WinForms. This led us to create the HWND's in native code and draw to those, with C# providing the rectangles they should be drawn to in screen coordinates.
What kind of funky behavior?
If you mean flickering or drawing delay, have you tried to lock() the panel or any other class for thread/drawing synchronisation?
Again: Whats the exact problem when you send the data to the decoder, receive a image, convert it and then draw it with an OnPaint handler. (Setup a different thread that loops at 25fps, call panel1.Invalidate() then)
I guess the crux of the problem is that we have HWND's created on a non-UI thread and we want to 'simulate' those being embedded in the C# application.
Don't do that. Try to draw the received data in your c# application.
In general, I wouldn't reccomend mixing native code and c#. Having the h264 decoder in native code is the only exception here.
Use your threads to decode the video packets (as you already do) then have one thread that loops and calls Invalidate(as said above). Then have an OnPaint handler for each panel you are displaying a video in. In this handler get the most recent video picture and draw it (e.Graphics).
I hope this helps, but also need more information about the problem...
I like the DirectShow answer posted earlier, but I wanted to include an additional option that might be easier for you to implement, based on this excerpt from your question:
While functional, all BitBlt'ing had to happen on the UI thread which caused it to bog down when displaying more than a couple videos
My idea is to start from that code, and use the Async CTP for Visual Studio 2010 that is currently available and includes a go-live license. From there it should be a relatively simple to modify this existing code to be more responsive: just add await and async keywords in a few places and the rest of the code should be largely unchanged.
I want to create a simple video renderer to play around, and do stuff like creating what would be a mobile OS just for fun. My father told me that in the very first computers, you would edit a specific memory address and the screen would update. I would like to simulate this inside a window in Windows. Is there any way I can do this with C#?
This used to be done because you could get direct access to the video buffer. This is typically not available with today's systems, as the video memory is managed by the video driver and OS. Further, there really isn't a 1:1 mapping of video memory buffer and what is displayed anymore. With so much memory available, it became possible to have multiple buffers and switch between them. The currently displayed buffer is called the "front buffer" and other, non-displayed buffers are called "back buffers" (for more, see https://en.wikipedia.org/wiki/Multiple_buffering). We typically write to back buffers and then have the video system update the front buffer for us. This provides smooth updates, as the video driver synchronizes the update with the scan rate of the monitor.
To write to back buffers using C#, my favorite technique is to use the WPF WritableBitmap. I've also used the System.Drawing.Bitmap to update the screen by writing pixels to it via LockBits.
It's a full featured topic that's outside the scope (it won't fit, not that i won't ramble about it for hours :-) of this answer..but this should get you started with drawing in C#
http://www.geekpedia.com/tutorial50_Drawing-with-Csharp.html
Things have come a bit from the old days of direct memory manipulation..although everything is still tied to pixels.
Edit: Oh, and if you run into flickering problems and get stuck, drop me a line and i'll send you a DoubleBuffered panel to paint with.