I'm building a C# application where 2 or more cameras are connected to a processing module that has one or more outputs. I need to be able to connect "monitor" windows to preview each camera and the processed output that can be hidden or shown independently, with additional processing filters added to the stream while the video program is running.
Conceptually, I'm trying to build something that looks like this:
(source: fkeinternet.net)
(Using the Video Mixing Filter from the Video Processing Project, I can actually build the above graph and have it run with the three video renderers displaying their respective video streams - in ActiveMovie windows, not C# form windows. Building a graph is not exactly the problem, building the complete application is the issue.)
Building on example code from the DirectShow .NET project, combined with code generated by GraphEditPlus, I can build a basic application with the video stream from a single camera displayed in a C# form window. I'm in the process of debugging multiple preview windows operating simultaneously, but I've realized there are other issues:
One of the problems with the graph illustrated above is that if any of the output windows are closed, the whole graph stops. Another is that it doesn't allow adding filters in the processing stage without stopping the whole graph and rebuilding it.
My idea is to break the monolithic graph into separate source, processing and display graphs so that each piece can be started or stopped as needed, something like this:
(source: fkeinternet.net)
I'm assuming I'd have to keep one graph running all the time to provide a "master clock" source for everything else (probably the "Processing Graph" component), but I'm not quite sure how to do that.
Is there a "standard" way for connecting multiple graphs together? For that matter, is it even possible? I've done a number of searches along the lines of "c# directshow connect two graphs" but all of the links returned are related to connecting filters together, not graphs. Am I asking the wrong question?
I stumbled across this post which pointed me to the answer I was looking for - GMFBridge
Related
I am making a C# app to capture graphic from an avermedia pcie capture card.
But it seems that there are no out of box tools to do so.
So I made a C++ directshow app to do the capture, which is a console app and opens a capture window when running.
How can I redirect the output to a C# app? for example, to a CaptureElement?
So you want to have XAML CaptureElement connected to AverMedia PCIe capture card. This sounds like a well-understood challenge overall, however every other piece of technology you mentioned is eventually a bad choice: DirectShow, multiple apps with piping, redirection and fitting of cutsom code to XAML CaptureElement control.
Microsoft has intentionally been limited ways you can integrate different APIs and so there are not so many ways to get everything together.
Let us go over the supposed integration path. The capture card is supposed to be shipped with a compatible driver:
Video capture devices are supported through the UVC class driver and must be compatible with UVC 1.1
When this is the case, such devices are visible to Media Foundation API handling video capture among tasks. XAML CaptureElement would be able to see a video capture device through this API and this way everything is supposed to work without need to fit anything from your end.
If this is not happening, it suggests you are dealing with an unsupported device coming without suitable or compatible driver.
Previous media API in Windows was DirectShow but its days are gone. It remains perfectly working as a legacy framework, a lot of applications out there are still relying on it. Specifically it will not integrate with new technology like XAML and UWP. More to that, even Media Foundation itself, the current media API, in its public offering is lagging behind when it comes to fitting with most recent technology. Having said that it is a good idea to stay well clear of DirectShow here if this is at all possible.
I see no need for cross-process design with video travelling between process through piping. There is no good reason for such design and even though this can work efficiently (Windows itself proves it can work great in terms of performance by having so called Frame Server service in it), this is not to be built on piping. In your case it is unlikely to be have to be built on multiple processes either. Instead you can develop a native code DLL project that takes care of video acquisition and connects to managed code via suitable glue layer: C++/CLI, COM, C++/WinRT and such.
Then next thing is fitting to XAML CaptureElement. The control is designed to work with Windows.Media.Capture.MediaCapture class that talks to hardware and you don't have suitable hardware as you plan to implement your own acquisition layer. Long story short you are not supposed to forward external data to CaptureElement and you would have hard time doing this. Your best strategy is to upload externally obtained data to Windows.Graphics.Imaging.SoftwareBitmap or alike and take involved performance impact as acceptable. That is, you will be dealing with video frames as images.
An alternative way is to upload acquired video frames into Direct 3D 11 textures and it would open you a more performant way of integration with video related controls, such as Windows.UI.Xaml.Controls.SwapChainPanel however it would also require that you put much more development effort in there.
I have problem with handling multiple video windows in directshow.
I have app in c# that captures video stream of my webcam and sends it through infinite pin tree to 2 separate renderers. When I run the app each of the renderers gets its own VideoWindow. Now when I set VideoWindow Style I can see changes only on one of them (looks like it is 2nd renderer window). Secound window is not effected at all. What should I do to access both of them? There is only one graph to build entire thing. Is there a way to reference videowindow by the renderer not by the graph? I was looking around web but there is nothing thats was able to point me in to good direction.
I've different angle 2D images(images of one particular object), I want to merge all that three images and create one 3D Image.
How can I do that in Unity 3D?
The process you're looking for is called Photogrammetry. To get good quality models, you'll want more than 3 images. This process is very CPU intensive and takes a long time, you won't want this built into your Unity3D Game/App. You'll want to generate the models with external tools, then import them.
If you actually are looking for a C# way to do this, I only know of one opensource library working on this. Epicycle.Photogrammetry-cs.
If you want to manage the whole process of the conversion, check out this tutorial using only open source software.
This YouTube tutorial is easy to follow (I've done it myself), and also uses only free software.
If you're looking for a more polished, less technical solution, try ReMake by Autodesk
If you're looking for a completely hands off process, try 123D Catch by Autodesk
As you might surmise from the question title, we are required to decode and display multiple (e.g., eight) H.264 encoded videos at the same time (and keep them all time synchronized, but that's another question for another time). The videos are usually at at 25 FPS with a resolution of 640x480.
I'm going to provide a bit of background before I get to the crux of the problem.
The feature needs to be baked into a fairly large C# 3.5 (WinForms) application. The videos will occupy rectangles in the application - the managed code needs to be able to specify where each video is drawn as well as it's size.
We get the H264 packets in C# and fire them into a native H264 decoder to get YUV12 image data.
An early attempt consisted of converting the YUV12 images to RGB24 and BitBlt'ing them to a HWND passed into the native code from C#. While functional, all BitBlt'ing had to happen on the UI thread which caused it to bog down when displaying more than a couple videos (on a 2.6 GHZ core 2 duo).
The current attempt spins up one-thread-per-cpu-core on startup and load balances the decoding/displaying of videos across these threads. The performance of this is mind-blasting (I find watching task manager much more interesting than the videos being displayed). UI-wise, it leaves a lot to be desired.
The millisecond we started drawing to an HWND created on the UI thread (e.g., a panel docked in a WinForms control) from a non-UI thread, we started getting all sorts of funky behavior due to the un-thread-safeness of WinForms. This led us to create the HWND's in native code and draw to those, with C# providing the rectangles they should be drawn to in screen coordinates.
Gah! CanOfWorms.Open().
Problem: When the C# application receives focus, it jumps to the front of the Z-Order and hides the video windows.
Solution: Place the video windows Always On Top.
Problem: When the user switches to another application, the video windows are still on top.
Solution: Detect activation and deactivation of the C# application and show/hide the video windows accordingly.
Problem: User says, "I want my videos playing on one monitor while I edit a Word document in the other!"
Solution: Tell user to shut up and that Word sucks anyways.
Problem: I get fired.
etc. etc.
I guess the crux of the problem is that we have HWND's created on a non-UI thread and we want to 'simulate' those being embedded in the C# application.
Any thoughts/suggestions? Am I completely out to lunch here?
I'm more than open to taking a completely different approach if one exists (This project required a lot of learning - winning the lottery would have a greater likelihood than me having picked the best approach at every step along the road).
Forget about BitBlt-ing and do this:
for each window you want your video to be played, create one DirectShow graph and attach the renderer of the graph to that window
before renderer in the graph put the samplegrabber filter. It will allow you to have callback in which you'll be able to fill the buffer
instead of blitting, decode to the buffer provided in samplegrabber.
In addition, I guess that you'll be able to put raw YUV12 into the buffer, as VMRenderer is able to display them directly.
Use DirectShowNet library.
EDIT:
And yes, BTW, if the videos are on the same 'canvas', you can use same technique with renderer and create only one large window, then shift decoded video rectangles 'by hand' and put them into the framebuffers buffer.
YET ANOTHER EDIT:
BitBlts are ALWAYS serialized, i.e. they can't run in parallel.
The millisecond we started drawing to an HWND created on the UI thread (e.g., a panel docked in a WinForms control) from a non-UI thread, we started getting all sorts of funky behavior due to the un-thread-safeness of WinForms. This led us to create the HWND's in native code and draw to those, with C# providing the rectangles they should be drawn to in screen coordinates.
What kind of funky behavior?
If you mean flickering or drawing delay, have you tried to lock() the panel or any other class for thread/drawing synchronisation?
Again: Whats the exact problem when you send the data to the decoder, receive a image, convert it and then draw it with an OnPaint handler. (Setup a different thread that loops at 25fps, call panel1.Invalidate() then)
I guess the crux of the problem is that we have HWND's created on a non-UI thread and we want to 'simulate' those being embedded in the C# application.
Don't do that. Try to draw the received data in your c# application.
In general, I wouldn't reccomend mixing native code and c#. Having the h264 decoder in native code is the only exception here.
Use your threads to decode the video packets (as you already do) then have one thread that loops and calls Invalidate(as said above). Then have an OnPaint handler for each panel you are displaying a video in. In this handler get the most recent video picture and draw it (e.Graphics).
I hope this helps, but also need more information about the problem...
I like the DirectShow answer posted earlier, but I wanted to include an additional option that might be easier for you to implement, based on this excerpt from your question:
While functional, all BitBlt'ing had to happen on the UI thread which caused it to bog down when displaying more than a couple videos
My idea is to start from that code, and use the Async CTP for Visual Studio 2010 that is currently available and includes a go-live license. From there it should be a relatively simple to modify this existing code to be more responsive: just add await and async keywords in a few places and the rest of the code should be largely unchanged.
I would like to have 2 video windows playing a (same file for now) video. As I'm still new to c# and DirectShow I'm having problems with this and haven't found any working examples for multi-video solutions. I can get it to work for 1 window from samples, and would like to know what else is required to get the 2nd window working. Do I need to create a separate filter for the 2nd one aswell, or just fiddling around with the handles is enough?
What problems are you facing for multi-video rendering ?
As far as I know, you need to create one graph per video stream you want to have rendered, i.e. one Source/(Transform)/Render chain per stream. I don't think using a single render filter and playing with the windows handles is going to work (but I may be mistaken).
If you need to play the same video file in two different windows, just connect a Smart Tee filter after your source filter (or after your decompression filters), and connect a render filter to each of the Smart Tee's output pins.
If you want an easy method to test Directshow graphs, use GraphEdit (available in the DirectX SDK and in many other places on the Internet).