Develop Reference

OpenGL ES Texture streaming or mapping

Situation
I have a video stream coming from a native library in iOS. I'm trying to display the image in an iPhoneOSGameView using glTexImage2D and glTexSubImage2D for updates. I can update subregions of the image, I receive a structure that tells me which rectangle has to be updated on the gpu.
The issue
Framerate is quite low. After much profiling both in OpenGLES and the application code, I have concluded that the application usually is waiting on the texture upload. The slow function is glClear, but I suspect there's an intrinsic glFlush in there.
My question
I've seen some people talking about glMapBuffer that could allow me to stream the video directly to the texture in user-space. I've looked at pixel buffer objects, but they require OpenGLES 3.0 or an extension in 2.0. Is there an efficient way (for mobile) to stream a texture with minimal memory copying OR a way to transfer the texture from different thread?
Additional information
I'm working in C# Xamarin and I'm testing on different devices such as an iPod Touch Gen3, a iPad Air 2 and a iPad Pro 12".

there an efficient way (for mobile) to stream a texture with minimal memory copying
Most operating systems have some media framework which allows import of images directly using the EGL_image_external extension avoiding the need to upload. I'm not sure how it works on iOS, but I strongly suspect it should be possible. It's OS-specific unfortunately, so no standard way of doing it.
OR a way to transfer the texture from different thread?
Just create two GL Contexts in the same share group.
https://developer.apple.com/library/ios/documentation/3DDrawing/Conceptual/OpenGLES_ProgrammingGuide/ConcurrencyandOpenGLES/ConcurrencyandOpenGLES.html

Drawing geographic map tiles with C#

I am creating a privately consumed custom map overlay.
I cannot use an open source server like MapServer, because of the sheer volumes of data and the format that it is in.
Originally it was going to be a client-side solution that pushed an ArrayBuffer to the client and render the data on a map using WebGL, however we later found out that our users' PCs would be minus a GPU, so they cannot smoothly run the WebGL rendering.
So I took the concepts and applied them to OpenTK - I created an IIS server handler that creates an OpenTK instance, and renders a requested tile.
For prototype's sake, it works - however I feel this is not the best solution.
What is the most efficient way to render out tiles?
I would love to pre-render the tiles, but there are just too many datasets (adding 1000 more per day!) to be able to efficiently do this.
Is OpenTK a good route to go down (because of the hardware acceleration it can take advantage of?), or is there too much overhead in setting up an instance?
Or are the C# Graphics libraries a better route to learn and use?
Or even - is it worth ditching IIS and C# all together and using a different language/framework for serving the images?

Your server only has a single GPU, so launching multiple instances of OpenTK will be significantly slower than launching a single instance and queuing tiles for rendering. Context switching inside the GPU drivers hurts. The latest version of OpenTK starts up in milliseconds so that should not be a problem (but you will have to measure.)

How can I edit individual pixels in a window?

I want to create a simple video renderer to play around, and do stuff like creating what would be a mobile OS just for fun. My father told me that in the very first computers, you would edit a specific memory address and the screen would update. I would like to simulate this inside a window in Windows. Is there any way I can do this with C#?

This used to be done because you could get direct access to the video buffer. This is typically not available with today's systems, as the video memory is managed by the video driver and OS. Further, there really isn't a 1:1 mapping of video memory buffer and what is displayed anymore. With so much memory available, it became possible to have multiple buffers and switch between them. The currently displayed buffer is called the "front buffer" and other, non-displayed buffers are called "back buffers" (for more, see https://en.wikipedia.org/wiki/Multiple_buffering). We typically write to back buffers and then have the video system update the front buffer for us. This provides smooth updates, as the video driver synchronizes the update with the scan rate of the monitor.
To write to back buffers using C#, my favorite technique is to use the WPF WritableBitmap. I've also used the System.Drawing.Bitmap to update the screen by writing pixels to it via LockBits.

It's a full featured topic that's outside the scope (it won't fit, not that i won't ramble about it for hours :-) of this answer..but this should get you started with drawing in C#
http://www.geekpedia.com/tutorial50_Drawing-with-Csharp.html
Things have come a bit from the old days of direct memory manipulation..although everything is still tied to pixels.
Edit: Oh, and if you run into flickering problems and get stuck, drop me a line and i'll send you a DoubleBuffered panel to paint with.

Developing a Video Chat Application with high quality video streaming

I am working for a company where we are developing video chat support on an existing application. I have looked at various solutions for this like
Using Managed Direct show for video capture and streaming in C#
Some code samples in code project where we take an image and pass it over the network (I would call it rather a crude solution as this would eat up lot of bandwidth.
Code a compression algorithm from scratch from scratch and use it to compress-decompress video.
Now the challenge is that we are looking to achieve very high quality video streaming and the container application is coded in C#.NET
This is what I have proposed so far. The network logic to stream data is written in C# , the video compression to be written in VC++ and call this VC++ dll using pinvoke or either CLI which way possible.
I am looking for some one more experienced that me in this field who can suggest me if Iam going correct or can this be still improved.
The ultimate goal is high quality video streaming.
The codec can be any anything like h.2633, h.264 etc.

I've used several ways to get video streaming/conferencing with .net easily, without need to dig into directshow. (ok, dig some, but not deep :)
1) Use of plain Windows Media Encoder components. It is documented with samples in Windows Media Encoder SDK. Good for any high resolution streaming, but delay is too big for realtime chat (0.5-2 seconds at best). Modern Express Encoder SDK another option.
2) Microsoft Research ConferenceXP http://cct.cs.washington.edu/ Full featured conferencing API including application streaming. They too low level Windows Media coded filters and wrapped them into managed code. Works well. Easily customizable. Looks bit abandoned now.
3) Microsoft RTC Client up to version 1.3 - core of windows messenger.
pros: managed samples from Microsoft, good docs, reliable performance, freely redistributable, microsoft compatible (good) SIP stack included. Major conferencing vendors like Emblaze VCON based their solutions on it in some near past, not sure about this days, but I know that Tandberg licensed Microsft's VC-1.
cons: version up to 1.3 support h261-h263 video only. modern version with support of VC-1(h264) codec does not allow direct serverless ip-ip connections. It does at require Microsoft Live Communications server. Newer version SDK does not cover well video conferencing calls.
http://msdn.microsoft.com/en-us/library/ms775892(VS.85).aspx
Please let us know what platform you have chosen. By the way, I've even used ConferenceXP video rtp part with RTC 1.3 voice/SIP features together to improve video quality, so you have wide choice of managed technologies here. Another thing is Live Meeting at which I had no chance to take good look yet.

Save yourself the trouble and use VLC. There are some decent .NET wrappers for it (http://forum.videolan.org/viewtopic.php?f=32&t=52021&start=30)
We are using C# and VLC for an IPTV network. We take input off DISH network satellites via Osprey-450 video capture devices on a Windows XP server. From there, we have a .NET server component that we wrote in C# that uses VLC behind the scenes (starting separate processes in .NET to control the vlc.exe instances). The VLC processes transcode and stream the signals over a network (.h264 or MPEG-4, we've successfully done both).
On the client side we have a C# WinForm application that uses an embedded VLC Viewer to view multicast signals. This application is mainly for command & control. The real use of the multicast signals happens when our set top boxes attached to our TV's decode and display the streams.
We thought we were going to have to write our own DirectX encoders too, but don't go to all the trouble. VLC works really well and has enough C# support to be very useful. Feel free to e-mail me if you have specific questions about implementation.

You should check out the Ucentrik SDK. This SDK will enable you to integrate rich-media functionality such as video, audio, chat, remote-desktop sharing and control and video recording in your applications. The video codecs supported are VP8 (Google), Theora, and x.264. Additionally, the rich media traffic is encapsulated within an HTTP protocol to enable it to traverse firewalls that enable normal web traffic. This technology is completely free and you can download the SDK and request an API key so that you can evaluate without investing any time on setting up the infrastructure. In the next few months, we are releasing a server component so that you can download and host the infrastructure your self or your customer. The technology supports 1-to-many connections which means that you can create video conferences if you like. The features are highly modular so that you can integrate just the video or audio or desktop share or a combination of the technologies using the same SDK... You should request for an SDK available here: www.ucentrik.com. Additionally, there are some video's here: http://www.youtube.com/user/ucentrik
good luck.

Ucentrik has just released an open-source call-center application that integrates the CTX technology. This call-center application implements the video, audio, desktop/application sharing (with control), text chat functionality available from the CTX API. The application also includes some business logic specifically around providing the ability to route the calls to an agent that is available or have specific skill-set. The project is available at http://vcca.codeplex.com - please note that you will require access to the Ucentrik CTX SDK which is available for request on the Ucentrik website (www.ucentrik.com). Good luck.

Pass texture using pointer across process

It's hard to put this into the title, so let me explain.
I have an application that uses Direct3D to display some mesh and directshow(vmr9 + allocator) to play some video, and then send the video frame as texture to the Direct3D portion to be applied onto the mesh. The application needs to run 24/7. At least it's allowed to be restarted every 24hours but not more frequent than that.
Now the problem is that directshow seems to be giving problem after a few hours of playback, either due to the codec, video driver or video file itself. At which point the application simply refuse playing anymore video. But the Direct3D portion is still running fine, mesh still displayed. Once the application is restarted, everything back to normal.
So, I'm thinking of splitting the 2 parts into 2 different process. So that when ever the video process failed to play video, at least I could restart it immediately, without loosing the Direct3D portion.
So here comes the actual question, whether it's possible to pass the texture from the video player to the direct3d process by passing the pointer, aka retrieve the texture of another process from pointer? My initial guess is not possible due to protected memory addressing.
I have TCP communication setup on both process, and let's not worry about communicating the pointer at this point.
This might be a crazy idea, but it will work wonder of it's ever possible

Yes you can do this with Direct3D 9Ex. This only works with Vista and you must use a Direct3DDevice9Ex. You can read about sharing resources here.

Now the problem is that directshow seems to be giving problem after a few hours of playback, either due to the codec, video driver or video file itself. At which point the application simply refuse playing anymore video.
Why not just fix this bug instead?

If you separate it out as a separate process then I suspect this would not be possible, but if it were a child thread then they would have shared memory addressing I believe.

Passing textures doesn't work.
I'd do it using the following methods:
Replace the VMR with a custom renderer+allocator that places the picture into memory
You allocate memory for pictures from a shared memory pool
Once you receive another picture you signal an event
The Direct3D process waits for this event and updates the mesh with the new texture
Note you'll need to transfer the picture data to the graphics card. The big difference is that this transfer now happens in the Direct3D app and not in the DirectShow app.
You could also try to use the VMR for this. I'm not sure if the custom allocator/renderer parts will allow you to render into shared memory.

Maybe you could use the Sample Grabber in your DirectShow host process to get the image as a system memory buffer. Then you could use WriteProcessMemory to write the data into a pre-agreed address (which you setup over TCP or something) in your Direct3D app.