Develop Reference

Generate a video stream from Basler camera grabbing loop

I am a beginner in video streaming and handling.
I need to take the stream from a Basler GiGE camera and display it in a web container.
I am using the Basler.Pylon C# API to access the camera and grab images one by one. from the IGrabResult object returned, I can access various parameters such as width, height, stride, and of course the byte buffer.
On my PC, I can easily display that in an image window, but what do i need to do to display that in an ASP.NET web application?
EDIT
I am not looking for code but more for guidelines, if someone could explain how video streaming works in general, that would work too

Video streaming is quite specialised and in general I would say if you want to stream high quality video over the internet to multiple end suers, its easiest to use a dedicated video streaming server rather than try to build one yourself.
Dedicated video streaming servers can be provided via a hosted service (e.g Vimeo), be a commercial server you install and run (e.g. Wowza) or a freeware streaming server you install and rune (e.g. GStreamer), so you have options there.
A a general rule, the streaming server will break your video into chunks and create multiple bit rate copies of your video. This allows a client to use Adaptive Bit Rate streaming (ABR) and download your video chunk by chunk, selecting the bit rate version for the next chunk depending on the current device and network conditions. HLS and MPEG-DASH are exmaples of ABR streaming protocols.
On a web page you then need a HTNML5 player which can understand this streaming protocol - again there are many examples such as the freeware Shaka and Dash.js players. Integrating these into a web page is very straightforward.
You can observe these in use on services like Netflix and YouTube which will often start at a lower bit rate to ensure a fast start and then 'step up' to higher bit rates until the optimal one for the current network conditions and device is reached. See here for some info on how you can see a graph of this when watching YouTube, for example:
https://stackoverflow.com/a/42365034/334402
Having said all the above, it is worth nothing that your case seems to be dealing with a stream of still images. Although all video is really a stream of still images under the covers, it may be that your image is changing infrequently, and hence you may not need some of the techniques above - a lot of the technology of video streaming is to deal with the very large amount of data streaming 30 or 60 high quality frames per second from a server to a client.
If your stream of images, for example, was one every 30 seconds then it may, as Nisus says, be just as easy to simply display the images in your web page and have the web page or app poll the server every 30 seconds (using ASP.NET AJAX in your case) to download the new image.

You have at least two options - first one is producing series of jpeg-images every few seconds and show them one by one on client using tag and simple javascript code. Second option is to generate and stream mp4-video and show it on client with some COM-windows media player or html5 control.

Show webcam stream with delay

I've been trying to find a way to show the stream of my webcam with a 30 second delay.
I found out that VLC is able to do this, but the problem is that the framerate is much too low and it's not smooth or viewable at all. I have to run this on a fairly old laptop with a better webcam that I own now, so I guess it's not an option.
I am able somewhat familiar with c#/c++ and python so I thought that I might make a solution of my own as the task seems fairly easy. Though, the problem is, I don't know where to start and any nudges in the right direction would be much appreciated.
My initial idea was to record first 30 seconds of the stream to the disk, then use VLC to view partial file (AFAIK it's able to do that). Is it an idea worth working on or should I scratch it and use some sort of a buffer for the video frames in the last 30 seconds?
Again, any nudges in the right direction will be much appreciated, thanks!

Take a look at OpenCV.
It can retrieve and show images from a webcam.
A lot a of tutorials are also available; e.g. http://opencv.willowgarage.com/wiki/CameraCapture
So simply create an array (or whatever) to hold the amount of pictures to be expected in 30 sec (FPS*30).
Start to retrieve images and as soon as the array is filled start to play from position 0.
Newly retrieved images can than be stored at the position from the "just" shown image.

Miguel Grinberg has written an excellent video streaming tutorial in Python which sends JPEG frames successively. Check his blog post here:
http://blog.miguelgrinberg.com/post/video-streaming-with-flask1
Each of these JPEG can be quickly reviewed and then broadcasted. [To take the required Delay in consideration]
As far as getting the input video feed is concerned, you can interface a webcam using OPENCV. OpenCV uses VideoCapture which returns raw images as bytes. These bytes needs to be encoded to JPEG and interfaced with Miguel's code.
import cv2
class VideoCamera(object):
def __init__(self):
# Using OpenCV to capture from device
self.video = cv2.VideoCapture(0)
def __del__(self):
self.video.release()
def get_frame(self):
success, image = self.video.read()
# We are using Motion JPEG, but OpenCV defaults to capture raw images,
# so we must encode it into JPEG in order to correctly display the
# video stream.
ret, jpeg = cv2.imencode('.jpg', image)
return jpeg.tobytes()
This approach will help you cater all the required features:
No internet required
Adjustable delay - Easily control the delay and the processing you want to perform on each frame
Good Quality
Record on Demand - Store the captured frames as per need
Have a record back feature, by just saving the previous 24*x frames (24fps stream)

How to save all depth frames in OpenNI kinect

I've wrote a program to save all the depth frames of the Kinect depth images in OpenNI. I've used the simple viewer sample. The problem that not all the frames are saved!! I run my program for 10 sec and around 20 images only are saved although the application is set with 30fps!!
Could anyone please advise?

My colleague uses a 2-phase extraction. First write the images in binary, in order to avoid losing time during encoding or conversions. (You can use System.IO.FileStream and BinaryWriter for that). And then in another program, read the binary files to get raw depth or color images. You might make use of Matlab, OpenCV or another utility for this second part.
But keep in mind that, even this approach might cause some skipped/dropped frames. Personally, I've never achieved to obtain a constant 30 fps for a long period.

Capturing Desktop activities as recorded movie in C#

My requirement is to create an application that records desktop activities, along with audio, as a movie. After searching, I found that Microsoft Expression Encoder can be used to record desktop activities as a movie but the output file size is very large. For 10 seconds of video, it occupied around 30 to 40 MB. Also, it uses xesc format.
Is there is any other free API available to do this job?

Before you give up on Expression Encoder try adjusting:
ScreenCaptureJob.ScreenCaptureVideoProfile.Quality
Reducing the quality can greatly reduce the file size. Try it and see if the results are acceptable for you.
Reducing the framerate is actually unhelpful; I guess it forces a fixed framerate, whereas the default is to use a variable framerate based on activity.
If you don't like .xesc files you can transcode the video after you've captured it.
But 30 to 40MB for ten seconds is still way too much. I recorded ten seconds of (admittedly not very large, 1366x768) full-screen video at the default quality. With not much going on it took 300K; with lots of activity (constantly switching between full-screen apps) it took at most 1.5MB.
Reducing the quality reduced file sizes by about 50%.
Unless you're playing a full-screen video and trying to record that, you shouldn't see anything like 30 to 40MB. Perhaps you should look at your audio settings.

ScreenRecorderLib from nuget is good.
SharpAVI is taking too much of my disc space.
You will need to be careful for ScreenRecorderLib, it would require some time saving mp4 file in the end.
Make sure your program won't end before that happens.
I use FileInfo.Length to check if the file size is not growing anymore. This will determine if the saving is finished or not.

Efficient way to send images via WCF?

I am learning WCF, LINQ and a few other technologies by writing, from scratch, a custom remote control application like VNC. I am creating it with three main goals in mind:
The server will provide 'remote control' on an application level (i.e. seamless windows) instead of full desktop access.
The client can select any number of applications that are running on the server and receive a stream of images of each of them.
A client can connect to more than one server simultaneously.
Right now I am using WCF to send an array of Bytes that represents the window being sent:
using (var ms = new MemoryStream()) {
window.GetBitmap().Save(ms, ImageFormat.Jpeg);
frame.Snapshot = ms.ToArray();
}
GetBitmap implementation:
var wRectangle = GetRectangle();
var image = new Bitmap(wRectangle.Width, wRectangle.Height);
var gfx = Graphics.FromImage(image);
gfx.CopyFromScreen(wRectangle.Left, wRectangle.Top, 0, 0, wRectangle.Size, CopyPixelOperation.SourceCopy);
return image;
It is then sent via WCF (TCPBinding and it will always be over LAN) to the client and reconstructed in a blank windows form with no border like this:
using (var ms = new MemoryStream(_currentFrame.Snapshot))
{
BackgroundImage = Image.FromStream(ms);
}
I would like to make this process as efficient as possible in both CPU and memory usage with bandwidth coming in third place. I am aiming to have the client connect to 5+ servers with 10+ applications per server.
Is my existing method the best approach (while continuing to use these technologies) and is there anything I can do to improve it?
Ideas that I am looking into (but I have no experience with):
Using an open source graphics library to capture and save the images instead of .Net solution.
Saving as PNG or another image type rather than JPG.
Send image deltas instead of a full image every time.
Try and 'record' the windows and create a compressed video stream instead of picture snapshots (mpeg?).

You should be aware for this points:
Transport: TCP/binary message encoding will be fastest way to transfer your image data
Image capture: you can rely on P/Invoke to access your screen data, as this can be faster and more memory consuming. Some examples: Capturing the Screen Image in C# [P/Invoke], How to take a screen shot using .NET [Managed] and Capturing screenshots using C# (Managed)
You should to reduce your image data before send it;
choose your image format wisely, as some formats have native compression (as JPG)
an example should be Find differences between images C#
sending only diff image, you can crop it and just send non-empty areas
Try to inspect your WCF messages. This will help you to understand how messages are formatted and will help you to identify how to make that messages smaller.
Just after passing through all this steps and being satisfied with your final code, you can download VncSharp source code. It implements the RFB Protocol (Wikipedia entry), "a simple protocol for remote access to graphical user interfaces. Because it works at the framebuffer level it is applicable to all windowing systems and applications, including X11, Windows and Macintosh. RFB is the protocol used in VNC (Virtual Network Computing)."

I worked on a similar project a while back. This was my general approach:
Rasterized the captured bitmap to tiles of 32x32
To determine which tiles had changed between frames I used unsafe code to compare them 64-bits at a time
On the set of delta tiles I applied one of the PNG filters to improve compressability and had the best results with the Paeth filter
Used DeflateStream to compress the filtered deltas
Used BinaryMessageEncoding custom binding to the service to transmit the data in Binary in stead of the default Base64 encoded version
Some client-side considerations. When dealing with large amounts of data being transferred through a WCF service I found that some parameters of the HttpTransportBinding and the XmlDictionaryRenderQuotas were set to pretty conservative values. So you will want to increase them.

Check out this: Large Data and Streaming (WCF)

The fastest way to send data between client/server is to send a byte array, or several byte arrays. That way WCF don't have to do any custom serialization on your data.
That said. You should use the new WPF/.Net 3.5 library to compress your images instead of the ones from System.Drawing. The functions in the System.Windows.Media.Imaging namespace are faster than the old ones, and can still be used in winforms.
In order to know if compression is the way to go you will have to benchmark your scenario to know how the compression/decompression time compares to transferring all the bytes uncompressed.
If you transfer the data over internet, then compression will help for sure. Between components on the same machine or on a LAN, the benefit might not be so obvious.
You could also try compressing the image, then chunk the data and send asynchronously with a chunk id which you puzzle together on the client. Tcp connections start slow and increase in bandwidth over time, so starting two or four at the same time should cut the total transfer time (all depending on how much data you are sending). Chunking the compressed images bytes is also easier logic wise compared to doing tiles in the actual images.
Summed up: System.Windows.Media.Imaging should help you both cpu and bandwidth wise compared to your current code. Memory wise I would guess about the same.

Instead of capturing the entire image just send smaller subsections of the image. Meaning: starting in the upper left corner, send a 10x10 pixel image, then 'move' ten pixels and send the next 10px square, and so on. You can then send dozens of small images and then update the painted full image on the client. If you've used RDC to view images on a remote machine you've probably seen it do this sort of screen painting.
Using the smaller image sections you can then split up the deltas as well, so if nothing has changed in the current section you can safely skip it, inform the client that you're skipping it, and then move onto the next section.
You'll definitely want to use compression for sending the images. However you should check to see if you get smaller file sizes from using compression similar to gZip, or if using an image codec gives you better results. I've never run a comparison, so I can't say for certain one way or another.

Your solution looks fine to me, but I suggest (as others did) you use tiles and compress the traffic when possible. In addition, I think you should send the entire image once a while, just to be sure that the client's deltas have a common "base".
Perhaps you can use an existing solution for streaming, such as RTP-H263 for video streaming. It works great, it uses compression, and it's well documented and widely used. You can then skip the WCF part and go directly to the streaming part (either over TCP or over UDP). If your solution should go to production, perhaps the H263 streaming approach would be better in terms of responsiveness and network usage.

Bitmap scrImg = new Bitmap(Screen.PrimaryScreen.Bounds.Width, Screen.PrimaryScreen.Bounds.Height);
Graphics scr;
scr.CopyFromScreen(new Point(0, 0), new Point(0, 0), Screen.PrimaryScreen.Bounds.Size);
testPictureBox.Image = (Image)scrImg;
I use this code to capture my screen.