thats how i wrote your beautiful code(some simple changes for me for easier understanding)
private void Form1_Load(object sender, EventArgs e)
{
prev = GetDesktopImage();//get a screenshot of the desktop;
cur = GetDesktopImage();//get a screenshot of the desktop;
var locked1 = cur.LockBits(new Rectangle(0, 0, cur.Width, cur.Height),
ImageLockMode.ReadWrite, PixelFormat.Format32bppArgb);
var locked2 = prev.LockBits(new Rectangle(0, 0, prev.Width, prev.Height),
ImageLockMode.ReadWrite, PixelFormat.Format32bppArgb);
ApplyXor(locked1, locked2);
compressionBuffer = new byte[1920* 1080 * 4];
// Compressed buffer -- where the data goes that we'll send.
int backbufSize = LZ4.LZ4Codec.MaximumOutputLength(this.compressionBuffer.Length) + 4;
backbuf = new CompressedCaptureScreen(backbufSize);
MessageBox.Show(compressionBuffer.Length.ToString());
int length = Compress();
MessageBox.Show(backbuf.Data.Length.ToString());//prints the new buffer size
}
the compression buffer length is for example 8294400
and the backbuff.Data.length is 8326947
I didn't like the compression suggestions, so here's what I would do.
You don't want to compress a video stream (so MPEG, AVI, etc are out of the question -- these don't have to be real-time) and you don't want to compress individual pictures (since that's just stupid).
Basically what you want to do is detect if things change and send the differences. You're on the right track with that; most video compressors do that. You also want a fast compression/decompression algorithm; especially if you go to more FPS that will become more relevant.
Differences. First off, eliminate all branches in your code, and make sure memory access is sequential (e.g. iterate x in the inner loop). The latter will give you cache locality. As for the differences, I'd probably use a 64-bit XOR; it's easy, branchless and fast.
If you want performance, it's probably better to do this in C++: The current C# implementation doesn't vectorize your code, and that will help you a great deal here.
Do something like this (I'm assuming 32bit pixel format):
for (int y=0; y<height; ++y) // change to PFor if you like
{
ulong* row1 = (ulong*)(image1BasePtr + image1Stride * y);
ulong* row2 = (ulong*)(image2BasePtr + image2Stride * y);
for (int x=0; x<width; x += 2)
row2[x] ^= row1[x];
}
Fast compression and decompression usually means simpler compression algorithms. https://code.google.com/p/lz4/ is such an algorithm, and there's a proper .NET port available for that as well. You might want to read on how it works too; there is a streaming feature in LZ4 and if you can make it handle 2 images instead of 1 that will probably give you a nice compression boost.
All in all, if you're trying to compress white noise, it simply won't work and your frame rate will drop. One way to solve this is to reduce the colors if you have too much 'randomness' in a frame. A measure for randomness is entropy, and there are several ways to get a measure of the entropy of a picture ( https://en.wikipedia.org/wiki/Entropy_(information_theory) ). I'd stick with a very simple one: check the size of the compressed picture -- if it's above a certain limit, reduce the number of bits; if below, increase the number of bits.
Note that increasing and decreasing bits is not done with shifting in this case; you don't need your bits to be removed, you simply need your compression to work better. It's probably just as good to use a simple 'AND' with a bitmask. For example, if you want to drop 2 bits, you can do it like this:
for (int y=0; y<height; ++y) // change to PFor if you like
{
ulong* row1 = (ulong*)(image1BasePtr + image1Stride * y);
ulong* row2 = (ulong*)(image2BasePtr + image2Stride * y);
ulong mask = 0xFFFCFCFCFFFCFCFC;
for (int x=0; x<width; x += 2)
row2[x] = (row2[x] ^ row1[x]) & mask;
}
PS: I'm not sure what I would do with the alpha component, I'll leave that up to your experimentation.
Good luck!
The long answer
I had some time to spare, so I just tested this approach. Here's some code to support it all.
This code normally run over 130 FPS with a nice constant memory pressure on my laptop, so the bottleneck shouldn't be here anymore. Note that you need LZ4 to get this working and that LZ4 is aimed at high speed, not high compression ratio's. A bit more on that later.
First we need something that we can use to hold all the data we're going to send. I'm not implementing the sockets stuff itself here (although that should be pretty simple using this as a start), I mainly focused on getting the data you need to send something over.
// The thing you send over a socket
public class CompressedCaptureScreen
{
public CompressedCaptureScreen(int size)
{
this.Data = new byte[size];
this.Size = 4;
}
public int Size;
public byte[] Data;
}
We also need a class that will hold all the magic:
public class CompressScreenCapture
{
Next, if I'm running high performance code, I make it a habit to preallocate all the buffers first. That'll save you time during the actual algorithmic stuff. 4 buffers of 1080p is about 33 MB, which is fine - so let's allocate that.
public CompressScreenCapture()
{
// Initialize with black screen; get bounds from screen.
this.screenBounds = Screen.PrimaryScreen.Bounds;
// Initialize 2 buffers - 1 for the current and 1 for the previous image
prev = new Bitmap(screenBounds.Width, screenBounds.Height, PixelFormat.Format32bppArgb);
cur = new Bitmap(screenBounds.Width, screenBounds.Height, PixelFormat.Format32bppArgb);
// Clear the 'prev' buffer - this is the initial state
using (Graphics g = Graphics.FromImage(prev))
{
g.Clear(Color.Black);
}
// Compression buffer -- we don't really need this but I'm lazy today.
compressionBuffer = new byte[screenBounds.Width * screenBounds.Height * 4];
// Compressed buffer -- where the data goes that we'll send.
int backbufSize = LZ4.LZ4Codec.MaximumOutputLength(this.compressionBuffer.Length) + 4;
backbuf = new CompressedCaptureScreen(backbufSize);
}
private Rectangle screenBounds;
private Bitmap prev;
private Bitmap cur;
private byte[] compressionBuffer;
private int backbufSize;
private CompressedCaptureScreen backbuf;
private int n = 0;
First thing to do is capture the screen. This is the easy part: simply fill the bitmap of the current screen:
private void Capture()
{
// Fill 'cur' with a screenshot
using (var gfxScreenshot = Graphics.FromImage(cur))
{
gfxScreenshot.CopyFromScreen(screenBounds.X, screenBounds.Y, 0, 0, screenBounds.Size, CopyPixelOperation.SourceCopy);
}
}
As I said, I don't want to compress 'raw' pixels. Instead, I'd much rather compress XOR masks of previous and the current image. Most of the times this will give you a whole lot of 0's, which is easy to compress:
private unsafe void ApplyXor(BitmapData previous, BitmapData current)
{
byte* prev0 = (byte*)previous.Scan0.ToPointer();
byte* cur0 = (byte*)current.Scan0.ToPointer();
int height = previous.Height;
int width = previous.Width;
int halfwidth = width / 2;
fixed (byte* target = this.compressionBuffer)
{
ulong* dst = (ulong*)target;
for (int y = 0; y < height; ++y)
{
ulong* prevRow = (ulong*)(prev0 + previous.Stride * y);
ulong* curRow = (ulong*)(cur0 + current.Stride * y);
for (int x = 0; x < halfwidth; ++x)
{
*(dst++) = curRow[x] ^ prevRow[x];
}
}
}
}
For the compression algorithm I simply pass the buffer to LZ4 and let it do its magic.
private int Compress()
{
// Grab the backbuf in an attempt to update it with new data
var backbuf = this.backbuf;
backbuf.Size = LZ4.LZ4Codec.Encode(
this.compressionBuffer, 0, this.compressionBuffer.Length,
backbuf.Data, 4, backbuf.Data.Length-4);
Buffer.BlockCopy(BitConverter.GetBytes(backbuf.Size), 0, backbuf.Data, 0, 4);
return backbuf.Size;
}
One thing to note here is that I make it a habit to put everything in my buffer that I need to send over the TCP/IP socket. I don't want to move data around if I can easily avoid it, so I'm simply putting everything that I need on the other side there.
As for the sockets itself, you can use a-sync TCP sockets here (I would), but if you do, you will need to add an extra buffer.
The only thing that remains is to glue everything together and put some statistics on the screen:
public void Iterate()
{
Stopwatch sw = Stopwatch.StartNew();
// Capture a screen:
Capture();
TimeSpan timeToCapture = sw.Elapsed;
// Lock both images:
var locked1 = cur.LockBits(new Rectangle(0, 0, cur.Width, cur.Height),
ImageLockMode.ReadWrite, PixelFormat.Format32bppArgb);
var locked2 = prev.LockBits(new Rectangle(0, 0, prev.Width, prev.Height),
ImageLockMode.ReadWrite, PixelFormat.Format32bppArgb);
try
{
// Xor screen:
ApplyXor(locked2, locked1);
TimeSpan timeToXor = sw.Elapsed;
// Compress screen:
int length = Compress();
TimeSpan timeToCompress = sw.Elapsed;
if ((++n) % 50 == 0)
{
Console.Write("Iteration: {0:0.00}s, {1:0.00}s, {2:0.00}s " +
"{3} Kb => {4:0.0} FPS \r",
timeToCapture.TotalSeconds, timeToXor.TotalSeconds,
timeToCompress.TotalSeconds, length / 1024,
1.0 / sw.Elapsed.TotalSeconds);
}
// Swap buffers:
var tmp = cur;
cur = prev;
prev = tmp;
}
finally
{
cur.UnlockBits(locked1);
prev.UnlockBits(locked2);
}
}
Note that I reduce Console output to ensure that's not the bottleneck. :-)
Simple improvements
It's a bit wasteful to compress all those 0's, right? It's pretty easy to track the min and max y position that has data using a simple boolean.
ulong tmp = curRow[x] ^ prevRow[x];
*(dst++) = tmp;
hasdata |= tmp != 0;
You also probably don't want to call Compress if you don't have to.
After adding this feature you'll get something like this on your screen:
Iteration: 0.00s, 0.01s, 0.01s 1 Kb => 152.0 FPS
Using another compression algorithm might also help. I stuck to LZ4 because it's simple to use, it's blazing fast and compresses pretty well -- still, there are other options that might work better. See http://fastcompression.blogspot.nl/ for a comparison.
If you have a bad connection or if you're streaming video over a remote connection, all this won't work. Best to reduce the pixel values here. That's quite simple: apply a simple 64-bit mask during the xor to both the previous and current picture... You can also try using indexed colors - anyhow, there's a ton of different things you can try here; I just kept it simple because that's probably good enough.
You can also use Parallel.For for the xor loop; personally I didn't really care about that.
A bit more challenging
If you have 1 server that is serving multiple clients, things will get a bit more challenging, as they will refresh at different rates. We want the fastest refreshing client to determine the server speed - not slowest. :-)
To implement this, the relation between the prev and cur has to change. If we simply 'xor' away like here, we'll end up with a completely garbled picture at the slower clients.
To solve that, we don't want to swap prev anymore, as it should hold key frames (that you'll refresh when the compressed data becomes too big) and cur will hold incremental data from the 'xor' results. This means you can basically grab an arbitrary 'xor'red frame and send it over the line - as long as the prev bitmap is recent.
H264 or Equaivalent Codec Streaming
There are various compressed streaming available which does almost everything that you can do to optimize screen sharing over network. There are many open source and commercial libraries to stream.
Screen transfer in Blocks
H264 already does this, but if you want to do it yourself, you have to divide your screens into smaller blocks of 100x100 pixels, and compare these blocks with previous version and send these blocks over network.
Window Render Information
Microsoft RDP does lot better, it does not send screen as a raster image, instead it analyzes screen and creates screen blocks based on the windows on the screen. It then analyzes contents of screen and sends image only if needed, if it is a text box with some text in it, RDP sends information to render text box with a text with font information and other information. So instead of sending image, it sends information on what to render.
You can combine all techniques and make a mixed protocol to send screen blocks with image and other rendering information.
Instead of handling data as an array of bytes, you can handle it as an array of integers.
int* p = (int*)((byte*)scan0.ToPointer() + y * stride);
int* p2 = (int*)((byte*)scan02.ToPointer() + y * stride2);
for (int x = 0; x < nWidth; x++)
{
//always get the complete pixel when differences are found
if (*p2 != 0)
*p = *p2
++p;
++p2;
}
Related
I am porting an application from C# (WinForms) to C++ and noticed that drawing an image using GDI+ is much slower in C++, even though it uses the same API.
The image is loaded at application startup into a System.Drawing.Image or Gdiplus::Image, respectively.
The C# drawing code is (directly in the main form):
public Form1()
{
this.SetStyle(ControlStyles.UserPaint | ControlStyles.AllPaintingInWmPaint | ControlStyles.OptimizedDoubleBuffer, true);
this.image = Image.FromFile(...);
}
private readonly Image image;
protected override void OnPaint(PaintEventArgs e)
{
base.OnPaint(e);
var sw = Stopwatch.StartNew();
e.Graphics.TranslateTransform(this.translation.X, this.translation.Y); /* NOTE0 */
e.Graphics.DrawImage(this.image, 0, 0, this.image.Width, this.image.Height);
Debug.WriteLine(sw.Elapsed.TotalMilliseconds.ToString()); // ~3ms
}
Regarding SetStyle: AFAIK, these flags (1) make WndProc ignore WM_ERASEBKGND, and (2) allocate a temporary HDC and Graphics for double buffered drawing.
The C++ drawing code is more bloated.
I have browsed the reference source of System.Windows.Forms.Control to see how it handles HDC and how it implements double buffering.
As far as I can tell, my implementation matches that closely (see NOTE1) (note that I implemented it in C++ first and then looked at how it's in the .NET source -- I may have overlooked things).
The rest of the program is more or less what you get when you create a fresh Win32 project in VS2019. All error handling omitted for readability.
// In wWinMain:
Gdiplus::GdiplusStartupInput gdiplusStartupInput;
Gdiplus::GdiplusStartup(&gdiplusToken, &gdiplusStartupInput, NULL);
gdip_bitmap = Gdiplus::Image::FromFile(...);
// In the WndProc callback:
case WM_PAINT:
// Need this for the back buffer bitmap
RECT client_rect;
GetClientRect(hWnd, &client_rect);
int client_width = client_rect.right - client_rect.left;
int client_height = client_rect.bottom - client_rect.top;
// Double buffering
HDC hdc0 = BeginPaint(hWnd, &ps);
HDC hdc = CreateCompatibleDC(hdc0);
HBITMAP back_buffer = CreateCompatibleBitmap(hdc0, client_width, client_height); /* NOTE1 */
HBITMAP dummy_buffer = (HBITMAP)SelectObject(hdc, back_buffer);
// Create GDI+ stuff on top of HDC
Gdiplus::Graphics *graphics = Gdiplus::Graphics::FromHDC(hdc);
QueryPerformanceCounter(...);
graphics->DrawImage(gdip_bitmap, 0, 0, bitmap_width, bitmap_height);
/* print performance counter diff */ // -> ~27 ms typically
delete graphics;
// Double buffering
BitBlt(hdc0, 0, 0, client_width, client_height, hdc, 0, 0, SRCCOPY);
SelectObject(hdc, dummy_buffer);
DeleteObject(back_buffer);
DeleteDC(hdc); // This is the temporary double buffer HDC
EndPaint(hWnd, &ps);
/* NOTE1 */: In the .NET source code they don't use CreateCompatibleBitmap, but CreateDIBSection instead.
That improves performance from 27 ms to 21 ms and is very cumbersome (see below).
In both cases I am calling Control.Invalidate or InvalidateRect, respectively, when the mouse moves (OnMouseMove, WM_MOUSEMOVE). The goal is to implement panning with the mouse using SetTransform - that's irrelevant for now as long as draw performance is bad.
NOTE2: https://stackoverflow.com/a/1617930/653473
This answer suggests that using Gdiplus::CachedBitmap is the trick. However, I can find no evidence in the C# WinForms source code that it makes use of cached bitmaps in any way - the C# code uses GdipDrawImageRectI which maps to GdipDrawImageRectI, which maps to Graphics::DrawImage(IN Image* image, IN INT x, IN INT y, IN INT width, IN INT height).
Regarding /* NOTE1 */, here is the replacement for CreateCompatibleBitmap (just substitute CreateVeryCompatibleBitmap):
bool bFillBitmapInfo(HDC hdc, BITMAPINFO *pbmi)
{
HBITMAP hbm = NULL;
bool bRet = false;
// Create a dummy bitmap from which we can query color format info about the device surface.
hbm = CreateCompatibleBitmap(hdc, 1, 1);
pbmi->bmiHeader.biSize = sizeof(BITMAPINFOHEADER);
// Call first time to fill in BITMAPINFO header.
GetDIBits(hdc, hbm, 0, 0, NULL, pbmi, DIB_RGB_COLORS);
if ( pbmi->bmiHeader.biBitCount <= 8 ) {
// UNSUPPORTED
} else {
if ( pbmi->bmiHeader.biCompression == BI_BITFIELDS ) {
// Call a second time to get the color masks.
// It's a GetDIBits Win32 "feature".
GetDIBits(hdc, hbm, 0, pbmi->bmiHeader.biHeight, NULL, pbmi, DIB_RGB_COLORS);
}
bRet = true;
}
if (hbm != NULL) {
DeleteObject(hbm);
hbm = NULL;
}
return bRet;
}
HBITMAP CreateVeryCompatibleBitmap(HDC hdc, int width, int height)
{
BITMAPINFO *pbmi = (BITMAPINFO *)LocalAlloc(LMEM_ZEROINIT, 4096); // Because otherwise I would have to figure out the actual size of the color table at the end; whatever...
bFillBitmapInfo(hdc, pbmi);
pbmi->bmiHeader.biWidth = width;
pbmi->bmiHeader.biHeight = height;
if (pbmi->bmiHeader.biCompression == BI_RGB) {
pbmi->bmiHeader.biSizeImage = 0;
} else {
if ( pbmi->bmiHeader.biBitCount == 16 )
pbmi->bmiHeader.biSizeImage = width * height * 2;
else if ( pbmi->bmiHeader.biBitCount == 32 )
pbmi->bmiHeader.biSizeImage = width * height * 4;
else
pbmi->bmiHeader.biSizeImage = 0;
}
pbmi->bmiHeader.biClrUsed = 0;
pbmi->bmiHeader.biClrImportant = 0;
void *dummy;
HBITMAP back_buffer = CreateDIBSection(hdc, pbmi, DIB_RGB_COLORS, &dummy, NULL, 0);
LocalFree(pbmi);
return back_buffer;
}
Using a very compatible bitmap as the back buffer improves performance from 27 ms to 21 ms.
Regarding /* NOTE0 */ in the C# code -- the code is only fast if the transformation matrix doesn't scale. C# performance drops slightly when upscaling (~9ms), and drops significantly (~22ms) when downsampling.
This hints to: DrawImage probably wants to BitBlt if possible. But it can't in my C++ case because the Bitmap format (that was loaded from disk) is different from the back buffer format or something.
If I create a new more compatible bitmap (this time no clear difference between CreateCompatibleBitmap and CreateVeryCompatibleBitmap), and then draw the original bitmap onto that, and then only use the more compatible bitmap in the DrawImage call, then performance increases to about 4.5 ms. It also has the same performance characteristics when scaling now as the C# code.
if (better_bitmap == NULL)
{
HBITMAP tmp_bitmap = CreateVeryCompatibleBitmap(hdc0, gdip_bitmap->GetWidth(), gdip_bitmap->GetHeight());
HDC copy_hdc = CreateCompatibleDC(hdc0);
HGDIOBJ old = SelectObject(copy_hdc, tmp_bitmap);
Gdiplus::Graphics *copy_graphics = Gdiplus::Graphics::FromHDC(copy_hdc);
copy_graphics->DrawImage(gdip_bitmap, 0, 0, gdip_bitmap->GetWidth(), gdip_bitmap->GetHeight());
// Now tmp_bitmap contains the image, hopefully in the device's preferred format
delete copy_graphics;
SelectObject(copy_hdc, old);
DeleteDC(copy_hdc);
better_bitmap = Gdiplus::Bitmap::FromHBITMAP(tmp_bitmap, NULL);
}
BUT it's still consistently slower, there must be something missing still. And it raises a new question: Why is this not necessary in C# (same image and same machine)? Image.FromFile does not convert the bitmap format on loading as far as I can tell.
Why is the DrawImage call in the C++ code still slower, and what do I need to do to make it as fast as in C#?
I ended up replicating more of the .NET code insanity.
The magic call that makes it go fast is GdipImageForceValidation in System.Drawing.Image.FromFile. This function is basically not documented at all, and it is not even [officially] callable from C++. It is merely mentioned here: https://learn.microsoft.com/en-us/windows/win32/gdiplus/-gdiplus-image-flat
Gdiplus::Image::FromFile and GdipLoadImageFromFile don't actually load the full image into memory. It effectively gets copied from the disk every time it is being drawn. GdipImageForceValidation forces the image to be loaded into memory, or so it seems...
My initial idea of copying the image into a more compatible bitmap was on the right track, but the way I did it does not yield the best performance for GDI+ (because I used a GDI bitmap from the original HDC). Loading the image directly into a new GDI+ bitmap, regardless of pixel format, yields the same performance characteristics as seen in the C# implementation:
better_bitmap = new Gdiplus::Bitmap(gdip_bitmap->GetWidth(), gdip_bitmap->GetHeight(), PixelFormat24bppRGB);
Gdiplus::Graphics *graphics = Gdiplus::Graphics::FromImage(better_bitmap);
graphics->DrawImage(gdip_bitmap, 0, 0, gdip_bitmap->GetWidth(), gdip_bitmap->GetHeight());
delete graphics;
Even better yet, using PixelFormat32bppPARGB further improves performance substantially - the premultiplied alpha pays off when the image is repeatedly drawn (regardless of whether the source image has an alpha channel).
It seems calling GdipImageForceValidation effectively does something similar internally, although I don't know what it really does. Because Microsoft made it as impossible as they could to call the GDI+ flat API from C++ user code, I just modified Gdiplus::Image in my Windows SDK headers to include an appropriate method. Copying the bitmap explicitly to PARGB seems cleaner to me (and yields better performance).
Of course, after one finds out which undocumented function to use, google would also give some additional information: https://photosauce.net/blog/post/image-scaling-with-gdi-part-5-push-vs-pull-and-image-validation
GDI+ is not my favorite API.
I'm using SharpDx to capture the screen (1 to 60fps). Some frames are all transparent and end up getting processed and saved by the code.
Is there any simple/fast way to detect these frames drops without having to open the generated bitmap and look for the alpha values?
Here's what I'm using (saves capture as image):
try
{
//Try to get duplicated frame within given time.
_duplicatedOutput.AcquireNextFrame(MinimumDelay, out var duplicateFrameInformation, out var screenResource);
//Copy resource into memory that can be accessed by the CPU.
using (var screenTexture2D = screenResource.QueryInterface<Texture2D>())
_device.ImmediateContext.CopySubresourceRegion(screenTexture2D, 0, new ResourceRegion(Left, Top, 0, Left + Width, Top + Height, 1), _screenTexture, 0);
//Get the desktop capture texture.
var mapSource = _device.ImmediateContext.MapSubresource(_screenTexture, 0, MapMode.Read, MapFlags.None); //, out var stream);
#region Get image data
var bitmap = new System.Drawing.Bitmap(Width, Height, PixelFormat.Format32bppArgb);
var boundsRect = new System.Drawing.Rectangle(0, 0, Width, Height);
//Copy pixels from screen capture Texture to GDI bitmap.
var mapDest = bitmap.LockBits(boundsRect, ImageLockMode.WriteOnly, bitmap.PixelFormat);
var sourcePtr = mapSource.DataPointer;
var destPtr = mapDest.Scan0;
for (var y = 0; y < Height; y++)
{
//Copy a single line
Utilities.CopyMemory(destPtr, sourcePtr, Width * 4);
//Advance pointers
sourcePtr = IntPtr.Add(sourcePtr, mapSource.RowPitch);
destPtr = IntPtr.Add(destPtr, mapDest.Stride);
}
//Release source and dest locks
bitmap.UnlockBits(mapDest);
//Bitmap is saved in here!!!
#endregion
_device.ImmediateContext.UnmapSubresource(_screenTexture, 0);
screenResource.Dispose();
_duplicatedOutput.ReleaseFrame();
}
catch (SharpDXException e)
{
if (e.ResultCode.Code != SharpDX.DXGI.ResultCode.WaitTimeout.Result.Code)
throw;
}
It's a modified version from this one.
I also have this version (saves capture as pixel array):
//Get the desktop capture texture.
var data = _device.ImmediateContext.MapSubresource(_screenTexture, 0, MapMode.Read, MapFlags.None, out var stream);
var bytes = new byte[stream.Length];
//BGRA32 is 4 bytes.
for (var height = 0; height < Height; height++)
{
stream.Position = height * data.RowPitch;
Marshal.Copy(new IntPtr(stream.DataPointer.ToInt64() + height * data.RowPitch), bytes, height * Width * 4, Width * 4);
}
I'm not sure if it's the best way of saving the screen capture as image and/or pixel array, but it's somewhat working.
Anyway, the problem is that some frames captured are fully transparent and they are useless to me. I need to somehow avoid saving them at all.
When capturing as pixel array, I can simply check the bytes array, to know if the 4th item is 255 or 0. When saving as image, I could use the bitmap.GetPixel(0,0).A to know if the image has content or not.
But with both ways I need to finish the capture and get the full image content before being capable of knowing if the frame was dropped or not.
Is there any way to know if the frame was correctly captured?
You propblem boils down to you trying to do this on a timer. There is no way to guarantee a minimum execution time for every single tick/frame. And if the user picks a too high value, you get effects like this. At worst you might have ticks queue up in the EventQueue, until you run into a Exception because the queue is overfilled.
What you need to do is limit the rate to a maximum. If it does not work as fast as the user wants, that is the reality. I wrote some simple rate limiting code just for such a case:
integer interval = 20;
DateTime dueTime = DateTime.Now.AddMillisconds(interval);
while(true){
if(DateTime.Now >= dueTime){
//insert code here
//Update next dueTime
dueTime = DateTime.Now.AddMillisconds(interval);
}
else{
//Just yield to not tax out the CPU
Thread.Sleep(1);
}
}
Just two notes:
this was designed to run in a seperate thread. You have to run it as such, or adapt the Thread.Sleep() to your multitasking option of choice.
DateTime.Now is not suited for such small timeframes (2 digit ms). Usually the returned value will only update every 18 ms or so. It actually varies over time. 60 FPS puts you around 16ms. You should be using Stopwatch or something similar.
In order to ignore frame drops, I'm using this code (so far it's working as expected):
//Try to get the duplicated frame within given time.
_duplicatedOutput.AcquireNextFrame(1000, out var duplicateFrameInformation, out var screenResource);
//Somehow, it was not possible to retrieve the resource.
if (screenResource == null || duplicateFrameInformation.AccumulatedFrames == 0)
{
//Mark the frame as dropped.
frame.WasDropped = true;
FrameList.Add(frame);
screenResource?.Dispose();
_duplicatedOutput.ReleaseFrame();
return;
}
I'm simply checking if the screenResource is not null and if there are frames accumulated.
Why am I getting an out of memory exception?
So this dies in C# on the first time through:
splitBitmaps.Add(neededImage.Clone(rectDimensions, neededImage.PixelFormat));
Where splitBitmaps is a List<BitMap> BUT this works in VB for at least 4 iterations:
arlSplitBitmaps.Add(Image.Clone(rectDimensions, Image.PixelFormat))
Where arlSplitBitmaps is a simple array list. (And yes I've tried arraylist in c#)
This is the fullsection:
for (Int32 splitIndex = 0; splitIndex <= numberOfResultingImages - 1; splitIndex++)
{
Rectangle rectDimensions;
if (splitIndex < numberOfResultingImages - 1)
{
rectDimensions = new Rectangle(splitImageWidth * splitIndex, 0,
splitImageWidth, splitImageHeight);
}
else
{
rectDimensions = new Rectangle(splitImageWidth * splitIndex, 0,
sourceImageWidth - (splitImageWidth * splitIndex), splitImageHeight);
}
splitBitmaps.Add(neededImage.Clone(rectDimensions, neededImage.PixelFormat));
}
neededImage is a Bitmap by the way.
I can't find any useful answers on the intarweb, especially not why it works just fine in VB.
Update:
I actually found a reason (sort of) for this working but forgot to post it. It has to do with converting the image to a bitmap instead of just trying to clone the raw image if I remember.
Clone() may also throw an Out of memory exception when the coordinates specified in the Rectangle are outside the bounds of the bitmap. It will not clip them automatically for you.
I found that I was using Image.Clone to crop a bitmap and the width took the crop outside the bounds of the original image. This causes an Out of Memory error. Seems a bit strange but can beworth knowing.
I got this too when I tried to use the Clone() method to change the pixel format of a bitmap. If memory serves, I was trying to convert a 24 bpp bitmap to an 8 bit indexed format, naively hoping that the Bitmap class would magically handle the palette creation and so on. Obviously not :-/
This is a reach, but I've often found that if pulling images directly from disk that it's better to copy them to a new bitmap and dispose of the disk-bound image. I've seen great improvement in memory consumption when doing so.
Dave M. is on the money too... make sure to dispose when finished.
I struggled to figure this out recently - the answers above are correct. Key to solving this issue is to ensure the rectangle is actually within the boundaries of the image. See example of how I solved this.
In a nutshell, checked to if the area that was being cloned was outside the area of the image.
int totalWidth = rect.Left + rect.Width; //think -the same as Right property
int allowableWidth = localImage.Width - rect.Left;
int finalWidth = 0;
if (totalWidth > allowableWidth){
finalWidth = allowableWidth;
} else {
finalWidth = totalWidth;
}
rect.Width = finalWidth;
int totalHeight = rect.Top + rect.Height; //think same as Bottom property
int allowableHeight = localImage.Height - rect.Top;
int finalHeight = 0;
if (totalHeight > allowableHeight){
finalHeight = allowableHeight;
} else {
finalHeight = totalHeight;
}
rect.Height = finalHeight;
cropped = ((Bitmap)localImage).Clone(rect, System.Drawing.Imaging.PixelFormat.DontCare);
Make sure that you're calling .Dispose() properly on your images, otherwise unmanaged resources won't be freed up. I wonder how many images are you actually creating here -- hundreds? Thousands?
I am writing a video player to play back frames captured by our ASIC. They are in a custom format, and I have been provided with a function that decodes the ASIC states. The video can be any size from 640x480 to 2560x1200 (!!!). The output of each state cycle is a 16x16 blocks of pixels, which I have to get into a video on the screen.
Every time the screen needs to be updated, I have the following information:
Block width
Block height
X coordinate of block start
Y coordinate of block start
A 1-D array of RGB32 pixel color info
Major limitations:
.NET 3.5
No unsafe code
I spent this morning trying out a WriteableBitmap, and using it as a source for an Image, something like this:
private WriteableBitmap ImageSource;
public MainWindow()
{
InitializeComponent();
ImageSource = new WriteableBitmap(FrameWidth, FrameHeight, 96, 96, PixelFormats.Bgr32, null);
ImagePanel.Source = ImageSource;
}
private void DrawBox(byte Red, byte Green, byte Blue, int X, int Y)
{
int BoxWidth = 16;
int BoxHeight = 16;
int BytesPerPixel = ImageSource.Format.BitsPerPixel / 8;
byte[] Pixels = new byte[BoxWidth * BoxHeight * BytesPerPixel];
for (int i = 0; i < Pixels.Length; i += 4)
{
Pixels[i + 0] = Blue;
Pixels[i + 1] = Green;
Pixels[i + 2] = Red;
Pixels[i + 3] = 0;
}
int Stride = BoxWidth * BytesPerPixel;
Int32Rect DrawBox = new Int32Rect(X, Y, BoxWidth, BoxHeight);
ImageSource.Lock();
ImageSource.WritePixels(DrawBox, Pixels, Stride, 0);
ImageSource.Unlock();
}
It works, my video plays on the screen, but it is sloooooooow. Nowhere near real-time play speed. Is there a better way, besides procedurally generating a series of bitmaps, to do this that I'm not seeing? I've read something about D3Dimage, but it seems that's more for exporting a 3D scene to a bitmap. Suggestions here?
If I understand correctly you receive a 16*16 pixel block as a capture, and in the example above you are then updating the writeable bitmap on each of these updates.
This looks like a lot of churn as a render is being triggered on each block.
Would it not be better to:
Maintain a single byte buffer that represents an entire frame.
Update the buffer with your 16*16 blocks as you receive them.
When you have received all blocks for a frame, write the buffer to the bitmap.
Re-use the frame buffer for the next frame by over-writing it.
In this way you will have far less churn on rendering as you will not trigger a render for each block you receive, but only for each frame.
Update 1
I would also consider using Bgr24 as your pixel format if you are not using an alpha channel. That way you will have 3 bytes per pixel, instead of 4, so quite a lot less overhead.
Update 2
If things are still too slow for larger frame sizes, you could also consider the following which adds complexity though.
Consider maintaining and flipping two frame buffers. In this way you could composite frames on a background thread and then blit the finished frame on the UI thread. Whist the UI thread is rendering the frame, you can continue compositing on the alternative buffer.
Update 3
If your pixel colour info array from your capture is in the correct format, you could try using Buffer.BlockCopy to bulk copy the source pixel array into the frame buffer array. You would have to keep your bitmap format as bgr32 though. This low level array copying technique could yield some performance gains.
i need to re-project a series of ariel images that have been referenced in a geographical coordinate system into a UTM projection. I had read that using getPixel and setPixel might be slow - should set up a series of 2 dimensional arrays for intermediate access and then flush the values to the destination image when I am done.
Is this normally this sort of image processing is done by the professionals?
Most image processing is feature detection, segmentation of a scene, fault finding, classification and tracking ....
You might want to take a peek at the book:
Image Processing in C (applicable for other languages too)
Image Processing - Principles and Applications
Which describes many fast and effective means of many image transformations. These two books helped me when I was processing images :)
If I understand your question ... If you are re-aligning or assembling many images, and you don't have orientation as well as position, you can use these algorithms for re-alignment of edges and common features. If you are stitching by position then these algorithms will help in re-sampling/resizing your images for more efficient assembly. There are also some open source libraries for these kinds of things. (OpenCV comes to mind)
edit: If I were re-projecting large images into new projections based on position conversion (and it were dynamic, not static) I would look into building an on-demand application that will refactor images given required resolution and desired position. The application can then pull the nearest resolution of the relative neighbourhood images and provide a result at the desired resolution.
Without more background, I hope this helps!
edit 2:
Comment from answer below:
Depends on the images. If they are fixed size then an array might be good. If they vary then it might be better to implement a system that provides get/setpixel using relative sampling/averaging to match up images of differing res?
I don't know the ins and outs of the images you are working with, and what you are doing, but often abstracting what a 'pixel' is rather than accessing values in an array is useful. This way you can implement conversion, sampling, rotating, correcting algorithms on the backend. Like GetVPixel() or SetVPixel(). This may be more useful when working with multiple, differing res/format images. Like
SetVPixel(img1, coord1, GetVPixel(img2, coord2))
Obviously in an OOP/C# manner. img1 and img2 can be different in size, res, geographics, alignment or anything else providing your backend understands both.
If you don't mind using unsafe code, you can wrap the Bitmap's BitmapData in an object that allows you to efficiently get and set pixels. The below code is mostly taken from a gaussian blur filter, with a couple of modifications of my own. It's not the most flexible code if your bitmap formats differ but I hope it illustrates how you can manipulate bitmaps more efficiently.
public unsafe class RawBitmap : IDisposable
{
private BitmapData _bitmapData;
private byte* _begin;
public RawBitmap(Bitmap originBitmap)
{
OriginBitmap = originBitmap;
_bitmapData = OriginBitmap.LockBits(new Rectangle(0, 0, OriginBitmap.Width, OriginBitmap.Height), ImageLockMode.ReadWrite, PixelFormat.Format24bppRgb);
_begin = (byte*)(void*)_bitmapData.Scan0;
}
#region IDisposable Members
public void Dispose()
{
OriginBitmap.UnlockBits(_bitmapData);
}
#endregion
public unsafe byte* Begin
{
get { return _begin; }
}
public unsafe byte* this[int x, int y]
{
get
{
return _begin + y * (_bitmapData.Stride) + x * 3;
}
}
public unsafe byte* this[int x, int y, int offset]
{
get
{
return _begin + y * (_bitmapData.Stride) + x * 3 + offset;
}
}
public unsafe void SetColor(int x, int y, Color color)
{
byte* p = this[x, y];
p[0] = color.B;
p[1] = color.G;
p[2] = color.R;
}
public unsafe Color GetColor(int x, int y)
{
byte* p = this[x, y];
return new Color
(
p[2],
p[1],
p[0]
);
}
public int Stride
{
get { return _bitmapData.Stride; }
}
public int Width
{
get { return _bitmapData.Width; }
}
public int Height
{
get { return _bitmapData.Height; }
}
public int GetOffset()
{
return _bitmapData.Stride - _bitmapData.Width * 3;
}
public Bitmap OriginBitmap { get; private set; }
}
The FreeImage library is pretty fast and offers a Cut and Paste that might be useful. The distribution comes with a C# wrapper.
AFAIK the overhead of GetPixel/SetPixel is the call to it, when accessing an array there is no call hence less overhead.
You should start with GetPixel/SetPixel, you can alway override those calls later to use direct data access.