TL;DR below
Hi,
I'm trying to make an in-game screen, whose purpose is to show the contents of another window running on the computer (under Windows). For that matter, I had to get my hands dirty and play with that good ol' Windows legacy API, the one with external references and everything. After a couple days of struggle, I was able to make it work! The only problem is that Unity doesn't seem to support it...
I already asked this on Unity answers, but had no answers.
I have a WindowOperation class which wraps all of the API calls into static methods, and also performs a few basic tasks. One of them consists in capturing the screen pixels and saving them in a System.Drawing.Bitmap:
public static Bitmap CaptureApplication(string procName)
{
// ...
var bmp = new Bitmap(width, height, PixelFormat.Format24bppRgb);
// ...
return bmp;
}
Unity fails declaring bmp and outputs the following message in the console:
ArgumentException: Parameter is not valid.
System.Drawing.Bitmap..ctor (Int32 width, Int32 height, PixelFormat format)
(wrapper remoting-invoke-with-check) System.Drawing.Bitmap:.ctor (int,int,System.Drawing.Imaging.PixelFormat)
[More info about the stack trace, which leads to the declaration of bmp in the above code]
According to MSDN and other threads on SO, such exceptions are thrown when .NET is unable to allocate a single block of contiguous memory as big as requested.
Since I didn't have such problems when I tested my code outside of Unity, my best guess is that (correct me if I'm wrong) Unity sandboxes the execution of the script and somehow doesn't allow the allocation of the memory bmp needs.
My screen dimension is 1600*900, and pixel depth is 3 bytes ( PixelFormat.Format24bppRgb ), so the amount of memory needed is 4,320,000 bytes, or just a bit more than 4MB, which doesn't seem excessive to me.
To capture the screen, I'm using .NET's Graphics.CopyFromScreen() function. I don't know how it works, but let's say it uses the same amount of memory as bmp, so the biggest amount of memory being used at one point of time in this script according to a pessimistic estimation rounds up to 10MB. However, I don't think this changes the root of the problem, and bmp is declared before any usage of CopyFromScreen() anyway.
TL;DR, is there a way I can tell Unity that it should let me allocate tons of memory in my scripts if I want to? Or maybe I'm completely off tracks and the problem lies somewhere else?
All answers are more than welcome!
Thank you in advance. :)
Related
I have to rotate JPG images lossless in .net (90°|180°|270°). The following articles show how to do it:
https://learn.microsoft.com/en-us/dotnet/api/system.drawing.imaging.encoder.transformation?view=netframework-4.7.2
https://www.codeproject.com/tips/64116/Using-GDIplus-code-in-a-WPF-application-for-lossle.aspx
The examples seem quite straightforward; however, I had no luck getting this to work. My source data comes as an array (various JPG files, from camera from internet etc.) and so I want to return the rotated images also as a byte array. Here the (simplified) code:
Image image;
using (var ms = new MemoryStream(originalImageData)) {
image = System.Drawing.Image.FromStream(ms);
}
// If I don't copy the image into a new bitmap, every try to save the image fails with a general GDI+ exception. This seems to be another bug of GDI+.
var bmp = new Bitmap(image);
// Creating the parameters for saving
var encParameters = new EncoderParameters(1);
encParameters.Param[0] = new EncoderParameter(Encoder.Transformation, (long)EncoderValue.TransformRotate90);
using (var ms = new MemoryStream()) {
// Now saving the image, what fails always with an ArgumentException from GDI+
// There is no difference, if I try to save to a file or to a stream.
bmp.Save(ms, GetJpgEncoderInfo(), encParameters);
return ms.ToArray();
}
I always get an ArgumentException from GDI+ without any useful information:
The operation failed with the final exception [ArgumentException].
Source: System.Drawing
I tried an awful lot of things, however never got it working.
The main code seems right, since if I change the EncoderParameter to Encoder.Quality, the code works fine:
encParameters.Param[0] = new EncoderParameter(Encoder.Quality, 50L);
I found some interesting posts about this problem in the internet, however no real solution. One particularly contains a statement from Hans Passant, that this seems to be really a bug, with a response from an MS employee, which I don't understand or which may be also simply weird:
https://social.msdn.microsoft.com/Forums/vstudio/en-US/de74ec2e-643d-41c7-9d04-254642a9775c/imagesave-quotparameter-is-not-validquot-in-windows-7?forum=netfxbcl
However this post is 10 years old and I cannot believe, that this is not fixed, especially since the transformation has an explicit example in the MSDN docs.
Does anyone have a hint, what I'm doing wrong, or, if this is really a bug, how can I circumvent it?
Please note that I have to make the transformation lossless (as far as the pixel-size allows it). Therefore, Image.RotateFlip is not an option.
Windows version is 10.0.17763, .Net is 4.7.2
using (var ms = new MemoryStream(originalImageData)) {
image = System.Drawing.Image.FromStream(ms);
}
This is the root of all evil and made the first attempt fail. It violates the rule stipulated in the Remarks section of the documentation, You must keep the stream open for the lifetime of the Image. Violating the rule does not cause consistent trouble, note how Save() call failed but the Bitmap(image) constructor succeeded. GDI+ is somewhat lazy, you have very nice evidence that the JPEG codec indeed tries to avoid recompressing the image. But that can't work, the raw data in the stream is no longer accessible since the stream got disposed. The exception is lousy because the native GDI+ code doesn't know beans about a MemoryStream. The fix is simple, just move the closing } bracket after the Save() call.
From there it went wrong another way, triggered primarily by the new bmp object. Neither the image nor the bmp objects are being disposed. This consumes address space in a hurry, the GC can't run often enough to keep you out of trouble since the data for the bitmap is stored in unmanaged memory. Now the Save() call fails when the MemoryStream can't allocate memory anymore.
You must use the using statement on these objects so this can't happen.
Ought to solve the problems, do get rid of Bitmap workaround since that forces the JPEG to be recompressed. Technically you can still get into trouble when the images are large, suffering from address space fragmentation in a 32-bit process. Keep an eye on the "Private bytes" memory counter for the process, ideally it stays below a gigabyte. If not then use Project > Properties > Build tab, untick "Prefer 32-bit".
I am trying to develop a basic screen sharing and collaboration app in C#. I am currently working on capturing the screen, finding areas of the screen that have changed and subsequently need to be transmitted to the end client.
I am having a problem in that the overall frame rate of the screen capture is too low. I have a fairly good algorithm for finding areas of the screen that have changed. Given a byte array of pixels on the screen it calculates areas that have changed in 2-4ms, however the overall frame rate I am getting is 15-18 fps (i.e. taking somewhere around 60ms per frame). The bottleneck is capturing the data on the screen as a byte array which is taking around 35-50ms. I have tried a couple of different techniques and can't push the fps past 20.
At first I tried something like this:
var _bmp = new Bitmap(screenSectionToMonitor.Width, screenSectionToMonitor.Height);
var _gfx = Graphics.FromImage(_bmp);
_gfx.CopyFromScreen(_screenSectionToMonitor.X, _screenSectionToMonitor.Y, 0, 0, new Size(_screenSectionToMonitor.Width, _screenSectionToMonitor.Height), CopyPixelOperation.SourceCopy);
var data = _bmp.LockBits(new Rectangle(0, 0, _screenSectionToMonitor.Width, _screenSectionToMonitor.Height), ImageLockMode.ReadOnly, _bmp.PixelFormat);
var ptr = data.Scan0;
Marshal.Copy(ptr, _screenshot, 0, _screenSectionToMonitor.Height * _screenSectionToMonitor.Width * _bytesPerPixel);
_bmp.UnlockBits(data);
This is too slow taking around 45ms just to run the code above for a single 1080p screen. This makes the overall frame rate too slow to be smooth, so I then tried using DirectX as per the example here:
http://www.codeproject.com/Articles/274461/Very-fast-screen-capture-using-DirectX-in-Csharp
However this didn't really net any results. It marginally increased the speed of the screen capture but it was still much too slow (taking around 25-40ms, and the small increase wasn't worth the overhead of the extra DLLs, code, etc.
After googling around a bit I couldn't really find any better solutions, so my question is what is the best way to capture the pixels currently displaying on the screen? An ideal solution would:
Capture the screen as an array of bytes as RGBA
Work on older windows platforms (e.g. Windows XP and above)
Work with multiple displays
Uses existing system libraries rather than 3rd party DLLs
All these points are negotiable for a solution that return a decent overall framerate, in the region of 5-10ms for the actual capturing so the framerate can be 40-60fps.
Alternatively, If there no solution that matches above, am I taking the wrong path to calculate screen changes. Is there a better way to calculate areas of the screen that have changed?
Perhaps you can access the screen buffers at a lower level of code and hook directly into the layers and regions Windows uses as part of its screen updates. It sounds like you are after the raw display changes and Windows already has to keep track of this data. Just offering a direction for you to pursue while you find someone more knowledgeable.
Recently, I was trying to answer another SO question about loading the frames (Bitmap and duration) of animated GIFs. The code can be found on pastenbin.
While doing additional tests on this code before moving it into my dev library, I noticed that there is a problem with this line of code:
//Get the times stored in the gif
//PropertyTagFrameDelay ((PROPID) 0x5100) comes from gdiplusimaging.h
//More info on http://msdn.microsoft.com/en-us/library/windows/desktop/ms534416(v=vs.85).aspx
var times = img.GetPropertyItem(0x5100).Value;
When running this on Windows .Net using this (example GIF), the array is of the same size as the amount of frames in the animated GIF and filled with the durations of the frames. In this case a byte[20] which converts to (BitConverter.ToInt32()) 5 durations:
[75,0,0,0,125,0,0,0,125,0,0,0,125,0,0,0,250,0,0,0]
On MonoMac however, this line of code for the same example GIF returns a byte[4] which converts to only one duration (the first):
[75,0,0,0]
I tested this for 10 different GIF's and the result is always the same. On Windows all durations are in the byte[], while MonoMac only lists the first duration:
[x,0,0,0]
[75,0,0,0]
[50,0,0,0]
[125,0,0,0]
Looking at the Mono System.Drawing.Image source code, the length seem to be set in this method, which is a GDI wrapper:
status = GDIPlus.GdipGetPropertyItemSize (nativeObject, propid,out propSize);
However, I don't really see any problems, not with the source as with my implementation. Am I missing something or is this a bug?
I don't see anything wrong in the mono source either. It would have been helpful if you would have posted one of the sample images you tried. One quirk about the GIF image format is that the Graphics Control Extension block that contains the frame time is optional and may be omitted before an image descriptor. Non-zero odds therefore that you have GIF files that just have one GCE that applies to all the frames, you are supposed to apply the same frame time to every frame.
Do note that you didn't get 4 values, the frame time is encoded as a 32-bit value and you are seeing the little endian encoding for it in a byte[]. You should use BitConverter.ToInt32(), as you correctly did in your sample code.
I therefore think you should probably use this instead:
//convert 4 bit value to integer
var duration = BitConverter.ToInt32(times, 4*i % times.Length);
Do note that there's another nasty implementation detail about GIF frames, frames #2 and up do not have to be the same size as the frame #1. And each frame has a metadata field that describes what should be done with the previous frame to merge it with the next one. There are no property IDs that I know of to obtain the frame offset, size and undraw method for each frame. I think you need to render each frame into a bitmap yourself to get a proper sequence of images. Very ugly details, GIF needs to die.
If you look into libgdiplus you will see that the properties are always read from the active bitmap:
if (gdip_bitmapdata_property_find_id(image->active_bitmap, propID, &index) != Ok) {
You can set the active bitmap by calling Image.SelectActiveFrame and then mono will return the correct durations, one by one. Since this is an incompatibility with windows, I'd call it a mono bug. As a simple workaround, you can of course just check the array length and handle both cases. This will be better than a check for mono, because if mono gets fixed this will continue to work.
I'm working on some university project and got stuck with memory issue.
I load a bitmap which takes about 1,5GB on HDD with code below:
Bitmap bmp = new Bitmap(pathToFile);
The issue is that the newly created Bitmap object uses about 3,5GB of RAM which is something I can't understand (that's really BIG wrapper :E). I need to get to the pixel array, and the use of Bitmap class is really helpful (I use LockBits() method later, and process the array byte per byte) but in this case it's total blocker. So here is my question:
Is there any easy way to extract the pixel array without lending additional 2gb?
I'm using c# just to extract the needed array, which is later processed in c++ - maybe I can extract all needed data in c++ (but conversion issue appears here - I'm concentrating on 24bgr format)?
PS: I need to keep the whole bitmap in memory so splitting it into parts is no solution.
PS2: Just to clarify some issues: I know the difference between file extension and file format. The loaded file is uncompressed bitmap 3 bytes per pixel of size ~1.42GB (16k x 32k pixels), so why Bitmap object is more than two times bigger? Any decompressing issues and converting into other format aren't taking place.
Consider using Memory Mapped Files to access your HUGE data :).
An example focused on what you need can be found here: http://visualstudiomagazine.com/articles/2010/06/23/memory-mapped-files.aspx
It's in managed code but you might as well use it from equivalent native code.
Let me know if you need more details.
You can use this solution , Work with bitmaps faster in C#
http://www.codeproject.com/Tips/240428/Work-with-bitmap-faster-with-Csharp
Or you can use memory mapped files
http://visualstudiomagazine.com/articles/2010/06/23/memory-mapped-files.aspx
You can stop memory caching.
Instead of
Bitmap bmp = new Bitmap(pathToFile);
Use
var bmp = (Bitmap)Image.FromStream(sourceFileStream, false, false);
see https://stackoverflow.com/a/47424918/887092
Hey there!
Here is my setting:
I've got a c# application that extracts features from a series of images. Due to the size of a dataset (several thousand images) it is heavily parallelized, that's why we have a high-end machine with ssd that runs on Windows7 x64 (.NET4 runtime) to lift the hard work. I'm developing it on a Windows XP SP3 x86 machine under Visual Studio 2008 (.NET3.5) with Windows Forms - no chance to move to WPF by the way.
Edit3:
It's weird but I think I finally found out what's going on. Seems to be the codec for the image format that yields different results on the two machines! I don't know exactly what is going on there but the decoder on the xp machine produces more sane results than the win7 one. Sadly the better version is still in the x86 XP system :(. I guess the only solution to this one is changing the input image format to something lossless like png or bmp (Stupid me not thinking about the file format in the first place :)).
Edit2:
Thank you for your efforts. I think I will stick to implementing a converter on my own, it's not exactly what I wanted but I have to solve it somehow :). If anybody is reading this who has some ideas for me please let me know.
Edit:
In the comments I was recommended to use a third party lib for this. I think I didn't made myself clear enough in that I don't really want to use the DrawImage approach anyway - it's just a flawed quickhack to get an actually working new Bitmap(tmp, ... myPixelFormat) that would hopefully use some interpolation. The thing I want to achieve is solely to convert the incoming image to a common PixelFormat with some standard interpolation.
My problem is as follows. Some of the source images are in Indexed8bpp jpg format that don't get along very well with the WinForms imaging stuff. Therefore in my image loading logic there is a check for indexed images that will convert the image to my applications default format (e.g. Format16bpp) like that:
Image GetImageByPath(string path)
{
Image result = null;
using (FileStream fs = new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.Read))
{
Image tmp = Image.FromStream(fs); // Here goes the same image ...
if (tmp.PixelFormat == PixelFormat.Format1bppIndexed ||
tmp.PixelFormat == PixelFormat.Format4bppIndexed ||
tmp.PixelFormat == PixelFormat.Format8bppIndexed ||
tmp.PixelFormat == PixelFormat.Indexed)
{
// Creating a Bitmap container in the application's default format
result = new Bitmap(tmp.Width, tmp.Height, DefConf.DefaultPixelFormat);
Graphics g = Graphics.FromImage(result);
g.InterpolationMode = InterpolationMode.HighQualityBicubic;
// We need not to scale anything in here
Rectangle drawRect = new Rectangle(0, 0, tmp.Width, tmp.Height);
// (*) Here is where the strange thing happens - I know I could use
// DrawImageUnscaled - that isn't working either
g.DrawImage(tmp, drawRect, drawRect, GraphicsUnit.Pixel);
g.Dispose();
}
else
{
result = new Bitmap(tmp); // Just copying the input stream
}
tmp.Dispose();
}
// (**) At this stage the x86 XP memory image differs from the
// the x64 Win7 image despite having the same settings
// on the very same image o.O
result.GetPixel(0, 0).B; // x86: 102, x64: 102
result.GetPixel(1, 0).B; // x86: 104, x64: 102
result.GetPixel(2, 0).B; // x86: 83, x64: 85
result.GetPixel(3, 0).B; // x86: 117, x64: 121
...
return result;
}
I tracked the problem down to (*). I think the InterpolationMode has something to do with it but there's no difference which of them I choose the results are different at (**) on the two systems anyway. I've been investigating test image data with some stupid copy&paste lines, to be sure it's not an issue with accessing the data in a wrong way.
The images all together look like this Electron Backscatter Diffraction Pattern. The actual color values differ subtly but they carry a lot of information - the interpolation even enhances it. It looks like the composition algorithm on the x86 machine uses the InterpolationMode property whereas the x64 thingy just spreads the palette values out without taking any interpolation into account.
I never noticed any difference between the output of the two machines until the day I implemented a histogram view feature on the data in my application. On the x86 machine it is balanced as one would expect it from watching the images. The x64 machine on the other hand would rather give some kind of sparse bar-diagram, an indication of indexed image data. It even effects the overall output data of the whole application - the output differs on both machines with the same data, that's not a good thing.
To me it looks like a bug in the x64 implementation, but that's just me :-). I just want the images on the x64 machine to have the same values as the x86 ones.
If anybody has an idea I'd be very pleased. I've been searching for similar behavior on the net for ages but resistance seems futile :)
Oh look out ... a whale!
If you want to make sure that this is always done the same way, you'll have to write your own code to handle it. Fortunately, it's not too difficult.
Your 8bpp image has a palette that contains the actual color values. You need to read that palette and convert the color values (which, if I remember correctly, are 24 bits) to 16-bit color values. You're going to lose information in the conversion, but you're already losing information in your conversion. At least this way, you'll lost the information in a predictable way.
Put the converted color values (there won't be more than 256 of them) into an array that you can use for lookup. Then ...
Create your destination bitmap and call LockBits to get a pointer to the actual bitmap data. Call LockBits to get a pointer to the bitmap data of the source bitmap. Then, for each pixel:
read the source bitmap pixel (8 bytes)
get the color value (16 bits) from your converted color array
store the color value in the destination bitmap
You could do this with GetPixel and SetPixel, but it would be very very slow.
I vaguely seem to recall that .NET graphics classes rely on GDI+. If that's still the case today, then there's no point in trying your app on different 64 bit systems with different video drivers. Your best bet would be to either do the interpolation using raw GDI operations (P/Invoke) or write your own pixel interpolation routine in software. Neither option is particularly attractive.
You really should use OpenCV for image handling like that, it's available in C# here: OpenCVSharp.
I use a standard method for the graphics object, and with this settings outperforms X86. Count performance at release runs, not debug. Also check optimize code at project properties, build tab. Studio 2017, framework 4.7.1
public static Graphics CreateGraphics(Image i)
{
Graphics g = Graphics.FromImage(i);
g.CompositingMode = CompositingMode.SourceOver;
g.CompositingQuality = CompositingQuality.HighSpeed;
g.InterpolationMode = InterpolationMode.NearestNeighbor;
g.SmoothingMode = SmoothingMode.HighSpeed;
return g;
}