Why is Mat not enough in EmguCV?
Why can't Matrix<> load an image from a file itself?
For instance,
Mat img = new Mat(path);
is a valid operation. But,
Matrix<byte> img = new Matrix<byte>(path);
or,
Matrix<byte> img = Matrix<byte>.FromFile(path);
aren't valid operations.
Based on the information from the Emgu Wiki the fundamental difference between the two types is whether the underlying data array is managed or not.
Mat is a wrapper around the C++ cv::Mat class. Generally this class acts as a smart pointer which manages the memory allocated for the data array it owns (although it's able to just observe as well -- a good example of this capability is the ability to return a Mat header for a Matrix instance in C#). This means that OpenCV is able to (re)allocate the memory as necesssary. The trade-off is that in such cases it's more difficult to access the underlying data effectively in C#.
The Matrix class uses a managed array to hold the data. That means you can easily access the underlying data array in C#.
Honestly, the best person to tell you why it's not possible to load Matrix from an image file would be the author. My guess would be that it's intended to represent other things than images. Technically this could be added in the same way as the ability to load an image file was given to the Mat wrapper (the C++ equivalent has no such feature).
Related
I have a particular problem which I cannot seem to reproduce in a minimal working example.
I have to deal with a large framework of legacy code and modifying all of that out of my scope. To deal with it I have to apply some particular patterns.
Overview of the codebase
I have a managed C# application (.NET 5.0). In this appliation I need to run some C++ code.
For this, there is a CLI-wrapper project. This wrapper contains most of the legacy framework which is out of my control and is why I can only transfer strings to my C++ class (more on this later). Based on config, this legacy framework uses the wrapper to instantiate C++ classes and calls methods on them, processes the results and finally, destroys all the C++ classes afterwards.
This CLI-wrapper allows me ONLY to pass strings as parameters to the C++ classes it creates.
All of my libraries are dynamically linked (using DLL's). The C# is a project which references the C++/CLI wrapper which in turn referenced the C++ project with my C++-class. This project references the external LargeLibrary (more on this later).
The root of the problem
The C++ code is called repeatedly, every few seconds. It should respond fast.
My C++ code needs to load some large file from disk (about 400 MB) and process it which takes quite some time.
Since the C++ classes are recreated each time, loading the file each time consumes so much time which is unacceptable.
As this data is essentially constant, I try to load it once during initialisation of the program. Then I pass a pointer to my C++ class which then can use the object. The object then remains in memory when the C++ class is destroyed so it can be used again later.
To complicate things, I need quite a large library to read and process my file (I reference this library here as LargeLibrary). If I make the CLI-wrapper dependent on this, it won't compile.
I can imagine this is because of the CLI stuff. Therefore, I use a void pointer, so the wrapper does not have to be aware of the actual type of behind the pointer. The actual object is created using a function inside my C++-class (so the correct destructor is linked to the shared pointer).
This all compiles fine.
My solution
I made a small extension to the CLI-wrapper to create the object which read my file from disk and keeps the information in memory.
This object is created using the method CreateInformationObject(). ptr_native is a smart pointer for using native objects in managed code. It's type is: CAutoNativePtr<std::shared_ptr<void>> ptr_native.
Creating my object inside the wrapper looks like:
// Create a shared_ptr on dynamic memory (i.e. heap).
std::shared_ptr<void>* objectPointer = new std::shared_ptr<void>();
// Load the module and store a shared pointer pointing to it in the dynamic memory.
*objectPointer = CppConsumerStuff::CppConsumer::CreateInformationObject(value);
// Load the module and store a shared pointer pointing to it in the dynamic memory.
ptr_native.Attach(objectPointer);
The CreateInformationObject() method inside my C++ class (the CppConsumerStuff::CppConsumer) is:
std::shared_ptr<void> CppConsumer::CreateInformationObject(std::string pathToFile)
{
std::shared_ptr<LargeLibrary::ActualObjectType> objectPtr = std::make_shared<LargeLibrary::ActualObjectType>();
*objectPtr = LargeLibrary::FileLoader::load(pathToFile)
return objectPtr;
}
Then, because of the legacy framework, I tried this longshot: convert the pointer address to string, pass it via the framework to my C++ class and convert it back to a pointer to the actual type of the object.
This goes like (in my CLI-wrapper extension):
//Cast void pointer to string.
String^ CliStorage::GetPointerString()
{
std::stringstream ss;
ss << (*ptr_native).get(); // Pointer to hex string.
std::string ptr_string = ss.str();
return StringToManaged(ptr_string);
}
Finally, (in my C++ class), I convert this pointer-string back to a pointer to the actual object as:
void DoWorkOnLargeObject(std::string ptr_string)
{
// Cast pointer to usable type
uint64_t raw_ptr = 0; // Define int size depending on system architecture.
std::stringstream ss;
ss << std::hex << ptr_string;
ss >> raw_ptr; //Hex string to int.
cppObjectPtr = reinterpret_cast<void*>(raw_ptr);
LargeLibrary::ActualObjectType* cppObjectPtrCasted = static_cast<LargeLibrary::ActualObjectType*>(cppObjectPtr);
// Use the object.
cppObjectPtrCasted->GetDataStuff();
// Rest of code doing work...
}
My results
I build all of this in Visual Studio 2019.
When I create a Debug build, all works :).
However, when I create a Release build, it does not work and throws the following Exception: ``
Minimal working example
I tried to create a minimal working example.
Both with and without the large external library.
However, in my minimum working Examples it always works, no matter the build type (debug / release).
My question
So my question is: Do my minimum working examples work by accident and am I relying on undefined behavior? Or should this concept (no matter how ugly it is) indeed work?
If it is undefined behavior, please explain, I want to learn. If it should work, the problem resides in the legacy framework and I will make inquiries about this.
I know these are very ugly patterns, but I try to get something working with the means I have within my scope.
Thank you
EDIT, I added CreateInformationObject() method code to my question. I think my hazard may be inside here. Maybe I do some illegal pointer stuff which results in undefined behavior?
I am not an expert on this so take my advise with a grain of salt. In my opinion directly sharing the memory address between the processes will in general fail due to memory protection (which forbids programs to just access memory that was not allocated for them).
You could used shared memory. This is memory shared between processes. Normally one would use this to share memory between concurrent processes but this is in no way necessary (and not having competing accesses is actually beneficial). Wikipedia lists boost and Qt as examples for libraries implementing cross-platform support for shared memory.
Looking into the boost documentation for sharing memory, it says "As shared memory has kernel or filesystem persistence, the user must explicitly destroy it.", which is exactly what you want, since it should persist between calls of the same program. Note that you should remove the shared memory in some other way since it will persist.
Adapting the example from the documentation, it could look something like this:
#include <boost/interprocess/shared_memory_object.hpp>
#include <boost/interprocess/mapped_region.hpp>
#include <cstring>
#include <cstdlib>
#include <string>
constexpr auto shm_name = "SharedMemoryCLI";
using namespace boost::interprocess;
auto create_shared_memory() {
// Compute your data and calculate the size needed:
shared_memory_object shm {create_only, shm_name, read_write};
// Either use an upper bound for the size needed or compute your data before.
shm.truncate(data_size);
//Map the whole shared memory in this process
mapped_region region{shm, read_write};
// Either write your data directly to region.get_address():
compute_data_to(region.get_address());
// Or have the data already computed and memcopy it:
std::memcpy(region.get_address(), data_ptr, data_size);
return region;
}
auto obtain_memory_region() {
try {
shared_memory_object shm{open_only, shm_name, read_only};
return mapped_region {shm, read_only};
} catch(const std::exception &er) {
// One should probably check if this is the "right" exception but boost does not say what type it uses here...
return create_shared_memory();
}
}
int main(int argc, char *argv[])
{
region = obtain_memory_region();
static_cast<char*>(region.get_address()); // can be used as a to your data.
return 0;
}
Note that you maybe have to persist the exact size of your shared memory in some other way (or maybe just as the first 8 byte of the region). You can then have to somehow get the char* back to your wanted type, but I think that a reinterpret_cast should work here.
The above code is not tested and I give no guarantees but I am pretty confident that it should work roughly in this way and be about as fast as just sharing the pointer (if that would work). You really should read the entirety of https://www.boost.org/doc/libs/1_48_0/doc/html/interprocess/sharedmemorybetweenprocesses.html before applying this in any way.
In my game user can choose multiple images from gallery and load them into game. I have created a function that get images from filepath. In that function I declare local variable byte[]. so after read file do i need to dispose that byte[] to freed memory. here is my code :
if (File.Exists(filePath))
{
byte[] fileData = File.ReadAllBytes(filePath);
Texture2D tex = new Texture2D(2, 2);
tex.LoadImage(fileData);
}
Do I need to clear byte[] after texture is loaded from byte[] ???? User can choose any image file from gallery so there is no limit of filesize.
Short answer: No
byte[] is a managed resource. It will be fully cleaned up by the GC.
File related classes often involve unmanaged resources - usually the OS filehandles. But File.ReadAllBytes(string) looks like it follows my advise "create, use, dispose. All in the same piece of code, ideally using a using statement." So I expect not issues form it.
Networking and DB classes involve unamanged resources - usually network connections.
A lot of drawing related classes use unmanaged resources - primarily some unmanaged memory for performance.
But this is just a byte[]. As managed as any int[]. Note that you do not have a choice on the mater anyway as neither byte nor array implement IDisposeable. If you try to dispose if it, the compiler will wonder what you are talking about as there is no such function.
Is there a way to access the underlying memory of a BitmapImage object with c# pointers?
I know that there's a CopyPixels method but it makes a copy of the pixels array into a new array (duplicating memory requirements). If you open a large image (2gb) it allocates a lot of un-useful stuff. And if you want to operate some sort of elaboration, like CCLA, it takes a huge amount of memory.
I need only to read the pixel array.
Is it possible to address pixels directly like you can do in System.Drawing.Bitmap?
I wrote a fast bitmap access for System.Drawing.Bitmap, but as I'm using WPF, I need the same functionality for BitmapSource. Otherwise I have to duplicate the image loading (Bitmap for my old method and BitmapSource to show the image in WPF) taking a lot of memory and time.
Thank you
Lorenzo
A bitmap source does not necessarily have backing memory for the entire image. An example of when it does not would be an image file on disk which is lazy loaded.
The only access you have with WIC, and therefore WPF, is the CopyPixels method. Certain subclasses of BitmapSource will allow access to a buffer, but they are internally just allocating memory and calling CopyPixels themselves.
I would assume that whatever operation does not require access to the entire image at a time. If so, you can call CopyPixels to a smaller buffer, and window your access to the image. Most decoders, when a single pixel is requested, will buffer the entire stride, or in the case of JPEG, then entire block.
I am not sure what CCLA is, and cannot find a definition that seems to fit, but if it is some sort of transform on the source image, you can implement it as a BitmapSource. That way, you can compose a full chain which will
read the image from disk (TiffBitmapEncoder et al.)
scale or translate them (TransformedBitmap)
then window it to only the portion you need (CroppedBitmap)
use a format conversion (FormatConvertedBitmap)
pass it to your algorithm (CclaTransformBitmap)
and finally render it to a WritableBitmap, which gives you access to the buffer
With careful attention to the CacheOption used on the source image, as well as the order of transforms, you should be able to access an arbitrarily large image without significant memory impact.
If you already have a performant algorithm for GDI (System.Drawing), there is no sense re-implementing it. You can still display your final bitmap using Imaging.CreateBitmapSourceFromHBitmap or by using a WindowsFormsHost to host the control you previously built.
I am able to copy a VideoInfoHeader which is part AMMediaType with the following lines:
AMMediaType mediaType = new AMMediaType();
VideoInfoHeader videoInfo = new VideoInfoHeader();
(pSampleGrabber as ISampleGrabber).GetConnectedMediaType(mediaType);
videoInfo = (VideoInfoHeader)Marshal.PtrToStructure(mediaType.formatPtr, typeof(VideoInfoHeader));
that I got from SampleGrabber Parameter is Incorrect and other ressources on the internet. However I am not able to copy the whole AMMediaType-Structure. The pointer to the VideoInfoHeader is lost.
So I was wondering if there exists a helper function that
copies an AMMediaType to a different location in memory, looks for pointers like VideoInfoHeader and copies the content of the VideoInfoHeader and other referenced data to a new location (including pointing formatPtr to the right location).
I am using this in the following scenario:
I retrieve all possible IPin mediatypes of a video input device and show the AMMediaTypes in a Windows.Forms - Combobox. When I read the SelectedObject from the Combobox, the pointer to the VideoInfoHeader is zero. This is where it fails to connect using my selected pin configuration when constructing the filtergraph.
The AMMediaType aka AM_MEDIA_TYPE structure consists of regular members and additionally allocated piece of memory pointed to by formatPtr. The latter is a byte array with a meaning dependent on format type. Both structure and slave memory block are typically using specific COM allocator.
Duplicating the structure is copying its members and duplicating the byte array behind formatPtr. The structure also includes pUnk member supposed to be COM interface, however it is typically null. It is valid for it to be non-null, but I cannot remember a single case over years when it was actually used, in particular with video and audio media types.
I found a blog entry which suggests that sometimes c# compiler may decide to put array on the stack instead of the heap:
Improving Performance Through Stack Allocation (.NET Memory Management: Part 2)
This guy claims that:
The compiler will also sometimes decide to put things on the stack on its own. I did an experiment with TestStruct2 in which I allocated it both an unsafe and normal context. In the unsafe context the array was put on the heap, but in the normal context when I looked into memory the array had actually been allocated on the stack.
Can someone confirm that?
I was trying to repeat his example, but everytime I tried array was allocated on the heap.
If c# compiler can do such trick without using 'unsafe' keyword I'm specially intrested in it. I have a code that is working on many small byte arrays (8-10 bytes long) and so using heap for each new byte[...] is a waste of time and memory (especially that each object on heap has 8 bytes overhead needed for garbage collector).
EDIT: I just want to describe why it's important to me:
I'm writing library that is communicating with Gemalto.NET smart card which can have .net code working in it. When I call a method that returns something, smart card return 8 bytes that describes me the exact Type of return value. This 8 bytes are calculated by using md5 hash and some byte arrays concatenations.
Problem is that when I have an array that is not known to me I must scan all types in all assemblies loaded in application and for each I must calculate those 8 bytes until I find the same array.
I don't know other way to find the type, so I'm trying to speed it up as much as possible.
Author of the linked-to article here.
It seems impossible to force stack allocation outside of an unsafe context. This is likely the case to prevent some classes of stack overflow condition.
Instead, I recommend using a memory recycler class which would allocate byte arrays as needed but also allow you to "turn them in" afterward for reuse. It's as simple as keeping a stack of unused byte arrays and, when the list is empty, allocating new ones.
Stack<Byte[]> _byteStack = new Stack<Byte[]>();
Byte[] AllocateArray()
{
Byte[] outArray;
if (_byteStack.Count > 0)
outArray = _byteStack.Pop();
else
outArray = new Byte[8];
return outArray;
}
void RecycleArray(Byte[] inArray)
{
_byteStack.Push(inArray);
}
If you are trying to match a hash with a type it seems the best idea would be to use a Dictionary for fast lookups. In this case you could load all relevant types at startup, if this causes program startup to become too slow you might want to consider caching them the first time each type is used.
From your line:
I have a code that is working on many small byte arrays (8-10 bytes long)
Personally, I'd be more interested in allocating a spare buffer somewhere that different parts of your code can re-use (while processing the same block). Then you don't have any creation/GC to worry about. In most cases (where the buffer is used for very discreet operations) with a scratch-buffer, you can even always assume that it is "all yours" - i.e. every method that needs it can assume that they can start writing at zero.
I use this single-buffer approach in some binary serialization code (while encoding data); it is a big boost to performance. In my case, I pass a "context" object between the layers of serialization (that encapsulates the scratch-buffer, the output-stream (with some additional local buffering), and a few other oddities).
System.Array (the class representing an array) is a reference type and lives on the heap. You can only have an array on the stack if you use unsafe code.
I can't see where it says otherwise in the article that you refer to. If you want to have a stack allocated array, you can do something like this:
decimal* stackAllocatedDecimals = stackalloc decimal[4];
Personally I wouldn't bother- how much performance do you think you will gain by this approach?
This CodeProject article might be useful to you though.