Develop Reference

Passing a C++ object pointer to a C++ class through C++/CLI

I have a particular problem which I cannot seem to reproduce in a minimal working example.
I have to deal with a large framework of legacy code and modifying all of that out of my scope. To deal with it I have to apply some particular patterns.
Overview of the codebase
I have a managed C# application (.NET 5.0). In this appliation I need to run some C++ code.
For this, there is a CLI-wrapper project. This wrapper contains most of the legacy framework which is out of my control and is why I can only transfer strings to my C++ class (more on this later). Based on config, this legacy framework uses the wrapper to instantiate C++ classes and calls methods on them, processes the results and finally, destroys all the C++ classes afterwards.
This CLI-wrapper allows me ONLY to pass strings as parameters to the C++ classes it creates.
All of my libraries are dynamically linked (using DLL's). The C# is a project which references the C++/CLI wrapper which in turn referenced the C++ project with my C++-class. This project references the external LargeLibrary (more on this later).
The root of the problem
The C++ code is called repeatedly, every few seconds. It should respond fast.
My C++ code needs to load some large file from disk (about 400 MB) and process it which takes quite some time.
Since the C++ classes are recreated each time, loading the file each time consumes so much time which is unacceptable.
As this data is essentially constant, I try to load it once during initialisation of the program. Then I pass a pointer to my C++ class which then can use the object. The object then remains in memory when the C++ class is destroyed so it can be used again later.
To complicate things, I need quite a large library to read and process my file (I reference this library here as LargeLibrary). If I make the CLI-wrapper dependent on this, it won't compile.
I can imagine this is because of the CLI stuff. Therefore, I use a void pointer, so the wrapper does not have to be aware of the actual type of behind the pointer. The actual object is created using a function inside my C++-class (so the correct destructor is linked to the shared pointer).
This all compiles fine.
My solution
I made a small extension to the CLI-wrapper to create the object which read my file from disk and keeps the information in memory.
This object is created using the method CreateInformationObject(). ptr_native is a smart pointer for using native objects in managed code. It's type is: CAutoNativePtr<std::shared_ptr<void>> ptr_native.
Creating my object inside the wrapper looks like:
// Create a shared_ptr on dynamic memory (i.e. heap).
std::shared_ptr<void>* objectPointer = new std::shared_ptr<void>();
// Load the module and store a shared pointer pointing to it in the dynamic memory.
*objectPointer = CppConsumerStuff::CppConsumer::CreateInformationObject(value);
// Load the module and store a shared pointer pointing to it in the dynamic memory.
ptr_native.Attach(objectPointer);
The CreateInformationObject() method inside my C++ class (the CppConsumerStuff::CppConsumer) is:
std::shared_ptr<void> CppConsumer::CreateInformationObject(std::string pathToFile)
{
std::shared_ptr<LargeLibrary::ActualObjectType> objectPtr = std::make_shared<LargeLibrary::ActualObjectType>();
*objectPtr = LargeLibrary::FileLoader::load(pathToFile)
return objectPtr;
}
Then, because of the legacy framework, I tried this longshot: convert the pointer address to string, pass it via the framework to my C++ class and convert it back to a pointer to the actual type of the object.
This goes like (in my CLI-wrapper extension):
//Cast void pointer to string.
String^ CliStorage::GetPointerString()
{
std::stringstream ss;
ss << (*ptr_native).get(); // Pointer to hex string.
std::string ptr_string = ss.str();
return StringToManaged(ptr_string);
}
Finally, (in my C++ class), I convert this pointer-string back to a pointer to the actual object as:
void DoWorkOnLargeObject(std::string ptr_string)
{
// Cast pointer to usable type
uint64_t raw_ptr = 0; // Define int size depending on system architecture.
std::stringstream ss;
ss << std::hex << ptr_string;
ss >> raw_ptr; //Hex string to int.
cppObjectPtr = reinterpret_cast<void*>(raw_ptr);
LargeLibrary::ActualObjectType* cppObjectPtrCasted = static_cast<LargeLibrary::ActualObjectType*>(cppObjectPtr);
// Use the object.
cppObjectPtrCasted->GetDataStuff();
// Rest of code doing work...
}
My results
I build all of this in Visual Studio 2019.
When I create a Debug build, all works :).
However, when I create a Release build, it does not work and throws the following Exception: ``
Minimal working example
I tried to create a minimal working example.
Both with and without the large external library.
However, in my minimum working Examples it always works, no matter the build type (debug / release).
My question
So my question is: Do my minimum working examples work by accident and am I relying on undefined behavior? Or should this concept (no matter how ugly it is) indeed work?
If it is undefined behavior, please explain, I want to learn. If it should work, the problem resides in the legacy framework and I will make inquiries about this.
I know these are very ugly patterns, but I try to get something working with the means I have within my scope.
Thank you
EDIT, I added CreateInformationObject() method code to my question. I think my hazard may be inside here. Maybe I do some illegal pointer stuff which results in undefined behavior?

I am not an expert on this so take my advise with a grain of salt. In my opinion directly sharing the memory address between the processes will in general fail due to memory protection (which forbids programs to just access memory that was not allocated for them).
You could used shared memory. This is memory shared between processes. Normally one would use this to share memory between concurrent processes but this is in no way necessary (and not having competing accesses is actually beneficial). Wikipedia lists boost and Qt as examples for libraries implementing cross-platform support for shared memory.
Looking into the boost documentation for sharing memory, it says "As shared memory has kernel or filesystem persistence, the user must explicitly destroy it.", which is exactly what you want, since it should persist between calls of the same program. Note that you should remove the shared memory in some other way since it will persist.
Adapting the example from the documentation, it could look something like this:
#include <boost/interprocess/shared_memory_object.hpp>
#include <boost/interprocess/mapped_region.hpp>
#include <cstring>
#include <cstdlib>
#include <string>
constexpr auto shm_name = "SharedMemoryCLI";
using namespace boost::interprocess;
auto create_shared_memory() {
// Compute your data and calculate the size needed:
shared_memory_object shm {create_only, shm_name, read_write};
// Either use an upper bound for the size needed or compute your data before.
shm.truncate(data_size);
//Map the whole shared memory in this process
mapped_region region{shm, read_write};
// Either write your data directly to region.get_address():
compute_data_to(region.get_address());
// Or have the data already computed and memcopy it:
std::memcpy(region.get_address(), data_ptr, data_size);
return region;
}
auto obtain_memory_region() {
try {
shared_memory_object shm{open_only, shm_name, read_only};
return mapped_region {shm, read_only};
} catch(const std::exception &er) {
// One should probably check if this is the "right" exception but boost does not say what type it uses here...
return create_shared_memory();
}
}
int main(int argc, char *argv[])
{
region = obtain_memory_region();
static_cast<char*>(region.get_address()); // can be used as a to your data.
return 0;
}
Note that you maybe have to persist the exact size of your shared memory in some other way (or maybe just as the first 8 byte of the region). You can then have to somehow get the char* back to your wanted type, but I think that a reinterpret_cast should work here.
The above code is not tested and I give no guarantees but I am pretty confident that it should work roughly in this way and be about as fast as just sharing the pointer (if that would work). You really should read the entirety of https://www.boost.org/doc/libs/1_48_0/doc/html/interprocess/sharedmemorybetweenprocesses.html before applying this in any way.

How to find implementation of an extern C# method [duplicate]

When use ILSpy to check the code of System.String, I found there are some methods marked as MethodImplOptions.InternalCall such as:
[SecurityCritical]
[MethodImpl(MethodImplOptions.InternalCall)]
internal static extern int nativeCompareOrdinalEx(string strA, int indexA, string strB, int indexB, int count);
I know MethodImplOptions.InternalCall means this method is implemented natively by the common language runtime to optimized code to improve performance.
My question is: Is that anyway can enable us to see the code marked as MethodImplOptions.InternalCall?

You'll need the source code for the CLR to see the implementation of these methods. That's a bit hard to come by, Microsoft doesn't publish it and it is not covered by the Reference Source.
As long as the method is "old", available since .NET 2.0, then you'll have a shot at it from the SSCLI20 source code. With a non-zero risk that you will be looking at an outdated version of the code of course. But good enough to get an idea what it looks like and often still accurate.
The starting point to start searching for the code is the clr/src/vm/ecall.cpp source code file. It contains the tables that the jitter searches for internal methods. The section that's relevant for nativeCompareOrdinalEx() looks like this:
FCFuncStart(gStringFuncs)
FCDynamic("FastAllocateString", CORINFO_INTRINSIC_Illegal, ECall::FastAllocateString)
FCDynamicSig(COR_CTOR_METHOD_NAME, &gsig_IM_ArrChar_RetVoid, CORINFO_INTRINSIC_Illegal, ECall::CtorCharArrayManaged)
FCDynamicSig(COR_CTOR_METHOD_NAME, &gsig_IM_ArrChar_Int_Int_RetVoid, CORINFO_INTRINSIC_Illegal, ECall::CtorCharArrayStartLengthManaged)
FCDynamicSig(COR_CTOR_METHOD_NAME, &gsig_IM_PtrChar_RetVoid, CORINFO_INTRINSIC_Illegal, ECall::CtorCharPtrManaged)
FCDynamicSig(COR_CTOR_METHOD_NAME, &gsig_IM_PtrChar_Int_Int_RetVoid, CORINFO_INTRINSIC_Illegal, ECall::CtorCharPtrStartLengthManaged)
FCDynamicSig(COR_CTOR_METHOD_NAME, &gsig_IM_Char_Int_RetVoid, CORINFO_INTRINSIC_Illegal, ECall::CtorCharCountManaged)
FCFuncElement("nativeCompareOrdinal", COMString::FCCompareOrdinal) // <=== Here
FCFuncElement("nativeCompareOrdinalWC", COMString::FCCompareOrdinalWC)
FCIntrinsic("get_Length", COMString::Length, CORINFO_INTRINSIC_StringLength)
// etc..
}
Note how the FCFuncElement has the method name as a string and a function pointer to the C++ method that implements the internal call. Grepping the source code tree then takes you to clr/src/vm/comstring.cpp. I won't bore everybody with the C++ code, just have a look for yourself.
/*================================CompareOrdinal===============================*/
FCIMPL3(INT32, COMString::FCCompareOrdinal, StringObject* strA, StringObject* strB, CLR_BOOL bIgnoreCase) {
// Yadayada
//...
}
Searching for CaseInsensitiveCompHelper() and FastCompareStringHelperAligned() takes you to the actual implementations of the, respectively, case-insensitive and case-sensitive comparison functions in the same source code file.
The only other thing notable about this is that CLR version 4 made some changes to this mechanism. Adding lots of new internal methods and supporting an entirely different additional interop mechanism through a [DllImport] attribute for a fake DLL named "QCall". There's no good way to see source for these additions that I know of.
UPDATE: source is now available from the CoreCLR project. The table was moved from ecall.cpp to ecalllist.h, the mechanics are still the same. Do keep in mind that this is the .NETCore version of the CLR, the desktop version source is still closed-source. The two version are however likely to have much in common.

Since now CoreCLR is open source, so we could check out the internal code.
You could search COMString::CompareOrdinalEx keyword in stringnative.cpp to see the internal implementation.

As the help string says, they are "implemented in the CLR itself", so you'll need to consult its C++ sources or disassembly.
Generally, the files that comprise the CLR engine are a few native DLLs in the %WINDIR%\Microsoft.NET\Framework\<.NET engine version> folder, mostly mscor*.dll and clr.dll. The root .NET DLL, mscoree.dll, is in System32 but it seems to only act as a launcher.
Since the InternalCall method implementations are implementation details, there's no guarantee that those methods are implemented in a consistent way, e.g. that there's even some global registry of them.
E.g. disassembling shows that .NET 4 System.String's native methods are implemented in clr.dll and referenced in a directory-like structure while System.Deployment.Application.NativeMethods.IClrRuntimeInfo is backed by the CLRRuntimeInfoImpl COM class in mscoreei.dll, the methods simply being its virtual functions.

Loading multiple instance of DLL (C#/.NET)

I am developing C#/.NET 3.5 application. I am using legacy dll written in C, signals.dll. I invoke it from a .NET wrapper using P/Invoke. I am calling 2 types of processing functions, type A and B. When I call only one type of processing, all works fine. When I interleave calls to A and B processing, data result is corrupted. I believe that dll, signals.dll is using C style global variables, and data gets corrupted.
To resolve that, I created 2 copies of dll on disk, signals.dll and signals2.dll. Then I modified .NET wrapper using P/Invoke to direct type A processing to one dll, type B processing to another instance. And now, all works fine.
Then I saw similar problem on forums and solution there. (Supporting multiple instances of a plugin DLL with global data ).
Basically, that proposed solution is dynamic lay from code, creating a new instance of .dll on disk (based on need), and loads it and invokes functions from it. Key part of code looks like this:
private IntPtr dllHandle;
string myDllPath = Path.Combine(dllDir, String.Format("mylib-{0}.dll", GetHashCode()));
File.Copy(origDllPath, myDllPath);
dllPath = myDllPath;
dllHandle = LoadLibrary(dllPath);
_getVersion = GetProcEntryDelegate<_getVersionDelegate>(dllHandle, "GetVersion");
private delegate int _getVersionDelegate();
private readonly _getVersionDelegate _getVersion;
public int GetVersion()
{
return _getVersion();
}
private static D GetProcEntryDelegate<D>(IntPtr hModule, string name)
where D: class
{
IntPtr addr = _getProcAddress(hModule, name);
if (addr == IntPtr.Zero)
throw new Win32Exception();
return Marshal.GetDelegateForFunctionPointer(addr, typeof(D)) as D;
}
What is coming to my mind, would it be possible to modify above code to create copy of dll IN MEMORY, not on disk and load it from there. I think just that IntPtr dllHandle needs to be fooled into getting value from memory, not from LoadLibrary. How to do that?

Both LoadLibrary and LoadLibraryEx requires a file path. You'll need a custom loading procedure, including memory mappings and what-not. I've found a blog post ("Loading a DLL from memory") describing the procedure, and a matching GitHub project; MemoryModule.

There is nothing just about it :) - it's far more complex and involving
Here is a link that might help - Load Library/Module from Memory
And as #Hans Passant said I'd discourage you to go that way - even though it may be a tempting solution for some scenarios (but I don't see that you really need that honestly, nice maybe).
It involves dealing with the portable executable format - and I doubt that project covers all that needs to be done.
You could try making your C++/CLI wrapper - or exporting the MemoryLoadLibrary and try P/Invoking - but I doubt that'd work easily.

Converting to C++ from C#

I have done pretty much all my programming using C# and very much a newbie to C++. However now I have to convert to C++ and is finding it a bit difficult. For example, I wrote a pretty simple program using C# to acquire a RegistryKey, then using a recursive function I iterate through my registry key to find a specific key and then get the values I want. No problem, I can write that program in 10 minutes using C#. Here is the code.
My primary function. It gets Bluetooth Registry Key and then call the recursive function.
private static void CheckOpenComPorts()
{
RegistryKey blueToothPorts = Registry.LocalMachine.OpenSubKey(#"SYSTEM\CurrentControlSet\Enum\Bluetooth");
List<string> foundPorts = new List<string>();
AddFoundPortsToList(blueToothPorts, ref foundPorts);
//Rest of the program; not relevant here.
}
Recursive Function. Iterates the passed Key to find out necessary values.
private static void AddFoundPortsToList(RegistryKey regKey, ref List<string> ports)
{
try
{
string[] subKeys = regKey.GetSubKeyNames();
if (subKeys != null)
{
foreach (string subKey in subKeys)
{
AddFoundPortsToList(regKey.OpenSubKey(subKey), ref ports);
}
}
if (regKey.Name.EndsWith("Device Parameters"))
{
string str = System.Convert.ToString(regKey.GetValue("PortName"));
if (String.IsNullOrEmpty(str) == false)
{
ports.Add(str);
}
}
}
catch (System.Security.SecurityException ex)
{
;
}
}
The above code works fine, but when I tried to convert it to C++, I'm pretty lost.
Note : I'm using a Win32 Console C++ Program.
I figured out that I can do something like the following to get the Bluetooth Registry Key.
RegOpenKeyEx(HKEY_LOCAL_MACHINE, L"SYSTEM\\CurrentControlSet\\Enum\\Bluetooth", 0, KEY_READ, &hKey)
But after that, I'm pretty lost about the recursive function. Specially, how do I get the available subkeys of the passed registry key when I do NOT know the subkey names?. Or in short, what is the equivalent behavior of RegistryKey.GetSubKeyNames() in C++?
As I am only beginning this thing a code sample with some explanations would be great.

Enumerate the subkeys of a key - RegEnumKeyEx
Enumerate the values of a key - RegEnumValue
You get all subkeys by calling RegEnumKeyEx in a loop until it returns ERROR_NO_MORE_ITEMS.
In the exact same way, you get all values by calling RegEnumValue in a loop until it returns ERROR_NO_MORE_ITEMS.

I'm assuming you want to transition from .NET to native C++ programming. (i.e. no CLI and no .NET framework, which you could still use if you enabled managed C++ compilation).
If you spent a ton of time in C# land, you are probably very used to a ton of very convenient classes for just about everything imaginable and all you have to do is hit "." and let the Intellisense list the methods. Well.... you can forget all those conveniences :)
There is no such (at least not complete) framework in C++ so often you have to turn to Win32 API. MSDN Library is YOUR FRIEND. If you want to get good at C++, learn how to read it and learn how to look things up (not just by name, but learn where different categories are). In this case, if you search for the function you found, you will find a whole set of functions that work on registry. So now, looking at other methods in the same category in MSDN library, you can find RegEnumKeyEx. (hint: make sure to switch MSDN library UI to classical view, that makes it much easier to navigate between topics. I don't know what MS was thinking with their "new" look and feel)
As you start using Win32 API you will realize what a pain it is, especially coming from C#. But you don't have to use it directly (or at least not every time). You can use other libraries, for example ATL provides you with CRegKey class which makes working with registry much simpler. If you can't find the class, do what C++ does best, write your own class. Constantly working directly with windows functions will make your code very, very long and a pain to read.
Another 2 libraries worth knowing as you get into C++ are STL (a must) and Boost (strong should). Boost especially has a lot of OS abstraction so you don't have to go directly to windows DLLs every time.

If you search for the MSDN help on RegOpenKeyEx and then go up in the contents you'll find all the related methods: Registry methods
You probably want to use RegEnumKeyEx to enumerate subkeys.

C# DllImport trouble

My question is a little general, so i'm not looking for an exact answer, but possibly some directions to look into that will help me...
At my work place I program mostly in C#.
We have this 3rd party company we work with, that gave us a Native C++ dll that we need to use.
Since the C++ method I needed wasn't exposed in a manner that was easy to reference from C#, I wrapped the dll in another Native C++ Dll.
So now i have 2 Native C++ dlls, one wrapping the other.
I created a small C# console application that calls the method I created in C++.
My method signature looks like this :
[DllImport("HashMethodWrapper.dll")]
[return: MarshalAs(UnmanagedType.LPStr)]
private static extern string CreateHash(
string input,
[MarshalAs(UnmanagedType.LPStr)]StringBuilder output);
In my console application, everything works fine, and i always receive the string im expecting in the result.
But when I move it to a web service or a Web Application i created (since this is where i really need it), I see that the string im receiving is garbage and not even consistent. It seems as if im getting just some reference to memory that is lost or something like that, but this is only a guess of mine...
I don't know why this happens, since in my console application everything works fine.
Does anyone have a direction that might help me ???...
Thanks in advance,
gillyb
Edit :
I thought it might have to do with some unpinned objects, so i tried calling the method in a fixed statement, something like :
unsafe public static string CreateHashWrap(string pass)
{
String bb;
StringBuilder outPass = new StringBuilder();
fixed (char* resultStr = CreateHash(pass, outPass))
{
bb = new String(resultStr);
}
return bb;
}
...but this still didn't do it for me. Is this the right way to pin objects ?
2nd Edit :
The method signature in C++ looks like this :
extern "C" __declspec(dllexport) char *CreateRsaHash(char *inputPass, char *hashPass);
3rd Edit :
I changed the signature of the method to be
extern "C" __declspec(dllexport) bool CreateRsaHash(char *inputPass, char *hashPass);
and the return value im looking for is placed in the *hashPass parameter.
Now, I created a simple Console application to test it. When insert the DllImport in my main class, and directly call the method everything works great, but when I move the DllImport and wrap the method in a different class and call that class from the Console 'Main' method, I get a StackOverflow exception!
Anyone got any ideas why this is happening ??

Try specifying the capacity of the StringBuilder before passing it to your interop method.

It's really hard to know from the sparse information but if I had to guess I would say you need to make sure you're pinning the output object. Also I would probably change the output parameter to some other type, it seems pretty strange that StringBuilder works at all frankly.
I do know that if you allocate an object, it will get a pointer but that doesn't mean that it won't move. So if you try to pass a pointer to a managed object into an unmanaged environment you need to make sure you tell the GC to "pin" the memory so it doesn't get moved out from under you.
Here is a really rough version of what I mean by pinning:
string input = "...";
StringBuilder output = new StringBuilder();
var handle = System.Runtime.InteropServices.GCHandle.Alloc(output, GCHandleType.Pinned);
try
{
CreateHash(input, output);
}
finally
{
handle.Free();
}

I would consider to warp inside a C# shared assembly/dll instead of a c++ dll and then try to get your console application to work with the dll. It is good practice to wrap external dependencies this way anyway.
Otherwise some traditional issues are 32 vs 64 bit, the load path to the shared library. Is it really only a string or something more complex?

I found the solution to my problem, and now i feel kinda (if not really!) stupid... :-|
I used the LoadLibrary() method in C++ to dynamically invoke a method from the other native dll. The problem was that I didn't give the method any path, and just the dll filename. In .net, it would've searched in the current folder, but seems like in native code this doesn't work this way.
The bigger problem in my programming practices is obviously the fact that i didn't fully cover error handling in my native C++ dll!
All the asnwers I received on this page weren't for nothing though...
Once I found out that i had problem with the directory path, I ran into different exceptions about trying to access corrupt memory, etc. And then I needed to create pinned objects, and declare a size for my StringBuilder object.
Thanks to everyone for your help!!
:)