After posting a question yesterday I thought I had this cleared up but I'm still having problems, I have a C++/CLI wrapper for a C++ class, some functions of the C++ class take buffers for recv as parameters, the packet structures are defined as C++ structs and that is what is taken as a parameter.
In C# I have replicated these C++ structs using structlayout so that I have equivalent structs in C# which are laid out the same in memory as my C++ structs. In my C++/CLI code I attempted the following
UINT GetValues(value class^ JPVals) // value class, as C# structs are value types
{
IntPtr ptr;
Marshal::StructureToPtr(JPVals,ptr,false);
return m_pComms->GetValues(ptr,0); // m_pComms is a wrapped unmanaged class
//GetValues takes a pointer to a C++ struct
}
The error I get is cannot convert parameter 1 from 'System::IntPtr' to 'SJPVal *', why is it not possible to Marshall from value class to C++ struct pointer? And in this case what should I be passing in and how should I be marshalling it?
You didn't get the serialization process:
// !! Note the % !!
UINT GetValues(value class% JPVals) // value class, as C# structs are value types
{
// Allocate a buffer for serialization, pointer to NULL otherwise
IntPtr ptr = Marshal::AllocHGlobal(Marshal::SizeOf(JPVals));
try {
// Serialize the managed object to "static" memory (not managed by the GC)
Marshal::StructureToPtr(JPVals, ptr, false);
// Pass it to unmanaged code that will modify it.
auto ret = m_pComms->GetValues(reinterpret_cast<SJPVal*>(ptr.ToPointer()), 0);
// Copies the modifications back
Marshal::PtrToStructure(ptr, JPVals);
// Free resources
Marshal::FreeHGlobal(ptr);
return ret;
} catch (...) {
// Make sure we free the memory
Marshal.FreeHGlobal(ptr);
throw;
}
}
EDIT: shown how to copy back the value.
As you are using a C# struct you need to pass it by reference to make sure the changes are copied back. Alternatively, the code will work the same with a C# class. The first step (StructureToPtr) is probably useless, now, since you probably don't care about what was in there before your call to GetValues.
By the way your naming convention is a bit bad. You should NOT start variable names by a capital letter in C++.
Related
I have a .NET program that uses a DLL export to get a name of a user.
public static extern string Name(byte[] buf);
This is the export, and I would really like to not change it, as a lot of code relies on it. So, I would like to know, in C++, how would I convert a char* array to the byte buffer?
I have tried this:
void Name(std::byte buf[256])
{
std::string s{ "test" };
std::byte* ptr = reinterpret_cast<std::byte*>(s.data());
buf = ptr;
return;
}
When I print out the string that I convert, I get nothing, it is empty.
Your C++ function implementation does not match the expectations of the C# function declaration.
The C# byte array is marshalled into the function as a pinned pointer to the array's raw memory. The syntax you are using in the C++ code for the parameter (std::byte buf[256]) is just syntax sugar, the compiler actually treats it as a pointer (std::byte* buf). Which is fine in this situation. However, your C++ function is not actually copying anything into the memory that the pointer is pointing at. You are simply changing the pointer itself to point at a different memory address. And the pointer itself is a local variable, so when the function exits, the pointer will no longer exist, and it won't matter what it is pointing at.
Also, the C# declaration is expecting the function to return something that can be marshalled to a .NET string, but the C++ function is not actually return'ing anything at all. The default marshalling behavior for a string return value is as UnmanagedType.LPStr, so the C++ function needs to return a pointer to a null-terminated char* string. The memory for that string must be allocated with CoTaskMemAlloc(), as the marshaller will take ownership of the memory and free it with CoTaskMemFree() after converting the char data to a string.
Also, the C++ function has no calling convention defined, so it is going to use the compiler's default (which is usually __cdecl, unless you change your compiler's settings). However, the default calling convention that .NET's DllImport expects the C++ function to use is __stdcall instead (for compatibility with Win32 APIs). This mismatch won't matter in 64bit, but it matters alot in 32bit.
Without changing your C# declaration, try this on the C++ side:
char* __stdcall Name(std::byte buf[256])
{
std::string s{ "test" };
size_t size = s.size() + 1;
memcpy(buf, s.c_str(), size);
void *ptr = CoTaskMemAlloc(size);
memcpy(ptr, s.c_str(), size);
return ptr;
}
That being said, it is weird to have a function that returns a string in both a byte array output parameter and in a return value. Are you sure the byte array is not meant to be used as an input parameter instead, where the function parses its content to extract amd a string? That would make more sense. It would really help if you would update your question to show how your .NET code is actually calling the C++ function and using the byte array.
When you write
void Name(std::byte buf[256])
that declares buf as pointer to 256 bytes, not an array. So when you later write
buf = ptr;
all you are doing is changing the local variable buf to now point at ptr. And at the end of the function the local variable dies.
Use std::array<std::byte, 256>& and copy the string contents into that. Or even better return a freshly made array instead of an in/out parameter.
Assume the following:
Let us have a C++ class:
class ExampleClass
{
private:
int var1;
public:
void DoSomething();
}
Assume also following C++ function which we want to call from .NET Core app with PInvoke or using EmitCalli:
extern "C" ExampleClass CreateObject()
So C++ function returns instance of ExampleClass by value. Is there any way to get this instance on managed part as byte array (assuming that size of ExampleClass is known).
As I remember, in most native x86(x64) calling conventions C++ functions which return structures actully have pointer to the structure to fill as one of the parameters. Will this hint work with NET Core: allocate byte array on managed part and pass pointer as first parameter to unmanaged call?
Thanks!
in C++
ExampleClass CreateClass();
will return a block of bytes of sizeof(ExampleClass) and in absence of virtual members and inheritance it will be plain object and equivalent to same struct definition (see more about this here: Structure of a C++ Object in Memory Vs a Struct )
P/Invoke marshaller should be able to deal with this without a problem.
It it not an IntPtr, it's an actual memory block for C and C++.
IntPtr equivalent would be
ExampleClass * CreateClass();
I am not sure why it returns the instance instead of a pointer but they probably had their reasons.
so if you define it
public struct ExampleClass
{
public int Var1;
}
you should be fine (except if class structure is more complex you would need to check what was layout in C++ compilation, specifically padding. For example class with 2 bools may yield size of 16 in some padding).
see more here:
https://learn.microsoft.com/en-us/dotnet/framework/interop/marshaling-classes-structures-and-unions
https://learn.microsoft.com/en-us/dotnet/api/system.runtime.interopservices.structlayoutattribute
I have an array of arrays of this struct (shown here in C#, but existing in C++ as well):
[StructLayout(LayoutKind.Sequential)]
public struct MyStruct
{
IntPtr name; //pointer to string, char* on C++ side
long pValues;
long jValues;
long eValues;
long kValues;
int cost;
};
and an algorithm in a C++ DLL that does work on it, being called from managed C# code. It's CPU-heavy, which is what necessitates this as it runs much faster in C++ than C#. The managed (C#) side never has to know the contents of the struct data, as the algorithm only returns a single array of ints.
So, how would I go about storing this data in the most efficient way (ie with the least overhead), for the lifetime of the application? I think I have it narrowed down to two options:
Initialize structs and set values in C#, pin memory with GCHandle and pass reference to C++ whenever I want to do work (see this post on Unity forums)
Initialize structs and set values in C++, have structs persist in memory on unmanaged side
So my questions are very specific:
With 1, I'm confused as to how marshalling works. It looks like in MSDN: Copying and Pinning that you are able to pass arrays of structures by pinning and passing a reference to the pinned data, without having to copy or convert any of it (and as long as the struct looks the same on both sides). Am I reading that correctly, is that how it actually works? Referring to the Unity3d forum post, I see Marshal.PtrToStructure being called; I thought that performs copying operations? As the data would be stored on the managed side in this instance, having to copy and/or convert the data every time the C++ function is called would cause a lot of overhead, unless I'm thinking that those type of operations are a lot more expensive than they actually are.
With 2, I'm wondering if it's possible to have persistence between C++ calls. To the best of my knowledge, if you're P/Invoking from a DLL, you can't have persistent data on the unmanaged side, so I can't just define and store my struct arrays there, making the only data transferred between managed and unmanaged the int array resulting from the unmanaged algorithm. Is this correct?
Thank you very much for taking the time to read and help!
If the C# code does not need to know the internals of the array and the structure, don't expose it to the C# code. Do all the work on this type in the unmanaged code and avoid marshalling overhead.
Essentially, you want to follow this basic pattern. I'm sure the details will differ, but this should give you the basic concept.
C++
MyStruct* newArray(const int len)
{
return new MyStruct[len];
}
void workOnArray(MyStruct* array, const int len)
{
// do stuff with the array
}
void deleteArray(const MyStruct* array)
{
delete[] array;
}
C#
[DllImport(dllname)]
static extern IntPtr newArray(int len);
[DllImport(dllname)]
static extern void workOnArray(IntPtr array int len);
[DllImport(dllname)]
static extern void deleteArray(IntPtr array);
I have the following struct declared (C++):
struct NativeOperationResult {
const INTEROP_BOOL Success; // INTEROP_BOOL = char
const char16_t* const ErrorMessage;
NativeOperationResult(const NativeOperationResult& c);
/* various constructors, omitted for brevity */
};
Now, I have an exported function definition elsewhere:
extern "C" __declspec(dllexport) NativeOperationResult ReturnFailureWithMessage() {
return { INTEROP_BOOL_FALSE, "Test" };
}
My expectation is to be calling ReturnFailureWithMessage (a test method in case you were wondering) from C# via P/Invoke. In the NativeOperationResult constructor, it takes a copy of "Test" and puts it in ErrorMessage.
The NativeOperationResult has ownership of the char16_t* so I need to delete it when the struct is destroyed. That's no problem, but I don't want to delete the memory before the .NET CLR has a chance to copy the string in to the managed heap.
Frankly I'm a bit fuzzy on where to delete that memory. What I think is that the C++ compiler will make a copy of my struct (or just move it) and then the CLR will use that copy... Which means I should delete the native memory from .NET with Marshal.FreeHGlobal.
Is that correct?
No, that is not correct. You need to differentiate between two cases:
1) You didn't do any allocations on C++ side. This is the case you are talking about right now.
2) You did do allocations on C++ side, you need to take care of deallocation.
Thus to answer your question: no, your example does not need any "deletion" of memory, since no-one has allocated the memory explictly.
Second case is a bit trickier. If you do memory allocations on C++ side with new char16_t[blah], you need to release the memory with delete[] nativeOperationResult.ErrorMessage. This is not possible to do on C# side. The memory can be allocated using different allocators(such as; malloc, new) and C# does not know how to deal with these pointers.
You need to add a new flag to NativeOperationResult, such as DeletionRequired, and export new function from unmanaged side: FreeNativeOperationResultIfNeeded(..). There is longer discussion here.
You can avoid all this non-sense with C++ strings. They work magically, and no deletion is required.
struct NativeOperationResult {
const INTEROP_BOOL Success; // INTEROP_BOOL = char
const string const ErrorMessage;
NativeOperationResult(const NativeOperationResult& c);
/* various constructors, omitted for brevity */
};
Instead of returning NativeOperationResult, you can make it an out parameter and expect the calling method to pass it the right amount of memory. That way the .Net can allocate memory and then clean it up after the pinvoke is done. But you will have to figure out a way to inform .Net the amount of memory it is expected to allocate.
extern "C" __declspec(dllexport) void ReturnFailureWithMessage(NativeOperationResult* result)
Here is a simple C code (VS 2013 C++ project, "compiled as C"):
typedef struct {
bool bool_value;
} BoolContainer;
BoolContainer create_bool_container (bool bool_value)
{
return (BoolContainer) { bool_value };
}
Here is my P/Invoke wrapper
public partial class NativeMethods
{
[DllImport(LibraryName, CallingConvention = CallingConvention.Cdecl)]
public static extern BoolContainer create_bool_container([MarshalAs(UnmanagedType.I1)] bool bool_value);
}
Here are the managed versions of the BoolContainer:
First is throwing MarshalDirectiveException: Method's type signature is not PInvoke compatible.:
public struct BoolContainer // Marshal.SizeOf(typeof(BoolContainer)) = 1
{
[MarshalAs(UnmanagedType.I1)] // same with UnmanagedType.U1
public bool bool_value;
}
Second is working:
public struct BoolContainer // Marshal.SizeOf(typeof(BoolContainer)) = 1
{
public byte bool_value;
}
Test code:
BoolContainer boolContainer = NativeMethods.create_bool_container(true);
Is seems like DllImport ignores MarshalAs for any boolean members in the returned structure. Is there a way to keep bool in the managed declaration?
Structs as return types are pretty difficult in interop, they don't fit a CPU register. This is handled in native code by the caller reserving space on its stack frame for the return value and passing a pointer to the callee. Which then copies the return value through the pointer. In general a pretty troublesome feature, different C compilers have different strategies to pass that pointer. The pinvoke marshaller assumes MSVC behavior.
This is a problem with structs that are not blittable, in other words when its layout is no longer an exact match with the unmanaged struct that the native code needs. That requires an extra step to convert the unmanaged struct to the managed struct, the equivalent of Marshal.PtrToStructure(). You see this for example if you add a string field to your struct. That makes it non-blittable since the C string has to be converted to a System.String, you'll get the exact same exception.
A similar kind of problem in your case, bool is also a very difficult type to interop with. Caused by the C language not having the type and it getting added later with highly incompatible choices. It is 1 byte in C++ and the CLR, 2 bytes in COM, 4 bytes in C and the winapi. The pinvoke marshaller picked the winapi version as the default, making it match BOOL, the most common reason to pinvoke. This forced you to add the [MarshalAs] attribute and that also requires extra work, again to be done by the equivalent of Marshal.PtrToStructure().
So what the pinvoke marshaller is missing is the support for this extra step, custom marshaling of the return value after the call. Not sure if this was a design constraint or a resource constraint. Probably both, ownership of non-blittable struct members is completely unspecified in unmanaged languages and guessing at it, especially as a return value, very rarely turns out well. So they may well have made the call that working on the feature just wasn't worth the low odds of success. This is a guess of course.
Using byte instead is a fine workaround.