Background
Hi all. I don't have a significant amount of c++ experience, but I have a lot of experience in other languages.
What I am trying to accomplish is create wrappers for a C# library to be used in a native C++ application. What I have so far is a C++/CLI wrapper that exposes many of the functions in the C# library so that I can use them in my native C++ application. I have done so in a way like what is demonstrated here:
http://pragmateek.com/using-c-from-native-c-with-the-help-of-ccli-v2/
Question
I am having an issue passing a managed C++/CLI pointer to the C# library. The function in the C# library looks like this:
public bool SomeFunction(out byte[] buffer)
For some reason I am having a hard time passing a pointer to this variable in my C++/CLI wrapper. The function is requesting a managed reference but the & doesn't work on managed references, and I can't seem to get it working by using % either.
I need to grab the byte array from the function, and I am pretty certain I can marshal the data into a type my C++ code can handle. Any suggestions?
Edit: Removed the unclear reference to the C++/CLI code. In response to Hans, I was referencing the C# library successfully, because I have been able to pass data back and forth from the C# library to the native C++ code. Also, I didn't intend for this to sound so mysterious. The C# library is for a microscope camera that I am trying to interface with Micromanager. The particular function I am working on has to do with grabbing the data from the image buffer.
Update
Thanks to Lucas, I have a solution. For whatever reason I thought I needed to pass a pointer to the C# library in my C++/CLI library. My final code for the C++/CLI wrapper was:
public: bool SomeFunction(unsigned char* cBuffer, int* count) {
cli::array<unsigned char>^ buffer;
bool isFull = _referenceToManagedCSharpClass->SomeFunction(buffer);
pin_ptr<unsigned char> pinnedArray = &buffer[0];
cBuffer = pinnedArray;
*count = buffer->Length;
return isFull;
}
It's still not clear why I didn't need to pass a pointer to the C# library, or why I need to use the pin_ptr to store the pointer in cBuffer, but I think I will figure this out with reading. Thanks to everyone that helped me with this.
Suppose you have the following class:
ref class SomeClass
{
public:
static bool SomeFunction([System::Runtime::InteropServices::OutAttribute] array<unsigned char>^% buffer)
{
buffer = gcnew array<unsigned char> { 'H', 'E', 'L', 'L', 'O' };
return true;
}
};
It could be implemented in C#, this doesn't matter as long as you can access it from C++/CLI (add a reference as Hans said).
To get the output value, you simply do the following:
array<unsigned char>^ clrArray;
SomeClass::SomeFunction(clrArray);
clrArray will be set to the value returned by SomeFunction from the out parameter.
If you want to access the raw bytes from native C++, you need to pin the array. As long as it's pinned, you're safe to reference the buffer. Here's an example:
Assuming you have the following native function:
void NativeFunc(unsigned char *data, size_t count)
{
std::copy(data, data + count, std::ostream_iterator<unsigned char>(std::cout, ""));
std::cout << std::endl;
}
You can call:
pin_ptr<unsigned char> pinnedArray = &clrArray[0];
NativeFunc(pinnedArray, clrArray->Length);
You have to consider the data pointer as invalid after pinnedArray goes out of scope. It may get relocated by the GC at any time. If you need pinning for longer periods of time, use a pinned GCHandle, but this is more heavyweight.
Related
I have a .NET program that uses a DLL export to get a name of a user.
public static extern string Name(byte[] buf);
This is the export, and I would really like to not change it, as a lot of code relies on it. So, I would like to know, in C++, how would I convert a char* array to the byte buffer?
I have tried this:
void Name(std::byte buf[256])
{
std::string s{ "test" };
std::byte* ptr = reinterpret_cast<std::byte*>(s.data());
buf = ptr;
return;
}
When I print out the string that I convert, I get nothing, it is empty.
Your C++ function implementation does not match the expectations of the C# function declaration.
The C# byte array is marshalled into the function as a pinned pointer to the array's raw memory. The syntax you are using in the C++ code for the parameter (std::byte buf[256]) is just syntax sugar, the compiler actually treats it as a pointer (std::byte* buf). Which is fine in this situation. However, your C++ function is not actually copying anything into the memory that the pointer is pointing at. You are simply changing the pointer itself to point at a different memory address. And the pointer itself is a local variable, so when the function exits, the pointer will no longer exist, and it won't matter what it is pointing at.
Also, the C# declaration is expecting the function to return something that can be marshalled to a .NET string, but the C++ function is not actually return'ing anything at all. The default marshalling behavior for a string return value is as UnmanagedType.LPStr, so the C++ function needs to return a pointer to a null-terminated char* string. The memory for that string must be allocated with CoTaskMemAlloc(), as the marshaller will take ownership of the memory and free it with CoTaskMemFree() after converting the char data to a string.
Also, the C++ function has no calling convention defined, so it is going to use the compiler's default (which is usually __cdecl, unless you change your compiler's settings). However, the default calling convention that .NET's DllImport expects the C++ function to use is __stdcall instead (for compatibility with Win32 APIs). This mismatch won't matter in 64bit, but it matters alot in 32bit.
Without changing your C# declaration, try this on the C++ side:
char* __stdcall Name(std::byte buf[256])
{
std::string s{ "test" };
size_t size = s.size() + 1;
memcpy(buf, s.c_str(), size);
void *ptr = CoTaskMemAlloc(size);
memcpy(ptr, s.c_str(), size);
return ptr;
}
That being said, it is weird to have a function that returns a string in both a byte array output parameter and in a return value. Are you sure the byte array is not meant to be used as an input parameter instead, where the function parses its content to extract amd a string? That would make more sense. It would really help if you would update your question to show how your .NET code is actually calling the C++ function and using the byte array.
When you write
void Name(std::byte buf[256])
that declares buf as pointer to 256 bytes, not an array. So when you later write
buf = ptr;
all you are doing is changing the local variable buf to now point at ptr. And at the end of the function the local variable dies.
Use std::array<std::byte, 256>& and copy the string contents into that. Or even better return a freshly made array instead of an in/out parameter.
I have an array of arrays of this struct (shown here in C#, but existing in C++ as well):
[StructLayout(LayoutKind.Sequential)]
public struct MyStruct
{
IntPtr name; //pointer to string, char* on C++ side
long pValues;
long jValues;
long eValues;
long kValues;
int cost;
};
and an algorithm in a C++ DLL that does work on it, being called from managed C# code. It's CPU-heavy, which is what necessitates this as it runs much faster in C++ than C#. The managed (C#) side never has to know the contents of the struct data, as the algorithm only returns a single array of ints.
So, how would I go about storing this data in the most efficient way (ie with the least overhead), for the lifetime of the application? I think I have it narrowed down to two options:
Initialize structs and set values in C#, pin memory with GCHandle and pass reference to C++ whenever I want to do work (see this post on Unity forums)
Initialize structs and set values in C++, have structs persist in memory on unmanaged side
So my questions are very specific:
With 1, I'm confused as to how marshalling works. It looks like in MSDN: Copying and Pinning that you are able to pass arrays of structures by pinning and passing a reference to the pinned data, without having to copy or convert any of it (and as long as the struct looks the same on both sides). Am I reading that correctly, is that how it actually works? Referring to the Unity3d forum post, I see Marshal.PtrToStructure being called; I thought that performs copying operations? As the data would be stored on the managed side in this instance, having to copy and/or convert the data every time the C++ function is called would cause a lot of overhead, unless I'm thinking that those type of operations are a lot more expensive than they actually are.
With 2, I'm wondering if it's possible to have persistence between C++ calls. To the best of my knowledge, if you're P/Invoking from a DLL, you can't have persistent data on the unmanaged side, so I can't just define and store my struct arrays there, making the only data transferred between managed and unmanaged the int array resulting from the unmanaged algorithm. Is this correct?
Thank you very much for taking the time to read and help!
If the C# code does not need to know the internals of the array and the structure, don't expose it to the C# code. Do all the work on this type in the unmanaged code and avoid marshalling overhead.
Essentially, you want to follow this basic pattern. I'm sure the details will differ, but this should give you the basic concept.
C++
MyStruct* newArray(const int len)
{
return new MyStruct[len];
}
void workOnArray(MyStruct* array, const int len)
{
// do stuff with the array
}
void deleteArray(const MyStruct* array)
{
delete[] array;
}
C#
[DllImport(dllname)]
static extern IntPtr newArray(int len);
[DllImport(dllname)]
static extern void workOnArray(IntPtr array int len);
[DllImport(dllname)]
static extern void deleteArray(IntPtr array);
I've read the various MSDN pages on C++ Interop with P/Invoke here and here but I am still confused.
I have some large arrays of doubles that I need to get into native code, and some resulting arrays that need to get back. I do not know the sizes of the output arrays in advance. For simplicity, I will use only a single array in the example. The platform is x64; I read that marshalling internals are quite different between 32- and 64-bit environments so this might be important.
C#
[DllImport("NativeLib.dll")]
public static extern void ComputeSomething(double[] inputs, int inlen,
[Out] out IntPtr outputs, [Out] out int outlen);
[DllImport("NativeLib.dll")]
public static extern void FreeArray(IntPtr outputs);
public void Compute(double[] inputs, out double[] outputs)
{
IntPtr output_ptr;
int outlen;
ComputeSomething(inputs, inputs.Length, out output_ptr, out outlen);
outputs = new double[outlen];
Marshal.Copy(output_ptr, outputs, 0, outlen);
FreeArray(output_ptr);
}
C++
extern "C"
{
void ComputeSomething(double* inputs, int input_length,
double** outputs, int* output_length)
{
//...
*output_length = ...;
*outputs = new double[output_length];
//...
}
void FreeArray(double* outputs)
{
delete[] outputs;
}
}
It works, that is, I can read out the doubles I wrote into the array on the C++ side. However, I wonder:
Is this really the right way to use P/Invoke?
Aren't my signatures needlessly complicated?
Can P/Invoke be used more efficiently to solve this problem?
I believe I read that marshalling for single dimensional arrays of built-in types can be avoided. Is there a way around Marshal.Copy?
Note that we have a working C++/Cli version, but there are some problems related to local statics in third-party library code that lead to crashes. Microsoft marked this issue as WONTFIX, which is why I am looking for alternatives.
It is okayish. The complete lack of a way to return an error code is pretty bad, that's going to hurt when the arrays are large and the program runs out of memory. The hard crash you get is pretty undiagnosable.
The need to copy the arrays and to explicitly release them doesn't win any prizes of course. You solve that by letting the caller pass a pointer to its own array and you just write the elements. You however need a protocol to let the caller figure out how large the array needs to be, that is going to require calling the method twice. The first call returns the required size, the second call gets the job done.
A boilerplate example would be:
[DllImport("foo.dll")]
private static int ReturnData(double[] data, ref int dataLength);
And a sample usage:
int len = 0;
double[] data = null;
int err = ReturnData(data, ref len);
if (err == ERROR_MORE_DATA) { // NOTE: expected
data = new double[len];
err = ReturnData(data, len);
}
No need to copy, no need to release memory, good thing. The native code can corrupt the GC heap if it doesn't pay attention to the passed len, not such a good thing. But of course easy to avoid.
If it were practical to separate the code that determines the output length from the code that populates the output then you could:
Export a function that returned the output length.
Call that from the C# code and then allocate the output buffer.
Call the unmanaged code again, this time asking it to populate the output buffer.
But I'm assuming that you have rejected this option because it is impractical. In which case your code is a perfectly reasonable way to solve your problem. In fact I would say that you've done a very good job.
The code will work just the same in x86 once you fix the calling convention mismatch. On the C++ side the calling convention is cdecl, but on the C# side it is stdcall. That doesn't matter on x64 since there is only one calling convention. But it would be a problem under x86.
Some comments:
You don't need to use [Out] as well as out. The latter implies the former.
You can avoid exporting the deallocator by allocating off a shared heap. For instance CoTaskMemAlloc on the C++ side, and then deallocate with Mashal.FreeCoTaskMem on the C# side.
If you knew the array size beforehand, you could write a C++/CLI DLL that takes the managed array as parameter, pins it, and calls the native C++ DLL on the pinned pointer it obtains.
But if it's output-only, I don't see any version without a copy. You can use a SAFEARRAY so P/Invoke does the copying instead of you, but that's all.
I have a C# array<System::Byte> and I want that to translate to a C++ byte*. How can I do that? I am using C++/CLR because it lets me use managed/unmanaged code in the same project. I'm basically writing a DLL and making a few methods that can be called via C#, but that contain unmanaged C++ code.
So basically, my C++/CLR method header is this:
void testMethod(array<Byte>^ bytesFromCSharp);
and inside of that testMethod I would like to translate the bytesFromCSharp into a byte* which can be used by other unmanaged C++ code. I malloced the byte* array and wrote a for loop to copy byte by byte but it feels like there should be a better solution.
edit: Example of Hans' technique from his answer below:
//C#
byte[] myBytes = {1,2,3,4};
//C++/CLI
void testMethod(array<byte>^ myBytes) {
pin_ptr<byte> thePtr = &myBytes[0];
byte* bPtr = thePtr;
nativeCode(bPtr);
...
}
Use the pin_ptr<> template class to pin the array and generate a pointer that native code can use. Pinning it ensures that the array cannot be moved by the garbage collector while the native code is using it.
Just make sure that the native code cannot use the array anymore after the pin_ptr<> variable goes out of scope. Which also means that native code storing the pointer for later use is not okay. If it does then you have to make a copy.
After posting a question yesterday I thought I had this cleared up but I'm still having problems, I have a C++/CLI wrapper for a C++ class, some functions of the C++ class take buffers for recv as parameters, the packet structures are defined as C++ structs and that is what is taken as a parameter.
In C# I have replicated these C++ structs using structlayout so that I have equivalent structs in C# which are laid out the same in memory as my C++ structs. In my C++/CLI code I attempted the following
UINT GetValues(value class^ JPVals) // value class, as C# structs are value types
{
IntPtr ptr;
Marshal::StructureToPtr(JPVals,ptr,false);
return m_pComms->GetValues(ptr,0); // m_pComms is a wrapped unmanaged class
//GetValues takes a pointer to a C++ struct
}
The error I get is cannot convert parameter 1 from 'System::IntPtr' to 'SJPVal *', why is it not possible to Marshall from value class to C++ struct pointer? And in this case what should I be passing in and how should I be marshalling it?
You didn't get the serialization process:
// !! Note the % !!
UINT GetValues(value class% JPVals) // value class, as C# structs are value types
{
// Allocate a buffer for serialization, pointer to NULL otherwise
IntPtr ptr = Marshal::AllocHGlobal(Marshal::SizeOf(JPVals));
try {
// Serialize the managed object to "static" memory (not managed by the GC)
Marshal::StructureToPtr(JPVals, ptr, false);
// Pass it to unmanaged code that will modify it.
auto ret = m_pComms->GetValues(reinterpret_cast<SJPVal*>(ptr.ToPointer()), 0);
// Copies the modifications back
Marshal::PtrToStructure(ptr, JPVals);
// Free resources
Marshal::FreeHGlobal(ptr);
return ret;
} catch (...) {
// Make sure we free the memory
Marshal.FreeHGlobal(ptr);
throw;
}
}
EDIT: shown how to copy back the value.
As you are using a C# struct you need to pass it by reference to make sure the changes are copied back. Alternatively, the code will work the same with a C# class. The first step (StructureToPtr) is probably useless, now, since you probably don't care about what was in there before your call to GetValues.
By the way your naming convention is a bit bad. You should NOT start variable names by a capital letter in C++.