We are using PInvoke to interop between C# and C++.
I have an interop struct as follows, with an identical layout C++ struct on the other side.
[StructLayout(LayoutKind.Sequential)]
public struct MeshDataStruct : IDisposable
{
public MeshDataStruct(double[] vertices, int[] triangles , int[] surfaces)
{
_vertex_count = vertices.Length / 3;
_vertices = Marshal.AllocHGlobal(_vertex_count*3*sizeof (double));
Marshal.Copy(vertices, 0, _vertices, _vertex_count);
}
// .. extract data methods to double[] etc.
private IntPtr _vertices;
private int _vertex_count;
public void Dispose()
{
if (_vertices != IntPtr.Zero)
{
Marshal.FreeHGlobal(_vertices);
_vertices = IntPtr.Zero;
}
}
}
Now I would like to add a second ctor
public MeshDataStruct(bool filled_in_by_native_codee)
{
_vertex_count = 0;
_vertices = IntPtr.Zero;
}
and then write a method in C++ that allows C++ to fill in the data. This would allow us to use the same structure for input as well as output data...
However, as far as I understand it, AllocHGlobal is available in C# and C++/Cli, but not pure C++.
So my question is: How can I allocate memory in C++ such that I can safely free it on the C# side with a call to Marshal.FreeHGlobal(...)?
This traditionally always ended up poorly, the Microsoft CRT created its own heap with HeapCreate() to service malloc/new calls in a C or C++ program. Can't deallocate such memory in C#, you don't have the heap handle.
That has changed however, starting with the CRT included with VS2012 (msvcr120.dll and up). It now uses the default process heap, the one returned by GetProcessHeap(). Also the one used by Marshal.Alloc/FreeHGlobal(). So you now have a shot at it, provided the native code doesn't use the debug allocator (crtdbg.h). Be careful throwing away that debug option.
The pinvoke marshaller was not changed, nor can it. If it has to release memory, like an array or string returned as a function return value, then it will call CoTaskMemFree(). It is not clear from your question which could apply. In case of doubt and if you have the choice in your native code then you can't go wrong with CoTaskMemAlloc(), paired to Marshal.FreeCoTaskMem() in your C# code.
From the documentation:
AllocHGlobal is one of two memory allocation methods in the Marshal
class. (Marshal.AllocCoTaskMem is the other.) This method exposes the
Win32 LocalAlloc function from Kernel32.dll.
When AllocHGlobal calls LocalAlloc, it passes a LMEM_FIXED flag, which
causes the allocated memory to be locked in place. Also, the allocated
memory is not zero-filled.
So, you can call LocalAlloc from your unmanaged code to allocate memory, and Marshal.FreeHGlobal from your managed code to deallocate it. Likewise, LocalFree can be be used in unmanaged code to deallocate memory allocated with Marshal.AllocHGlobal.
As the documentation also intimates, you could do the same with CoTaskMemAlloc/CoTaskMemFree and Marshal.AllocCoTaskMem/FreeCoTaskMem.
Having said that, you are setting yourself up for a fall doing it this way. It is far cleaner to keep the allocation and deallocation in the same modules. Mixing an matching in this way is very likely to lead to great confusion over who is responsible for deallocating the memory.
Related
I have the following struct declared (C++):
struct NativeOperationResult {
const INTEROP_BOOL Success; // INTEROP_BOOL = char
const char16_t* const ErrorMessage;
NativeOperationResult(const NativeOperationResult& c);
/* various constructors, omitted for brevity */
};
Now, I have an exported function definition elsewhere:
extern "C" __declspec(dllexport) NativeOperationResult ReturnFailureWithMessage() {
return { INTEROP_BOOL_FALSE, "Test" };
}
My expectation is to be calling ReturnFailureWithMessage (a test method in case you were wondering) from C# via P/Invoke. In the NativeOperationResult constructor, it takes a copy of "Test" and puts it in ErrorMessage.
The NativeOperationResult has ownership of the char16_t* so I need to delete it when the struct is destroyed. That's no problem, but I don't want to delete the memory before the .NET CLR has a chance to copy the string in to the managed heap.
Frankly I'm a bit fuzzy on where to delete that memory. What I think is that the C++ compiler will make a copy of my struct (or just move it) and then the CLR will use that copy... Which means I should delete the native memory from .NET with Marshal.FreeHGlobal.
Is that correct?
No, that is not correct. You need to differentiate between two cases:
1) You didn't do any allocations on C++ side. This is the case you are talking about right now.
2) You did do allocations on C++ side, you need to take care of deallocation.
Thus to answer your question: no, your example does not need any "deletion" of memory, since no-one has allocated the memory explictly.
Second case is a bit trickier. If you do memory allocations on C++ side with new char16_t[blah], you need to release the memory with delete[] nativeOperationResult.ErrorMessage. This is not possible to do on C# side. The memory can be allocated using different allocators(such as; malloc, new) and C# does not know how to deal with these pointers.
You need to add a new flag to NativeOperationResult, such as DeletionRequired, and export new function from unmanaged side: FreeNativeOperationResultIfNeeded(..). There is longer discussion here.
You can avoid all this non-sense with C++ strings. They work magically, and no deletion is required.
struct NativeOperationResult {
const INTEROP_BOOL Success; // INTEROP_BOOL = char
const string const ErrorMessage;
NativeOperationResult(const NativeOperationResult& c);
/* various constructors, omitted for brevity */
};
Instead of returning NativeOperationResult, you can make it an out parameter and expect the calling method to pass it the right amount of memory. That way the .Net can allocate memory and then clean it up after the pinvoke is done. But you will have to figure out a way to inform .Net the amount of memory it is expected to allocate.
extern "C" __declspec(dllexport) void ReturnFailureWithMessage(NativeOperationResult* result)
Do I need to do something like this:
TEXTMETRIC tm;
bool isTrueType = false;
if (NativeMethods.GetTextMetrics(hDC, out tm))
{
isTrueType = ((PitchAndFamily)tm.tmPitchAndFamily & PitchAndFamily.TMPF_TRUETYPE) == PitchAndFamily.TMPF_TRUETYPE;
IntPtr ptr = Marshal.AllocHGlobal(Marshal.SizeOf<TEXTMETRIC>(tm));
Marshal.StructureToPtr<TEXTMETRIC>(tm, ptr, true);
Marshal.FreeHGlobal(ptr);
}
or will the allocated memeory be automatically cleaned up once the function exits?
I think (from what I've read) it's the latter, but I'm not sure!
Any elucidation appreciated!
Marshal.StructureToPtr<TEXTMETRIC>(tm, ptr, true);
Using true here is very wrong. The memory you allocated with AllocHGlobal() is not initialized and contains random bytes. It does not contain a previous version of the structure that needs to be released before it is overwritten by the method.
This can technically cause very hard to diagnose random crashes, depending on the random byte values. You got away with it because TEXTMETRIC does not contain any members that require cleanup. The FreeHGlobal() call is sufficient, no need for Marshal.DestroyStructure(), you ought to put it in a finally block so it is exception-safe. Which answers your question.
To be complete, cleanup is only required if the structure contains a BSTR, SAFEARRAY or a COM interface pointer. Resources that have explicit release calls and require the [MarshalAs] attribute in the structure declaration. That's exceedingly rare when you use pinvoke. Not rare at all when you use COM interop, also uses StructureToPtr() under the hood, but the CLR makes the call automatically.
The function you call, GetTextMetrics, expects the caller to allocate and deallocate the memory for the structure. If you allocate with AllocHGlobal you must deallocate with FreeHGlobal.
However all this is unnecessary. You are allocating the structure when you declare tm. There's nothing more needed. Avoid the need to call FreeHGlobal by never calling AllocHGlobal.
TEXTMETRIC tm;
bool isTrueType = false;
if (NativeMethods.GetTextMetrics(hDC, out tm))
{
isTrueType = ((PitchAndFamily)tm.tmPitchAndFamily & PitchAndFamily.TMPF_TRUETYPE) == PitchAndFamily.TMPF_TRUETYPE;
}
Avoiding manual allocation would also allow you to avoid making the broken call, or indeed any call, to StructureToPtr that other answers describe.
No, if you allocate memory using AllocHGlobal you must free it yourself. otherwise that memory will be leaked.
Here is the relevant part from documentation
A pointer to the newly allocated memory. This memory must be released
using the Marshal.FreeHGlobal method.
If you manually allocate unmanaged memory (and you are doing that) you need to manually free it. Add a finally block to make sure that no exception can interfere with freeing memory.
You example doesn't make much sense, though, because you copy tm to a memory block that you never use.
I've read the various MSDN pages on C++ Interop with P/Invoke here and here but I am still confused.
I have some large arrays of doubles that I need to get into native code, and some resulting arrays that need to get back. I do not know the sizes of the output arrays in advance. For simplicity, I will use only a single array in the example. The platform is x64; I read that marshalling internals are quite different between 32- and 64-bit environments so this might be important.
C#
[DllImport("NativeLib.dll")]
public static extern void ComputeSomething(double[] inputs, int inlen,
[Out] out IntPtr outputs, [Out] out int outlen);
[DllImport("NativeLib.dll")]
public static extern void FreeArray(IntPtr outputs);
public void Compute(double[] inputs, out double[] outputs)
{
IntPtr output_ptr;
int outlen;
ComputeSomething(inputs, inputs.Length, out output_ptr, out outlen);
outputs = new double[outlen];
Marshal.Copy(output_ptr, outputs, 0, outlen);
FreeArray(output_ptr);
}
C++
extern "C"
{
void ComputeSomething(double* inputs, int input_length,
double** outputs, int* output_length)
{
//...
*output_length = ...;
*outputs = new double[output_length];
//...
}
void FreeArray(double* outputs)
{
delete[] outputs;
}
}
It works, that is, I can read out the doubles I wrote into the array on the C++ side. However, I wonder:
Is this really the right way to use P/Invoke?
Aren't my signatures needlessly complicated?
Can P/Invoke be used more efficiently to solve this problem?
I believe I read that marshalling for single dimensional arrays of built-in types can be avoided. Is there a way around Marshal.Copy?
Note that we have a working C++/Cli version, but there are some problems related to local statics in third-party library code that lead to crashes. Microsoft marked this issue as WONTFIX, which is why I am looking for alternatives.
It is okayish. The complete lack of a way to return an error code is pretty bad, that's going to hurt when the arrays are large and the program runs out of memory. The hard crash you get is pretty undiagnosable.
The need to copy the arrays and to explicitly release them doesn't win any prizes of course. You solve that by letting the caller pass a pointer to its own array and you just write the elements. You however need a protocol to let the caller figure out how large the array needs to be, that is going to require calling the method twice. The first call returns the required size, the second call gets the job done.
A boilerplate example would be:
[DllImport("foo.dll")]
private static int ReturnData(double[] data, ref int dataLength);
And a sample usage:
int len = 0;
double[] data = null;
int err = ReturnData(data, ref len);
if (err == ERROR_MORE_DATA) { // NOTE: expected
data = new double[len];
err = ReturnData(data, len);
}
No need to copy, no need to release memory, good thing. The native code can corrupt the GC heap if it doesn't pay attention to the passed len, not such a good thing. But of course easy to avoid.
If it were practical to separate the code that determines the output length from the code that populates the output then you could:
Export a function that returned the output length.
Call that from the C# code and then allocate the output buffer.
Call the unmanaged code again, this time asking it to populate the output buffer.
But I'm assuming that you have rejected this option because it is impractical. In which case your code is a perfectly reasonable way to solve your problem. In fact I would say that you've done a very good job.
The code will work just the same in x86 once you fix the calling convention mismatch. On the C++ side the calling convention is cdecl, but on the C# side it is stdcall. That doesn't matter on x64 since there is only one calling convention. But it would be a problem under x86.
Some comments:
You don't need to use [Out] as well as out. The latter implies the former.
You can avoid exporting the deallocator by allocating off a shared heap. For instance CoTaskMemAlloc on the C++ side, and then deallocate with Mashal.FreeCoTaskMem on the C# side.
If you knew the array size beforehand, you could write a C++/CLI DLL that takes the managed array as parameter, pins it, and calls the native C++ DLL on the pinned pointer it obtains.
But if it's output-only, I don't see any version without a copy. You can use a SAFEARRAY so P/Invoke does the copying instead of you, but that's all.
I'm writing a C# library where the calling app will pass in a large amount of contiguous, unmanaged memory. This calling app can be either from .Net or Visual C++ (it will go through an intermediate C++/CLI library before calling my library if from C++). It would be useful to validate that there is sufficient memory, so I decided to call the _msize() function. Unfortunately, _msize always seems to give me the wrong size back.
I went back and modified my allocation routine in my sample app and then immediately call _msize. Here is my code snipet:
public unsafe class MyMemory
{
/// <returns></returns>
[DllImport("msvcrt.dll", SetLastError = true)]
public static extern int _msize(IntPtr handle);
public static IntPtr MyAlloc(int size)
{
IntPtr retVal = Marshal.AllocHGlobal(size);
...
int memSize = MyMemory._msize(retVal);
if (memSize < size)
{
...
}
return retVal;
}
When I pass in the size 199229440, I get back memSize of 199178885. I've seen similar results for different numbers. It is less than 0.01% off, which I would totally understand if it was over, but the fact is it is under, meaning _msize thinks the allocated memory is less than what was asked for. Anyone have any clue why this is? And any recommendations on what I should do instead would be appreciated as well.
P/Invoke the LocalSize function instead.
_msize is for determining the size of a block allocated with malloc (and its friends). AllocHGlobal is a wrapper around GlobalAlloc or LocalAlloc (depending on what reference you believe; but I think the two are equivalent), and you want the LocalSize function to determine the size of the block that actually returned. So far as I can tell, Marshal doesn't contain a wrapper for LocalSize, but you can call it using P/Invoke.
So it seems like it's only by sheer good luck that _msize is returning anything useful for you at all. Perhaps malloc uses GlobalAlloc (or LocalAlloc), either always or just when asked for large blocks, and requests a bit of extra space for bookkeeping; in which case _msize would be trying to compensate for that.
I have a quite strange problem:
I am testing several function calls to a unmanaged C dll with NUnit. The odd thing is, the test fails when it runs normally, but when i run it with the debugger (even with no break point) it passes fine.
So, has the debugger a wider memory access as the plain NUnit application?
i have isolated the call which fails. its passing back a char pointer to a string, which the marshaller should convert to a C# string. the C side looks like this:
#define get_symbol(a) ((a).a_w.w_symbol->s_name)
EXTERN char *atom_get_symbol(t_atom *a);
...
char *atom_get_symbol(t_atom *a) {
return get_symbol(*a);
}
and the C# code:
[DllImport("csharp.dll", EntryPoint="atom_get_symbol")]
[return:MarshalAs(UnmanagedType.LPStr)]
private static extern string atom_get_symbol(IntPtr a);
the pointer which is returned from c is quite deep inside the code and part of a list. so do i just miss some security setting?
EDIT: here is the exception i get:
System.AccessViolationException : (translated to english:) there was an attempt to read or write protected memory. this might be an indication that other memory is corrupt.
at Microsoft.Win32.Win32Native.CoTaskMemFree(IntPtr ptr)
at ....atom_get_symbol(IntPtr a)
SOLUTION:
the problem was, that the marshaller wanted to free the memory which was part of a C struct. but it sould just make a copy of the string and leave the memory as is:
[DllImport("csharp.dll", EntryPoint="atom_get_symbol")]
private static extern IntPtr atom_get_symbol(IntPtr a);
and then in the code get a copy of the string with:
var string = Marshal.PtrToStringAnsi(atom_get_symbol(ptrToStruct));
great!
This will always cause a crash on Vista and up, how you avoided it at all isn't very clear. The stack trace tells the tale, the pinvoke marshaller is trying to release the string buffer that was allocated for the string. It always uses CoTaskMemFree() to do so, the only reasonable guess at an allocator that might have been used to allocate the memory for the string. But that rarely works out well, C or C++ code almost always uses the CRT's private heap. This doesn't crash on XP, it has a much more forgiving memory manager. Which produces undiagnosable memory leaks.
Notable is that the C declaration doesn't give much promise that you can pinvoke the function, it doesn't return a const char*. The only hope you have is to declare the return type as IntPtr instead of string so the pinvoke marshaller doesn't try to release the pointed-to memory. You'll need to use Marshal.PtrToStringAnsi() to convert the returned IntPtr to a string.
You'll need to test the heck out of it, call the function a billion times to ensure that you don't leak memory. If that test crashes with an OutOfMemoryException then you have a big problem. The only alternative then is to write a wrapper in the C++/CLI language and make sure that it uses the exact same version of the CRT as the native code so that they both use the same heap. Which is tricky and impossible if you don't have the source code. This function is just plain difficult to call from any language, including C. It should have been declared as int atom_get_symbol(t_atom* a, char* buf, size_t buflen) so it can be called with a buffer that's allocated by the client code.