Develop Reference

What's the correct way to marshal a const float**?

I'm trying to read an array that's created by pinvoking a dll function from C#. When I print out the array's contents it's actually full of junk.
I suspect this is happening because I am incorrectly marshalling a const float** to an out IntPtr. How do you properly marshal a const float**?
DLL C++ Interface
int Foo(void *objPtr, uint64_t *resultLen, const float **result);
DLL Import Statement
[DllImport("foo.dll", CharSet = CharSet.Auto)]
public static extern int Foo(IntPtr objPtr, out ulong resultLen, out IntPtr result);
Calling Code
IntPtr objPtr = getObj();
IntPtr result;
ulong resultLen;
int output = Foo(objPtr, out resultLen, out result);

Because there is no way to tell the marshaler the size of the array ahead of time you will have to copy the array manually. So out IntPtr is correct.
Note you will have a problem with very large arrays. See https://msdn.microsoft.com/en-us/library/hh285054(v=vs.110).aspx and How to get around Marshal.Copy (32bit) length limit? . This snippet will use int as the resulting array length. You will need to figure out what to do in your particular case.
Also note your DLL must be responsible for releasing the memory it allocates. See Release unmanaged memory from managed C# with pointer of it .
IntPtr objPtr = getObj();
IntPtr result;
int resultLen;
// call your external function
int output = Foo(objPtr, out resultLen, out result);
// create an array to hold the output data
float[] array = new float[resultLen];
// copy the data
Marshal.Copy(result, array, 0, resultLen);
// since the memory was allocated by the DLL only it knows how to free it
// so call the free function exported by the DLL
FreeBufferAfterFoo(result);

How do I call this c function in c# (unmarshalling return struct)?

I want to use c# interop to call a function from a dll written in c. I have the header files.
Take a look at this:
enum CTMBeginTransactionError {
CTM_BEGIN_TRX_SUCCESS = 0,
CTM_BEGIN_TRX_ERROR_ALREADY_IN_PROGRESS,
CTM_BEGIN_TRX_ERROR_NOT_CONNECTED
};
#pragma pack(push)
#pragma pack(1)
struct CTMBeginTransactionResult {
char * szTransactionID;
enum CTMBeginTransactionError error;
};
struct CTMBeginTransactionResult ctm_begin_customer_transaction(const char * szTransactionID);
How do I call ctm_begin_customer_transaction from c#. The const char * mapps well to string, but despite various attempts (looking at stackoverflow and other sites), I fail to marshal the return structure. If I define the function to return IntPtr it works ok...
Edit
I changed the return type to IntPtr and use:
CTMBeginTransactionResult structure = (CTMBeginTransactionResult)Marshal.PtrToStructure(ptr, typeof(CTMBeginTransactionResult));
but it throws AccessViolationException
I also tried:
IntPtr ptr = Transactions.ctm_begin_customer_transaction("");
int size = 50;
byte[] byteArray = new byte[size];
Marshal.Copy(ptr, byteArray, 0, size);
string stringData = Encoding.ASCII.GetString(byteArray);
stringData == "70e3589b-2de0-4d1e-978d-55e22225be95\0\"\0\0\a\0\0\b\b?" at this point. The "70e3589b-2de0-4d1e-978d-55e22225be95" is the szTransactionID from the struct. Where is the Enum? Is it the next byte?

There's a memory management problem hidden in this struct. Who owns the C string pointer? The pinvoke marshaller will always assume that the caller owns it so it will try to release the string. And passes the pointer to CoTaskMemFree(), same function as the one called by Marshal.FreeCoTaskMem(). These functions use the COM memory allocator, the universal interop memory manager in Windows.
This rarely comes to a good end, C code does not typically use that allocator unless the programmer designed his code with interop in mind. In which case he'd never have used a struct as a return value, interop always works much less trouble-free when the caller supplies buffers.
So you cannot afford to let the marshaller do its normal duty. You must declare the return value type as IntPtr so it doesn't try to release the string. And you must marshal it yourself with Marshal.PtrToStructure().
That however still leaves the question unanswered, who owns the string? There is nothing you can do to release the string buffer, you don't have access to the allocator used in the C code. The only hope you have is that the string wasn't actually allocated on the heap. That's possible, the C program might be using string literals. You need to verify that guess. Call the function a billion times in a test program. If that doesn't explode the program then you're good. If not then only C++/CLI can solve your problem. Given the nature of the string, a "transaction ID" ought to change a lot, I'd say you do have a problem.

I hate to answer my own question, but I found the solution to marshal the resulting struct. The struct is 8 bytes long (4 bytes for the char * and 4 bytes for enum). Marshalling the string does not work automatically, but the following works:
// Native (unmanaged)
public enum CTMBeginTransactionError
{
CTM_BEGIN_TRX_SUCCESS = 0,
CTM_BEGIN_TRX_ERROR_ALREADY_IN_PROGRESS,
CTM_BEGIN_TRX_ERROR_NOT_CONNECTED
};
// Native (unmanaged)
[StructLayout(LayoutKind.Sequential, Pack = 1, CharSet = CharSet.Ansi)]
internal struct CTMBeginTransactionResult
{
public IntPtr szTransactionID;
public CTMBeginTransactionError error;
};
// Managed wrapper around native struct
public class BeginTransactionResult
{
public string TransactionID;
public CTMBeginTransactionError Error;
internal BeginTransactionResult(CTMBeginTransactionResult nativeStruct)
{
// Manually marshal the string
if (nativeStruct.szTransactionID == IntPtr.Zero) this.TransactionID = "";
else this.TransactionID = Marshal.PtrToStringAnsi(nativeStruct.szTransactionID);
this.Error = nativeStruct.error;
}
}
[DllImport("libctmclient-0.dll")]
internal static extern CTMBeginTransactionResult ctm_begin_customer_transaction(string ptr);
public static BeginTransactionResult BeginCustomerTransaction(string transactionId)
{
CTMBeginTransactionResult nativeResult = Transactions.ctm_begin_customer_transaction(transactionId);
return new BeginTransactionResult(nativeResult);
}
The code works, but I still need to investigate, if calling the unmanaged code results in memory leaks.

Marshalling a structure in an out returns garbage

I am trying to store and retrieve some data into/from an unmanaged dll. I have tried to narrow down my problem by simplifying the struct as much as possible and here is what I am getting down to:
Structure definition
[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Ansi, Pack = 1)]
public class MyStruct
{
private UInt32 size;
public UInt16 SomeData;
public MyStruct()
{
size = (UInt32)Marshal.SizeOf(this);
this.SomeData = 66; //just put any non 0 value for test
}
}
DLL imports:
[DllImport(MY_DLL, CallingConvention = CallingConvention.Cdecl, CharSet = CharSet.Ansi)]
[return:MarshalAs(UnmanagedType.U1)]
public static extern bool SetData(ref MyStruct ms);
[DllImport(MY_DLL, CallingConvention = CallingConvention.Cdecl, CharSet = CharSet.Ansi)]
public static extern IntPtr GetData();
Function calls:
MyStruct ms_in = new MyStruct();
bool b = Wrapper.SetData(ref ms_in);
IntPtr ptr = Wrapper.GetData();
MyStruct ms_out = (MyStruct)Marshal.PtrToStructure(ptr, typeof(MyStruct));
Simple enough I guess. I know that charset and packing are ok as I simply pasted the struct layout attributes from another struct definition for the same dll as I did for most of the code actually.
When reading the content of ms_out it is just full of garbage (random large numbers).
I finally found the answer to my question by trial and error but I can't understand it much. Here is the working version:
[DllImport(MY_DLL, CallingConvention = CallingConvention.Cdecl, CharSet = CharSet.Ansi)]
[return:MarshalAs(UnmanagedType.I1)]
public static extern bool SetData( [In, MarshalAs(UnmanagedType.LPStruct)] MyStruct ms);
Replacing ref by [In, MarshalAs(UnmanagedType.LPStruct)] did the trick but why?
Thank you for your answers, happy coding.

Now why is the structure being full of garbage on return from the unmanaged dll?
This occurs when I set the data on the unmanaged dll and then get it back but not when I locally create an unmanaged pointer, set the data and read them back:
MyClass x = new MyClass(); //create class
IntPtr ptr2 = Marshal.AllocHGlobal(Marshal.SizeOf(x)); //allocate unmanaged memory
Marshal.StructureToPtr(x, ptr2, false); //marshall to unmanaged memory
MyClass xOut = (MyClass)Marshal.PtrToStructure(ptr2, typeof(MyClass)); //marshall from unmanaged memory
Marshal.FreeHGlobal(ptr2); //free unmanaged memory
If your data "survives" the above test then all the StructLayout, charset, marshalling, etc is OK. In my case it is.
If you simply create the data in c# and pass a pointer to it to the unmanaged code you are only certain that the pointer address will be valid, the data it points to will be invalid as soon as the c# variable is out of scope.
The following method will work with any kind of data I tested including strings:
[return:MarshalAs(UnmanagedType.U1)]
public static extern bool SetData( IntPtr data); //no marshalling in, no ref, no problem...
MyClass x = new MyClass(); //create class
//...store some data in x....
IntPtr ptrIn = Marshal.AllocHGlobal(Marshal.SizeOf(x)); //allocate unmanaged memory
Marshal.StructureToPtr(x, ptrIn, false); //marshall to unmanaged memory
bool b = Wrapper.SetData(ref ms_in); //store data in unmanaged dll
IntPtr ptrOut = Wrapper.GetData(); //get data back from unmanaged dll
MyClass xOut = (MyClass)Marshal.PtrToStructure(ptrOut, typeof(MyClass)); //marshall from unmanaged memory
Marshal.FreeHGlobal(ptrIn); //free unmanaged memory
The problem here is that the caller must release the memory allocated. In my scenario this is used to pass the data on to yet another unmanaged dll (don't ask...) but the calling program has no way to make sure that the final recipient has actually read the data.
Who is going to take care of cleaning up the memory in the end?
I will probably go with the callee cleaning up the data and the caller checking that the data is released/cleaning if not before creating a new block and same thing on exit unless I find a scenario a bit simpler.

Investigating this further I found that marshalling can be used with both c# structs or classes as destination.
Using a struct:
[StructLayout...
struct MyStruct
{
//some properties
//can't have parameterless constructor
public void MakeStruct()
{
size = ...;
//initialize properties as needed
}
}
...
public static extern bool SetData(ref MyStruct ms); //ref is ok for struct, equivalent to c &struct
Using a class:
[StructLayout...
class MyClass
{
//some properties
//parameterless constructor
public void MyClass()
{
size = ...;
//initialize properties as needed
}
}
...
public static extern bool SetData([In, MarshalAs(UnmanagedType.LPStruct)] MyClass ms); //no ref for class
Using a class has the advantage of automatically setting the size in the constructor instead of the caller having to set it explicitly with the struct version.

AccessViolationException in P/Invoke call

I'm writing a small zlib wrapper via P/Invoke calls. It runs perfectly on a 64-bit target (64-bit C# build, 64-bit DLL), but throws an AccessViolationException on a 32-bit target (32-bit C# build, 32-bit DLL).
Here's the C# signature and code which throws the exception:
[DllImport(Program.UnmanagedDll, CallingConvention = CallingConvention.Cdecl)]
private static extern ZLibResult ZLibDecompress(byte[] inStream, uint inLength, byte[] outStream, ref uint outLength);
internal enum ZLibResult : byte {
Success = 0,
Failure = 1,
InvalidLevel = 2,
InputTooShort = 3
}
internal static ZLibResult Decompress(byte[] compressed, out byte[] data, uint dataLength) {
var len = (uint) compressed.Length;
fixed (byte* c = compressed) {
var buffer = new byte[dataLength];
ZLibResult result;
fixed (byte* b = buffer) {
result = ZLibDecompress(c, len, b, &dataLength);
}
if(result == ZLibResult.Success) {
data = buffer;
return result;
}
data = null;
return result;
}
}
And here's the C code (compiled with MinGW-w64):
#include <stdint.h>
#include "zlib.h"
#define ZLibCompressSuccess 0
#define ZLibCompressFailure 1
__cdecl __declspec(dllexport) uint8_t ZLibDecompress(uint8_t* inStream, uint32_t inLength,
uint8_t* outStream, uint32_t* outLength)
{
uLongf oL = (uLongf)*outLength;
int result = uncompress(outStream, &oL, inStream, inLength);
*outLength = (uint32_t)oL;
if(result == Z_OK)
return ZLibCompressSuccess;
return ZLibCompressFailure;
}
I've looked over everything and can't figure out why an access violation would be happening on a 32-bit build and not on a 64-bit build. ZLibDecompress works fine decompressing the same stream when called from a C app, but throws an access violation when called from my C# app.
Does anyone know why this could be happening?
EDIT:
Updated my code, still getting an access violation on 32-bit builds, but not 64-bit.
C# Code:
[DllImport(Program.UnmanagedDll, CallingConvention = CallingConvention.Cdecl)]
private static extern ZLibResult ZLibDecompress(
[MarshalAs(UnmanagedType.LPArray)]byte[] inStream, uint inLength,
[MarshalAs(UnmanagedType.LPArray)]byte[] outStream, ref uint outLength);
internal static ZLibResult Decompress(byte[] compressed, out byte[] data, uint dataLength) {
var buffer = new byte[dataLength];
var result = ZLibDecompress(compressed, (uint)compressed.Length, buffer, ref dataLength);
if(result == ZLibResult.Success) {
data = buffer;
return result;
}
data = null;
return result;
}
C Code:
__declspec(dllexport) uint8_t __cdecl ZLibDecompress(uint8_t* inStream, uint32_t inLength,
uint8_t* outStream, uint32_t* outLength) {
uLongf oL = (uLongf)*outLength;
int result = uncompress(outStream, &oL, inStream, inLength);
*outLength = (uint32_t)oL;
if(result == Z_OK)
return ZLibCompressSuccess;
return ZLibCompressFailure;
}

fixed (byte* b = buffer) {
result = ZLibDecompress(c, len, b, &dataLength);
}
No, that can't work. The fixed keyword provides a highly optimized way to ensure that the garbage collector moving objects doesn't cause trouble. It doesn't do it by pinning the object (like the documentation says), it does it by exposing the b variable to the garbage collector. Which then sees it referencing the buffer and updates the value of b when it moves buffer.
That however can't work in this case, a copy of the b value was passed to ZlibDecompress(). The garbage collector cannot update that copy. The outcome will be poor when a GC occurs while ZLibDecompress() is running, the native code will destroy the integrity of the garbage collected heap and that will eventually cause an AV.
You cannot use fixed, you must use GCHandle.Alloc() to pin the buffer.
But don't do that either, you are helping too much. The pinvoke marshaller is already very good at pinning objects when necessary. Declare the instream and outstream arguments as byte[] instead of byte*. And pass the arrays directly without doing anything special. Also, the outlength argument should be declared ref int.

In 64-bit there's only one ABI for Windows (no cdecl/stdcall), so the problem for 32-bit seems to be with the calling conventions. Your parameter pointers are going into wrong registers and the native function accesses the wrong memory region.
To resolve the issue:
Try commenting out the lines in the native function (see if it crashes - it yes, it's not the calling convention)
Try playing with the calling conventions "cdecl/stdcall"
To check everything, try dumping the pointer values and see if they coincide in native/managed functions.
EDIT:
Then it is a problem with pointers. You are allocating the arrays in C# (thus they reside in a managed heap). You have to marshal them using the "[MarshalAs(UnmanagedType.LPArray)]" attribute.
[DllImport(Program.UnmanagedDll, CallingConvention = CallingConvention.Cdecl)]
private static extern ZLibResult ZLibDecompress(
[MarshalAs(UnmanagedType.LPArray)] byte[] inStream,
uint inLength,
[MarshalAs(UnmanagedType.LPArray)] byte[] outStream,
ref UInt32 outLength);
The [In,Out] modifier might be of help also.
And yes, as Hans says, pin the pointers and do not allow them to be garbage-collected.
byte[] theStream = new byte[whateveyouneed];
// Pin down the byte array
GCHandle handle = GCHandle.Alloc(theStream, GCHandleType.Pinned);
IntPtr address = handle.AddrOfPinnedObject();
and then pass it as IntPtr.

The actual issue was caused by MinGW-w64 generating a buggy DLL. I had been passing -ftree-vectorize to gcc when building zlib, which was generating code that the 32-bit CLR didn't like. The code ran fine after using less aggressive optimization options.

converting a byte array from a native method to a managed structure

I have a c# .net 2.0 CF application that interfaces with a native DLL implementing a function like this:
struct NATIVE_METHOD_REPLY
{
int other_irrelevant_data;
int data_size;
void* data;
}
// reply_buffer will contain an array of NATIVE_METHOD_REPLY structures
// and their data.
//
// returns an error code
int Foo(NATIVE_METHOD_REPLY* reply_buffer, int reply_size);
I've implemented it in C# like this:
[StructLayout(LayoutKind.Sequential)]
internal struct NATIVE_METHOD_REPLY
{
public Int32 OtherIrrelevantData;
public Int16 DataSize;
public IntPtr DataPtr;
}
[DllImport("my_lib.dll", SetLastError = true)]
internal static extern Int32 Foo(byte[] replyBuffer, Int32 replySize);
public byte[] void Bar()
{
// data returned to the user. May be an arbitrary size.
byte[] result_buffer = new byte[256];
// data sent to Foo()
byte[] reply_buffer =
new byte[Marshal.SizeOf(typeof(NativeMethods.NATIVE_METHOD_REPLY)) +
result_buffer.Length];
NativeMethods.Foo(reply_buffer, reply_buffer.Length);
// is there a better way of doing this?
NativeMethods.NATIVE_METHOD_REPLY reply;
GCHandle pinned_reply = GCHandle.Alloc(reply_buffer,
GCHandleType.Pinned);
try
{
reply = (NativeMethods.NATIVE_METHOD_REPLY)Marshal.PtrToStructure(
pinned_reply.AddrOfPinnedObject(),
typeof(NativeMethods.NATIVE_METHOD_REPLY));
Marshal.Copy(reply.DataPtr, result_buffer, 0, reply.DataSize);
}
finally
{
pinned_reply.Free();
}
// bonus point*: is this okay to do after the Free() call?
int test = reply.OtherIrrelevantData;
return result_buffer;
}
While this works correctly, I would like to know if this is the most efficient / most correct way of implementing this function.
Is there some method converting a managed byte array to a managed structure that doesn't involve an intermediate native handle and a copy? For instance, in C++, I would just do this:
NATIVE_METHOD_REPLY* reply = reinterpret_cast< NATIVE_METHOD_REPLY* >( reply.DataPtr );
*For a bonus point, is it okay to use data in the structure after the native handle has been freed?
Thanks,
PaulH
Edit: Updated solution
[DllImport("my_lib.dll", SetLastError = true)]
internal static extern Int32 Foo(IntPtr replyBuffer, Int32 replySize);
public byte[] void Bar()
{
byte[] result_buffer = new byte[256];
int reply_buffer_len = Marshal.SizeOf(typeof(NativeMethods.NATIVE_METHOD_REPLY)) + result_buffer.Length;
IntPtr reply_buffer = Marshal.AllocCoTaskMem(reply_buffer_len);
NativeMethods.NATIVE_METHOD_REPLY reply;
try
{
NativeMethods.Foo(reply_buffer, reply_buffer_len);
reply = (NativeMethods.NATIVE_METHOD_REPLY)Marshal.PtrToStructure(
reply_buffer,
typeof(NativeMethods.NATIVE_METHOD_REPLY));
Marshal.Copy(reply.DataPtr, result_buffer, 0, reply.DataSize);
}
finally
{
Marshal.FreeCoTaskMem(reply_buffer);
}
return result_buffer;
}

The structure has a fixed size. There's no point in passing an array, just pass the structure:
[DllImport("my_lib.dll", SetLastError = true)]
internal static extern Int32 Foo(out NATIVE_METHOD_REPLY replyBuffer, Int32 replySize);
You do have a memory management problem. Who owns the pointer?
Okay, the structure is actually variable sized and the pointer points into the array. You need nother approach. Simply allocate a chunk of unmanaged memory up front instead of letting the P/Invoke marshaller copy the data into a managed array. Which is in fact a hard requirement since the garbage collector can move the array, invalidating the pointer. Call Marshal.CoTaskMemAlloc() to reserve the memory, you'll have to free it later. And change the first argument of the function to IntPtr (not out).
You'll find that marshaling the structure a lot easier too, no need to pin the memory. Don't forget to Marshal.FreeCoTaskMem() when you're done.

In C# under the full framework, you can marshal the array directly. See Default Marshaling for Arrays. I don't know what the limitations are on the Compact Framework.