Develop Reference

Converting std::vector to array an then p/Invoking it causes a Access Violation Exception in mscorlib.dll during marshalling

In C++ I have the following struct from 3rd-party code:
typedef struct NodeInfoTag
{
long lResult;
int bComplete;
char *pszNodeAddr;
char *pszParentAddr;
RTS_WCHAR *pwszNodeName;
RTS_WCHAR *pwszDeviceName;
RTS_WCHAR *pwszVendorName;
unsigned long ulTargetType;
unsigned long ulTargetId;
unsigned long ulTargetVersion;
unsigned short wMaxChannels;
}NodeInfotyp;
And the definition to RTS_WCHAR:
# ifndef RTS_WCHAR_DEFINED
# define RTS_WCHAR_DEFINED
typedef wchar_t RTS_WCHAR; /* wide character value */
# endif
(So it's basically a wchar_t)
Then I have my own class called CScanNetworkCallback, which extends the CPLCHandlerCallback class, a class from the same vendor:
.h file:
class CScanNetworkCallback : public CPLCHandlerCallback
{
public:
bool bScanComplete;
NodeInfotyp* pNodeInfo;
NodeInfotyp* pNodeInfoList;
std::vector<NodeInfotyp> vList;
CScanNetworkCallback();
virtual ~CScanNetworkCallback(void);
virtual long Notify(CPLCHandler *pPlcHandler, CallbackAddInfoTag CallbackAdditionalInfo);
};
The implementation follows their own guidelines with some of my own stuff thrown in:
CScanNetworkCallback::CScanNetworkCallback(void) : CPLCHandlerCallback()
{
bScanComplete = false;
}
CScanNetworkCallback::~CScanNetworkCallback()
{
delete pNodeInfo;
delete pNodeInfoList;
}
long CScanNetworkCallback::Notify(CPLCHandler *pPlcHandler, CallbackAddInfoTag CallbackAdditionalInfo)
{
if (pPlcHandler != NULL)
{
if (CallbackAdditionalInfo.ulType == PLCH_SCAN_NETWORK_CALLBACK)
{
pNodeInfo = CallbackAdditionalInfo.AddInf.pNodeInfo;
if (pNodeInfo->lResult == RESULT_OK)
{
vList.push_back(*pNodeInfo);
bScanComplete = false;
}
else
{
pNodeInfoList = &vList[0]; //New pointer points to the vector elements, which will be used as an array later on
// I have also tried copying it, to the same result:
//std::copy(vList.begin(), vList.end(), pNodeInfoList);
bScanComplete = true;
}
}
}
return RESULT_OK;
}
So basically, the Notify method in the class above is called every time a "node" is found in the network, assigning the node's information to pNodeInfo (please disregard what a node is, it isn't relevant ATM). Since it is called to every node in the network during the scanning process and I must send this information to C++, I couldn't find any other way to do so other than using a std::vector to store every callback info for latter use, as I don't know how many nodes there will be at compile time. The else part is called after all nodes have been found. In order to make sense out of the C# code, I must describe the implementation of some other C++ methods that are p/Invoked:
PROASADLL __declspec(dllexport) void scanNetwork(){
pScanHandler->ScanNetwork(NULL, &scanNetworkCallback);
}
The object scanNetworkCallback is static. pScanHandler is a pointer to another class from the 3rd party vendor and its ScanNetwork method runs on a separate thread. Internally (and I only know that due to this API Guidelines, I don't have its source code), it calls the Notify method whenever a node is found in the network, or something to that effect
And finally:
PROASADLL __declspec(dllexport) NodeInfotyp* getScanResult(int* piSize) {
*piSize = scanNetworkCallback.vList.size();
return scanNetworkCallback.pNodeInfoList;
}
That returns the pointer that points to all nodes' information and the amount in as an out parameter. Now let's take a look at the C# code:
public static List<NodeInfoTag> AsaScanNetworkAsync()
{
Console.WriteLine("SCANNING NETWORK");
scanNetwork(); // C++ Method
while (!isScanComplete()) // Holds the C# thread until the scan is complete
Thread.Sleep(50);
int size = 0;
IntPtr pointer = getScanResult(out size); // works fine, I get some IntPtr and the correct size
List<NodeInfoTag> list = Marshaller.MarshalPointerToList<NodeInfoTag>(pointer, size); // PROBLEM!!!
// Continue doing stuff
}
This is the class NodeInfoTag, to match the C++ NodeInfotyp struct:
[StructLayout(LayoutKind.Sequential)]
public class NodeInfoTag
{
public int Result;
public int Complete;
[MarshalAs(UnmanagedType.LPStr)] //char*
public string NodeAddress;
[MarshalAs(UnmanagedType.LPStr)] //char*
public string ParentAddress;
[MarshalAs(UnmanagedType.LPWStr)] //wchar_t
public string VendorName;
public uint TargetType;
public uint TargetId;
public uint TargetVersion;
public short MaxChannels;
}
And this is where I get my Memory Access Violation:
internal class Marshaller
{
public static List<T> MarshalPointerToList<T>(IntPtr pointer, int size)
{
if (size == 0)
return null;
List<T> list = new List<T>();
var symbolSize = Marshal.SizeOf(typeof(T));
for (int i = 0; i < size; i++)
{
var current = (T)Marshal.PtrToStructure(pointer, typeof(T));
list.Add(current);
pointer = new IntPtr(pointer.ToInt32() + symbolSize);
}
return list;
}
}
The error occurs specifically when marshaling should take place, at the line var current = (T)Marshal.PtrToStructure(pointer, typeof(T));. This C# code used to work just fine, but the C++ part was terrible, convoluted and error-prone, so I decided to make things more simple but I can't figure out for the life of me why I'm getting this Exception as I'm making sure that all C++ resources are available for C#, since for testing purposes, I don't delete anything in C++ and I'm only using variables with global scope within the class, which is allocated to static memory. So, what did I miss?
Edit: I removed pNodeInfoList = &vList[0]; and rewrote getScanResult as follows:
static NodeInfotyp pNodeInfoList;
//(...)
PROASADLL __declspec(dllexport) NodeInfotyp* getScanResult(int* piSize) {
*piSize = scanNetworkCallback.vList.size();
std::move(scanNetworkCallback.vList.begin(),
scanNetworkCallback.vList.end(), &pNodeInfoList);
return &pNodeInfoList;
}
No dice. I don't use new or malloc in any of the variables involved, and even changed pNodeInfoList (the array) from a class member to a global variable. Also, I'm using move, as I've been told, could be used to solve ownership problems. Any other tips?

Ownership is not part of the naive C++ type system, so you will not get an error when you delete a pointer you do not own or transfer ownership away without giving it up.
However, semantically certain values and pointers and data blocks are owned by certain types or values.
In this case the vector owns its block of memory. There is no way to ask it or make it give up ownership.
Calling .data() onky provides you a pointer, it does not give that pointer semantic ownership.
You store the return value of .data() in a member variable. You later call delete on that member variable. This indicates to me that member variable is supposed to own its data. So you double delete (as both the vector and the pointer think they own the data pointed to), and your compiler crashes the program for you.
You need to rewite your code taking into account liefetime and ownership of every block of memory you are working with. One approach is to never ever call new, malloc or delete or free directly, and always use memory managing types like vector and unique ptr. Avoid persisting raw pointers, as their ownership semantics are not clear from the type.

void ** handles with P/Invoke

I am using a C API DLL from a 3rd party vendor. The problem I have is that I can't seem to find a good template for Marshalling the following C code:
API_Open( void ** handle );
API_Close( void * handle );
The calls are simplified, but the handle is a void *, which is (in C) passed into the API_Open call as &handle, and then passed into API_Close as handle.
I've tried to do the same in C#, but can't figure out how to Marshal handle properly. My C# version (latest try) is:
[DllImport("External.dll",EntryPoint="API_Open")]
public static extern int API_Open( out IntPtr handle );
[DllImport("External.dll",EntryPoint="API_Close")]
public static extern int API_Close( IntPtr handle );
public static int Wrapper_API_Open( ref Int32 handle )
{
int rc = SUCCESS;
// Get a 32bit region to act as our void**
IntPtr voidptrptr = Marshal.AllocHGlobal(sizeof(Int32));
// Call our function.
rc = API_Open(out voidptrptr);
// In theory, the value that voidptrptr points to should be the
// RAM address of our handle.
handle = Marshal.ReadInt32( Marshal.ReadIntPtr(voidptrptr) );
return rc;
}
public static int Wrapper_API_Close(ref Int32 handle)
{
int rc = SUCCESS;
// Get a 32bit region to act as our void *
IntPtr voidptr = Marshal.AllocHGlobal(sizeof(Int32));
// Write the handle into it.
Marshal.WriteInt32(voidptr,handle);
// Call our function.
rc = API_Close(voidptr);
return rc;
}
public void SomeRandomDrivingFunction()
{
.
.
.
Int32 handle;
Wrapper_API_Open( ref handle );
.
.
.
Wrapper_API_Close( ref handle );
.
.
.
}
The API return code is always INVALID_DEVICE_OBJECT when I call API_Close. Any thoughts? I thought this would be pretty straightforward, but I'm having trouble wrapping my head around the void** and void* parts of the function calls.
Thanks

You seem to be over-complicating this significantly. I don't know why you want to introduce Int32 for the handles since they do need to be pointer sized. You should use IntPtr.
The API_Open accepts the address of the variable where the handle is returned. The caller allocates that variable and passes it to the callee, which populates the variable. The C function might look like this:
int API_Open(void **handle)
{
*handle = InternalCreateHandle();
return CODE_SUCCESS;
}
You'd call that in C like this:
void *handle;
int retval = API_Open(&handle);
if (retval != CODE_SUCCESS)
// handle error
// go one and use handle
Translated to C#, the void* maps to IntPtr, and the use of the double pointer is just a means to get around the fact that C only supports pass-by-value. In C# you would use pass-by-reference.
For API_Close it is even simpler because we are passing the handle by value:
int API_Close(void *handle)
{
InternalCloseHandle(handle);
return CODE_SUCCESS;
}
And the calling code is simply:
int retval = API_Close(handle);
if (retval != CODE_SUCCESS)
// handle error
So, the C# wrapper functions should be:
public static int Wrapper_API_Open(out IntPtr handle)
{
return API_Open(out handle);
}
public static int Wrapper_API_Close(IntPtr handle)
{
return API_Close(handle);
}
At which point these wrapper methods do look somewhat pointless!

How do I write a custom marshaler which allows data to flow from native to managed?

In attempting to write a custom marshaler related to this question (P/Invoke from C to C# without knowing size of array), I have come across something I cannot understand. This is the first ever custom marshaler that I have written so no doubt I'm missing something obvious due to my ignorance.
Here's my C# code:
using System;
using System.Runtime.InteropServices;
using System.Text;
namespace CustomMarshaler
{
public class MyCustomMarshaler : ICustomMarshaler
{
static MyCustomMarshaler static_instance;
public IntPtr MarshalManagedToNative(object managedObj)
{
if (managedObj == null)
return IntPtr.Zero;
if (!(managedObj is int[]))
throw new MarshalDirectiveException("VariableLengthArrayMarshaler must be used on an int array.");
int[] arr = (int[])managedObj;
int size = sizeof(int) + arr.Length * sizeof(int);
IntPtr pNativeData = Marshal.AllocHGlobal(size);
Marshal.WriteInt32(pNativeData, arr.Length);
Marshal.Copy(arr, 0, pNativeData + sizeof(int), arr.Length);
return pNativeData;
}
public object MarshalNativeToManaged(IntPtr pNativeData)
{
int len = Marshal.ReadInt32(pNativeData);
int[] arr = new int[len];
Marshal.Copy(pNativeData + sizeof(int), arr, 0, len);
return arr;
}
public void CleanUpNativeData(IntPtr pNativeData)
{
Marshal.FreeHGlobal(pNativeData);
}
public void CleanUpManagedData(object managedObj)
{
}
public int GetNativeDataSize()
{
return -1;
}
public static ICustomMarshaler GetInstance(string cookie)
{
if (static_instance == null)
{
return static_instance = new MyCustomMarshaler();
}
return static_instance;
}
}
class Program
{
[DllImport(#"MyLib.dll")]
private static extern void Foo(
[In, Out, MarshalAs(UnmanagedType.CustomMarshaler, MarshalTypeRef = typeof(MyCustomMarshaler))]
int[] arr
);
static void Main(string[] args)
{
int[] colorTable = new int[] { 1, 2, 3, 6, 12 };
Foo(colorTable);
foreach (int value in colorTable)
Console.WriteLine(value);
}
}
}
On the other side is a trivial native DLL, written in Delphi as it happens.
library MyLib;
procedure Foo(P: PInteger); stdcall;
var
i, len: Integer;
begin
len := P^;
Writeln(len);
for i := 1 to len do begin
inc(P);
Writeln(P^);
inc(P^);
end;
end;
exports
Foo;
begin
end.
The idea is that the array is passed to the DLL which then prints out the length field, and the values of the array. The native code also increments each value of the array by 1.
So, I expect to see this output:
5
1
2
3
6
12
2
3
4
7
13
But unfortunately I see this output:
5
1
2
3
6
12
1
2
3
6
12
Under the debugger I can see that MarshalNativeToManaged is executing, and that the values that it returns have been incremented. But these incremented values don't find there way back into the object that is passed to Foo.
What do I need to do to fix this?

I had a similar problem many years ago and found that there was very little documentation on Custom Marshaling. I suspect using ICustomMarshaler never really took off since it can always be done using manual marshaling in the course of your regular code. And so there was never really a need for any documentation of advanced custom marshaling scenarios.
Anyway, through a variety of sources and much trial and error I think I teased out a practical understanding of how most of Custom Marshaling works.
In your case, you have set up the ManagedToNative method correctly for [In] marshaling and the NativeToManaged method correctly for most [Out] marshaling but [In, Out] marshaling is actually a bit trickier. [In, Out] marshaling is actually in-place marshaling. So on the way back out you must marshal the data back to the same instance that was provided in the [In] side of the operation.
There are a number of small variations on this depending on whether using reference or value types, whether the call is a normal pInvoke call or a callback on a delegate, etc. But thinking about what needs to end up where is the key.
The following variation on your code works the way you want it to (and it seems to works the same way for .Net 2.0 and up):
//This must be thread static since, in theory, the marshaled
//call could be executed simultaneously on two or more threads.
[ThreadStatic] int[] marshaledObject;
public IntPtr MarshalManagedToNative(object managedObj)
{
if (managedObj == null)
return IntPtr.Zero;
if (!(managedObj is int[]))
throw new MarshalDirectiveException("VariableLengthArrayMarshaler must be used on an int array.");
//This is called on the way in so we must keep a reference to
//the original object so we can marshal to it on the way out.
marshaledObject = (int[])managedObj;
int size = sizeof(int) + marshaledObject.Length * sizeof(int);
IntPtr pNativeData = Marshal.AllocHGlobal(size);
Marshal.WriteInt32(pNativeData, marshaledObject.Length);
Marshal.Copy(marshaledObject, 0, (IntPtr)(pNativeData.ToInt64() + sizeof(int)), marshaledObject.Length);
return pNativeData;
}
public object MarshalNativeToManaged(IntPtr pNativeData)
{
if (marshaledObject == null)
throw new MarshalDirectiveException("This marshaler can only be used for in-place ([In. Out]) marshaling.");
int len = Marshal.ReadInt32(pNativeData);
if (marshaledObject.Length != len)
throw new MarshalDirectiveException("The size of the array cannot be changed when using in-place marshaling.");
Marshal.Copy((IntPtr)(pNativeData.ToInt64() + sizeof(int)), marshaledObject, 0, len);
//Reset to null for next call;
marshalledObject = null;
return marshaledObject;
}

int len = Marshal.ReadInt32(pNativeData);
int[] arr = new int[len];
Your problem is located here, you are creating a new array. But you need to update the colorTable array instead. You got a reference to it in your MarshalManagedToNative() method, you'll need to store it so you can use it again in your MarshalNativeToManaged() method.
Note that this has a number of consequences. Your custom marshaller object becomes stateful, you can't use a static instance anymore. And you'll need a different approach if the unmanaged code modifies the array length. You allow for this by reading the len back but didn't actually implement it, so it's okay. Just assert that the length didn't change.

Thanks a lot to Stephen and Hans for their excellent answers. I can see clearly now that I must keep hold of the managed object that was passed to MarshalManagedToNative, and then return the same object from the subsequent call to MarshalNativeToManaged.
It's a bit of a bind that the framework provides no mechanism for managing such state. That's because the marshaller uses the same instance of the custom marshaller for each call to the function.
Stephen's approach of using thread local storage will work I believe. I'm personally not a fan of thread local storage. Another option is to use a dictionary keyed on the unmanaged pointer. Here's an illustration:
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Runtime.InteropServices;
using System.Text;
namespace CustomMarshaler
{
public class MyCustomMarshaler : ICustomMarshaler
{
private Dictionary<IntPtr, object> managedObjects = new Dictionary<IntPtr, object>();
public IntPtr MarshalManagedToNative(object managedObj)
{
if (managedObj == null)
return IntPtr.Zero;
if (!(managedObj is int[]))
throw new MarshalDirectiveException("MyCustomMarshaler must be used on an int array.");
int[] arr = (int[])managedObj;
int size = sizeof(int) + arr.Length * sizeof(int);
IntPtr pNativeData = Marshal.AllocHGlobal(size);
Marshal.WriteInt32(pNativeData, arr.Length);
Marshal.Copy(arr, 0, pNativeData + sizeof(int), arr.Length);
lock (managedObjects)
{
managedObjects.Add(pNativeData, managedObj);
}
return pNativeData;
}
public object MarshalNativeToManaged(IntPtr pNativeData)
{
int[] arr;
lock (managedObjects)
{
arr = (int[])managedObjects[pNativeData];
managedObjects.Remove(pNativeData);
}
int len = Marshal.ReadInt32(pNativeData);
Debug.Assert(len == arr.Length);
Marshal.Copy(pNativeData + sizeof(int), arr, 0, len);
return arr;
}
public void CleanUpNativeData(IntPtr pNativeData)
{
Marshal.FreeHGlobal(pNativeData);
}
public void CleanUpManagedData(object managedObj)
{
}
public int GetNativeDataSize()
{
return -1;
}
public static ICustomMarshaler GetInstance(string cookie)
{
return new MyCustomMarshaler();
}
}
class Program
{
[DllImport(#"MyLib.dll")]
private static extern void Foo(
[In, Out, MarshalAs(UnmanagedType.CustomMarshaler, MarshalTypeRef = typeof(MyCustomMarshaler))]
int[] arr
);
static void Main(string[] args)
{
int[] colorTable = new int[] { 1, 2, 3, 6, 12 };
Foo(colorTable);
foreach (int value in colorTable)
Console.WriteLine(value);
}
}
}

Marshalling a char** in C#

I am interfacing with code that takes a char** (that is, a pointer to a string):
int DoSomething(Whatever* handle, char** error);
Basically, it takes a handle to its state, and if something goes wrong, it returns an error code and optionally an error message (the memory is allocated externally and freed with a second function. That part I've figued out :) ).
I, however, am unsure how to handle in in C#. What I have currently:
[DllImport("mydll.dll", CallingConvention = CallingConvention.Cdecl)]
private static unsafe extern int DoSomething(IntPtr handle, byte** error);
public static unsafe int DoSomething(IntPtr handle, out string error) {
byte* buff;
int ret = DoSomething(handle, &buff);
if(buff != 0) {
// ???
} else {
error = "";
}
return ret;
}
I've poked around, but I can't figure out how to turn that into a byte[], suitable for feeding to UTF8Encoding.UTF8.GetString()
Am I on the right track?
EDIT: To make more explicit, the library function allocates memory, which must be freed by calling another library function. If a solution does not leave me with a pointer I can free, the solution is unacceptable.
Bonus question: As implied above, this library uses UTF-8 for its strings. Do I need to do anything special in my P/Invokes, or just use string for normal const char* parameters?

You should just be able to use a ref string and have the runtime default marshaller take care of this conversion for you. You can hint the char width on the parameter with [MarshalAs(UnmanagedType.LPStr)] to make sure that you are using 8-bit characters.
Since you have a special deallocation method to call, you'll need to keep the pointer, like you've already shown in your question's example.
Here's how I'd write it:
[DllImport("mydll.dll", CallingConvention = CallingConvention.Cdecl)]
private static unsafe extern int DoSomething(
MySafeHandle handle, void** error); // byte** should work, too, I'm just lazy
Then you can get a string:
var errorMsg = Marshal.PtrToStringAnsi(new IntPtr(*error));
And cleanup:
[DllImport("mydll.dll", CallingConvention = CallingConvention.Cdecl)]
private static extern int FreeMyMemory(IntPtr h);
// ...
FreeMyMemory(new IntPtr(error));
And now we have the marshalled error, so just return it.
return errorMsg;
Also note the MySafeHandle type, which would inherit from System.Runtime.InteropServices.SafeHandle. While not strictly needed (you can use IntPtr), it gives you a better handle management when interoping with native code. Read about it here: http://msdn.microsoft.com/en-us/library/system.runtime.interopservices.safehandle.aspx.

For reference, here is code that compiles (but, not tested yet, working on that next tested, works 100%) that does what I need. If anyone can do better, that's what I'm after :D
public static unsafe int DoSomething(IntPtr handle, out string error) {
byte* buff;
int ret = DoSomething(handle, &buff);
if(buff != null) {
int i = 0;
//count the number of bytes in the error message
while (buff[++i] != 0) ;
//allocate a managed array to store the data
byte[] tmp = new byte[i];
//(Marshal only works with IntPtrs)
IntPtr errPtr = new IntPtr(buff);
//copy the unmanaged array over
Marshal.Copy(buff, tmp, 0, i);
//get the string from the managed array
error = UTF8Encoding.UTF8.GetString(buff);
//free the unmanaged array
//omitted, since it's not important
//take a shot of whiskey
} else {
error = "";
}
return ret;
}
Edit: fixed the logic in the while loop, it had an off by one error.

C++ Access violation when calling dll function

I have a function definition in my VC++ Win32 DLL
DEMO2_API void ProcessData(char* i_buff, unsigned short i_len, char* o_buf,
unsigned *o_len, unsigned short *errorCode)
{
__describe (i_buff,&i_len,o_buf,o_len,errorCode);
}
This dll function is called by a c# application.
When called, it generate access violation exception.
After reasearching i found, the cause for my problem.
http://social.msdn.microsoft.com/Forums/en-US/csharplanguage/thread/6e843243-baf4-4eb1-8a20-c691ad47762c
But could not understand what exactly they are doinng in example code.
Can someone explain it so me?
And what would be P/Invoke signature in c# after externally allocating memory?

C# uses IntPtr to represent externally allocated memory. C# pointers and references can only be used with memory provided by the garbage collector.
The System.InteropServices.Marshal class provides some methods for interacting with native memory areas represented by IntPtr, of course they aren't typesafe.
But I don't see anything in your function that could return a pointer to allocated memory. You'd need a double-pointer argument, or a pointer return value, and you have neither.
EDIT to add example as requested:
// this doesn't work right
void external_alloc_and_fill(int n, int* result)
{
result = new int[n];
while (n-- > 0) { result[n] = n; }
}
extern external_alloc_and_fill(int n, int* result)
int a = 5;
fixed (int* p = &a) {
external_alloc_and_fill(17, p);
// p still points to a, a is still 5
}
better:
// works fine
void external_alloc_and_fill2(int n, int** presult)
{
int* result = *presult = new int[n];
while (n-- > 0) { result[n] = n; }
}
extern external_alloc_and_fill2(int n, ref IntPtr result)
int a 5;
IntPtr p = &a;
external_alloc_and_fill2(17, ref p);
// a is still 5 but p is now pointing to the memory created by 'new'
// you'll have to use Marshal.Copy to read it though

I changed passing mode of O_len to out instead of ref and it works.
Thnaks everyone for giving nice answers and comments. I hope this would be useful for other community members ( plus those googling...)