Develop Reference

Converting std::vector to array an then p/Invoking it causes a Access Violation Exception in mscorlib.dll during marshalling

In C++ I have the following struct from 3rd-party code:
typedef struct NodeInfoTag
{
long lResult;
int bComplete;
char *pszNodeAddr;
char *pszParentAddr;
RTS_WCHAR *pwszNodeName;
RTS_WCHAR *pwszDeviceName;
RTS_WCHAR *pwszVendorName;
unsigned long ulTargetType;
unsigned long ulTargetId;
unsigned long ulTargetVersion;
unsigned short wMaxChannels;
}NodeInfotyp;
And the definition to RTS_WCHAR:
# ifndef RTS_WCHAR_DEFINED
# define RTS_WCHAR_DEFINED
typedef wchar_t RTS_WCHAR; /* wide character value */
# endif
(So it's basically a wchar_t)
Then I have my own class called CScanNetworkCallback, which extends the CPLCHandlerCallback class, a class from the same vendor:
.h file:
class CScanNetworkCallback : public CPLCHandlerCallback
{
public:
bool bScanComplete;
NodeInfotyp* pNodeInfo;
NodeInfotyp* pNodeInfoList;
std::vector<NodeInfotyp> vList;
CScanNetworkCallback();
virtual ~CScanNetworkCallback(void);
virtual long Notify(CPLCHandler *pPlcHandler, CallbackAddInfoTag CallbackAdditionalInfo);
};
The implementation follows their own guidelines with some of my own stuff thrown in:
CScanNetworkCallback::CScanNetworkCallback(void) : CPLCHandlerCallback()
{
bScanComplete = false;
}
CScanNetworkCallback::~CScanNetworkCallback()
{
delete pNodeInfo;
delete pNodeInfoList;
}
long CScanNetworkCallback::Notify(CPLCHandler *pPlcHandler, CallbackAddInfoTag CallbackAdditionalInfo)
{
if (pPlcHandler != NULL)
{
if (CallbackAdditionalInfo.ulType == PLCH_SCAN_NETWORK_CALLBACK)
{
pNodeInfo = CallbackAdditionalInfo.AddInf.pNodeInfo;
if (pNodeInfo->lResult == RESULT_OK)
{
vList.push_back(*pNodeInfo);
bScanComplete = false;
}
else
{
pNodeInfoList = &vList[0]; //New pointer points to the vector elements, which will be used as an array later on
// I have also tried copying it, to the same result:
//std::copy(vList.begin(), vList.end(), pNodeInfoList);
bScanComplete = true;
}
}
}
return RESULT_OK;
}
So basically, the Notify method in the class above is called every time a "node" is found in the network, assigning the node's information to pNodeInfo (please disregard what a node is, it isn't relevant ATM). Since it is called to every node in the network during the scanning process and I must send this information to C++, I couldn't find any other way to do so other than using a std::vector to store every callback info for latter use, as I don't know how many nodes there will be at compile time. The else part is called after all nodes have been found. In order to make sense out of the C# code, I must describe the implementation of some other C++ methods that are p/Invoked:
PROASADLL __declspec(dllexport) void scanNetwork(){
pScanHandler->ScanNetwork(NULL, &scanNetworkCallback);
}
The object scanNetworkCallback is static. pScanHandler is a pointer to another class from the 3rd party vendor and its ScanNetwork method runs on a separate thread. Internally (and I only know that due to this API Guidelines, I don't have its source code), it calls the Notify method whenever a node is found in the network, or something to that effect
And finally:
PROASADLL __declspec(dllexport) NodeInfotyp* getScanResult(int* piSize) {
*piSize = scanNetworkCallback.vList.size();
return scanNetworkCallback.pNodeInfoList;
}
That returns the pointer that points to all nodes' information and the amount in as an out parameter. Now let's take a look at the C# code:
public static List<NodeInfoTag> AsaScanNetworkAsync()
{
Console.WriteLine("SCANNING NETWORK");
scanNetwork(); // C++ Method
while (!isScanComplete()) // Holds the C# thread until the scan is complete
Thread.Sleep(50);
int size = 0;
IntPtr pointer = getScanResult(out size); // works fine, I get some IntPtr and the correct size
List<NodeInfoTag> list = Marshaller.MarshalPointerToList<NodeInfoTag>(pointer, size); // PROBLEM!!!
// Continue doing stuff
}
This is the class NodeInfoTag, to match the C++ NodeInfotyp struct:
[StructLayout(LayoutKind.Sequential)]
public class NodeInfoTag
{
public int Result;
public int Complete;
[MarshalAs(UnmanagedType.LPStr)] //char*
public string NodeAddress;
[MarshalAs(UnmanagedType.LPStr)] //char*
public string ParentAddress;
[MarshalAs(UnmanagedType.LPWStr)] //wchar_t
public string VendorName;
public uint TargetType;
public uint TargetId;
public uint TargetVersion;
public short MaxChannels;
}
And this is where I get my Memory Access Violation:
internal class Marshaller
{
public static List<T> MarshalPointerToList<T>(IntPtr pointer, int size)
{
if (size == 0)
return null;
List<T> list = new List<T>();
var symbolSize = Marshal.SizeOf(typeof(T));
for (int i = 0; i < size; i++)
{
var current = (T)Marshal.PtrToStructure(pointer, typeof(T));
list.Add(current);
pointer = new IntPtr(pointer.ToInt32() + symbolSize);
}
return list;
}
}
The error occurs specifically when marshaling should take place, at the line var current = (T)Marshal.PtrToStructure(pointer, typeof(T));. This C# code used to work just fine, but the C++ part was terrible, convoluted and error-prone, so I decided to make things more simple but I can't figure out for the life of me why I'm getting this Exception as I'm making sure that all C++ resources are available for C#, since for testing purposes, I don't delete anything in C++ and I'm only using variables with global scope within the class, which is allocated to static memory. So, what did I miss?
Edit: I removed pNodeInfoList = &vList[0]; and rewrote getScanResult as follows:
static NodeInfotyp pNodeInfoList;
//(...)
PROASADLL __declspec(dllexport) NodeInfotyp* getScanResult(int* piSize) {
*piSize = scanNetworkCallback.vList.size();
std::move(scanNetworkCallback.vList.begin(),
scanNetworkCallback.vList.end(), &pNodeInfoList);
return &pNodeInfoList;
}
No dice. I don't use new or malloc in any of the variables involved, and even changed pNodeInfoList (the array) from a class member to a global variable. Also, I'm using move, as I've been told, could be used to solve ownership problems. Any other tips?

Ownership is not part of the naive C++ type system, so you will not get an error when you delete a pointer you do not own or transfer ownership away without giving it up.
However, semantically certain values and pointers and data blocks are owned by certain types or values.
In this case the vector owns its block of memory. There is no way to ask it or make it give up ownership.
Calling .data() onky provides you a pointer, it does not give that pointer semantic ownership.
You store the return value of .data() in a member variable. You later call delete on that member variable. This indicates to me that member variable is supposed to own its data. So you double delete (as both the vector and the pointer think they own the data pointed to), and your compiler crashes the program for you.
You need to rewite your code taking into account liefetime and ownership of every block of memory you are working with. One approach is to never ever call new, malloc or delete or free directly, and always use memory managing types like vector and unique ptr. Avoid persisting raw pointers, as their ownership semantics are not clear from the type.

Marshalling an array of strings from managed to native code

I have a managed function with the following declaration (both interface and implementation):
[return: MarshalAs(UnmanagedType.SafeArray, SafeArraySubType = VarEnum.VT_BSTR)]
String[] ManagedFunction()
{
String[] foo = new String[1];
foo[0] = "bar";
return foo;
}
There is also a native C++ interface with the same methods as the managed interface, inside of that interface, this method has the following declaration:
void ManagedFunction(SAFEARRAY* foo);
This function is called by native code in the following way:
void NativeFunction(ManagedBinding binding)
{
CComSafeArray<BSTR> cComSafeArray;
cComSafeArray.Create();
LPSAFEARRAY safeArray = cComSafeArray.Detach();
binding.comObject->ManagedFunction(safeArray);
}
I'm not sure what I'm doing wrong but after my managed function has been called, safeArray appears to have garbage values, somethings going wrong while marshalling the return value back to native code. Could someone with more experience than me with .Net interop please shed some light on this? Also, It might be relevant to mention that I haven't had problems with returning ValueTypes from my managed function (boolean if you're curious), something about returning a String array is messing things up. Thanks!

1) Your function return a SAFEARRAY, so why you allocate it before calling function?
2) ManagedFunction supposed to return a SAFEARRAY, so it should get a SAFEARRAY* to be able to return it! so you should say:
LPSAFEARRAY lpsa;
binding.comObject->ManagedFunction(&lpsa);
CComSafeArray<BSTR> cComSafeArray;
cComSafeArray.Attach(lpsa);

Well I finally got it to work. I created a managed representation of SAFEARRAY called ManagedSafeArray (stolen from here : http://social.msdn.microsoft.com/Forums/en-US/clr/thread/6641abfc-3a9c-4976-a523-43890b2b79a2/):
[StructLayout(LayoutKind.Sequential)]
struct ManagedSafeArray
{
public ushort dimensions; // Count of dimensions in the SAFEARRAY
public ushort features; // Flags to describe SAFEARRAY usage
public uint elementSize; // Size of an array element
public uint locks; // Number of times locked without unlocking
public IntPtr dataPtr; // Pointer to the array data
public uint elementCount; // Element count for first (only) dimension
public int lowerBound; // Lower bound for first (only) dimension
}
I changed the signature of my method to:
void ManagedMethod(ref ManagedSafeArray foo);
Inside my method, I manually updated the dataPtr field by calling Marshal.AllocCoTaskMem(...) and then copied over the strings I wanted the SAFEARRAY to contain.
I have no idea as to why the CLR wasn't able to automatically marshal the parameters to and from native code and I'd still appreciate it if someone could try to explain that.

Marshalling a char** in C#

I am interfacing with code that takes a char** (that is, a pointer to a string):
int DoSomething(Whatever* handle, char** error);
Basically, it takes a handle to its state, and if something goes wrong, it returns an error code and optionally an error message (the memory is allocated externally and freed with a second function. That part I've figued out :) ).
I, however, am unsure how to handle in in C#. What I have currently:
[DllImport("mydll.dll", CallingConvention = CallingConvention.Cdecl)]
private static unsafe extern int DoSomething(IntPtr handle, byte** error);
public static unsafe int DoSomething(IntPtr handle, out string error) {
byte* buff;
int ret = DoSomething(handle, &buff);
if(buff != 0) {
// ???
} else {
error = "";
}
return ret;
}
I've poked around, but I can't figure out how to turn that into a byte[], suitable for feeding to UTF8Encoding.UTF8.GetString()
Am I on the right track?
EDIT: To make more explicit, the library function allocates memory, which must be freed by calling another library function. If a solution does not leave me with a pointer I can free, the solution is unacceptable.
Bonus question: As implied above, this library uses UTF-8 for its strings. Do I need to do anything special in my P/Invokes, or just use string for normal const char* parameters?

You should just be able to use a ref string and have the runtime default marshaller take care of this conversion for you. You can hint the char width on the parameter with [MarshalAs(UnmanagedType.LPStr)] to make sure that you are using 8-bit characters.
Since you have a special deallocation method to call, you'll need to keep the pointer, like you've already shown in your question's example.
Here's how I'd write it:
[DllImport("mydll.dll", CallingConvention = CallingConvention.Cdecl)]
private static unsafe extern int DoSomething(
MySafeHandle handle, void** error); // byte** should work, too, I'm just lazy
Then you can get a string:
var errorMsg = Marshal.PtrToStringAnsi(new IntPtr(*error));
And cleanup:
[DllImport("mydll.dll", CallingConvention = CallingConvention.Cdecl)]
private static extern int FreeMyMemory(IntPtr h);
// ...
FreeMyMemory(new IntPtr(error));
And now we have the marshalled error, so just return it.
return errorMsg;
Also note the MySafeHandle type, which would inherit from System.Runtime.InteropServices.SafeHandle. While not strictly needed (you can use IntPtr), it gives you a better handle management when interoping with native code. Read about it here: http://msdn.microsoft.com/en-us/library/system.runtime.interopservices.safehandle.aspx.

For reference, here is code that compiles (but, not tested yet, working on that next tested, works 100%) that does what I need. If anyone can do better, that's what I'm after :D
public static unsafe int DoSomething(IntPtr handle, out string error) {
byte* buff;
int ret = DoSomething(handle, &buff);
if(buff != null) {
int i = 0;
//count the number of bytes in the error message
while (buff[++i] != 0) ;
//allocate a managed array to store the data
byte[] tmp = new byte[i];
//(Marshal only works with IntPtrs)
IntPtr errPtr = new IntPtr(buff);
//copy the unmanaged array over
Marshal.Copy(buff, tmp, 0, i);
//get the string from the managed array
error = UTF8Encoding.UTF8.GetString(buff);
//free the unmanaged array
//omitted, since it's not important
//take a shot of whiskey
} else {
error = "";
}
return ret;
}
Edit: fixed the logic in the while loop, it had an off by one error.

Pinning an updateble struct before passing to unmanaged code?

I using some old API and need to pass the a pointer of a struct to unmanaged code that runs asynchronous.
In other words, after i passing the struct pointer to the unmanaged code, the unmanaged code copies the pointer and returns immediately. The unmanaged code can access that struct in background, in another thread. I have no control over the unmanaged code that runs in another thread nor the thread itself.
The fixed { } statement can't be used for pinning because it not designed for async unmanaged pinning.
GCHandle can pin only references, so the struct must be boxed to use GCHandle. I tried it, and it works. The main problem with it, is that you can't update the struct from managed code. To update a struct, first of all we need to unbox it, then update, then box again, but... oops ... box again?!? this means the previous pointer in the memory still point to the old non-up-to-date struct, and the new struct have another pointer, and this means that i need to pass new pointer to the unmanaged code... inapplicable in my case.
How can i pin a struct in the memory without fixed { } statement, and so that i can update it from managed code without change it's pointer?
Thanks.
Edit:
Just thought... is there a way to pin the parent object that contains the struct, and then get the pointer of the struct rather than the container object?

Using pinned memory in this case is not a good idea, given that the memory for the struct needs to be valid for a long time. GCHandle.Alloc() will box the structure and store it on the heap. With it being pinned, it will be a long term burden to the garbage collector as it needs to constantly find a way around the rock in the road.
The simple solution is to allocate memory for the struct in unmanaged memory. Use Marshal.SizeOf() to get the size of the structure and Marshal.AllocCoTaskMem() to allocate the memory. That gets you the pointer you need to pass to the unmanaged code. Initialize the memory with Marshal.StructureToPtr(). And read updates to the structure written by the unmanaged code with PtrToStructure().
If you do this frequently, you'll be constantly copying the structure. That could be expensive, depending on the size of the structure. To avoid that, use an unsafe pointer to access the unmanaged memory directly. Some basic syntax:
using System;
using System.Runtime.InteropServices;
class Program {
unsafe static void Main(string[] args) {
int len = Marshal.SizeOf(typeof(Test));
IntPtr mem = Marshal.AllocCoTaskMem(len);
Test* ptr = (Test*)mem;
ptr->member1 = 42;
// call method
//..
int value = ptr->member1;
Marshal.FreeCoTaskMem(mem);
}
public struct Test {
public int member1;
}
}

Is unsafe code an option?
// allocate unmanaged memory
Foo* foo = (Foo*)Marshal.AllocHGlobal(sizeof(Foo));
// initialize struct
foo->bar = 0;
// invoke unmanaged function which remembers foo
UnsafeNativeMethods.Bar(foo);
Console.WriteLine(foo->bar);
// update struct
foo->bar = 10;
// invoke unmanaged function which uses remembered foo
UnsafeNativeMethods.Qux();
Console.WriteLine(foo->bar);
// free unmanaged memory
Marshal.FreeHGlobal((IntPtr)foo);
This compiles and doesn't throw an exception, but I don't have an unmanaged function at hand to test if it works.
From MSDN:
When AllocHGlobal calls LocalAlloc, it passes a LMEM_FIXED flag, which causes the allocated memory to be locked in place. Also, the allocated memory is not zero-filled.

Instead of pinning, you need to use Marshal.StructureToPtr and Marshal.PtrToStructure to marshal the struct into memory that's usable in native code.

Struct example:
[StructLayout(LayoutKind.Sequential)]
public struct OVERLAPPED_STRUCT
{
public IntPtr InternalLow;
public IntPtr InternalHigh;
public Int32 OffsetLow;
public Int32 OffsetHigh;
public IntPtr EventHandle;
}
How to pin it to the struct and use it:
OVERLAPPED_STRUCT over_lapped = new OVERLAPPED_STRUCT();
// edit struct in managed code
over_lapped.OffsetLow = 100;
IntPtr pinned_overlap_struct = Marshal.AllocHGlobal(Marshal.SizeOf(over_lapped));
Marshal.StructureToPtr(over_lapped, pinned_overlap_struct, true);
// Pass pinned_overlap_struct to your unmanaged code
// pinned_overlap_struct changes ...
// Get resulting new struct
OVERLAPPED_STRUCT nat_ov = (OVERLAPPED_STRUCT)Marshal.PtrToStructure(pinned_overlap_struct, typeof(OVERLAPPED_STRUCT));
// See what new value is
int offset_low = nat_ov.OffsetLow;
// Clean up
Marshal.FreeHGlobal(pinned_overlap_struct);

How about having the struct include an ActOnMe() interface and method something like:
delegate void ActByRef<T1,T2>(ref T1 p1, ref T2 p2);
interface IActOnMe<TT> {ActOnMe<T>(ActByRef<TT,T> proc, ref T param);}
struct SuperThing : IActOnMe<SuperThing>
{
int this;
int that;
...
void ActOnMe<T>(ActByRef<SuperThing,T>, ref T param)
{
proc(ref this, ref param);
}
}
Because the delegate takes a generic parameter by reference, it should be possible in most cases to avoid the overhead of creating closures by passing a delegate to a static method along with a reference to a struct to carry data to or from that method. Further, casting an already-boxed instance of SuperThing to IActOnMe<SuperThing> and calling ActOnMe<T> on it will expose the fields of that boxed instance for updating, as opposed to creating another copy of them as would occur with a typecast to the struct.

To answer your edit:
Just thought... is there a way to pin the parent object that contains the struct, and then get the pointer of the struct rather than the container object?
I think so. If anything, you should be able to with a managed array of structures (possibly an array of one).
Here is an example code:
[StructLayout(LayoutKind.Sequential)]
struct SomeStructure
{
public int first;
public int second;
public SomeStructure(int first, int second) { this.first=first; this.second=second; }
}
/// <summary>
/// For this question on Stack Overflow:
/// https://stackoverflow.com/questions/1850488/pinning-an-updateble-struct-before-passing-to-unmanaged-code
/// </summary>
private static void TestModifiableStructure()
{
SomeStructure[] objArray = new SomeStructure[1];
objArray[0] = new SomeStructure(10, 10);
GCHandle hPinned = GCHandle.Alloc(objArray, GCHandleType.Pinned);
//Modify the pinned structure, just to show we can
objArray[0].second = 42;
Console.WriteLine("Before unmanaged incrementing: {0}", objArray[0].second);
PlaceholderForUnmanagedFunction(hPinned.AddrOfPinnedObject());
Console.WriteLine("Before unmanaged incrementing: {0}", objArray[0].second);
//Cleanup
hPinned.Free();
}
//Simulates an unmanaged function that accesses ptr->second
private static void PlaceholderForUnmanagedFunction(IntPtr ptr)
{
int secondInteger = Marshal.ReadInt32(ptr, 4);
secondInteger++;
Marshal.WriteInt32(ptr, 4, secondInteger);
}
And its output:
Before unmanaged incrementing: 42
Before unmanaged incrementing: 43

Invalid Variant crash

I have a situation where I've wrapped a Native C++ DLL with C++/CLI for eventual use in C#.
There are a few callback functions that are causing some issues at run time. Particularly, I get the following exception:
An unhandled exception of type
'System.Runtime.InteropServices.InvalidOleVariantTypeException'
occurred in ToadWrapTest.dll
Additional information: Specified OLE
variant is invalid.
On this line of code (C++/CLI):
public delegate int ManagedCallbackFunction (Object^ inst, const Object^ data);
public delegate int UnManagedCallbackFunction (void* inst, const void* data);
ManagedCallbackFunction^ m_callbackFn;
int intermidiaryCallback(void * pInstance, const void * pData)
{
void* temp = (void*)pData;
System::IntPtr ip1 = IntPtr(pInstance);
System::IntPtr ip2 = IntPtr(temp);
Object^ oInst = Marshal::GetObjectForNativeVariant(ip1);
Object^ oData = Marshal::GetObjectForNativeVariant(ip2);
//invoke the callback to c#
//return m_callbackFn::Invoke(oInst, oData);
return 0;
};
The reason I've made this "intermediary callback" was an attempt to circumvent the Invalid variant exception being thrown when I tried to directly map the delegate from C# to the native C++ code. As an attempted work-around, I declare a delegate on the C# side and pass that funcptr to the C++/CLI wrapper. I then pass the intermediary funcptr to the native C++ and just daisy chain the calls together.
What I know is that it all works in native C++ world. The problem is mapping the void* to the managed world. The following code shows the native C++ version of the callback:
int (*CallbackFunction) (void *inst, const void *data);
If anyone can help here, I'd really appreciate it.

Are pInstance and pData really VARIANT? If they are, I would expect your callback function to be more strongly typed:
int (*CallbackFunction)(VARIANT *inst, VARIANT *data);
If that's the case, in your code you should be able to look at the actual VARIANT to hand check it. If you are not really getting VARIANTs (ie, you are really just getting void * pointers), you shouldn't try to turn them into C# objects since there is no inherent meaning to them. They should get passed through as IntPtr. If you know that they should have some other type of inherent meaning, you need to marshal them as appropriate types.

Big Thanks to plinth on this one! I am posting the final solution below to anyone else who has to deal with 3rd party fun like this one! Please feel free to critique, as I am not done optimizing the code. This may still be to roundabout a solution.
First, the callback functions became:
public delegate int ManagedCallbackFunction (IntPtr oInst, IntPtr oData);
public delegate int UnManagedCallbackFunction (void* inst, const void* data);
ManagedCallbackFunction^ m_callbackFn;
Big props on this one. It just plain won't work if you try to cast from void* directly to Object^. Using the IntPtr and my intermediary callback:
int intermidiaryCallback(void * pInstance, const void * pData)
{
void* temp = (void*)pData;
return m_callbackFn->Invoke(IntPtr(pInstance), IntPtr(temp));
};
We finally get a working model on the C# side with some massaging of the objects:
public static int hReceiveTestMessage(IntPtr pInstance, IntPtr pData)
{
// provide object context for static member function
helloworld2 hw = (helloworld2)GCHandle.FromIntPtr(pInstance).Target;
if (hw == null || pData == null)
{
Console.WriteLine("hReceiveTestMessage received NULL data or instance pointer\n");
return 0;
}
// populate message with received data
IntPtr ip2 = GCHandle.ToIntPtr(GCHandle.Alloc(new DataPacketWrap(pData)));
DataPacketWrap dpw = (DataPacketWrap)GCHandle.FromIntPtr(ip2).Target;
uint retval = hw.m_testData.load_dataSets(ref dpw);
// display message contents
hw.displayTestData();
return 1;
}
I mention "massaging" the objects because the delegate is not specific to this callback function and I don't know what object pData will be until run time(from the delegates POV). Because of this issue, I have to do some extra work with the pData object. I basically had to overload the constructor in my wrapper to accept an IntPtr. Code is provided for full "clarity":
DataPacketWrap (IntPtr dp)
{
DataPacket* pdp = (DataPacket*)(dp.ToPointer());
m_NativeDataPacket = pdp;
};