Develop Reference

.NET 6 IDispatch client implementation crash

.NET 6 has trouble calling its own IDispatch objects, if marshaled.
To reproduce:
if (Array.IndexOf(Environment.GetCommandLineArgs(), "/s")>=0) //Server
{
Thread t = new Thread(new ThreadStart(() =>
{
Hello h = new Hello();
new RunningObjectTable().Register("HelloTest", h);
Thread.Sleep(1000 * 3600);
}));
t.SetApartmentState(ApartmentState.MTA);
t.Start();
Thread.Sleep(1000 * 3600);
}
else //Client
{
object o = new RunningObjectTable().Get("HelloTest");
IHello h = o as IHello;
int f = h.Foo();
Console.WriteLine(h);
}
[ComImport]
[Guid("00020400-0000-0000-C000-000000000046")] //IID_IDispatch
[InterfaceType(ComInterfaceType.InterfaceIsIDispatch)]
[TypeLibType(TypeLibTypeFlags.FDispatchable)]
public interface IHello
{
[MethodImpl(MethodImplOptions.PreserveSig | MethodImplOptions.InternalCall)]
[DispId(1)]
public int Foo();
}
public class Hello: IHello
{
public int Foo()
{
Debug.WriteLine("Hello from server");
return 19;
}
}
Run the program with /s, that's the object server. Then run another copy to become the client.
The method invocation line in the client crashes with "Member not found" - HRESULT 0x80020003, DISP_E_MEMBERNOTFOUND. Normally it would mean a bogus DISPID, but where would the discrepancy possibly come from?
One minor infraction in that code is that the interface is decorated with the IID of IDispatch. With a custom IID, as COM prescribes, it doesn't even unmarshal as the dispinterface (the as IHello line returns null); internally, there's a QueryInterface call across processes with said custom IID, the dispinterface is not registered under HKCR\Interfaces, so the interprocess COM machinery doesn't know how to marshal it. At least that's my theory.
A similar piece of logic works fine if the server is a native (C++) one. If the same C# piece is recompiled against .NET framework 4.72, it doesn't even get that far, the o as IHello; line returns null.
RunningObjectTable is a helper class around the ROT. For completeness' sake, here:
internal class RunningObjectTable
{
#region API
[DllImport("ole32.dll")]
private static extern int CreateItemMoniker([MarshalAs(UnmanagedType.LPWStr)] string
lpszDelim, [MarshalAs(UnmanagedType.LPWStr)] string lpszItem,
out IMoniker ppmk);
[DllImport("ole32.dll")]
private static extern int GetRunningObjectTable(int reserved, out IRunningObjectTable prot);
#endregion
private IRunningObjectTable m_rot;
public RunningObjectTable()
{
GetRunningObjectTable(0, out m_rot);
}
private IMoniker CreateItemMoniker(string s)
{
IMoniker mon;
CreateItemMoniker("", s, out mon);
return mon;
}
public int Register(string ItemName, object o)
{
return m_rot.Register(0, o, CreateItemMoniker(ItemName));
}
public void Unregister(int ROTCookie)
{
m_rot.Revoke(ROTCookie);
}
public object Get(string ItemName)
{
object o;
m_rot.GetObject(CreateItemMoniker(ItemName), out o);
return o;
}
}
Can someone with access to .NET Core internals please tell me what's going on?

Here's what is going on. The .NET object supports two dispinterfaces - one corresponding to IHello, another corresponding to Hello itself, and the latter is treated as the default one. I've confirmed that by adding an extra method Bar() to Hello, and checking the type in a native C++ client:
IUnknownPtr pUnk;
ROTGet(L"HelloTest", &pUnk);
IDispatchPtr pDisp(pUnk);
LPOLESTR Name = (LPOLESTR)L"Bar";
DISPID dispid;
hr = pDisp->GetIDsOfNames(IID_NULL, &Name, 1, 0, &dispid);
And I got S_OK and a valid, if bogus, DISPID. I then requested a DISPID for Foo in the same way, and got another bogus DISPID.
I've found two ways to fix that.
One, tell the framework that it should not implicitly generate a dispinterface for Hello. Annotate the class with [ClassInterface(ClassInterfaceType.None)], and it won't. The explicit check for the DISPID of Foo confirms that. The unmarshaled dispinterface in the client corresponds to IHello, and the methods work as expected.
Another approach, give IHello an identity (an IID) and make it marshalable. If I slap a custom GUID on that interface, and add a key with that GUID to HKCR\Interfaces, providing ProxyStubClsid32 equals to {00020420-0000-0000-C000-000000000046} (that's PSDispatch, the proxy/stub CLSID for IDispatch), the fragment also works.
One other way to tell COM that an IID corresponds to a dispinterface and should be marshaled as one is via implementing IStdMarshalInfo. I didn't try this.
One more note: the Hello class has to be declared public for this to work. If it's not, for some reason the as IHello returns null.

How to use D3D12GetDebugInterface API from c#?

I am currently trying to use Direct3D from c# and I've started by using the D3D12GetDebugInterface API.
The C++ syntax of the function is as follows (copied from the Microsoft documentation):
HRESULT D3D12GetDebugInterface(
REFIID riid,
void **ppvDebug
);
I'm having trouble importing the function into C#. I thought that maybe riid should be a pointer to a Guid struct, and I'm just using an IntPtr for **ppvDebug. Since **ppvDebug is a pointer to a pointer to an ID3D12Debug interface, I tried reimplementing the ID3D12Debug interface in C# code and using Marshal.PtrToStructure() to resolve the IntPtr to a usable ID3D12Debug interface instance, but that won't work. I remember reading about the ID3D12Debug interface being a COM object, but don't you need an ID for a COM object so you can import it? I haven't found any sort of COM ID anywhere in the documentation.
Anyway here's my latest attempt at getting something out of the function:
[DllImport("D3D12.dll")]
static extern int D3D12GetDebugInterface(IntPtr riid, IntPtr ppvDebug);
void func() {
unsafe
{
IntPtr DebugControllerPtr = IntPtr.Zero;
Type InterfaceType = typeof(ID3D12Debug);
Guid ID = InterfaceType.GUID;
IntPtr ptr = Marshal.AllocHGlobal(sizeof(Guid));
Marshal.StructureToPtr(ID, ptr, false);
D3D12GetDebugInterface(ptr, DebugControllerPtr);
Marshal.FreeHGlobal(ptr);
ID3D12Debug DebugController = null;
Marshal.PtrToStructure(DebugControllerPtr, DebugController);
DebugController.EnableDebugLayer();
}
}
In case you want to see my ID3D12Debug interface:
interface ID3D12Debug
{
void EnableDebugLayer();
}
As I said, I think Direct3D makes use of COM, which I am completely missing here, so maybe that's why it doesn't work.

There are usually many ways to declare interop code. Here is one that should work:
public static void Main()
{
D3D12GetDebugInterface(typeof(ID3D12Debug).GUID, out var obj);
var debug = (ID3D12Debug)obj;
debug.EnableDebugLayer(); // for example
}
[DllImport("D3D12")]
public static extern int D3D12GetDebugInterface([MarshalAs(UnmanagedType.LPStruct)] Guid riid, [MarshalAs(UnmanagedType.IUnknown)] out object ppvDebug);
[Guid("344488b7-6846-474b-b989-f027448245e0"), InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
public interface ID3D12Debug
{
[PreserveSig]
void EnableDebugLayer();
}
You must add interface Guid and InterfaceType attributes for COM interfaces.
Guid can be passed easily by reference using UnmanagedType.LPStruct
You don't need unsafe code here.
You should check for errors (undone in my code)
If you need .NET interface definitions with DirectX, you can use this open source project here: https://github.com/smourier/DirectN, for example: https://github.com/smourier/DirectN/blob/master/DirectN/DirectN/Generated/ID3D12Debug.cs

COM interop and marshaling of VARIANT(VT_PTR)

We use a 3rd party COM object, one of which methods under certain conditions returns a VARIANT of VT_PTR type. That upsets the default .NET marshaler, which throws the following error:
Managed Debugging Assistant 'InvalidVariant' : 'An invalid VARIANT was
detected during a conversion from an unmanaged VARIANT to a managed
object. Passing invalid VARIANTs to the CLR can cause unexpected
exceptions, corruption or data loss.
Method signatures:
// (Unmanaged) IDL:
HRESULT getAttribute([in] BSTR strAttributeName, [retval, out] VARIANT* AttributeValue);
// C#:
[return: MarshalAs(UnmanagedType.Struct)]
object getAttribute([In, MarshalAs(UnmanagedType.BStr)] string strAttributeName);
Is there an elegant way to bypass such marshaler's behavior and obtain the underlying unmanaged pointer on the managed side?
What I've considered/tried so far:
A custom marshaler:
[return: MarshalAs(UnmanagedType.CustomMarshaler,
MarshalTypeRef = typeof(IntPtrMarshaler))]
object getAttribute([In, MarshalAs(UnmanagedType.BStr)] string strAttributeName);
I did implement IntPtrMarshaler, just to find the interop layer crashing the process even before any of my ICustomMarshaler methods gets called. Perhaps, the VARIANT* argument type is not compatible with custom marshalers.
Rewrite (or clone) the C# interface definition with getAttribute method redefined (like below) and do all the marshaling for output VARIANT manually:
void getAttribute(
[In, MarshalAs(UnmanagedType.BStr)],
string strAttributeName,
IntPtr result);
This doesn't seem nice (the interface itself has 30+ other methods). It'd also break existing, unrelated pieces of code which already make use of getAttribute without issues.
Obtain an unmanaged method address of getAttribute from vtable (using Marshal.GetComSlotForMethodInfo etc), then do the manual invocation and marshaling against my own custom delegate type (using Marshal.GetDelegateForFunctionPointer etc).
So far, I've taken this approach and it seem to work fine, but it feels as such an overkill for what should be a simple thing.
Am I missing some other feasible interop options for this scenario? Or, maybe there is a way to make CustomMarshaler work here?

What I would do is define a simple VARIANT structure like this:
[StructLayout(LayoutKind.Sequential)]
public struct VARIANT
{
public ushort vt;
public ushort r0;
public ushort r1;
public ushort r2;
public IntPtr ptr0;
public IntPtr ptr1;
}
And the interface like this;
[Guid("39c16a44-d28a-4153-a2f9-08d70daa0e22"), InterfaceType(ComInterfaceType.InterfaceIsDual)]
public interface MyInterface
{
VARIANT getAttributeAsVARIANT([MarshalAs(UnmanagedType.BStr)] string strAttributeName);
}
Then, add an extension method somewhere in a static class like this, so the caller can have the same coding experience using MyInterface:
public static object getAttribute(this MyInterface o, string strAttributeName)
{
return VariantSanitize(o.getAttributeAsVARIANT(strAttributeName));
}
private static object VariantSanitize(VARIANT variant)
{
const int VT_PTR = 26;
const int VT_I8 = 20;
if (variant.vt == VT_PTR)
{
variant.vt = VT_I8;
}
var ptr = Marshal.AllocCoTaskMem(Marshal.SizeOf<VARIANT>());
try
{
Marshal.StructureToPtr(variant, ptr, false);
return Marshal.GetObjectForNativeVariant(ptr);
}
finally
{
Marshal.FreeCoTaskMem(ptr);
}
}
This will do nothing for normal variants, but will just patch it for VT_PTR cases.
Note this only works if the caller and the callee are in the same COM apartement.
If they are not, you will get the DISP_E_BADVARTYPE error back because marshaling must be done, and by default, it will be done by the COM universal marshaler (OLEAUT) which only support Automation compatible data types (just like .NET).
In this case, theoratically, you could replace this marshaler by another one (at COM level, not at NET level), but that would mean to add some code on C++ side and possibly in the registry (proxy/stub, IMarshal, etc.).

For my own future reference, here's how I ended up doing it, using the 3rd option mentioned in the question:
[ComImport, Guid("75A67021-058A-4E2A-8686-52181AAF600A"), InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
public interface IInterface
{
[return: MarshalAs(UnmanagedType.Struct)]
object getAttribute([In, MarshalAs(UnmanagedType.BStr)] string strAttributeName);
}
private delegate int IInterface_getAttribute(
IntPtr pInterface,
[MarshalAs(UnmanagedType.BStr)] string name,
IntPtr result);
public static object getAttribute(this IInterface obj, string name)
{
var ifaceType = typeof(IInterface);
var ifaceMethodInfo = ((Func<string, object>)obj.getAttribute).Method;
var slot = Marshal.GetComSlotForMethodInfo(ifaceMethodInfo);
var ifacePtr = Marshal.GetComInterfaceForObject(obj, ifaceType);
try
{
var vtablePtr = Marshal.ReadIntPtr(ifacePtr);
var methodPtr = Marshal.ReadIntPtr(vtablePtr, IntPtr.Size * slot);
var methodWrapper = Marshal.GetDelegateForFunctionPointer<IInterface_getAttribute>(methodPtr);
var resultVar = new VariantClass();
var resultHandle = GCHandle.Alloc(resultVar, GCHandleType.Pinned);
try
{
var pResultVar = resultHandle.AddrOfPinnedObject();
VariantInit(pResultVar);
var hr = methodWrapper(ifacePtr, name, pResultVar);
if (hr < 0)
{
Marshal.ThrowExceptionForHR(hr);
}
if (resultVar.vt == VT_PTR)
{
return resultVar.ptr;
}
try
{
return Marshal.GetObjectForNativeVariant(pResultVar);
}
finally
{
VariantClear(pResultVar);
}
}
finally
{
resultHandle.Free();
}
}
finally
{
Marshal.Release(ifacePtr);
}
}

How to Instantiate a ComVisible Class into Its Own AppDomain in a Single-Threaded Client?

The Problem
When instantiating two, independent .NET COM-visible classes within the same, single-threaded COM client, .NET loads them both into the same AppDomain.
I am guessing that this is because they are being loaded into the same thread/process.
An example of this behavior is shown in this GitHub repository.
Essentially, the demonstration is as follows:
Instantiate one COM class
Set an attribute on the first COM object which, in the back-end calls SetData on the CurrentDomain.
Instantiate a second, independent COM class (different interface name, GUIDs, etc)
Read the AppDomain attribute
Demonstrate that it appears the same
Also, get the hash code from both AppDomains, noting that it is also the same
Why is this a problem?
When both classes have the AppDomain.CurrentDomain.AssemblyResolve event implemented (or any other AppDomain event, for that matter), the events can interfere with one another. This is at least one complication; I am guessing that there may be others as well.
An Idea
I thought the best way of handling this would be to create a new AppDomain for each COM object. Because I could not find (or Google) a way of doing this in a managed way, I thought it might be necessary to do it in unmanaged code.
I did a little detective work. In OleView, the InprocServer32 attribute for a .NET COM-visible class is mscoree.dll. So, I created a "shim" DLL which forwarded all of its EXPORTS to mscoree.dll. By process of elimination (eliminating exports until the COM would no longer load), I discovered that DllGetClassObject in mscoree was responsible for starting up the .NET runtime, and returning the instantiated COM object.
So, what I can do is implement my own DllGetClassObject, like so:
Host the .NET runtime in an unmanaged assembly using CLRCreateInstance
Create the object in a new AppDomain, and return it
(I'm guessing it's not as simple as it sounds, though)
The Question
Before I embark on this potentially difficult and lengthy process, I'd like to know:
Is there a managed way of getting a .NET COM-visible class to run in its own AppDomain?
If not, is this the "right" way of doing it, or am I missing an obvious solution?

If the code does not have to run in the same process, an out-of-process server would be the easiest fix. Pass CLSCTX_LOCAL_SERVER to CoCreateInstance and each class will be created in a dllhost hosting process.
For example on the client:
public static object CreateLocalServer(Guid clsid)
{
return CoCreateInstance(clsid, null, CLSCTX.LOCAL_SERVER, IID_IUnknown);
}
public static object CreateLocalServer(string progid)
{
Contract.Requires(!string.IsNullOrEmpty(progid));
Guid clsid;
CLSIDFromProgID(progid, out clsid);
return CreateLocalServer(clsid);
}
enum CLSCTX : uint
{
INPROC_SERVER = 0x1,
INPROC_HANDLER = 0x2,
LOCAL_SERVER = 0x4,
INPROC_SERVER16 = 0x8,
REMOTE_SERVER = 0x10,
INPROC_HANDLER16 = 0x20,
RESERVED1 = 0x40,
RESERVED2 = 0x80,
RESERVED3 = 0x100,
RESERVED4 = 0x200,
NO_CODE_DOWNLOAD = 0x400,
RESERVED5 = 0x800,
NO_CUSTOM_MARSHAL = 0x1000,
ENABLE_CODE_DOWNLOAD = 0x2000,
NO_FAILURE_LOG = 0x4000,
DISABLE_AAA = 0x8000,
ENABLE_AAA = 0x10000,
FROM_DEFAULT_CONTEXT = 0x20000,
ACTIVATE_32_BIT_SERVER = 0x40000,
ACTIVATE_64_BIT_SERVER = 0x80000
}
[DllImport(Ole32, ExactSpelling = true, PreserveSig = false)]
[return: MarshalAs(UnmanagedType.Interface)]
public static extern object CoCreateInstance(
[In, MarshalAs(UnmanagedType.LPStruct)] Guid rclsid,
[MarshalAs(UnmanagedType.IUnknown)] object pUnkOuter,
CLSCTX dwClsContext,
[In, MarshalAs(UnmanagedType.LPStruct)] Guid riid);
[DllImport(Ole32, CharSet = CharSet.Unicode, PreserveSig = false)]
public static extern void CLSIDFromProgID(string progId, out Guid rclsid);
You can also register a custom host, and swap the standard InProcServer32 for LocalServer32. For an example server
// StandardOleMarshalObject keeps us single-threaded on the UI thread
// https://msdn.microsoft.com/en-us/library/74169f59(v=vs.110).aspx
[ComVisible(true)]
[ClassInterface(ClassInterfaceType.None)]
[ProgId(IpcConstants.CoordinatorProgID)]
public sealed class Coordinator : StandardOleMarshalObject, ICoordinator
{
public Coordinator()
{
// required for regasm
}
#region Registration
[ComRegisterFunction]
internal static void RegasmRegisterLocalServer(string path)
{
// path is HKEY_CLASSES_ROOT\\CLSID\\{clsid}", we only want CLSID...
path = path.Substring("HKEY_CLASSES_ROOT\\".Length);
using (RegistryKey keyCLSID = Registry.ClassesRoot.OpenSubKey(path, writable: true))
{
// Remove the auto-generated InprocServer32 key after registration
// (REGASM puts it there but we are going out-of-proc).
keyCLSID.DeleteSubKeyTree("InprocServer32");
// Create "LocalServer32" under the CLSID key
using (RegistryKey subkey = keyCLSID.CreateSubKey("LocalServer32"))
{
subkey.SetValue("", Assembly.GetExecutingAssembly().Location, RegistryValueKind.String);
}
}
}
[ComUnregisterFunction]
internal static void RegasmUnregisterLocalServer(string path)
{
// path is HKEY_CLASSES_ROOT\\CLSID\\{clsid}", we only want CLSID...
path = path.Substring("HKEY_CLASSES_ROOT\\".Length);
Registry.ClassesRoot.DeleteSubKeyTree(path, throwOnMissingSubKey: false);
}
#endregion
}

Well... here's a managed proof-of-concept using RGiesecke.DllExport that works; whether or not it's a good solution remains to be seen... so: use at your own risk. I'm still looking for better answers.
One thing that could be improved is that we don't need a new AppDomain for each instantiation; only for each object. I'm sure that there are other subtleties that I'm missing.
We compile and register the DLL, then using OleView (or registry), we change the default ProcServer32 value to point to the managed DLL itself. This can be automated by providing a method in the DLL that is decorated with [ComRegisterFunction()].
using System;
using System.ComponentModel;
using System.Reflection;
using System.Runtime.InteropServices;
using RGiesecke.DllExport;
using System.IO;
namespace Com_1
{
[Guid("F35D5D5D-4A3C-4042-AC35-CE0C57AF8383")]
[InterfaceType(ComInterfaceType.InterfaceIsIDispatch)]
[ComVisible(true)]
public interface IComClass1
{
void SetAppDomainData(string data);
string GetAppDomainData();
int GetAppDomainHash();
}
//https://gist.github.com/jjeffery/1568627
[Guid("00000001-0000-0000-c000-000000000046")]
[InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
[ComImport]
internal interface IClassFactory
{
void CreateInstance([MarshalAs(UnmanagedType.IUnknown)] object pUnkOuter, ref Guid riid, [MarshalAs(UnmanagedType.IUnknown)] out object ppvObject);
void LockServer(bool fLock);
}
[ComVisible(true)]
[ClassInterface(ClassInterfaceType.None)]
[Guid("3CA12D49-CFE5-45A3-B114-22DF2D7A0CAB")]
[Description("Sample COM Class 1")]
[ProgId("Com1.ComClass1")]
public class ComClass1 : MarshalByRefObject, IComClass1, IClassFactory
{
public void SetAppDomainData(string data)
{
AppDomain.CurrentDomain.SetData("CurrentDomainCustomData", data);
}
public string GetAppDomainData()
{
return (string)AppDomain.CurrentDomain.GetData("CurrentDomainCustomData");
}
public int GetAppDomainHash()
{
return AppDomain.CurrentDomain.GetHashCode();
}
[DllExport]
public static uint DllGetClassObject(Guid rclsid, Guid riid, out IntPtr ppv)
{
ppv = IntPtr.Zero;
try
{
if (riid.CompareTo(Guid.Parse("00000001-0000-0000-c000-000000000046")) == 0)
{
//Call to DllClassObject is requesting IClassFactory.
var instance = new ComClass1();
IntPtr iUnk = Marshal.GetIUnknownForObject(instance);
//return instance;
Marshal.QueryInterface(iUnk, ref riid, out ppv);
return 0;
}
else
return 0x80040111; //CLASS_E_CLASSNOTAVAILABLE
}
catch
{
return 0x80040111; //CLASS_E_CLASSNOTAVAILABLE
}
}
public void CreateInstance([MarshalAs(UnmanagedType.IUnknown)] object pUnkOuter, ref Guid riid, [MarshalAs(UnmanagedType.IUnknown)] out object ppvObject)
{
IntPtr ppv = IntPtr.Zero;
//http://stackoverflow.com/a/13355702/864414
AppDomainSetup domaininfo = new AppDomainSetup();
domaininfo.ApplicationBase = Directory.GetParent(Assembly.GetExecutingAssembly().Location).FullName;
var curDomEvidence = AppDomain.CurrentDomain.Evidence;
AppDomain newDomain = AppDomain.CreateDomain("MyDomain", curDomEvidence, domaininfo);
Type type = typeof(ComClass1);
var instance = newDomain.CreateInstanceAndUnwrap(
type.Assembly.FullName,
type.FullName);
ppvObject = instance;
}
public void LockServer(bool fLock)
{
//Do nothing
}
}
}

What's wrong with this interop C#/managed/unmanaged C++ parameter passing?

I'm trying to make a non-static vendor C++ DLL accessible via C#. In order to do this, I'm writing a managed C++ wrapper DLL which basically creates static variables for the vendor DLL and makes those accessible to the C# application.
Here's an example:
typedef void(__stdcall *LPLISTENER_FUNC)(VENDORHANDLE hModule, VENDORWPARAM wParam, VENDORLPARAM lParam);
public delegate void VENDOR_Delegate(VENDORHANDLE hModule,
VENDORWPARAM wParam, VENDORLPARAM lParam);
public class VENDORWrapper
{
private:
static VENDORHSTORAGE _hStorage;
static VENDOR_Delegate^ _hOpenCallback;
void static Initialize()
{
_hStorage=storage_initialize();
}
void static registerCallback(unsigned int type, VENDOR_Delegate^ callback)
{
if (type == 2)
{
_hOpenCallback = callback;
::storage_register_callback(_hStorage, type, (LPLISTENER_FUNC)&_hOpenCallback);
}
}
bool static Open(String^ file)
{
bool retval=false;
filePath = file;
IntPtr ip = Marshal::StringToHGlobalAuto(filePath);
LPCWSTR str = static_cast<LPCWSTR>(ip.ToPointer());
//ERROR OCCURS HERE
retval = storage_open(_hStorage, str);
Marshal::FreeHGlobal( ip );
return retval;
}
void static Close()
{
storage_close(_hStorage);
}
}
The C# is skeletal:
public static VENDORStorageWrapper.VENDOR_Delegate openCallback
= new VENDORStorageWrapper.VENDOR_Delegate(fileOpened);
static void Main(string[] args)
{
VENDORStorageWrapper.VENDORStorageWrapper.Initialize();
Debug.WriteLine("DLL initalized");
VENDORStorageWrapper.VENDORStorageWrapper.registerCallback(2,
openCallback);
Debug.WriteLine("Callback registered");
VENDORStorageWrapper.VENDORStorageWrapper.Open("blah_file");
Debug.WriteLine("File opened");
}
public static void fileOpened(System.Int32 hstorage, System.UInt32 wParam, System.Int32 lParam)
{
Debug.WriteLine("file opened");
}
The vendor DLL's functions are specified as __stdcall, so I think I'm compliant on that front. The vendor's initialize call (_storage_initialize above) seems to be properly setting the handle, which is statically scoped. The storage_open call that's leading into the exception accepts a VENDORHANDLE (really a long) and an LPCWSTR, which I'm trying to convert the string passed from C# to. I think that's where the problem is...
When run, the app throws an unhandled exception "System.Runtime.InteropServices.SEHException" at the commented line above. The exception's coming from inside the vendor DLL, which I have no source code for. The vendor library works perfectly when called in an unmanaged C++ context and the file is known to be good. I think I'm missing something obvious in how I'm handling the parameters, but I can't see what it is.
I also don't think I have the callback set up properly, but I'm not the point where I can test that yet. Any ideas?

I'm not sure of the true big picture, but my experience using native DLLs with .net c# or vb, create a simple c# wrapper (just declarations) to the native DLL, instead of a c++ wrapper. Maybe this will help if the vendor DLL is really a vanilla DLL, as most are, like Windows api calls.
namespace MyNameSpace
{
public class MyWrapper
{
// passing an int to the native DLL
[DllImport("Vendor.DLL")]
public static extern int DllFunc1(int hModule, int nData);
// passing a string to a native DLL expecting null terminated raw wide characters //
[DllImport("Vendor.DLL", CharSet=CharSet.Unicode )]
public static extern int Dllszset(int hModule, string text);
}
}
Then .net will handle it for you and you call your vendor function as...
MyNameSpace.MyWrapper.Dllszset(h, "hello");
Hope this helps you or someone.