Develop Reference

Marshalling Function Pointers with .NET 7 LibraryImport

I'm trying to implement some P/Invoke code using the new LibraryImport attribute, as opposed to the old DllImport. Specifically, I am trying to marshal a WNDCLASSEXW struct for use in RegisterClassEx.
Here is a simplified, shortened version of my managed implementation of WNDCLASSEXW:
[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Unicode)]
public struct WindowClass
{
private uint StructSize;
public WindowClassStyle Style;
[MarshalAs(UnmanagedType.FunctionPtr)]
public Win32API.WindowProcedure? WindowProcedure;
private int ClassAdditionalBytes;
private int WindowAdditionalBytes;
public IntPtr Instance;
public IntPtr Icon;
public IntPtr Cursor;
public IntPtr BackgroundBrush;
[MarshalAs(UnmanagedType.LPWStr)]
public string? ClassMenuResourceName;
[MarshalAs(UnmanagedType.LPWStr)]
public string? ClassName;
public IntPtr SmallIcon;
}
And my definition of Win32API.WindowProcedure:
[UnmanagedFunctionPointer(CallingConvention.StdCall)]
public delegate nint WindowProcedure(IntPtr windowHandle, MessageID messageID, nuint wParam, nint lParam);
And finally my definition of RegisterClassEx:
[LibraryImport("user32.dll", SetLastError = true, EntryPoint = "RegisterClassExW")]
public static partial ushort RegisterClassEx(in WindowClass classDefinition);
However, this results in the error:
Error SYSLIB1051: The type 'xxx.WindowClass' is not supported by source-generated P/Invokes. The generated source will not handle marshalling of parameter 'classDefinition'.
Therefore I believe I require custom marshalling for the WindowClass struct.
However, with this system being relatively new, I'm having difficulty finding good guidance on how to do this correctly and optimally. Previously, DllImport would magically marshal most types with little guidance, but LibraryImport appears to require more information, and be a bit stricter.
I could circumvent the problem by changing the type to IntPtr and requiring conversion of the delegate to IntPtr elsewhere in the program, but I would much prefer to do it as close to the managed/unmanaged boundary as possible and keep the structs and exposed native functions usable with more descriptive types.
Some of the resources I have found while searching:
The old P/Invoke documentation regarding delegates/function pointers
The new information regarding CustomMarshaller
The design documentation for the new source generator-based system
Primary Question: How do I correctly implement custom marshalling for my WNDPROC and the LP(C)WSTRs?
Question 2:
I would prefer to use a readonly struct, and turn all of the members into { get; init; } properties instead of fields, due to the nicer semantics. However I've noticed that the MarshalAs attribute cannot be applied to properties. Is there a good way to both use readonly structs with properties, while also providing the necessary information to ensure everything gets marshalled in/out correctly? Specifically for more complex types such as string? <-> LPCWSTR, delegate? <-> void*, and other such types I may encounter.
Bonus Question:
It appears that LibraryImport de-emphasizes the importance of specifying the correct calling convention. It's no longer part of the main attribute like DllImport, instead using a secondary attribute that looks like this: [UnmanagedCallConv(CallConvs = new[] { typeof(CallConvStdcall) })] which frankly looks terrible. Is specifying calling convention necessary or beneficial now?

I was able to get it working with custom marshaling. While Simon's advice of changing the struct to contain the native types makes sense in the general case, in my case it doesn't, as these will be exposed for others to use.
The answer might be different for faster, more frequently called methods, but in this case registering a class and creating a window is inherently quite an expensive operation, so the added overhead of copying data to/from a different struct isn't worth any concern.
The marshaler was implemented like this:
[CustomMarshaller(typeof(WindowClass), MarshalMode.UnmanagedToManagedIn, typeof(WindowClassMarshaler))]
[CustomMarshaller(typeof(WindowClass), MarshalMode.ManagedToUnmanagedIn, typeof(ManagedToUnmanagedIn))]
internal static unsafe class WindowClassMarshaler
{
[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Unicode)]
internal unsafe struct WindowClassUnmanaged
{
public uint StructSize;
public uint Style;
public IntPtr WindowProcedure;
public int ClassAdditionalBytes;
public int WindowAdditionalBytes;
public IntPtr Instance;
public IntPtr Icon;
public IntPtr Cursor;
public IntPtr BackgroundBrush;
public char* ClassMenuResourceName;
public char* ClassName;
public IntPtr SmallIcon;
}
internal static unsafe WindowClass ConvertToManaged(WindowClassUnmanaged unmanaged)
{
return new()
{
WindowProcedure = Marshal.GetDelegateForFunctionPointer<Win32API.WindowProcedure>(unmanaged.WindowProcedure),
ClassMenuResourceName = MarshalHelpers.Win32WideCharArrToString(unmanaged.ClassMenuResourceName),
ClassName = MarshalHelpers.Win32WideCharArrToString(unmanaged.ClassName),
// (remainder omitted, just simple copies)
};
}
internal unsafe ref struct ManagedToUnmanagedIn
{
public static int BufferSize => sizeof(WindowClassUnmanaged);
private byte* UnmanagedBufferStruct;
private char* UnmanagedStrResourceName, UnmanagedStrClassName;
public void FromManaged(WindowClass managed, Span<byte> buffer)
{
IntPtr WindowProcedure = (managed.WindowProcedure == null) ? IntPtr.Zero : Marshal.GetFunctionPointerForDelegate(managed.WindowProcedure);
this.UnmanagedStrResourceName = (managed.ClassMenuResourceName == null) ? null : (char*)Marshal.StringToHGlobalUni(managed.ClassMenuResourceName);
this.UnmanagedStrClassName = (managed.ClassName == null) ? null : (char*)Marshal.StringToHGlobalUni(managed.ClassName);
WindowClassUnmanaged Result = new()
{
WindowProcedure = WindowProcedure,
ClassMenuResourceName = this.UnmanagedStrResourceName,
ClassName = this.UnmanagedStrClassName,
// (remainder omitted, just simple copies)
};
Span<byte> ResultByteView = MemoryMarshal.Cast<WindowClassUnmanaged, byte>(MemoryMarshal.CreateSpan(ref Result, 1));
Debug.Assert(buffer.Length >= ResultByteView.Length, "Target buffer isn't large enough to hold the struct data.");
ResultByteView.CopyTo(buffer);
this.UnmanagedBufferStruct = (byte*)Unsafe.AsPointer(ref MemoryMarshal.GetReference(buffer));
}
public byte* ToUnmanaged() => this.UnmanagedBufferStruct;
public void Free()
{
if (this.UnmanagedStrResourceName != null)
{
Marshal.FreeHGlobal((nint)this.UnmanagedStrResourceName);
this.UnmanagedStrResourceName = null;
}
if (this.UnmanagedStrClassName != null)
{
Marshal.FreeHGlobal((nint)this.UnmanagedStrClassName);
this.UnmanagedStrClassName = null;
}
}
}
}
With this helper function to convert a Win32 LP(C)WSTR into a regular .NET string:
public static unsafe string? Win32WideCharArrToString(char* unmanagedArr)
{
if (unmanagedArr == null) { return null; }
int Length = 0;
while (*(unmanagedArr + Length) != 0x0000) { Length++; }
return Encoding.Unicode.GetString((byte*)unmanagedArr, Length * sizeof(char));
}
The nicer WindowClass struct is pretty much the same as before, except readonly, and with all elements being { get; init; }. The MarshalAs attributes on members are no longer required, as the custom marshaling handles everything.
Finally, the actual extern function now looks like this:
[LibraryImport("user32.dll", SetLastError = true, EntryPoint = "RegisterClassExW")]
[UnmanagedCallConv(CallConvs = new[] { typeof(CallConvStdcall) })]
public static partial ushort RegisterClassEx([MarshalUsing(typeof(WindowClassMarshaler))] WindowClass classDefinition);
Note that this has been corrected. Previously I used the in keyword on the parameter, but this causes it to pass a pointer to the pointer to the struct data, which is an extra level of indirection that will cause the code to fail. Above is the updated version that works correctly.
I've tested and verified this works both in regular publish modes, as well as with AOT compilation, which was the reason for using LibraryImport in this case.
My bonus question still stands however, is there any benefit in specifying stdcall using UnmanagedCallConv?

COM interop and marshaling of VARIANT(VT_PTR)

We use a 3rd party COM object, one of which methods under certain conditions returns a VARIANT of VT_PTR type. That upsets the default .NET marshaler, which throws the following error:
Managed Debugging Assistant 'InvalidVariant' : 'An invalid VARIANT was
detected during a conversion from an unmanaged VARIANT to a managed
object. Passing invalid VARIANTs to the CLR can cause unexpected
exceptions, corruption or data loss.
Method signatures:
// (Unmanaged) IDL:
HRESULT getAttribute([in] BSTR strAttributeName, [retval, out] VARIANT* AttributeValue);
// C#:
[return: MarshalAs(UnmanagedType.Struct)]
object getAttribute([In, MarshalAs(UnmanagedType.BStr)] string strAttributeName);
Is there an elegant way to bypass such marshaler's behavior and obtain the underlying unmanaged pointer on the managed side?
What I've considered/tried so far:
A custom marshaler:
[return: MarshalAs(UnmanagedType.CustomMarshaler,
MarshalTypeRef = typeof(IntPtrMarshaler))]
object getAttribute([In, MarshalAs(UnmanagedType.BStr)] string strAttributeName);
I did implement IntPtrMarshaler, just to find the interop layer crashing the process even before any of my ICustomMarshaler methods gets called. Perhaps, the VARIANT* argument type is not compatible with custom marshalers.
Rewrite (or clone) the C# interface definition with getAttribute method redefined (like below) and do all the marshaling for output VARIANT manually:
void getAttribute(
[In, MarshalAs(UnmanagedType.BStr)],
string strAttributeName,
IntPtr result);
This doesn't seem nice (the interface itself has 30+ other methods). It'd also break existing, unrelated pieces of code which already make use of getAttribute without issues.
Obtain an unmanaged method address of getAttribute from vtable (using Marshal.GetComSlotForMethodInfo etc), then do the manual invocation and marshaling against my own custom delegate type (using Marshal.GetDelegateForFunctionPointer etc).
So far, I've taken this approach and it seem to work fine, but it feels as such an overkill for what should be a simple thing.
Am I missing some other feasible interop options for this scenario? Or, maybe there is a way to make CustomMarshaler work here?

What I would do is define a simple VARIANT structure like this:
[StructLayout(LayoutKind.Sequential)]
public struct VARIANT
{
public ushort vt;
public ushort r0;
public ushort r1;
public ushort r2;
public IntPtr ptr0;
public IntPtr ptr1;
}
And the interface like this;
[Guid("39c16a44-d28a-4153-a2f9-08d70daa0e22"), InterfaceType(ComInterfaceType.InterfaceIsDual)]
public interface MyInterface
{
VARIANT getAttributeAsVARIANT([MarshalAs(UnmanagedType.BStr)] string strAttributeName);
}
Then, add an extension method somewhere in a static class like this, so the caller can have the same coding experience using MyInterface:
public static object getAttribute(this MyInterface o, string strAttributeName)
{
return VariantSanitize(o.getAttributeAsVARIANT(strAttributeName));
}
private static object VariantSanitize(VARIANT variant)
{
const int VT_PTR = 26;
const int VT_I8 = 20;
if (variant.vt == VT_PTR)
{
variant.vt = VT_I8;
}
var ptr = Marshal.AllocCoTaskMem(Marshal.SizeOf<VARIANT>());
try
{
Marshal.StructureToPtr(variant, ptr, false);
return Marshal.GetObjectForNativeVariant(ptr);
}
finally
{
Marshal.FreeCoTaskMem(ptr);
}
}
This will do nothing for normal variants, but will just patch it for VT_PTR cases.
Note this only works if the caller and the callee are in the same COM apartement.
If they are not, you will get the DISP_E_BADVARTYPE error back because marshaling must be done, and by default, it will be done by the COM universal marshaler (OLEAUT) which only support Automation compatible data types (just like .NET).
In this case, theoratically, you could replace this marshaler by another one (at COM level, not at NET level), but that would mean to add some code on C++ side and possibly in the registry (proxy/stub, IMarshal, etc.).

For my own future reference, here's how I ended up doing it, using the 3rd option mentioned in the question:
[ComImport, Guid("75A67021-058A-4E2A-8686-52181AAF600A"), InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
public interface IInterface
{
[return: MarshalAs(UnmanagedType.Struct)]
object getAttribute([In, MarshalAs(UnmanagedType.BStr)] string strAttributeName);
}
private delegate int IInterface_getAttribute(
IntPtr pInterface,
[MarshalAs(UnmanagedType.BStr)] string name,
IntPtr result);
public static object getAttribute(this IInterface obj, string name)
{
var ifaceType = typeof(IInterface);
var ifaceMethodInfo = ((Func<string, object>)obj.getAttribute).Method;
var slot = Marshal.GetComSlotForMethodInfo(ifaceMethodInfo);
var ifacePtr = Marshal.GetComInterfaceForObject(obj, ifaceType);
try
{
var vtablePtr = Marshal.ReadIntPtr(ifacePtr);
var methodPtr = Marshal.ReadIntPtr(vtablePtr, IntPtr.Size * slot);
var methodWrapper = Marshal.GetDelegateForFunctionPointer<IInterface_getAttribute>(methodPtr);
var resultVar = new VariantClass();
var resultHandle = GCHandle.Alloc(resultVar, GCHandleType.Pinned);
try
{
var pResultVar = resultHandle.AddrOfPinnedObject();
VariantInit(pResultVar);
var hr = methodWrapper(ifacePtr, name, pResultVar);
if (hr < 0)
{
Marshal.ThrowExceptionForHR(hr);
}
if (resultVar.vt == VT_PTR)
{
return resultVar.ptr;
}
try
{
return Marshal.GetObjectForNativeVariant(pResultVar);
}
finally
{
VariantClear(pResultVar);
}
}
finally
{
resultHandle.Free();
}
}
finally
{
Marshal.Release(ifacePtr);
}
}

Marshal bool return as custom Return<bool>

I am currently wrapping Un4seen BASS audio library in .Net
BASS functions returning a bool value, require checking LastError to know the Error that occurred if false is returned.
So I implemented a Return<T> class to make it object-oriented:
using ManagedBass.Dynamics;
namespace ManagedBass
{
/// <summary>
/// Wraps a Bass Error in a function return value
/// </summary>
/// <typeparam name="T">The Type of the function return value</typeparam>
public class Return<T>
{
public Errors ErrorCode { get; }
Return(T Value)
{
ErrorCode = Bass.LastError;
this.Value = Value;
}
public T Value { get; }
public static implicit operator T(Return<T> e) => e.Value;
public static implicit operator Return<T>(T e) => new Return<T>(e);
public bool Success => ErrorCode == Errors.OK;
}
}
Now for methods like:
[DllImport(DllName, EntryPoint = "BASS_StreamFree")]
public static extern bool StreamFree(int Handle);
I would instead want to return a Return<bool> so that the user can easily check the ErrorCode in object-oriented way.
Now, you may say that I create another method like:
// bool implicitly converts to Return<bool>
public static Return<bool> StreamFree2(int Handle) => StreamFree(Handle)
But, the problem is that there are already thousands of DllImports and doing this for all of them would be a nightmare.
So, I tried:
[DllImport(DllName, EntryPoint = "BASS_StreamFree")]
[return: MarshalAs(UnmanagedType.Bool)]
public static extern Return<bool> StreamFree(int Handle);
It compiled, but didn't work at all.
Can anyone please point me out where I am wrong or is something like this even possible.
Thanks in Advance.

There is no way to do this without writing wrapper functions. You cannot simply change the function's return type. The marshaller isn't going to be able to create a new Return<T> object automatically. Your final attempt at a declaration tells the marshaller that the function returns a BOOL, but the function is not declared as returning a BOOL. Obviously that's not going to work; it fails at runtime when the mismatch is detected.
There is likely another problem. I don't know what type the BASS library functions actually return, but if it is bool, then UnmanagedType.Bool is the wrong type. You actually want UnmanagedType.U1. UnmanagedType.Bool refers to the 4-byte BOOL type, which is actually an integer. UnmanagedType.U1 refers to C++'s 1-byte bool type.
More generally, I don't think your "object-oriented" approach would offer any real advantage. Sure, you'd be returning an object, but the client code you write wouldn't look any different because you'd still have to check the error code each time.
An elegant, .NET-conforming design would throw an exception when the library function returned an error. That exception (BassException or whatever) would then wrap the error code. Of course, this will require writing wrapper functions for each of the library functions, but at least there will be some payoff once you get done. Consider automating the generation of such functions with a tool.

What's wrong with this interop C#/managed/unmanaged C++ parameter passing?

I'm trying to make a non-static vendor C++ DLL accessible via C#. In order to do this, I'm writing a managed C++ wrapper DLL which basically creates static variables for the vendor DLL and makes those accessible to the C# application.
Here's an example:
typedef void(__stdcall *LPLISTENER_FUNC)(VENDORHANDLE hModule, VENDORWPARAM wParam, VENDORLPARAM lParam);
public delegate void VENDOR_Delegate(VENDORHANDLE hModule,
VENDORWPARAM wParam, VENDORLPARAM lParam);
public class VENDORWrapper
{
private:
static VENDORHSTORAGE _hStorage;
static VENDOR_Delegate^ _hOpenCallback;
void static Initialize()
{
_hStorage=storage_initialize();
}
void static registerCallback(unsigned int type, VENDOR_Delegate^ callback)
{
if (type == 2)
{
_hOpenCallback = callback;
::storage_register_callback(_hStorage, type, (LPLISTENER_FUNC)&_hOpenCallback);
}
}
bool static Open(String^ file)
{
bool retval=false;
filePath = file;
IntPtr ip = Marshal::StringToHGlobalAuto(filePath);
LPCWSTR str = static_cast<LPCWSTR>(ip.ToPointer());
//ERROR OCCURS HERE
retval = storage_open(_hStorage, str);
Marshal::FreeHGlobal( ip );
return retval;
}
void static Close()
{
storage_close(_hStorage);
}
}
The C# is skeletal:
public static VENDORStorageWrapper.VENDOR_Delegate openCallback
= new VENDORStorageWrapper.VENDOR_Delegate(fileOpened);
static void Main(string[] args)
{
VENDORStorageWrapper.VENDORStorageWrapper.Initialize();
Debug.WriteLine("DLL initalized");
VENDORStorageWrapper.VENDORStorageWrapper.registerCallback(2,
openCallback);
Debug.WriteLine("Callback registered");
VENDORStorageWrapper.VENDORStorageWrapper.Open("blah_file");
Debug.WriteLine("File opened");
}
public static void fileOpened(System.Int32 hstorage, System.UInt32 wParam, System.Int32 lParam)
{
Debug.WriteLine("file opened");
}
The vendor DLL's functions are specified as __stdcall, so I think I'm compliant on that front. The vendor's initialize call (_storage_initialize above) seems to be properly setting the handle, which is statically scoped. The storage_open call that's leading into the exception accepts a VENDORHANDLE (really a long) and an LPCWSTR, which I'm trying to convert the string passed from C# to. I think that's where the problem is...
When run, the app throws an unhandled exception "System.Runtime.InteropServices.SEHException" at the commented line above. The exception's coming from inside the vendor DLL, which I have no source code for. The vendor library works perfectly when called in an unmanaged C++ context and the file is known to be good. I think I'm missing something obvious in how I'm handling the parameters, but I can't see what it is.
I also don't think I have the callback set up properly, but I'm not the point where I can test that yet. Any ideas?

I'm not sure of the true big picture, but my experience using native DLLs with .net c# or vb, create a simple c# wrapper (just declarations) to the native DLL, instead of a c++ wrapper. Maybe this will help if the vendor DLL is really a vanilla DLL, as most are, like Windows api calls.
namespace MyNameSpace
{
public class MyWrapper
{
// passing an int to the native DLL
[DllImport("Vendor.DLL")]
public static extern int DllFunc1(int hModule, int nData);
// passing a string to a native DLL expecting null terminated raw wide characters //
[DllImport("Vendor.DLL", CharSet=CharSet.Unicode )]
public static extern int Dllszset(int hModule, string text);
}
}
Then .net will handle it for you and you call your vendor function as...
MyNameSpace.MyWrapper.Dllszset(h, "hello");
Hope this helps you or someone.

Using TaskDialogIndirect in C#

I've been working for a while with the regular Windows Vista/7 TaskDialog for a while, and I wanted to add some additional functionality (like custom buttons and a footer), so I need to use TaskDialogIndirect.
Following the MSDN documentation for TaskDialogIndirect, I got this signature:
[DllImport("comctl32.dll",CharSet = CharSet.Unicode,EntryPoint="TaskDialogIndirect")]
static extern int TaskDialogIndirect (TASKDIALOGCONFIG pTaskConfig, out int pnButton, out int pnRadioButton, out bool pfVerificationFlagChecked);
The TASKDIALOGCONFIG class is shown below:
public class TASKDIALOGCONFIG
{
public UInt16 cbSize;
public IntPtr hwndParent;
public IntPtr hInstance;
public String dwFlags;
public String dwCommonButtons;
public IntPtr hMainIcon;
public String pszMainIcon;
public String pszMainInstruction;
public String pszContent;
public UInt16 cButtons;
public TASKDIALOG_BUTTON pButtons;
public int nDefaultButton;
public UInt16 cRadioButtons;
public TASKDIALOG_BUTTON pRadioButtons;
public int nDefaultRadioButton;
public String pszVerificationText;
public String pszExpandedInformation;
public String pszExpandedControlText;
public String pszCollapsedControlText;
public IntPtr hFooterIcon;
public IntPtr pszFooterText;
public String pszFooter;
// pfCallback;
// lpCallbackData;
public UInt16 cxWidth;
}
The TASKDIALOG_BUTTON implementation:
public class TASKDIALOG_BUTTON
{
public int nButtonID;
public String pszButtonText;
}
I am not entirely sure if I am on the right track here. Did anyone use TaskDialogIndirect from managed code directly through WinAPI (without VistaBridge or Windows API Code Pack)? I am curious about the possible implementations, as well as the callback declarations (I am not entirely sure how to implement TaskDialogCallbackProc).
PS: I am looking for a direct WinAPI implementation, not one through a wrapper.

Look into the VistaBridge library. It will give you a pleasant wrapper around all this stuff, including TaskDialogIndirect..
http://code.msdn.microsoft.com/VistaBridge

It's worth looking through the Windows API code pack source code as it contains a reasonably complete implementation of TaskDialogIndirect using WinAPI - including callbacks - which would be a good starting point for your own implementation.

PInvoke.NET is a great resource for PInvoke techniques. Unfortunately, they only have a TODO stub for TaskDialogIndirect so far.

Look at http://www.codeproject.com/Articles/21276/Vista-TaskDialog-Wrapper-and-Emulator. Specifically the file VistaUnsafeNativeMethods.cs has the relevant DllImport bits, including VistaTaskDialogCallback which I think is the callback you are interested in.