I have been trying to create a handle to a structure type because I need a pinned pointer to it, but I am getting the error "Object contains non-primitive or non-blittable data"
My structure looks like this:
[StructLayout(LayoutKind.Sequential)]
public struct MyStruct
{
[MarshalAs(UnmanagedType.U1)]
public bool Test;
}
Now, when I call,
var mystruct = new MyStruct();
var handle = GCHandle.Alloc(mystruct, GCHandleType.Pinned);
I get the error "Object contains non-primitive or non-blittable data". Now I understand that the bool field is a non-blittable type. But I was under the impression that by adding the MarshalAs attribute, I could tell the marshaller how to convert the type. (I also tried UnmanagedType.Bool)
This structure has to be defined globally, because it is needed throughout my class. The only reason I need the pointer is because I have an unmanaged API that must pass this structure as a pointer. Then I have to get that structure in a callback and read/update members.
So this is the basic scenario.
Structure is created globally in a managed class
Pointer to structure is obtained
Pointer to the structure is passed into the API
The API calls a static method callback where I then need to get my structure and read/update members.
I tried to use Marshal.StructureToPtr but this only creates a copy, so if in my managed class I update the member, when the callback is raised, the updated value is not there.
Does anyone know how I can get a pinned pointer to my structure so I can read/modify the public members and have them available in the callback?
Thanks
You've got more than one problem here. Using a struct is highly inadvisable. It will get boxed before the GCHandle.Alloc() call and that boxed object gets pinned. You cannot see any updates to it through your mystruct variable. Use a class instead.
And avoid bool, it is a non-blittable type due to its highly variable implementation. It is 4 bytes in C, 1 byte in C++, 2 bytes in COM. Just make it a byte instead. You can write a property to get it back to a bool.
So:
[StructLayout(LayoutKind.Sequential)]
public class MyStruct
{
private byte _test;
public bool Test {
get { return _test != 0; }
set { _test = value ? 1 : 0; }
}
}
You're right that you're telling the marshaller how to marshal the type.
But that won't do you any good when you then attempt to bypass the marshaller.
You need to decide whether you want to use the marshaller, or whether you want the unmanaged code to directly write into managed memory.
If you want to use the marshaller:
Generally, a good way to handle this is to use it in both directions. You can use Marshal.StructureToPtr (as you've found), call the external function, and then use Marshal.PtrToStructure to convert it back into your managed representation.
Or you can use methods that are set up in such a way that marshalling happens automatically, without you needing to specify this manually. For example, calling a native method taking a ref MyStruct parameter will allow for that to happen.
If you don't want to use the marshaller:
Don't use any types that require marshalling. As Hans Passant comments, use a different type instead, byte would probably be a good choice.
(I'll refrain from commenting on the advantages and disadvantages of using structs here, except that the points already made about it are well worth reading and understanding.)
Related
I have a device that transmits binary data. To interpret the data I have defined a struct that matches the data format. The struct has a StuctLayoutAttribute with LayoutKind.Sequential. This works as expected, e.g:
[StructLayout(LayoutKind.Sequential)]
struct DemoPlain
{
public int x;
public int y;
}
Marshal.OffsetOf<DemoPlain>("x"); // yields 0, as expected
Marshal.OffsetOf<DemoPlain>("y"); // yields 4, as expected
Marshal.SizeOf<DemoPlain>(); // yields 8, as expected
Now I wish to treat one struct similar to an other struct, so I experimented with the structure implementing an interface:
interface IDemo
{
int Product();
}
[StructLayout(LayoutKind.Sequential)]
struct DemoWithInterface: IDemo
{
public int x;
public int y;
public int Product() => x * y;
}
Marshal.OffsetOf<DemoWithInterface>("x").Dump(); // yields 0
Marshal.OffsetOf<DemoWithInterface>("y").Dump(); // yields 4
Marshal.SizeOf<DemoWithInterface>().Dump(); // yields 8
To my surprise the offsets and size of DemoWithInterface remain the same as DemoPlain and converting the same binary data from the device to either an an array of DemoPlain or an array of DemoWithInterface both work. How is this possible?
C++ implementations often use a vtable (see Where in memory is vtable stored?) to sore virtual methods. I believe that in C# methods published in an interface, and methods that are declared virtual, are similar to virtual methods in C++ and that it requires something similar to a vtable to find the correct method. Is this correct or does C# do it completely different? If correct, where is the vtable like structure stored? If different, how is C# implemented with respect to interface inheritance and virtual methods?
Basically, "does not apply". Structs in C# - as has been discussed - do not support inheritance, and so no v-table is required.
The field layout is the field layout. It is simply: where the actual fields are. Implementing interfaces doesn't change the fields at all, and doesn't require any change to the layout. So that's why the size and layout isn't impacted.
There are some virtual methods that structs can (and usually should) override - ToString() etc. So you can legitimately ask "so how does that work?" - and the answer is: smoke and mirrors. Also known as constrained call. This defers the question of "virtual call vs static call" to the JIT. The JIT has full knowledge as to whether the method is overridden or not, and can emit appropriate opcodes - either a box and virtual call (a box is an object, so has a v-table), or a direct static call.
It might be tempting to think that the compiler should do this, not the JIT - but often the struct is in an external assembly, and it would be catastrophic if the compiler emitted a static call because it could see the overridden ToString() etc, and then someone updates the library without rebuilding the app, and it gets a version that doesn't override (MissingMethodException) - so constrained call is more reliable. And doing the same thing even for in-assembly types is just simpler and easier to support.
This constrained call also happens for generic (<T>) methods - since the T could be a struct. Recall that the JIT executes per T for value-typed T on a generic method, so it can apply this logic per-type, and bake in the actual known static call locations. And if you're using something like .ToString() and your T is a struct that doesn't override that: it will box and virtual-call instead.
Note that once you assign a struct to an interface variable - for example:
DemoWithInterface foo = default;
IDemo bar = foo;
var i = bar.Product();
you have "boxed" it, and everything is now virtual-call on the box. A box has a full v-table. That is why generic methods with generic type constraints are often preferable:
DemoWithInterface foo = default;
DoSomething(foo);
void DoSomething<T>(T obj) where T : IDemo
{
//...
int i = obj.Product();
//...
}
will use constrained call throughout and will not require a box, despite accessing interface members. The JIT resolves the static call options for the specific T at execution.
Default Marshalling Behavior | Microsoft Docs and especially the section Value Types Used in Platform Invoke provides the answer:
When marshaled to unmanaged code, these formatted types are marshaled as C-style structures.
and
When a formatted type is marshaled as a structure, only the fields within the type are accessible. If the type has methods, properties, or events, they are inaccessible from unmanaged code.
So the C# struct's (virtual) methods are stripped and only a plain C-struct in transmitted. In the OP's case the device sends the bytes which comprise a plain C-struct, Marshal.PtrToStructure<T>(IntPtr) converts the bytes to a C#-struct and, in case of DemoWithInterface, attaches the Product-method and the vtable (or other meta-data) to make the struct implement IDemo.
I am currently writing a thin C# binding for OpenGL. I've just recently implemented the OpenGL GenVertexArrays function, which has the following signature:
OpenGL Documentation on glGenVertexArrays.
Essentially, you pass it an array in which to store generated object values for the vertex arrays created by OpenGL.
In order to create the binding, I use delegates as glGenVertexArrays is an OpenGL extension function, so I have to load it dynamically using wglGetProcAddress. The delegate signature I have defined in C# looks like this:
[SuppressUnmanagedCodeSecurity]
[UnmanagedFunctionPointer(CallingConvention.StdCall)]
private delegate void glGenVertexArrays(uint amount, uint[] array);
The function pointer is retrieved and converted to this delegate using Marshal.GetDelegateForFunctionPointer, like this:
IntPtr proc = wglGetProcAddress(name);
del = Marshal.GetDelegateForFunctionPointer(proc, delegateType);
Anyways, here's what bothers me:
In any official documentation I can find on default marshalling behaviour for reference types (which includes arrays), is this:
By default, reference types (classes, arrays, strings, and interfaces)
passed by value are marshaled as In parameters for performance
reasons. You do not see changes to these types unless you apply
InAttribute and OutAttribute (or just OutAttribute) to the method
parameter.
This is taken from this MSDN page: MSDN page on directional attributes
However, as can be seen from my delegate signatures, the [In] and [Out] directional attributes have not been used on the array of unsigned integers, meaning when I call this function I should actually not be able to see the generated object values which OpenGL should have stored in them. Except, I am. Using this signature, I can the following result when running the debugger:
As can be seen, the call absolutely did affect the array, even though I did not explicitly use the [Out] attribute. This is not, from what I understand, a result I should expect.
Does anyone know the reason behind this? I know it might seem as a minor deal, but I am very curious to know why this seems to break the default marshalling behaviour described by Microsoft. Is there some behind-the-scenes stuff going on when invoking delegates compared to pure platform invoke prototypes? Or am I misinterpreting the documentation?
[EDIT]
For anyone curious, the public method that invokes the delegate is defined on a static "GL" class, and is as followed:
public static void GenVertexArrays(uint amount, uint[] array)
{
InvokeExtensionFunction<glGenVertexArrays>()(amount, array);
}
It is not mentioned on the documentation page you linked, but there is another topic dedicated to the marshaling of arrays, where it says:
With pinning optimization, a blittable array can appear to operate as an In/Out parameter when interacting with objects in the same apartment.
Both conditions are met in your case: array of uint is blittable, and there is no machine-to-machine marshaling. It is still a good idea to declare it [Out], so your intention is documented within the code.
The documentation is correct in the general case. But uint is a bit special, it is a blittable type. An expensive word that means that the pinvoke marshaller does not have to do anything special to convert the array element values. An uint in C# is exactly the same type as an unsigned int in C. Not a coincidence at all, it is the kind of type that a processor can handle natively.
So the marshaller can simply pin the array and pass a pointer to the first array element as the second argument. Very fast, always what you want. And the function scribbles directly into the managed array, so copying the values back is not necessary. A bit dangerous too, you never ever want to lie about the amount argument, GC heap corruption is an excessively ugly bug to diagnose.
Most simple value types and structs of simple values types are blittable. bool is a notable exception. You'll otherwise never have to be sorry for using [Out] even if it is not necessary. The marshaller simply ignores it here.
I've been going round and round in circles on Google on this, and I can find all kinds of discussion, lots of suggestions, but nothing seems to work. I have an ActiveX component which takes an image as a byte array. When I do a TLB import, it comes in with this signature:
int HandleImage([MarshalAs(UnmanagedType.Struct)] ref object Bitmap);
How do I pass a byte[] to that?
There's another function which can return the data with a similar signature, and it works because I can pass "null" in. The type that comes back is a byte[1..size] (non-zero bounded byte[]). But even if I try to pass in what came back, it still gets a type mismatch exception.
More details:
I've been editing the method in the IDispatch interface signature (using ILSpy to extract the interface from the auto-generated interop assembly). I've tried just about every combination of the following, it always gets Type mismatch exception:
Adding and removing the "ref"
Changing the parameter datatype to "byte[]" or "Array"
Marshalling as [MarshalAs(UnmanagedType.SafeArray, SafeArraySubType = VarEnum.VT_UI1)]. After playing around with MarshalAs quite a bit, I'm becoming convinced that IDispatch does not use those attributes.
Also tried using the "ref object" interface as is, and passing it different types: byte[], Array.CreateInstance(typeof(byte) (which I think are both identical, but I found someone suggesting it, so it couldn't hurt to try).
Here's an example of Delphi code that creates a proper array to pass in:
var
image: OLEVariant;
buf: Pointer;
image := VarArrayCreate([0, Stream.Size], VarByte);
Buf := VarArrayLock(image);
Stream.ReadBuffer(Buf^, Stream.Size);
VarArrayUnlock(image);
Here's the C code to do the same thing. I guess if I can't get it to work from C#, I can invoke it through managed C++, although I'd rather have everything in one project:
long HandleImage(unsigned char* Bitmap, int Length)
{
VARIANT vBitmap;
VariantInit (&vBitmap);
VariantClear(&vBitmap);
SAFEARRAYBOUND bounds[1];
bounds[0].cElements = Length;
bounds[0].lLbound = 1;
SAFEARRAY* arr = SafeArrayCreate(VT_UI1, 1, bounds);
SafeArrayLock(arr);
memcpy(arr->pvData, Bitmap, Length);
SafeArrayUnlock(arr);
vBitmap.parray = arr;
vBitmap.vt = VT_ARRAY | VT_UI1;
long result;
static BYTE parms[] = VTS_PVARIANT;
InvokeHelper(0x5e, DISPATCH_METHOD, VT_I4, (void*)&result, parms,
&vBitmap);
SafeArrayDestroy(arr);
VariantClear(&vBitmap);
return result;
}
I finally figured out how to do it in 100% C# code. Apparently Microsoft never considered the idea that someone might use a method with this signature to pass data in, since it marshals correctly going the other direction (it properly comes back as a byte[]).
Also, ICustomMarshaler doesn't get called on IDispatch calls, it never hit the breakpoints in the custom marshaler (except the static method to get an instance of it).
The answer by Hans Passant in this question got me on the right track: Calling a member of IDispatch COM interface from C#
The copy of IDispatch there doesn't contain the "Invoke" method on IUnknown, but it can be added to the interface, using types in System.Runtime.InteropServices.ComTypes as appropriate: http://msdn.microsoft.com/en-us/library/windows/desktop/ms221479%28v=vs.85%29.aspx
That means you get 100% control over marshaling arguments. Microsoft doesn't expose an implementation of the VARIANT structure, so you have to define your own: http://limbioliong.wordpress.com/2011/09/19/defining-a-variant-structure-in-managed-code-part-2/
The input parameters of Invoke are a variant array, so you have to marshal those to an unmanaged array, and there's a variant output parameter.
So now that we have a variant, what should it contain? This is where automatic marshaling falls down. Instead of directly embedding a pointer to the SAFEARRAY, it needs a pointer to another variant, and that variant should point to the SAFEARRAY.
You can build SAFEARRAYs via P/Invoking these methods: http://msdn.microsoft.com/en-us/library/windows/desktop/ms221145%28v=vs.85%29.aspx
So main variant should be VT_VARIANT | VT_BYREF, it should point to another variant VT_UI8 | VT_ARRAY, and that should point to a SAFEARRAY generated via SafeArrayCreate(). That outermost variant should then be copied into a block of memory, and its IntPtr sent to the Invoke method.
Here is a C# program that tries Marshal.SizeOf on a few different types:
using System;
using System.Runtime.InteropServices;
[StructLayout(LayoutKind.Sequential)]
class AClass { }
[StructLayout(LayoutKind.Sequential)]
struct AStruct { }
[StructLayout(LayoutKind.Sequential)]
class B { AClass value; }
[StructLayout(LayoutKind.Sequential)]
class C<T> { T value; }
class Program
{
static void M(object o) { Console.WriteLine(Marshal.SizeOf(o)); }
static void Main()
{
M(new AClass());
M(new AStruct());
M(new B());
M(new C<AStruct>());
M(new C<AClass>());
}
}
The first four calls to M() succeed, but on the last one, SizeOf throws an ArgumentException:
"Type 'C`1[AClass]' cannot be marshaled as an unmanaged structure; no meaningful size or offset can be computed."
Why? Specifically, why does SizeOf choke on C<AClass>, but not on B or on C<AStruct>?
EDIT: Because it was asked about in the comments, here's the "real-world" problem that inspired this mostly-academic question:
I'm calling into a C API that is basically one C function that operates on (pointers to) lots of different types of simple C structures. All contain a common header followed by one field, but the type of that field is different in different structures. A flag in the header indicates the type of the field. (Strange, yes, but that's what I have to work with).
If I could define a single generic type C<T> and a single C# extern declaration M(C<T>), and then call M(C<int>) on one line, and M(C<double>) on another, I'd have a short and sweet interop solution. But given JaredPar's answer, it appears that I have to make a separate C# type for each structure (though inheritance can provide the common header).
Generics as a rule are not supported in any interop scenario. Both P/Invoke and COM Interop will fail if you attempt to marshal a generic type or value. Hence I would expect Marshal.SizeOf to be untested or unsupported for this scenario as it is a marshal-specific function.
It is not known what size the aggregated object T would have (it would be the size of a pointer if T is a referenece type and mostly any value if it is value type).
I think you can solve this problem by setting the MarshalAs attribute on the field 'value' specifying the most matching type (for example, Unmanagedtype.SysInt). Note that it still won't work for so-called non-mappable types (i.e. types for which fields offsets and sizes can't be deduced easily).
But AFAIK, it's not recommended to use generics in interop.
In my C# application, I have a large struct (176 bytes) that is passed potentially a hundred thousand times per second to a function. This function then simply takes a pointer to the struct and passes the pointer to unmanaged code. Neither the function nor the unmanaged code will make any modifications to the struct.
My question is, should I pass the struct to the function by value or by reference? In this particular case, my guess is that passing by reference would be much faster than pushing 176 bytes onto the call stack, unless the JIT happens to recognize that the struct is never modified (my guess is it can't recognize this since the struct's address is passed to unmanaged code) and optimizes the code.
Since we're at it, let's also answer the more general case where the function does not pass the struct's pointer to unmanaged code, but instead performs some read-only operation on the contents of the struct. Would it be faster to pass the struct by reference? Would in this case the JIT recognize that the struct is never modified and thus optimize? Presumably it is not more efficient to pass a 1-byte struct by reference, but at what struct size does it become better to pass a struct by reference, if ever?
Thanks.
EDIT:
As pointed out below, it's also possible to create an "equivalent" class for regular use, and then use a struct when passing to unmanaged code. I see two options here:
1) Create a "wrapper" class that simply contains the struct, and then pin and pass a pointer to the struct to the unmanaged code when necessary. A potential issue I see is that pinning has its own performance consequences.
2) Create an equivalent class whose fields are copied to the struct when the struct is needed. But copying would take a lot of time and seems to me to defeat the point of passing by reference in the first place.
EDIT:
As mentioned a couple times below, I could certainly just measure the performance of each of these methods. I will do this and post the results. However, I am still interested in seeing people's answers and reasonings from an intellectual perspective.
I did some very informal profiling, and the results indicate that, for my particular application, there is a modest performance gain for passing by reference. For by-value I got about 10,050,000 calls per second, whereas for by-reference I got about 11,200,000 calls per second.
Your mileage may vary.
Before you ask whether or not you should pass the struct by reference, you should ask yourself why you've got such an enormous struct in the first place. Does it really need to be a struct? If you need to use a struct at some point for P/Invoke, would it be worth having a struct just for that, and then the equivalent class elsewhere?
A struct that big is very, very unusual...
See the Design Guidelines for Developing Class Libraries section on Choosing Between Classes and Structures for more guidance on this.
The only way you can get an answer to this question is to code up both and measure the performance.
You mention unmanaged/managed interop. My experience is that it takes a surprisingly long time to to the interop. You could try changing your code from:
void ManagedMethod(MyStruct[] items) {
foreach (var item in items) {
unmanagedHandle.ProcessOne(item);
}
}
To:
void ManagedMethod(MyStruct[] items) {
unmanagedHandle.ProcessMany(items, items.Count);
}
This technique helped me in a similar case, but only measurements will tell you if it works for your case.
Why not just use a class instead, and pass your class to your P/Invoke function?
Using class will pass nicely around in your managed code, and will work the same as passing a struct by reference to a P/Invoke function.
e.g.
// What you have
public struct X
{
public int data;
}
[DllImport("mylib.dll")]
static extern void Foo( ref X arg);
// What you could do
[StructLayout(LayoutKind.Sequential)]
public class Y
{
public int data;
}
[DllImport("mylib.dll")]
static extern void Bar( Y arg );