Develop Reference

C# marshal struct with complex array to bytes

I have to interop with a device, through a C++ interface that accepts byte[] for both in/out streams.
Many of the data being passed around are in the form of an outer struct containing some numbers, and a dynamic array of inner structs (which in turn may contain arrays of their own). A simple form would be:
[StructLayout(LayoutKind.Sequential, Pack = 1)]
public struct Outer
{
public float Outer_1;
//...
public double Outer_N;
public Inner[] InnerArray; // variable length
}
[StructLayout(LayoutKind.Sequential, Pack = 1)]
public struct Inner
{
public Int16 Inner_1;
//...
public Int32 Inner_N;
}
Simple nested (non-arrayed) structs work fine, either using a fixed(byte*) + Marshal.StructureToPtr(), or GCHandle.Alloc() + Marshal.Copy().
I do know that the array is just a pointer, and its contents would need separate marshalling; but I was hoping I could at least start by marshalling the non-array Outer_1 .. Outer_N part of the structure, followed by a loop to marshal the array contents to different locations of the byte array. I'm trying to avoid having to marshal Outer_1..N one-by-one.
And this is where things fall apart. Neither of the usual methods work.
[Fact]
public unsafe void TestSerialize()
{
Outer managed = new()
{
Outer_1 = 1.0f,
Outer_N = 3.14,
InnerArray = new[] {
new Inner { Inner_1 = 1, Inner_N = Int32.MinValue },
new Inner { Inner_1 = 10, Inner_N = Int32.MaxValue },
},
};
var buffer = new byte[500]; // Could be calculated, but won't bother right now.
//--- Attempt 1:
fixed (byte* pBuf = buffer)
{
//System.NotSupportedException: 'Operation is not supported. (0x80131515)'
Marshal.StructureToPtr(managed, new IntPtr(pBuf), false);
}
//--- Attempt 2:
//System.ArgumentException: 'Object contains non-primitive or non-blittable data'
var managedHandle = GCHandle.Alloc(managed, GCHandleType.Pinned);
Marshal.Copy(
source: managedHandle.AddrOfPinnedObject(),
destination: buffer,
startIndex: 0, length: 12);
managedHandle.Free();
}
Is there a way to marshal the Outer struct without me marshalling Outer_1, Outer_1 ... Outer_N by hand? Probably fine if I'm just dealing with the one, but there's tons of different messages I need to marshal.

Casting a managed array to an array of structs without copying

I have a managed array provided by a third party library. The type of the array does not directly reflect the data stored in the array, but instead it's the data interpreted as integers.
So int[] data = Lib.GetData(); gives me an integer array, that I would like to cast into an array of DataStructure, that could look like this.
struct DataStructure {
public int Id;
public double Value;
}
Current I use Marshal.Copy (an implementation can been seen here), but it seems a bit excessive to copy the entire thing.
Does something like this exists:
int[] data = Lib.GetData();
DataStructure[] dataStructs = InterpretAs<DataStructure>(data);
where no copying is required, but access to dataStruct elements can be done like dataStruct[1].Id?
EDIT 2:
If I just want a single DataStructure, I can use
public static T ToStruct<T>(byte[] bytes) where T : struct
{
GCHandle handle = GCHandle.Alloc(bytes, GCHandleType.Pinned);
T something = Marshal.PtrToStructure<T>(handle.AddrOfPinnedObject());
handle.Free();
return something;
}
where no copying is required.
EDIT:
The answers possible duplicate are currently about 7 years old, and they consists of either copying the data or implementing a hack.

There is actually a cheat, but it is an ugly unsafe totally unsafe cheat:
[StructLayout(LayoutKind.Sequential)]
//[StructLayout(LayoutKind.Sequential, Pack = 4)]
public struct DataStructure
{
public int Id;
public double Value;
}
[StructLayout(LayoutKind.Explicit)]
public struct DataStructureConverter
{
[FieldOffset(0)]
public int[] IntArray;
[FieldOffset(0)]
public DataStructure[] DataStructureArray;
}
and then you can convert it without problems:
var myarray = new int[8];
myarray[0] = 1;
myarray[3] = 2;
//myarray[4] = 2;
DataStructure[] ds = new DataStructureConverter { IntArray = myarray }.DataStructureArray;
int i1 = ds[0].Id;
int i2 = ds[1].Id;
Note that depending on the size of DataStructure (if it is 16 bytes or 12 bytes), you have to use Pack = 4 (if it is 12 bytes) or you don't need anything (see explanation (1) later)
I'll add that this technique is undocumented and totally unsafe. It even has a problem: ds.Length isn't the length of the DataStructure[] but is the length of the int[] (so in the example given it is 8, not 2)
The "technique" is the same I described here and originally described here.
explanation (1)
The sizeof(double) is 8 bytes, so Value is normally aligned on the 8 bytes boundary, so normally there is a "gap" between Id (that has sizeof(int) == 4) and Value of 4 bytes. So normally sizeof(DataStructure) == 16. Depending on how the DataStructure is built, there could not be this gap, so the Pack = 4 that forces alignment on the 4 byte boundary.

PInvoke Pointer to struct including float array

I'm attempting to P Invoke a C library for use on a Xamarin android app.
Consider the following C structure:
typedef struct
{
bool myBool;
myOtherStruct sOtherStruct;
int myInt;
float myFloat;
float myFloatArray[1024];
float myFloatArray2[1024];
float myFloatArray3[20];
float myFloatArray4[30];
} sMyStruct;
This is called using the following function:
unsigned int initialise(sMyStruct* a_pMyStruct)
I've put this into a C# structure:
[StructLayout(LayoutKind.Sequential)]
public unsafe struct SMyStruct
{
bool myBool;
myOtherStruct sOtherStruct;
int myInt;
float myFloat;
public fixed float myFloatArray[1024];
public fixed float myFloatArray2[1024];
public fixed float myFloatArray3[20];
public fixed float myFloatArray4[30];
public unsafe float[] MyFloatArray
{
get
{
fixed (float* ptr = myFloatArray)
{
float[] array = new float[1024];
Marshal.Copy((IntPtr)ptr, array, 0, 1024 * sizeof(float));
return array;
}
}
}
public SMyStruct (bool MyBool, myOtherStruct otherStruct, int MyInt, float MyFloat)
{
myBool = MyBool;
sOtherStruct = myOtherStruct;
myInt = MyInt;
myFloat = MyFloat;
}
Here's my function in C# to invoke this:
[DllImport("libMylib")]
private static extern unsafe uint initialise(SMyStruct* a_pMyStruct);
I then call this function with:
public unsafe void init ()
{
SMyStruct initStruct;
uint result = initialise(&initStruct);
}
So what happens is the C function will return my structure with A LOT of data. I then pass the structure again into another C routine which uses these parameters for the rest of the program.
My issue is how do I get the float array data back into the correct structure variable so that I can pass it again? At current my code has been based on these questions:
Marshalling float Array to c# and
Marshalling complex struct to c#
But I've not managed to code this so I can pass the float array back to my struct without even seeing a compiler error (let alone failing when I test it!)
How do I get the float array data into my structure?
EDIT
After a couple of answers and comments, I'm adding what I was doing initially to try and add some clarity.
I get a compiler error when rather than using the "public unsafe float[]..." routine above I do this (within the struct):
public SMyStruct (bool MyBool, myOtherStruct otherStruct, int MyInt, float MyFloat, float* MyFloatArray, float* MyFloatArray2, float* MyFloatArray3, float* MyFloatArray4)
{
myBool = MyBool;
sOtherStruct = myOtherStruct;
myInt = MyInt;
myFloat = MyFloat;
myFloatArray = MyFloatArray;
myFloatArray2 = MyFloatArray2;
myFloatArray3 = MyFloatArray3;
myFloatArray4 = MyFloatArray4;
}
With this code I get the error "You cannot use fixed size buffers contained in unfixed expressions. Try using the fixed statement"
At this point I attempted to use the copy routine instead.
What I want is to ensure the fields myFloatArray, myFloatArray2 etc. to be populated with whatever the initialise function returns. FYI myBool, sOtherStruct etc. are populated as I expect.

If you don't need to access the data you can leave it as a pointer. Although it looks like you're responsible for the memory, so you'll need to alloc and later free the unmanaged memory you are using. Something like...
[DllImport("libMylib")]
private static extern uint initialise(IntPtr a_pMyStruct);
[DllImport("libMylib")]
private static extern uint anotherNativeMethod(IntPtr a_pMyStruct);
//...
//How big is myOtherStruct??
int size = 1 + ?? + 4 + 4 + (1024*4) + (1024*4) + (20*4) + (30*4);
//int size = Marhsal.SizeOf(SMyStruct);
IntPtr pMyStruct = Marshal.AllocHGlobal(size);
initialise(pMyStruct);
anotherNativeMethod(pMyStruct);
Marshal.FreeHGlobal(pMyStruct);
Note that you can still use the get the Marshaller to copy the pointer to your structure using Marshal.PtrToStructure but you no longer need to depend on it for your code.

I suspect that many of your problems are caused by you attempting to run before you can walk. You have attempted to make a complex struct with many members. Make a single mistake in one place, and nothing works anywhere.
So, how can we simplify? Well, the question you ask relates to the fixed buffers. To paraphrase the question you are asking:
How can I copy an array to and from a fixed buffer?
Let's deal with that by working with a simplified type that only contains a fixed buffer, and prove that we can copy to and from that buffer.
Your property getter is along the right lines. The biggest problem is that you pass an incorrect length. The final argument of the Marshal.Copy overload that you are calling is the number of elements. You erroneously pass the number of bytes. The setter is very similar in nature to the getter. It all looks like this:
using System;
using System.Text;
using System.Runtime.InteropServices;
namespace ConsoleApplication1
{
public unsafe struct MyStruct
{
private const int floatArrayLength = 4;
private fixed float _floatArray[floatArrayLength];
public float[] floatArray
{
get
{
float[] result = new float[floatArrayLength];
fixed (float* ptr = _floatArray)
{
Marshal.Copy((IntPtr)ptr, result, 0, floatArrayLength);
}
return result;
}
set
{
int length = Math.Min(floatArrayLength, value.Length);
fixed (float* ptr = _floatArray)
{
Marshal.Copy(value, 0, (IntPtr)ptr, length);
for (int i = length; i < floatArrayLength; i++)
ptr[i] = 0;
}
}
}
}
class Program
{
static void WriteArray(float[] arr)
{
foreach (float value in arr)
{
Console.Write(value);
Console.Write(" ");
}
Console.WriteLine();
}
static void Main(string[] args)
{
MyStruct myStruct = new MyStruct();
WriteArray(myStruct.floatArray);
myStruct.floatArray = new float[] { 1, 2, 3, 4 };
WriteArray(myStruct.floatArray);
myStruct.floatArray = new float[] { 5, 6 };
WriteArray(myStruct.floatArray);
Console.ReadLine();
}
}
}
The output of the program is:
0 0 0 0
1 2 3 4
5 6 0 0
This building block shows you how to handle your fixed buffers. You can use the code from this program and be sure that the fixed buffers are handled correctly. When you move to a more complex structure, if you have any problems you can be confident that they are not related to the fixed buffer code.

Why is this struct cleared after being created?

I have a struct that looks something like this:
[StructLayout(LayoutKind.Sequential)]
public struct in_addr {
public Anonymous1 S_un;
[StructLayoutAttribute(LayoutKind.Explicit)]
public struct Anonymous1 {
[FieldOffsetAttribute(0)]
public Anonymous2 S_un_b;
[FieldOffsetAttribute(0)]
public Anonymous3 S_un_w;
[FieldOffsetAttribute(0)]
public uint S_addr;
}
[StructLayoutAttribute(LayoutKind.Sequential)]
public struct Anonymous2 {
public byte s_b1;
public byte s_b2;
public byte s_b3;
public byte s_b4;
}
[StructLayoutAttribute(LayoutKind.Sequential)]
public struct Anonymous3 {
public ushort s_w1;
public ushort s_w2;
}
public in_addr(byte[] address) {
S_un.S_addr = (uint)BitConverter.ToInt32(address, 0);
S_un.S_un_b.s_b1 = address[0];
S_un.S_un_b.s_b2 = address[1];
S_un.S_un_b.s_b3 = address[2];
S_un.S_un_b.s_b4 = address[3];
S_un.S_un_w.s_w1 = 0;
S_un.S_un_w.s_w2 = 0;
}
}
When I try to create a new instance of this struct, every byte field is set to 0.
in_addr temp = new in_addr(bytes);
I've stepped through the struct's constructor and verified that the bytes are indeed getting assigned to the fields. But when I step out of the constructor and check the value of temp, everything is 0.
What's causing this?

Because of these lines:
S_un.S_un_w.s_w1 = 0;
S_un.S_un_w.s_w2 = 0;
They are mapped to your 4 bytes via
[FieldOffsetAttribute(0)]
public Anonymous2 S_un_b;
[FieldOffsetAttribute(0)]
public Anonymous3 S_un_w;

I can't reproduce this; your code works for me. I’m using Mono but I very much doubt that this is a bug in the Microsoft C# compiler, it's more likely that you've got an error elsewhere.

I just tested this in VisualStudio 2008 using C# and got the expected output (1, 2).
Try posting the actual example code that you're having an issue with.
EDIT This is why example code can be bad ;)
Your issue was with the StructLayout.Explicit on Anonymous1. Is there a particular reason that
You did explicit instead of sequential and putting them in the order you wanted
You left the indices as 0 for all of them
When I changed it to Sequential and removed the attributes, it worked fine for me.
EDIT 2 (Deleted)
EDIT 3
Your issue is with the assignment in the constructor. Not sure how I missed this. You don't need to set all of those variables, because your FieldOffset attribute makes them all stored in the same location. The struct itself is only occupying 4 bytes of memory (in theory, anyway). Whether you're accessing it via the int, the bytes, or the two shorts, they all go to the same place. As a result, your first two sets of assignments (to the int and to the bytes) are redundant, and the last set (to the shorts setting them to 0) clears out what you just did.
Unfortunately the C# compiler doesn't know this, so I'm sure you added the last assignment because it complained about the struct not being fully assigned. Add the dummy assignments first for the shorts and the int, then assign the bytes explicitly from the array.

If you are using C# 3.0, just try the following:
MyStruct ms = new MyStruct
{
MyByte = 1,
MyNestedStruct.NestedStryctByte = 2
}
That should work, and would alleviate the need to have a constructor in your struct. If you can't directly initialize the NestedStructByte, create it first:
MyStruct ms = new MyStruct
{
MyByte = 1,
MyNestedStruct = new MyStruct.NestedStruct
{
NestedStryctByte = 2
}
}

I tested your code - everything is OK.
public struct MyStruct
{
public byte MyByte;
public NestedStruct MyNestedStruct;
public struct NestedStruct
{
public byte NestedStructByte;
}
public MyStruct(byte[] bytes)
{
MyByte = bytes[0];
MyNestedStruct.NestedStructByte = bytes[1];
}
}
class Program
{
static void Main(string[] args)
{
MyStruct ms = new MyStruct(new byte[] { 1, 2 });
//ms.MyByte; // 0, but should be 1
//ms.MyNestedStruct.NestedStructByte; // 0, but should be 2
}
}

reinterpret_cast in C#

I'm looking for a way to reinterpret an array of type byte[] as a different type, say short[]. In C++ this would be achieved by a simple cast but in C# I haven't found a way to achieve this without resorting to duplicating the entire buffer.
Any ideas?

You can achieve this but this is a relatively bad idea. Raw memory access like this is not type-safe and can only be done under a full trust security environment. You should never do this in a properly designed managed application. If your data is masquerading under two different forms, perhaps you actually have two separate data sets?
In any case, here is a quick and simple code snippet to accomplish what you asked:
byte[] bytes = new byte[] { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 };
int byteCount = bytes.Length;
unsafe
{
// By using the fixed keyword, we fix the array in a static memory location.
// Otherwise, the garbage collector might move it while we are still using it!
fixed (byte* bytePointer = bytes)
{
short* shortPointer = (short*)bytePointer;
for (int index = 0; index < byteCount / 2; index++)
{
Console.WriteLine("Short {0}: {1}", index, shortPointer[index]);
}
}
}

There are four good answers to this question. Each has different downsides. Of course, beware of endianness and realize that all of these answers are holes in the type system, just not particularly treacherous holes. In short, don't do this a lot, and only when you really need to.
Sander's answer. Use unsafe code to reinterpret pointers. This is the fastest solution, but it uses unsafe code. Not always an option.
Leonidas' answer. Use StructLayout and FieldOffset(0) to turn a struct into a union. The downsides to this are that some (rare) environments don't support StructLayout (eg Flash builds in Unity3D) and that StructLayout cannot be used with generics.
ljs' answer. Use BitConverter methods. This has the disadvantage that most of the methods allocate memory, which isn't great in low-level code. Also, there isn't a full suite of these methods, so you can't really use it generically.
Buffer.BlockCopy two arrays of different types. The only downside is that you need two buffers, which is perfect when converting arrays, but a pain when casting a single value. Just beware that length is specified in bytes, not elements. Buffer.ByteLength helps. Also, it only works on primitives, like ints, floats and bools, not structs or enums.
But you can do some neat stuff with it.
public static class Cast {
private static class ThreadLocalType<T> {
[ThreadStatic]
private static T[] buffer;
public static T[] Buffer
{
get
{
if (buffer == null) {
buffer = new T[1];
}
return buffer;
}
}
}
public static TTarget Reinterpret<TTarget, TSource>(TSource source)
{
TSource[] sourceBuffer = ThreadLocalType<TSource>.Buffer;
TTarget[] targetBuffer = ThreadLocalType<TTarget>.Buffer;
int sourceSize = Buffer.ByteLength(sourceBuffer);
int destSize = Buffer.ByteLength(targetBuffer);
if (sourceSize != destSize) {
throw new ArgumentException("Cannot convert " + typeof(TSource).FullName + " to " + typeof(TTarget).FullName + ". Data types are of different sizes.");
}
sourceBuffer[0] = source;
Buffer.BlockCopy(sourceBuffer, 0, targetBuffer, 0, sourceSize);
return targetBuffer[0];
}
}
class Program {
static void Main(string[] args)
{
Console.WriteLine("Float: " + Cast.Reinterpret<int, float>(100));
Console.ReadKey();
}
}

c# supports this so long as you are willing to use unsafe code but only on structs.
for example : (The framework provides this for you but you could extend this to int <-> uint conversion
public unsafe long DoubleToLongBits(double d)
{
return *((long*) (void*) &d);
}
Since the arrays are reference types and hold their own metadata about their type you cannot reinterpret them without overwriting the metadata header on the instance as well (an operation likely to fail).
You can howveer take a foo* from a foo[] and cast that to a bar* (via the technique above) and use that to iterate over the array. Doing this will require you pin the original array for the lifetime of the reinterpreted pointer's use.

You could wrap your shorts/bytes into a structure which allows you to access both values:
See also here: C++ union in C#
using System;
using System.Collections.Generic;
using System.Runtime.InteropServices;
namespace TestShortUnion {
[StructLayout(LayoutKind.Explicit)]
public struct shortbyte {
public static implicit operator shortbyte(int input) {
if (input > short.MaxValue)
throw new ArgumentOutOfRangeException("input", "shortbyte only accepts values in the short-range");
return new shortbyte((short)input);
}
public shortbyte(byte input) {
shortval = 0;
byteval = input;
}
public shortbyte(short input) {
byteval = 0;
shortval = input;
}
[FieldOffset(0)]
public byte byteval;
[FieldOffset(0)]
public short shortval;
}
class Program {
static void Main(string[] args) {
shortbyte[] testarray = new shortbyte[] { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 1111 };
foreach (shortbyte singleval in testarray) {
Console.WriteLine("Byte {0}: Short {1}", singleval.byteval, singleval.shortval);
}
System.Console.ReadLine();
}
}
}

You can use System.Memory to do this in a safe way.
public static TTo[] Cast<TFrom, TTo>(this TFrom[] source) where TTo : struct where TFrom : struct =>
MemoryMarshal.Cast<TFrom, TTo>(source).ToArray();
private byte[] CastToBytes(int[] foo) => foo.Cast<int, byte>(foo);

This kind of behaviour would result in C# being rather type-unsafe. You can easily achieve this in a type-safe manner, however (though of course you are copying the array in doing so).
If you want one byte to map to one short then it's simple using ConvertAll, e.g.:-
short[] shorts = Array.ConvertAll(bytes, b => (short)b);
If you want to simply map every 2 bytes to a short then the following should work:-
if (bytes.Length % 2 != 0)
{
throw new ArgumentException("Byte array must have even rank.");
}
short[] shorts = new short[bytes.Length / 2];
for (int i = 0; i < bytes.Length / 2; ++i)
{
shorts[i] = BitConverter.ToInt16(bytes, 2*i);
}
It may be possible to use the marshaller to do some weird bit-twiddling to achieve this, probably using an unsafe { ... } code block, though this would be prone to errors and make your code unverifiable.
I suspect what you're trying to do can be achieved better using a type-safe idiom rather than a type-unsafe C/C++ one!
Update: Updated to take into account comment.

Casting like this is fundamentally unsafe and not permitted in a managed language. That's also why C# doesn't support unions. Yes, the workaround is to use the Marshal class.

Wouldn't it be possible to create a collection class that implements an interface for both bytes and shorts? Maybe implement both IList< byte > and IList< short >? Then you can have your underlying collection contain bytes, but implement IList< short > functions that work on byte pairs.

I used the code from FastArraySerializer to create a type converter to get from SByte[] to Double[]
public unsafe class ConvertArrayType
{
[StructLayout(LayoutKind.Explicit)]
private struct Union
{
[FieldOffset(0)] public sbyte[] sbytes;
[FieldOffset(0)] public double[] doubles;
}
private Union _union;
public double[] doubles {
get { return _union.doubles; }
}
public sbyte[] sbytes
{
get { return _union.sbytes; }
}
[StructLayout(LayoutKind.Sequential, Pack = 1)]
private struct ArrayHeader
{
public UIntPtr type;
public UIntPtr length;
}
private readonly UIntPtr SBYTE_ARRAY_TYPE;
private readonly UIntPtr DOUBLE_ARRAY_TYPE;
public ConvertArrayType(Array ary, Type newType)
{
fixed (void* pBytes = new sbyte[1])
fixed (void* pDoubles = new double[1])
{
SBYTE_ARRAY_TYPE = getHeader(pBytes)->type;
DOUBLE_ARRAY_TYPE = getHeader(pDoubles)->type;
}
Type typAry = ary.GetType();
if (typAry == newType)
throw new Exception("No Type change specified");
if (!(typAry == typeof(SByte[]) || typAry == typeof(double[])))
throw new Exception("Type Not supported");
if (newType == typeof(Double[]))
{
ConvertToArrayDbl((SByte[])ary);
}
else if (newType == typeof(SByte[]))
{
ConvertToArraySByte((Double[])ary);
}
else
{
throw new Exception("Type Not supported");
}
}
private void ConvertToArraySByte(double[] ary)
{
_union = new Union { doubles = ary };
toArySByte(_union.doubles);
}
private void ConvertToArrayDbl(sbyte[] ary)
{
_union = new Union { sbytes = ary };
toAryDouble(_union.sbytes);
}
private ArrayHeader* getHeader(void* pBytes)
{
return (ArrayHeader*)pBytes - 1;
}
private void toAryDouble(sbyte[] ary)
{
fixed (void* pArray = ary)
{
var pHeader = getHeader(pArray);
pHeader->type = DOUBLE_ARRAY_TYPE;
pHeader->length = (UIntPtr)(ary.Length / sizeof(double));
}
}
private void toArySByte(double[] ary)
{
fixed (void* pArray = ary)
{
var pHeader = getHeader(pArray);
pHeader->type = SBYTE_ARRAY_TYPE;
pHeader->length = (UIntPtr)(ary.Length * sizeof(double));
}
}
} // ConvertArrayType{}
Here's the VB usage:
Dim adDataYgch As Double() = Nothing
Try
Dim nGch As GCHandle = GetGch(myTag)
If GCHandle.ToIntPtr(nGch) <> IntPtr.Zero AndAlso nGch.IsAllocated Then
Dim asb As SByte()
asb = CType(nGch.Target, SByte())
Dim cvt As New ConvertArrayType(asb, GetType(Double()))
adDataYgch = cvt.doubles
End If
Catch ex As Exception
Debug.WriteLine(ex.ToString)
End Try