Related
There have been variants on this question, but the ones I found were "I have a struct in a C DLL". In this case, I have 100% C# code. I have a struct that contains a variable length array of structs that I am trying to marshal to a tightly packed byte array. I'm using structs and Marshal.StructureToPtr because I need a tightly packed array without all of the metadata that BinaryReader/Writer use to help it serialize and deserialize.
Here is the struct definition:
[StructLayout(LayoutKind.Sequential, Pack = 1)]
public struct CharacterSelect_Struct
{
public uint CharCount;
public uint TotalChars;
[MarshalAs(UnmanagedType.ByValArray, ArraySubType = UnmanagedType.Struct)]
public CharacterSelectEntry_Struct[] Entries;
public static CharacterSelect_Struct Initialize(uint totalChars, uint charCount)
{
return new CharacterSelect_Struct
{
CharCount = charCount,
TotalChars = totalChars,
Entries = 0 != charCount ? new CharacterSelectEntry_Struct[charCount] : null
};
}
}
This works great if Entries contains 1 element. If it contains 2 or more I still only ever get the contents of the first element.
Is there a way to serialize the above and get all of the contents of Entries, or do I have to manually serialize the above, and if so, other than doing, "Serialize each Entry separately and append to a list, then serialize the outer struct"?
I went with a struct and Marshal.StructureToPtr approach because it seemed easier than going with classes and manually writing all of the serialization and deserialization using BitConverter, but I'm wondering if there's a simpler way to give me what I need..
I've found questions such as this one, which have come close to solving my dilemma. However, I've yet to find a clean approach to solving this issue in a generic manner.
I have a project that has a lot of structs that will be used for binary data transmission. This data needs to be Big Endian and, of course, most .Net architecture is Little Endian. This means that when I convert my structs to bytes, the byte order for my values are reversed.
Is there a fairly straight-forward approach to either forcing my structs to contain data in Big Endian format, or is there a way to generically write these structs to byte arrays (and byte arrays to structs) that output Big Endian data?
Here is some sample code for what I've already done.
[StructLayout(LayoutKind.Sequential, Pack = 1)]
public unsafe struct StructType_1
{
short shortVal;
ulong ulongVal;
int intVal;
}
[StructLayout(LayoutKind.Sequential, Pack = 1)]
public unsafe struct StructType_2
{
long longVal_1;
long longVal_2;
long longVal;
int intVal;
}
...
public static class StructHelper
{
//How can I change the following methods so that the struct data
//is converted to and from BigEndian data?
public static byte[] StructToByteArray<T>(T structVal) where T : struct
{
int size = Marshal.SizeOf(structVal);
byte[] arr = new byte[size];
IntPtr ptr = Marshal.AllocHGlobal(size);
Marshal.StructureToPtr(structVal, ptr, true);
Marshal.Copy(ptr, arr, 0, size);
Marshal.FreeHGlobal(ptr);
return arr;
}
public static T StructFromByteArray<T>(byte[] bytes) where T : struct
{
int sz = Marshal.SizeOf(typeof(T));
IntPtr buff = Marshal.AllocHGlobal(sz);
Marshal.Copy(bytes, 0, buff, sz);
T ret = (T)Marshal.PtrToStructure(buff, typeof(T));
Marshal.FreeHGlobal(buff);
return ret;
}
}
If you don't mind reading and writing each field to a stream (which may have performance implications) you could use Jon Skeet's EndianBinaryWriter: https://stackoverflow.com/a/1540269/106159
The code would look something like this:
public unsafe struct StructType_2
{
long longVal_1;
long longVal_2;
long longVal;
int intVal;
}
using (MemoryStream memory = new MemoryStream())
{
using (EndianBinaryWriter writer = new EndianBinaryWriter(EndianBitConverter.Big, stream))
{
writer.Write(longVal_1);
writer.Write(longVal_2);
writer.Write(longVal);
writer.Write(intVal);
}
byte[] buffer = memory.ToArray();
// Use buffer
}
You would use the EndianBinaryReader for data going in the opposite direction.
This does of course have two fairly large drawbacks:
It's fiddly writing code to read and write each field.
It may be too slow to do it this way, depending on performance requirements.
Also have a look at this answers to this similar question: Marshalling a big-endian byte collection into a struct in order to pull out values
Given the example on the BitConverter that shows converting a uint, I would suspect not.
How do I marshal this C++ type?
The ABS_DATA structure is used to associate an arbitrarily long data block with the length information. The declared length of the Data array is 1, but the actual length is given by the Length member.
typedef struct abs_data {
ABS_DWORD Length;
ABS_BYTE Data[ABS_VARLEN];
} ABS_DATA;
I tried the following code, but it's not working. The data variable is always empty and I'm sure it has data in there.
[System.Runtime.InteropServices.StructLayoutAttribute(System.Runtime.InteropServices.LayoutKind.Sequential, CharSet = System.Runtime.InteropServices.CharSet.Ansi)]
public struct abs_data
{
/// ABS_DWORD->unsigned int
public uint Length;
/// ABS_BYTE[1]
[System.Runtime.InteropServices.MarshalAsAttribute(System.Runtime.InteropServices.UnmanagedType.ByValTStr, SizeConst = 1)]
public string Data;
}
Old question, but I recently had to do this myself and all the existing answers are poor, so...
The best solution for marshaling a variable-length array in a struct is to use a custom marshaler. This lets you control the code that the runtime uses to convert between managed and unmanaged data. Unfortunately, custom marshaling is poorly-documented and has a few bizarre limitations. I'll cover those quickly, then go over the solution.
Annoyingly, you can't use custom marshaling on an array element of a struct or class. There's no documented or logical reason for this limitation, and the compiler won't complain, but you'll get an exception at runtime. Also, there's a function that custom marshalers must implement, int GetNativeDataSize(), which is obviously impossible to implement accurately (it doesn't pass you an instance of the object to ask its size, so you can only go off the type, which is of course variable size!) Fortunately, this function doesn't matter. I've never seen it get called, and it the custom marshaler works fine even if it returns a bogus value (one MSDN example has it return -1).
First of all, here's what I think your native prototype might look like (I'm using P/Invoke here, but it works for COM too):
// Unmanaged C/C++ code prototype (guess)
//void DoThing (ABS_DATA *pData);
// Guess at your managed call with the "marshal one-byte ByValArray" version
//[DllImport("libname.dll")] public extern void DoThing (ref abs_data pData);
Here's the naïve version of how you might have used a custom marshaler (which really ought to have worked). I'll get to the marshaler itself in a bit...
[StructLayout(LayoutKind.Sequential)]
public struct abs_data
{
// Don't need the length as a separate filed; managed arrays know it.
[MarshalAs(UnmanagedType.CustomMarshaler, MarshalTypeRef=typeof(ArrayMarshaler<byte>))]
public byte[] Data;
}
// Now you can just pass the struct but it takes arbitrary sizes!
[DllImport("libname.dll")] public extern void DoThing (ref abs_data pData);
Unfortunately, at runtime, you apparently can't marshal arrays inside data structures as anything except SafeArray or ByValArray. SafeArrays are counted, but they look nothing like the (extremely common) format that you're looking for here. So that won't work. ByValArray, of course, requires that the length be known at compile time, so that doesn't work either (as you ran into). Bizarrely, though, you can use custom marshaling on array parameters, This is annoying because you have to put the MarshalAsAttribute on every parameter that uses this type, instead of just putting it on one field and having that apply everywhere you use the type containing that field, but c'est la vie. It looks like this:
[StructLayout(LayoutKind.Sequential)]
public struct abs_data
{
// Don't need the length as a separate filed; managed arrays know it.
// This isn't an array anymore; we pass an array of this instead.
public byte Data;
}
// Now you pass an arbitrary-sized array of the struct
[DllImport("libname.dll")] public extern void DoThing (
// Have to put this huge stupid attribute on every parameter of this type
[MarshalAs(UnmanagedType.CustomMarshaler, MarshalTypeRef=typeof(ArrayMarshaler<abs_data>))]
// Don't need to use "ref" anymore; arrays are ref types and pass as pointer-to
abs_data[] pData);
In that example, I preserved the abs_data type, in case you want to do something special with it (constructors, static functions, properties, inheritance, whatever). If your array elements consisted of a complex type, you would modify the struct to represent that complex type. However, in this case, abs_data is basically just a renamed byte - it's not even "wrapping" the byte; as far as the native code is concerned it's more like a typedef - so you can just pass an array of bytes and skip the struct entirely:
// Actually, you can just pass an arbitrary-length byte array!
[DllImport("libname.dll")] public extern void DoThing (
// Have to put this huge stupid attribute on every parameter of this type
[MarshalAs(UnmanagedType.CustomMarshaler, MarshalTypeRef=typeof(ArrayMarshaler<byte>))]
byte[] pData);
OK, so now you can see how to declare the array element type (if needed), and how to pass the array to an unmanaged function. However, we still need that custom marshaler. You should read "Implementing the ICustomMarshaler Interface" but I'll cover this here, with inline comments. Note that I use some shorthand conventions (like Marshal.SizeOf<T>()) that require .NET 4.5.1 or higher.
// The class that does the marshaling. Making it generic is not required, but
// will make it easier to use the same custom marshaler for multiple array types.
public class ArrayMarshaler<T> : ICustomMarshaler
{
// All custom marshalers require a static factory method with this signature.
public static ICustomMarshaler GetInstance (String cookie)
{
return new ArrayMarshaler<T>();
}
// This is the function that builds the managed type - in this case, the managed
// array - from a pointer. You can just return null here if only sending the
// array as an in-parameter.
public Object MarshalNativeToManaged (IntPtr pNativeData)
{
// First, sanity check...
if (IntPtr.Zero == pNativeData) return null;
// Start by reading the size of the array ("Length" from your ABS_DATA struct)
int length = Marshal.ReadInt32(pNativeData);
// Create the managed array that will be returned
T[] array = new T[length];
// For efficiency, only compute the element size once
int elSiz = Marshal.SizeOf<T>();
// Populate the array
for (int i = 0; i < length; i++)
{
array[i] = Marshal.PtrToStructure<T>(pNativeData + sizeof(int) + (elSiz * i));
}
// Alternate method, for arrays of primitive types only:
// Marshal.Copy(pNativeData + sizeof(int), array, 0, length);
return array;
}
// This is the function that marshals your managed array to unmanaged memory.
// If you only ever marshal the array out, not in, you can return IntPtr.Zero
public IntPtr MarshalManagedToNative (Object ManagedObject)
{
if (null == ManagedObject) return IntPtr.Zero;
T[] array = (T[])ManagedObj;
int elSiz = Marshal.SizeOf<T>();
// Get the total size of unmanaged memory that is needed (length + elements)
int size = sizeof(int) + (elSiz * array.Length);
// Allocate unmanaged space. For COM, use Marshal.AllocCoTaskMem instead.
IntPtr ptr = Marshal.AllocHGlobal(size);
// Write the "Length" field first
Marshal.WriteInt32(ptr, array.Length);
// Write the array data
for (int i = 0; i < array.Length; i++)
{ // Newly-allocated space has no existing object, so the last param is false
Marshal.StructureToPtr<T>(array[i], ptr + sizeof(int) + (elSiz * i), false);
}
// If you're only using arrays of primitive types, you could use this instead:
//Marshal.Copy(array, 0, ptr + sizeof(int), array.Length);
return ptr;
}
// This function is called after completing the call that required marshaling to
// unmanaged memory. You should use it to free any unmanaged memory you allocated.
// If you never consume unmanaged memory or other resources, do nothing here.
public void CleanUpNativeData (IntPtr pNativeData)
{
// Free the unmanaged memory. Use Marshal.FreeCoTaskMem if using COM.
Marshal.FreeHGlobal(pNativeData);
}
// If, after marshaling from unmanaged to managed, you have anything that needs
// to be taken care of when you're done with the object, put it here. Garbage
// collection will free the managed object, so I've left this function empty.
public void CleanUpManagedData (Object ManagedObj)
{ }
// This function is a lie. It looks like it should be impossible to get the right
// value - the whole problem is that the size of each array is variable!
// - but in practice the runtime doesn't rely on this and may not even call it.
// The MSDN example returns -1; I'll try to be a little more realistic.
public int GetNativeDataSize ()
{
return sizeof(int) + Marshal.SizeOf<T>();
}
}
Whew, that was long! Well, there you have it. I hope people see this, because there's a lot of bad answers and misunderstanding out there...
It is not possible to marshal structs containing variable-length arrays (but it is possible to marshal variable-length arrays as function parameters). You will have to read your data manually:
IntPtr nativeData = ... ;
var length = Marshal.ReadUInt32 (nativeData) ;
var bytes = new byte[length] ;
Marshal.Copy (new IntPtr ((long)nativeData + 4), bytes, 0, length) ;
If the data being saved isn't a string, you don't have to store it in a string. I usually do not marshal to a string unless the original data type was a char*. Otherwise a byte[] should do.
Try:
[MarshalAs(UnmanagedType.ByValArray, SizeConst=[whatever your size is]]
byte[] Data;
If you need to convert this to a string later, use:
System.Text.Encoding.UTF8.GetString(your byte array here).
Obviously, you need to vary the encoding to what you need, though UTF-8 usually is sufficient.
I see the problem now, you have to marshal a VARIABLE length array. The MarshalAs does not allow this and the array will have to be sent by reference.
If the array length is variable, your byte[] needs to be an IntPtr, so you would use,
IntPtr Data;
Instead of
[MarshalAs(UnmanagedType.ByValArray, SizeConst=[whatever your size is]]
byte[] Data;
You can then use the Marshal class to access the underlying data.
Something like:
uint length = yourABSObject.Length;
byte[] buffer = new byte[length];
Marshal.Copy(buffer, 0, yourABSObject.Data, length);
You may need to clean up your memory when you are finished to avoid a leak, though I suspect the GC will clean it up when yourABSObject goes out of scope. Anyway, here is the cleanup code:
Marshal.FreeHGlobal(yourABSObject.Data);
You are trying to marshal something that is a byte[ABS_VARLEN] as if it were a string of length 1. You'll need to figure out what the ABS_VARLEN constant is and marshal the array as:
[MarshalAs(UnmanagedType.LPArray, SizeConst = 1024)]
public byte[] Data;
(The 1024 there is a placeholder; fill in whatever the actual value of ASB_VARLEN is.)
In my opinion, it's simpler and more efficient to pin the array and take its address.
Assuming you need to pass abs_data to myNativeFunction(abs_data*):
public struct abs_data
{
public uint Length;
public IntPtr Data;
}
[DllImport("myDll.dll")]
static extern void myNativeFunction(ref abs_data data);
void CallNativeFunc(byte[] data)
{
GCHandle pin = GCHandle.Alloc(data, GCHandleType.Pinned);
abs_data tmp;
tmp.Length = data.Length;
tmp.Data = pin.AddrOfPinnedObject();
myNativeFunction(ref tmp);
pin.Free();
}
How do I marshal this C++ type?
The ABS_DATA structure is used to associate an arbitrarily long data block with the length information. The declared length of the Data array is 1, but the actual length is given by the Length member.
typedef struct abs_data {
ABS_DWORD Length;
ABS_BYTE Data[ABS_VARLEN];
} ABS_DATA;
I tried the following code, but it's not working. The data variable is always empty and I'm sure it has data in there.
[System.Runtime.InteropServices.StructLayoutAttribute(System.Runtime.InteropServices.LayoutKind.Sequential, CharSet = System.Runtime.InteropServices.CharSet.Ansi)]
public struct abs_data
{
/// ABS_DWORD->unsigned int
public uint Length;
/// ABS_BYTE[1]
[System.Runtime.InteropServices.MarshalAsAttribute(System.Runtime.InteropServices.UnmanagedType.ByValTStr, SizeConst = 1)]
public string Data;
}
Old question, but I recently had to do this myself and all the existing answers are poor, so...
The best solution for marshaling a variable-length array in a struct is to use a custom marshaler. This lets you control the code that the runtime uses to convert between managed and unmanaged data. Unfortunately, custom marshaling is poorly-documented and has a few bizarre limitations. I'll cover those quickly, then go over the solution.
Annoyingly, you can't use custom marshaling on an array element of a struct or class. There's no documented or logical reason for this limitation, and the compiler won't complain, but you'll get an exception at runtime. Also, there's a function that custom marshalers must implement, int GetNativeDataSize(), which is obviously impossible to implement accurately (it doesn't pass you an instance of the object to ask its size, so you can only go off the type, which is of course variable size!) Fortunately, this function doesn't matter. I've never seen it get called, and it the custom marshaler works fine even if it returns a bogus value (one MSDN example has it return -1).
First of all, here's what I think your native prototype might look like (I'm using P/Invoke here, but it works for COM too):
// Unmanaged C/C++ code prototype (guess)
//void DoThing (ABS_DATA *pData);
// Guess at your managed call with the "marshal one-byte ByValArray" version
//[DllImport("libname.dll")] public extern void DoThing (ref abs_data pData);
Here's the naïve version of how you might have used a custom marshaler (which really ought to have worked). I'll get to the marshaler itself in a bit...
[StructLayout(LayoutKind.Sequential)]
public struct abs_data
{
// Don't need the length as a separate filed; managed arrays know it.
[MarshalAs(UnmanagedType.CustomMarshaler, MarshalTypeRef=typeof(ArrayMarshaler<byte>))]
public byte[] Data;
}
// Now you can just pass the struct but it takes arbitrary sizes!
[DllImport("libname.dll")] public extern void DoThing (ref abs_data pData);
Unfortunately, at runtime, you apparently can't marshal arrays inside data structures as anything except SafeArray or ByValArray. SafeArrays are counted, but they look nothing like the (extremely common) format that you're looking for here. So that won't work. ByValArray, of course, requires that the length be known at compile time, so that doesn't work either (as you ran into). Bizarrely, though, you can use custom marshaling on array parameters, This is annoying because you have to put the MarshalAsAttribute on every parameter that uses this type, instead of just putting it on one field and having that apply everywhere you use the type containing that field, but c'est la vie. It looks like this:
[StructLayout(LayoutKind.Sequential)]
public struct abs_data
{
// Don't need the length as a separate filed; managed arrays know it.
// This isn't an array anymore; we pass an array of this instead.
public byte Data;
}
// Now you pass an arbitrary-sized array of the struct
[DllImport("libname.dll")] public extern void DoThing (
// Have to put this huge stupid attribute on every parameter of this type
[MarshalAs(UnmanagedType.CustomMarshaler, MarshalTypeRef=typeof(ArrayMarshaler<abs_data>))]
// Don't need to use "ref" anymore; arrays are ref types and pass as pointer-to
abs_data[] pData);
In that example, I preserved the abs_data type, in case you want to do something special with it (constructors, static functions, properties, inheritance, whatever). If your array elements consisted of a complex type, you would modify the struct to represent that complex type. However, in this case, abs_data is basically just a renamed byte - it's not even "wrapping" the byte; as far as the native code is concerned it's more like a typedef - so you can just pass an array of bytes and skip the struct entirely:
// Actually, you can just pass an arbitrary-length byte array!
[DllImport("libname.dll")] public extern void DoThing (
// Have to put this huge stupid attribute on every parameter of this type
[MarshalAs(UnmanagedType.CustomMarshaler, MarshalTypeRef=typeof(ArrayMarshaler<byte>))]
byte[] pData);
OK, so now you can see how to declare the array element type (if needed), and how to pass the array to an unmanaged function. However, we still need that custom marshaler. You should read "Implementing the ICustomMarshaler Interface" but I'll cover this here, with inline comments. Note that I use some shorthand conventions (like Marshal.SizeOf<T>()) that require .NET 4.5.1 or higher.
// The class that does the marshaling. Making it generic is not required, but
// will make it easier to use the same custom marshaler for multiple array types.
public class ArrayMarshaler<T> : ICustomMarshaler
{
// All custom marshalers require a static factory method with this signature.
public static ICustomMarshaler GetInstance (String cookie)
{
return new ArrayMarshaler<T>();
}
// This is the function that builds the managed type - in this case, the managed
// array - from a pointer. You can just return null here if only sending the
// array as an in-parameter.
public Object MarshalNativeToManaged (IntPtr pNativeData)
{
// First, sanity check...
if (IntPtr.Zero == pNativeData) return null;
// Start by reading the size of the array ("Length" from your ABS_DATA struct)
int length = Marshal.ReadInt32(pNativeData);
// Create the managed array that will be returned
T[] array = new T[length];
// For efficiency, only compute the element size once
int elSiz = Marshal.SizeOf<T>();
// Populate the array
for (int i = 0; i < length; i++)
{
array[i] = Marshal.PtrToStructure<T>(pNativeData + sizeof(int) + (elSiz * i));
}
// Alternate method, for arrays of primitive types only:
// Marshal.Copy(pNativeData + sizeof(int), array, 0, length);
return array;
}
// This is the function that marshals your managed array to unmanaged memory.
// If you only ever marshal the array out, not in, you can return IntPtr.Zero
public IntPtr MarshalManagedToNative (Object ManagedObject)
{
if (null == ManagedObject) return IntPtr.Zero;
T[] array = (T[])ManagedObj;
int elSiz = Marshal.SizeOf<T>();
// Get the total size of unmanaged memory that is needed (length + elements)
int size = sizeof(int) + (elSiz * array.Length);
// Allocate unmanaged space. For COM, use Marshal.AllocCoTaskMem instead.
IntPtr ptr = Marshal.AllocHGlobal(size);
// Write the "Length" field first
Marshal.WriteInt32(ptr, array.Length);
// Write the array data
for (int i = 0; i < array.Length; i++)
{ // Newly-allocated space has no existing object, so the last param is false
Marshal.StructureToPtr<T>(array[i], ptr + sizeof(int) + (elSiz * i), false);
}
// If you're only using arrays of primitive types, you could use this instead:
//Marshal.Copy(array, 0, ptr + sizeof(int), array.Length);
return ptr;
}
// This function is called after completing the call that required marshaling to
// unmanaged memory. You should use it to free any unmanaged memory you allocated.
// If you never consume unmanaged memory or other resources, do nothing here.
public void CleanUpNativeData (IntPtr pNativeData)
{
// Free the unmanaged memory. Use Marshal.FreeCoTaskMem if using COM.
Marshal.FreeHGlobal(pNativeData);
}
// If, after marshaling from unmanaged to managed, you have anything that needs
// to be taken care of when you're done with the object, put it here. Garbage
// collection will free the managed object, so I've left this function empty.
public void CleanUpManagedData (Object ManagedObj)
{ }
// This function is a lie. It looks like it should be impossible to get the right
// value - the whole problem is that the size of each array is variable!
// - but in practice the runtime doesn't rely on this and may not even call it.
// The MSDN example returns -1; I'll try to be a little more realistic.
public int GetNativeDataSize ()
{
return sizeof(int) + Marshal.SizeOf<T>();
}
}
Whew, that was long! Well, there you have it. I hope people see this, because there's a lot of bad answers and misunderstanding out there...
It is not possible to marshal structs containing variable-length arrays (but it is possible to marshal variable-length arrays as function parameters). You will have to read your data manually:
IntPtr nativeData = ... ;
var length = Marshal.ReadUInt32 (nativeData) ;
var bytes = new byte[length] ;
Marshal.Copy (new IntPtr ((long)nativeData + 4), bytes, 0, length) ;
If the data being saved isn't a string, you don't have to store it in a string. I usually do not marshal to a string unless the original data type was a char*. Otherwise a byte[] should do.
Try:
[MarshalAs(UnmanagedType.ByValArray, SizeConst=[whatever your size is]]
byte[] Data;
If you need to convert this to a string later, use:
System.Text.Encoding.UTF8.GetString(your byte array here).
Obviously, you need to vary the encoding to what you need, though UTF-8 usually is sufficient.
I see the problem now, you have to marshal a VARIABLE length array. The MarshalAs does not allow this and the array will have to be sent by reference.
If the array length is variable, your byte[] needs to be an IntPtr, so you would use,
IntPtr Data;
Instead of
[MarshalAs(UnmanagedType.ByValArray, SizeConst=[whatever your size is]]
byte[] Data;
You can then use the Marshal class to access the underlying data.
Something like:
uint length = yourABSObject.Length;
byte[] buffer = new byte[length];
Marshal.Copy(buffer, 0, yourABSObject.Data, length);
You may need to clean up your memory when you are finished to avoid a leak, though I suspect the GC will clean it up when yourABSObject goes out of scope. Anyway, here is the cleanup code:
Marshal.FreeHGlobal(yourABSObject.Data);
You are trying to marshal something that is a byte[ABS_VARLEN] as if it were a string of length 1. You'll need to figure out what the ABS_VARLEN constant is and marshal the array as:
[MarshalAs(UnmanagedType.LPArray, SizeConst = 1024)]
public byte[] Data;
(The 1024 there is a placeholder; fill in whatever the actual value of ASB_VARLEN is.)
In my opinion, it's simpler and more efficient to pin the array and take its address.
Assuming you need to pass abs_data to myNativeFunction(abs_data*):
public struct abs_data
{
public uint Length;
public IntPtr Data;
}
[DllImport("myDll.dll")]
static extern void myNativeFunction(ref abs_data data);
void CallNativeFunc(byte[] data)
{
GCHandle pin = GCHandle.Alloc(data, GCHandleType.Pinned);
abs_data tmp;
tmp.Length = data.Length;
tmp.Data = pin.AddrOfPinnedObject();
myNativeFunction(ref tmp);
pin.Free();
}
I'm trying to read binary data using C#. I have all the information about the layout of the data in the files I want to read. I'm able to read the data "chunk by chunk", i.e. getting the first 40 bytes of data converting it to a string, get the next 40 bytes.
Since there are at least three slightly different version of the data, I would like to read the data directly into a struct. It just feels so much more right than by reading it "line by line".
I have tried the following approach but to no avail:
StructType aStruct;
int count = Marshal.SizeOf(typeof(StructType));
byte[] readBuffer = new byte[count];
BinaryReader reader = new BinaryReader(stream);
readBuffer = reader.ReadBytes(count);
GCHandle handle = GCHandle.Alloc(readBuffer, GCHandleType.Pinned);
aStruct = (StructType) Marshal.PtrToStructure(handle.AddrOfPinnedObject(), typeof(StructType));
handle.Free();
The stream is an opened FileStream from which I have began to read from. I get an AccessViolationException when using Marshal.PtrToStructure.
The stream contains more information than I'm trying to read since I'm not interested in data at the end of the file.
The struct is defined like:
[StructLayout(LayoutKind.Explicit)]
struct StructType
{
[FieldOffset(0)]
public string FileDate;
[FieldOffset(8)]
public string FileTime;
[FieldOffset(16)]
public int Id1;
[FieldOffset(20)]
public string Id2;
}
The examples code is changed from original to make this question shorter.
How would I read binary data from a file into a struct?
The problem is the strings in your struct. I found that marshaling types like byte/short/int is not a problem; but when you need to marshal into a complex type such as a string, you need your struct to explicitly mimic an unmanaged type. You can do this with the MarshalAs attrib.
For your example, the following should work:
[StructLayout(LayoutKind.Explicit)]
struct StructType
{
[FieldOffset(0)]
[MarshalAs(UnmanagedType.ByValTStr, SizeConst = 8)]
public string FileDate;
[FieldOffset(8)]
[MarshalAs(UnmanagedType.ByValTStr, SizeConst = 8)]
public string FileTime;
[FieldOffset(16)]
public int Id1;
[FieldOffset(20)]
[MarshalAs(UnmanagedType.ByValTStr, SizeConst = 66)] //Or however long Id2 is.
public string Id2;
}
Here is what I am using.This worked successfully for me for reading Portable Executable Format.It's a generic function, so T is your struct type.
public static T ByteToType<T>(BinaryReader reader)
{
byte[] bytes = reader.ReadBytes(Marshal.SizeOf(typeof(T)));
GCHandle handle = GCHandle.Alloc(bytes, GCHandleType.Pinned);
T theStructure = (T)Marshal.PtrToStructure(handle.AddrOfPinnedObject(), typeof(T));
handle.Free();
return theStructure;
}
As Ronnie said, I'd use BinaryReader and read each field individually. I can't find the link to the article with this info, but it's been observed that using BinaryReader to read each individual field can be faster than Marshal.PtrToStruct, if the struct contains less than 30-40 or so fields. I'll post the link to the article when I find it.
The article's link is at: http://www.codeproject.com/Articles/10750/Fast-Binary-File-Reading-with-C
When marshaling an array of structs, PtrToStruct gains the upper-hand more quickly, because you can think of the field count as fields * array length.
I don't see any problem with your code.
just out of my head, what if you try to do it manually? does it work?
BinaryReader reader = new BinaryReader(stream);
StructType o = new StructType();
o.FileDate = Encoding.ASCII.GetString(reader.ReadBytes(8));
o.FileTime = Encoding.ASCII.GetString(reader.ReadBytes(8));
...
...
...
also try
StructType o = new StructType();
byte[] buffer = new byte[Marshal.SizeOf(typeof(StructType))];
GCHandle handle = GCHandle.Alloc(buffer, GCHandleType.Pinned);
Marshal.StructureToPtr(o, handle.AddrOfPinnedObject(), false);
handle.Free();
then use buffer[] in your BinaryReader instead of reading data from FileStream to see whether you still get AccessViolation exception.
I had no luck using the
BinaryFormatter, I guess I have to
have a complete struct that matches
the content of the file exactly.
That makes sense, BinaryFormatter has its own data format, completely incompatible with yours.
I had no luck using the BinaryFormatter, I guess I have to have a complete struct that matches the content of the file exactly. I realised that in the end I wasn't interested in very much of the file content anyway so I went with the solution of reading part of stream into a bytebuffer and then converting it using
Encoding.ASCII.GetString()
for strings and
BitConverter.ToInt32()
for the integers.
I will need to be able to parse more of the file later on but for this version I got away with just a couple of lines of code.
Try this:
using (FileStream stream = new FileStream(fileName, FileMode.Open))
{
BinaryFormatter formatter = new BinaryFormatter();
StructType aStruct = (StructType)formatter.Deserialize(filestream);
}
Reading straight into structs is evil - many a C program has fallen over because of different byte orderings, different compiler implementations of fields, packing, word size.......
You are best of serialising and deserialising byte by byte. Use the build in stuff if you want or just get used to BinaryReader.
I had structure:
[StructLayout(LayoutKind.Explicit, Size = 21)]
public struct RecordStruct
{
[FieldOffset(0)]
public double Var1;
[FieldOffset(8)]
public byte var2
[FieldOffset(9)]
[MarshalAs(UnmanagedType.ByValTStr, SizeConst = 12)]
public string String1;
}
}
and I received "incorrectly aligned or overlapped by non-object".
Based on that I found:
https://social.msdn.microsoft.com/Forums/vstudio/en-US/2f9ffce5-4c64-4ea7-a994-06b372b28c39/strange-issue-with-layoutkindexplicit?forum=clr
OK. I think I understand what's going on here. It seems like the
problem is related to the fact that the array type (which is an object
type) must be stored at a 4-byte boundary in memory. However, what
you're really trying to do is serialize the 6 bytes separately.
I think the problem is the mix between FieldOffset and serialization
rules. I'm thinking that structlayout.sequential may work for you,
since it doesn't actually modify the in-memory representation of the
structure. I think FieldOffset is actually modifying the in-memory
layout of the type. This causes problems because the .NET framework
requires object references to be aligned on appropriate boundaries (it
seems).
So my struct was defined as explicit with:
[StructLayout(LayoutKind.Explicit, Size = 21)]
and thus my fields had specified
[FieldOffset(<offset_number>)]
but when you change your struct to Sequentional, you can get rid of those offsets and the error will disappear. Something like:
[StructLayout(LayoutKind.Sequential, Size = 21)]
public struct RecordStruct
{
public double Var1;
public byte var2;
[MarshalAs(UnmanagedType.ByValTStr, SizeConst = 12)]
public string String1;
}
}