Develop Reference

Creating a managed structure for data

Last night I was working on cleaning up some code I found that was a port of an old game I used to play. One of the tricks I used to clean up data was to get rid of the home-brew DataOffsetAttribute that was made for a struct and just make it a plain jane struct and then I can (later) convert it to something a little more useful. Kind of like a person would do with a DataLayer. This worked really good for fixed sized data from a file. Using this method I was about to convert between the two "data types".
public static byte[] ToByteArray<T>(this T dataStructure) where T : struct
{
int size = Marshal.SizeOf(dataStructure);
byte[] arr = new byte[size];
IntPtr ptr = Marshal.AllocHGlobal(size);
Marshal.StructureToPtr(dataStructure, ptr, true);
Marshal.Copy(ptr, arr, 0, size);
Marshal.FreeHGlobal(ptr);
return arr;
}
public static T MarshalAs<T>(this byte[] rawDataStructure) where T : struct
{
var type = typeof(T);
int size = Marshal.SizeOf(type);
IntPtr ptr = Marshal.AllocHGlobal(size);
Marshal.Copy(rawDataStructure, 0, ptr, size);
T structure = (T)Marshal.PtrToStructure(ptr, type);
Marshal.FreeHGlobal(ptr);
return structure;
}
So then I started to wonder if this would work on another project I worked on a long time ago where the data was variable. Here is what I was hoping my data structure would look like
[StructLayout(LayoutKind.Sequential, Pack = 1)]
public struct RfidReaderResponse
{
public byte PreambleA;
public byte PreambleB;
public byte Length;
public byte LengthLRC;
public ushort ReaderId;
public byte CommandId;
public byte ErrorCode;
public byte[] Data;
public byte LRC;
}
I would probably combine the preamble bytes into a ushort and check if it is a specific value... but that is a different discussion. I have 3 known good responses that I used for testing back in the day. I put those into linqpad as well as my hopeful datastructure. So here is what I am currently using for testing
void Main()
{
var readerResponses = new string[]
{
//data is null because of error EC
"AA-BB-05-05-02-39-0C-EC-21",
//data is 44-00
"AA-BB-07-07-02-39-0C-00-44-00-8B",
//data is 44-00-20-04-13-5E-1A-A4-33-80
"AA-BB-0F-0F-02-39-10-00-44-00-20-04-13-5E-1A-A4-33-80-FB",
};
readerResponses
.Select(x=> x.ToByteArray().MarshalAs<RfidReaderResponse>())
.Dump();
}
now if I comment out the last two fields of the struct I get back what I expect for the first part of the response, but I am just lost on how to get back the data portion. I would prefer to have it as I have it with some magical attribute that the Marshaler understands but so far I can't figure it out. I have tried
[StructLayout(LayoutKind.Sequential, Pack = 1)]
public struct RfidReaderResponse
{
public byte PreambleA;
public byte PreambleB;
public byte Length;
public byte LengthLRC;
[MarshalAs(UnmanagedType.LPArray, SizeParamIndex = 2)]
public IntPtr Data;
}
and
[StructLayout(LayoutKind.Sequential, Pack = 1)]
public struct RfidReaderResponse
{
public byte PreambleA;
public byte PreambleB;
public byte Length;
public byte LengthLRC;
public ushort ReaderId;
public byte CommandId;
public byte ErrorCode;
[MarshalAs(UnmanagedType.SafeArray, SafeArraySubType=VarEnum.VT_I1)]
public byte[] Data;
public byte LRC;
}
after a few hours of research. Both of which didn't work stating that my method couldn't make a size for my structure. So I dunno. What am I doing wrong?
EDIT
forgot the method for converting hex strings to byte arrays
public static byte[] ToByteArray(this string hexString)
{
hexString = System.Text.RegularExpressions.Regex.Replace(hexString, "[^0-9A-F.]", "").ToUpper();
if (hexString.Length % 2 == 1)
throw new Exception("The binary key cannot have an odd number of digits");
byte[] arr = new byte[hexString.Length >> 1];
for (int i = 0; i < hexString.Length >> 1; ++i)
{
arr[i] = (byte)((GetHexVal(hexString[i << 1]) << 4) + (GetHexVal(hexString[(i << 1) + 1])));
}
return arr;
}
private static int GetHexVal(char hex)
{
int val = (int)hex;
return val - (val < 58 ? 48 : 55);
}

Reading binary file into struct

I know this has been answered but after reading the other questions I'm still with no solution. I have a file which was written with the following C++ struct:
typedef struct myStruct{
char Name[127];
char s1[2];
char MailBox[149];
char s2[2];
char RouteID[10];
} MY_STRUCT;
My approach was to be able to parse one field at a time in the struct, but my issue is that I cannot get s1 and MailBox to parse correctly. In the file, the s1 field contains "\r\n" (binary 0D0A), and this causes my parsing code to not parse the MailBox field correctly. Here's my parsing code:
[StructLayout(LayoutKind.Explicit, Size = 0x80 + 0x2 + 0x96)]
unsafe struct MY_STRUCT
{
[FieldOffset(0)]
[MarshalAs(UnmanagedType.ByValTStr, SizeConst = 0x80)]
public string Name;
[FieldOffset(0x80)]
public fixed char s1[2];
/* Does not work, "Could not load type 'MY_STRUCT' ... because it contains an object field at offset 130 that is incorrectly aligned or overlapped by a non-object field." */
[FieldOffset(0x80 + 0x2)]
[MarshalAs(UnmanagedType.ByValTStr, SizeConst = 0x96)]
public string MailBox;
}
If I comment out the last field and reduce the struct's size to 0x80+0x2 it will work correctly for the first two variables.
One thing to note is that the Name and Mailbox strings contain the null terminating character, but since s1 doesn't have the null-terminating character it seems to be messing up the parser, but I don't know why because to me it looks like the code is explicitly telling the Marshaler that the s1 field in the struct is only a fixed 2-char buffer, not a null-terminated string.
Here is a pic of my test data (in code I seek past the first row in the BinaryReader, so "Name" begins at 0x0, not 0x10).

Here's one way, it doesn't use unsafe (nor is it particularly elegant/efficient)
using System.Text;
using System.IO;
namespace ReadCppStruct
{
/*
typedef struct myStruct{
char Name[127];
char s1[2];
char MailBox[149];
char s2[2];
char RouteID[10];
} MY_STRUCT;
*/
class MyStruct
{
public string Name { get; set; }
public string MailBox { get; set; }
public string RouteID { get; set; }
}
class Program
{
static string GetString(Encoding encoding, byte[] bytes, int index, int count)
{
string retval = encoding.GetString(bytes, index, count);
int nullIndex = retval.IndexOf('\0');
if (nullIndex != -1)
retval = retval.Substring(0, nullIndex);
return retval;
}
static MyStruct ReadStruct(string path)
{
byte[] bytes = File.ReadAllBytes(path);
var utf8 = new UTF8Encoding();
var retval = new MyStruct();
int index = 0; int cb = 127;
retval.Name = GetString(utf8, bytes, index, cb);
index += cb + 2;
cb = 149;
retval.MailBox = GetString(utf8, bytes, index, cb);
index += cb + 2;
cb = 10;
retval.RouteID = GetString(utf8, bytes, index, cb);
return retval;
} // http://stackoverflow.com/questions/30742019/reading-binary-file-into-struct
static void Main(string[] args)
{
MyStruct ms = ReadStruct("MY_STRUCT.data");
}
}
}

Here's how I got it to work for me:
public static unsafe string BytesToString(byte* bytes, int len)
{
return new string((sbyte*)bytes, 0, len).Trim(new char[] { ' ' }); // trim trailing spaces (but keep newline characters)
}
[StructLayout(LayoutKind.Explicit, Size = 127 + 2 + 149 + 2 + 10)]
unsafe struct USRRECORD_ANSI
{
[FieldOffset(0)]
public fixed byte Name[127];
[FieldOffset(127)]
public fixed byte s1[2];
[FieldOffset(127 + 2)]
public fixed byte MailBox[149];
[FieldOffset(127 + 2 + 149)]
public fixed byte s2[2];
[FieldOffset(127 + 2 + 149 + 2)]
public fixed byte RouteID[10];
}
After the struct has been parsed, I can access the strings by calling the BytesToString method, e.g. string name = BytesToString(record.Name, 127);
I did notice that I don't need the Size attribute in the StructLayout, I'm not sure if it's the best practice to keep it or remove it, ideas?

Your struct sizes are not adding up correctly. The MailBox size is 0x95 as listed in MY_STRUCT, not 0x96 as you're calling it in the C# code.

Marshalling part of a struct

I've been trying to figure out whether it is possible to marshal a part of a struct into an array of bytes without any errors. The reason for this is because the part not going to be marshalled has a variable sized array. This is the the code I'm using.
C#:
public struct Random {
[MarshalAs(UnmanagedType.I4)]
public int a;
[MarshalAs(UnmanagedType.I4)]
public int b;
[MarshalAs(UnmanagedType.SafeArray, SafeArraySubType = VarEnum.VT_RECORD)]
public Random1[] r;
}
public struct Random1 {
[MarshalAs(UnmanagedType.I4)]
public int c;
}
private void Form1_Load(object sender, EventArgs e) {
int size = Marshal.SizeOf(typeof(Random));
int s = 4;
Random r = new Random();
Random r1 = new Random();
r.a = 1;
r.b = 5;
r.r = new Random1[2];
r.r[0].c = 10;
r.r[1].c = 12;
IntPtr p = Marshal.AllocHGlobal(size);
try {
byte[] arr = new byte[size];
Marshal.StructureToPtr(r, p, false);
Marshal.Copy(p, arr, 0, 8);
Marshal.FreeHGlobal(p);
p = Marshal.AllocHGlobal(size);
Marshal.Copy(arr, 0, p, 8);
r1 = (Random) Marshal.PtrToStructure(p, typeof(Random));
Marshal.FreeHGlobal(p);
p = IntPtr.Zero;
Debug.WriteLine(r1.a);
Debug.WriteLine(r1.b);
Debug.WriteLine(r1.r[0].c);
Debug.WriteLine(r1.r[1].c);
} finally {
if (p != IntPtr.Zero) {
Marshal.FreeHGlobal(p);
}
}
}
When I try this code it gives me an ArgumentException in the StructureToPtr. What am I doing wrong or can this be done? If not I've read something about using IntPtr. Could someone tell me how to do this?

I've been thinking long and hard about this one, and after trying several (very hackish) options, I have arrived at the conclusion that what you are doing is not possible.
First of all, it isn't possible to directly marshal an array of variable length, because no meaningful size offset can be computed by the Marshal, and therefore the memory layout can not be determined.
If the struct wouldn't contain any non-blittable, non-primitive types, you could theoretically do something like this:
byte[] buffer = new byte[8];
Marshal.Copy(new IntPtr(&r), buffer, 0, 8);
IntegersOnlyStruct partialStruct;
fixed (byte* b = buffer)
partialStruct = *(IntegersOnlyStruct*) b;
And obtain the two integers contained in the struct that way. However, since your Random1 is a non-primitive, non-blittable type this approach won't get you very far either.
The IntPtr approach to keep a pointer to a memory location where your Random1 struct is kept in memory is an alternative that could possibly work, but I haven't gone in-depth on that, yet.
I think you may want to re-think the way you approach this, as it's not possible in its current form. If anybody can prove otherwise, I'd be happy to see how it can be done.

Convert a struct to bytes, save to file and read back to struct by another application in C#

I tried writing structures to a binary file but I can't read them correctly.
This is my struct. It has a dynamic number of "values". If the number of values is 3, then GetSize() will return 8 + (8*3) = 32
[StructLayout (LayoutKind.Sequential)]
public struct Sample
{
public long timestamp;
public double[] values;
public int GetSize()
{
return sizeof(long) + sizeof(double) * values.Length;
}
}
First, I convert the structure to bytes by:
public static byte[] SampleToBytes(Sample samp)
{
int size = samp.GetSize();
byte[] arr = new byte[size];
IntPtr ptr = Marshal.AllocHGlobal(size);
Marshal.StructureToPtr(samp, ptr, true);
Marshal.Copy(ptr, arr, 0, size);
Marshal.FreeHGlobal(ptr);
return arr;
}
Then, I write the bytes using BinaryWriter, and exit.
When I have to run the program again and read the file I saved. I use BinaryReader. I get every 32 bytes from the file and convert each array of 32 bytes back to struct using:
public static Sample BytesToSample(byte[] arr)
{
Sample samp = new Sample();
IntPtr ptr = Marshal.AllocHGlobal(32);
Marshal.Copy(arr, 0, ptr, 32);
samp = (Sample)Marshal.PtrToStructure(ptr, samp.GetType());
Marshal.FreeHGlobal(ptr);
return samp;
}
However, a SafeArrayTypeMismatchException occurs at PtrToStructure().
Could anyone please tell me what I am doing wrong?
Thanks.

You do realize that double[], being an array type, is a reference type? The struct holds the long followed by a reference to the heap object.
That would be the reason why it doesn't work, I think.
I believe you should simply write the elements of your array to a binary writer.

C# Marshalling vs Serialization

I have a C# app which read/write messages of some predefined protocol from/to USB. So I have at least 2 options here. One is serialize/Deserialize. The other is marshal. At the beginning I was picking the first option and it worked fine. But after more and more message types are introduced I am annoyed by the cumbersome implementation for each message type and feel the marshal may be a better way to go. I haven't use marshal before. Is it the right way to go? I did some test. One problem I have is when I have a structure with an array, how to write to it? Such as the following:-
[StructLayout(LayoutKind.Sequential, Size=TotalBytesInStruct),Serializable]
public struct SomeData
{
/// char[15]
[MarshalAsAttribute(UnmanagedType.ByValTStr, SizeConst = 15)]
public sbyte[] data;
/// int[15]
[MarshalAsAttribute(UnmanagedType.ByValArray, SizeConst = 15)]
public int[] prob;
}
It appears that the data and prob are both reference and I have to new a object to use them, which sounds not quite right to me. I need the whole struct to be a continuous block of memory, which I don't think the new operator will do that for me.
Any suggestion is really appreciated
The following is how I did by serialization.
List<byte> rawData = new List<byte>();
rawData.AddRange(BitConverter.GetBytes(ProtocolVersion));
// 16 bytes for operator ID
byte[] temp = new byte[16];
Array.Copy(Encoding.ASCII.GetBytes(OperatorId), temp, OperatorId.Length > temp.Length ? temp.Length : OperatorId.Length);
rawData.AddRange(temp);
// 16 bytes for operator password
Array.Clear(temp, 0, temp.Length);
Array.Copy(Encoding.ASCII.GetBytes(OperatorPassword), temp, OperatorPassword.Length > temp.Length ? temp.Length : OperatorPassword.Length);
rawData.AddRange(temp);
The following is how I did by marshaling
static byte[] RawSerialize<T>(T obj)
{
if (obj == null)
{
return null;
}
int size = Marshal.SizeOf(obj);
byte[] result = new byte[size];
IntPtr buffer = Marshal.AllocHGlobal(size);
Marshal.StructureToPtr(obj, buffer, false);
Marshal.Copy(buffer, result, 0, size);
return result;
}

It seems that I only need to new an array like the following and it works fine.
[StructLayout(LayoutKind.Sequential, Pack = 1)]
public class SomeData
{
/// char[15]
[MarshalAsAttribute(UnmanagedType.ByValArray, SizeConst = 15)]
public byte[] data = new byte[15];
/// int[15]
[MarshalAsAttribute(UnmanagedType.ByValArray, SizeConst = 15)]
public int[] prob = new int[15];
}