I recently came across the issue of needing to frequently (& easily) convert & extract bytes from a BigEndian byte array that was being received over TCP. For those unfamiliar with Big/Little Endian, the short version is that each number was sent as MSB then LSB.
When processing and extracting values from the byte array using the default BitConverter on a Little Endian system (read: just about any C#/windows machine), this is a problem.
Here are some handy extension methods that I created in my project that I thought I would share with fellow readers. The trick here is to only reverse the bytes required for the requested data size.
As a bonus, the byte[ ] extension methods makes the code tidy (in my opinion).
Comments & improvements are welcome.
Syntax to use:
var bytes = new byte[] { 0x01, 0x02, 0x03, 0x04 };
var uint16bigend = bytes.GetUInt16BigE(2); // MSB-LSB ... 03-04 = 772
var uint16bitconv = bytes.GetUInt16(2); // LSB-MSB ... 04-03 = 1027
Here is the class with the initial set of extensions. Easy for readers to extend & customize:
public static class BitConverterExtensions
{
public static UInt16 GetUInt16BigE(this byte[] bytes, int startIndex)
{
return BitConverter.ToUInt16(bytes.Skip(startIndex).Take(2).Reverse().ToArray(), 0);
}
public static UInt32 GetUInt32BigE(this byte[] bytes, int startIndex)
{
return BitConverter.ToUInt32(bytes.Skip(startIndex).Take(4).Reverse().ToArray(), 0);
}
public static Int16 GetInt16BigE(this byte[] bytes, int startIndex)
{
return BitConverter.ToInt16(bytes.Skip(startIndex).Take(2).Reverse().ToArray(), 0);
}
public static Int32 GetInt32BigE(this byte[] bytes, int startIndex)
{
return BitConverter.ToInt32(bytes.Skip(startIndex).Take(4).Reverse().ToArray(), 0);
}
public static UInt16 GetUInt16(this byte[] bytes, int startIndex)
{
return BitConverter.ToUInt16(bytes, startIndex);
}
public static UInt32 GetUInt32(this byte[] bytes, int startIndex)
{
return BitConverter.ToUInt32(bytes, startIndex);
}
public static Int16 GetInt16(this byte[] bytes, int startIndex)
{
return BitConverter.ToInt16(bytes, startIndex);
}
public static Int32 GetInt32(this byte[] bytes, int startIndex)
{
return BitConverter.ToInt32(bytes, startIndex);
}
}
You might want to look at using an endian-aware BinaryReader/BinaryWriter implemention. Here are links to some:
Jon Skeet's answer to the question, BinaryWriter Endian issue
Anculus.Core.IO has an endian-aware BinaryReader, BinaryWriter and BitConverter: https://code.google.com/p/libanculus-sharp/source/browse/trunk/src/Anculus.Core/IO/?r=227
The Iso-Parser project (parser for parsing ISO disk images) has an endian-aware BitConverter
Rabbit MQ (Rabbit Message Queue) has an big-endian BinaryReader and BinaryWriter on Github at https://github.com/rabbitmq/rabbitmq-dotnet-client.
Related
I want to use my laptop to communicate with MES(Manufacturing Execution System).
And when I serialized the data (struct type), something happen.
The code below is what I have done:
[StructLayout(LayoutKind.Sequential, Pack = 4)]
struct DataPackage
{
public int a;
public ushort b;
public byte c;
[MarshalAs(UnmanagedType.ByValTStr, SizeConst = 5)] public string d;
}
class Program
{
static void Main(string[] args)
{
DataPackage pack1 = new DataPackage();
pack1.a = 0x33333301;
pack1.b = 200;
pack1.c = 21;
pack1.d = "hello";
byte[] pack1_serialized = getBytes(pack1);
Console.WriteLine(BitConverter.ToString(pack1_serialized));
byte[] getBytes(DataPackage str)
{
int size = Marshal.SizeOf(str);
byte[] arr = new byte[size];
IntPtr ptr = Marshal.AllocHGlobal(size);
Marshal.StructureToPtr(str, ptr, true);
Marshal.Copy(ptr, arr, 0, size);
Marshal.FreeHGlobal(ptr);
return arr;
}
}
}
And here is the outcome:
I want the outcome to be like this:
33-33-33-01-00-C8-15-68-65-6C-6C-6F
So the questions are:
Why is the uint / ushort type data reverse after Marshalling?
Is there any other way that I can send the data in the sequence that I want ?
Why is the last word "o" in string "hello" disappear in the byte array ?
Thanks.
1 - Because your expected outcome is big endian, and your system appears to use little endian, so basically reversed order of bytes compared to what you expect.
2- Easiest way is to "convert" your numbers to big endian before marshalling (that is change them in a way which will produce desired result while converting them using little endian), for example like this:
static int ToBigEndianInt(int x) {
if (!BitConverter.IsLittleEndian)
return x; // already fine
var ar = BitConverter.GetBytes(x);
Array.Reverse(ar);
return BitConverter.ToInt32(ar, 0);
}
static ushort ToBigEndianShort(ushort x) {
if (!BitConverter.IsLittleEndian)
return x; // already fine
var ar = BitConverter.GetBytes(x);
Array.Reverse(ar);
return BitConverter.ToUInt16(ar, 0);
}
And then:
pack1.a = ToBigEndianInt(0x33333301);
pack1.b = ToBigEndianShort(200);
Note that this way of conversion is not very efficient and if you need more perfomance you can do this with some bit manipulations.
3 - Because string is null terminated, and this null terminator counts in SizeConst. Since you have it 5, there will be 4 characters of your string + 1 null terminator. Just increase SizeConst = 6 (that might add additional zeroes at the end because of Pack = 4).
Last night I was working on cleaning up some code I found that was a port of an old game I used to play. One of the tricks I used to clean up data was to get rid of the home-brew DataOffsetAttribute that was made for a struct and just make it a plain jane struct and then I can (later) convert it to something a little more useful. Kind of like a person would do with a DataLayer. This worked really good for fixed sized data from a file. Using this method I was about to convert between the two "data types".
public static byte[] ToByteArray<T>(this T dataStructure) where T : struct
{
int size = Marshal.SizeOf(dataStructure);
byte[] arr = new byte[size];
IntPtr ptr = Marshal.AllocHGlobal(size);
Marshal.StructureToPtr(dataStructure, ptr, true);
Marshal.Copy(ptr, arr, 0, size);
Marshal.FreeHGlobal(ptr);
return arr;
}
public static T MarshalAs<T>(this byte[] rawDataStructure) where T : struct
{
var type = typeof(T);
int size = Marshal.SizeOf(type);
IntPtr ptr = Marshal.AllocHGlobal(size);
Marshal.Copy(rawDataStructure, 0, ptr, size);
T structure = (T)Marshal.PtrToStructure(ptr, type);
Marshal.FreeHGlobal(ptr);
return structure;
}
So then I started to wonder if this would work on another project I worked on a long time ago where the data was variable. Here is what I was hoping my data structure would look like
[StructLayout(LayoutKind.Sequential, Pack = 1)]
public struct RfidReaderResponse
{
public byte PreambleA;
public byte PreambleB;
public byte Length;
public byte LengthLRC;
public ushort ReaderId;
public byte CommandId;
public byte ErrorCode;
public byte[] Data;
public byte LRC;
}
I would probably combine the preamble bytes into a ushort and check if it is a specific value... but that is a different discussion. I have 3 known good responses that I used for testing back in the day. I put those into linqpad as well as my hopeful datastructure. So here is what I am currently using for testing
void Main()
{
var readerResponses = new string[]
{
//data is null because of error EC
"AA-BB-05-05-02-39-0C-EC-21",
//data is 44-00
"AA-BB-07-07-02-39-0C-00-44-00-8B",
//data is 44-00-20-04-13-5E-1A-A4-33-80
"AA-BB-0F-0F-02-39-10-00-44-00-20-04-13-5E-1A-A4-33-80-FB",
};
readerResponses
.Select(x=> x.ToByteArray().MarshalAs<RfidReaderResponse>())
.Dump();
}
now if I comment out the last two fields of the struct I get back what I expect for the first part of the response, but I am just lost on how to get back the data portion. I would prefer to have it as I have it with some magical attribute that the Marshaler understands but so far I can't figure it out. I have tried
[StructLayout(LayoutKind.Sequential, Pack = 1)]
public struct RfidReaderResponse
{
public byte PreambleA;
public byte PreambleB;
public byte Length;
public byte LengthLRC;
[MarshalAs(UnmanagedType.LPArray, SizeParamIndex = 2)]
public IntPtr Data;
}
and
[StructLayout(LayoutKind.Sequential, Pack = 1)]
public struct RfidReaderResponse
{
public byte PreambleA;
public byte PreambleB;
public byte Length;
public byte LengthLRC;
public ushort ReaderId;
public byte CommandId;
public byte ErrorCode;
[MarshalAs(UnmanagedType.SafeArray, SafeArraySubType=VarEnum.VT_I1)]
public byte[] Data;
public byte LRC;
}
after a few hours of research. Both of which didn't work stating that my method couldn't make a size for my structure. So I dunno. What am I doing wrong?
EDIT
forgot the method for converting hex strings to byte arrays
public static byte[] ToByteArray(this string hexString)
{
hexString = System.Text.RegularExpressions.Regex.Replace(hexString, "[^0-9A-F.]", "").ToUpper();
if (hexString.Length % 2 == 1)
throw new Exception("The binary key cannot have an odd number of digits");
byte[] arr = new byte[hexString.Length >> 1];
for (int i = 0; i < hexString.Length >> 1; ++i)
{
arr[i] = (byte)((GetHexVal(hexString[i << 1]) << 4) + (GetHexVal(hexString[(i << 1) + 1])));
}
return arr;
}
private static int GetHexVal(char hex)
{
int val = (int)hex;
return val - (val < 58 ? 48 : 55);
}
I tried writing structures to a binary file but I can't read them correctly.
This is my struct. It has a dynamic number of "values". If the number of values is 3, then GetSize() will return 8 + (8*3) = 32
[StructLayout (LayoutKind.Sequential)]
public struct Sample
{
public long timestamp;
public double[] values;
public int GetSize()
{
return sizeof(long) + sizeof(double) * values.Length;
}
}
First, I convert the structure to bytes by:
public static byte[] SampleToBytes(Sample samp)
{
int size = samp.GetSize();
byte[] arr = new byte[size];
IntPtr ptr = Marshal.AllocHGlobal(size);
Marshal.StructureToPtr(samp, ptr, true);
Marshal.Copy(ptr, arr, 0, size);
Marshal.FreeHGlobal(ptr);
return arr;
}
Then, I write the bytes using BinaryWriter, and exit.
When I have to run the program again and read the file I saved. I use BinaryReader. I get every 32 bytes from the file and convert each array of 32 bytes back to struct using:
public static Sample BytesToSample(byte[] arr)
{
Sample samp = new Sample();
IntPtr ptr = Marshal.AllocHGlobal(32);
Marshal.Copy(arr, 0, ptr, 32);
samp = (Sample)Marshal.PtrToStructure(ptr, samp.GetType());
Marshal.FreeHGlobal(ptr);
return samp;
}
However, a SafeArrayTypeMismatchException occurs at PtrToStructure().
Could anyone please tell me what I am doing wrong?
Thanks.
You do realize that double[], being an array type, is a reference type? The struct holds the long followed by a reference to the heap object.
That would be the reason why it doesn't work, I think.
I believe you should simply write the elements of your array to a binary writer.
I have a C# app which read/write messages of some predefined protocol from/to USB. So I have at least 2 options here. One is serialize/Deserialize. The other is marshal. At the beginning I was picking the first option and it worked fine. But after more and more message types are introduced I am annoyed by the cumbersome implementation for each message type and feel the marshal may be a better way to go. I haven't use marshal before. Is it the right way to go? I did some test. One problem I have is when I have a structure with an array, how to write to it? Such as the following:-
[StructLayout(LayoutKind.Sequential, Size=TotalBytesInStruct),Serializable]
public struct SomeData
{
/// char[15]
[MarshalAsAttribute(UnmanagedType.ByValTStr, SizeConst = 15)]
public sbyte[] data;
/// int[15]
[MarshalAsAttribute(UnmanagedType.ByValArray, SizeConst = 15)]
public int[] prob;
}
It appears that the data and prob are both reference and I have to new a object to use them, which sounds not quite right to me. I need the whole struct to be a continuous block of memory, which I don't think the new operator will do that for me.
Any suggestion is really appreciated
The following is how I did by serialization.
List<byte> rawData = new List<byte>();
rawData.AddRange(BitConverter.GetBytes(ProtocolVersion));
// 16 bytes for operator ID
byte[] temp = new byte[16];
Array.Copy(Encoding.ASCII.GetBytes(OperatorId), temp, OperatorId.Length > temp.Length ? temp.Length : OperatorId.Length);
rawData.AddRange(temp);
// 16 bytes for operator password
Array.Clear(temp, 0, temp.Length);
Array.Copy(Encoding.ASCII.GetBytes(OperatorPassword), temp, OperatorPassword.Length > temp.Length ? temp.Length : OperatorPassword.Length);
rawData.AddRange(temp);
The following is how I did by marshaling
static byte[] RawSerialize<T>(T obj)
{
if (obj == null)
{
return null;
}
int size = Marshal.SizeOf(obj);
byte[] result = new byte[size];
IntPtr buffer = Marshal.AllocHGlobal(size);
Marshal.StructureToPtr(obj, buffer, false);
Marshal.Copy(buffer, result, 0, size);
return result;
}
It seems that I only need to new an array like the following and it works fine.
[StructLayout(LayoutKind.Sequential, Pack = 1)]
public class SomeData
{
/// char[15]
[MarshalAsAttribute(UnmanagedType.ByValArray, SizeConst = 15)]
public byte[] data = new byte[15];
/// int[15]
[MarshalAsAttribute(UnmanagedType.ByValArray, SizeConst = 15)]
public int[] prob = new int[15];
}
I need some help with the following line in perl:
pack "NN", $b, $a;
I can't really understand how to convert that to C#. where a & b are both int
Thanks
In .NET the result would not be a string, but a byte array. A string is composed of 16-bit char values, so that is not convenient for representing 8-bit data.
Use a method like this to get the "network" representation of an integer:
public static byte[] ToNetwork(int value) {
byte[] data = BitConverter.GetBytes(value);
if (BitConverter.IsLittleEndian) {
Array.Reverse(data);
}
return data;
}
Write the byte arrays to a memory stream:
int a = 1;
int b = 42;
byte[] result;
using (MemoryStream m = new MemoryStream()) {
m.Write(ToNetwork(a), 0, 4);
m.Write(ToNetwork(b), 0, 4);
result = m.ToArray();
}
Now you have an array with eight bytes; each integer in big endian format.
Mono provides a nice Pack/Unpack API, DataConverter (inspired after the Perl pack and unpack functions). Source code is available in git repro (BSD License)
{
byte [] data = DataConverter.Pack("^II", 12345678, 87654321);
var result = DataConverter.Unpack("^II", data, 0);
Console.Write ("{0}, {1}\n", result[0], result[1]);
}