Develop Reference

byte array from specific index as struct in c# without making a copy

Currently I code client-server junk and deal a lot with C++ structs passed over network.
I know about ways provided here Reading a C/C++ data structure in C# from a byte array, but they all about making a copy.
I want to have something like that:
struct/*or class*/ SomeStruct
{
public uint F1;
public uint F2;
public uint F3;
}
Later in my code I want to have something like that:
byte[] Data; //16 bytes that I got from network
SomeStruct PartOfDataAsSomeStruct { get { return /*make SomeStruct instance based on this.Data starting from index 4, without copying it. So when I do PartOfDataAsSomeStruct.F1 = 132465; it also changes bytes 4, 5, 6 and 7 in this.Data.*/; } }
If this is possible, please, tell how?

Like so?
byte[] data = new byte[16];
// 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
Console.WriteLine(BitConverter.ToString(data));
ref SomeStruct typed = ref Unsafe.As<byte, SomeStruct>(ref data[4]);
typed.F1 = 42;
typed.F2 = 3;
typed.F3 = 9;
// 00-00-00-00-2A-00-00-00-03-00-00-00-09-00-00-00
Console.WriteLine(BitConverter.ToString(data));
This coerces the data from the middle of the byte-array using a ref-local that is an "interior managed pointer" to the data. Zero copies.
If you need multiple items (like how a vector would work), you can do the same thing with spans and MemoryMarshal.Cast
Note that it uses CPU-endian rules for the elements - little endian in my case.
For spans:
byte[] data = new byte[256];
// create a span of some of it
var span = new Span<byte>(data, 4, 128);
// now coerce the span
var typed = MemoryMarshal.Cast<byte, SomeStruct>(span);
Console.WriteLine(typed.Length); // 10 of them fit
typed[3].F1 = 3; // etc

Thank you for the correction, Marc Gravell. And thank you for the example.
Here is a way using Class and Bitwise Operators, without pointers, to do the samething:
class SomeClass
{
public byte[] Data;
public SomeClass()
{
Data = new byte[16];
}
public uint F1
{
get
{
uint ret = (uint)(Data[4] << 24 | Data[5] << 16 | Data[6] << 8 | Data[7]);
return ret;
}
set
{
Data[4] = (byte)(value >> 24);
Data[5] = (byte)(value >> 16);
Data[6] = (byte)(value >> 8);
Data[7] = (byte)value;
}
}
}
Testing:
SomeClass sc = new SomeClass();
sc.F1 = 0b_00000001_00000010_00000011_00000100;
Console.WriteLine(sc.Data[3].ToString() + " " + sc.Data[4].ToString() + " " + sc.Data[5].ToString() + " " + sc.Data[6].ToString());
Console.WriteLine(sc.F1.ToString());
//Output:
//1 2 3 4
//16909060

Getting rid of pointers and unmanaged code

I've been trying to get rid of the unmanaged code that I'm currently using on my source code after a friend suggested why I shouldn't be using unmanaged code. However I seem to keep facing a couple of issues here and there. For most case scenarios I used Buffer.BlockCopy() as it seemed to be the most adequate method but there are still a few in which I'm not sure what to use.
Basically this is used to handle packets that are sent between a server and a client. WriteInt16 is a function used to write ushortvalues in the byte array for example. (1. is basically just doing this at offset 20 and 22).
I'll leave a couple examples below:
1.
fixed (byte* p = Data)
{
*((ushort*)(p + 20)) = X;//x is an ushort
*((ushort*)(p + 22)) = Y;//y is an ushort
}
2.
private byte* Ptr
{
get
{
fixed (byte* p = PData)
return p;
}
}
3.
public unsafe void WriteInt16(ushort val)
{
try
{
*((ushort*)(Ptr + Count)) = val;
Count += 2;
}
catch{}
}

Assuming little Endian platform and byte arrays (not tested):
*((ushort*)(p + 20)) = X;
is
Data[20] = (byte)X;
Data[21] = (byte)(X >> 8);
and
*((ushort*)(Ptr + Count)) = val;
is
PData[Count] = (byte)val;
PData[Count + 1] = (byte)(val >> 8);

int to byte[] consistency over network

I have a struct that gets used all over the place and that I store as byteArray on the hd and also send to other platforms.
I used to do this by getting a string version of the struct and using getBytes(utf-8) and getString(utf-8) during serialization. With that I guess I avoided the little and big endian problems?
However that was quite a bit of overhead and I am now using this:
public static explicit operator byte[] (Int3 self)
{
byte[] int3ByteArr = new byte[12];//4*3
int x = self.x;
int3ByteArr[0] = (byte)x;
int3ByteArr[1] = (byte)(x >> 8);
int3ByteArr[2] = (byte)(x >> 0x10);
int3ByteArr[3] = (byte)(x >> 0x18);
int y = self.y;
int3ByteArr[4] = (byte)y;
int3ByteArr[5] = (byte)(y >> 8);
int3ByteArr[6] = (byte)(y >> 0x10);
int3ByteArr[7] = (byte)(y >> 0x18);
int z = self.z;
int3ByteArr[8] = (byte)z;
int3ByteArr[9] = (byte)(z >> 8);
int3ByteArr[10] = (byte)(z >> 0x10);
int3ByteArr[11] = (byte)(z >> 0x18);
return int3ByteArr;
}
public static explicit operator Int3(byte[] self)
{
int x = self[0] + (self[1] << 8) + (self[2] << 0x10) + (self[3] << 0x18);
int y = self[4] + (self[5] << 8) + (self[6] << 0x10) + (self[7] << 0x18);
int z = self[8] + (self[9] << 8) + (self[10] << 0x10) + (self[11] << 0x18);
return new Int3(x, y, z);
}
It works quite well for me, but I am not quite sure how little/big endian works,. do I still have to take care of something here to be safe when some other machine receives an int I sent as a bytearray?

Your current approach will not work for the case when your application running on system which use Big-Endian. In this situation you don't need reordering at all.
You don't need to reverse byte arrays by your self
And you don't need check for endianess of the system by your self
Static method IPAddress.HostToNetworkOrder will convert integer to the integer with big-endian order.
Static method IPAddress.NetworkToHostOrder will convert integer to the integer with order your system using
Those methods will check for Endianness of the system and will do/or not reordering of integers.
For getting bytes from integer and back use BitConverter
public struct ThreeIntegers
{
public int One;
public int Two;
public int Three;
}
public static byte[] ToBytes(this ThreeIntegers value )
{
byte[] bytes = new byte[12];
byte[] bytesOne = IntegerToBytes(value.One);
Buffer.BlockCopy(bytesOne, 0, bytes, 0, 4);
byte[] bytesTwo = IntegerToBytes(value.Two);
Buffer.BlockCopy(bytesTwo , 0, bytes, 4, 4);
byte[] bytesThree = IntegerToBytes(value.Three);
Buffer.BlockCopy(bytesThree , 0, bytes, 8, 4);
return bytes;
}
public static byte[] IntegerToBytes(int value)
{
int reordered = IPAddress.HostToNetworkOrder(value);
return BitConverter.GetBytes(reordered);
}
And converting from bytes to struct
public static ThreeIntegers GetThreeIntegers(byte[] bytes)
{
int rawValueOne = BitConverter.ToInt32(bytes, 0);
int valueOne = IPAddress.NetworkToHostOrder(rawValueOne);
int rawValueTwo = BitConverter.ToInt32(bytes, 4);
int valueTwo = IPAddress.NetworkToHostOrder(rawValueTwo);
int rawValueThree = BitConverter.ToInt32(bytes, 8);
int valueThree = IPAddress.NetworkToHostOrder(rawValueThree);
return new ThreeIntegers(valueOne, valueTwo, valueThree);
}
If you will use BinaryReader and BinaryWriter for saving and sending to another platforms then BitConverter and byte array manipulating can be dropped off.
// BinaryWriter.Write have overload for Int32
public static void SaveThreeIntegers(ThreeIntegers value)
{
using(var stream = CreateYourStream())
using (var writer = new BinaryWriter(stream))
{
int reordredOne = IPAddress.HostToNetworkOrder(value.One);
writer.Write(reorderedOne);
int reordredTwo = IPAddress.HostToNetworkOrder(value.Two);
writer.Write(reordredTwo);
int reordredThree = IPAddress.HostToNetworkOrder(value.Three);
writer.Write(reordredThree);
}
}
For reading value
public static ThreeIntegers LoadThreeIntegers()
{
using(var stream = CreateYourStream())
using (var writer = new BinaryReader(stream))
{
int rawValueOne = reader.ReadInt32();
int valueOne = IPAddress.NetworkToHostOrder(rawValueOne);
int rawValueTwo = reader.ReadInt32();
int valueTwo = IPAddress.NetworkToHostOrder(rawValueTwo);
int rawValueThree = reader.ReadInt32();
int valueThree = IPAddress.NetworkToHostOrder(rawValueThree);
}
}
Of course you can refactor methods above and get more cleaner solution.
Or add as extension methods for BinaryWriter and BinaryReader.

Yes you do. With changes endianness your serialization which preserves bit ordering will run into trouble.
Take the int value 385
In a bigendian system it would be stored as
000000000000000110000001
Interpreting it as littleendian would read it as
100000011000000000000000
And reverse translate to 8486912
If you use the BitConverter class there will be a book property desiring the endianness of the system. The bitconverter can also produce the bit arrays for you.
You will have to decide to use either endianness and reverse the byte arrays according to the serializing or deserializing systems endianness.
The description on MSDN is actually quite detailed. Here they use Array.Reverse for simplicity. I am not certain that your casting to/from byte in order to do the bit manipulation is in fact the fastest way of converting, but that is easily benchmarked.

C# - Cast a byte array to an array of struct and vice-versa (reverse)

I would like to save a Color[] to a file. To do so, I found that saving a byte array to a file using "System.IO.File.WriteAllBytes" should be very efficient.
I would like to cast my Color[] (array of struct) to a byte array into a safe way considering:
Potential problem of little endian / big endian (I think it is impossible to happen but want to be sure)
Having 2 differents pointer to the same memory which point to different type. Does the garbage collection will know what to do - moving objects - deleting a pointer ???
If it is possible, it would be nice to have a generic way to cast array of byte to array of any struct (T struct) and vice-versa.
If not possible, why ?
Thanks,
Eric
I think that those 2 solutions make a copy that I would like to avoid and also they both uses Marshal.PtrToStructure which is specific to structure and not to array of structure:
Reading a C/C++ data structure in C# from a byte array
How to convert a structure to a byte array in C#?

Since .NET Core 2.1, yes we can! Enter MemoryMarshal.
We will treat our Color[] as a ReadOnlySpan<Color>. We reinterpret that as a ReadOnlySpan<byte>. Finally, since WriteAllBytes has no span-based overload, we use a FileStream to write the span to disk.
var byteSpan = MemoryMarshal.AsBytes(colorArray.AsSpan());
fileStream.Write(byteSpan);
As an interesting side note, you can also experiment with the [StructLayout(LayoutKind.Explicit)] as an attribute on your fields. It allows you to specify overlapping fields, effectively allowing the concept of a union.
Here is a blog post on MSDN that illustrates this. It shows the following code:
[StructLayout(LayoutKind.Explicit)]
public struct MyUnion
{
[FieldOffset(0)]
public UInt16 myInt;
[FieldOffset(0)]
public Byte byte1;
[FieldOffset(1)]
public Byte byte2;
}
In this example, the UInt16 field overlaps with the two Byte fields.
This seems to be strongly related to what you are trying to do. It gets you very close, except for the part of writing all the bytes (especially of multiple Color objects) efficiently. :)

Regarding Array Type Conversion
C# as a language intentionally makes the process of flattening objects or arrays into byte arrays difficult because this approach goes against the principals of .NET strong typing. The conventional alternatives include several serialization tools which are generally seen a safer and more robust, or manual serialization coding such as BinaryWriter.
Having two variables of different types point to the same object in memory can only be performed if the types of the variables can be cast, implicitly or explicitly. Casting from an array of one element type to another is no trivial task: it would have to convert the internal members that keep track of things such as array length, etc.
A simple way to write and read Color[] to file
void WriteColorsToFile(string path, Color[] colors)
{
BinaryWriter writer = new BinaryWriter(File.OpenWrite(path));
writer.Write(colors.Length);
foreach(Color color in colors)
{
writer.Write(color.ToArgb());
}
writer.Close();
}
Color[] ReadColorsFromFile(string path)
{
BinaryReader reader = new BinaryReader(File.OpenRead(path));
int length = reader.ReadInt32();
Colors[] result = new Colors[length];
for(int n=0; n<length; n++)
{
result[n] = Color.FromArgb(reader.ReadInt32());
}
reader.Close();
}

You could use pointers if you really want to copy each byte and not have a copy but the same object, similar to this:
var structPtr = (byte*)&yourStruct;
var size = sizeof(YourType);
var memory = new byte[size];
fixed(byte* memoryPtr = memory)
{
for(int i = 0; i < size; i++)
{
*(memoryPtr + i) = *structPtr++;
}
}
File.WriteAllBytes(path, memory);
I just tested this and after adding the fixed block and some minor corrections it looks like it is working correctly.
This is what I used to test it:
public static void Main(string[] args)
{
var a = new s { i = 1, j = 2 };
var sPtr = (byte*)&a;
var size = sizeof(s);
var mem = new byte[size];
fixed (byte* memPtr = mem)
{
for (int i = 0; i < size; i++)
{
*(memPtr + i) = *sPtr++;
}
}
File.WriteAllBytes("A:\\file.txt", mem);
}
struct s
{
internal int i;
internal int j;
}
The result is the following:
(I manually resolved the hex bytes in the second line, only the first line was produced by the program)

public struct MyX
{
public int IntValue;
[MarshalAs(UnmanagedType.ByValArray, SizeConst = 3, ArraySubType = UnmanagedType.U1)]
public byte[] Array;
MyX(int i, int b)
{
IntValue = b;
Array = new byte[3];
}
public MyX ToStruct(byte []ar)
{
byte[] data = ar;//= { 1, 0, 0, 0, 9, 8, 7 }; // IntValue = 1, Array = {9,8,7}
IntPtr ptPoit = Marshal.AllocHGlobal(data.Length);
Marshal.Copy(data, 0, ptPoit, data.Length);
MyX x = (MyX)Marshal.PtrToStructure(ptPoit, typeof(MyX));
Marshal.FreeHGlobal(ptPoit);
return x;
}
public byte[] ToBytes()
{
Byte[] bytes = new Byte[Marshal.SizeOf(typeof(MyX))];
GCHandle pinStructure = GCHandle.Alloc(this, GCHandleType.Pinned);
try
{
Marshal.Copy(pinStructure.AddrOfPinnedObject(), bytes, 0, bytes.Length);
return bytes;
}
finally
{
pinStructure.Free();
}
}
}
void function()
{
byte[] data = { 1, 0, 0, 0, 9, 8, 7 }; // IntValue = 1, Array = {9,8,7}
IntPtr ptPoit = Marshal.AllocHGlobal(data.Length);
Marshal.Copy(data, 0, ptPoit, data.Length);
var x = (MyX)Marshal.PtrToStructure(ptPoit, typeof(MyX));
Marshal.FreeHGlobal(ptPoit);
var MYstruc = x.ToStruct(data);
Console.WriteLine("x.IntValue = {0}", x.IntValue);
Console.WriteLine("x.Array = ({0}, {1}, {2})", x.Array[0], x.Array[1], x.Array[2]);
}

Working code for reference (take care, I did not need the alpha channel in my case):
// ************************************************************************
// If someday Microsoft make Color serializable ...
//public static void SaveColors(Color[] colors, string path)
//{
// BinaryFormatter bf = new BinaryFormatter();
// MemoryStream ms = new MemoryStream();
// bf.Serialize(ms, colors);
// byte[] bytes = ms.ToArray();
// File.WriteAllBytes(path, bytes);
//}
// If someday Microsoft make Color serializable ...
//public static Colors[] LoadColors(string path)
//{
// Byte[] bytes = File.ReadAllBytes(path);
// BinaryFormatter bf = new BinaryFormatter();
// MemoryStream ms2 = new MemoryStream(bytes);
// return (Colors[])bf.Deserialize(ms2);
//}
// ******************************************************************
public static void SaveColorsToFile(Color[] colors, string path)
{
var formatter = new BinaryFormatter();
int count = colors.Length;
using (var stream = File.OpenWrite(path))
{
formatter.Serialize(stream, count);
for (int index = 0; index < count; index++)
{
formatter.Serialize(stream, colors[index].R);
formatter.Serialize(stream, colors[index].G);
formatter.Serialize(stream, colors[index].B);
}
}
}
// ******************************************************************
public static Color[] LoadColorsFromFile(string path)
{
var formatter = new BinaryFormatter();
Color[] colors;
using (var stream = File.OpenRead(path))
{
int count = (int)formatter.Deserialize(stream);
colors = new Color[count];
for (int index = 0; index < count; index++)
{
byte r = (byte)formatter.Deserialize(stream);
byte g = (byte)formatter.Deserialize(stream);
byte b = (byte)formatter.Deserialize(stream);
colors[index] = Color.FromRgb(r, g, b);
}
}
return colors;
}
// ******************************************************************

passing c++ char* to c# via shared-memory

Sorry for probably simple question but I'm newbie in shared memory and trying to learn by example how to do things.
On c++ side I receive such pair: const unsigned char * value, size_t length
On c# side I need to have regular c# string. Using shared memory what is the best way to do that?

It's not that easy to using the string.
If it's me, I'll try these ways:
1.simply get a copy of the string. System.Text.Encoding.Default.GetString may convert from a byte array to a string.
You may try in a unsafe code block (for that you could use pointer type) to do:
(1) create a byte array, size is your "length"
byte[] buf = new byte[length];
(2) copy your data to the array
for(int i = 0; i < length; ++i) buf[i] = value[i];
(3) get the string
string what_you_want = System.Text.Encoding.Default.GetString(buf);
2.write a class, having a property "string what_you_want", and each time you access it, the above process will perform.
before all, you should first using P/Invoke feature to get the value of that pair.
edit: this is an example.
C++ code:
struct Pair {
int length;
unsigned char value[1024];
};
#include <windows.h>
#include <stdio.h>
int main()
{
const char* s = "hahaha";
HANDLE handle = CreateFileMappingW(INVALID_HANDLE_VALUE, NULL, PAGE_READWRITE, 0, sizeof(Pair), L"MySharedMemory");
struct Pair* p = (struct Pair*) MapViewOfFile(handle, FILE_MAP_READ|FILE_MAP_WRITE, 0, 0, sizeof(Pair));
if (p != 0) {
p->length = lstrlenA(s);
lstrcpyA((char*)p->value, s);
puts("plz start c# program");
getchar();
} else
puts("create shared memory error");
if (handle != NULL)
CloseHandle(handle);
return 0;
}
and C# code:
using System;
using System.IO.MemoryMappedFiles;
class Program
{
static void Main(string[] args)
{
MemoryMappedFile mmf = MemoryMappedFile.OpenExisting("MySharedMemory");
MemoryMappedViewStream mmfvs = mmf.CreateViewStream();
byte[] blen = new byte[4];
mmfvs.Read(blen, 0, 4);
int len = blen[0] + blen[1] * 256 + blen[2] * 65536 + blen[3] * 16777216;
byte[] strbuf = new byte[len];
mmfvs.Read(strbuf, 0, len);
string s = System.Text.Encoding.Default.GetString(strbuf);
Console.WriteLine(s);
}
}
just for example.
you may also add error-check part.