Develop Reference

C# BitConverter.GetBytes() padding is incorrect?

I am working on writing my own DNS server in .net core. I'm at the stage where I am encoding the response payload to send back, and the schema shows that most of the numbers are encoded as 16 bit numbers. C#'s ints are 32 bit numbers. Not a big deal, I'm just dropping off the remaining 16 bits from the front of the number I have no problem with that.
I was doing this by hand until I discovered the System.BitConverter class. I tried using it, however, and the results I came up with were reversed of what it came up with.
For example:
using System;
var myInt = 15;
byte[] data = new byte[2];
data[0] = (byte)(myInt >> 8);
data[1] = (byte)(myInt & 255);
var myIntStr = "";
foreach(var b in data)
{
myIntStr += System.Convert.ToHexString(new byte[]{ b });
myIntStr += " ";
}
Console.WriteLine(myIntStr);
var myShort = System.Convert.ToInt16(myInt);
byte[] data2 = System.BitConverter.GetBytes(myShort);
myIntStr = "";
foreach(var b in data2)
{
myIntStr += System.Convert.ToHexString(new byte[]{ b });
myIntStr += " ";
}
Console.WriteLine(myIntStr);
This code produces the following result:
00 0F
0F 00
It's my understanding that 000F is 15 where as 0F00 is 3840. Am I not understanding bit shifting correctly? I literally just started working with actual bits last night lol.
Thanks for reading this and thanks in advance for your help!

As per the comments on the Question, the answer resides in Endianness.
Network byte order sent from the dig command I am using to test with uses Big Endian order. However, my CPU architecture is Small Endian.
Dotnet behind the scenes in their UDPClient class reverses the bytes if your system is Small Endian when sending bytes, and vice verse when receiving bytes. But because I was creating the bytes by hand using bit shifting in the Big Endian format, they were then reversed to be in Non-Network Byte order while everything else was in Network Byte order.
The solution here is to either have conditional logic to test if your system is IsLittleEndian According to the Microsoft dotnet docs, or let the System.BitConverter class handle it for you.
For instance: in my above example I was trying to convert a 32 bit int into a 16 bit unsigned bit. I ended up replacing the above code with:
public static byte[] IntTo16Bit(int input)
{
ushort input16;
if (!UInt16.TryParse(input.ToString(), out input16))
{
throw new Exception($"Input was {input}");
}
if (BitConverter.IsLittleEndian)
{
return BitConverter.GetBytes(input16).Reverse().ToArray();
}
return BitConverter.GetBytes(input16);
}
and plan on better handling when the i32 cannot be converted into a u16.

How to convert an int to a byte array of size 3?

I need to convert an int to a byte array of size 3. This means dropping the last byte, for example:
var temp = BitConverter.GetBytes(myNum).Take(3).ToArray());
However, is there a better way to do is? Maybe by creating a custom struct?
EDIT
For this requirement I have a predefined max value of 16777215 for this new data type.

Something like this (no Linq, just getting bytes)
int value = 123;
byte[] result = new byte[] {
(byte) (value & 0xFF),
(byte) ((value >> 8) & 0xFF),
(byte) ((value >> 16) & 0xFF),
};

Sounds like you want to create a new struct that represents a 3 byte unsigned integer (based solely on the max value quoted).
Using your original method is very prone to failure, firstly, Take(3) is dependent on whether the system you're running on is big-endian or little-endian, secondly, it doesn't take into account what happens when you get passed a negative int which your new struct can't handle.
You will need to write the conversions yourself, I would take in the int as given, check if it's negative, check if it's bigger than 16777215, if it passes those checks then it's between 0 and 16777215 and you can store it in your new struct, simply execute a Where(b => b != 0) instead of Take(3) to get around the endian-ness problem. Obviously take into account the 0 case where all bytes = 0.

Fast casting in C# using BitConverter, can it be any faster?

In our application, we have a very large byte-array and we have to convert these bytes into different types. Currently, we use BitConverter.ToXXXX() for this purpose. Our heavy hitters are, ToInt16 and ToUInt64.
For UInt64, our problem is that the data stream has actually 6-bytes of data to represent a large integer. Since there is no native function to convert 6-bytes of data to UInt64, we do:
UInt64 value = BitConverter.ToUInt64() & 0x0000ffffffffffff;
Our use of ToInt16 is simpler, do don't have to do any bit manipulation.
We do so many of these 2 operations that I wanted to ask the SO community whether there's a faster way to do these conversions. Right now, approximately 20% of our entire CPU cycles is consumed by these two functions.

Have you thought about using memory pointers directly. I can't vouch for its performance but it is a common trick in C++\C...
byte[] arr = { 1, 2, 3, 4, 5, 6, 7, 8 ,9,10,11,12,13,14,15,16};
fixed (byte* a2rr = &arr[0])
{
UInt64* uint64ptr = (UInt64*) a2rr;
Console.WriteLine("The value is {0:X2}", (*uint64ptr & 0x0000FFFFFFFFFFFF));
uint64ptr = (UInt64*) ((byte*) uint64ptr+6);
Console.WriteLine("The value is {0:X2}", (*uint64ptr & 0x0000FFFFFFFFFFFF));
}
You'll need to make your assembly "unsafe" in the build settings as well as mark the method in which you'd be doing this unsafe aswell. You are also tied to little endian with this approach.

You can use the System.Buffer class to copy a whole array over to another array of a different type as a fast, 'block copy' operation:
The BlockCopy method accesses the bytes in the src parameter array using offsets into memory, not programming constructs such as indexes or upper and lower array bounds.
The array types must be of 'primitive' types, they must align, and the copy operation is endian-sensitive. In your case of 6-bytes integers, it can't align with any of .NET's 'primitive' types, unless you can obtain the source array with two bytes of padding for each six, which will then align to Int64. But this method will work for arrays of Int16, which may speed up some of your operations.

Why not:
UInt16 valLow = BitConverter.ToUInt16();
UInt64 valHigh = (UInt64)BitConverter.ToUInt32();
UInt64 Value = (valHigh << 16) | valLow;
You can make that a single statement, although the JIT compiler will probably do that for you automatically.
That will prevent you from reading those extra two bytes that you end up throwing away.
If that doesn't reduce CPU, then you'll probably want to write your own converter that reads the bytes directly from the buffer. You can either use array indexing or, if you think it's necessary, unsafe code with pointers.
Note that, as a commenter pointed out, if you use any of these suggestions, then either you're limited to a particular "endian-ness", or you'll have to write your code to detect little/big endian and react accordingly. The code sample I showed above works for little endian (x86).

See my answer for a similar question here.
It's the same unsafe memory manipulation as in Jimmy's answer, but in a more "friendly" way for consumers. It'll allow you to view your byte array as UInt64 array.

For anyone else who stumbles across this if you only need little endian and do not need to auto detect big endian and convert from that. Then I've written an extended version of bitconverter with a number of additions to handle Span as well as converting arrays of type T for example int[] or timestamp[]
Also extended the types supported to include timestamp, decimal and datetime.
https://github.com/tcwicks/ChillX/blob/master/src/ChillX.Serialization/BitConverterExtended.cs
Example usage:
Random rnd = new Random();
RentedBuffer<byte> buffer = RentedBuffer<byte>.Shared.Rent(BitConverterExtended.SizeOfUInt64
+ (20 * BitConverterExtended.SizeOfUInt16)
+ (20 * BitConverterExtended.SizeOfTimeSpan)
+ (10 * BitConverterExtended.SizeOfSingle);
UInt64 exampleLong = long.MaxValue;
int startIndex = 0;
startIndex += BitConverterExtended.GetBytes(exampleLong, buffer.BufferSpan, startIndex);
UInt16[] shortArray = new UInt16[20];
for (int I = 0; I < shortArray.Length; I++) { shortArray[I] = (ushort)rnd.Next(0, UInt16.MaxValue); }
//When using reflection / expression trees CLR cannot distinguish between UInt16 and Int16 or Uint64 and Int64 etc...
//Therefore Uint methods are renamed.
startIndex += BitConverterExtended.GetBytesUShortArray(shortArray, buffer.BufferSpan, startIndex);
TimeSpan[] timespanArray = new TimeSpan[20];
for (int I = 0; I < timespanArray.Length; I++) { timespanArray[I] = TimeSpan.FromSeconds(rnd.Next(0, int.MaxValue)); }
startIndex += BitConverterExtended.GetBytes(timespanArray, buffer.BufferSpan, startIndex);
float[] floatArray = new float[10];
for (int I = 0; I < floatArray.Length; I++) { floatArray[I] = MathF.PI * rnd.Next(short.MinValue, short.MaxValue); }
startIndex += BitConverterExtended.GetBytes(floatArray, buffer.BufferSpan, startIndex);
//Do stuff with buffer and then
buffer.Return(); //always better to return it as soon as possible
//Or in case you forget
buffer = null;
//and let RentedBufferContract do this automatically
it supports reading from and writing to both byte[] or RentedBuffer however using the RentedBuffer class greatly reduces GC collection overheads.
RentedBufferContract class internally handles returning buffers to the pool to prevent memory leaks.
Also includes a serializer which is similar to messagepack.
Note: MessagePack is a faster serializer with more features however this serializer reduces GC collection overheads by reading from and writing to rented byte buffers.
https://github.com/tcwicks/ChillX/blob/master/src/ChillX.Serialization/ChillXSerializer.cs

I need to write a big-endian single-precision floating point number to file in C#

I have a file format I'm trying to write to using C#. The format encodes integer-based RGB color values as a floating point. It also stores the values as big-endian. I found an example of what I'm trying to do, written in php, here: http://www.colourlovers.com/ase.phps
I can convert the endian-ness easily. I know this code is very verbose, but I'm just using it to watch the bits swap during troubleshooting.
private uint SwapEndian(uint host) {
uint ReturnValue;
uint FirstHalf;
uint LastHalf;
FirstHalf = (host & 0xFFFF0000);
FirstHalf = (FirstHalf >> 16);
LastHalf = (host & 0x0000FFFF);
LastHalf = (LastHalf << 16);
ReturnValue = (FirstHalf | LastHalf);
return ReturnValue;
}
C# won't let me perform bit-shifts on floats. Converting to an int or uint to call my SwapEndian method above loses the encoding information the file format requires.
So, how would you take a floating point number and change its endian-ness without losing the exponent data?

You could just use:
var bytes = BitConverter.GetBytes(floatVal);
And reverse the array (assuming the CPU is little-endian, which you an check), or simply just access the array in the order you need.
There is also a way to do this with unsafe code, treating the float* as a byte* so you can get the bytes in the order you want.

I'm slightly confused as to why your shifting by 16 bits rather than 8. Naw do I understand why you need RGB as a float (they are generaly 1 byte each). But anywho.
you can use a 'fake' union to treat a float as an uint
[StructLayout(LayoutKind.Explicit)]
public struct FloatIntUnion
{
[FieldOffset(0)]
public float f;
[FieldOffset(0)]
public uint i;
}
this allows you to assign the float and then provide do bitwise operations on the uint part of the 'union' then use the float again as the final value.
However I would probably just use:
var bytes = BitConverter.GetBytes (RGB);
if (BitConverter.IsLittleEndian)
Array.Reverse (bytes);
return bytes;
until performance started to become an issue (because of THIS method (read profile first)).

Convert 2 bytes to a number

I have a control that has a byte array in it.
Every now and then there are two bytes that tell me some info about number of future items in the array.
So as an example I could have:
...
...
Item [4] = 7
Item [5] = 0
...
...
The value of this is clearly 7.
But what about this?
...
...
Item [4] = 0
Item [5] = 7
...
...
Any idea on what that equates to (as an normal int)?
I went to binary and thought it may be 11100000000 which equals 1792. But I don't know if that is how it really works (ie does it use the whole 8 items for the byte).
Is there any way to know this with out testing?
Note: I am using C# 3.0 and visual studio 2008

BitConverter can easily convert the two bytes in a two-byte integer value:
// assumes byte[] Item = someObject.GetBytes():
short num = BitConverter.ToInt16(Item, 4); // makes a short
// out of Item[4] and Item[5]

A two-byte number has a low and a high byte. The high byte is worth 256 times as much as the low byte:
value = 256 * high + low;
So, for high=0 and low=7, the value is 7. But for high=7 and low=0, the value becomes 1792.
This of course assumes that the number is a simple 16-bit integer. If it's anything fancier, the above won't be enough. Then you need more knowledge about how the number is encoded, in order to decode it.
The order in which the high and low bytes appear is determined by the endianness of the byte stream. In big-endian, you will see high before low (at a lower address), in little-endian it's the other way around.

You say "this value is clearly 7", but it depends entirely on the encoding. If we assume full-width bytes, then in little-endian, yes; 7, 0 is 7. But in big endian it isn't.
For little-endian, what you want is
int i = byte[i] | (byte[i+1] << 8);
and for big-endian:
int i = (byte[i] << 8) | byte[i+1];
But other encoding schemes are available; for example, some schemes use 7-bit arithmetic, with the 8th bit as a continuation bit. Some schemes (UTF-8) put all the continuation bits in the first byte (so the first has only limited room for data bits), and 8 bits for the rest in the sequence.

If you simply want to put those two bytes next to each other in binary format, and see what that big number is in decimal, then you need to use this code:
if (BitConverter.IsLittleEndian)
{
byte[] tempByteArray = new byte[2] { Item[5], Item[4] };
ushort num = BitConverter.ToUInt16(tempByteArray, 0);
}
else
{
ushort num = BitConverter.ToUInt16(Item, 4);
}
If you use short num = BitConverter.ToInt16(Item, 4); as seen in the accepted answer, you are assuming that the first bit of those two bytes is the sign bit (1 = negative and 0 = positive). That answer also assumes you are using a big endian system. See this for more info on the sign bit.

If those bytes are the "parts" of an integer it works like that. But beware, that the order of bytes is platform specific and that it also depends on the length of the integer (16 bit=2 bytes, 32 bit=4bytes, ...)

In case that item[5] is the MSB
ushort result = BitConverter.ToUInt16(new byte[2] { Item[5], Item[4] }, 0);
int result = 256 * Item[5] + Item[4];