In our application, we have a very large byte-array and we have to convert these bytes into different types. Currently, we use BitConverter.ToXXXX() for this purpose. Our heavy hitters are, ToInt16 and ToUInt64.
For UInt64, our problem is that the data stream has actually 6-bytes of data to represent a large integer. Since there is no native function to convert 6-bytes of data to UInt64, we do:
UInt64 value = BitConverter.ToUInt64() & 0x0000ffffffffffff;
Our use of ToInt16 is simpler, do don't have to do any bit manipulation.
We do so many of these 2 operations that I wanted to ask the SO community whether there's a faster way to do these conversions. Right now, approximately 20% of our entire CPU cycles is consumed by these two functions.
Have you thought about using memory pointers directly. I can't vouch for its performance but it is a common trick in C++\C...
byte[] arr = { 1, 2, 3, 4, 5, 6, 7, 8 ,9,10,11,12,13,14,15,16};
fixed (byte* a2rr = &arr[0])
{
UInt64* uint64ptr = (UInt64*) a2rr;
Console.WriteLine("The value is {0:X2}", (*uint64ptr & 0x0000FFFFFFFFFFFF));
uint64ptr = (UInt64*) ((byte*) uint64ptr+6);
Console.WriteLine("The value is {0:X2}", (*uint64ptr & 0x0000FFFFFFFFFFFF));
}
You'll need to make your assembly "unsafe" in the build settings as well as mark the method in which you'd be doing this unsafe aswell. You are also tied to little endian with this approach.
You can use the System.Buffer class to copy a whole array over to another array of a different type as a fast, 'block copy' operation:
The BlockCopy method accesses the bytes in the src parameter array using offsets into memory, not programming constructs such as indexes or upper and lower array bounds.
The array types must be of 'primitive' types, they must align, and the copy operation is endian-sensitive. In your case of 6-bytes integers, it can't align with any of .NET's 'primitive' types, unless you can obtain the source array with two bytes of padding for each six, which will then align to Int64. But this method will work for arrays of Int16, which may speed up some of your operations.
Why not:
UInt16 valLow = BitConverter.ToUInt16();
UInt64 valHigh = (UInt64)BitConverter.ToUInt32();
UInt64 Value = (valHigh << 16) | valLow;
You can make that a single statement, although the JIT compiler will probably do that for you automatically.
That will prevent you from reading those extra two bytes that you end up throwing away.
If that doesn't reduce CPU, then you'll probably want to write your own converter that reads the bytes directly from the buffer. You can either use array indexing or, if you think it's necessary, unsafe code with pointers.
Note that, as a commenter pointed out, if you use any of these suggestions, then either you're limited to a particular "endian-ness", or you'll have to write your code to detect little/big endian and react accordingly. The code sample I showed above works for little endian (x86).
See my answer for a similar question here.
It's the same unsafe memory manipulation as in Jimmy's answer, but in a more "friendly" way for consumers. It'll allow you to view your byte array as UInt64 array.
For anyone else who stumbles across this if you only need little endian and do not need to auto detect big endian and convert from that. Then I've written an extended version of bitconverter with a number of additions to handle Span as well as converting arrays of type T for example int[] or timestamp[]
Also extended the types supported to include timestamp, decimal and datetime.
https://github.com/tcwicks/ChillX/blob/master/src/ChillX.Serialization/BitConverterExtended.cs
Example usage:
Random rnd = new Random();
RentedBuffer<byte> buffer = RentedBuffer<byte>.Shared.Rent(BitConverterExtended.SizeOfUInt64
+ (20 * BitConverterExtended.SizeOfUInt16)
+ (20 * BitConverterExtended.SizeOfTimeSpan)
+ (10 * BitConverterExtended.SizeOfSingle);
UInt64 exampleLong = long.MaxValue;
int startIndex = 0;
startIndex += BitConverterExtended.GetBytes(exampleLong, buffer.BufferSpan, startIndex);
UInt16[] shortArray = new UInt16[20];
for (int I = 0; I < shortArray.Length; I++) { shortArray[I] = (ushort)rnd.Next(0, UInt16.MaxValue); }
//When using reflection / expression trees CLR cannot distinguish between UInt16 and Int16 or Uint64 and Int64 etc...
//Therefore Uint methods are renamed.
startIndex += BitConverterExtended.GetBytesUShortArray(shortArray, buffer.BufferSpan, startIndex);
TimeSpan[] timespanArray = new TimeSpan[20];
for (int I = 0; I < timespanArray.Length; I++) { timespanArray[I] = TimeSpan.FromSeconds(rnd.Next(0, int.MaxValue)); }
startIndex += BitConverterExtended.GetBytes(timespanArray, buffer.BufferSpan, startIndex);
float[] floatArray = new float[10];
for (int I = 0; I < floatArray.Length; I++) { floatArray[I] = MathF.PI * rnd.Next(short.MinValue, short.MaxValue); }
startIndex += BitConverterExtended.GetBytes(floatArray, buffer.BufferSpan, startIndex);
//Do stuff with buffer and then
buffer.Return(); //always better to return it as soon as possible
//Or in case you forget
buffer = null;
//and let RentedBufferContract do this automatically
it supports reading from and writing to both byte[] or RentedBuffer however using the RentedBuffer class greatly reduces GC collection overheads.
RentedBufferContract class internally handles returning buffers to the pool to prevent memory leaks.
Also includes a serializer which is similar to messagepack.
Note: MessagePack is a faster serializer with more features however this serializer reduces GC collection overheads by reading from and writing to rented byte buffers.
https://github.com/tcwicks/ChillX/blob/master/src/ChillX.Serialization/ChillXSerializer.cs
Related
I have a method to convert Span<byte> to int:
int ConvertToInt32(Span<byte> buffer)
{
if (buffer.Length != 4)
{
// add zero-padding to the buffer
}
return BitConverter.ToInt32(buffer);
// or MemoryMarshal....
}
If the input parameter (buffer) length is 4, so the conversion works, but if it has less than 4 bytes it doesn't.
I'm looking for a non-allocating method to do this.
So how can I add zero-padding to the buffer before conversion, or is there any other solution?
You can allocate on stack with stackalloc which should be cheaper then allocating on the heap and is usual approach in high-performance scenarios AFAIK:
Span<byte> buffer = stackalloc byte[3] {0,0,1};
if (buffer.Length != 4)
{
Span<byte> interimBuffer = stackalloc byte[4];
// ..
}
Note that you can just convert to type with corresponding size and then cast to int:
if (buffer.Length == 2)
{
var int16 = (int)BitConverter.ToInt16(buffer);
}
But due to branching it can have worse performance.
Also do not forget about endianness of computer's architecture as written in the docs.
You could waste four bytes on a static buffer:
// buffer used for converting fewer than 4 bytes
private static byte convertBuffer[] = new byte[4];
int ConvertToInt32(Span<byte> buffer)
{
if (buffer.Length != 4)
{
// copy the bytes from buffer to convertBuffer
// and then . . .
return BitConverter.ToInt32(buffer);
}
return BitConverter.ToInt32(buffer);
// or MemoryMarshal....
}
The question here will be where to copy the bytes. Do you pad left or pad right? As the documentation says:
The order of bytes in the array must reflect the endianness of the computer system's architecture. For more information, see the Remarks section of BitConverter.
In truth, that 4-byte buffer will likely require more than 4 bytes, what with allocation overhead and such. I'd figure somewhere between 12 and 24 bytes, depending on if you're running on the 32-bit or 64-bit runtime. Still, it prevents having to do a memory allocation every time you use it.
The only other way I can think of is to write a loop that'll do the conversion: basically, re-create the logic of BitConverter.ToInt32(), but modified to handle fewer than 4 bytes. That's not too tough, but again you need to make allowances for the endianness of the computer. If you're interested in how BitConverter works, take a look at the source.
is there any way of calculate manually the memory that an array is goin to consume.
i am using for languaje C# in a 64 bit OS
let say i have the next array:
int number[][]= new int[2][2];
number[0][0]=25;
number[0][1]=60;
....
...
so my fist question is, each dimension of the array has the same bit asignation? lets say number[0][0] has a 12 bit asing (i dont now if 12 bits is the right answer) so this will make the first line a 24 bit of memory asing?
how much fisical and virtual memory does each dimension takes?
if i use int, double or string for the array is there any diference of memory to been used?
at the end if i used GC.GetTotalMemory will i recibe the same result of the total of memory been used by array?
You need to use the sizeof function to get how many bytes are allocated to your Type.
int number[][] = new int[2][];
for (int i = 0; i < number.Length; i++)
{
number[i] = new int[2];
}
int size = sizeof(int) * number.Length * number[0].Length;
Which is the fastest way to convert a byte[] to float[] and vice versa (without a loop of course).
I'm using BlockCopy now, but then I need the double memory. I would like some kind of cast.
I need to do this conversion just to send the data through a socket and reconstruct the array in the other end.
Surely msarchet's proposal makes copies too. You are talking about just changing the way .NET thinks about a memory area, if you dont' want to copy.
But, I don't think what you want is possible, as bytes and floats are represented totally different in memory. A byte uses exactly a byte in memory, but a float uses 4 bytes (32 bits).
If you don't have the memory requirements to store your data, just represent the data as the data type you will be using the most in memory, and convert the values you actually use, when you use them.
How do you want to convert a float (which can represent a value between ±1.5 × 10−45 and±3.4 × 10^38) into a byte (which can represent a value between 0 and 255) anyway?
(see more info her about:
byte: http://msdn.microsoft.com/en-us/library/5bdb6693(v=VS.100).aspx
float: http://msdn.microsoft.com/en-us/library/b1e65aza.aspx
More about floating types in .NET here: http://csharpindepth.com/Articles/General/FloatingPoint.aspx
You can use StructLayout to achieve this (from Stack Overflow question C# unsafe value type array to byte array conversions):
[StructLayout(LayoutKind.Explicit)]
struct UnionArray
{
[FieldOffset(0)]
public Byte[] Bytes;
[FieldOffset(0)]
public float[] Floats;
}
static void Main(string[] args)
{
// From bytes to floats - works
byte[] bytes = { 0, 1, 2, 4, 8, 16, 32, 64 };
UnionArray arry = new UnionArray { Bytes = bytes };
for (int i = 0; i < arry.Bytes.Length / 4; i++)
Console.WriteLine(arry.Floats[i]);
}
IEnumerable<float> ToFloats(byte[] bytes)
{
for(int i = 0; i < bytes.Length; i+=4)
yield return BitConverter.ToSingle(bytes, i);
}
Two ways if you have access to LINQ:
var floatarray = ByteArry.AsEnumerable.Cast<float>().ToArray();
or just using Array Functions
var floatarray = Array.ConvertAll(ByteArray, item => (float)item);
I'm using the MD5 algorithm to hash the key for an on-disk hash table (I know it's questionable whether this is the best algorithm to use for this, but I'm going with it for now. The problem is generalizable to any algorithm that produces a byte array). My problem is this:
The size of the hash code determines the number of combinations (buckets) in the hash table. Since MD5 is 128 bit, there are a huge number of combinations (~ 3.4e38) which is way too big for my purpose. So what I want to do is pick off the first n bits of the byte array that MD5 produces, and convert those into a long (or ulong) value. Since MD5 produces a byte array, it would be easy to do if I wanted an integral number of bytes, but this leads to too big a jump in the number of combinations. I'm finding the single bit version to be a lot trickier.
Goal:
n = 10 // I.e. I want 2^10 combinations
long pos = someFcn(byte[] key, n)
where key is the value being hashed, and n is the number of bits of the MD5 result I want to use. Pos, then, will be an integer from 0 to 1023 (in the case of n = 10). If n = 11, the code will be from 0 to 2^11-1 = 2027, etc. Has to be somewhat fast/efficient.
Doesn't seem that hard but it's eluding me. Any help would be much appreciated. Thanks.
First, convert the first four bytes into an integer, with BitConverter.ToInt32. It's getting four bytes no matter what, but this probably won't make it measurably slower, since you're working with 32-bit registers for the rest of the calculations anyway, and complex stuff like "if it's < 16 then do this with the first two bytes" will just make it more complicated
Then, given that integer, take the lowest N bits. If you really want a specific number of bits [a power of two number of buckets] not known at compile time, ~((-1)<<N) is a nice trick to get 2^N-1.
Or you could simply use ToUInt32 instead and modulo a prime number [it might be slightly better to convert to UInt64 instead, then you've got fully half the bits to start with, in this case]
To obtain the first 10 bits, for example:
int result = ((int)key[0] << 2) | (((int)key[1] >> 6) & 0x03)
If you have an array like this,
unsigned char data[2000];
then you can just scrape off the first n bits into an integer like so:
typedef unsigned long long int MyInt;
MyInt scrape(size_t n, unsigned char * data)
{
MyInt result = 0;
size_t b;
for (b = 0; b < n / 8; ++b)
{
result <<= 8;
result += data[b];
}
const size_t remaining_bits = n % 8;
result <<= remaining_bits;
result += (data[b] >> (8 - remaining_bits));
return result;
}
I'm assuming that CHAR_BITS == 8, feel free to generalize the code if you like. Also the size of the array times 8 must be at least n.
In the documentation of hardware that allows us to control it via UDP/IP,
I found the following fragment:
In this communication protocol, DWORD is a 4 bytes data, WORD is a 2 bytes data,
BYTE is a single byte data. The storage format is little endian, namely 4 bytes (32bits) data is stored as: d7-d0, d15-d8, d23-d16, d31-d24; double bytes (16bits) data is stored as: d7-d0 , d15-d8.
I am wondering how this translates to C#?
Do I have to convert stuff before sending it over?
For example, if I want to send over a 32 bit integer, or a 4 character string?
C# itself doesn't define the endianness. Whenever you convert to bytes, however, you're making a choice. The BitConverter class has an IsLittleEndian field to tell you how it will behave, but it doesn't give the choice. The same goes for BinaryReader/BinaryWriter.
My MiscUtil library has an EndianBitConverter class which allows you to define the endianness; there are similar equivalents for BinaryReader/Writer. No online usage guide I'm afraid, but they're trivial :)
(EndianBitConverter also has a piece of functionality which isn't present in the normal BitConverter, which is to do conversions in-place in a byte array.)
You can also use
IPAddress.NetworkToHostOrder(...)
For short, int or long.
Re little-endian, the short answer (to do I need to do anything) is "probably not, but it depends on your hardware". You can check with:
bool le = BitConverter.IsLittleEndian;
Depending on what this says, you might want to reverse portions of your buffers. Alternatively, Jon Skeet has specific-endian converters here (look for EndianBitConverter).
Note that itaniums (for example) are big-endian. Most Intels are little-endian.
Re the specific UDP/IP...?
You need to know about network byte order as well as CPU endian-ness.
Typically for TCP/UDP comms, you always convert data to network byte order using the htons function (and ntohs, and their related functions).
Normally network order is big-endian, but in this case (for some reason!) the comms is little endian, so those functions are not very useful. This is important as you cannot assume the UDP comms they have implemented follow any other standards, it also makes life difficult if you have a big-endian architecture as you just can't wrap everything with htons as you should :-(
However, if you're coming from an intel x86 architecture, then you're already little-endian, so just send the data without conversion.
I'm playing around with packed data in UDP Multicast and I needed something to reorder UInt16 octets since I noticed an error in packet header (Wireshark), so I made this:
private UInt16 swapOctetsUInt16(UInt16 toSwap)
{
Int32 tmp = 0;
tmp = toSwap >> 8;
tmp = tmp | ((toSwap & 0xff) << 8);
return (UInt16) tmp;
}
In case of UInt32,
private UInt32 swapOctetsUInt32(UInt32 toSwap)
{
UInt32 tmp = 0;
tmp = toSwap >> 24;
tmp = tmp | ((toSwap & 0xff0000) >> 8);
tmp = tmp | ((toSwap & 0xff00) << 8);
tmp = tmp | ((toSwap & 0xff) << 24);
return tmp;
}
This is just for testing
private void testSwap() {
UInt16 tmp1 = 0x0a0b;
UInt32 tmp2 = 0x0a0b0c0d;
SoapHexBinary shb1 = new SoapHexBinary(BitConverter.GetBytes(tmp1));
SoapHexBinary shb2 = new SoapHexBinary(BitConverter.GetBytes(swapOctetsUInt16(tmp1)));
Debug.WriteLine("{0}", shb1.ToString());
Debug.WriteLine("{0}", shb2.ToString());
SoapHexBinary shb3 = new SoapHexBinary(BitConverter.GetBytes(tmp2));
SoapHexBinary shb4 = new SoapHexBinary(BitConverter.GetBytes(swapOctetsUInt32(tmp2)));
Debug.WriteLine("{0}", shb3.ToString());
Debug.WriteLine("{0}", shb4.ToString());
}
from which output was this:
0B0A: {0}
0A0B: {0}
0D0C0B0A: {0}
0A0B0C0D: {0}
If you're parsing and performance is not critical, consider this very simple code:
private static byte[] NetworkToHostOrder (byte[] array, int offset, int length)
{
return array.Skip (offset).Take (length).Reverse ().ToArray ();
}
int foo = BitConverter.ToInt64 (NetworkToHostOrder (queue, 14, 8), 0);