Develop Reference

Porting FNV-1a algorithm from C# to Lua, multiplication result don't match

I'm trying to port the Accidental Noise library from C# to Lua. I encounter an issue when trying to port the FNV-1A algorithm. The result of the multiplication with the prime doesn't match when using same input values.
First I'd like to show the C# code of the algorithm:
// The "new" FNV-1A hashing
private const UInt32 FNV_32_PRIME = 0x01000193;
private const UInt32 FNV_32_INIT = 2166136261;
public static UInt32 FNV32Buffer(Int32[] uintBuffer, UInt32 len)
{
//NOTE: Completely untested.
var buffer = new byte[len];
Buffer.BlockCopy(uintBuffer, 0, buffer, 0, buffer.Length);
var hval = FNV_32_INIT;
for (var i = 0; i < len; i++)
{
hval ^= buffer[i];
hval *= FNV_32_PRIME;
}
return hval;
}
This function is called as such (simplified) elsewhere in the codebase:
public static UInt32 HashCoordinates(Int32 x, Int32 y, Int32 seed)
{
Int32[] d = { x, y, seed };
return FNV32Buffer(d, sizeof(Int32) * 3);
}
I noticed the sizeof(Int32) result is always multiplied by the number of elements in the Int32[] array. In this case (on my machine) the result is 12, which causes the buffer size in the FNV32Buffer function to be an array of 12 bytes.
Inside the for loop we see the following:
A bitwise XOR operation is performed on hval
hval is multiplied by a prime number
The result of the multiply operation doesn't match with the result of my Lua implementation.
My Lua implementation is as such:
local FNV_32_PRIME = 0x01000193
local FNV_32_INIT = 0x811C9DC5
local function FNV32Buffer(buffer)
local bytes = {}
for _, v in ipairs(buffer) do
local b = toBits(v, 32)
for i = 1, 32, 8 do
bytes[#bytes + 1] = string.sub(b, i, i + 7)
end
end
local hash = FNV_32_INIT
for i, v in ipairs(bytes) do
hash = bit.bxor(hash, v)
hash = hash * FNV_32_PRIME
end
return hash
end
I don't supply the buffer length in my implementation as Lua's Bitwise operators always work on 32-bit signed integers.
In my implementation I create a bytes array and for each number in the buffer table I extract the bytes. When comparing the C# and Lua byte arrays I get mostly similar results:
byte #
C#
Lua
1
00000000
00000000
2
00000000
00000000
3
00000000
00000000
4
00000000
00000000
5
00000000
00000000
6
00000000
00000000
7
00000000
00000000
8
00000000
00000000
9
00101100
00000000
10
00000001
00000000
11
00000000
00000001
12
00000000
00101100
It seems due to endianness the byte ordering is different, but this I can change. I don't believe this has anything to do with my issue right now.
For the C# and Lua byte arrays, I loop through each byte and perform the FNV-1A algorithm on each byte.
When using the values {0, 0, 300} (x, y, seed) as input for the C# and Lua functions I get the following results after the first iteration of the FNV hashing loop is finished:
C#: 00000101_00001100_01011101_00011111 (84696351)
Lua: 01111110_10111100_11101000_10111000 (2126309560)
As can be seen the result after just the first hashing loop are very different. From debugging I can see the numbers diverge when multiplying with the prime. I believe the cause could be that Lua uses signed numbers by default, whereas the C# implementation works on unsigned integers. Or perhaps the results are different due to differences in endianness?
I did read that Lua uses unsigned integers when working with hex literals. Since FNV_32_PRIME is a hex literal, I guess it should work the same as the C# implementation, yet the end result differs.
How can I make sure the Lua implementation matches the results of the C# implementation?

LuaJIT supports CPU native datatypes.
64-bit values (suffixed with LL) are used to avoid precision loss of multiplication result.
-- LuaJIT 2.1 required
local ffi = require'ffi'
-- The "new" FNV-1A hashing
local function FNV32Buffer(data, size_in_bytes)
data = ffi.cast("uint8_t*", data)
local hval = 0x811C9DC5LL
for j = 0, size_in_bytes - 1 do
hval = bit.bxor(hval, data[j]) * 0x01000193LL
end
return tonumber(bit.band(2^32-1, hval))
end
local function HashCoordinates(x, y, seed)
local d = ffi.new("int32_t[?]", 3, x, y, seed)
return FNV32Buffer(d, ffi.sizeof(d))
end
print(HashCoordinates(0, 0, 300)) --> 3732851086

Arithmetic on 32 bit unsigned numbers does not necessarily produce a 32 bit number.
Not tested, but I think the result of the multiplication with the prime number should be normalized using bit.toBit() as stated in the reference you provide.

BitArray stores bits in reverse order when initialized with an Int32 array

Best way to describe my miss understanding is with the code itself:
var emptyByteArray = new byte[2];
var specificByteArray = new byte[] {150, 105}; //0x96 = 150, 0x69 = 105
var bitArray1 = new BitArray(specificByteArray);
bitArray1.CopyTo(emptyByteArray, 0); //[0]: 150, [1]:105
var hexString = "9669";
var intValueForHex = Convert.ToInt32(hexString, 16); //16 indicates to convert from hex
var bitArray2 = new BitArray(new[] {intValueForHex}) {Length = 16}; //Length=16 truncates the BitArray
bitArray2.CopyTo(emptyByteArray, 0); //[0]:105, [1]:150 (inversed, why??)
I've been reading that the bitarray iterates from the LSB to the MSB, what's the best way for me to initialize the bitarray starting from a hex string then?

I think you are thinking about it wrong. Why are you even using a BitArray? Endianness is a byte-related convention, BitArray is just an array of bits. Since it is least-significant bit first, the correct way to store a 32-bit number in a bit array is with bit 0 at index 0 and bit 31 at index 31. This isn't just just my personal bias towards little-endianness (bit 0 should be in byte 0 not byte 3 for goodness sake), it's because BitArray stores bit 0 of a byte at index 0 in the array. It also stores bit 0 of a 32-bit integer in bit 0 of the array, no matter the endianness of the platform you are on.
For example, instead of your integer 9669, let's look at 1234. No matter what platform you are on, that 16-bit number has the following bit representation, because we write a hex number with the most significant hex digit 1 to the left and the least significant hex digit 4 to the right, bit 0 is on the right (a human convention):
1 2 3 4
0001 0010 0011 0100
No matter how an architecture orders the bytes, bit 0 of a 16-bit number always means the least-significant bit (the right-most here) and bit 15 means the most-significant bit (the left-most here). Due to this, your bit array will always be like this, with bit 0 on the left because that's the way I read an array (with index 0 being bit 0 and index 15 being bit 15):
---4--- ---3--- ---2--- ---1---
0 0 1 0 1 1 0 0 0 1 0 0 1 0 0 0
What you are doing is trying to impose the byte order you want onto an array of bits where it doesn't belong. If you want to reverse the bytes, then you'll get this in the bit array which makes a lot less sense, and means you'll have to reverse the bytes again when you get the integer back out:
---2--- ---1--- ---4--- ---3---
0 1 0 0 1 0 0 0 0 0 1 0 1 1 0 0
I don't think this makes any kind of sense for storing an integer. If you want to store the big-endian representation of a 32-bit number in the BitArray then what you are really storing is a byte array that just happens to be the big-endian representation of a 32-bit number and you should convert to a byte array first make it big-endian if necessary before putting it in the BitArray:
int number = 0x1234;
byte[] bytes = BitConverter.GetBytes(number);
if (BitConverter.IsLittleEndian)
{
bytes = bytes.Reverse().ToArray();
}
BitArray ba = new BitArray(bytes);

Strange behavior with << operator

When dealing with the C# shift operators, I encountered a unexpected behavior of the shift-left operator.
Then I tried this simple function:
for (int i = 5; i >= -10; i--) {
int s = 0x10 << i;
Debug.WriteLine(i.ToString().PadLeft(3) + " " + s.ToString("x8"));
}
With this result:
5 00000200
4 00000100
3 00000080
2 00000040
1 00000020
0 00000010
-1 00000000 -> 00000008 expected
-2 00000000 -> 00000004 expected
-3 00000000 -> 00000002 expected
-4 00000000 -> 00000001 expected
-5 80000000
-6 40000000
-7 20000000
-8 10000000
-9 08000000
-10 04000000
Until today, I expected that the << operator can deal with negative values of the second operand.
MSDN tells nothing about the behavior when using negative values of the second operand. But MSDN says that only the lower 5 bits (0-31) are used by the operator, which should fit for negative values.
I also tried with long values: long s = 0x10L << i;, but with the same result.
So what happens here?
EDIT
As stated in your answers, the negative value representation is not the reason for this.
I got the same wrong result for all cases:
0x10<<-3 = 0x00000000 (wrong!)
0x10<<(int)(0xfffffffd) = 0x00000000 (wrong!)
0x10<<(0x0000001d = 0x00000000 (wrong!)
expected = 0x00000002
EDIT #2
One of these two should be true:
1) The shift operator is a real shift-operator, so the results should be:
1a) 0x10 << -3 = 00000002
1b) 0x10 << -6 = 00000000
2) The shift operator is a rotate-operator, so the results should be:
2a) 0x10 << -3 = 00000002 (same as 1a)
2b) 0x10 << -6 = 40000000
But the shown results does not fit either to 1) nor to 2) !!!

Negative numbers are two complemented, so -1 == 0xFFFFFFFF, and 0xFFFFFFFF & 31 == 31, -2 == 0xFFFFFFFE, and 0xFFFFFFFE & 31 == 30 and so on.
-10 == 0xFFFFFFF6, and 0xFFFFFFF6 & 31 == 22, in fact:
(0x10 << 22) == 04000000
Some code to show:
const int num = 0x10;
int maxShift = 31;
for (int i = 5; i >= -10; i--)
{
int numShifted = num << i;
uint ui = (uint)i;
int uiWithMaxShift = (int)(ui & maxShift);
int numShifted2 = num << uiWithMaxShift;
Console.WriteLine("{0,3}: {1,8:x} {2,2} {3,8:x} {4,8:x} {5}",
i,
ui,
uiWithMaxShift,
numShifted,
numShifted2,
numShifted == numShifted2);
}
With long it's the same, but now instead of & 31 you have & 63. -1 == 63, -2 == 62 and -10 == 54
Some example code:
const long num = 0x10;
int maxShift = 63;
for (int i = 5; i >= -10; i--)
{
long numShifted = num << i;
uint ui = (uint)i;
int uiWithMaxShift = (int)(ui & maxShift);
long numShifted2 = num << uiWithMaxShift;
Console.WriteLine("{0,3}: {1,8:x} {2,2} {3,16:x} {4,16:x} {5}",
i,
ui,
uiWithMaxShift,
numShifted,
numShifted2,
numShifted == numShifted2);
}
Just to be clear:
(int x) << y == (int x) << (int)(((uint)y) & 31)
(long x) << y == (long x) << (int)(((uint)y) & 63)
and not
(int x) << y == (int x) << (Math.Abs(y) & 63)
(long x) << y == (long x) << (Math.Abs(y) & 63)
And what you think "should be" "it would be beautiful if it was" "has to be" ecc is irrelevant. While 1 and 0 are "near" (their binary representation has a "distance" of 1 in number of different bits), 0 and -1 are "far" (their binary representation has a "distance" of 32 or 64 in number of different bits)
You think you should get this:
-1 00000000 -> 00000008 expected
-2 00000000 -> 00000004 expected
-3 00000000 -> 00000002 expected
-4 00000000 -> 00000001 expected
but in truth what you don't see is that you are getting this:
-1 (00000008) 00000000
-2 (00000004) 00000000
-3 (00000002) 00000000
-4 (00000001) 00000000
-5 (00000000) 80000000 <-- To show that "symmetry" and "order" still exist
-6 (00000000) 40000000 <-- To show that "symmetry" and "order" still exist
where the part in (...) is the part that is to the "left" of the int and that doesn't exist.

It has to do with the representation of negative numbers. -1 corresponds to all-ones, so its five least significant bits sum to 31 and left-shifting 0x10 by 31 bits gives all zeroes (the high-order bit that is already set is discarded as per the documentation).
Increasingly larger negative numbers correspond to shifting by 30, 29, etc bits. The bit that is already set in 0x10 is in zero-based position 4, so in order for it to not be discarded the shift has to be at most 31 - 4 = 27 bits, which happens when i == -5.
You can easily see what's going on if you try e.g. Console.WriteLine((-1).ToString("x8")):
ffffffff
Update: when first operand is a long you see similar behavior because now six least significant bits are counted from the second operand: 0x10L << -1 shifts left 63 bits, etc.

The left shift operator doesn't see a negative second operand as a right shift. It will just use the lower five bits of the value and make a left shift using that.
The lower five bits of the value -1 (0xFFFFFFFF) will be 31 (0x0000001F), so the first operand 0x10 is shifted to the left 31 steps leaving just the least significant bit in the most significant bit of the result.
In other words, 0x10 << -1 is the same as 0x10 << 31 which will be 0x800000000, but the result is just 32 bits so it will be truncated to 0x00000000.
When you are using a long value, the six least significant bits are used of the second operand. The value -1 becomes 63, and the bits are still shifted out outside the range of the long.

C# byte arrays - signed and unsigned dilemma

I start with a signed byte array and convert to unsigned.. so is the printed result correct?
byte[] unsigned = new byte[] {10,100,120,180,200,220,240};
sbyte[] signed = Utils.toSignedByteArray(unsigned);
And the print (I just append them with a StringBuilder):
signed: [10,100,120,-76,-56,-36,-16]
unsigned : [10,100,120,180,200,220,240]
where:
public static sbyte[] toSignedByteArray(byte[] unsigned){
sbyte[] signed = new sbyte[unsigned.Length];
Buffer.BlockCopy(unsigned, 0, signed, 0, unsigned.Length);
return signed;
}
If I change to this I get the same result.
sbyte[] signed = (sbyte[])(Array)unsigned;
Shouldn't -128 (signed) become 0, -118 become 10, and so on.. and not 10 (signed) = 10 (unsigned)!?
Because
sbyte -128 to 127
byte 0 to 255
So??

Signed integers are represented in the Two's complement system.
Examples:
Bits Unsigned 2's complement
value value
00000000 0 0
00000001 1 1
00000010 2 2
01111110 126 126
01111111 127 127
10000000 128 −128
10000001 129 −127
10000010 130 −126
11111110 254 −2
11111111 255 −1

Result of addition of two bytes when it cross its limit

let i have got two byte variable:
byte a= 255;
byte b= 121;
byte c= (byte) (a + b);
Console.WriteLine(c.ToString());
output:120
please explain me how this is adding values. i know that its crossing size limit of byte but don't know what exactly operation it performs in such situation because its not looking like its chopping the result.
Thanks
EDIT: sorry its 120 as a answer.

You are overflowing the byte storage of 255 so it starts from 0.
So: a + b is an integer = 376
Your code is equivalent to:
byte c = (byte)376;
That's one of the reasons why adding two bytes returns an integer. Casting it back to a byte should be done at your own risk.
If you want to store the integer 376 into bytes you need an array:
byte[] buffer = BitConverter.GetBytes(376);
As you can see the resulting array contains 4 bytes now which is what is necessary to store a 32 bit integer.

It gets obvious when you look at the binary representation of the values:
var | decimal | binary
----|----------------------
a | 255 | 1111 1111
b | 121 | 0111 1001
| |
a+b | 376 | 1 0111 1000
This gets truncated to 8 bits, the overflow bit is disregarded when casting the result to byte:
c | | 0111 1000 => 120

As others are saying, you are overflowing; the a+b operation results in an int, which you are explicitly casting to a byte. Documentation is here, essentially in an unchecked context, the cast is done by truncating the most significant bits.

I guess you mean byte c= (byte)(a + b);
On my end the result here is 120, and that is what I would expect.
a+b equals 376, and all bits that represent 256 and up gets stripped (since byte actually only hold 1 byte), then 120 is what you are left with inside your byte.