Develop Reference

Comparing bits efficiently ( overlap set of x )

I want to compare a stream of bits of arbitrary length to a mask in c# and return a ratio of how many bits were the same.
The mask to check against is anywhere between 2 bits long to 8k (with 90% of the masks being 5 bits long), the input can be anywhere between 2 bits up to ~ 500k, with an average input string of 12k (but yeah, most of the time it will be comparing 5 bits with the first 5 bits of that 12k)
Now my naive implementation would be something like this:
bool[] mask = new[] { true, true, false, true };
float dendrite(bool[] input) {
int correct = 0;
for ( int i = 0; i<mask.length; i++ ) {
if ( input[i] == mask[i] )
correct++;
}
return (float)correct/(float)mask.length;
}
but I expect this is better handled (more efficient) with some kind of binary operator magic?
Anyone got any pointers?
EDIT: the datatype is not fixed at this point in my design, so if ints or bytearrays work better, I'd also be a happy camper, trying to optimize for efficiency here, the faster the computation, the better.
eg if you can make it work like this:
int[] mask = new[] { 1, 1, 0, 1 };
float dendrite(int[] input) {
int correct = 0;
for ( int i = 0; i<mask.length; i++ ) {
if ( input[i] == mask[i] )
correct++;
}
return (float)correct/(float)mask.length;
}
or this:
int mask = 13; //1101
float dendrite(int input) {
return // your magic here;
} // would return 0.75 for an input
// of 101 given ( 1100101 in binary,
// matches 3 bits of the 4 bit mask == .75
ANSWER:
I ran each proposed answer against each other and Fredou's and Marten's solution ran neck to neck but Fredou submitted the fastest leanest implementation in the end. Of course since the average result varies quite wildly between implementations I might have to revisit this post later on. :) but that's probably just me messing up in my test script. ( i hope, too late now, going to bed =)
sparse1.Cyclone
1317ms 3467107ticks 10000iterations
result: 0,7851563
sparse1.Marten
288ms 759362ticks 10000iterations
result: 0,05066964
sparse1.Fredou
216ms 568747ticks 10000iterations
result: 0,8925781
sparse1.Marten
296ms 778862ticks 10000iterations
result: 0,05066964
sparse1.Fredou
216ms 568601ticks 10000iterations
result: 0,8925781
sparse1.Marten
300ms 789901ticks 10000iterations
result: 0,05066964
sparse1.Cyclone
1314ms 3457988ticks 10000iterations
result: 0,7851563
sparse1.Fredou
207ms 546606ticks 10000iterations
result: 0,8925781
sparse1.Marten
298ms 786352ticks 10000iterations
result: 0,05066964
sparse1.Cyclone
1301ms 3422611ticks 10000iterations
result: 0,7851563
sparse1.Marten
292ms 769850ticks 10000iterations
result: 0,05066964
sparse1.Cyclone
1305ms 3433320ticks 10000iterations
result: 0,7851563
sparse1.Fredou
209ms 551178ticks 10000iterations
result: 0,8925781
( testscript copied here, if i destroyed yours modifying it lemme know. https://dotnetfiddle.net/h9nFSa )

how about this one - dotnetfiddle example
using System;
namespace ConsoleApplication1
{
public class Program
{
public static void Main(string[] args)
{
int a = Convert.ToInt32("0001101", 2);
int b = Convert.ToInt32("1100101", 2);
Console.WriteLine(dendrite(a, 4, b));
}
private static float dendrite(int mask, int len, int input)
{
return 1 - getBitCount(mask ^ (input & (int.MaxValue >> 32 - len))) / (float)len;
}
private static int getBitCount(int bits)
{
bits = bits - ((bits >> 1) & 0x55555555);
bits = (bits & 0x33333333) + ((bits >> 2) & 0x33333333);
return ((bits + (bits >> 4) & 0xf0f0f0f) * 0x1010101) >> 24;
}
}
}
64 bits one here - dotnetfiddler
using System;
namespace ConsoleApplication1
{
public class Program
{
public static void Main(string[] args)
{
// 1
ulong a = Convert.ToUInt64("0000000000000000000000000000000000000000000000000000000000001101", 2);
ulong b = Convert.ToUInt64("1110010101100101011001010110110101100101011001010110010101100101", 2);
Console.WriteLine(dendrite(a, 4, b));
}
private static float dendrite(ulong mask, int len, ulong input)
{
return 1 - getBitCount(mask ^ (input & (ulong.MaxValue >> (64 - len)))) / (float)len;
}
private static ulong getBitCount(ulong bits)
{
bits = bits - ((bits >> 1) & 0x5555555555555555UL);
bits = (bits & 0x3333333333333333UL) + ((bits >> 2) & 0x3333333333333333UL);
return unchecked(((bits + (bits >> 4)) & 0xF0F0F0F0F0F0F0FUL) * 0x101010101010101UL) >> 56;
}
}
}

I came up with this code:
static float dendrite(ulong input, ulong mask)
{
// get bits that are same (0 or 1) in input and mask
ulong samebits = mask & ~(input ^ mask);
// count number of same bits
int correct = cardinality(samebits);
// count number of bits in mask
int inmask = cardinality(mask);
// compute fraction (0.0 to 1.0)
return inmask == 0 ? 0f : correct / (float)inmask;
}
// this is a little hack to count the number of bits set to one in a 64-bit word
static int cardinality(ulong word)
{
const ulong mult = 0x0101010101010101;
const ulong mask1h = (~0UL) / 3 << 1;
const ulong mask2l = (~0UL) / 5;
const ulong mask4l = (~0UL) / 17;
word -= (mask1h & word) >> 1;
word = (word & mask2l) + ((word >> 2) & mask2l);
word += word >> 4;
word &= mask4l;
return (int)((word * mult) >> 56);
}
This will check 64-bits at a time. If you need more than that you can just split the input data into 64-bit words and compare them one by one and compute the average result.
Here's a .NET fiddle with the code and a working test case:
https://dotnetfiddle.net/5hYFtE

I would change the code to something along these lines:
// hardcoded bitmask
byte mask = 255;
float dendrite(byte input) {
int correct = 0;
// store the xor:ed result
byte xored = input ^ mask;
// loop through each bit
for(int i = 0; i < 8; i++) {
// if the bit is 0 then it was correct
if(!(xored & (1 << i)))
correct++;
}
return (float)correct/(float)mask.length;
}
The above uses a mask and input of 8 bits, but of course you could modify this to use a 4 byte integer and so on.
Not sure if this will work as expected, but it might give you some clues on how to proceed.
For example if you only would like to check the first 4 bits you could change the code to something like:
float dendrite(byte input) {
// hardcoded bitmask i.e 1101
byte mask = 13;
// number of bits to check
byte bits = 4;
int correct = 0;
// store the xor:ed result
byte xored = input ^ mask;
// loop through each bit, notice that we only checking the first 4 bits
for(int i = 0; i < bits; i++) {
// if the bit is 0 then it was correct
if(!(xored & (1 << i)))
correct++;
}
return (float)correct/(float)bits;
}
Of course it might be faster to actually use a int instead of a byte.

Rounding value to nearest power of two

I am looking for the fastest way in C# to round a value to the nearest power of two.
I've discovered that the fastest way to round a value to the next power of two if to use bitwise operators like this.
int ToNextNearest(int x)
{
if (x < 0) { return 0; }
--x;
x |= x >> 1;
x |= x >> 2;
x |= x >> 4;
x |= x >> 8;
x |= x >> 16;
return x + 1;
}
But this gives the next nearest and not the nearest and I would like to only have the nearest power of two.
Here is a simple way to do that.
int ToNearest(int x)
{
Math.Pow(2, Math.Round(Math.Log(x) / Math.Log(2)));
}
But is there a better optimized version of finding the nearest power of two value ?
Thanks a lot.

Surely the best way is to use your bitwise routine to find the next power of two, then divide that result by two. This gives you the previous power of two. Then a simple comparison will tell you which of the two is closer.
int ToNearest(int x)
{
int next = ToNextNearest(x);
int prev = next >> 1;
return next - x < x - prev ? next : prev;
}
Untested code but you get the idea.

I'm using this:
public static int CeilPower2(int x)
{
if (x < 2) {
return 1;
}
return (int) Math.Pow(2, (int) Math.Log(x-1, 2) + 1);
}
public static int FloorPower2(int x)
{
if (x < 1) {
return 1;
}
return (int) Math.Pow(2, (int) Math.Log(x, 2));
}

On .Net Core, the fastest way to do this would probably to use the intrinsics operations:
private static int NearestPowerOf2(uint x)
{
return 1 << (sizeof(uint) * 8 - BitOperations.LeadingZeroCount(x - 1));
}
On CPU supporting the LZCNT instructions, it is just 6 CPU instructions, without branching.

.Net6 has introduced a method for this
using System.Numerics;
var nearestPowOf2 = BitOperations.RoundUpToPowerOf2(100); //returns 128

How about this:
int ToNearest(int val, int pow)
{
if (pow < 0) return 0;
if (pow == 0) return val;
if (val & (1 << (pow - 1))) {
return ((val >> pow) + 1) << pow;
} else {
return (val >> pow) << pow;
}
}

Haven't tested but i think this could work
int ToNearest(value x)
{
int num = 0;
for(int i=1; i < 65; i++)
{
int cur = Math.Abs(value - 0<<i);
if(Math.Abs(value - 0<<i) < Math.Abs(value - num))
num = cur;
else if(num != 0) break;
}
return num;
}

This is the full implementation of the suggested solution of #john, with the change that it will round up if the value is exactly in between the next and previous power of two.
public static int RoundToNextPowerOfTwo(int a)
{
int next = CeilToNextPowerOfTwo(a);
int prev = next >> 1;
return next - a <= a - prev ? next : prev;
}
public static int CeilToNextPowerOfTwo(int number)
{
int a = number;
int powOfTwo = 1;
while (a > 1)
{
a = a >> 1;
powOfTwo = powOfTwo << 1;
}
if (powOfTwo != number)
{
powOfTwo = powOfTwo << 1;
}
return powOfTwo;
}

Since C# requires IEEE754 floats there is probably a faster way on any platform that does not emulate the floating point functions:
int ToNearestPowerOf2(int x) =>
1 << (int)(BitConverter.DoubleToInt64Bits(x + x/3) >> 52) - 1023;
Rationale:
x + x/3
nearest power of 2, basically *4/3
(BitConverter.DoubleToInt64Bits(x) >> 52) - 1023
take floating point exponent for floor(ln2(x))
1 << x
exponential function with base 2
The function obviously requires a positive value for x.
0 won't work because the closest power of 2 is -∞,
and negative values have a complex logarithms.
Whether this is the fastest way will probably highly depend on what the JIT optimizer squeezes out of the code, more specifically how it handles the hard pointer cast in DoubleToInt64Bits. This may prevent other optimizations.
You do not have to use any comparison to get the nearest power of 2. Since all powers of two are separated by the same factor the rounding point is always at 3/4 of the next power of 2 (i.e. exactly the topmost 2 bits are set). So multiplication by the reciprocal followed by truncation will do the job.

Operator '&' cannot be applied to operands of type 'ulong' and 'ulong*'

Operator '&' cannot be applied to operands of type 'ulong' and 'ulong*'
What am I doing wrong? I'm trying to find which masks a integer consists of, if that makes sense.
e.g.
63 = 1+2+4+8+16+32
unsafe
{
UInt64 n = Convert.ToUInt64(textAttributes.Text);
UInt64* p = &n;
for(UInt64 i = 1; i <= n; i <<= 1)
{
if (i & p)
{
switch(i)
{
default:
break;
}
}
}
}

You dont need unsafe code for this.
The compiler error is legitimate since you apply the operator & between a pointer and an integer
You probably want :
UInt64 n = 63;
for(int i = 0; i < 64; i++)
{
UInt64 j = ((UInt64) 1) << i;
if ((j & n) != 0)
{
Console.WriteLine(1 << i);
}
}

What you're trying to do there is a bitwise AND against a memory address. You need to dereference that pointer if you want to do anything with it:
if ((i & *p) != 0)
// ^^ dereference
Dereferencing, via an asterisk prefix, will retrieve the value at that memory address. Without it.. its the memory address itself 1
1. In C# it is a compiler error. But that is what it is.

Well you do not need unsafe context for such operation
Try this :
static void Main(string[] args)
{
UInt64 n = Convert.ToUInt64(63);
int size = Marshal.SizeOf(n) * 8;
for (int i = size - 1; i >= 0; i--)
{
Console.Write((n >> i) & 1);
}
}
This will print 0000000000000000000000000000000000000000000000000000000000111111 so you will know which bit are set!

Find bit position without using Log()

I have an integer input that is power of 2 (1, 2, 4, 8 etc). I want the function to return bit position without using log(). For example, for inputs above will return {0, 1, 2, 3} respectively This for C#. Plus if this can be done in SQL.
Thanks!

I don't have VS on my Mac to test this out, but did you want something like this?
public static int Foo(int n)
{
if (n <= 0)
{
return -1; // A weird value is better than an infinite loop.
}
int i = 0;
while (n % 2 == 0)
{
n /= 2;
i++;
}
return i;
}

The fastest code I found to do this is from the Bit Twiddling Hacks site. Specifically, the lookup based on the DeBruijn sequence. See http://graphics.stanford.edu/~seander/bithacks.html#IntegerLogDeBruijn
I tested a naive method, a switch-based method, and two of the Bit Twiddling Hacks methods: the DeBruijn sequence, and the other that says, "if you know your value is a power of two."
I ran all of these against an array of 32 million powers of two. That is, integers of the form 2^N, where N is in the range 0..30. A value of 2^31 is a negative number, which causes the naive method to go into an infinite loop.
I compiled the code with Visual Studio 2010 in release mode and ran it without the debugger (i.e. Ctrl+F5). On my system, the averages over several dozen runs are:
Naive method: 950 ms
Switch method: 660 ms
Bithack method 1: 1,154 ms
DeBruijn: 105 ms
It's clear that the DeBruijn sequence method is much faster than any of the others. The other Bithack method is inferior here because the conversion from C to C# results in some inefficiencies. For example, the C statement int r = (v & b[0]) != 0; ends up requiring an if or a ternary operator (i.e. ?:) in C#.
Here's the code.
class Program
{
const int Million = 1000 * 1000;
static readonly int NumValues = 32 * Million;
static void Main(string[] args)
{
// Construct a table of integers.
// These are random powers of two.
// That is 2^N, where N is in the range 0..31.
Console.WriteLine("Constructing table");
int[] values = new int[NumValues];
Random rnd = new Random();
for (int i = 0; i < NumValues; ++i)
{
int pow = rnd.Next(31);
values[i] = 1 << pow;
}
// Run each one once to make sure it's JITted
GetLog2_Bithack(values[0]);
GetLog2_DeBruijn(values[0]);
GetLog2_Switch(values[0]);
GetLog2_Naive(values[0]);
Stopwatch sw = new Stopwatch();
Console.Write("GetLog2_Naive ... ");
sw.Restart();
for (int i = 0; i < NumValues; ++i)
{
GetLog2_Naive(values[i]);
}
sw.Stop();
Console.WriteLine("{0:N0} ms", sw.ElapsedMilliseconds);
Console.Write("GetLog2_Switch ... ");
sw.Restart();
for (int i = 0; i < NumValues; ++i)
{
GetLog2_Switch(values[i]);
}
sw.Stop();
Console.WriteLine("{0:N0} ms", sw.ElapsedMilliseconds);
Console.Write("GetLog2_Bithack ... ");
sw.Restart();
for (int i = 0; i < NumValues; ++i)
{
GetLog2_Bithack(values[i]);
}
Console.WriteLine("{0:N0} ms", sw.ElapsedMilliseconds);
Console.Write("GetLog2_DeBruijn ... ");
sw.Restart();
for (int i = 0; i < NumValues; ++i)
{
GetLog2_DeBruijn(values[i]);
}
sw.Stop();
Console.WriteLine("{0:N0} ms", sw.ElapsedMilliseconds);
Console.ReadLine();
}
static int GetLog2_Naive(int v)
{
int r = 0;
while ((v = v >> 1) != 0)
{
++r;
}
return r;
}
static readonly int[] MultiplyDeBruijnBitPosition2 = new int[32]
{
0, 1, 28, 2, 29, 14, 24, 3, 30, 22, 20, 15, 25, 17, 4, 8,
31, 27, 13, 23, 21, 19, 16, 7, 26, 12, 18, 6, 11, 5, 10, 9
};
static int GetLog2_DeBruijn(int v)
{
return MultiplyDeBruijnBitPosition2[(uint)(v * 0x077CB531U) >> 27];
}
static readonly uint[] b = new uint[] { 0xAAAAAAAA, 0xCCCCCCCC, 0xF0F0F0F0, 0xFF00FF00, 0xFFFF0000};
static int GetLog2_Bithack(int v)
{
int r = (v & b[0]) == 0 ? 0 : 1;
int x = 1 << 4;
for (int i = 4; i > 0; i--) // unroll for speed...
{
if ((v & b[i]) != 0)
r |= x;
x >>= 1;
}
return r;
}
static int GetLog2_Switch(int v)
{
switch (v)
{
case 0x00000001: return 0;
case 0x00000002: return 1;
case 0x00000004: return 2;
case 0x00000008: return 3;
case 0x00000010: return 4;
case 0x00000020: return 5;
case 0x00000040: return 6;
case 0x00000080: return 7;
case 0x00000100: return 8;
case 0x00000200: return 9;
case 0x00000400: return 10;
case 0x00000800: return 11;
case 0x00001000: return 12;
case 0x00002000: return 13;
case 0x00004000: return 14;
case 0x00008000: return 15;
case 0x00010000: return 16;
case 0x00020000: return 17;
case 0x00040000: return 18;
case 0x00080000: return 19;
case 0x00100000: return 20;
case 0x00200000: return 21;
case 0x00400000: return 22;
case 0x00800000: return 23;
case 0x01000000: return 24;
case 0x02000000: return 25;
case 0x04000000: return 26;
case 0x08000000: return 27;
case 0x10000000: return 28;
case 0x20000000: return 29;
case 0x40000000: return 30;
case int.MinValue: return 31;
default:
return -1;
}
}
}
If I optimize the Bithack code by unrolling the loop and using constants instead of array lookups, its time is the same as the time for the switch statement method.
static int GetLog2_Bithack(int v)
{
int r = ((v & 0xAAAAAAAA) != 0) ? 1 : 0;
if ((v & 0xFFFF0000) != 0) r |= (1 << 4);
if ((v & 0xFF00FF00) != 0) r |= (1 << 3);
if ((v & 0xF0F0F0F0) != 0) r |= (1 << 2);
if ((v & 0xCCCCCCCC) != 0) r |= (1 << 1);
return r;
}

0] if number is zero or negative, return/throw error
1] In your language, find the construct that converts a number to base 2.
2] convert the base-2 value to string
3] return the length of the string minus 1.

Verbose code, but probably the fastest:
if (x < 1)
throw SomeException();
if (x < 2)
return 0;
if (x < 4)
return 1;
if (x < 8)
return 2;
//.... etc.
This involves no division, nor conversion to-from double. It requires only comparisons, which are very speedy. See Code Complete, 2nd edition, page 633, for a discussion.
If you know that the input will always be a power of two, you might get better performance from a switch block:
switch (input)
{
case 1:
return 0;
case 2:
return 1;
case 4:
return 2;
case 8:
return 3;
//...
default:
throw SomeException();
}
I tested the performance on 10 million random ints, and on 10 million randomly-selected powers of two. The results:
Bithacks 1: 1360 milliseconds
Bithacks 2: 1103 milliseconds
If: 1320 milliseconds
Bithacks 1 (powers of 2): 1335 milliseconds
Bithacks 2 (powers of 2): 1060 milliseconds
Bithacks 3 (powers of 2): 1286 milliseconds
If (powers of 2): 1026 milliseconds
Switch (powers of 2): 896 milliseconds
I increased the number of iterations by ten times, and got these results:
Bithacks 1: 13347 milliseconds
Bithacks 2: 10370 milliseconds
If: 12918 milliseconds
Bithacks 1 (powers of 2): 12528 milliseconds
Bithacks 2 (powers of 2): 10150 milliseconds
Bithacks 3 (powers of 2): 12384 milliseconds
If (powers of 2): 9969 milliseconds
Switch (powers of 2): 8937 milliseconds
Now I didn't do any profiling to see if I did something stupid in translating the bit hacks to C# code, nor to see how much of the execution time is spent in the function that computes the log. So this is just a back-of-the-envelope kind of calculation, but it does suggest that the if approach is about the same as the bit hacks algorithms, and switch is a bit faster. Additionally, the if and switch approaches are far easier to understand and maintain.

This is a CPU friendly way to do it:
int bitpos=-1;
while(num>0) {
num = num>>1;
bitpos++;
}
return bitpos;
For SQL, use CASE. You can do binary search using nested IF....ELSE if performance is a concern. But with just 32 possible values, the overheads of implementing it could be much more than something simple sequential search.

find bit position in a power-of-two byte (uint8 Flag with only one bit set) coded in C (don't know about C#)
This returns the bit position 1..8 - you may need to decrement the result for indexing 0..7.
int pos(int a){
int p = (a & 3) | ((a >> 1) & 6) | ((a >> 2) & 5) | ((a >> 3) & 4) | ((a >> 4) & 15) | ((a >> 5) & 2) | ((a >> 6) & 1);
return p;
}
For info on the "evolution" of the code snippet see my blogpost here http://blog.edutoolbox.de/?p=263
EDIT:
deBruijn style is about 100x faster ...

set some consecutive bits in a byte

I need an efficient method with the following signature:
public byte SetBits(byte oldValue, byte newValue, int startBit, int bitCount)
Which returns oldValue, only that starting from its startbit bit up to its startbit + bitcount bit (zero-based), it's replaced with the first bitcount bits of newValue
For example, if:
oldValue = 11101101
newValue = 10000011
startBit = 1
bitCount = 2
Then the result would be: 11101111 (the segment 10 in oldValue is replaced with the corresponding 11 segment in newValue)

Here you go... Bitshift both directions to get the mask... then use it to generate the new byte
public static byte SetBits(byte oldValue, byte newValue, int startBit, int bitCount)
{
if (startBit < 0 || startBit > 7 || bitCount < 0 || bitCount > 7
|| startBit + bitCount > 8)
throw new OverflowException();
int mask = (255 >> 8 - bitCount) << startBit;
return Convert.ToByte((oldValue & (~mask)) | ((newValue << startBit) & mask));
}

startBit--; //account for 0 indexing
byte flag = 1 << startBit;
for (int i = startBit; i < bitCount; i++, flag <<= 1)
{
byte mask = newValue & flag;
if (mask != 0)
oldValue |= mask;
else
oldValue &= ~(flag);
}
return oldValue;
Some brain compiled code here but it should be along the lines that you want if I read the question correctly.

If I understood your question, I think this is what you are after:
byte mask = 0xFF;
for (int i = startPos-1; i < numBits; i++)
{
if ((newValue & (1 << i)) == 1)
{
mask = (byte)(mask | (1 << i));
}
else
{
mask = (byte)(mask &~(1<<i));
}
}
return (byte)(oldValue & mask);
This code is based on some neat tricks from Low Level Bit Hacks You Absolutely Must Know
I know setting a bit in a byte initialized to 0xFF is really a no-op, but I felt the code should be left in as it can help show off what is really going on. I encourage users of the code to optimize it as needed.