Develop Reference

Speed up processing 32 bit numbers in combinations (k from n)

I have a list of 128 32 bit numbers, and I want to know, is there any combination of 12 numbers, so that all numbers XORed give the 32 bit number with all bits set to 1.
So I have started with naive approach and took combinations generator like that:
private static IEnumerable<int[]> Combinations(int k, int n)
{
var state = new int[k];
var stack = new Stack<int>();
stack.Push(0);
while (stack.Count > 0)
{
var index = stack.Count - 1;
var value = stack.Pop();
while (value < n)
{
state[index++] = value++;
if (value < n)
{
stack.Push(value);
}
if (index == k)
{
yield return state;
break;
}
}
}
}
and used it like that (data32 is an array of given 32bit numbers)
foreach (var probe in Combinations(12, 128))
{
int p = 0;
foreach (var index in probe)
{
p = p ^ data32[index];
}
if (p == -1)
{
//print out found combination
}
}
Of course it takes forever to check all 23726045489546400 combinations...
So my question(s) are - am I missing something in options how to speedup the check process?
Even if I do the calculation of combinations in partitions (e.g. I could start like 8 threads each will check combination started with numbers 0..8), or speed up the XORing by storing the perviously calculated combination - it is still slow.
P.S. I'd like it to run in reasonable time - minutes, hours not years.
Adding a list of numbers as was requested in one of the comments:
1571089837
2107702069
466053875
226802789
506212087
484103496
1826565655
944897655
1370004928
748118360
1000006005
952591039
2072497930
2115635395
966264796
1229014633
827262231
1276114545
1480412665
2041893083
512565106
1737382276
1045554806
172937528
1746275907
1376570954
1122801782
2013209036
1650561071
1595622894
425898265
770953281
422056706
477352958
1295095933
1783223223
842809023
1939751129
1444043041
1560819338
1810926532
353960897
1128003064
1933682525
1979092040
1987208467
1523445101
174223141
79066913
985640026
798869234
151300097
770795939
1489060367
823126463
1240588773
490645418
832012849
188524191
1034384571
1802169877
150139833
1762370591
1425112310
2121257460
205136626
706737928
265841960
517939268
2070634717
1703052170
1536225470
1511643524
1220003866
714424500
49991283
688093717
1815765740
41049469
529293552
1432086255
1001031015
1792304327
1533146564
399287468
1520421007
153855202
1969342940
742525121
1326187406
1268489176
729430821
1785462100
1180954683
422085275
1578687761
2096405952
1267903266
2105330329
471048135
764314242
459028205
1313062337
1995689086
1786352917
2072560816
282249055
1711434199
1463257872
1497178274
472287065
246628231
1928555152
1908869676
1629894534
885445498
1710706530
1250732374
107768432
524848610
2791827620
1607140095
1820646148
774737399
1808462165
194589252
1051374116
1802033814

I don't know C#, I did something in Python, maybe interesting anyway. Takes about 0.8 seconds to find a solution for your sample set:
solution = {422056706, 2791827620, 506212087, 1571089837, 827262231, 1650561071, 1595622894, 512565106, 205136626, 944897655, 966264796, 477352958}
len(solution) = 12
solution.issubset(nums) = True
hex(xor(solution)) = '0xffffffff'
There are 128C12 combinations, that's 5.5 million times as many as the 232 possible XOR values. So I tried being optimistic and only tried a subset of the possible combinations. I split the 128 numbers into two blocks of 28 and 100 numbers and try combinations with six numbers from each of the two blocks. I put all possible XORs of the first block into a hash set A, then go through all XORs of the second block to find one whose bitwise inversion is in that set. Then I reconstruct the individual numbers.
This way I cover (28C6)2 × (100C6)2 = 4.5e14 combinations, still over 100000 times as many as there are possible XOR values. So probably still a very good chance to find a valid combination.
Code (Try it online!):
from itertools import combinations
from functools import reduce
from operator import xor as xor_
nums = list(map(int, '1571089837 2107702069 466053875 226802789 506212087 484103496 1826565655 944897655 1370004928 748118360 1000006005 952591039 2072497930 2115635395 966264796 1229014633 827262231 1276114545 1480412665 2041893083 512565106 1737382276 1045554806 172937528 1746275907 1376570954 1122801782 2013209036 1650561071 1595622894 425898265 770953281 422056706 477352958 1295095933 1783223223 842809023 1939751129 1444043041 1560819338 1810926532 353960897 1128003064 1933682525 1979092040 1987208467 1523445101 174223141 79066913 985640026 798869234 151300097 770795939 1489060367 823126463 1240588773 490645418 832012849 188524191 1034384571 1802169877 150139833 1762370591 1425112310 2121257460 205136626 706737928 265841960 517939268 2070634717 1703052170 1536225470 1511643524 1220003866 714424500 49991283 688093717 1815765740 41049469 529293552 1432086255 1001031015 1792304327 1533146564 399287468 1520421007 153855202 1969342940 742525121 1326187406 1268489176 729430821 1785462100 1180954683 422085275 1578687761 2096405952 1267903266 2105330329 471048135 764314242 459028205 1313062337 1995689086 1786352917 2072560816 282249055 1711434199 1463257872 1497178274 472287065 246628231 1928555152 1908869676 1629894534 885445498 1710706530 1250732374 107768432 524848610 2791827620 1607140095 1820646148 774737399 1808462165 194589252 1051374116 1802033814'.split()))
def xor(vals):
return reduce(xor_, vals)
A = {xor(a)^0xffffffff: a
for a in combinations(nums[:28], 6)}
for b in combinations(nums[28:], 6):
if a := A.get(xor(b)):
break
solution = {*a, *b}
print(f'{solution = }')
print(f'{len(solution) = }')
print(f'{solution.issubset(nums) = }')
print(f'{hex(xor(solution)) = }')

Arrange your numbers into buckets based on the position of the first 1 bit.
To set the first bit to 1, you will have to use an odd number of the items in the corresponding bucket....
As you recurse, try to maintain an invariant that the number of leading 1 bits is increasing and then select the bucket that will change the next 0 to a 1, this will greatly reduce the number of combinations that you have to try.

I have found a possible solution, which could work for my particular task.
As main issue to straitforward approach I see a number of 2E16 combinations.
But, if I want to check if combination of 12 elements equal to 0xFFFFFFFF, I could check if 2 different combinations of 6 elements with opposit values exists.
That will reduce number of combinations to "just" 5E9, which is achievable.
On first attempt I think to store all combinations and then find opposites in the big list. But, in .NET I could not find quick way of storing more then Int32.MaxValue elements.
Taking in account idea with bits from comments and answer, I decided to store at first only xor sums with leftmost bit set to 1, and then by definition I need to check only sums with leftmost bit set to 0 => reducing storage by 2.
In the end it appears that many collisions could appear, so there are many combinations with the same xor sum.
Current version which could find such combinations, need to be compiled in x64 mode and use (any impovements welcomed):
static uint print32(int[] comb, uint[] data)
{
uint p = 0;
for (int i = 0; i < comb.Length; i++)
{
Console.Write("{0} ", comb[i]);
p = p ^ data[comb[i]];
}
Console.WriteLine(" #[{0:X}]", p);
return p;
}
static uint[] data32;
static void Main(string[] args)
{
int n = 128;
int k = 6;
uint p = 0;
uint inv = 0;
long t = 0;
//load n numbers from a file
init(n);
var lookup1x = new Dictionary<uint, List<byte>>();
var lookup0x = new Dictionary<uint, List<byte>>();
Stopwatch watch = new Stopwatch();
watch.Start();
//do not use IEnumerable generator, use function directly to reuse xor value
var hash = new uint[k];
var comb = new int[k];
var stack = new Stack<int>();
stack.Push(0);
while (stack.Count > 0)
{
var index = stack.Count - 1;
var value = stack.Pop();
if (index == 0)
{
p = 0;
Console.WriteLine("Start {0} sequence, combinations found: {1}",value,t);
}
else
{
//restore previous xor value
p = hash[index - 1];
}
while (value < n)
{
//xor and store
p = p ^ data32[value];
hash[index] = p;
//remember current state (combination)
comb[index++] = value++;
if (value < n)
{
stack.Push(value);
}
//combination filled to end
if (index == k)
{
//if xor have MSB set, put it to lookup table 1x
if ((p & 0x8000000) == 0x8000000)
{
lookup1x[p] = comb.Select(i => (byte)i).ToList();
inv = p ^ 0xFFFFFFFF;
if (lookup0x.ContainsKey(inv))
{
var full = lookup0x[inv].Union(lookup1x[p]).OrderBy(x=>x).ToArray();
if (full.Length == 12)
{
print32(full, data32);
}
}
}
else
{
//otherwise put it to lookup table 2, but skip all combinations which are started with 0
if (comb[0] != 0)
{
lookup0x[p] = comb.Select(i => (byte)i).ToList();
inv = p ^ 0xFFFFFFFF;
if (lookup1x.ContainsKey(inv))
{
var full = lookup0x[p].Union(lookup1x[inv]).OrderBy(x=>x).ToArray();
if (full.Length == 12)
{
print32(full, data32);
}
}
}
}
t++;
break;
}
}
}
Console.WriteLine("Check was done in {0} ms ", watch.ElapsedMilliseconds);
//end
}

C# Get Bits from Byte

I don't know why I am getting wrong bits when reading my byte. I have followed some indications found here, bit it is not working. I show you my code:
byte[] byte22And23 = _packet.ReadBytes(2);
DataFromGTAPackets.Packet8.ControlFlags.rightEngineSmoke = _packet.GetBit(byte22And23[0], 0);
DataFromGTAPackets.Packet8.ControlFlags.leftEngineSmoke = _packet.GetBit(byte22And23[0], 1);
DataFromGTAPackets.Packet8.ControlFlags.rightEngineFire = _packet.GetBit(byte22And23[0], 2);
DataFromGTAPackets.Packet8.ControlFlags.leftEngineFire = _packet.GetBit(byte22And23[0], 3);
DataFromGTAPackets.Packet8.ControlFlags.rightRotorFail = _packet.GetBit(byte22And23[0], 4);
DataFromGTAPackets.Packet8.ControlFlags.leftRotorFail = _packet.GetBit(byte22And23[0], 5);
etc...
public bool GetBit(byte b, int bitNumber)
{
bool bit = (b & (1 << bitNumber - 1)) != 0;
return bit;
}
My byte 22 has a value of 2, it is: 00000010. And my byte 23 has a value of 0. When I use this code to read byte 22, my 3rd variable (rightEngineFire) is getting "true" (1). It has no sense, it is wrong obviously. But I don't know what is wrong.
Thank you!

You GetBit method considers bit numbers to be 1...32, instead of being 0...31.
Simple test:
bool b1 = GetBit(1, 0); // false
bool b2 = GetBit(1, 1); // true
You should change the method to
public static bool GetBit(byte b, int bitNumber)
{
bool bit = (b & (1 << bitNumber)) != 0;
return bit;
}
Or you could write:
DataFromGTAPackets.Packet8.ControlFlags.rightEngineSmoke = _packet.GetBit(byte22And23[0], 1);
But in C-derived languages (and C# is a C-derived language) we count from 0, so the first solution is better (for me).
And as a sidenote:
1 << bitNumber - 1
is quite unreadable, because probably 9 out of 10 programmers don't know/don't remember that the expression means
1 << (bitNumber - 1)
because in the operator precedence table the - comes before the <<, so when you mix operators you should use (...) to make clear the priority.

The GetBit() implementation is wrong, you can check with the following test:
[TestMethod]
public void MyTestMethod()
{
var byte22And23 = new byte[] { 2, 0 };
var sb = new StringBuilder();
for (int i = 7; i >=0; i--)
{
var r = GetBit(byte22And23[0], i);
sb.Append((r) ? "1" : "0");
}
// result: 00000100
Assert.AreEqual("00000010", sb.ToString());
}
It looks like there is no need of -1 in GetBit() because you are indexing bits 0-based and not 1-based:
public bool GetBit(byte b, int bitNumber)
{
// no need of -1 here ------------ˇ
bool bit = (b & (1 << bitNumber - 1)) != 0;
return bit;
}
After the modification the tests runs green.

Generating Schröder Paths

I want to generate schröder paths from (0, 0) to (2n, 0) with
no peaks, i.e., no up step followed immediately by a down step.
Some examples are for n=3 : shröder paths.
/ is coded as U, -- is coded as R and \ is coded as D. Here is my code to generate these paths :
public static void addParen(List<String> list, int upstock,int rightstock,int
downstock,bool B, char[] str, int count,int total,int n)
{
if (total == n && downstock == 0)
{
String s = copyvalueof(str);
list.Add(s);
}
if (total > n || (total==n && downstock>0) )
return;
else
{
if (upstock > 0 && total<n)
{
str[count] = 'U';
addParen(list, upstock - 1,rightstock, downstock+1,B=true, str, count + 1,total+1,n);
}
if (downstock > 0 && total<n && B==false)
{
str[count] = 'D';
addParen(list, upstock,rightstock, downstock - 1,B=false, str, count + 1,total+1,n);
}
if (rightstock > 0 && total < n)
{
str[count] = 'R';
addParen(list, upstock, rightstock-1, downstock, B = false, str, count + 1, total + 2,n);
}
}
}
public static List<String> generatePaths(int count)
{
char[] str = new char[count * 2];
bool B = false;
List<String> list = new List<String>();
addParen(list, count-1, count, 0,B,str, 0, 0,count*2);
return list;
}
The total is 2n. I start with n-1 ups n rights and zero downs.Since there is no Up yet my bool B is false ( if there comes an up then down can not come after it,so to prevent this I put B=true which prevents it.) If an up comes, then there should be a corresponding down, and total should be incremented by one. If right comes then, total should be incremented by 2. My algorithm in general works like that but I couldn't get a correct result with this implementation.

The original solution didn't adapt to OP's needs, as it was too complicated to port to javascript and the intention was to show better practices solving these kind of problems more than actually solving easily this one in particular.
But in the spirit of using immutable types to solve path algorithms, we can still do so in a much simpler fashion: we'll use string.
Ok as always, lets build up our infrastructure: tools that will make our life easier:
private const char Up = 'U';
private const char Down = 'D';
private const char Horizontal = 'R';
private static readonly char[] upOrHorizontal = new[] { Up, Horizontal };
private static readonly char[] downOrHorizontal = new[] { Down, Horizontal };
private static readonly char[] all = new[] { Up, Horizontal, Down };
And a handy little helper method:
private static IList<char> GetAllPossibleDirectionsFrom(string path)
{
if (path.Length == 0)
return upOrHorizontal;
switch (path.Last())
{
case Up: return upOrHorizontal;
case Down: return downOrHorizontal;
case Horizontal: return all;
default:
Debug.Assert(false);
throw new NotSupportedException();
}
}
Remember, break down your problems to smaller problems. All difficult problems can be solved solving smaller easier problems. This helper method is hard to get wrong; thats good, its hard to write a bug inside easy short methods.
And now, we solve the bigger problem. We'll not use iterator blocks so porting is easier. We'll concede here using a mutable list to keep track of all the valid paths we find.
Our recursive solution is the following:
private static void getPaths(IList<string> allPaths,
string currentPath,
int height,
int maxLength,
int maxHeight)
{
if (currentPath.Length == maxLength)
{
if (height == 0)
{
allPaths.Add(currentPath);
}
}
else
{
foreach (var d in GetAllPossibleDirectionsFrom(currentPath))
{
int newHeight;
switch (d)
{
case Up:
newHeight = height + 1;
break;
case Down:
newHeight = height - 1;
break;
case Horizontal:
newHeight = height;
break;
default:
Debug.Assert(false);
throw new NotSupportedException();
}
if (newHeight < 0 /*illegal path*/ ||
newHeight >
maxLength - (currentPath.Length + 1)) /*can not possibly
end with zero height*/
continue;
getPaths(allPaths,
currentPath + d.ToString(),
newHeight,
maxLength,
maxHeight);
}
}
}
Not much to say, its pretty self explanatory. We could cut back some on the arguments; height is not strictly necessary, we could count ups and downs in the current path and figure out the height we are currently at, but that seems wasteful. maxLength could also, and probably should be removed, we have enough information with maxHeight.
Now we just need a method to kick this off:
public static IList<string> GetSchroderPathsWithoutPeaks(int n)
{
var allPaths = new List<string>();
getPaths(allPaths, "", 0, 2 * n, n);
return allPaths;
}
And we're set! If we take this out for a test drive:
var paths = GetSchroderPathsWithoutPeaks(2);
Console.WriteLine(string.Join(Environment.NewLine, paths));
We get the expected results:
URRD
URDR
RURD
RRRR
As to why your solution doesn't work? Well, just the fact that you can't figure it out says it all about how unecessary complicated your current solution is starting to look. When that happens, its normally a good idea to take a step back, rethink your approach, write down a clear specification of what your program should be doing step by step and start over.

Sign extending from a constant bit width in C#

I have a value thats 5 bits in length. 4 bits determine the number and the 5th bit determines the sign, there by holding any value between -16 and +15. How can I accomplish sign extending from a constant bit width in C#? I know in C, I can use something like the follow to accomplish this:
int x; // convert this from using 5 bits to a full int
int r; // resulting sign extended number goes here
struct {signed int x:5;} s;
r = s.x = x;
How can I do something similar to this in C#?

It's not really clear what you mean, but it could be as simple as:
int fiveBits = normal & 0x1f;
and for the reverse:
int normal = fiveBits < 16 ? fiveBits : fiveBits | -32;
If you could suggest some original input and desired output, that would help.

Perform a left shift followed by an arithmetic right shift to move the sign bit into the high position and then back. The arithmetic right shift will perform the sign extension for you.
Of course this depends on having a working arithmetic shift operation. The abstract C language does not (it's implementation-defined whether it works or not), but most implementations do. I'm not sure about C# but I would guess it has one.

I know this is an old question, but for future searchers I have more info.
C# does not support custom bit widths, but it does support binary operations and getters/setters, which makes it relatively easy to add a compatibility layer. For instance, if you want to store the raw data in a byte _num, but want to be able to interact with it using a standard C# sbyte, you can use the following:
byte _num;
sbyte num {
get
{
return (sbyte)(((_num & 0x10) << 3) | (_num & 0x0F));
}
set
{
_num = (byte)((value & 0x0F) | ((value & 0x80) >> 3));
}
}
This kind of shell is especially useful when interacting with low level firmware or embedded projects.

I'm just writing a C function (because I don't really know C#) that will do this using operations that I know are available in C#.
int five_bit_to_signed(int five_bit) {
int sh = (sizeof(int)*8)-5;
int x = five_bit << sh; // puts your sign bit in the highest bit.
return x >> sh; // since x is signed this is an arithmatic signed shift
}

From your question, it appears you wish to have a structure that can readily be converted to and from an int type:
struct FiveBit
{
public int bits;
public static implicit operator int(FiveBit f)
{
return (f.bits & 0x10) == 0 ? f.bits : f.bits | -32;
}
public static implicit operator FiveBit(int r)
{
return new FiveBit() { bits = r & 0x1f };
}
}
And here's an example of usage:
class FiveBitTest
{
static void Main(string[] args)
{
FiveBit f = new FiveBit();
int r; // resulting sign extended number goes here
f.bits = 0;
r = f;
Console.WriteLine("r = {0}, f.bits = 0x{1:X}", r, f.bits);
f.bits = 0x1f;
r = f;
Console.WriteLine("r = {0}, f.bits = 0x{1:X}", r, f.bits);
r = -2;
f = r;
Console.WriteLine("r = {0}, f.bits = 0x{1:X}", r, f.bits);
}
The output of the above is:
r = 0, f.bits = 0x0
r = -1, f.bits = 0x1F
r = -2, f.bits = 0x1E

how to convert an int to a bitmask?

Is there a way to convert an int to a bitmask?
example:
int i = 33;
should be converted to (not sure of the datatype)
bool[] bitmask = new[] {true, false, false, false, false, true};
Update
In reaction to most answers:
I need to do this:
BitArray bits = new BitArray(BitConverter.GetBytes(showGroup.Value));
List<String> showStrings = new List<string>();
for (int i = 0; i < bits.Length; i++)
{
if(bits[i])
showStrings.Add((i+1).ToString().PadLeft(2, '0'));
}
How would that go without converting it to a bitarray?

An int already is a bitmask. If you want to twiddle the bits, you can use bitwise operators freely on ints. If you want to convert the int to an enum that has the Flags attribute, a simple cast will suffice.

Found it
BitArray bits = new BitArray(System.BitConverter.GetBytes(showGroup.Value));

You could construct a bool[32] and loop through all bits in the int, masking it with 2^(loop counter) and setting the bools in the array appropriately.
Are you sure you need this, though? Most operations with bitmasks work with ints directly.
To answer the question in your edit:
int val = 35;
List<string> showStrings = new List<string>();
for (int i = 0; i < 32; i++)
{
if (( (1 << i) & val) > 0)
{
showStrings.Add((i + 1).ToString().PadLeft(2, '0'));
}
}
prints:
01
02
06
Not the most obvious solution if you're not used to bit arithmetic, true. Mask each bit in the integer value with 2^(bit-index), and if the resulting value is greater than zero (indicating that the bit at that index is set), do something. 1 << i (left-shifting) is equivalent to 2^i, and may have the same performance characteristics once JITted, but I'm used to this form.
Expressed as a macro-like method:
bool IsSet(int val, int index)
{
return (( (1 << (index-1)) & val) > 0);
}

int val = 33;
var bitarray = new BitArray(new[] { val });
var att = bitarray.Cast<bool>().ToArray();

Since you asked, here is a solution without using BitArray:
// First define a bitmask enum for the bits you are interested in
[Flags]
public enum BitFlags
{
Flag1 = 1,
Flag2 = 2,
Flag3 = 4,
Flag4 = 8,
Flag5 = 16
// ...
}
int index = 0;
List<string> showStrings = new List<string>();
foreach(int flag in Enum.GetValues(typeof(BitFlags))cast<int>())
{
index += 1;
if ((input & flag) == flag)
showStrings.Add(index.ToString().PadLeft(2, '0'));
}
It is about the same amount of code, with negligible performance difference. It does however let you strongly define your bit values and you can choose to omit bits in the BitFlags enum that you don't care about.