I have a use case where I need to scramble an input in such a way that:
Each specific input always maps to a specific pseudo-random output.
The output must shuffle the input sufficiently so that an incrementing input maps to a pseudo-random output.
For example, if the input is 64 bits, there must be exactly 2^64 unique outputs, and these must break incrementing inputs as much as possible (arbitrary requirement).
I will code this in C#, but can translate from Java or C, so long as there are not SIMD intrinsics. What I am looking for is some already existing code, rather than reinventing the wheel.
I have looked on Google, but haven't found anything that does a 1:1 mapping.
This seems to work fairly well:
const long multiplier = 6364136223846793005;
const long mulinv_multiplier = -4568919932995229531;
const long offset = 1442695040888963407;
static long Forward(long x)
{
return x * multiplier + offset;
}
static long Reverse(long x)
{
return (x - offset) * mulinv_multiplier;
}
You can change the constants to whatever as long as multiplier is odd and mulinv_multiplier is the modular multiplicative inverse (see wiki:modular multiplicative inverse or Hackers Delight 10-15 Exact Division by Constants) of multiplier (modulo 2^64, obviously - and that's why multiplier has to be odd, otherwise it has no inverse).
The offset can be anything, but make it relatively prime with 2^64 just to be on the safe side.
These specific constants come from Knuths linear congruential generator.
There's one small thing: it puts the complement of the LSB of the input in the LSB of the result. If that's a problem, you could just rotate it by any nonzero amount.
For 32 bits, the constants can be multiplier = 0x4c957f2d, offset = 0xf767814f, mulinv_multiplier = 0x329e28a5.
For 64 bits, multiplier = 12790229573962758597, mulinv_multiplier = 16500474117902441741 may work better.
Or, you could use a CRC, which is reversible for this use (ie the input is the same size as the CRC) for CRC64 it requires some modifications of course.
Just from the top of my head:
Shift the input: Make sure you keep every bit, i.e. use two shift operations in different directions and OR the result together.
Apply an static XOR.
Everything else that comes to my mind won't be bijective. However, a search for bijective might bring up something useful ;D
Related
I want to change a value, of, let's say, type int to be of type short, and making the value itself be "normalized" to the maximum value short can store - that is, so int.MaxValue would convert into short.MaxValue, and vice versa.
Here's an example using floating-point math to demonstrate:
public static short Rescale(int value){
float normalized = (float)value / int.MaxValue; // normalize the value to -1.0 to 1.0
float rescaled = normalized * (float)(short.MaxValue);
return (short)(rescaled);
}
While this works, it seems like using floating-point math is really inefficient, and can be improved, as we're dealing with binary data here. I tried using bit-shifting, but with to no avail.
Both signed and unsigned values are going to be processed - that isn't really an issue with the floating point solution, but when bit-shifting and doing other bit-manipulation, that makes things much more difficult.
This code will be used in quite a performance heavy context - it will be called 512 times every ~20 milliseconds, so performance is pretty important here.
How can I do this with bit-manipulation (or plain old integer algebra, if bit manipulation isn't necessary) and avoid floating-point math when we're operating on integer values?
You should use the shift operator. It is very fast.
int is 32bits, short is 16, so shift 16 bits right to scale your int to a short:
int x = 208908324 ;
//32 bits vs 16 bits.
short k = (short) (x >> 16);
Just reverse the process for scaling up. Obviously the lower bits will be filled with zeros.
In C++ I was able to use std::map<double, T> which is an ordered dictionary for its keys, but is a Red-Black tree which gives me O(lg n) for both insert and search. I was able to look up whether a value existed within some epsilon by using std::lower_bound and std::upper_bound together.
I have not been able to find the same thing while using C# 7+/.NET Core. Does such a thing exist?
In pseudocode, I'd like to do something like this
Map<float, T> map = ...
// key epsilon newValue
map.Insert(0.5f, 0.1f, someObj); // No values in the map, inserts fine
map.Get( 0.45f, 0.1f); // 0.45 +/- 0.1 contains 0.5, would return someObj
map.Get( 0.3f, 0.1f); // 0.3 +/- 0.1 does not include 0.5, it is not found
map.Insert(0.55f, 0.1f, anotherObj); // 0.55 +/- 0.1 includes 0.5, replace someObj
map.Insert(0.35f, 0.1f, anObj); // 0.35 +/- 0.1 doesn't overlap, insert new value
The way I'd have to do it would be to roll my own self-balancing binary search tree, but I'd rather not reinvent the wheel if such a thing exists.
I've been looking at SortedDictionary, however its Keys field is a collection so I can't jump around in it. Same issue for OrderedDictionary, unless I missed something.
I may not be able to use a SortedList since there will be more insertions than lookups, and due to the random order I'm worried that I'll end up getting a lot of O(n) swaps that need to be done when insertions. I'm assuming a uniform distribution in my input (which is very likely the case because of the data I'm working with), which means the insertions towards the middle and the front would cause a lot of shifting if it implements it the way I think it does... which would give me on average a cost of n/2 insertions and leave me at O(n). At least with a binary search tree, I'm getting O(lg n). Therefore the good solution here may not be applicable.
Most importantly, this is an algorithm that is used in a very hot section of the code. Performance is extremely important, choosing something that is not fast will likely drastically damage the performance of the application. I really need O(lg n) or some novel way of doing this that I didn't think of before.
My idea is to combine two data structures, SortedSet and a regular map.
SortedSet has GetViewBetween method, which has expected performance.
https://github.com/dotnet/corefx/pull/30921
Note: the expected performance of this method is met only in .NET core, it was much slower in the past: Why SortedSet<T>.GetViewBetween isn't O(log N)?
In this set you keep only the float keys.
Additionally, you have a Map from float to your desired type. You perform operations on the map only after checking your SortedSet.
I realize there are some rough edges (when an interval gives a few entries in the SortedSet), but I believe this is equivalent to the cpp implementation.
Hope you find this helpful, good luck with the implementation.
Now while this answer I'm about to give is a C++ profiled answer and not with C#, it solves the problem in a much better and faster way.
The better way to solve this is multiplying the floating point by the inverse of the epsilon. For example if your epsilon is 0.25, then you'd want to multiply all your floats/doubles by 4 and then cast it to an integer (or floor/ceil it if you care about things collecting around zero). The following uses int as the key but this would be fine for longs as well. My data fits in the +/- 2^31 range after quantizing (on computers with at least sizeof int being 4 bytes) so this is sufficient for me.
// Consider using std::is_floating_point_v for K
template <typename K, typename V>
class QuantizedGrid {
int quantizer;
std::unordered_map<int, V> map;
public:
explicit QuantizedGrid(const double epsilon) {
quantizer = 1.0 / epsilon;
}
V& operator[](const K k) {
return map[static_cast<int>(quantizer * k)];
}
bool contains(const K k) const {
int key = static_cast<int>(quantizer * k);
return map.count(key) > 0;
}
};
Compared to using upper/lower bound checks, the performance from that to the above code is as follows:
or rather it was 650% faster to convert to an integer and insert into a dictionary that supports O(1) amortized insertion/lookup/delete.
It is also way less code than implementing a custom upper/lower bound.
My guess is the O(lg n) BST lookup time is much worse by the O(1) dictionary time, and the cost of casting a float to and int is small enough to make this bound by data structure lookups/cache issues.
I have tried BigInteger, decimal, float and long but no luck.
Screenshot of required output example
It is a fairly easy task to write your own rational class; remember, rationals are just pairs of integers, and you already have BigInteger.
In this series of articles I show how to devise your own big integer and big rational classes starting from absolutely nothing, not even integers. Note that this is not fast and not intended to be fast; it is intended to be educational. You can use the techniques I describe in this series to help you when designing your arithmetic class.
https://ericlippert.com/2013/09/16/math-from-scratch-part-one/
Or, if you don't want to write it yourself, you can always use the one from Microsoft:
http://bcl.codeplex.com/wikipage?title=BigRational&referringTitle=Home
But that said...
I need a minimum of 128 decimal places to calculate precise probabilities of events between different time steps
Do you need 128 decimal places to represent 128 digits of precision, or of magnitude? Because if it is just magnitude, then simply do a transformation of your probability math into logarithms and do the math in doubles.
The easiest way to achieve arbitrary precision numbers is to combine the BigInteger class from System.Numerics with an int exponent. You could use BigInteger for your exponent, but this is likely overkill as the numbers would be well beyong meaningful in scale.
So if you create a class along these lines:
public class ArbDecimal
{
BigInteger value;
int exponent;
public override string ToString()
{
StringBuilder sb = new StringBuilder();
int place;
foreach (char digit in value.ToString())
{
if (place++ == value.ToString().Length - exponent)
{
sb.Append('.');
}
sb.Append(digit);
}
return sb.ToString();
}
}
You should then be able to define your mathematical operations using the laws of indices with the value and exponent fields.
For instance, to achieve addition, you would scale the larger value to have the same exponent as the smaller one by multiplying it by 10^(largerExp-smallerExp) then adding the two values and rescaling.
In your class, the number 0.01 would be represented like:
value = 1
exponent = -2
Due to the fact that 1*10^-2 = 0.01.
Utilising this method, you can store arbitrarily precise (and large) numbers limited only by the available ram and the .NET framework's object size limit.
I refer to the Rabin Karp Wikipedia article on Hash use.
In the example, the string "hi" is hashed using a prime number 101 as the base.
hash("hi")= ASCII("h")*101^1+ASCII("i")*101^0 = 10609
Can such an algorithm be used practically in Java or C# where long has a maximum value of 9,223,372,036,854,775,807? Naively, to me it seems that the hash value grows exponentially and with a large enough N (being string length) will result in overflow of the long type. For example, say I have 65 characters in my string input for the hash?
Is this correct, or are there methods of implementation which will never need to overflow (I can imagine possibly some lazy evaluation which merely stores the ascii and unit place in the prime base)?
hash("hi")= ASCII("h")*101^1+ASCII("i")*101^0 = 10609
That's only half the truth. In reality, if you would actually compute the value s_0 * p^0 + s_1 * p^1 + ... + s_n * p^n, the result would be a number whose representation would be about as long as the string itself, so you haven't gained anything. So what you actually do is to compute
(s_0 * p^0 + s_1 * p^1 + ... + s_n * p^n) mod M
where M is reasonably small. Thus your hash value will always be smaller than M.
So what you do in practice is you choose M = 2^64 and make use of the fact that unsigned integer overflow is well-defined in most programming languages. In fact, multiplication and addition of 64-bit integers in Java, C++ and C# is equivalent to multiplication and addition modulo 2^64.
It's not necessarily a wise choice to use 2^64 as the modulus. In fact you can easily construct a string with lots of collisions, thus provoking the worst case behaviour of Rabin-Karp, which is Ω(n * m) matching instead of O(n + m).
It would be better to use a large prime as the modulus and get much better collision resistance. The reason why this is usually not done is performance: We would need to explicitely use modular reduction (add a % M) to every addition and multiplication. What's worse, we can't even use the builtin multiplication anymore, because it could overflow if M > 2^32. So we need a custom MultiplyMod function, which is bound to be a lot slower than machine-level multiplication.
Is this correct, or are there methods of implementation which will never need to overflow (I can imagine possibly some lazy evaluation which merely stores the ascii and unit place in the prime base)?
As I already mentioned, if you don't reduce using a modulus, your hash value will grow as large as the string itself, thus rendering it useless to use a hash function in the first place. So yes, using controlled overflow modulo 2^64 is correct and even necessary if we don't manually reduce.
If your goal is a type of storage which contains only "small" number,
but where the sum can be compared:
You could view this simply as 101 - number system,
like 10=decimal, 16=hex. and so on.
Ie.
a) You have to store a set of { ascii value and it´s 101-power }
(without possibility for multiple entries with the same power).
b) When creating the data from a string,
values >101 have to be propagated (is this the right word?) to the next power.
Example 1:
"a" is 97*101^0
(trivial)
Example 2:
"g" is 1*101^1 + 2*101^0
because g is 103. 103>=101 ie. take only 103%101 for 101^0
(modulo, remainder of division)
and (int)(103/101) for the next power.
(if the ascii numers could be higher or the prime number is lower than 101
it could be possible that (int)(103/101) would exceed the prime numer too.
In this case, it would continue to prime^2 and so on, until the value is smaller
than the prime number)
Example 3:
"ag" is 98*101^1 + 2*101^0
Compared to above, 97*101^1 is added because of a.
and so on...
To compare without calculating the full sum,
just compare the values of one power to each other, for each power.
Equal if all "power values" are the same.
Side note: Be aware that ^ is not exponentiation in languages like C# and Java.
Given two float values (fLow and fHigh), how could you calculate the greatest or maximum stride/gap between the two successive values?
For example:
In the range 16777217f to 20000000f the answer would be 2, as values are effectively rounded to the nearest two.
Generalizing this to an arbitrary range has got me scratching my head - any suggestions?
cheers,
This should be language neutral, but I'm using C# (which conforms to IEEE-754 for this, I think).
This is in C. It requires some IEEE 754 behavior, for rounding and such. For IEEE 754 64-bit binary (double), SmallestPositive is 2-1074, approximately 4.9406564584124654417656879286822137236505980261e-324, and DBL_EPSILON is 2-52, 2.220446049250313080847263336181640625e-16. For 32-bit binary (float), change DBL to FLT and double to float wherever they appear (and fabs to fabsf and fmax to fmaxf, although it should work without these changes). Then SmallestPositive is 2-149, approximately 1.401298464324817070923729583289916131280261941876515771757068283889791e-45, and FLT_EPSILON is 2-23, 1.1920928955078125e-07.
For an interval between two values, the greatest step size is of course the step size at the endpoint with larger magnitude. (If that endpoint is exactly a power of two, the step size from that point to the next does not appear in the interval itself, so that would be a special case.)
#include <float.h>
#include <math.h>
/* Return the ULP of q.
This was inspired by Algorithm 3.5 in Siegfried M. Rump, Takeshi Ogita, and
Shin'ichi Oishi, "Accurate Floating-Point Summation", _Technical Report
05.12_, Faculty for Information and Communication Sciences, Hamburg
University of Technology, November 13, 2005.
*/
double ULP(double q)
{
// SmallestPositive is the smallest positive floating-point number.
static const double SmallestPositive = DBL_EPSILON * DBL_MIN;
/* Scale is .75 ULP, so multiplying it by any significand in [1, 2) yields
something in [.75 ULP, 1.5 ULP) (even with rounding).
*/
static const double Scale = 0.75 * DBL_EPSILON;
q = fabs(q);
return fmax(SmallestPositive, q - (q - q * Scale));
}
Well, machine accuracy is, as the name indicates, really something that might in general depend on the machine and even on the compiler. So, to be really sure you will typically have to write a program that actually tests what is going on.
However, I suspect that you are really looking for some handy formulas that you can use to approximate the maximum distance in a given interval. The Wikipedia article on machine epsilon gives a really nice overview over this topic and I'm mostly quoting from this source in the following.
Let s be the machine epsilon of your floating point representation (i.e., about 2^(-24) in the case of standard floats), then the maximum spacing between a normalised number x and its neighbors is 2*s*|x|. The word normalised is really crucial here and I will not even try to consider the situation for de-normalised numbers because this is where things get really nasty...
That is, in your particular case the maximum spacing h in the interval you propose is given by h = 2*s*max(|fLow|, |fHigh|).