Develop Reference

Alternative to hashing for quick comparison to avoid conflicts

I'm implementing a caching table to avoid having to perform a costly operation that creates a generic object from a set of parameters describing it. Once an object is requested, an hash of these parameters is computed, and a Dictionary containing the already created objects is queried to check if a copy has already been created, in which case its returned without the need of creating it again.
My problem lies in the fact that since the parameters describing these objects can be many, collisions in the hashing function are unavoidable (and too frequent), but on the other hand retrieving these objects is a performance-critical operation and i cannot afford full comparisons on all existing descriptions to search among already created objects.
I've tried to solve with many different hashing functions but since the nature of the parameters is unknown the results are unreliable.
What solutions other than hashing are there to this caching problem, or can hashing be used differently to avoid conflicts?
C# description of the problem:
class ObjectDescriptor
{
// description made of a list of parameters of unknown type
public object[] Fields;
// hashing procedure that may have conflicts
public override int GetHashCode()
{
int hash = 1009;
for (int i = 0; i < Fields.Length; i++)
{
unchecked { hash = hash * 9176 + Fields[i].GetHashCode(); }
}
return hash;
}
}
abstract class ObjectCache<T>
{
private Dictionary<int, T> indexedObjects;
// this operation is called many times and must be fast
public T Get(ObjectDescriptor descr)
{
T cachedValue;
if(!indexedObjects.TryGetValue(descr.GetHashCode(), out cachedValue))
{
cachedValue = CreateObject(descr);
indexedObjects[descr.GetHashCode()] = cachedValue;
}
return cachedValue;
}
// costly operation
protected abstract T CreateObject(ObjectDescriptor desc);
}

I'll leave the solution I ended up using. This is based on the fact that conflicts can be avoided by storing whole values from multiple fields in a single hash where possible:
byte b1 = 42, b2 = 255;
int naiveHash = CombineHash(b1.GetHashCode(), b2.GetHashCode()); // will always have conflicts
int typeAwareHash = b1 << 8 + b2; // no conflicts
To know how many bits are required by a field I required the implementation of IObjectDescriptorField:
interface IObjectDescriptorField
{
int GetHashCodeBitCount();
}
I then updated the ObjectDescriptor class with an HashCodeBuilder class:
class ObjectDescriptor
{
public IObjectDescriptorField[] Fields;
public override int GetHashCode()
{
HashCodeBuilder hash = new HashCodeBuilder();
for (int i = 0; i < Fields.Length; i++)
{
hash.AddBits(Fields[i].GetHashCode(), Fields[i].GetHashCodeBitCount());
}
return hash.GetHashCode();
}
}
HashCodeBuilder stack up bits until all 32 are used, and then uses a simple hash combination function like before:
public class HashCodeBuilder
{
private const int HASH_SEED = 352654597;
private static int Combine(int hash1, int hash2)
{
return ((hash1 << 5) + hash1 + (hash1 >> 27)) ^ hash2;
}
private int hashAccumulator;
private int bitAccumulator;
private int bitsLeft;
public HashCodeBuilder()
{
hashAccumulator = HASH_SEED;
bitAccumulator = 0;
bitsLeft = 32;
}
public void AddBits(int bits, int bitCount)
{
if (bitsLeft < bitCount)
{
hashAccumulator = Combine(hashAccumulator, bitAccumulator);
bitsLeft = 32;
hashAccumulator = 0;
}
bitAccumulator = bitAccumulator << bitCount + bits;
bitsLeft -= bitCount;
}
public override int GetHashCode()
{
return Combine(hashAccumulator, bitAccumulator);
}
}
This solution of course still have conflicts if more then 32 bits are used, but it worked for me because many of the fields where just bools or Enums with few values, which greatly benefit to be combined like this.

How to override Equals and GetHash of HashSet

I have a HashSet<int[]> foo where the int[] represents the coordinates of a point in a plane. The value at position 0 represents the x and the value at position 1 represents the y. I want to override the Equals and GetHashCode methods to be able to remove an element (the point represented as an array of size two) if its internal values are equals to a given one.
Already tried:
public override int GetHashCode(){
return this.GetHashCode();
}
public override bool Equals(object obj){
if (obj == null || ! (obj is int[]))
return false;
HashSet<int[]> item = obj as HashSet<int[]>;
return item == this;
}
In my class Maze.
Thanks in advance.
EDIT
I found a way to do that
class SameHash : EqualityComparer<int[]>
{
public override bool Equals(int[] i1, int[] i2)
{
return i1[0] == i2[0] && i1[1] == i2[1];
}
public override int GetHashCode(int[] i)
{
return base.GetHashCode();
}
}

It may seems like you solved what you asked for, but there is something important that should be pointed out. When you implemented the EqualityComparer<int[]> you coded the GetHashCode(int[] i) as return base.GetHashCode(); which is not correct even when it works. I took the time to provide you with the code below for you to see the results of your implementation and I also gave you a possible solution.
Copy this code and run it in a Console Project. Comment your line of code, uncomment the line right below it and run it again. You will see the difference!
Summarizing, when you return base.GetHashCode() you are returning the same hash code for every item. This causes collisions inside the hash set for all insertions ending up in a behavior as slow as if you were using a List<int[]> and you were asking if it contains an element before inserting it. That is why you will see that by using the function I provided you and for the range of numbers I'm generating you will be able to insert up to one million times in less than 1 sec. However, using yours, no matter the range, it spends 1 sec in around ten thousand insertions. This happens because for all n insertions there are collisions and the resulting time complexity is O(n^2) when the expected for a HashSet and an even distributed Hash Function is O(n).
Check this out:
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
namespace hashExample
{
class Program
{
static void Main(string[] args)
{
List<int[]> points = new List<int[]>();
Random random = new Random();
int toInsert = 20000;
for (int i = 0; i < toInsert; i++)
{
int x = random.Next(1000);
int y = random.Next(1000);
points.Add(new int[]{ x,y });
}
HashSet<int[]> set = new HashSet<int[]>(new SameHash());
Stopwatch clock = new Stopwatch();
clock.Start();
foreach (var item in points)
{
set.Add(item);
}
clock.Stop();
Console.WriteLine("Elements inserted: " + set.Count + "/" + toInsert);
Console.WriteLine("Time taken: " + clock.ElapsedMilliseconds);
}
public class SameHash : EqualityComparer<int[]>
{
public override bool Equals(int[] p1, int[] p2)
{
return p1[0] == p2[0] && p1[1] == p2[1];
}
public override int GetHashCode(int[] i)
{
return base.GetHashCode();
//return i[0] * 10000 + i[1];
//Notice that this is a very basic implementation of a HashCode function
}
}
}
}

The only way I found it possible was by creating a class MyPair instead of using an array (int[]) like you did. Notice that I used X*10000 + Y in the GetHashCode() function but you can change the constant value in order to get a better HashCode for every item or you can create you own. I just provided this one as a simple example and because is an easy way of having different hashCodes when the bounds for X and Y are relative small (less than the root of Int.MaxValue).
Here you have the working code:
using System;
using System.Collections.Generic;
using System.Linq;
namespace hash
{
public class MyPair
{
public int X { get; set; }
public int Y { get; set; }
public override int GetHashCode()
{
return X * 10000 + Y;
}
public override bool Equals(object obj)
{
MyPair other = obj as MyPair;
return X == other.X && Y == other.Y;
}
}
class Program
{
static void Main(string[] args)
{
HashSet<MyPair> hash = new HashSet<MyPair>();
MyPair one = new MyPair { X = 10, Y = 2 };
MyPair two = new MyPair { X = 1, Y = 24 };
MyPair three = new MyPair { X = 111, Y = 266 };
MyPair copyOfOne = new MyPair { X = 10, Y = 2 };
Console.WriteLine(hash.Add(one));
Console.WriteLine(hash.Add(two));
Console.WriteLine(hash.Add(three));
Console.WriteLine(hash.Add(copyOfOne));
}
}
}

fastest way for accessing double array as key in dictionary

I have a double[] array, i want to use it as key （not literally, but in the way that the key is matched when all the doubles in the double array need to be matched)
What is the fastest way to use the double[] array as key to dictionary?
Is it using
Dictionary<string, string> (convert double[] to a string)
or
anything else like converting it

Given that all key arrays will have the same length, either consider using a Tuple<,,, ... ,>, or use a structural equality comparer on the arrays.
With tuple:
var yourDidt = new Dictionary<Tuple<double, double, double>, string>();
yourDict.Add(Tuple.Create(3.14, 2.718, double.NaN), "da value");
string read = yourDict[Tuple.Create(3.14, 2.718, double.NaN)];
With (strongly typed version of) StructuralEqualityComparer:
class DoubleArrayStructuralEqualityComparer : EqualityComparer<double[]>
{
public override bool Equals(double[] x, double[] y)
{
return System.Collections.StructuralComparisons.StructuralEqualityComparer
.Equals(x, y);
}
public override int GetHashCode(double[] obj)
{
return System.Collections.StructuralComparisons.StructuralEqualityComparer
.GetHashCode(obj);
}
}
...
var yourDict = new Dictionary<double[], string>(
new DoubleArrayStructuralEqualityComparer());
yourDict.Add(new[] { 3.14, 2.718, double.NaN, }, "da value");
string read = yourDict[new[] { 3.14, 2.718, double.NaN, }];
Also consider the suggestion by Sergey Berezovskiy to create a custom class or (immutable!) struct to hold your set of doubles. In that way you can name your type and its members in a natural way that makes it more clear what you do. And your class/struct can easily be extended later on, if needed.

Thus all arrays have same length and each item in array have specific meaning, then create class which holds all items as properties with descriptive names. E.g. instead of double array with two items you can have class Point with properties X and Y. Then override Equals and GetHashCode of this class and use it as key (see What is the best algorithm for an overriding GetHashCode):
Dictionary<Point, string>
Benefits - instead of having array, you have data structure which makes its purpose clear. Instead of referencing items by indexes, you have nice named property names, which also make their purpose clear. And also speed - calculating hash code is fast. Compare:
double[] a = new [] { 12.5, 42 };
// getting first coordinate a[0];
Point a = new Point { X = 12.5, Y = 42 };
// getting first coordinate a.X

[Do not consider this a separate answer; this is an extension of #JeppeStigNielsen's answer]
I'd just like to point out that you make Jeppe's approach generic as follows:
public class StructuralEqualityComparer<T>: IEqualityComparer<T>
{
public bool Equals(T x, T y)
{
return StructuralComparisons.StructuralEqualityComparer.Equals(x, y);
}
public int GetHashCode(T obj)
{
return StructuralComparisons.StructuralEqualityComparer.GetHashCode(obj);
}
public static StructuralEqualityComparer<T> Default
{
get
{
StructuralEqualityComparer<T> comparer = _defaultComparer;
if (comparer == null)
{
comparer = new StructuralEqualityComparer<T>();
_defaultComparer = comparer;
}
return comparer;
}
}
private static StructuralEqualityComparer<T> _defaultComparer;
}
(From an original answer here: https://stackoverflow.com/a/5601068/106159)
Then you would declare the dictionary like this:
var yourDict = new Dictionary<double[], string>(new StructuralEqualityComparer<double[]>());
Note: It might be better to initialise _defaultComparer using Lazy<T>.
[EDIT]
It's possible that this might be faster; worth a try:
class DoubleArrayComparer: IEqualityComparer<double[]>
{
public bool Equals(double[] x, double[] y)
{
if (x == y)
return true;
if (x == null || y == null)
return false;
if (x.Length != y.Length)
return false;
for (int i = 0; i < x.Length; ++i)
if (x[i] != y[i])
return false;
return true;
}
public int GetHashCode(double[] data)
{
if (data == null)
return 0;
int result = 17;
foreach (var value in data)
result += result*23 + value.GetHashCode();
return result;
}
}
...
var yourDict = new Dictionary<double[], string>(new DoubleArrayComparer());

Ok this is what I found so far:
I input an entry (length 4 arrray) to the dictionary, and access it for 999999 times on my machine:
Dictionary<double[], string>(
new DoubleArrayStructuralEqualityComparer()); takes 1.75 seconds
Dictionary<Tuple<double...>,string> takes 0.85 seconds
The code below takes 0.1755285 seconds, which is the fastest now! (in line with the comment with Sergey.)
The fastest - The code of DoubleArrayComparer by Matthew Watson takes 0.15 seconds!
public class DoubleArray
{
private double[] d = null;
public DoubleArray(double[] d)
{
this.d = d;
}
public override bool Equals(object obj)
{
if (!(obj is DoubleArray)) return false;
DoubleArray dobj = (DoubleArray)obj;
if (dobj.d.Length != d.Length) return false;
for (int i = 0; i < d.Length; i++)
{
if (dobj.d[i] != d[i]) return false;
}
return true;
}
public override int GetHashCode()
{
unchecked // Overflow is fine, just wrap
{
int hash = 17;
for (int i = 0; i < d.Length;i++ )
{
hash = hash*23 + d[i].GetHashCode();
}
return hash;
}
}
}

What happens when ==, CompareTo(), and Equals() do not agree?

I have a program I wrote some years back to find "good" binary operators for bytes; byte A is left multiplied by byte B to yield byte C. The operator is defined as 256x256 byte matrix. A stripped down version of the class implementation is below.
Equals() is true IFF all 65536 bytes in the array are the same.
CompareTo() compares the Linearity of the operator into a continuum of more linear (bad for cryto) to less linear (good for crypto).
So it is possible for two instances, A and B, that both of the following are true:
A.Equals(B) = false
(A.ComparesTo(B) == 0) = true
My question is less: Is this a good idea? I know the answer is No, but given the large computational cost of measuring linearity and the narrow nature of my problem this design works. Also code similar to:
if (localMinimumOperator < globalMinimumOperator)
{
localMinimumOperator = globalMinimumOperator;
}
is easier for me to read.
My question is: What are the consequences of this divergence among: ==, CompareTo()== 0, and Equals()? or alternately:
Is there list of which LINQ extensions methods describing which extension use which interface (IEquatable or IComparable)?
Something more concise than this MSDN article on Enumerable?
For example:
IEnumerable<BinaryOperation> distinct = orgList.Distinct();
calls Equals(BinaryOperator) as per: Enumerable.Distinct<TSource> Method as does Contains(). I understand that Sort() and OrderBy() use calls to CompareTo().
But what about FindFirst() and BinarySearch()?
My example class:
using System;
using System.Collections.Generic;
using System.Linq;
namespace Jww05
{
public class BinaryOperation : IEquatable<BinaryOperation>, IComparable<BinaryOperation>
{
#region ClassMembers
public List<List<byte>> TruthTable
{
get
{
// I don't like giving out the underlying list if I help it
var retVal = new List<List<byte>>(OperatorDefintion);
return retVal;
}
}
// private data store for TruthTable
private List<List<byte>> OperatorDefintion { get; set; }
public BinaryOperation()
{
// initial state is the Identity operator
OperatorDefintion = new List<List<byte>>();
for (int i = 0; i < 256; i++)
{
var curRow = new List<byte>();
for (int j = 0; j < 256; j++)
{
curRow.Add((byte)(i + j));
}
OperatorDefintion.Add(curRow);
}
}
private long MeasureOperatorLinearity()
{
var diagonalOffsets = new byte[] { 255, 0, 1 };
/*
* Code that measures linearity in the original code used the Fast Walsh Hadamard Transform.
* That should go here, but it is removed because the FWHT is clutter for the purposes of this question.
*
* Since I needed a stub for this, I decided to exacerbate the differnece
* between CompareTo() == 0 and Equals()
* by returning an arbitrary int in lieu of the long CPU intensive Fast Walsh Hadamard Transform.
*
* If the matrices are identical on an element-by-element basis, then the Faux Linearity will be the the same.
* If the faux linearity (sum of terms on the main diagonal and corners) are the same, the underlying matrices could be different on an element-by-element basis.
*/
long fauxLinearityMeasure = 0;
for (var currRow = 0; currRow < OperatorDefintion.Count(); ++currRow)
{
fauxLinearityMeasure *= 5;
fauxLinearityMeasure = diagonalOffsets.Select(diagonalOffset => (byte)(currRow + diagonalOffset))
.Aggregate(fauxLinearityMeasure, (current, offestedIndex) => current + (OperatorDefintion[offestedIndex][currRow]));
}
return (int)fauxLinearityMeasure;
}
#endregion ClassMembers
#region ComparisonOperations
public int CompareTo(BinaryOperation other)
{
long otherLinearity = other.MeasureOperatorLinearity();
long thisLinearity = MeasureOperatorLinearity();
long linearityDiff = thisLinearity - otherLinearity;
// case the differnece of the linarity measures into {-1, 0, 1}
return (0 < linearityDiff) ? 1
: (0 > linearityDiff) ? -1
: 0;
}
public static bool operator >(BinaryOperation lhs, BinaryOperation rhs)
{
if (ReferenceEquals(null, lhs) ||
ReferenceEquals(null, rhs))
{
return false;
}
return (0 < lhs.CompareTo(rhs));
}
public static bool operator <(BinaryOperation lhs, BinaryOperation rhs)
{
if (ReferenceEquals(null, lhs) ||
ReferenceEquals(null, rhs))
{
return false;
}
return (0 > lhs.CompareTo(rhs));
}
public static bool operator <=(BinaryOperation lhs, BinaryOperation rhs)
{
if (ReferenceEquals(null, lhs) ||
ReferenceEquals(null, rhs))
{
return false;
}
// equals is cheap
if (lhs.Equals(rhs))
{
return true;
}
return (0 > lhs.CompareTo(rhs));
}
public static bool operator >=(BinaryOperation lhs, BinaryOperation rhs)
{
if (ReferenceEquals(null, lhs) ||
ReferenceEquals(null, rhs))
{
return false;
}
// equals is cheap
if (lhs.Equals(rhs))
{
return true;
}
return (0 < lhs.CompareTo(rhs));
}
#endregion ComparisonOperations
#region EqualityOperators
public bool Equals(BinaryOperation other)
{
if (ReferenceEquals(null, other))
{
return false;
}
var otherTruthTable = other.TruthTable;
var thisTruthTable = TruthTable;
var isEquals = true;
for (int currRow = 0; currRow < thisTruthTable.Count(); ++currRow)
{
isEquals = isEquals && thisTruthTable[currRow].SequenceEqual(otherTruthTable[currRow]);
}
return isEquals;
}
public override bool Equals(object obj)
{
return Equals(obj as BinaryOperation);
}
public override int GetHashCode()
{
return OperatorDefintion.SelectMany(currRow => currRow)
.Aggregate(1, (current, currByte) => current * 5 + currByte);
}
public static bool operator ==(BinaryOperation lhs, BinaryOperation rhs)
{
if (ReferenceEquals(null, lhs) ||
ReferenceEquals(null, rhs))
{
return false;
}
return (0 == lhs.CompareTo(rhs));
}
public static bool operator !=(BinaryOperation lhs, BinaryOperation rhs)
{
if (ReferenceEquals(null, lhs) ||
ReferenceEquals(null, rhs))
{
return false;
}
return (0 != lhs.CompareTo(rhs));
}
#endregion EqualityOperators
}
}

What are the consequences of this divergence among: ==, CompareTo()== 0, and Equals()?
Someone looking at your code in the future will truly hate you.
or alternately: Is there list of which linq extensions methods describing which extension use which interface (IEquitable or IComparable)?
I think that you've found most of it by yourself. A good rule of thumb is that usually there is nothing surprising in what interface is used by which LINQ function (no surprises is one of features of good design - unlike yours). For example: it's quite obvious that to sort elements it is necessary to know in which particular order should elements go, equality/inequality alone are not sufficient for this. BinarySearch also needs to know "which way to go" during search - if element is larger than current it re-curses into upper part of sorted array, if smaller it goes into lower. Again: obviously it needs IComparable. For Distinct Equals and GetHashCode suffice - sorting is not needed to determine a set of unique elements. And so on.

C# Compiler "Optimize code" : disable on a code fragment only

I have a C# code which is working good when the "optimize code" option is off, but fails otherwise. Is there any function or class attribute which can prevent the optimisation of a function or class, but let the compiler optimize the others ?
(I tried unsafe or MethodImpl, but without success)
Thanks
Edit :
I have done some more test...
The code is like this :
double arg = (Math.PI / 2d - Math.Atan2(a, d));
With a = 1 and d = 0, arg should be 0.
Thid code is a function which is called by Excel via ExcelDNA.
Calling an identical code from an optimized console app : OK
Calling this code from Excel without optimization : OK
Calling this code from Excel with optimization : Not OK, arg == 0 is false (instead arg is a very small value near 0, but not 0)
Same result with [MethodImpl(MethodImplOptions.NoOptimization)] before the called function.

This is very likely to do with the floating point mode which Excel likely has set - meaning that your program is calculating floating points slightly different because of the program (Excel) hosting your assembly (DLL). This might impact how your results are calculated, or how/what values are automatically coerced to zero.
To be absolutely sure you are not going to run into issues with different floating point modes and/or errors you should check for equality rather by checking if the values are very close together. This is not really a hack.
public class AlmostDoubleComparer : IComparer<double>
{
public static readonly AlmostDoubleComparer Default = new AlmostDoubleComparer();
public const double Epsilon = double.Epsilon * 64d; // 0.{322 zeroes}316
public static bool IsZero(double x)
{
return Compare(x, 0) == 0;
}
public static int Compare(double x, double y)
{
// Very important that cmp(x, y) == cmp(y, x)
if (Double.IsNaN(x) || Double.IsNaN(y))
return 1;
if (Double.IsInfinity(x) || Double.IsInfinity(y))
return 1;
var absX = Math.Abs(x);
var absY = Math.Abs(y);
var diff = absX > absY ? absX - absY : absY - absX;
if (diff < Epsilon)
return 0;
if (x < y)
return -1;
else
return 1;
}
int IComparer<double>.Compare(double x, double y)
{
return Compare(x, y);
}
}
// E.g.
double arg = (Math.PI / 2d - Math.Atan2(a, d));
if (AlmostDoubleComparer.IsZero(arg))
// Regard it as zero.
I also ported the re-interpret integer comparison, in case you find that more suitable (it deals with larger values more consistently).
public class AlmostDoubleComparer : IComparer<double>
{
public static readonly AlmostDoubleComparer Default = new AlmostDoubleComparer();
public const double MaxUnitsInTheLastPlace = 3;
public static bool IsZero(double x)
{
return Compare(x, 0) == 0;
}
public static int Compare(double x, double y)
{
// Very important that cmp(x, y) == cmp(y, x)
if (Double.IsNaN(x) || Double.IsNaN(y))
return 1;
if (Double.IsInfinity(x) || Double.IsInfinity(y))
return 1;
var ix = DoubleInt64.Reinterpret(x);
var iy = DoubleInt64.Reinterpret(y);
var diff = Math.Abs(ix - iy);
if (diff < MaxUnitsInTheLastPlace)
return 0;
if (ix < iy)
return -1;
else
return 1;
}
int IComparer<double>.Compare(double x, double y)
{
return Compare(x, y);
}
}
[StructLayout(LayoutKind.Explicit)]
public struct DoubleInt64
{
[FieldOffset(0)]
private double _double;
[FieldOffset(0)]
private long _int64;
private DoubleInt64(long value)
{
_double = 0d;
_int64 = value;
}
private DoubleInt64(double value)
{
_int64 = 0;
_double = value;
}
public static double Reinterpret(long value)
{
return new DoubleInt64(value)._double;
}
public static long Reinterpret(double value)
{
return new DoubleInt64(value)._int64;
}
}
Alternatively you could try and NGen the assembly and see if you can work around the either the mode Excel has, or how it is hosting the CLR.

That is what you get when working with floating point datatypes. You don't get exactly 0, but a very close value, since a double has limited precision and not every value can be represented and sometimes those tiny precision errors add up. You either need to expect that (check that the value is close enough to 0).