Develop Reference

Array as Dictionary key gives a lot of collisions

I need to use a list of numbers (longs) as a Dictionary key in order to do some group calculations on them.
When using the long array as a key directly, I get a lot of collisions. If I use string.Join(",", myLongs) as a key, it works as I would expect it to, but that's much, much slower (because the hash is more complicated, I assume).
Here's an example demonstrating my problem:
Console.WriteLine("Int32");
Console.WriteLine(new[] { 1, 2, 3, 0}.GetHashCode());
Console.WriteLine(new[] { 1, 2, 3, 0 }.GetHashCode());
Console.WriteLine("String");
Console.WriteLine(string.Join(",", new[] { 1, 2, 3, 0}).GetHashCode());
Console.WriteLine(string.Join(",", new[] { 1, 2, 3, 0 }).GetHashCode());
Output:
Int32
43124074
51601393
String
406954194
406954194
As you can see, the arrays return a different hash.
Is there any way of getting the performance of the long array hash, but the uniqeness of the string hash?
See my own answer below for a performance comparison of all the suggestions.
About the potential duplicate -- that question has a lot of useful information, but as this question was primarily about finding high performance alternatives, I think it still provides some useful solutions that are not mentioned there.

That the first one is different is actually good. Arrays are a reference type and luckily they are using the reference (somehow) during hash generation. I would guess that is something like the Pointer that is used on machine code level, or some Garbage Colletor level value. One of the things you have no influence on but is copied if you assign the same instance to a new reference variable.
In the 2nd case you get the hash value on a string consisting of "," and whatever (new[] { 1, 2, 3, 0 }).ToString(); should return. The default is something like teh class name, so of course in both cases they will be the same. And of course string has all those funny special rules like "compares like a value type" and "string interning", so the hash should be the same.

Another alternative is to leverage the lesser known IEqualityComparer to implement your own hash and equality comparisons. There are some notes you'll need to observe about building good hashes, and it's generally not good practice to have editable data in your keys, as it'll introduce instability should the keys ever change, but it would certainly be more performant than using string joins.
public class ArrayKeyComparer : IEqualityComparer<int[]>
{
public bool Equals(int[] x, int[] y)
{
return x == null || y == null
? x == null && y == null
: x.SequenceEqual(y);
}
public int GetHashCode(int[] obj)
{
var seed = 0;
if(obj != null)
foreach (int i in obj)
seed %= i.GetHashCode();
return seed;
}
}
Note that this still may not be as performant as a tuple, since it's still iterating the array rather than being able to take a more constant expression.

Your strings are returning the same hash codes for the same strings correctly because string.GetHashCode() is implemented that way.
The implementation of int[].GetHashCode() does something with its memory address to return the hash code, so arrays with identical contents will nevertheless return different hash codes.
So that's why your arrays with identical contents are returning different hash codes.
Rather than using an array directly as a key, you should consider writing a wrapper class for an array that will provide a proper hash code.
The main disadvantage with this is that it will be an O(N) operation to compute the hash code (it has to be - otherwise it wouldn't represent all the data in the array).
Fortunately you can cache the hash code so it's only computed once.
Another major problem with using a mutable array for a hash code is that if you change the contents of the array after using it for the key of a hashing container such as Dictionary, you will break the container.
Ideally you would only use this kind of hashing for arrays that are never changed.
Bearing all that in mind, a simple wrapper would look like this:
public sealed class IntArrayKey
{
public IntArrayKey(int[] array)
{
Array = array;
_hashCode = hashCode();
}
public int[] Array { get; }
public override int GetHashCode()
{
return _hashCode;
}
int hashCode()
{
int result = 17;
unchecked
{
foreach (var i in Array)
{
result = result * 23 + i;
}
}
return result;
}
readonly int _hashCode;
}
You can use that in place of the actual arrays for more sensible hash code generation.
As per the comments below, here's a version of the class that:
Makes a defensive copy of the array so that it cannot be modified.
Implements equality operators.
Exposes the underlying array as a read-only list, so callers can access its contents but cannot break its hash code.
Code:
public sealed class IntArrayKey: IEquatable<IntArrayKey>
{
public IntArrayKey(IEnumerable<int> sequence)
{
_array = sequence.ToArray();
_hashCode = hashCode();
Array = new ReadOnlyCollection<int>(_array);
}
public bool Equals(IntArrayKey other)
{
if (other is null)
return false;
if (ReferenceEquals(this, other))
return true;
return _hashCode == other._hashCode && equals(other.Array);
}
public override bool Equals(object obj)
{
return ReferenceEquals(this, obj) || obj is IntArrayKey other && Equals(other);
}
public static bool operator == (IntArrayKey left, IntArrayKey right)
{
return Equals(left, right);
}
public static bool operator != (IntArrayKey left, IntArrayKey right)
{
return !Equals(left, right);
}
public IReadOnlyList<int> Array { get; }
public override int GetHashCode()
{
return _hashCode;
}
bool equals(IReadOnlyList<int> other) // other cannot be null.
{
if (_array.Length != other.Count)
return false;
for (int i = 0; i < _array.Length; ++i)
if (_array[i] != other[i])
return false;
return true;
}
int hashCode()
{
int result = 17;
unchecked
{
foreach (var i in _array)
{
result = result * 23 + i;
}
}
return result;
}
readonly int _hashCode;
readonly int[] _array;
}
If you wanted to use the above class without the overhead of making a defensive copy of the array, you can change the constructor to:
public IntArrayKey(int[] array)
{
_array = array;
_hashCode = hashCode();
Array = new ReadOnlyCollection<int>(_array);
}

If you know the length of the arrays you're using, you could use a Tuple.
Console.WriteLine("Tuple");
Console.WriteLine(Tuple.Create(1, 2, 3, 0).GetHashCode());
Console.WriteLine(Tuple.Create(1, 2, 3, 0).GetHashCode());
Outputs
Tuple
1248
1248

I took all the suggestions from this question and the similar byte[].GetHashCode() question, and made a simple performance test.
The suggestions are as follows:
int[] as key (original attempt -- does not work at all, included as a benchmark)
string as key (original solution -- works, but slow)
Tuple as key (suggested by David)
ValueTuple as key (inspired by the Tuple)
Direct int[] hash as key
IntArrayKey (suggested by Matthew Watson)
int[] as key with Skeet's IEqualityComparer
int[] as key with David's IEqualityComparer
I generated a List containing one million int[]-arrays of length 7 containing random numbers between 100 000 and 999 999 (which is an approximation of my current use case). Then I duplicated the first 100 000 of these arrays, so that there are 900 000 unique arrays, and 100 000 that are listed twice (to force collisions).
For each solution, I enumerated the list, and added the keys to a Dictionary, OR incremented the Value if the key already existed. Then I printed how many keys had a Value more than 1**, and how much time it took.
The results are as follows (ordered from best to worst):
Algorithm Works? Time usage
NonGenericSkeetEquality YES 392 ms
SkeetEquality YES 422 ms
ValueTuple YES 521 ms
QuickIntArrayKey YES 747 ms
IntArrayKey YES 972 ms
Tuple YES 1 609 ms
string YES 2 291 ms
DavidEquality YES 1 139 200 ms ***
int[] NO 336 ms
IntHash NO 386 ms
The Skeet IEqualityComparer is only slightly slower than using the int[] as key directly, with the huge advantage that it actually works, so I'll use that.
** I'm aware that this is not a completely fool proof solution, as I could theoretically get the expected number of collisions without it actually being the collisions I expected, but having run the test a lot of times, I'm fairly certain I don't.
*** Did not finish, probably due to poor hashing algorithm and a lot of equality checks. Had to reduce the number of arrays to 10 000, then multiply the time usage by 100 to compare with the others.

C# - Creating a recursive function to calculate the sum of a list. Is it possible using only the list as the only parameter?

So in my attempt to start learning c# one challenge I've come across is to create a recursive function that will calculate the sum of a list. I'm wondering if it's possible to do this using a list as the only argument of the function? Or would I need to apply an index size as well to work through the list?
int addRecursively(List<int> numList)
{
int total = numList[0];
if (numList.Count > 1)
{
numList.RemoveAt(0);
return total += addRecursively(numList);
}
Console.WriteLine(total);
return total;
}
List<int> numbers = new<List<int> {1,2,3,4,5,6,7,8};
addRecursively(numbers); //returns only the last element of whichever list I enter.
I was hoping by assigning the total to the first index of the list before deleting the first index of the list that when passed into the next instance of the function the index of each element in the list would move down one, allowing me to get each value in the list and totalling them up. However using the function will only ever return the last element of whichever list of integers I enter.
My thought process came from arrays and the idea of the shift method on an array in JS, removing the first element and bringing the whole thing down.
Am I attempting something stupid here? Is there another similar method I should be using or would I be better off simply including a list size as another parameter?
Thanks for your time

So in my attempt to start learning c# one challenge I've come across is to create a recursive function that will calculate the sum of a list. I'm wondering if it's possible to do this using a list as the only argument of the function? Or would I need to apply an index size as well to work through the list?
That's a great exercise for a beginner. However, you would never, ever do this with a List<int> in a realistic program. First, because you'd simply call .Sum() on it. But that's a cop-out; someone had to write Sum, and that person could be you.
The reason you would never do this recursively is List<T> is not a recursive data structure. As you note, every time you recurse there has to be something different. If there is not something different then you have an unbounded recursion!
That means you have to change one of the arguments, either by mutating it, if it is a reference type, or passing a different argument. Neither is correct in this case where the argument is a list.
For a list, you never want to mutate the list, by removing items, say. You don't own that list. The caller owns the list and it is rude to mutate it on them. When I call your method to sum a list, I don't want the list to be emptied; I might want to use it for something else.
And for a list, you never want to pass a different list in a recursion because constructing the new list from the old list is very expensive.
(There is also the issue of deep recursion; presumably we wish to sum lists of more than a thousand numbers, but that will eat up all the stack space if you go with a recursive solution; C# is not a guaranteed-tail-recursive language like F# is. However, for learning purposes let's ignore this issue and assume we are dealing with only small lists.)
Since both of the techniques for avoiding unbounded recursions are inapplicable, you must not write recursive algorithms on List<T> (or, as you note, you must pass an auxiliary parameter such as an index, and that's the thing you change). But your exercise is still valid; we just have to make it a better exercise by asking "what would we have to change to make a list that is amenable to recursion?"
We need to change two things: (1) make the list immutable, and (2) make it a recursively defined data structure. If it is immutable then you cannot change the caller's data by accident; it's unchangeable. And if it is a recursively defined data structure then there is a natural way to do recursion on it that is cheap.
So this is your new exercise:
An ImmutableList is either (1) empty, or (2) a single integer, called the "head", and an immutable list, called the "tail". Implement these in the manner of your choosing. (Abstract base class, interface implemented by multiple classes, single class that does the whole thing, whatever you think is best. Pay particular attention to the constructors.)
ImmutableList has three public read-only properties: bool IsEmpty, int Head and ImmutableList Tail. Implement them.
Now we can define int Sum(ImmutableList) as a recursive method: the base case is the sum of an empty list is zero; the inductive case is the sum of a non-empty list is the head plus the sum of the tail. Implement it; can you do it as a single line of code?
You will learn much more about C# and programming in a functional style with this exercise. Use iterative algorithms on List<T>, always; that is what it was designed for. Use recursion on data structures that are designed for recursion.
Bonus exercises:
Write Sum as an extension method, so that you can call myImmutableList.Sum().
Sum is a special case of an operation called Aggregate. It returns an integer, and takes three parameters: an immutable list, an integer called the accumulator, and a Func<int, int, int>. If the list is empty, the result is the accumulator. Otherwise, the result is the recursion on the tail and calling the function on the head and the accumulator. Write a recursive Aggregate; if you've done it correctly then int Sum(ImmutableList items) => Aggregate(items, 0, (acc, item) => acc + item); should be a correct implementation of Sum.
Genericize ImmutableList to ImmutableList<T>; genericize Aggregate to Aggregate<T, R> where T is the list element type and R is the accumulator type.

Try this way:
int addRecursively(List<int> lst)
{
if(lst.Count() == 0) return 0;
return lst.Take(1).First() + addRecursively(lst.Skip(1).ToList());
}

one more example:
static public int RecursiveSum(List<int> ints)
{
int nextIndex = 0;
if(ints.Count == 0)
return 0;
return ints[0] + RecursiveSum(ints.GetRange(++nextIndex, ints.Count - 1));
}

These are some ways to get the sum of integers in a list.
You don't need a recursive method, it spends more system resources when it isn't needed.
class Program
{
static void Main(string[] args)
{
List<int> numbers = new List<int>() { 1, 2, 3, 4, 5 };
int sum1 = numbers.Sum();
int sum2 = GetSum2(numbers);
int sum3 = GetSum3(numbers);
int sum4 = GetSum4(numbers);
}
private static int GetSum2(List<int> numbers)
{
int total = 0;
foreach (int number in numbers)
{
total += number;
}
return total;
}
private static int GetSum3(List<int> numbers)
{
int total = 0;
for (int i = 0; i < numbers.Count; i++)
{
total += numbers[i];
}
return total;
}
private static int GetSum4(List<int> numbers)
{
int total = 0;
numbers.ForEach((number) =>
{
total += number;
});
return total;
}
}

Sort List<T> by reference

I would like to sort a List<T> by reference. Why? I want to compare several lists and find which elements are unique to a given list. Sorting them first in some canonical fashion and then stepping through all lists simultaneously seems a good way to do this. This essentially means that I need to make aa IComparer<T> that returns a non-zero integer for different references (and is consistent). The order does have to mean anything, but just provide a fixed order for all T for at least the time it takes me to get the job done.
GetHashCode can return the same value for two different objects on a 64-bit system, so that is out as a way to do a comparison.
IntPtr from AddrOfPinnedObject() might work since it has the correct size, but I have to pin the object for the duration of the process.
Making a Dictionary<T, long> where the long is issued first-come first-served is what I'm going with now, but I want to know if there are better options. The hash lookup seems unnecessary.
What is the best way to define ReferenceComparer<T> : IComparer<T> to give me that fixed ordering?
Here is the dictionary implementation:
public class ReferenceComparer<T> : IComparer<T>
{
private long nextAvailable = 0;
private readonly Dictionary<T, long> objectToLong = new Dictionary<T, long>(new AsObjectEqualityComparer<T>());
public int Compare(T x, T y)
{
long xLong;
if (!objectToLong.TryGetValue(x, out xLong))
{
xLong = nextAvailable;
objectToLong[x] = xLong;
nextAvailable = nextAvailable + 1;
}
long yLong;
if (!objectToLong.TryGetValue(y, out yLong))
{
yLong = nextAvailable;
objectToLong[y] = yLong;
nextAvailable = nextAvailable + 1;
}
return xLong.CompareTo(yLong);
}
}

best data structure for storing large number of numeric fields

I am working with a class, say Widget, that has a large number of numeric real world attributes (eg, height, length, weight, cost, etc.). There are different types of widgets (sprockets, cogs, etc.), but each widget shares the exact same attributes (the values will be different by widget, of course, but they all have a weight, weight, etc.). I have 1,000s of each type of widget (1,000 cogs, 1,000 sprockets, etc.)
I need to perform a lot of calculations on these attributes (say calculating the weighted average of the attributes for 1000s of different widgets). For the weighted averages, I have different weights for each widget type (ie, I may care more about length for sprockets than for cogs).
Right now, I am storing all the attributes in a Dictionary< string, double> within each widget (the widgets have an enum that specifies their type: cog, sprocket, etc.). I then have some calculator classes that store weights for each attribute as a Dictionary< WidgetType, Dictionary< string, double >>. To calculate the weighted average for each widget, I simply iterate through its attribute dictionary keys like:
double weightedAvg = 0.0;
foreach (string attibuteName in widget.Attributes.Keys)
{
double attributeValue = widget.Attributes[attributeName];
double attributeWeight = calculator.Weights[widget.Type][attributeName];
weightedAvg += (attributeValue * attributeWeight);
}
So this works fine and is pretty readable and easy to maintain, but is very slow for 1000s of widgets based on some profiling. My universe of attribute names is known and will not change during the life of the application, so I am wondering what some better options are. The few I can think of:
1) Store attribute values and weights in double []s. I think this is probably the most efficient option, but then I need to make sure the arrays are always stored in the correct order between widgets and calculators. This also decouples the data from the metadata so I will need to store an array (?) somewhere that maps between the attribute names and the index into double [] of attribute values and weights.
2) Store attribute values and weights in immutable structs. I like this option because I don't have to worry about the ordering and the data is "self documenting". But is there an easy way to loop over these attributes in code? I have almost 100 attributes, so I don't want to hardcode all those in the code. I can use reflection, but I worry that this will cause even a larger penalty hit since I am looping over so many widgets and will have to use reflection on each one.
Any other alternatives?

Three possibilities come immediately to mind. The first, which I think you rejected too readily, is to have individual fields in your class. That is, individual double values named height, length, weight, cost, etc. You're right that it would be more code to do the calculations, but you wouldn't have the indirection of dictionary lookup.
Second is to ditch the dictionary in favor of an array. So rather than a Dictionary<string, double>, you'd just have a double[]. Again, I think you rejected this too quickly. You can easily replace the string dictionary keys with an enumeration. So you'd have:
enum WidgetProperty
{
First = 0,
Height = 0,
Length = 1,
Weight = 2,
Cost = 3,
...
Last = 100
}
Given that and an array of double, you can easily go through all of the values for each instance:
for (int i = (int)WidgetProperty.First; i < (int)WidgetProperty.Last; ++i)
{
double attributeValue = widget.Attributes[i];
double attributeWeight = calculator.Weights[widget.Type][i];
weightedAvg += (attributeValue * attributeWeight);
}
Direct array access is going to be significantly faster than accessing a dictionary by string.
Finally, you can optimize your dictionary access a little bit. Rather than doing a foreach on the keys and then doing a dictionary lookup, do a foreach on the dictionary itself:
foreach (KeyValuePair<string, double> kvp in widget.Attributes)
{
double attributeValue = kvp.Value;
double attributeWeight = calculator.Weights[widget.Type][kvp.Key];
weightedAvg += (attributeValue * attributeWeight);
}

To calculate weighted averages without looping or reflection, one way would be to calculate the weighted average of the individual attributes and store them in some place. This should happen while you are creating instance of the widget. Following is a sample code which needs to be modified to your needs.
Also, for further processing of the the widgets themselves, you can use data parallelism. see my other response in this thread.
public enum WidgetType { }
public class Claculator { }
public class WeightStore
{
static Dictionary<int, double> widgetWeightedAvg = new Dictionary<int, double>();
public static void AttWeightedAvgAvailable(double attwightedAvg, int widgetid)
{
if (widgetWeightedAvg.Keys.Contains(widgetid))
widgetWeightedAvg[widgetid] += attwightedAvg;
else
widgetWeightedAvg[widgetid] = attwightedAvg;
}
}
public class WidgetAttribute
{
public string Name { get; }
public double Value { get; }
public WidgetAttribute(string name, double value, WidgetType type, int widgetId)
{
Name = name;
Value = value;
double attWeight = Calculator.Weights[type][name];
WeightStore.AttWeightedAvgAvailable(Value*attWeight, widgetId);
}
}
public class CogWdiget
{
public int Id { get; }
public WidgetAttribute height { get; set; }
public WidgetAttribute wight { get; set; }
}
public class Client
{
public void BuildCogWidgets()
{
CogWdiget widget = new CogWdiget();
widget.Id = 1;
widget.height = new WidgetAttribute("height", 12.22, 1);
}
}

As it is always the case with data normalization, is that choosing your normalization level determines a good part of the performance. It looks like you would have to go from your current model to another model or a mix.
Better performance for your scenario is possible when you do not process this with the C# side, but with the database instead. You then get the benefit of indexes, no data transfer except the wanted result, plus 100000s of man hours already spent on performance optimization.

Use Data Parallelism supported by the .net 4 and above.
https://msdn.microsoft.com/en-us/library/dd537608(v=vs.110).aspx
An excerpt from the above link
When a parallel loop runs, the TPL partitions the data source so that the loop can operate on multiple parts concurrently. Behind the scenes, the Task Scheduler partitions the task based on system resources and workload. When possible, the scheduler redistributes work among multiple threads and processors if the workload becomes unbalanced

Is there a class in C# to handle a couple of INT (range of 2 INT- 1-10)

I am quite new to C# and I was wondering if there is a Class or a data structure or the best way to handle the following requirement...
I need to handle a COUPLE of int that represent a range of data (eg. 1 - 10 or 5-245) and I need a method to verify if an Int value is contained in the range...
I believe that in C# there is a class built in the framework to handle my requirement...
what I need to do is to verify if an INT (eg. 5) is contained in the range of values Eg (1-10) ...
in the case that I should discover that there is not a class to handle it, I was thinking to go with a Struct that contain the 2 numbers and make my own Contain method to test if 5 is contained in the range 1-10)

in the case that I should discover that there is not a class to handle
it, I was thinking to go with a Struct that contain the 2 numbers and
make my own Contain method to test if 5 is contained in the range
1-10)
That's actually a great idea as there's no built-in class for your scenario in the BCL.

You're looking for a range type; the .Net framework does not include one.
You should make an immutable (!) Int32Range struct, as you suggested.
You may want to implement IEnumerable<int> to allow users to easily loop through the numbers in the range.
You need to decide whether each bound should be inclusive or exclusive.
[Start, End) is probably the most obvious choice.
Whatever you choose, you should document it clearly in the XML comments.

Nothing exists that meets your requirements exactly.
Assuming I understood you correctly, the class is pretty simple to write.
class Range
{
public int Low {get; set;}
public int High {get; set;}
public bool InRange(int val) { return val >= Low && val <= High; }
}
A Tuple<int,int> would get you part of the way but you'd have to add an extension method to get the extra behavior. The downside is that the lower- and upper-bounds are implicitly Item1 and Item2 which could be confusing.
// written off-the-cuff, may not compile
public static class TupleExtension
{
public static bool InRange(Tuple<int, int> this, int queryFor)
{
return this.Item1 >= queryFor && this.Item2 <= queryFor;
}
}

You could create an extension if you want to avoid making a new type:
public static class Extensions
{
public static bool IsInRange(this int value, int min, int max)
{
return value >= min && value <= max;
}
}
Then you could do something like:
if(!value.IsInRange(5, 545))
throw new Exception("Value is out of range.");

i think you can do that with an array.
some nice examples and explanation can be found here:
http://www.dotnetperls.com/int-array

Nothing built in AFAIK, but (depending on the size of the range) an Enumerable.Range would work (but be less than optimal, as you're really storing every value in the range, not just the endpoints). It does allow you to use the LINQ methods (including Enumerable.Contains), though - which may come in handy.
const int START = 5;
const int END = 245;
var r = Enumerable.Range(START, (END - START)); // 2nd param is # of integers
return r.Contains(100);
Personally, I'd probably go ahead and write the class, since it's fairly simple (and you can always expose an IEnumerable<int> iterator via Enumerable.Range if you want to do LINQ over it)