Develop Reference

best data structure for storing large number of numeric fields

I am working with a class, say Widget, that has a large number of numeric real world attributes (eg, height, length, weight, cost, etc.). There are different types of widgets (sprockets, cogs, etc.), but each widget shares the exact same attributes (the values will be different by widget, of course, but they all have a weight, weight, etc.). I have 1,000s of each type of widget (1,000 cogs, 1,000 sprockets, etc.)
I need to perform a lot of calculations on these attributes (say calculating the weighted average of the attributes for 1000s of different widgets). For the weighted averages, I have different weights for each widget type (ie, I may care more about length for sprockets than for cogs).
Right now, I am storing all the attributes in a Dictionary< string, double> within each widget (the widgets have an enum that specifies their type: cog, sprocket, etc.). I then have some calculator classes that store weights for each attribute as a Dictionary< WidgetType, Dictionary< string, double >>. To calculate the weighted average for each widget, I simply iterate through its attribute dictionary keys like:
double weightedAvg = 0.0;
foreach (string attibuteName in widget.Attributes.Keys)
{
double attributeValue = widget.Attributes[attributeName];
double attributeWeight = calculator.Weights[widget.Type][attributeName];
weightedAvg += (attributeValue * attributeWeight);
}
So this works fine and is pretty readable and easy to maintain, but is very slow for 1000s of widgets based on some profiling. My universe of attribute names is known and will not change during the life of the application, so I am wondering what some better options are. The few I can think of:
1) Store attribute values and weights in double []s. I think this is probably the most efficient option, but then I need to make sure the arrays are always stored in the correct order between widgets and calculators. This also decouples the data from the metadata so I will need to store an array (?) somewhere that maps between the attribute names and the index into double [] of attribute values and weights.
2) Store attribute values and weights in immutable structs. I like this option because I don't have to worry about the ordering and the data is "self documenting". But is there an easy way to loop over these attributes in code? I have almost 100 attributes, so I don't want to hardcode all those in the code. I can use reflection, but I worry that this will cause even a larger penalty hit since I am looping over so many widgets and will have to use reflection on each one.
Any other alternatives?

Three possibilities come immediately to mind. The first, which I think you rejected too readily, is to have individual fields in your class. That is, individual double values named height, length, weight, cost, etc. You're right that it would be more code to do the calculations, but you wouldn't have the indirection of dictionary lookup.
Second is to ditch the dictionary in favor of an array. So rather than a Dictionary<string, double>, you'd just have a double[]. Again, I think you rejected this too quickly. You can easily replace the string dictionary keys with an enumeration. So you'd have:
enum WidgetProperty
{
First = 0,
Height = 0,
Length = 1,
Weight = 2,
Cost = 3,
...
Last = 100
}
Given that and an array of double, you can easily go through all of the values for each instance:
for (int i = (int)WidgetProperty.First; i < (int)WidgetProperty.Last; ++i)
{
double attributeValue = widget.Attributes[i];
double attributeWeight = calculator.Weights[widget.Type][i];
weightedAvg += (attributeValue * attributeWeight);
}
Direct array access is going to be significantly faster than accessing a dictionary by string.
Finally, you can optimize your dictionary access a little bit. Rather than doing a foreach on the keys and then doing a dictionary lookup, do a foreach on the dictionary itself:
foreach (KeyValuePair<string, double> kvp in widget.Attributes)
{
double attributeValue = kvp.Value;
double attributeWeight = calculator.Weights[widget.Type][kvp.Key];
weightedAvg += (attributeValue * attributeWeight);
}

To calculate weighted averages without looping or reflection, one way would be to calculate the weighted average of the individual attributes and store them in some place. This should happen while you are creating instance of the widget. Following is a sample code which needs to be modified to your needs.
Also, for further processing of the the widgets themselves, you can use data parallelism. see my other response in this thread.
public enum WidgetType { }
public class Claculator { }
public class WeightStore
{
static Dictionary<int, double> widgetWeightedAvg = new Dictionary<int, double>();
public static void AttWeightedAvgAvailable(double attwightedAvg, int widgetid)
{
if (widgetWeightedAvg.Keys.Contains(widgetid))
widgetWeightedAvg[widgetid] += attwightedAvg;
else
widgetWeightedAvg[widgetid] = attwightedAvg;
}
}
public class WidgetAttribute
{
public string Name { get; }
public double Value { get; }
public WidgetAttribute(string name, double value, WidgetType type, int widgetId)
{
Name = name;
Value = value;
double attWeight = Calculator.Weights[type][name];
WeightStore.AttWeightedAvgAvailable(Value*attWeight, widgetId);
}
}
public class CogWdiget
{
public int Id { get; }
public WidgetAttribute height { get; set; }
public WidgetAttribute wight { get; set; }
}
public class Client
{
public void BuildCogWidgets()
{
CogWdiget widget = new CogWdiget();
widget.Id = 1;
widget.height = new WidgetAttribute("height", 12.22, 1);
}
}

As it is always the case with data normalization, is that choosing your normalization level determines a good part of the performance. It looks like you would have to go from your current model to another model or a mix.
Better performance for your scenario is possible when you do not process this with the C# side, but with the database instead. You then get the benefit of indexes, no data transfer except the wanted result, plus 100000s of man hours already spent on performance optimization.

Use Data Parallelism supported by the .net 4 and above.
https://msdn.microsoft.com/en-us/library/dd537608(v=vs.110).aspx
An excerpt from the above link
When a parallel loop runs, the TPL partitions the data source so that the loop can operate on multiple parts concurrently. Behind the scenes, the Task Scheduler partitions the task based on system resources and workload. When possible, the scheduler redistributes work among multiple threads and processors if the workload becomes unbalanced

c# Best type/collection/list/dataset to handle super large data (csv/tab files)

I am building one WPF (MVVM) app that handles really large csv files. We are talking about 1GB to 10GB.
I open the file and parse it with File.ReadLines into a List of following class:
public class FileLine
{
public DateTime Time { get; set; }
public string Message { get; set; } //Usually around 256 characters
public string Info1 { get; set; } //Exact 56 characters
public string Info2 { get; set; } //Exact 4 characters
//and so on
}
... then I do all sort of data manipulation, queries, charts... you name it... everything using Linq.
We are testing a 1.8GB file and when it is opened, the process takes around 2GB of memory.
Eventually, when my customer needs to open his 10GB file it will be impossible, because it is going to take 12GB+ of Memory.
What is the best type/collection/list/dataset to this kind of work?

When i've had to do something like this before I handled it by having a container object that held a list of dictionaries. At the time I thought the limit would/should be 2^32 number of elements, but an exception for exceeding the collection was thrown well before getting 2^32 elements and still had many GB of ram left. Say you want a Dictionary, something like this should work until you really do exhaust all physical and virtual memory... A possible solution for you follows... I remember when I worked on this a few years ago the server actually had 512Gb of ram, I'm sure they have ones with more now... Anyway that's a separate story.
public class MyHugeDictionary
{
List<Dictionary<typea, typeb> allDict= null;
Dictionary<typea, typeb> currDictionary ;
MyHugeDictiionary()
{
allDict = new List<Dictionary<typea, typeb>();
currDictionary = new Dictionary<typea, typeb);
allDict.Add(currDictionary);
}
public bool ItemExists( typea, typeb)
{
foreach( KeyValue<Dictionary<typea, typeb> kv in allDict)
{
if( kv.ContainsKey(typea) )
{
return true;
}
}
return false;
}
public Add( typea a, typeb b)
{
try
{
if( !ItemExist( tyepa, typeb) ) // find if items is in any other dictionary first
{
currDictionary.Add( a, b) ;
}
else { // handle dups... ; }
}
catch( CollectionSizeError x) // look-up for actual exception
{
currDictionary = CreateDictiionary();
allDict.Add( currDictionary ) ;
currDictionary.Add( a,b);
}
catch( OutOfMemory y) // look-up for actual exception
{
// oops game over for real now :(
}
}
}

After some discussion the best thing is to read the file, process it, and dispose all the rest, sticking only with the result.
Another possibility was to use database, but it would add too much complexity, although it is possible.

See this:
https://github.com/aumcode/nfx/tree/master/Source/NFX/ApplicationModel/Pile
https://www.infoq.com/articles/Big-Memory-Part-3
You can store whatever you want - no pauses.
The problem with large collections is:
a. They are not really designed to hold very many entries (i.e. Dictionary never shrinks back to zero size)
b. You get GC stalls/pauses when you have too many objects
see the links above - what we did is "hiding" of data from GC as described in the article. This way you can store millions of objects using LocalCache class as a dictionary.
For large memory apps in net - remember to enable 64 bit and set GC to SERVER mode in your app config file

How to implement a SearchByID?

Good afternoon all!
As a part of getting a better grip on some of the most aspects of object based programming, I've started to attempt something far larger than I have done in the past. Hereby I'm trying to learn about inheritance, code reuse, using classes far more extensively, and so on.
For this purpose I am trying to piece together all the parts required for a basic RPG/dungeon crawler.
I know this has been done a billion times before, but I find that actually trying to code something like it takes you through a lot more problems than you might think, which is a great way to learn (I think).
For now I have only loaded up a WPF application, since my interest is 95% on being able to piece together the working classes, routines, functions, etc. And not so much interested in how it will look. I am actually reading up on XNA, but since I am mostly trying to get a grip on the basic workings, I don't want to complicate those aspects with the graphical side of things just yet.
The problem I am now facing is that when I would a character to attack or defend, it should know from which other character it came, or to which one it should be pointed. I figured I could either use a GUID, or a manually appointed ID. But the problem is that I don't really know how I can implement such a thing.
The thing that I figured was that I could maybe add a reference to an array (Character[]), and have a SearchByID function loop through them to find the right one, and return it. Like so:
internal Character SearchByID(string _ID)
{
foreach(Character charToFind in Character[])
{
if(charToFind.ID == _ID)
return charToFind;
}
}
This of course has to be altered a bit due to the return at the moment, but just to give you an idea.
What I am stuck on is how to create the appropriate array outside of the "Character"-class? I can fill it up just fine, but how do I go about having it added above class level?
The way the "Character"-class is built up is that every new character instantiates from the Character class. The constructor then loads the appropriate values. But other than this, I see no possibility to initialize an array outside of this.
If it is preferable to post the entire code that I have, that will be no problem at all!
Thanks for any insights you may provide me with.

I think you can just use the Character-class and pass other Characters to it, for example:
public class Character
{
public string Name { get; private set; }
public int HitPoints { get; private set; }
public int Offense { get; private set; }
public int Defense { get; private set; }
public Character(string name, int hitPoints, int offense, int defense)
{
Name = name;
HitPoints = hitPoints;
Offense = offense;
Defense = defense;
}
public void Defend(Character source)
{
HitPoints = HitPoints - (source.Offense - Defense);
if (HitPoints <= 0)
{
Console.WriteLine("{0} died", Name);
}
}
public void Attack(Character target)
{
// Here you can call the other character's defend with this char as an attacker
target.Defend(this);
if (target.HitPoints <= 0)
{
Console.WriteLine("{0} killed {1}", Name, target.Name);
}
}
}
The thing with object oriented programming is that you have to start thinking in objects. Objects are like boxes when they're concrete. You can make new ones and give them some properties, like a name, height, width, hitpoints, whatever. You can also let these objects perform actions. Now a simple box won't do much itself, but a character can do various things, so it makes sense to put these actions in the Character-class.
Besides having Characters, you might have a Game-class which manages the game-state, characters, monsters, treasure chests etc...
Now this simple example may cause you to gain HitPoints when your defense is higher than the attacker's offense, but that's details, I'll leave the exact implementation up to you.

I guess you want a way to insert characters in an array when they are instantiated..
You can make a static array or list
So,your class in my opinion should be
class Character
{
static List<Character> characterList=new List<Character>();//all characters are here
public Character(string id,...)
{
//initialize your object
characterList.Add(this);//store them in the list as and when created
}
internal Character SearchByID(string _ID)
{
foreach(Character charToFind in characterList)
{
if(charToFind.ID == _ID)
return charToFind;
}
}
}
As you may be knowing static members are associated with the class not with the object.So,when you create a new character object it would be automatically added to the characterList

Unless you are dealing with seperate processes, e.g. client-server, you probably don't want to use "Id"s at all.
Whereever you are passing string _ID around, pass the actual Character instead. This saves you looking up in an array or whatever.
Post more code, and I can show you what I mean.

You could use a dictionary, instantiated in your controller class:
Dictionary<Guid, Character> _characterList = new Dictionary<Guid, Character>();
Initialise:
var someCharacter = new Character() { stats = something };
var otherCharacter = new Character() { stats = anotherThing };
var char1Id = Guid.NewGuid();
var char2Id = Guid.NewGuid();
_characterList.Add(char1Id, someCharacter);
_characterList.Add(char2Id, otherCharacter);
then, to access characters:
var charToFind = _characterList[char1Id];
or
var charToFind = _characterList.Single(c => c.Name = "Fred The Killer");
or whatever else...

Check out keyed collection
KeyedCollection
It is like a dictionary where the key is a property of class.
You will be able to reference a Character with
Characters[id]
Syntax
On your Character class overrite GetHashCode and Equals for performance.
If you use Int32 for the ID then you will get a perfect hash.
Very fast and O(1).

Merge 2 sorted time series algorithm

I have 2 time series that contain Bar objects, each Bar object contains a member variable of type long and each time series is stored within its own BlockingCollection. The time series is sorted in ascending order of the long values.
I like to devise a merge algorithm that allows me to take away the Bar that contains the long member variable of lowest value relative to the same comparison element in the other BlockingCollection.
Example, if the long value contained in the first Bar (bar1) in BlockingCollection1 is lower than the long value contained in the first Bar (bar2) in BlockingCollection2, then Take() from BlockingCollection1 and Add() to a MasterBlockingCollection, essentially ending up with a merged stream of Bar objects sorted by the value of each Bar's long member variable.
I like to later on extend to n BlockingCollections, not just 2. I played around with arrays that hold the long values to make the mapping easier but I think arrays are handier when working with pointers pertaining to this specific target algorithm.
I wonder whether anyone can point me to a Linq implementation and comment on how computationally expensive such approach is. I am asking because throughput is of importance as there are hundreds of millions of Bar objects flowing through the collections. If someone has a more clever idea than using Linq that would be very welcomed. I came across some ideas re merge algorithm at DrDobbs some time ago but cannot find the article anymore. In case it is not apparent by now, I target C# (.Net4.0)
Thanks a lot
Edit: I forgot to mention that the merging process is supposed to happen at the same time than workers who add new items to the blockingcollections (running on different tasks)

Here's an implementation of Merge. It should run in O(cN) time, where c is the number of collections. Is this what you're looking for?
public static BlockingCollection<Bar> Merge(IEnumerable<BlockingCollection<Bar>> collections)
{
BlockingCollection<Bar> masterCollection = new BlockingCollection<Bar>();
LinkedList<BarWrapper> orderedLows = new LinkedList<BarWrapper>();
foreach (var c in collections)
OrderedInsert(new BarWrapper { Value = c.Take(), Source = c }, orderedLows);
while (orderedLows.Any())
{
BarWrapper currentLow = orderedLows.First.Value;
orderedLows.RemoveFirst();
BlockingCollection<Bar> collection = currentLow.Source;
if (collection.Any())
OrderedInsert(new BarWrapper { Value = collection.Take(), Source = collection }, orderedLows);
masterCollection.Add(currentLow.Value);
}
return masterCollection;
}
private static void OrderedInsert(BarWrapper bar, LinkedList<BarWrapper> orderedLows)
{
if (!orderedLows.Any())
{
orderedLows.AddFirst(bar);
return;
}
var iterator = orderedLows.First;
while (iterator != null && iterator.Value.Value.LongValue < bar.Value.LongValue)
iterator = iterator.Next;
if (iterator == null)
orderedLows.AddLast(bar);
else
orderedLows.AddBefore(iterator, bar);
}
class BarWrapper
{
public Bar Value { get; set; }
public BlockingCollection<Bar> Source { get; set; }
}
class Bar
{
public Bar(long l)
{
this.LongValue = l;
}
public long LongValue { get; set; }
}

Generating the next available unique name in C#

If you were to have a naming system in your app where the app contains say 100 actions, which creates new objects, like:
Blur
Sharpen
Contrast
Darken
Matte
...
and each time you use one of these, a new instance is created with a unique editable name, like Blur01, Blur02, Blur03, Sharpen01, Matte01, etc. How would you generate the next available unique name, so that it's an O(1) operation or near constant time. Bear in mind that the user can also change the name to custom names, like RemoveFaceDetails, etc.
It's acceptable to have some constraints, like restricting the number of characters to 100, using letters, numbers, underscores, etc...
EDIT: You can also suggest solutions without "filling the gaps" that is without reusing the already used, but deleted names, except the custom ones of course.

I refer you to Michael A. Jackson's Two Rules of Program Optimization:
Don't do it.
For experts only: Don't do it yet.
Simple, maintainable code is far more important than optimizing for a speed problem that you think you might have later.
I would start simple: build a candidate name (e.g. "Sharpen01"), then loop through the existing filters to see if that name exists. If it does, increment and try again. This is O(N2), but until you get thousands of filters, that will be good enough.
If, sometime later, the O(N2) does become a problem, then I'd start by building a HashSet of existing names. Then you can check each candidate name against the HashSet, rather than iterating. Rebuild the HashSet each time you need a unique name, then throw it away; you don't need the complexity of maintaining it in the face of changes. This would leave your code easy to maintain, while only being O(N).
O(N) will be good enough. You do not need O(1). The user is not going to click "Sharpen" enough times for there to be any difference.

I would create a static integer in action class that gets incremented and assigned as part of each new instance of the class. For instance:
class Blur
{
private static int count = 0;
private string _name;
public string Name
{
get { return _name; }
set { _name = value; }
}
public Blur()
{
_name = "Blur" + count++.ToString();
}
}
Since count is static, each time you create a new class, it will be incremented and appended to the default name. O(1) time.
EDIT
If you need to fill in the holes when you delete, I would suggest the following. It would automatically queue up numbers when items are renamed, but it would be more costly overall:
class Blur
{
private static int count = 0;
private static Queue<int> deletions = new Queue<int>();
private string _name;
public string Name
{
get { return _name; }
set
{
_name = value;
Delete();
}
}
private int assigned;
public Blur()
{
if (deletions.Count > 0)
{
assigned = deletions.Dequeue();
}
else
{
assigned = count++;
}
_name = "Blur" + assigned.ToString();
}
public void Delete()
{
if (assigned >= 0)
{
deletions.Enqueue(assigned);
assigned = -1;
}
}
}
Also, when you delete an object, you'll need to call .Delete() on the object.
CounterClass Dictionary version
class CounterClass
{
private int count;
private Queue<int> deletions;
public CounterClass()
{
count = 0;
deletions = new Queue<int>();
}
public string GetNumber()
{
if (deletions.Count > 0)
{
return deletions.Dequeue().ToString();
}
return count++.ToString();
}
public void Delete(int num)
{
deletions.Enqueue(num);
}
}
you can create a Dictionary to look up counters for each string. Just make sure you parse out the index and call .Delete(int) whenever you rename or delete a value.

You can easily do it in O(m) where m is the number of existing instances of the name (and not dependent on n, the number of items in the list.
Look up the string S in question. If S isn't in the list, you're done.
S exists, so construct S+"01" and check for that. Continue incrementing (e.g. next try S+"02" until it doesn't exist.
This gives you unique names but they're still "pretty" and human-readable.
Unless you expect a large number of duplicates, this should be "near-constant" time because m will be so small.
Caveat: What if the string naturally ends with e.g. "01"? In your case this sounds unlikely so perhaps you don't care. If you do care, consider adding more of a suffix, e.g. "_01" instead of just "01" so it's easier to tell them apart.

You could do something like this:
private Dictionary<string, int> instanceCounts = new Dictionary<string, int>();
private string GetNextName(string baseName)
{
int count = 1;
if (instanceCounts.TryGetValue(baseName, out count))
{
// the thing already exists, so add one to it
count++;
}
// update the dictionary with the new value
instanceCounts[baseName] = count;
// format the number as desired
return baseName + count.ToString("00");
}
You would then just use it by calling GetNextName(...) with the base name you wanted, such as
string myNextName = GetNextName("Blur");
Using this, you wouldn't have to pre-init the dictionary.
It would fill in as you used the various base words.
Also, this is O(1).

I would create a dictionary with a string key and a integer value, storing the next number to use for a given action. This will be almost O(1) in practice.
private IDictionary<String, Int32> NextFreeActionNumbers = null;
private void InitializeNextFreeActionNumbers()
{
this.NextFreeActionNumbers = new Dictionary<String, Int32>();
this.NextFreeActionNumbers.Add("Blur", 1);
this.NextFreeActionNumbers.Add("Sharpen", 1);
this.NextFreeActionNumbers.Add("Contrast", 1);
// ... and so on ...
}
private String GetNextActionName(String action)
{
Int32 number = this.NextFreeActionNumbers[action];
this.NextFreeActionNumbers[action] = number + 1;
return String.Format("{0} {1}", action, number);
}
And you will have to check against collisions with user edited values. Again a dictionary might be a smart choice. There is no way around that. What ever way you generate your names, the user can always change a existing name to the next one you generate unless you include all existing names into the generation schema. (Or use a special character that is not allowed in user edited names, but that would be not that nice.)
Because of the comments on reusing the holes I want to add it here, too. Don't resuse the holes generated be renaming or deletion. This will confuse the user because names he deleted or modified will suddenly reappear.

I would look for ways to simplify the problem.
Are there any constraints that can be applied? As an example, would it be good enough if each user can only have one (active) type of action? Then, the actions could be distinguished using the name (or ID) of the user.
Blur (Ben F)
Blur (Adrian H)
Focus (Ben F)
Perhaps this is not an option in this case, but maybe something else would be possible. I would go to great lengths in order to avoid the complexity in some of the proposed solutions!

If you want O(1) time then just track how many instances of each you have. Keep a hashtable with all of the possible objects, when you create an object, increment the value for that object and use the result in the name.

You're definitely not going to want to expose a GUID to the user interface.
Are you proposing an initial name like "Blur04", letting the user rename it, and then raising an error message if the user's custom name conflicts? Or silently renaming it to "CustomName01" or whatever?
You can use a Dictionary to check for duplicates in O(1) time. You can have incrementing counters for each effect type in the class that creates your new effect instances. Like Kevin mentioned, it gets more complex if you have to fill in gaps in the numbering when an effect is deleted.