Develop Reference

How to handle values of values of values

I'm reading in a text file that contains data for 3D elements and store them in a dictionary dict in C#. The main objects are OPEN_SHELLs and CLOSED_SHELLs. These contain multiple ADVANCED_FACEs. These again contain a single FACE_OUTER_BOUND and multiple FACE_BOUNDs. These again contain more values and so on until there are finally numerical values.
For now I have a class Step that contains
List<List>string>> closedShell; //contains all closed shells with their values
List<List<string>> openShell; //contains all open shells with their values
List<List<string>> closedShellAdvFace; //contains all closed advanced faces...
List<List<string>> openShellAdvFace; //contains all open advanced faces...
...
I iterate through each list to get the next values and so on. Now this doesn't seem really efficient as I'm using duplicate code for closed and open lists.
An examplary code for this:
string closedShellKey = "";
string closedShellValue = "";
string openShellKey = "";
string openShellValue = "";
// For CLOSED_SHELLs
for (int shellListIndex = 0; shellListIndex < stepObj.GetClosedShells().Count; shellListIndex++)
{
for (int valuesCount = 1; valuesCount < stepObj.GetClosedShells()[shellListIndex].Count - 1; valuesCount++)
{
if (dict.ContainsKey(stepObj.GetClosedShells()[shellListIndex][valuesCount]))
{
closedShellKey = stepObj.GetClosedShells()[shellListIndex][valuesCount];
dict.TryGetValue(closedShellKey, out closedShellValue);
stepObj.SetCsAdvFace(SplitValues(closedShellValue));
} else
{
//Throw Exception
}
}
}
// For OPEN_SHELLs
for (int shellListIndex = 0; shellListIndex < stepObj.GetOpenShells().Count; shellListIndex++)
{
for (int valuesCount = 1; valuesCount < stepObj.GetOpenShells()[shellListIndex].Count - 1; valuesCount++)
{
if (dict.ContainsKey(stepObj.GetOpenShells()[shellListIndex][valuesCount]))
{
openShellKey = stepObj.GetOpenShells()[shellListIndex][valuesCount];
dict.TryGetValue(openShellKey, out openShellValue);
stepObj.SetOsAdvFace(SplitValues(openShellValue));
} else
{
//Throw Exception
}
}
}
This goes on for the next values, etc.
What would be a really good and efficient way to implement each of these steps?
Maybe create an openShellObject and a closedShellObject to further seperate?
How would I handle data that contains different data that again contains further different data, etc.?
Hope this is clear enough

First, note that Dictionary.TryGetValue already does the work of Dictionary.ContainsKey so you only need the former.
If I understand this, you need to iterate over multiple collections, applying an operation that varies only in one step, according to each collection category, e.g. closed face, open face, etc. How to separate that step from the iteration code? I'd suggest either Template Method pattern or Method As Parameter (MAP). For this problem I'd probably choose MAP because the collection items vary by category not data type, and probably it means less coding.
In the pseudocode below I've assumed that the final step in each iteration always involves a method like stepObj.SetCsAdvFace that takes a string[] value returned by SplitValues. This is why, in the Apply method below, the parameter method is a delegate that takes a string[] parameter, so it matches either stepObj.SetCsAdvFace or stepObj.SetOsAdvFace, whichever is required for the relevant collection.
private void Apply(List<List<string>> list, Action<string[]> method)
{
foreach (var items in list)
{
for (int valuesIndex = 1; valuesIndex < items.Count - 1; valuesIndex++)
{
var key = items[valuesIndex];
string values;
if (dict.TryGetValue(key, out values))
{
method(SplitValues(values));
}
else
{
//Throw Exception
}
}
}
}
Apply(stepObj.GetClosedShells(), stepObj.SetCsAdvFace);
Apply(stepObj.GetOpenShells(), stepObj.SetOsAdvFace);
...

First get rid of two for loops, use IEnumerable instead:
var allShellKeys = stepObj.GetClosedShells().Union(stepObj.GetOpenShells()).SelectMany(i => i.Skip(1).Take(i.Count() - 2))
Then you can iterate over all values in one loop:
string anyShellValue;
foreach (var anyShellKey in allShellKeys)
{
if (dict.TryGetValue(anyShellKey, out anyShellValue))
{
stepObj.SetCsAdvFace(SplitValues(anyShellValue));
}
else
{
//Throw Exception
}
}

Index out of range error when using a list as a parameter for thread start routine

I am writing a C# program that requires giving a thread parameters to a function so that the function will run properly on the separate thread. Specifically one of the parameters is a string name to a file that it is supposed to access. The problem is that I am storing the names of the files in a list and I am accessing the value from the list. However, when I do this I get an index out of range error after one or two threads are created. I think that this is list of strings is my issue, but I know that the index is not out of range.
I am not sure if I am doing something wrong with the way I am passing in the parameters or what else could be wrong.
Here is a sample of my C# code (excluding the code for the functions called):
for (int i = 0; i < 5; i++)
{
surfaceGraphDataNames.Add(String.Format(surfacePlotDataLocation+"ThreadData{0}.txt", i));
try
{
generateInputFile(masterDataLocation);
}
catch
{
MessageBox.Show("Not enough data remaining to create an input file");
masterDataLocation = masterDataSet.Count - ((graphData.NumRootsUsed + 1) * (graphData.Polynomial + 1) - 1);
this.dataSetLabel.Text = String.Format("Current Data Set: {0}", masterDataLocation + 1);
return;
}
try
{
//creates the data in a specific text file I hope
createSurfaceGraph(surfaceGraphDataNames[i]);
//start threads
threadsRunning.Add(new Thread(() => runGnuplotClicks(surfaceGraphDataNames[i], masterDataLocation)));
threadsRunning[i].Start();
}
catch
{
this.graphPictureBox1.Image = null;//makes image go away if data fails
MessageBox.Show("Gridgen failed to generate good data");
}
masterDataLocation++;
}

Looks like that you have to do something like this:
threadsRunning.Add(new Thread(() => {
var k = i;
runGnuplotClicks(surfaceGraphDataNames[k], masterDataLocation)
}
));
The reason is that when you use the variable i, it's not safe because when your i++, and the surfaceGraphDataNames has not been added with new item yet, the exception will throw because your Thread run nearly simultaneously.
Here is the context which leads to the exception:
for(int i = 0; i < 5; i++){
//Suppose i is increased to 3 at here
//Here is where your Thread running code which accesses to the surfaceGraphDataNames[i]
//That means it's out of range at this time because
//the surfaceGraphDataNames has not been added with new item by the code below
surfaceGraphDataNames.Add(String.Format(surfacePlotDataLocation+"ThreadData{0}.txt", i));
//....
}
UPDATE
Looks like that the code above even can't work possibly because the i is increased before the actual ThreadStart is called. I think you can do this to make it safer:
var j = i;
threadsRunning.Add(new Thread(() => {
var k = j;
runGnuplotClicks(surfaceGraphDataNames[k], masterDataLocation)
}
));
Synchronization Attempt:
Queue<int> q = new Queue<int>();
for(int i = 0; i < 5; i++){
//.....
q.Enqueue(i);
threadsRunning.Add(new Thread(() => {
runGnuplotClicks(surfaceGraphDataNames[q.Dequeue()], masterDataLocation)
}
));
threadsRunning[i].Start();
}

I had a problem like this then I use Thread. I was sure that the index is not out of range and this situation not happened if I tried to stop by break-point and then continued.
Try to use Task instead of Thread. It works

The most obvious problem is that you are closing over the loop variable. When you construct a lambda expression any variable references are to the variable itself and not its value. Consider the following code taken from your example.
for (int i = 0; i < 5; i++)
{
// Code omitted for brevity.
new Thread(() => runGnuplotClicks(surfaceGraphDataNames[i], masterDataLocation))
// Code omitted for brevity.
}
What this is actually doing is capturing the variable i. But, by the time the thread starts executing the i could have been incremented several times possibly (and even likely) to the point where its value is now 5. It is possible that the IndexOutOfRangeException is being thrown because surfaceGraphDataNames does not have 6 slots. Nevermind the fact that your thread is not using the value of i that you thought it was.
To fix this you need to create a special capturing variable.
for (int i = 0; i < 5; i++)
{
// Code omitted for brevity.
int capture = i;
new Thread(() => runGnuplotClicks(surfaceGraphDataNames[capture], masterDataLocation))
// Code omitted for brevity.
}

Why does the same algorithm work in Scala much slower than in C#? And how to make it faster?

The algorithm creates all possible variants of the sequence from variants for each member of the sequence.
C# code :
static void Main(string[] args)
{
var arg = new List<List<int>>();
int i = 0;
for (int j = 0; j < 5; j++)
{
arg.Add(new List<int>());
for (int j1 = i; j1 < i + 3; j1++)
{
//if (j1 != 5)
arg[j].Add(j1);
}
i += 3;
}
List<Utils<int>.Variant<int>> b2 = new List<Utils<int>.Variant<int>>();
//int[][] bN;
var s = System.Diagnostics.Stopwatch.StartNew();
//for(int j = 0; j < 10;j++)
b2 = Utils<int>.Produce2(arg);
s.Stop();
Console.WriteLine(s.ElapsedMilliseconds);
}
public class Variant<T>
{
public T element;
public Variant<T> previous;
}
public static List<Variant<T>> Produce2(List<List<T>> input)
{
var ret = new List<Variant<T>>();
foreach (var form in input)
{
var newRet = new List<Variant<T>>(ret.Count * form.Count);
foreach (var el in form)
{
if (ret.Count == 0)
{
newRet.Add(new Variant<T>{ element = el, previous = null });
}
else
{
foreach (var variant in ret)
{
var buf = new Variant<T> { previous = variant, element = el };
newRet.Add(buf);
}
}
}
ret = newRet;
}
return ret;
}
Scala code :
object test {
def main() {
var arg = new Array[Array[Int]](5)
var i = 0
var init = 0
while (i<5)
{
var buf = new Array[Int](3)
var j = 0
while (j<3)
{
buf(j) = init
init = init+1
j = j + 1
}
arg(i)=buf
i = i + 1
}
println("Hello, world!")
val start = System.currentTimeMillis
var res = Produce(arg)
val stop = System.currentTimeMillis
println(stop-start)
/*for(list <- res)
{
for(el <- list)
print(el+" ")
println
}*/
println(res.length)
}
def Produce[T](input:Array[Array[T]]):Array[Variant[T]]=
{
var ret = new Array[Variant[T]](1)
for(val forms <- input)
{
if(forms!=null)
{
var newRet = new Array[Variant[T]](forms.length*ret.length)
if(ret.length>0)
{
for(val prev <-ret)
if(prev!=null)
for(val el <-forms)
{
newRet = newRet:+new Variant[T](el,prev)
}
}
else
{
for(val el <- forms)
{
newRet = newRet:+new Variant[T](el,null)
}
}
ret = newRet
}
}
return ret
}
}
class Variant[T](var element:T, previous:Variant[T])
{
}

As others have said, the difference is in how you're using the collections. Array in Scala is the same thing as Java's primitive array, [], which is the same as C#'s primitive array []. Scala is clever enough to do what you ask (namely, copy the entire array with a new element on the end), but not so clever as to tell you that you'd be better off using a different collection. For example, if you just change Array to ArrayBuffer it should be much faster (comparable to C#).
Actually, though, you'd be better off not using for loops at all. One of the strengths of Scala's collections library is that you have a wide variety of powerful operations at your disposal. In this case, you want to take every item from forms and convert it into a Variant. That's what map does.
Also, your Scala code doesn't seem to actually work.
If you want all possible variants from each member, you really want to use recursion. This implementation does what you say you want:
object test {
def produce[T](input: Array[Array[T]], index: Int = 0): Array[List[T]] = {
if (index >= input.length) Array()
else if (index == input.length-1) input(index).map(elem => List(elem))
else {
produce(input, index+1).flatMap(variant => {
input(index).map(elem => elem :: variant)
})
}
}
def main() {
val arg = Array.tabulate(5,3)((i,j) => i*3+j)
println("Hello, world!")
val start = System.nanoTime
var res = produce(arg)
val stop = System.nanoTime
println("Time elapsed (ms): " + (stop-start)/1000000L)
println("Result length: " + res.length)
println(res.deep)
}
}
Let's unpack this a little. First, we've replaced your entire construction of the initial variants with a single tabulate instruction. tabulate takes a target size (5x3, here), and then a function that maps from the indices into that rectangle into the final value.
We've also made produce a recursive function. (Normally we'd make it tail-recursive, but let's keep things as simple as we can for now.) How do you generate all variants? Well, all variants is clearly (every possibility at this position) + (all variants from later positions). So we write that down recursively.
Note that if we build variants recursively like this, all the tails of the variants end up the same, which makes List a perfect data structure: it's a singly-linked immutable list, so instead of having to copy all those tails over and over again, we just point to them.
Now, how do we actually do the recursion? Well, if there's no data at all, we had better return an empty array (i.e. if index is past the end of the array). If we're on the last element of the array of variations, we basically want each element to turn into a list of length 1, so we use map to do exactly that (elem => List(elem)). Finally, if we are not at the end, we get the results from the rest (which is produce(input, index+1)) and make variants with each element.
Let's take the inner loop first: input(index).map(elem => elem :: variant). This takes each element from variants in position index and sticks them onto an existing variant. So this will give us a new batch of variants. Fair enough, but where do we get the new variant from? We produce it from the rest of the list: produce(input, index+1), and then the only trick is that we need to use flatMap--this takes each element, produces a collection out of it, and glues all those collections together.
I encourage you to throw printlns in various places to see what's going on.
Finally, note that with your test size, it's actually an insigificant amount of work; you can't accurately measure that, even if you switch to using the more accurate System.nanoTime as I did. You'd need something like tabulate(12,3) before it gets significant (500,000 variants produced).

The :+ method of the Array (more precisely of ArrayOps) will always create a copy of the array. So instead of a constant time operation you have one that is more or less O(n).
You do it within nested cycles => your whole stuff will be an order of magnitude slower.
This way you more or less emulate an immutable data structure with a mutable one (which was not designed for it).
To fix it you can either use Array as a mutable data structure (but then try to avoid endless copying), or you can switch to a immutable one. I did not check your code very carefully, but the first bet is usually List, check the scaladoc of the various methods to see their performance behaviour.

ret.length is not 0 all the time, right before return it is 243. The size of array should not be changed, and List in .net is an abstraction on top of array. BUT thank you for the point - problem was that I used :+ operator with array which as I understand caused implicit use of type LinkedList

modify a list while doing foreach inside lock()

atm I do it like this:
lock (LockObj)
{
foreach (var o in Oo)
{
var val = o.DateActive;
if (val.AddSeconds(30) < DateTime.Now) Oo.Remove(o);
}
}
and I get this error:
Collection was modified; enumeration operation may not execute
how this should be done?

You have to use a regular for loop.
for (int i = 0; i < Oo.Length; ++i)
{
var val = Oo[i];
if (val.AddSeconds(30) < DateTime.Now)
{
Oo.RemoveAt(i);
i--; // since we just removed an element
}
}
The reason you cannot edit a collection with a foreach loop is because foreach uses a readonly IEnumerator of the collection you are iterating.

you can't modify a collection you are enumerating..
to change it get a copy of it and change it.
for(var k in OO.ToList())
.....
or
use count and iterate the collection with index,
for (int i=0;i<OO.Count();i++)
.....

You simply cannot modify the collection if you are iterating with foreach. You have two options, Loop with For instead of foreach or create another Collection and modify that.

This problem is completely unrelated to locking.
If you add/remove elements from a List all iterators pointing to that list become invalid.
One alternative to using an iterator is manually working with indices. Then you can iterate backwards and remove elements with RemoveAt.
for(int i=Oo.Count-1;i>=0;i--)
{
var o=Oo[i];
if (o.DateActive.AddSeconds(30)<DateTime.Now)
Oo.RemoveAt(i);
}
Unfortunately this native implementation is O(n^2). If you write it in a more complex way where you first assign the elements to their new position and then truncate the list it becomes O(n).
Buf if Oo is a List<T> there is a much better solution. You can use Oo.RemoveAll(o=>o.DateActive.AddSeconds(30)<DateTime.Now). Unfortunately you there is no such extension method on IList<T> by default.
I'd write the code like this:
lock (LockObj)
{
DateTime deleteTime=DateTime.Now.AddSeconds(-30);
Oo.RemoveAll(o=>o.DateActive<deleteTime);
}
As a sidenote I'd personally use UTC times instead of local times for such code.

class Program
{
static void Main(string[] args)
{
List<OOItem> oo = new List<OOItem>();
oo.Add( new OOItem() { DateActive = DateTime.Now.AddSeconds(-31) });
lock(LockObj)
{
foreach( var item in oo.Where( ooItem => ooItem.DateActive.AddSeconds(30) < DateTime.Now ).ToArray())
{
oo.Remove(item);
}
}
Debug.Assert( oo.Count == 0);
}
}
public class OOItem
{
public DateTime DateActive { get; set; }
}

I'm going to suggest an approach that avoids messing around with decrementing loop indexes and other stuff that makes code difficult to understand.
I think the best bet is to write a nice query and then do a foreach over the result of turning the query into an array:
var inactives = from o in Oo
where o.DateActive < DateTime.Now
select o;
foreach (var o in inactives.ToArray())
{
Oo.Remove(o);
}
This avoids the issue of the collection changing and makes the code quite a bit more readable.
If you're a little more "functionally" oriented then here's another choice:
(from o in Oo
where o.DateActive < DateTime.Now
select o)
.ToList()
.ForEach(o => Oo.Remove(o));
Enjoy!

The problem is not related to the lock.
Use a for() loop instead of foreach().
I can't 100% replace your code because your code provides no hint of what collection type "Oo" is. Neither does the name "Oo". Perhaps one of the evils of var keyword overuse? Or maybe I just can't see enough of your code ;)
int size = Oo.Length();
for(int i = 0; i < size; i++){
if (Oo[i].AddSeconds(30) < DateTime.Now){
Oo[i].RemoveAt(i);
size--; // Compensate for new size after removal.
}
}

you can use Parallel.ForEach(oO, val=> { oO.Remove(val); })
Parallel doesn't have the IEnumerator problem !

Is one of these for loops faster than the other?

for (var keyValue = 0; keyValue < dwhSessionDto.KeyValues.Count; keyValue++)
{...}
var count = dwhSessionDto.KeyValues.Count;
for (var keyValue = 0; keyValue < count; keyValue++)
{...}
I know there's a difference between the two, but is one of them faster than the other? I would think the second is faster.

Yes, the first version is much slower. After all, I'm assuming you're dealing with types like this:
public class SlowCountProvider
{
public int Count
{
get
{
Thread.Sleep(1000);
return 10;
}
}
}
public class KeyValuesWithSlowCountProvider
{
public SlowCountProvider KeyValues
{
get { return new SlowCountProvider(); }
}
}
Here, your first loop will take ~10 seconds, whereas your second loop will take ~1 second.
Of course, you might argue that the assumption that you're using this code is unjustified - but my point is that the right answer will depend on the types involved, and the question doesn't state what those types are.
Now if you're actually dealing with a type where accessing KeyValues and Count is cheap (which is quite likely) I wouldn't expect there to be much difference. Mind you, I'd almost always prefer to use foreach where possible:
foreach (var pair in dwhSessionDto.KeyValues)
{
// Use pair here
}
That way you never need the count. But then, you haven't said what you're trying to do inside the loop either. (Hint: to get more useful answers, provide more information.)

it depends how difficult it is to compute dwhSessionDto.KeyValues.Count if its just a pointer to an int then the speed of each version will be the same. However, if the Count value needs to be calculated, then it will be calculated every time, and therefore impede perfomance.
EDIT -- heres some code to demonstrate that the condition is always re-evaluated
public class Temp
{
public int Count { get; set; }
}
static void Main(string[] args)
{
var t = new Temp() {Count = 5};
for (int i = 0; i < t.Count; i++)
{
Console.WriteLine(i);
t.Count--;
}
Console.ReadLine();
}
The output is 0, 1, 2 - only !

See comments for reasons why this answer is wrong.
If there is a difference, it’s the other way round: Indeed, the first one might be faster. That’s because the compiler recognizes that you are iterating from 0 to the end of the array, and it can therefore elide bounds checks within the loop (i.e. when you access dwhSessionDTo.KeyValues[i]).
However, I believe the compiler only applies this optimization to arrays so there probably will be no difference here.

It is impossible to say without knowing the implementation of dwhSessionDto.KeyValues.Count and the loop body.
Assume a global variable bool foo = false; and then following implementations:
/* Loop body... */
{
if(foo) Thread.Sleep(1000);
}
/* ... */
public int Count
{
get
{
foo = !foo;
return 10;
}
}
/* ... */
Now, the first loop will perform approximately twice as fast as the second ;D
However, assuming non-moronic implementation, the second one is indeed more likely to be faster.

No. There is no performance difference between these two loops. With JIT and Code Optimization, it does not make any difference.

There is no difference but why you think that thereis difference , can you please post your findings?
if you see the implementation of insert item in Dictionary using reflector
private void Insert(TKey key, TValue value, bool add)
{
int freeList;
if (key == null)
{
ThrowHelper.ThrowArgumentNullException(ExceptionArgument.key);
}
if (this.buckets == null)
{
this.Initialize(0);
}
int num = this.comparer.GetHashCode(key) & 0x7fffffff;
int index = num % this.buckets.Length;
for (int i = this.buckets[index]; i >= 0; i = this.entries[i].next)
{
if ((this.entries[i].hashCode == num) && this.comparer.Equals(this.entries[i].key, key))
{
if (add)
{
ThrowHelper.ThrowArgumentException(ExceptionResource.Argument_AddingDuplicate);
}
this.entries[i].value = value;
this.version++;
return;
}
}
if (this.freeCount > 0)
{
freeList = this.freeList;
this.freeList = this.entries[freeList].next;
this.freeCount--;
}
else
{
if (this.count == this.entries.Length)
{
this.Resize();
index = num % this.buckets.Length;
}
freeList = this.count;
this.count++;
}
this.entries[freeList].hashCode = num;
this.entries[freeList].next = this.buckets[index];
this.entries[freeList].key = key;
this.entries[freeList].value = value;
this.buckets[index] = freeList;
this.version++;
}
Count is a internal member to this class which is incremented each item you insert an item into dictionary
so i beleive that there is no differenct at all.

The second version can be faster, sometimes. The point is that the condition is reevaluated after every iteration, so if e.g. the getter of "Count" actually counts the elements in an IEnumerable, or interogates a database /etc, this will slow things down.
So I'd say that if you dont affect the value of "Count" in the "for", the second version is safer.