We are looking through our code trying to identify high CPU usage, and I am looking at several areas where we use while loops. I would like to take the risk of infinite loops out of the code shown below, but I am not sure what the best solution would be.
IDictionaryEnumerator codeEnumerator = Resources.Error_Codes.ResourceManager.GetResourceSet(CultureInfo.CurrentCulture, true, true).GetEnumerator();
IDictionaryEnumerator messageEnumerator = Resources.Error_Messages.ResourceManager.GetResourceSet(CultureInfo.CurrentCulture, true, true).GetEnumerator();
bool codeDone = false;
bool messageDone = false;
while (codeEnumerator.MoveNext() && !codeDone)
{
string value = codeEnumerator.Value.ToString();
if (value == failedResponse.Code.ToString())
{
key = codeEnumerator.Key.ToString();
codeDone = true;
}
}
while (messageEnumerator.MoveNext() && !messageDone)
{
if (messageEnumerator.Key.ToString() == key)
{
message = messageEnumerator.Value.ToString();
messageDone = true;
}
}
Assuming that the underlying sequence are finite, not infinite, (which you have said is the case) then the loops will not run forever.
Eventually you can be sure that one of the following things will happen:
The if will be true for a given item, thus setting the boolean and breaking out of the loop.
You will advance to the end of the sequence, thus resulting in MoveNext being false.
An exception will be thrown from somewhere, such as from the underlying collection being modified by another thread, by a null value in the sequence, or anything else. As you have no try/catch, this will break you out of the loop.
In particular, since each iteration of the loop must advance the iterator (due to MoveNext) you can be sure that you will eventually end.
Looking at your code and trying to figure out what your dictionaries are actually named, I think you're looking for something like this:
var key = Error_Codes.FirstOrDefault(kvp => kvp.Value.ToString =
failedResponse.Code.ToString()).Select(kvp => kvp.Key);
string message = string.Empty;
if(null != key)
message = Error_Messages[key];
This assumes your dictionaries are Error_Codes and Error_Messages.
As Eric pointed out in the comments, there were issues with the way you were using iterators and dictionaries. This does away with the iterator problems, but this still isn't an ideal way to use dictionaries.
If you have a unique list of error messages and error code keys, then you could have a dictionary that maps the two together. Alternatively, you could combine the dictionaries with a common key set for the dictionary key, and a tuple of error codes and error messages as the dictionary values.
You could start a timer/other thread that sets a second contidion to false if the loop takes more than x time. But i dont think that this is a clean solution either.
You could add a counter that counts down in the while loop. Set the counter real high to a value like 100 or so and When the counter reaches zero, exit the loop. It means that its possible it might terminate before the operation is completely carried out, but it means it will eventually exit.
Related
foreach (var thing in things)
{
tryagain:
string thing.var1 = ThisCanReturnNullSometimes();
if (thing.var1 == null)
{
goto tryagain;
}
}
I know ideally you don't want a method that can "fail" but I'm working with the youtube data API and for some reason some calls just.. don't follow through.
It seems like a short and sweet way to reattempt the iteration, but i've never used a goto before and I've only heard people say don't use it.
Most programs can be expressed without goto. In this particular case, a loop is a far more readable construct, because it says pretty much what you want it to say:
string x;
do {
x=CanReturnNullSometimes();
} while (x==null);
One nice thing about this loop is that the readers always know its post-condition: the only way this loop can terminate is that x becomes non-null. You can also add safety check to ensure that you are not calling the method too many times.
Your goto is safe, but is generally not used. Essentially you have written an implementation of a while loop.
But your code does have an interesting trait, your variable can be scoped and assigned and still be available after the loop... which can be concisely done like :-
tryagain: var s = ThisCanReturnNullSometimes();
if (s == null) goto tryagain;
however, while that is interesting.... I'd stick with a while loop or helper method if you'd like it more concise
Of course, it also has the additional problem of being an infinite loop in the case of null being returned all the time.
You would probably be better off using something like a while loop to monitor the status of your method and keep trying it. You could add a maximum iteration check to make sure it doesn't just loop forever.
string thing.var1 = ThisCanReturnNullSometimes();
int iteration = 0;
while (thing.var1 == null && iteration < 5)
{
Thread.Sleep(5000); // sleep for a bit to give the remove service time to "work"
thing.var1 = ThisCanReturnNullSometimes();
iteration++;
}
This will sleep for 5 seconds then try the method again and repeat up to 5 times before continuing on.
Of course, the best way would be to work out why your method fails, if it's a common problem or something that can be fixed.
I ran across a weird issue, but there is certainly a miss from my side. Check if you can point that out. I was doing this in foreach loop but then I switched to a for loop to avoid looping ambiguities, but still it fails.
// Update each quantum.
for (int i = 0; i < Quantums.Where(a => a.Language != "en-US").Count(); i++)
{
Quantums[i].ListQType.AddRange(AddedOnes);
}
What happens is that the list AddedOnes gets added to all quantums in Each iteration and not only to the one we are dealing in the loop body. I suspect references mess.
Any pointers?
Thanks
This might not have caused your problem, but your for loop's condition seems wrong. If you're going to do a Where() on the condition, you should make sure the list you are working on in the for loop is on the same condition.
I suspect references mess
However, in spite of that, as you suspect, each of your Quantums seem to refer to the same object. It's probably in the way you created each object. I suspect you were doing something like:
Quantum q1 = new Quantum(){Property1=something /*etc*/};
Quantum q2 = q1; // assuming your Quantums is a list of Quantum
Or it could be just the ListQType that is reference copied on each of the elements of your Quantums.
You should look into deep copying, or at least instantiate new objects and copy values individually (safe if they are value types) if that was what you were trying to do.
Calling Quantums[i] is the problem. The filtered list will have different indexes and the orignal list will have different index i. Here you are modifying the original list with the the index of the filtered list.
Better use the for-each lambda expression or the for-each loop.
Quantums.Where(a => a.Language != "en-US").ForEach(x=>x.ListQType.AddRange(AddedOnes));
or
foreach(var item in Quantums.Where(a => a.Language != "en-US").)
{
item.ListQType.AddRange(AddedOnes);
}
I'm implementing a simple Dictionary<String, Int> that keeps track of picture files I download, and also the renaming of the files.
String - original filename
Int - new filename
I read up on TryGetValue vs ContainsKey and came across this:
TryGetValue approach is faster than ContainsKey approach but only when
you want to check the key in collection and also want to get the value
associated with it. If you only want to check the key is present or
not use ContainsKey only.
from here
As such, I was wondering what were other people's views on the following:
Should I use TryGetValue even if I do not plan to use the returned value, assuming the Dictonary size would grow to 1000 entries, and I do duplicate checks everytime I download ie. frequently?
In theory, follow the documentation. If you don't want the value then use ContainsKey because there's no code to go and actually grab the value out of memory.
Now, in practice, it probably doesn't matter because you're micro-optimizing on a Dictionary that's probably very small in the grand scheme of things. So, in practice, do what is best for you and the readability of your code.
And just to help you get a good idea, would grow to 1000 entries is really small so it really isn't going to matter in practice.
If you only want to check the key is present or not use ContainsKey only.
I think you answered the question for yourself.
Let's see the implementation of both under Reflector
public bool TryGetValue(TKey key, out TValue value)
{
int index = this.FindEntry(key);
if (index >= 0)
{
value = this.entries[index].value;
return true;
}
value = default(TValue);
return false;
}
public bool ContainsKey(TKey key)
{
return (this.FindEntry(key) >= 0);
}
this is how both the methods are implemented.
Now you can decide yourself which method is best.
I think that performance gains (if any) aren't worth the cost of obfuscating your code with this optimization.
Balance the scale your targeting with code maintainability. E.g.:
~10K concurrent calls average vs. < 5 developer team size GO FOR IT!
~500 concurrent call average vs. > 50 developer team size DON'T DO IT!
Let's think of it as a family tree, a father has kids, those kids have kids, those kids have kids, etc...
So I have a recursive function that gets the father uses Recursion to get the children and for now just print them to debug output window...But at some point ( after one hour of letting it run and printing like 26000 rows) it gives me a StackOverFlowException.
So Am really running out of memory? hmmm? then shouldn't I get an "Out of memory exception"? on other posts I found people were saying if the number of recursive calls are too much, you might still get a SOF exception...
Anyway, my first thought was to break the tree into smaller sub-strees..so I know for a fact that my root father always has these five kids, so Instead of Calling my method one time with root passed to it, I said ok call it five times with Kids of root Passes to it.. It helped I think..but still one of them is so big - 26000 rows when it crashes - and still have this issue.
How about Application Domains and Creating new Processes at run time at some certain level of depth? Does that help?
How about creating my own Stack and using that instead of recursive methods? does that help?
here is also a high-level of my code, please take a look, maybe there is actually something silly wrong with this that causes SOF error:
private void MyLoadMethod(string conceptCKI)
{
// make some script calls to DB, so that moTargetConceptList2 will have Concept-Relations for the current node.
// when this is zero, it means its a leaf.
int numberofKids = moTargetConceptList2.ConceptReltns.Count();
if (numberofKids == 0)
return;
for (int i = 1; i <= numberofKids; i++)
{
oUCMRConceptReltn = moTargetConceptList2.ConceptReltns.get_ItemByIndex(i, false);
//Get the concept linked to the relation concept
if (oUCMRConceptReltn.SourceCKI == sConceptCKI)
{
oConcept = moTargetConceptList2.ItemByKeyConceptCKI(oUCMRConceptReltn.TargetCKI, false);
}
else
{
oConcept = moTargetConceptList2.ItemByKeyConceptCKI(oUCMRConceptReltn.SourceCKI, false);
}
//builder.AppendLine("\t" + oConcept.PrimaryCTerm.SourceString);
Debug.WriteLine(oConcept.PrimaryCTerm.SourceString);
MyLoadMethod(oConcept.ConceptCKI);
}
}
How about creating my own Stack and using that instead of recursive methods? does that help?
Yes!
When you instantiate a Stack<T> this will live on the heap and can grow arbitrarily large (until you run out of addressable memory).
If you use recursion you use the call stack. The call stack is much smaller than the heap. The default is 1 MB of call stack space per thread. Note this can be changed, but it's not advisable.
StackOverflowException is quite different to OutOfMemoryException.
OOME means that there is no memory available to the process at all. This could be upon trying to create a new thread with a new stack, or in trying to create a new object on the heap (and a few other cases).
SOE means that the thread's stack - by default 1M, though it can be set differently in thread creation or if the executable has a different default; hence ASP.NET threads have 256k as a default rather than 1M - was exhausted. This could be upon calling a method, or allocating a local.
When you call a function (method or property), the arguments of the call are placed on the stack, the address the function should return to when it returns are put on the stack, then execution jumps to the function called. Then some locals will be placed on the stack. Some more may be placed on it as the function continues to execute. stackalloc will also explicitly use some stack space where otherwise heap allocation would be used.
Then it calls another function, and the same happens again. Then that function returns, and execution jumps back to the stored return address, and the pointer within the stack moves back up (no need to clean up the values placed on the stack, they're just ignored now) and that space is available again.
If you use up that 1M of space, you get a StackOverflowException. Because 1M (or even 256k) is a large amount of memory for these such use (we don't put really large objects in the stack) the three things that are likely to cause an SOE are:
Someone thought it would be a good idea to optimise by using stackalloc when it wasn't, and they used up that 1M fast.
Someone thought it would be a good idea to optimise by creating a thread with a smaller than usual stack when it wasn't, and they use up that tiny stack.
A recursive (whether directly or through several steps) call falls into an infinite loop.
It wasn't quite infinite, but it was large enough.
You've got case 4. 1 and 2 are quite rare (and you need to be quite deliberate to risk them). Case 3 is by far the most common, and indicates a bug in that the recursion shouldn't be infinite, but a mistake means it is.
Ironically, in this case you should be glad you took the recursive approach rather than iterative - the SOE reveals the bug and where it is, while with an iterative approach you'd probably have an infinite loop bringing everything to a halt, and that can be harder to find.
Now for case 4, we've got two options. In the very very rare cases where we've got just slightly too many calls, we can run it on a thread with a larger stack. This doesn't apply to you.
Instead, you need to change from a recursive approach to an iterative one. Most of the time, this isn't very hard thought it can be fiddly. Instead of calling itself again, the method uses a loop. For example, consider the classic teaching-example of a factorial method:
private static int Fac(int n)
{
return n <= 1 ? 1 : n * Fac(n - 1);
}
Instead of using recursion we loop in the same method:
private static int Fac(int n)
{
int ret = 1;
for(int i = 1; i <= n, ++i)
ret *= i;
return ret;
}
You can see why there's less stack space here. The iterative version will also be faster 99% of the time. Now, imagine we accidentally call Fac(n) in the first, and leave out the ++i in the second - the equivalent bug in each, and it causes an SOE in the first and a program that never stops in the second.
For the sort of code you're talking about, where you keep producing more and more results as you go based on previous results, you can place the results you've got in a data-structure (Queue<T> and Stack<T> both serve well for a lot of cases) so the code becomes something like):
private void MyLoadMethod(string firstConceptCKI)
{
Queue<string> pendingItems = new Queue<string>();
pendingItems.Enqueue(firstConceptCKI);
while(pendingItems.Count != 0)
{
string conceptCKI = pendingItems.Dequeue();
// make some script calls to DB, so that moTargetConceptList2 will have Concept-Relations for the current node.
// when this is zero, it means its a leaf.
int numberofKids = moTargetConceptList2.ConceptReltns.Count();
for (int i = 1; i <= numberofKids; i++)
{
oUCMRConceptReltn = moTargetConceptList2.ConceptReltns.get_ItemByIndex(i, false);
//Get the concept linked to the relation concept
if (oUCMRConceptReltn.SourceCKI == sConceptCKI)
{
oConcept = moTargetConceptList2.ItemByKeyConceptCKI(oUCMRConceptReltn.TargetCKI, false);
}
else
{
oConcept = moTargetConceptList2.ItemByKeyConceptCKI(oUCMRConceptReltn.SourceCKI, false);
}
//builder.AppendLine("\t" + oConcept.PrimaryCTerm.SourceString);
Debug.WriteLine(oConcept.PrimaryCTerm.SourceString);
pendingItems.Enque(oConcept.ConceptCKI);
}
}
}
(I haven't completely checked this, just added the queuing instead of recursing to the code in your question).
This should then do more or less the same as your code, but iteratively. Hopefully that means it'll work. Note that there is a possible infinite loop in this code if the data you are retrieving has a loop. In that case this code will throw an exception when it fills the queue with far too much stuff to cope. You can either debug the source data, or use a HashSet to avoid enqueuing items that have already been processed.
Edit: Better add how to use a HashSet to catch duplicates. First set up a HashSet, this could just be:
HashSet<string> seen = new HashSet<string>();
Or if the strings are used case-insensitively, you'd be better with:
HashSet<string> seen = new HashSet<string>(StringComparison.InvariantCultureIgnoreCase) // or StringComparison.CurrentCultureIgnoreCase if that's closer to how the string is used in the rest of the code.
Then before you go to use the string (or perhaps before you go to add it to the queue, you have one of the following:
If duplicate strings shouldn't happen:
if(!seen.Add(conceptCKI))
throw new InvalidOperationException("Attempt to use \" + conceptCKI + "\" which was already seen.");
Or if duplicate strings are valid, and we just want to skip performing the second call:
if(!seen.Add(conceptCKI))
continue;//skip rest of loop, and move on to the next one.
I think you have a recursion's ring (infinite recursion), not a really stack overflow error. If you are got more memory for stack - you will get the overflow error too.
For test it:
Declare a global variable for storing a operable objects:
private Dictionary<int,object> _operableIds = new Dictionary<int,object>();
...
private void Start()
{
_operableIds.Clear();
Recurtion(start_id);
}
...
private void Recurtion(int object_id)
{
if(_operableIds.ContainsKey(object_id))
throw new Exception("Have a ring!");
else
_operableIds.Add(object_id, null/*or object*/);
...
Recurtion(other_id)
...
_operableIds.Remove(object_id);
}
I am not sure if the title formulates it well so sorry.
I basically have a bunch of elements listing targets for a communication. I placed them in a dictionary though i am open to moving them to a different data structure. My problem is that i have a tree-like structure where a key is a branch and each branch has many leaves. Both the branch and the leaves have names stored in strings (cannot be numeral).
private Dictionary < string, string[]> targets;
For each element in the dictionary i must send a communication, and when the target answers i go to the next target and start over. So after searching i am faced with these dilemmas:
I cannot use the usual foreach because i need to keep the pointer in memory to pass it in between threads.
Since dictionaries are random access it is difficult to keep a pointer
When i receive a communication i must verify if the origins are from a target, so i like the dictionary.contains method for that.
I am fairly new at C#, so the answer is probably obvious but i am finding a hard time finding a data structure that fits my needs. What would be the simplest solution? Can somebody suggest anything?
Thank you.
EDIT
I think my post has confused many, and they are sort of stuck on the terms pointers and threads. By threads i don`t mean that they are parallel, simply that i cannot use a foreach or a loop as the next thread that does the next iteration is triggered by incoming communication. This mechanism cannot be changed at the moment, just the iteration must be. By pointer i wasn't referring to the memory pointers often used in C, i just meant something that points to where you are in a list. Sorry i am a Java programmer so i might be using confusing terms.
I noticed the Enumerator is often inherited and that it can be used with structures such as Dictionary and Linked List. Examples i find talk about this sub structure being encapsulated, and shows foreach loops as examples.
Would it be possible to use GetEnumerator() in some way that the enumerator would remember the current position even when accessed through a different thread?
I am off to test these on my own, but if any input from more experienced people is always appreciated!
I think you need to re-work your architecture a bit, the Dictionary itself is probably not the data structure you need to use for a ordered iteration.
I would consider moving your tree into a linked list instead.
When you kick off your communications I would suggest having your threads callback a delegate to update your list data, or another shared datastructure that keeps track of where you are in the communication process.
static LinkedList<LeafItem> TreeList = new LinkedList<LeafItem>( );
foreach (LeafItem li in TreeList) {
Thread newThread = new Thread(
new ParameterizedThreadStart(Work.DoWork));
newThread.Start(li);
}
You can enumerate over this in parallel using Parallel.ForEach method (from .NET 4). It has been backported as part of the Rx Framework for use in .NET 3.5sp1.
Note - this doesn't actually use one thread per item, but rather partitions the work using the thread pool, based on the hardware thread count of the system on which you're executing (which is usually better...). In .NET 4, it takes advantage of the ThreadPool's new hill climbing and work stealing algorithms, so is very efficient.
this one is a slight long shot, and I suspect I've messed it up somewhere here :/
basically the idea is to create a custom IEnumerator for your dictionary. The idea being that it contains a static variable that keeps the "location" of the enumeration, for continuing.
the following is some skeleton code for something that does work for pausing and restarting.
public class MyDictEnumerator<T> : IEnumerator<T>
{
private List<T> Dict;
private static int curLocation = -1;
public MyDictEnumerator(List<T> dictionary)
{
Dict = dictionary;
}
public T Current
{
get { return Dict[curLocation]; }
}
public void Dispose()
{ }
object System.Collections.IEnumerator.Current
{
get { return Dict[curLocation]; }
}
public bool MoveNext()
{
curLocation++;
if (curLocation >= Dict.Count)
return false;
return true;
}
public void Reset()
{
curLocation = -1;
}
}
Then to use:
MyDictEnumerator<KeyValuePair<string, int>> enumer = new MyDictEnumerator<KeyValuePair<string, int>>(test.ToList());
while (enumer.MoveNext())
{
Console.WriteLine(enumer.Current.Value);
}
I'll admit that this isn't the cleanest way of doing it. But if you break out of the enumerator, and create a new one on another thread, then it will continue at the same point (i think :/)
I hope this helps.
Edit: from your comments:
My alogrithm is more like: Get the
first target Send the message to the
first target Thread DIES - Catch a
port reception event check if its the
right target do some actions - go to
the next target start the loop over.
If you want to process the items asynchronously but not in parallel, you should be able to achieve this by copying the dictionary's keys to a Queue<string> and passing both to the callback that handles your asynchronous responses.
Your completion handler pseduo-code might look like this:
// first extract your dictionary, key, and queue from whatever state
// object you're using to pass data back to the completion event
if (dictionary.Contains(key)) {
// process the response
}
if (queue.Count > 0) {
string key = queue.Dequeue();
string[] messages = dictionary[key];
// send the messages, along with your state data and this callback
}