The following function I wrote causing the program to crash due to the stack overflow, although the recursion is finite.
public static void Key(char[] chars, int i, int l, string str) {
string newStr=null;
for(int j=0; j<l; j++)
newStr+=chars[i/(int)Math.Pow(68, j)%68];
if(newStr==str)
return;
Key(chars, ++i, l, newStr);
}
When I call the method with these parameters, all goes fine:
Key(chars, 0, 4, "aaaa");
But when it comes to bigger number of calls, it throws the StackOverflowException. So I assume the problem is that althogh the method is finite the call stack gets filled up before the the methods' job is done. So I have a few questions about that:
Why the functions don't get clear from the stack, they are no longer needed, they don't return any value.
And if so, is there a way i could clear the stack manually? I tried the StackTrace class but it's helpless in this case.
1) The function clears when it has ended execution. Calling Key inside of itself means that every call to it will be on the stack until the last call ends, at which stage they will all end in reverse order.
2) You can't clear the stack and carry on with the call.
The stack is still limited. For standard C# applications it is 1 MB. For ASP it is 256 KB. If you need more stack space than that, you'll see the exception.
If you create the thread yourself, you can adjust the stack size using this constructor.
Alternatively, you can rewrite your algorithm so it keeps track of state without using recursion.
It looks like the exit condition of NewStr == Str will never happen and eventually, you will run out of stack.
So, regardless of whether your code reaches its base case or not, your code should never get a stack overflow exception in production.
For example, this should give us a stack overflow right?
private static void Main(string[] args)
{
RecursorKey(0);
}
public static int RecursorKey(int val)
{
return RecursorKey(val ++);
}
In fact it doesn't, if you use .NET 4 and are not debugging and your binary is compiled as release.
This is because the clr is smart enough to do what is called tail recursion. Tail recursion isn't applicable everywhere, but in your case it is, and I was easily able to reproduce your exact problem. In your case, each time you call the function it pushes another stack frame onto the stack, so you get the overflow, even though the algorithm would theoretically end at some point.
To solve your issue, here is how you can enable optimizations.
However, I should point out that Jon Skeet recommends to not rely on tail call optimizations. Given that Jon's a smart guy, I'd listen to it him. If your code is going to encounter large stack depths, try rewriting it without recursion.
Related
I want to ask user to enter a value less than 10. I am using the following code. Which one is better to use? Loop or Recursive method. Someone said me using Recursive function method may cause Memory Leakage. Is it true?
class Program
{
static void Main(string[] args)
{
int x;
do
{
Console.WriteLine("Please Enther a value less than 10.");
x = int.Parse(Console.ReadLine());
} while (x > 10);
//Uncomment the bellow method and comment previous to test the Recursive method
//Value();
}
static string Value()
{
Console.WriteLine("Please Enther a value less than 10.");
return int.Parse(Console.ReadLine()) > 9 ? Value() : "";
}
}
It would probably be a long time before recursion became an issue in this example.
Recursive methods run the risk of causing stack overflow exceptions if they keep running for a long time without completing. This is because each method call results in data being stored on the stack (which has very limited space) - more info here:
https://en.wikipedia.org/wiki/Stack_overflow#:~:text=The%20most%2Dcommon%20cause%20of%20stack%20overflow%20is%20excessively%20deep%20or%20infinite%20recursion%2C%20in%20which%20a%20function%20calls%20itself%20so%20many%20times%20that%20the%20space%20needed%20to%20store%20the%20variables%20and%20information%20associated%20with%20each%20call%20is%20more%20than%20can%20fit%20on%20the%20stack.
In your case unless they enter a number greater than or equal to 10 loads of times or you have very little memory it should be fine.
Generally it's better to use loops than recursion as they are simpler to understand. Recursion is a useful tool for achieving good performance in certain scenarios but generally loops should be preferred.
Recursive functions are used when you have a base case and a regular case. The base case is vital, since it marks the end of recursion. For example, you can create a function factorial(n) to calculate the factorial of a number n. The base case happens when n reaches 1 and you just return 1, while in the regular case you just multiply n by factorial(n - 1).
In general (there are a few optimization cases in which you can save memory), recursive functions create a new stack frame for each call. So, for factorial(3), there are at least three stack frames being created, the factorial(3) itself, the factorial(2) one, and finally the one that finished recursions which would be factorial(1).
At least, in that case you know when you are going to finish, and how much memory you need, so a compiler can work with that in advance.
All that discussion above means that you are misunderstanding recursion if you think you can use it in order to validate user input. There can be only one call with the correct answer, which would be the base case, or even hundreds or thousands or regular case instances as well. This has the potential of overflowing the stack of your program, without any way to prevent that, on the part of the compiler or on your part.
Another way to see this is that recursion is used as a way of abstraction: you specify what needs to happen, even near to the mathematical problem, and not how it should happen. In your example, that level of abstraction is unneeded.
Let's think of it as a family tree, a father has kids, those kids have kids, those kids have kids, etc...
So I have a recursive function that gets the father uses Recursion to get the children and for now just print them to debug output window...But at some point ( after one hour of letting it run and printing like 26000 rows) it gives me a StackOverFlowException.
So Am really running out of memory? hmmm? then shouldn't I get an "Out of memory exception"? on other posts I found people were saying if the number of recursive calls are too much, you might still get a SOF exception...
Anyway, my first thought was to break the tree into smaller sub-strees..so I know for a fact that my root father always has these five kids, so Instead of Calling my method one time with root passed to it, I said ok call it five times with Kids of root Passes to it.. It helped I think..but still one of them is so big - 26000 rows when it crashes - and still have this issue.
How about Application Domains and Creating new Processes at run time at some certain level of depth? Does that help?
How about creating my own Stack and using that instead of recursive methods? does that help?
here is also a high-level of my code, please take a look, maybe there is actually something silly wrong with this that causes SOF error:
private void MyLoadMethod(string conceptCKI)
{
// make some script calls to DB, so that moTargetConceptList2 will have Concept-Relations for the current node.
// when this is zero, it means its a leaf.
int numberofKids = moTargetConceptList2.ConceptReltns.Count();
if (numberofKids == 0)
return;
for (int i = 1; i <= numberofKids; i++)
{
oUCMRConceptReltn = moTargetConceptList2.ConceptReltns.get_ItemByIndex(i, false);
//Get the concept linked to the relation concept
if (oUCMRConceptReltn.SourceCKI == sConceptCKI)
{
oConcept = moTargetConceptList2.ItemByKeyConceptCKI(oUCMRConceptReltn.TargetCKI, false);
}
else
{
oConcept = moTargetConceptList2.ItemByKeyConceptCKI(oUCMRConceptReltn.SourceCKI, false);
}
//builder.AppendLine("\t" + oConcept.PrimaryCTerm.SourceString);
Debug.WriteLine(oConcept.PrimaryCTerm.SourceString);
MyLoadMethod(oConcept.ConceptCKI);
}
}
How about creating my own Stack and using that instead of recursive methods? does that help?
Yes!
When you instantiate a Stack<T> this will live on the heap and can grow arbitrarily large (until you run out of addressable memory).
If you use recursion you use the call stack. The call stack is much smaller than the heap. The default is 1 MB of call stack space per thread. Note this can be changed, but it's not advisable.
StackOverflowException is quite different to OutOfMemoryException.
OOME means that there is no memory available to the process at all. This could be upon trying to create a new thread with a new stack, or in trying to create a new object on the heap (and a few other cases).
SOE means that the thread's stack - by default 1M, though it can be set differently in thread creation or if the executable has a different default; hence ASP.NET threads have 256k as a default rather than 1M - was exhausted. This could be upon calling a method, or allocating a local.
When you call a function (method or property), the arguments of the call are placed on the stack, the address the function should return to when it returns are put on the stack, then execution jumps to the function called. Then some locals will be placed on the stack. Some more may be placed on it as the function continues to execute. stackalloc will also explicitly use some stack space where otherwise heap allocation would be used.
Then it calls another function, and the same happens again. Then that function returns, and execution jumps back to the stored return address, and the pointer within the stack moves back up (no need to clean up the values placed on the stack, they're just ignored now) and that space is available again.
If you use up that 1M of space, you get a StackOverflowException. Because 1M (or even 256k) is a large amount of memory for these such use (we don't put really large objects in the stack) the three things that are likely to cause an SOE are:
Someone thought it would be a good idea to optimise by using stackalloc when it wasn't, and they used up that 1M fast.
Someone thought it would be a good idea to optimise by creating a thread with a smaller than usual stack when it wasn't, and they use up that tiny stack.
A recursive (whether directly or through several steps) call falls into an infinite loop.
It wasn't quite infinite, but it was large enough.
You've got case 4. 1 and 2 are quite rare (and you need to be quite deliberate to risk them). Case 3 is by far the most common, and indicates a bug in that the recursion shouldn't be infinite, but a mistake means it is.
Ironically, in this case you should be glad you took the recursive approach rather than iterative - the SOE reveals the bug and where it is, while with an iterative approach you'd probably have an infinite loop bringing everything to a halt, and that can be harder to find.
Now for case 4, we've got two options. In the very very rare cases where we've got just slightly too many calls, we can run it on a thread with a larger stack. This doesn't apply to you.
Instead, you need to change from a recursive approach to an iterative one. Most of the time, this isn't very hard thought it can be fiddly. Instead of calling itself again, the method uses a loop. For example, consider the classic teaching-example of a factorial method:
private static int Fac(int n)
{
return n <= 1 ? 1 : n * Fac(n - 1);
}
Instead of using recursion we loop in the same method:
private static int Fac(int n)
{
int ret = 1;
for(int i = 1; i <= n, ++i)
ret *= i;
return ret;
}
You can see why there's less stack space here. The iterative version will also be faster 99% of the time. Now, imagine we accidentally call Fac(n) in the first, and leave out the ++i in the second - the equivalent bug in each, and it causes an SOE in the first and a program that never stops in the second.
For the sort of code you're talking about, where you keep producing more and more results as you go based on previous results, you can place the results you've got in a data-structure (Queue<T> and Stack<T> both serve well for a lot of cases) so the code becomes something like):
private void MyLoadMethod(string firstConceptCKI)
{
Queue<string> pendingItems = new Queue<string>();
pendingItems.Enqueue(firstConceptCKI);
while(pendingItems.Count != 0)
{
string conceptCKI = pendingItems.Dequeue();
// make some script calls to DB, so that moTargetConceptList2 will have Concept-Relations for the current node.
// when this is zero, it means its a leaf.
int numberofKids = moTargetConceptList2.ConceptReltns.Count();
for (int i = 1; i <= numberofKids; i++)
{
oUCMRConceptReltn = moTargetConceptList2.ConceptReltns.get_ItemByIndex(i, false);
//Get the concept linked to the relation concept
if (oUCMRConceptReltn.SourceCKI == sConceptCKI)
{
oConcept = moTargetConceptList2.ItemByKeyConceptCKI(oUCMRConceptReltn.TargetCKI, false);
}
else
{
oConcept = moTargetConceptList2.ItemByKeyConceptCKI(oUCMRConceptReltn.SourceCKI, false);
}
//builder.AppendLine("\t" + oConcept.PrimaryCTerm.SourceString);
Debug.WriteLine(oConcept.PrimaryCTerm.SourceString);
pendingItems.Enque(oConcept.ConceptCKI);
}
}
}
(I haven't completely checked this, just added the queuing instead of recursing to the code in your question).
This should then do more or less the same as your code, but iteratively. Hopefully that means it'll work. Note that there is a possible infinite loop in this code if the data you are retrieving has a loop. In that case this code will throw an exception when it fills the queue with far too much stuff to cope. You can either debug the source data, or use a HashSet to avoid enqueuing items that have already been processed.
Edit: Better add how to use a HashSet to catch duplicates. First set up a HashSet, this could just be:
HashSet<string> seen = new HashSet<string>();
Or if the strings are used case-insensitively, you'd be better with:
HashSet<string> seen = new HashSet<string>(StringComparison.InvariantCultureIgnoreCase) // or StringComparison.CurrentCultureIgnoreCase if that's closer to how the string is used in the rest of the code.
Then before you go to use the string (or perhaps before you go to add it to the queue, you have one of the following:
If duplicate strings shouldn't happen:
if(!seen.Add(conceptCKI))
throw new InvalidOperationException("Attempt to use \" + conceptCKI + "\" which was already seen.");
Or if duplicate strings are valid, and we just want to skip performing the second call:
if(!seen.Add(conceptCKI))
continue;//skip rest of loop, and move on to the next one.
I think you have a recursion's ring (infinite recursion), not a really stack overflow error. If you are got more memory for stack - you will get the overflow error too.
For test it:
Declare a global variable for storing a operable objects:
private Dictionary<int,object> _operableIds = new Dictionary<int,object>();
...
private void Start()
{
_operableIds.Clear();
Recurtion(start_id);
}
...
private void Recurtion(int object_id)
{
if(_operableIds.ContainsKey(object_id))
throw new Exception("Have a ring!");
else
_operableIds.Add(object_id, null/*or object*/);
...
Recurtion(other_id)
...
_operableIds.Remove(object_id);
}
I'm getting a stack overflow exception for a segment of code that doesn't seem to be able to produce a stackoverflow... It looks like this:
public String WriteToFile(XmlDocument pDoc, String pPath)
{
string source = "";
string seq = "";
string sourcenet = "";
XmlNodelist sourceNode = pDoc.GetElementsByTagName(XmlUtils.Nodes.Source);
source = sourceNode.Item(0).InnerText;
XmlNodelist sqList= pDoc.GetElementsByTagName(XmlUtils.Nodes.Seq);
seq = sqList.Item(0).InnerText;
XmlNodelist sourceNets = pDoc.GetElementsByTagName(XmlUtils.Nodes.SourceNets);
sourcenet = sourceNets.Item(0).InnerText;
string fileName = Folders.GetMyFileName(source, seq, sourcenet);
string fullPath = Path.Combine(pPath, fileName);
pDoc.Save(pFullPathFile); <--- Stackoverflow is raised here
return pFullPathFile;
}
There are no recursive calls, if you examine the call stack it has a depth of 2 before going to "external code" (which I'm guessing is not that external but part of the framework that starts the thread, which has debugging turn off).
¿Is there anyway the exception can be risen because anything other than a recursive call? It does ALWAYS fails in the pDoc.Save method call... and pDoc isn't actually that big... more like 32KB of data...
A stack overflow exception can occur any time that the stack exceeds it's maximum size. This is mostly commonly done with by ...
Having a deeply nested stack which is not recursive. Think of event storms where event A leads to event B which leads to event C all of which have handlers that deeply grow the stack.
Having a shallow stack which occurs after some large stack allocations
Stack overflow simply means you have exhausted the stack, it doesn't need to be caused by recursion. Of course, because recursion utilizes the stack, it is often the cause of a stack overflow exception, but it doesn't need to be.
That being said, with the information you provided, it doesn't sound like there should be anything causing a stack overflow in the code you provided.
Threads in C# have a 1MB stack by default, but you can create a new thread with a smaller stack. Do you create threads yourself in this program, and are you setting the stack size ?
Also, have a look at the external code section (right click where it says External Code in the Call Stack window, choose "Show external code"). See if something looks wrong, is the framework for some reason going through a lot of method calls to do the save ?
There is indeed a recursive call.
pDoc.Save() calls WriteTo(XmlWriter w) on the document, which calls WriteContentTo(XmlWriter w).
This then calls WriteTo(XmlWriter w) on the all the nodes at the root level, which will contain one element node (possibly also some comments, whitespace, processing instructions, a document declarataion...).
On that element, this will cause it to write its tag ('<', element name, and then any attributes) followed by calling WriteContentTo(XmlWriter w) which calls WriteTo(XmlWriter w) on every child element, which calls WriteContentTo(XmlWriter w), and so on and so on.
Hence this is indeed recursive in how each element calls the same method on its child elements, and with a sufficiently deep document on sufficiently small stack space (default is 1MB on most applications, but 256KB on ASP.NET), you'll have a stack overflow.
For the record, you can also have a stack overflow without recursion as long as you burn through your stack space one way or another. stackalloc is a great way to find yourself doing this while only a few calls deep.
If you're in trouble due to this recursion, then remember that the implementation of WriteTo is essentially (manually inlining WriteContentTo into it):
w.WriteStartElement(this.Prefix, this.LocalName, this.NamespaceURI);
if (this.HasAttributes)
{
XmlAttributeCollection attributes = this.Attributes;
for (int i = 0; i < attributes.Count; i++)
{
attributes[i].WriteTo(w);
}
}
if (this.IsEmpty)
{
w.WriteEndElement();
}
else
{
for (XmlNode node = this.FirstChild; node != null; node = node.NextSibling)
{
node.WriteTo(w);
}
w.WriteFullEndElement();
}
Replace this with an iterative version, and you won't overflow the stack. Of course if you've somehow managed to put the document into a condition where it's got an element that's an ancestor to itself (does XmlDocument protect against that? I don't know off the top of my head), then it'll turn a stack-overflow into an infinite loop, which if anything is worse.
In some languages/runtimes a stack overflow can happen because of large memory allocations that are unrelated to the call stack itself. It's entirely possible that the 'external code' (I assume the framework) is running either into that situation or has actually a classic recursion overflow problem you can't see because you can't necessarily debug into it.
I have a recursive function in a BaseClass which relies on a protected virtual function for it's return condition.
It's possible for a child class to override this function incorrectly and leads to a StackOverFlow exception. The worst thing is there are some slow network calls and the exception is not gonna happen soon (many resources waste for a long time).
I'm looking for a method to check StackOverFlow in early stages some way in the base class (maybe using Reflection and the current recursion level).
Any idea ?
You could pass a simple integer 'depth' to the recursive function and increment it with each subsequent call. If it gets larger than the maximum allowed depth throw an Exception right then instead of waiting until it's too late and the dreaded StackOverflow exception has occurred.
Such safety mechanisms (increment counter, check it's not stupidly large) can also be handy in while loops where a small error can cause an infinite loop consuming vast quantities of CPU.
In large systems with many users (e.g. web sites) sometimes it's best to take precautionary measures like these with recursion and while loops because the consequences can reach far beyond one web page or one user of the system. It's not pretty code and the purists will no doubt balk at it, but it's efficient, it's defensive and it's pragmatic.
Solve the problem instead of creating a workaround. Create a private function which is recursive which calls the protected virtual function.
Although you probably can read the call stack and analyze it I wouldn't do that.
It will slow down execution
It is not your base class' responsibility
Document your base class' behaviour
An alternative could be to do the call stack analysis in DEBUG mode only. Here is a little code to see how to get the call stack.
using System.Diagnostics;
[STAThread]
public static void Main()
{
StackTrace stackTrace = new StackTrace(); // get call stack
StackFrame[] stackFrames = stackTrace.GetFrames(); // get method calls (frames)
// write call stack method names
foreach (StackFrame stackFrame in stackFrames)
{
Console.WriteLine(stackFrame.GetMethod().Name); // write method name
}
}
From this site
I've been trying to find an answer to this question for a few hours now on the web and on this site, and I'm not quite there.
I understand that .NET allocates 1MB to apps, and that it's best to avoid stack overflow by recoding instead of forcing stack size.
I'm working on a "shortest path" app that works great up to about 3000 nodes, at which point it overflows. Here's the method that causes problems:
public void findShortestPath(int current, int end, int currentCost)
{
if (!weight.ContainsKey(current))
{
weight.Add(current, currentCost);
}
Node currentNode = graph[current];
var sortedEdges = (from entry in currentNode.edges orderby entry.Value ascending select entry);
foreach (KeyValuePair<int, int> nextNode in sortedEdges)
{
if (!visited.ContainsKey(nextNode.Key) || !visited[nextNode.Key])
{
int nextNodeCost = currentCost + nextNode.Value;
if (!weight.ContainsKey(nextNode.Key))
{
weight.Add(nextNode.Key, nextNodeCost);
}
else if (weight[nextNode.Key] > nextNodeCost)
{
weight[nextNode.Key] = nextNodeCost;
}
}
}
visited.Add(current, true);
foreach (KeyValuePair<int, int> nextNode in sortedEdges)
{
if(!visited.ContainsKey(nextNode.Key) || !visited[nextNode.Key]){
findShortestPath(nextNode.Key, end, weight[nextNode.Key]);
}
}
}//findShortestPath
For reference, the Node class has one member:
public Dictionary<int, int> edges = new Dictionary<int, int>();
graph[] is:
private Dictionary<int, Node> graph = new Dictonary<int, Node>();
I've tried to opimize the code so that it isn't carrying any more baggage than needed from one iteration (recursion?) to the next, but with a 100K-Node graph with each node having between 1-9 edges it's going to hit that 1MB limit pretty quickly.
Anyway, I'm new to C# and code optimization, if anyone could give me some pointers (not like this) I would appreciate it.
The classic technique to avoid deep recursive stack dives is to simply avoid recursion by writing the algorithm iteratively and managing your own "stack" with an appropriate list data structure. Most likely you will need this approach here given the sheer size of your input set.
A while back I explored this problem in my blog. Or, rather, I explored a related problem: how do you find the depth of a binary tree without using recursion? A recursive tree depth solution is trivial, but blows the stack if the tree is highly imbalanced.
My recommendation is to study ways of solving this simpler problem, and then decide which of them, if any, could be adapted to your slightly more complex algorithm.
Note that in these articles the examples are given entirely in JScript. However, it should not be difficult to adapt them to C#.
Here we start by defining the problem.
http://blogs.msdn.com/ericlippert/archive/2005/07/27/recursion-part-one-recursive-data-structures-and-functions.aspx
The first attempt at a solution is the classic technique that you'll probably adopt: define an explicit stack; use it rather than relying upon the operating system and compiler implementing the stack for you. This is what most people do when faced with this problem.
http://blogs.msdn.com/ericlippert/archive/2005/08/01/recursion-part-two-unrolling-a-recursive-function-with-an-explicit-stack.aspx
The problem with that solution is that it's a bit of a mess. We can go even farther than simply making our own stack. We can make our own little domain-specific virtual machine that has its own heap-allocated stack, and then solve the problem by writing a program that targets that machine! This is actually easier than it sounds; the operations of the machine can be extremely high level.
http://blogs.msdn.com/ericlippert/archive/2005/08/04/recursion-part-three-building-a-dispatch-engine.aspx
And finally, if you are really a glutton for punishment (or a compiler developer) you can rewrite your program in Continuation Passing Style, thereby eliminating the need for a stack at all:
http://blogs.msdn.com/ericlippert/archive/2005/08/08/recursion-part-four-continuation-passing-style.aspx
http://blogs.msdn.com/ericlippert/archive/2005/08/11/recursion-part-five-more-on-cps.aspx
http://blogs.msdn.com/ericlippert/archive/2005/08/15/recursion-part-six-making-cps-work.aspx
CPS is a particularly clever way of moving the implicit stack data structure off the system stack and onto the heap by encoding it in the relationships between a bunch of delegates.
Here are all of my articles on recursion:
http://blogs.msdn.com/ericlippert/archive/tags/Recursion/default.aspx
You could convert the code to use a 'work queue' rather than being recursive. Something along the following pseudocode:
Queue<Task> work;
while( work.Count != 0 )
{
Task t = work.Dequeue();
... whatever
foreach(Task more in t.MoreTasks)
work.Enqueue(more);
}
I know that is cryptic but it's the basic concept of what you'll need to do. Since your only getting 3000 nodes with your current code, you will at best get to 12~15k without any parameters. So you need to kill the recursion completely.
Is your Node a struct or a class? If it's the former, make it a class so that it's allocated on the heap instead of on the stack.
I would first verify that you are actually overflowing the stack: you actually see a StackOverflowException get thrown by the runtime.
If this is indeed the case, you have a few options:
Modify your recursive function so that the .NET runtime can convert it into a tail-recursive function.
Modify your recursive function so that it is iterative and uses a custom data structure rather than the managed stack.
Option 1 is not always possible, and assumes that the rules the CLR uses to generate tail recursive calls will remain stable in the future. The primary benefit, is that when possible, tail recursion is actually a convenient way of writing recursive algorithms without sacrificing clarity.
Option 2 is a more work, but is not sensitive to the implementation of the CLR and can be implemented for any recursive algorithm (where tail recursion may not always be possible). Generally, you need to capture and pass state information between iterations of some loop, together with information on how to "unroll" the data structure that takes the places of the stack (typically a List<> or Stack<>). One way of unrolling recursion into iteration is through continuation passing pattern.
More resources on C# tail recursion:
Why doesn't .NET/C# optimize for tail-call recursion?
http://geekswithblogs.net/jwhitehorn/archive/2007/06/06/113060.aspx
I would first make sure I know why I'm getting a stack overflow. Is it actually because of the recursion? The recursive method isn't putting much onto the stack. Maybe it's because of the storage of the nodes?
Also, BTW, I don't see the end parameter ever changing. That suggests it doesn't need to be a parameter, carried on each stack frame.