Infinite state machine with an IDisposable

Infinite state machine with an IDisposable - c#

Lets say I have an infinite state machine to generate random md5 hashes:
public static IEnumerable<string> GetHashes()
{
using (var hash = System.Security.Cryptography.MD5.Create())
{
while (true)
yield return hash.ComputeHash(Guid.NewGuid().ToByteArray());
}
}
In the above example I use an using statement. Will the .Dispose() method ever be called? CQ, will the unmanaged resources ever be freed?
For example, if I use the machine as follows:
public static void Test()
{
int counter = 0;
var hashes = GetHashes();
foreach(var md5 in hashes)
{
Console.WriteLine(md5);
counter++;
if (counter > 10)
break;
}
}
Since the hashes variable will go out of scope (and I presume garbage collected) will the dispose method be called to free the resources used by System.Security.Cryptography.MD5 or is this a memory leak?

Let's change your original code blocks a bit to boil it down to the essentials, while still keeping it interesting enough to analyze. This is not exactly equivalent to what you posted, but we're still using the value of the iterator.
class Disposable : IDisposable {
public void Dispose() {
Console.WriteLine("Disposed!");
}
}
IEnumerable<int> CreateEnumerable() {
int i = 0;
using (var d = new Disposable()) {
while (true) yield return ++i;
}
}
void UseEnumerable() {
foreach (int i in CreateEnumerable()) {
Console.WriteLine(i);
if (i == 10) break;
}
}
This will print the numbers from 1 to 10 before printing Disposed!
What actually happens under the covers? A whole lot more. Let's tackle the outer layer first, UseEnumerable. The foreach is syntactic sugar for the following:
var e = CreateEnumerable().GetEnumerator();
try {
while (e.MoveNext()) {
int i = e.Current;
Console.WriteLine(i);
if (i == 10) break;
}
} finally {
e.Dispose();
}
For the exact details (because even this is simplified, a little) I refer you to the C# language specification, section 8.8.4. The important bit here is that a foreach entails an implicit call to the Dispose of the enumerator.
Next, the using statement in CreateEnumerable is syntactic sugar as well. In fact, let's write out the whole thing in primitive statements so we can make more sense of the translation later:
IEnumerable<int> CreateEnumerable() {
int i = 0;
Disposable d = new Disposable();
try {
repeat:
i = i + 1;
yield return i;
goto repeat;
} finally {
d.Dispose();
}
}
The exact rules for implementation of iterator blocks are detailed in section 10.14 of the language specification. They're given in terms of abstract operations, not code. A good discussion on what kind of code is generated by the C# compiler and what each part does is given in C# in Depth, but I'm going to give a simple translation instead that still complies with the specification. To reiterate, this is not what the compiler will actually produce, but it's a good enough approximation to illustrate what's happening and leaves out the more hairy bits that deal with threading and optimization.
class CreateEnumerable_Enumerator : IEnumerator<int> {
// local variables are promoted to instance fields
private int i;
private Disposable d;
// implementation of Current
private int current;
public int Current => current;
object IEnumerator.Current => current;
// State machine
enum State { Before, Running, Suspended, After };
private State state = State.Before;
// Section 10.14.4.1
public bool MoveNext() {
switch (state) {
case State.Before: {
state = State.Running;
// begin iterator block
i = 0;
d = new Disposable();
i = i + 1;
// yield return occurs here
current = i;
state = State.Suspended;
return true;
}
case State.Running: return false; // can't happen
case State.Suspended: {
state = State.Running;
// goto repeat
i = i + 1;
// yield return occurs here
current = i;
state = State.Suspended;
return true;
}
case State.After: return false;
default: return false; // can't happen
}
}
// Section 10.14.4.3
public void Dispose() {
switch (state) {
case State.Before: state = State.After; break;
case State.Running: break; // unspecified
case State.Suspended: {
state = State.Running;
// finally occurs here
d.Dispose();
state = State.After;
}
break;
case State.After: return;
default: return; // can't happen
}
}
public void Reset() { throw new NotImplementedException(); }
}
class CreateEnumerable_Enumerable : IEnumerable<int> {
public IEnumerator<int> GetEnumerator() {
return new CreateEnumerable_Enumerator();
}
IEnumerator IEnumerable.GetEnumerator() {
return GetEnumerator();
}
}
IEnumerable<int> CreateEnumerable() {
return new CreateEnumerable_Enumerable();
}
The essential bit here is that the code block is split up at the occurrences of a yield return or yield break statement, with the iterator responsible for remembering "where we were" at the time of the interruption. Any finally blocks in the body are deferred until the Dispose. The infinite loop in your code is really not an infinite loop anymore, because it's interrupted by periodic yield return statements. Note that, because the finally block isn't actually a finally block anymore, it getting executed is a little less certain when you're dealing with iterators. This is why using foreach (or any other way that ensures the Dispose method of the iterator is called in a finally block) is essential.
This is a simplified example; things get much more interesting when you make the loop more complex, introduce exceptions, etcetera. The burden of "just making this work" is on the compiler.

Largely, it depends on how you code it. But in your example, Dispose will be called.
Here's an explanation on how iterators get compiled.
And specifically, talking about finally:
Iterators pose an awkward problem. Instead of the whole method executing before the stack frame is popped, execution effectively pauses each time a value is yielded. There's no way of guaranteeing that the caller will ever use the iterator again, in any way, shape or form. If you require some more code to be executed at some point after the value is yielded, you're in trouble: you can't guarantee it will happen. To cut to the chase, code in a finally block which would normally be executed in almost all circumstances before leaving the method can't be relied on quite as much.
...
The state machine is built so that finally blocks are executed when an iterator is used properly, however. That's because IEnumerator implements IDisposable, and the C# foreach loop calls Dispose on iterators (even the nongeneric IEnumerator ones, if they implement IDisposable). The IDisposable implementation in the generated iterator works out which finally blocks are relevant to the current position (based on the state, as always) and execute the appropriate code.

Related

Trying to merge sort a linked list and getting a stackoverflow exception. How might I prevent that? [duplicate]

I would like to either prevent or handle a StackOverflowException that I am getting from a call to the XslCompiledTransform.Transform method within an Xsl Editor I am writing. The problem seems to be that the user can write an Xsl script that is infinitely recursive, and it just blows up on the call to the Transform method. (That is, the problem is not just the typical programmatic error, which is usually the cause of such an exception.)
Is there a way to detect and/or limit how many recursions are allowed? Or any other ideas to keep this code from just blowing up on me?

From Microsoft:
Starting with the .NET Framework
version 2.0, a StackOverflowException
object cannot be caught by a try-catch
block and the corresponding process is
terminated by default. Consequently,
users are advised to write their code
to detect and prevent a stack
overflow. For example, if your
application depends on recursion, use
a counter or a state condition to
terminate the recursive loop.
I'm assuming the exception is happening within an internal .NET method, and not in your code.
You can do a couple things.
Write code that checks the xsl for infinite recursion and notifies the user prior to applying a transform (Ugh).
Load the XslTransform code into a separate process (Hacky, but less work).
You can use the Process class to load the assembly that will apply the transform into a separate process, and alert the user of the failure if it dies, without killing your main app.
EDIT: I just tested, here is how to do it:
MainProcess:
// This is just an example, obviously you'll want to pass args to this.
Process p1 = new Process();
p1.StartInfo.FileName = "ApplyTransform.exe";
p1.StartInfo.UseShellExecute = false;
p1.StartInfo.WindowStyle = ProcessWindowStyle.Hidden;
p1.Start();
p1.WaitForExit();
if (p1.ExitCode == 1)
Console.WriteLine("StackOverflow was thrown");
ApplyTransform Process:
class Program
{
static void Main(string[] args)
{
AppDomain.CurrentDomain.UnhandledException += new UnhandledExceptionEventHandler(CurrentDomain_UnhandledException);
throw new StackOverflowException();
}
// We trap this, we can't save the process,
// but we can prevent the "ILLEGAL OPERATION" window
static void CurrentDomain_UnhandledException(object sender, UnhandledExceptionEventArgs e)
{
if (e.IsTerminating)
{
Environment.Exit(1);
}
}
}

NOTE The question in the bounty by #WilliamJockusch and the original question are different.
This answer is about StackOverflow's in the general case of third-party libraries and what you can/can't do with them. If you're looking about the special case with XslTransform, see the accepted answer.
Stack overflows happen because the data on the stack exceeds a certain limit (in bytes). The details of how this detection works can be found here.
I'm wondering if there is a general way to track down StackOverflowExceptions. In other words, suppose I have infinite recursion somewhere in my code, but I have no idea where. I want to track it down by some means that is easier than stepping through code all over the place until I see it happening. I don't care how hackish it is.
As I mentioned in the link, detecting a stack overflow from static code analysis would require solving the halting problem which is undecidable. Now that we've established that there is no silver bullet, I can show you a few tricks that I think helps track down the problem.
I think this question can be interpreted in different ways, and since I'm a bit bored :-), I'll break it down into different variations.
Detecting a stack overflow in a test environment
Basically the problem here is that you have a (limited) test environment and want to detect a stack overflow in an (expanded) production environment.
Instead of detecting the SO itself, I solve this by exploiting the fact that the stack depth can be set. The debugger will give you all the information you need. Most languages allow you to specify the stack size or the max recursion depth.
Basically I try to force a SO by making the stack depth as small as possible. If it doesn't overflow, I can always make it bigger (=in this case: safer) for the production environment. The moment you get a stack overflow, you can manually decide if it's a 'valid' one or not.
To do this, pass the stack size (in our case: a small value) to a Thread parameter, and see what happens. The default stack size in .NET is 1 MB, we're going to use a way smaller value:
class StackOverflowDetector
{
static int Recur()
{
int variable = 1;
return variable + Recur();
}
static void Start()
{
int depth = 1 + Recur();
}
static void Main(string[] args)
{
Thread t = new Thread(Start, 1);
t.Start();
t.Join();
Console.WriteLine();
Console.ReadLine();
}
}
Note: we're going to use this code below as well.
Once it overflows, you can set it to a bigger value until you get a SO that makes sense.
Creating exceptions before you SO
The StackOverflowException is not catchable. This means there's not much you can do when it has happened. So, if you believe something is bound to go wrong in your code, you can make your own exception in some cases. The only thing you need for this is the current stack depth; there's no need for a counter, you can use the real values from .NET:
class StackOverflowDetector
{
static void CheckStackDepth()
{
if (new StackTrace().FrameCount > 10) // some arbitrary limit
{
throw new StackOverflowException("Bad thread.");
}
}
static int Recur()
{
CheckStackDepth();
int variable = 1;
return variable + Recur();
}
static void Main(string[] args)
{
try
{
int depth = 1 + Recur();
}
catch (ThreadAbortException e)
{
Console.WriteLine("We've been a {0}", e.ExceptionState);
}
Console.WriteLine();
Console.ReadLine();
}
}
Note that this approach also works if you are dealing with third-party components that use a callback mechanism. The only thing required is that you can intercept some calls in the stack trace.
Detection in a separate thread
You explicitly suggested this, so here goes this one.
You can try detecting a SO in a separate thread.. but it probably won't do you any good. A stack overflow can happen fast, even before you get a context switch. This means that this mechanism isn't reliable at all... I wouldn't recommend actually using it. It was fun to build though, so here's the code :-)
class StackOverflowDetector
{
static int Recur()
{
Thread.Sleep(1); // simulate that we're actually doing something :-)
int variable = 1;
return variable + Recur();
}
static void Start()
{
try
{
int depth = 1 + Recur();
}
catch (ThreadAbortException e)
{
Console.WriteLine("We've been a {0}", e.ExceptionState);
}
}
static void Main(string[] args)
{
// Prepare the execution thread
Thread t = new Thread(Start);
t.Priority = ThreadPriority.Lowest;
// Create the watch thread
Thread watcher = new Thread(Watcher);
watcher.Priority = ThreadPriority.Highest;
watcher.Start(t);
// Start the execution thread
t.Start();
t.Join();
watcher.Abort();
Console.WriteLine();
Console.ReadLine();
}
private static void Watcher(object o)
{
Thread towatch = (Thread)o;
while (true)
{
if (towatch.ThreadState == System.Threading.ThreadState.Running)
{
towatch.Suspend();
var frames = new System.Diagnostics.StackTrace(towatch, false);
if (frames.FrameCount > 20)
{
towatch.Resume();
towatch.Abort("Bad bad thread!");
}
else
{
towatch.Resume();
}
}
}
}
}
Run this in the debugger and have fun of what happens.
Using the characteristics of a stack overflow
Another interpretation of your question is: "Where are the pieces of code that could potentially cause a stack overflow exception?". Obviously the answer of this is: all code with recursion. For each piece of code, you can then do some manual analysis.
It's also possible to determine this using static code analysis. What you need to do for that is to decompile all methods and figure out if they contain an infinite recursion. Here's some code that does that for you:
// A simple decompiler that extracts all method tokens (that is: call, callvirt, newobj in IL)
internal class Decompiler
{
private Decompiler() { }
static Decompiler()
{
singleByteOpcodes = new OpCode[0x100];
multiByteOpcodes = new OpCode[0x100];
FieldInfo[] infoArray1 = typeof(OpCodes).GetFields();
for (int num1 = 0; num1 < infoArray1.Length; num1++)
{
FieldInfo info1 = infoArray1[num1];
if (info1.FieldType == typeof(OpCode))
{
OpCode code1 = (OpCode)info1.GetValue(null);
ushort num2 = (ushort)code1.Value;
if (num2 < 0x100)
{
singleByteOpcodes[(int)num2] = code1;
}
else
{
if ((num2 & 0xff00) != 0xfe00)
{
throw new Exception("Invalid opcode: " + num2.ToString());
}
multiByteOpcodes[num2 & 0xff] = code1;
}
}
}
}
private static OpCode[] singleByteOpcodes;
private static OpCode[] multiByteOpcodes;
public static MethodBase[] Decompile(MethodBase mi, byte[] ildata)
{
HashSet<MethodBase> result = new HashSet<MethodBase>();
Module module = mi.Module;
int position = 0;
while (position < ildata.Length)
{
OpCode code = OpCodes.Nop;
ushort b = ildata[position++];
if (b != 0xfe)
{
code = singleByteOpcodes[b];
}
else
{
b = ildata[position++];
code = multiByteOpcodes[b];
b |= (ushort)(0xfe00);
}
switch (code.OperandType)
{
case OperandType.InlineNone:
break;
case OperandType.ShortInlineBrTarget:
case OperandType.ShortInlineI:
case OperandType.ShortInlineVar:
position += 1;
break;
case OperandType.InlineVar:
position += 2;
break;
case OperandType.InlineBrTarget:
case OperandType.InlineField:
case OperandType.InlineI:
case OperandType.InlineSig:
case OperandType.InlineString:
case OperandType.InlineTok:
case OperandType.InlineType:
case OperandType.ShortInlineR:
position += 4;
break;
case OperandType.InlineR:
case OperandType.InlineI8:
position += 8;
break;
case OperandType.InlineSwitch:
int count = BitConverter.ToInt32(ildata, position);
position += count * 4 + 4;
break;
case OperandType.InlineMethod:
int methodId = BitConverter.ToInt32(ildata, position);
position += 4;
try
{
if (mi is ConstructorInfo)
{
result.Add((MethodBase)module.ResolveMember(methodId, mi.DeclaringType.GetGenericArguments(), Type.EmptyTypes));
}
else
{
result.Add((MethodBase)module.ResolveMember(methodId, mi.DeclaringType.GetGenericArguments(), mi.GetGenericArguments()));
}
}
catch { }
break;
default:
throw new Exception("Unknown instruction operand; cannot continue. Operand type: " + code.OperandType);
}
}
return result.ToArray();
}
}
class StackOverflowDetector
{
// This method will be found:
static int Recur()
{
CheckStackDepth();
int variable = 1;
return variable + Recur();
}
static void Main(string[] args)
{
RecursionDetector();
Console.WriteLine();
Console.ReadLine();
}
static void RecursionDetector()
{
// First decompile all methods in the assembly:
Dictionary<MethodBase, MethodBase[]> calling = new Dictionary<MethodBase, MethodBase[]>();
var assembly = typeof(StackOverflowDetector).Assembly;
foreach (var type in assembly.GetTypes())
{
foreach (var member in type.GetMembers(BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Static | BindingFlags.Instance).OfType<MethodBase>())
{
var body = member.GetMethodBody();
if (body!=null)
{
var bytes = body.GetILAsByteArray();
if (bytes != null)
{
// Store all the calls of this method:
var calls = Decompiler.Decompile(member, bytes);
calling[member] = calls;
}
}
}
}
// Check every method:
foreach (var method in calling.Keys)
{
// If method A -> ... -> method A, we have a possible infinite recursion
CheckRecursion(method, calling, new HashSet<MethodBase>());
}
}
Now, the fact that a method cycle contains recursion, is by no means a guarantee that a stack overflow will happen - it's just the most likely precondition for your stack overflow exception. In short, this means that this code will determine the pieces of code where a stack overflow can occur, which should narrow down most code considerably.
Yet other approaches
There are some other approaches you can try that I haven't described here.
Handling the stack overflow by hosting the CLR process and handling it. Note that you still cannot 'catch' it.
Changing all IL code, building another DLL, adding checks on recursion. Yes, that's quite possible (I've implemented it in the past :-); it's just difficult and involves a lot of code to get it right.
Use the .NET profiling API to capture all method calls and use that to figure out stack overflows. For example, you can implement checks that if you encounter the same method X times in your call tree, you give a signal. There's a project clrprofiler that will give you a head start.

I would suggest creating a wrapper around XmlWriter object, so it would count amount of calls to WriteStartElement/WriteEndElement, and if you limit amount of tags to some number (f.e. 100), you would be able to throw a different exception, for example - InvalidOperation.
That should solve the problem in the majority of the cases
public class LimitedDepthXmlWriter : XmlWriter
{
private readonly XmlWriter _innerWriter;
private readonly int _maxDepth;
private int _depth;
public LimitedDepthXmlWriter(XmlWriter innerWriter): this(innerWriter, 100)
{
}
public LimitedDepthXmlWriter(XmlWriter innerWriter, int maxDepth)
{
_maxDepth = maxDepth;
_innerWriter = innerWriter;
}
public override void Close()
{
_innerWriter.Close();
}
public override void Flush()
{
_innerWriter.Flush();
}
public override string LookupPrefix(string ns)
{
return _innerWriter.LookupPrefix(ns);
}
public override void WriteBase64(byte[] buffer, int index, int count)
{
_innerWriter.WriteBase64(buffer, index, count);
}
public override void WriteCData(string text)
{
_innerWriter.WriteCData(text);
}
public override void WriteCharEntity(char ch)
{
_innerWriter.WriteCharEntity(ch);
}
public override void WriteChars(char[] buffer, int index, int count)
{
_innerWriter.WriteChars(buffer, index, count);
}
public override void WriteComment(string text)
{
_innerWriter.WriteComment(text);
}
public override void WriteDocType(string name, string pubid, string sysid, string subset)
{
_innerWriter.WriteDocType(name, pubid, sysid, subset);
}
public override void WriteEndAttribute()
{
_innerWriter.WriteEndAttribute();
}
public override void WriteEndDocument()
{
_innerWriter.WriteEndDocument();
}
public override void WriteEndElement()
{
_depth--;
_innerWriter.WriteEndElement();
}
public override void WriteEntityRef(string name)
{
_innerWriter.WriteEntityRef(name);
}
public override void WriteFullEndElement()
{
_innerWriter.WriteFullEndElement();
}
public override void WriteProcessingInstruction(string name, string text)
{
_innerWriter.WriteProcessingInstruction(name, text);
}
public override void WriteRaw(string data)
{
_innerWriter.WriteRaw(data);
}
public override void WriteRaw(char[] buffer, int index, int count)
{
_innerWriter.WriteRaw(buffer, index, count);
}
public override void WriteStartAttribute(string prefix, string localName, string ns)
{
_innerWriter.WriteStartAttribute(prefix, localName, ns);
}
public override void WriteStartDocument(bool standalone)
{
_innerWriter.WriteStartDocument(standalone);
}
public override void WriteStartDocument()
{
_innerWriter.WriteStartDocument();
}
public override void WriteStartElement(string prefix, string localName, string ns)
{
if (_depth++ > _maxDepth) ThrowException();
_innerWriter.WriteStartElement(prefix, localName, ns);
}
public override WriteState WriteState
{
get { return _innerWriter.WriteState; }
}
public override void WriteString(string text)
{
_innerWriter.WriteString(text);
}
public override void WriteSurrogateCharEntity(char lowChar, char highChar)
{
_innerWriter.WriteSurrogateCharEntity(lowChar, highChar);
}
public override void WriteWhitespace(string ws)
{
_innerWriter.WriteWhitespace(ws);
}
private void ThrowException()
{
throw new InvalidOperationException(string.Format("Result xml has more than {0} nested tags. It is possible that xslt transformation contains an endless recursive call.", _maxDepth));
}
}

This answer is for #WilliamJockusch.
I'm wondering if there is a general way to track down
StackOverflowExceptions. In other words, suppose I have infinite
recursion somewhere in my code, but I have no idea where. I want to
track it down by some means that is easier than stepping through code
all over the place until I see it happening. I don't care how hackish
it is. For example, It would be great to have a module I could
activate, perhaps even from another thread, that polled the stack
depth and complained if it got to a level I considered "too high." For
example, I might set "too high" to 600 frames, figuring that if the
stack were too deep, that has to be a problem. Is something like that
possible. Another example would be to log every 1000th method call
within my code to the debug output. The chances this would get some
evidence of the overlow would be pretty good, and it likely would not
blow up the output too badly. The key is that it cannot involve
writing a check wherever the overflow is happening. Because the entire
problem is that I don't know where that is. Preferrably the solution
should not depend on what my development environment looks like; i.e,
it should not assumet that I am using C# via a specific toolset (e.g.
VS).
It sounds like you're keen to hear some debugging techniques to catch this StackOverflow so I thought I would share a couple for you to try.
1. Memory Dumps.
Pro's: Memory Dumps are a sure fire way to work out the cause of a Stack Overflow. A C# MVP & I worked together troubleshooting a SO and he went on to blog about it here.
This method is the fastest way to track down the problem.
This method wont require you to reproduce problems by following steps seen in logs.
Con's: Memory Dumps are very large and you have to attach AdPlus/procdump the process.
2. Aspect Orientated Programming.
Pro's: This is probably the easiest way for you to implement code that checks the size of the call stack from any method without writing code in every method of your application. There are a bunch of AOP Frameworks that allow you to Intercept before and after calls.
Will tell you the methods that are causing the Stack Overflow.
Allows you to check the StackTrace().FrameCount at the entry and exit of all methods in your application.
Con's: It will have a performance impact - the hooks are embedded into the IL for every method and you cant really "de-activate" it out.
It somewhat depends on your development environment tool set.
3. Logging User Activity.
A week ago I was trying to hunt down several hard to reproduce problems. I posted this QA User Activity Logging, Telemetry (and Variables in Global Exception Handlers) . The conclusion I came to was a really simple user-actions-logger to see how to reproduce problems in a debugger when any unhandled exception occurs.
Pro's: You can turn it on or off at will (ie subscribing to events).
Tracking the user actions doesn't require intercepting every method.
You can count the number of events methods are subscribed too far more simply than with AOP.
The log files are relatively small and focus on what actions you need to perform to reproduce the problem.
It can help you to understand how users are using your application.
Con's: Isn't suited to a Windows Service and I'm sure there are better tools like this for web apps.
Doesn't necessarily tell you the methods that cause the Stack Overflow.
Requires you to step through logs manually reproducing problems rather than a Memory Dump where you can get it and debug it straight away.
Maybe you might try all techniques I mention above and some that #atlaste posted and tell us which one's you found were the easiest/quickest/dirtiest/most acceptable to run in a PROD environment/etc.
Anyway good luck tracking down this SO.

If you application depends on 3d-party code (in Xsl-scripts) then you have to decide first do you want to defend from bugs in them or not.
If you really want to defend then I think you should execute your logic which prone to external errors in separate AppDomains.
Catching StackOverflowException is not good.
Check also this question.

I had a stackoverflow today and i read some of your posts and decided to help out the Garbage Collecter.
I used to have a near infinite loop like this:
class Foo
{
public Foo()
{
Go();
}
public void Go()
{
for (float i = float.MinValue; i < float.MaxValue; i+= 0.000000000000001f)
{
byte[] b = new byte[1]; // Causes stackoverflow
}
}
}
Instead let the resource run out of scope like this:
class Foo
{
public Foo()
{
GoHelper();
}
public void GoHelper()
{
for (float i = float.MinValue; i < float.MaxValue; i+= 0.000000000000001f)
{
Go();
}
}
public void Go()
{
byte[] b = new byte[1]; // Will get cleaned by GC
} // right now
}
It worked for me, hope it helps someone.

With .NET 4.0 You can add the HandleProcessCorruptedStateExceptions attribute from System.Runtime.ExceptionServices to the method containing the try/catch block. This really worked! Maybe not recommended but works.
using System;
using System.Reflection;
using System.Runtime.InteropServices;
using System.Runtime.ExceptionServices;
namespace ExceptionCatching
{
public class Test
{
public void StackOverflow()
{
StackOverflow();
}
public void CustomException()
{
throw new Exception();
}
public unsafe void AccessViolation()
{
byte b = *(byte*)(8762765876);
}
}
class Program
{
[HandleProcessCorruptedStateExceptions]
static void Main(string[] args)
{
Test test = new Test();
try {
//test.StackOverflow();
test.AccessViolation();
//test.CustomException();
}
catch
{
Console.WriteLine("Caught.");
}
Console.WriteLine("End of program");
}
}
}

#WilliamJockusch, if I understood correctly your concern, it's not possible (from a mathematical point of view) to always identify an infinite recursion as it would mean to solve the Halting problem. To solve it you'd need a Super-recursive algorithm (like Trial-and-error predicates for example) or a machine that can hypercompute (an example is explained in the following section - available as preview - of this book).
From a practical point of view, you'd have to know:
How much stack memory you have left at the given time
How much stack memory your recursive method will need at the given time for the specific output.
Keep in mind that, with the current machines, this data is extremely mutable due to multitasking and I haven't heard of a software that does the task.
Let me know if something is unclear.

By the looks of it, apart from starting another process, there doesn't seem to be any way of handling a StackOverflowException. Before anyone else asks, I tried using AppDomain, but that didn't work:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Reflection;
using System.Text;
using System.Threading;
namespace StackOverflowExceptionAppDomainTest
{
class Program
{
static void recrusiveAlgorithm()
{
recrusiveAlgorithm();
}
static void Main(string[] args)
{
if(args.Length>0&&args[0]=="--child")
{
recrusiveAlgorithm();
}
else
{
var domain = AppDomain.CreateDomain("Child domain to test StackOverflowException in.");
domain.ExecuteAssembly(Assembly.GetEntryAssembly().CodeBase, new[] { "--child" });
domain.UnhandledException += (object sender, UnhandledExceptionEventArgs e) =>
{
Console.WriteLine("Detected unhandled exception: " + e.ExceptionObject.ToString());
};
while (true)
{
Console.WriteLine("*");
Thread.Sleep(1000);
}
}
}
}
}
If you do end up using the separate-process solution, however, I would recommend using Process.Exited and Process.StandardOutput and handle the errors yourself, to give your users a better experience.

You can read up this property every few calls, Environment.StackTrace , and if the stacktrace exceded a specific threshold that you preset, you can return the function.
You should also try to replace some recursive functions with loops.

Early exit from calling code when using yield return on a generic IEnumerable

What happens when calling code exits prior to completing enumeration of a IEnumerable that is yield returning.
A simplified example:
public void HandleData()
{
int count = 0;
foreach (var datum in GetFileData())
{
//handle datum
if (++count > 10)
{
break;//early exit
}
}
}
public static IEnumerable<string> GetFileData()
{
using (StreamReader sr = _file.BuildStreamer())
{
string line = String.Empty;
while ((line = sr.ReadLine()) != null)
{
yield return line;
}
}
}
In this case it seems quite important that the StreamReader is closed in a timely manner. Is there a pattern needed to handle this scenario?

That's a good question.
You see, while using foreach() to iterate resulting IEnumerable, you're safe. The Enumerator below implements IDisposable itself, which gets called in case of foreach (even if loop is exited with break) and takes care of cleaning the state of your in GetFileData.
But if you will play with Enumerator.MoveNext directly, you're in trouble and Dispose will never be called if exited earlier (of course, if you'll complete manual iteration, it will be).For manual Enumerator-based iteration, you can consider placing enumerator in using statement as well (as mentioned in code below).
Hope this example with different usecases covered will provide you some feedback for your question.
static void Main(string[] args)
{
// Dispose will be called
foreach(var value in GetEnumerable())
{
Console.WriteLine(value);
break;
}
try
{
// Dispose will be called even here
foreach (var value in GetEnumerable())
{
Console.WriteLine(value);
throw new Exception();
}
}
catch // Lame
{
}
// Dispose will not be called
var enumerator = GetEnumerable().GetEnumerator();
// But if enumerator and this logic is placed inside the "using" block,
// like this: using(var enumerator = GetEnumerable().GetEnumerable(){}), it will be.
while(enumerator.MoveNext())
{
Console.WriteLine(enumerator.Current);
break;
}
Console.WriteLine("{0}Here we'll see dispose on completion of manual enumeration.{0}", Environment.NewLine);
// Dispose will be called: ended enumeration
var enumerator2 = GetEnumerable().GetEnumerator();
while (enumerator2.MoveNext())
{
Console.WriteLine(enumerator2.Current);
}
}
static IEnumerable<string> GetEnumerable()
{
using (new MyDisposer())
{
yield return "First";
yield return "Second";
}
Console.WriteLine("Done with execution");
}
public class MyDisposer : IDisposable
{
public void Dispose()
{
Console.WriteLine("Disposed");
}
}
Originally observed by: https://blogs.msdn.microsoft.com/dancre/2008/03/15/yield-and-usings-your-dispose-may-not-be-called/
Author calls this (the fact that manual MoveNext() and early break will not trigger Dipose()) "a bug", but this is intended implementation.

Why does the using statement work on IEnumerator and what does it do?

Back when I was learning about foreach, I read somewhere that this:
foreach (var element in enumerable)
{
// do something with element
}
is basically equivalent to this:
using (var enumerator = enumerable.GetEnumerator())
{
while (enumerator.MoveNext())
{
var element = enumerator.Current;
// do something with element
}
}
Why does this code even compile if neither IEnumerator nor IEnumerator<T> implement IDisposable? C# language specification only seems to mention the using statement in the context of IDisposable.
What does such an using statement do?

Please, check the following link about foreach statement. It uses try/finally block with Dispose call if it's possible. That's the code which is behind using statement.

IEnumerator may not implement IDisposable but GetEnumerator() returns a IEnumerator<T> which does. From the docs on IEnumerator<T>:
In addition, IEnumerator implements IDisposable, which requires you to implement the Dispose method. This enables you to close database connections or release file handles or similar operations when using other resources. If there are no additional resources to dispose of, provide an empty Dispose implementation.
This is of course assuming that your enumeration is an IEnumerable<T> and not just a IEnumerable. If your original enumeration was just an IEnumerable then it wouldn't compile.

It appears no one answered the main question yet:
What does such a using statement do?
I'll go with an example from one of my favorite books - C# in Depth by Jon Skeet. You may have a result of IEnumerator<T> produced by a function like this:
static IEnumerable<string> Iterator()
{
try
{
Console.WriteLine("Before first yield");
yield return "first";
Console.WriteLine("Between yields");
yield return "second";
Console.WriteLine("After second yield");
}
finally
{
Console.WriteLine("In finally block");
}
}
And then you would use it like:
foreach (string value in Iterator())
{
Console.WriteLine("Received value: {0}", value);
if (value != null)
{
break;
}
}
The iterator returns only the first "first" value from the sequence as we break on the very first iteration. Now how your iterator might 'know' that you aren't going to proceed till the end of the loop so it would fire it's finally block? Here comes the hidden using statement. That's how the code from the example above would look like if you couldn’t use a foreach loop:
IEnumerable<string> enumerable = Iterator();
using (IEnumerator<string> enumerator = enumerable.GetEnumerator())
{
while (enumerator.MoveNext())
{
string value = enumerator.Current;
Console.WriteLine("Received value: {0}", value);
if (value != null)
{
break;
}
}
}
When we leave the scope of the hidden using statement, the Dispose() method fires:
finally
{
Console.WriteLine("In finally block");
}
*All of the code lines and some of the text above are provided by Jon Skeet. I could have rewritten it but decided that I should rather leave it intact. If the author doesn't want me to share it - I'll delete the answer or rewrite it ASAP.

Iterator function for IDisposable items

Suppose I want to create an iterator function that yields IDisposable items.
IEnumerable<Disposable> GetItems()
{
yield return new Disposable();
yield return new Disposable();
}
This does not seem ideal for the client code:
foreach (var item in source.GetItems())
{
using (item)
{
// ...
}
}
Intuitively, the using comes too late. Things could get moved around. One could accidentally insert code between the foreach and the using. Not ideal.
I am looking for a better alternative!
One approach that comes to mind is creating the following API instead of an iterator function:
// Client
while (source.HasItem)
{
using (var item = source.GetNextItem())
{
// ...
}
}
// Source
private IEnumerator<Disposable> Enumerator { get; }
private bool? _hasItem;
bool HasItem
{
get
{
if (this._hasItem == null) this._hasItem = this.Enumerator.MoveNext();
return this._hasItem;
}
}
Disposable GetNextItem()
{
if (!this.HasItem) throw new IndexOutOfBoundsException();
this._hasItem = null;
return this.Enumerator.Current;
}
But now it seems that the source has to become IDisposable! How else would it know when to dispose Enumerator? That can be an unpleasant side-effect.
I am looking for an alternative that feels solid in the client, but that keeps the source from becoming IDisposable too.
Edit - Clarification: I forgot to mention that some of the content that we need comes from an iterator. Concretely, imagine that we are returning IDbCommand instances, which are IDisposable. Before returning each command, we need to populate it with some query data, which in turn comes from a simple iterator method.

If I understand correctly, the following pattern works for you
foreach (var item in source.GetItems())
{
using (item)
{
// ...
}
}
but the potential problem is putting some code outside the using block. So why don't you just wrap that logic in a custom extensions method:
public static class EnumerableExtennisons
{
public static IEnumerable<T> WithUsing<T>(this IEnumerable<T> source)
where T : IDisposable
{
foreach (var item in source)
{
using (item)
yield return item;
}
}
}
This way you ensure the item is wrapped in using block *before** returning it to the caller, so there is no way the caller to insert code before/after it. The C# compiler generated code ensures the item.Dispose is called in either MoveNext or Dispose method of the IEnumerator<T> (in case the iteration ends earlier).
The usage would be to append .WithUsing() call instead of using block where needed:
foreach (var item in source.GetItems().WithUsing())
{
// ...
}

We can expose only an IEnumerator<T>, which supports nothing but MoveNext() and Current.
Now, the underlying, private iterator function can be streaming, taking care of disposing the items. No invalid operations are introduced to the client - unless they try to store the borrowed objects and try to use them later, where it becomes clear that the objects are already disposed.
// Client
using (var itemEnumerator = source.GetItemEnumerator())
{
while (itemEnumerator.MoveNext())
var current = itemEnumerator.Current;
}
// Source
IEnumerator<Disposable> GetItemEnumerator() => this.StreamItems().GetEnumerator();
private IEnumerable<Disposable> StreamItems()
{
while (this.ShouldCreateItem())
{
using (var item = this.CreateItem())
{
yield return item;
}
}
}

The iterator function itself can take responsibility for placing a using (i.e. disposing the current item when the next is requested).
IEnumerable<Disposable> GetItems()
{
// Assumming CreateItems() is an iterator as well
foreach (var item in this.CreateItems())
using (item) yield return item;
}
As a result, we will essentially have a streaming-only iterator. Methods like ToList() become meaningless, because only one item is usable at a time.
Unfortunately, this does create room for error.

Is this a dangerous locking pattern?

I have an enumerator written in C#, which looks something like this:
try
{
ReadWriteLock.EnterReadLock();
yield return foo;
yield return bar;
yield return bash;
}
finally
{
if (ReadWriteLock.IsReadLockHeld)
ReadWriteLock.ExitReadLock();
}
I believe this may be a dangerous locking pattern, as the ReadWriteLock will only be released if the enumeration is complete, otherwise the lock is left hanging and is never released, am I correct? If so, what's the best way to combat this?

No, the finally block will always be executed, pretty much unless somebody pulls the plug from the computer (well and a few other exceptions).
public static IEnumerable<int> GetNumbers() {
try
{
Console.WriteLine("Start");
yield return 1;
yield return 2;
yield return 3;
}
finally
{
Console.WriteLine("Finish");
}
}
...
foreach(int i in GetNumbers()) {
Console.WriteLine(i);
if(i == 2) break;
}
The output of the above will be
Start12Finish
Note that in C# you write yield return, not just yield. But I guess that was just a typo.

I think David's answered the question you intended to ask (about the enumeration aspect), but two additional points to consider:
What would happen if ReadWriteLock.EnterReadLock threw an exception?
What would happen if ReadWriteLock.ExitReadLock threw an exception?
In #1, you'll call ReadWriteLock.ExitReadLock inappropriately. In #2, you may hide an existing exception that's been thrown (since finally clauses happen either because the mainline processing reached the end of the try block or because an exception was thrown; in the latter case, you probably don't want to obscure the exception). Perhaps both of those things are unlikely in this specific case, but you asked about the pattern, and as a pattern it has those issues.

Finally will be executed in any way, but for locking in may not be safe. Compare following methods:
class Program
{
static IEnumerable<int> meth1()
{
try
{
Console.WriteLine("Enter");
yield return 1;
yield return 2;
yield return 3;
}
finally
{
Console.WriteLine("Exit");
}
}
static IEnumerable<int> meth2()
{
try
{
Console.WriteLine("Enter");
return new int[] { 1, 2, 3 };
}
finally
{
Console.WriteLine("Exit");
}
}
static public void Main()
{
foreach (int i in meth1())
{
Console.WriteLine("In");
}
Console.WriteLine();
foreach (int i in meth2())
{
Console.WriteLine("In");
}
}
}
Output is:
Enter
In
In
In
Exit
Enter
Exit
In
In
In
If your processing takes much time (per iteration) it is more reasonable to fill collection first, then process, but not yield.

Develop Reference

C# (C-Sharp) is a programming language developed by Microsoft that runs on the .NET Framework.