I have stumbled across a situation where garbage collection seems to be behaving differently between the same code running written as a Unit Test vs written in the Main method of a Console Application. I am wondering the reason behind this difference.
In this situation, a co-worker and I were in disagreement over the effects of registering an event handler on garbage collection. I thought that a demonstration would be better accepted than simply sending him a link to a highly rated SO answer. As such I wrote a simple demonstration as a unit test.
My unit test showed things worked as I said they should. However, my coworker wrote a console application that showed things working his way, which meant that GC was not occurring as I expected on the local objects in the Main method. I was able to reproduce the behavior he saw simply by moving the code from my test into the Main method of a Console Application project.
What I would like to know is why GC does not seem to collecting objects as expected when running in the Main method of a Console Application. By extracting methods so that the call to GC.Collect and the object going out of scope occurred in different methods, the expected behavior was restored.
These are the objects I used to define my test. There is simply an object with an event and an object providing a suitable method for an event handler. Both have finalizers setting a global variable so that you can tell when they have been collected.
private static string Log;
public const string EventedObjectDisposed = "EventedObject disposed";
public const string HandlingObjectDisposed = "HandlingObject disposed";
private class EventedObject
{
public event Action DoIt;
~EventedObject()
{
Log = EventedObjectDisposed;
}
protected virtual void OnDoIt()
{
Action handler = DoIt;
if (handler != null) handler();
}
}
private class HandlingObject
{
~HandlingObject()
{
Log = HandlingObjectDisposed;
}
public void Yeah()
{
}
}
This is my test (NUnit), which passes:
[Test]
public void TestReference()
{
{
HandlingObject subscriber = new HandlingObject();
{
{
EventedObject publisher = new EventedObject();
publisher.DoIt += subscriber.Yeah;
}
GC.Collect(GC.MaxGeneration);
GC.WaitForPendingFinalizers();
Thread.MemoryBarrier();
Assert.That(Log, Is.EqualTo(EventedObjectDisposed));
}
//Assertion needed for foo reference, else optimization causes it to already be collected.
Assert.IsNotNull(subscriber);
}
GC.Collect(GC.MaxGeneration);
GC.WaitForPendingFinalizers();
Thread.MemoryBarrier();
Assert.That(Log, Is.EqualTo(HandlingObjectDisposed));
}
I pasted the body above in to the Main method of a new console application, and converted the Assert calls to Trace.Assert invocations. Both equality asserts fail then fail. Code of resulting Main method is here if you want it.
I do recognize that when GC occurs should be treated as non-deterministic and that generally an application should not be concerning itself with when exactly it occurs.
In all cases the code was compiled in Release mode and targeting .NET 4.5.
Edit: Other things I tried
Making the test method static since NUnit supports that; test still worked.
I also tried extracting the whole Main method into an instance method on program and calling that. Both asserts still failed.
Attributing Main with [STAThread] or [MTAThread] in case this made a difference. Both asserts still failed.
Based on #Moo-Juice's suggestions:
I referenced NUnit to the Console app so that I could use the NUnit asserts, they failed.
I tried various changes to visibility to the both the test, test's class, Main method, and the class containing the Main method static. No change.
I tried making the Test class static and the class containing the Main method static. No change.
If the following code was extracted to a separate method, the test would be more likely to behave as you expected. Edit: Note that the wording of the C# language specification does not require this test to pass, even if you extract the code to a separate method.
{
EventedObject publisher = new EventedObject();
publisher.DoIt += subscriber.Yeah;
}
The specification allows but does not require that publisher be eligible for GC immediately at the end of this block, so you should not write code in such a way that you are assuming it can be collected here.
Edit: from ECMA-334 (C# language specification) §10.9 Automatic memory management (emphasis mine)
If no part of the object can be accessed by any possible continuation of execution, other than the running of finalizers, the object is considered no longer in use and it becomes eligible for finalization. [Note: Implementations might choose to analyze code to determine which references to an object can be used in the future. For instance, if a local variable that is in scope is the only existing reference to an object, but that local variable is never referred to in any possible continuation of execution from the current execution point in the procedure, an implementation might (but is not required to) treat the object as no longer in use. end note]
The problem isn't that it's a console application - the problem is that you're likely running it through Visual Studio - with a debugger attached! And/or you're compiling the console app as a Debug build.
Make sure you're compiling a Release build. Then go to Debug -> Start Without Debugging, or press Ctrl+F5, or run your console application from a command line. The Garbage Collector should now behave as expected.
This is also why Eric Lippert reminds you not to run any perf benchmarks in a debugger in C# Performance Benchmark Mistakes, Part One.
The jit compiler knows that a debugger is attached, and it deliberately de-optimizes the code it generates to make it easier to debug. The garbage collector knows that a debugger is attached; it works with the jit compiler to ensure that memory is cleaned up less aggressively, which can greatly affect performance in some scenarios.
Lots of the reminders in Eric's series of articles apply to your scenario. If you're interested in reading more, here are the links for parts two, three and four.
Related
According 8th step of this post I wrote following simple unit test to sure my Test class doesn't cause memory leak:
private class TestClass
{
}
[TestMethod]
public void MemoryLeakTest()
{
vat testObj = new TestClass();
var weakRef = new WeakReference(testObj)
testObj = null;
GC.Collect();
GC.WaitForPendingFinalizers();
GC.Collect();
Assert.IsFalse(weakRef.IsAlive);
}
The test pass in the master branch of my code repository, but when I run the test in another feature branch, the test fails. So I have two questions:
Is this method reliable to detect memory leak of my classes?
What conditions can cause this test pass in one branch and fail in another branch?
There are very few possibilities where this test would do something useful. All of them would involve that the constructor of Test does some kind of registration on a global variable (either register itself on a global event or store the this pointer in a static member). Since that is something that's very rarely done, doing the above test for all classes is overshooting. Also, the test does not cover the far more typical cases for a memory leak in C#: Building up a data structure (e.g. a List) and never clean it up.
This test may fail for a number of reasons: GC.Collect() does not necessarily force all garbage to be cleaned. It should, but there's no guarantee that it will always happen. Also, since testObj is a local variable, it is not yet out of scope when you call GC.Collect(), so depending on how the code is optimized, the variable cannot be considered garbage yet, which makes the test fail.
We're dealing with the GC being too quick in a .Net program.
Because we use a class with native resources and we do not call GC.KeepAlive(), the GC collects the object before the Native access ends. As a result the program crashes.
We have exactly the problem as described here:
Does the .NET garbage collector perform predictive analysis of code?
Like so:
{ var img = new ImageWithNativePtr();
IntPtr p = img.GetData();
// DANGER!
ProcessData(p);
}
The point is: The JIT generates information that shows the GC that img is not used at the point when GetData() runs. If a GC-Thread comes at the right time, it collects img and the program crashes. One can solve this by appending GC.KeepAlive(img);
Unfortunately there is already too much code written (at too many places) to rectify the issue easily.
Therefore: Is there for example an Attribute (i.e. for ImageWithNativePtr) to make the JIT behave like in a Debug build? In a Debug build, the variable img will remain valid until the end of the scope ( } ), while in Release it looses validity at the comment DANGER.
To the best of my knowledge there is no way to control jitter's behavior based on what types a method references. You might be able to attribute the method itself, but that won't fill your order. This being so, you should bite the bullet and rewrite the code. GC.KeepAlive is one option. Another is to make GetData return a safe handle which will contain a reference to the object, and have ProcessData accept the handle instead of IntPtr — that's good practice anyway. GC will then keep the safe handle around until the method returns. If most of your code has var instead of IntPtr as in your code fragment, you might even get away without modifying each method.
You have a few options.
(Requires work, more correct) - Implement IDisposable on your ImageWithNativePtr class as it compiles down to try { ... } finally { object.Dispose() }, which will keep the object alive provided you update your code with usings. You can ease the pain of doing this by installing something like CodeRush (even the free Xpress supports this) - which supports creating using blocks.
(Easier, not correct, more complicated build) - Use Post Sharp or Mono.Cecil and create your own attribute. Typically this attribute would cause GC.KeepAlive() to be inserted into these methods.
The CLR has nothing built in for this functionality.
I believe you can emulate what you want with a container that implements IDispose, and a using statement. The using statement allows for defining the scope and you can place anything you want in it that needs to be alive over that scope. This can be a helpful mechanism if you have no control over the implementation of ImageWithNativePtr.
The container of things to dispose is a useful idiom. Particularly when you really should be disposing of something ... which is probably the case with an image.
using(var keepAliveContainer = new KeepAliveContainer())
{
var img = new ImageWithNativePtr();
keepAliveContainer.Add(img);
IntPtr p = img.GetData();
ProcessData(p);
// anything added to the container is still referenced here.
}
For some time now, I've observed an intermittent COM problem in my VSIX package for Visual Studio 2010. Attempting to subscribe to one of the IDE's COM-based event sinks randomly throws the following error:
"COM object that has been separated from its underlying RCW cannot be used"
A repro case boils down to this code (which must be used in VSIX, obviously):
using System;
using EnvDTE;
using EnvDTE80;
class Test
{
private readonly Events _events;
private readonly Events2 _events2;
private readonly BuildEvents _buildEvents;
private readonly ProjectItemsEvents _projectItemsEvents;
public Test(IServiceProvider provider)
{
var dte = (DTE)provider.GetService(typeof(DTE));
var dte2 = (DTE2)dte;
// Store all references in fields as a GC precaution.
_events = dte.Events;
_events2 = (Events2)dte2.Events;
_buildEvents = _events.BuildEvents;
_projectItemsEvents = _events2.ProjectItemsEvents;
// Proceed to subscribe to event sinks.
_buildEvents.OnBuildBegin += BuildBeginHandler; // BOOM!
_projectItemsEvents.ItemAdded += ItemAddedHandler;
}
private void ItemAddedHandler(ProjectItem projectItem) { }
private void BuildBeginHandler(vsBuildScope scope, vsBuildAction action) { }
}
I've learned about a possible cause from numerous descriptions of similar problems that can be found on the net. It's basically a side effect of the way Runtime Callable Wrappers and GC interact during COM interop. Here's a link to a similar problem complete with explanation.
I'm fine with that explanation, especially because it suggests an easy workaround - storing the event sink reference in a field in order to prevent it from being prematurely GC'ed. Indeed, many people seem to have solved their problem this way.
What bothers me is that it doesn't work in my case. I'm really stumped as to why. As you can plainly see, I already store all object references in fields as a precaution. Yet the error still occurs. I tried being even more explicit using GC.KeepAlive() calls at the end of the ctor, but to no avail. Is there anything else left to do?
Without a solution, my VSIX randomly fails to load, leaving the user with a single option: to restart Visual Studio and hope it doesn't happen the next time.
Any help will truly be appreciated!
Well, I gave up and simply did the only thing that crossed my mind. I figured that since this is obviously a race condition I can't affect in a predictable manner, I might as well reenter the race if I lose.
So I moved the subscription lines into a while loop that try..catch-es them and retries after a bit of Thread.Sleep(). The loop exits either when both subscriptions succeed or when I've been continuously losing the race for more than 2 seconds.
The kicker is, I haven't lost the race once since I've implemented the change. A true Heisenbug, if I ever saw one.
Anyway, I'm going to stick with this until a proper solution occurs to me or the bug reappears.
I suspect that your problem is really that you are attempting to wire up your event handlers too soon. You normally need to be doing these sorts of things in the Initialize method of your package / toolwindow / whatever - generally speaking if you need to use a service you need to do if after the Initialize method has been called, definitely don't do this in the constructor of your Package.
(This is just a hunch - your Test class doesn't implement any VSX interfaces and so I can't see from your sample when the constructor is being called)
When I run my tests in Visual Studio individually, they all pass without a problem. However, when I run all of them at once some pass and some fail. I tried putting in a pause of 1 second in between each test method with no success.
Any ideas? Thanks in advance for your help...
It's possible that you have some shared data. Check for static member variables in the classes in use that means one test sets a value that causes a subsequent test to fail.
You can also debug unit tests. Depending on the framework you're using, you should be able to run the framework tool as a debug start application passing the path to the compiled assembly as a parameter.
It's very possible that some modifications/instantiations done in one test affect the others. That indicates poor test design and lack of proper isolation.
Everyone is probably right, some shared date is being modified between tests. But note the order of MS Test execution. Simply pausing between tests is not a solution. Each test is executed in it's own instance of the test class on a separate thread.
as per the other responses. It sounds like you have a singleton or a global variable that is causing the interaction.
Other unit test frameworks that I have used work hard to ensure that a test produces identical results whether the test is run individually or is run as part of the 'run them all' alternative. The goal is to prevent one test from having an effect on another due to side effects such as (for example) having one test leave the static state of a class in a configuration that another test is not expecting. The VS unit test framework does not appear to provide this isolation. I have 2 suggestions for minimizing the kinds of problems that the question implies. First, use non-static class in preference to a static class if the class has state (has anything other than static methods). Create a single instance of this class and have it keep the state info that was being kept in the static class. Second, if you do elect to have static class(es) with static state, have a static method that sets the static state back to 'empty' (e.g., a method that sets all static properties to null/zero/etc.). Call this at the end of each unit test to undo any effects that the test has imposed on the static state. (This is admittedly less than elegant but can be workable if done in moderation). Or do what I plan to do - find a unit test framework that provides isolation across tests.
I also ran into this problem although my issue ended up being a threading problem. In my case I was faking the HttpContext object since the tests relied on it's existence. However, I was setting this in the ClassInitialize method thinking this would be used for each method like below:
[ClassInitialize]
public static void ClassInit(TestContext testContext)
{
HttpContext.Current = new HttpContext(new HttpRequest(null, "http://tempuri.org", null), new HttpResponse(null));
}
However, it turns out that each test method in the class runs in a separate thread. So I had to add this code to every test method that relied upon it to fix the issue.
[TestMethod]
public void TestMethod1()
{
HttpContext.Current = new HttpContext(new HttpRequest(null, "http://tempuri.org", null), new HttpResponse(null));
...
}
[TestMethod]
public void TestMethod2()
{
HttpContext.Current = new HttpContext(new HttpRequest(null, "http://tempuri.org", null), new HttpResponse(null));
...
}
See link for more information on this.
http://blogs.msdn.com/b/nnaderi/archive/2007/02/17/explaining-execution-order.aspx
I faced a similar issue here how I solved it:
Copy the code of the second test inside the first test (after it).
Try to test the first test. The first test will probably fails, and then you can debug the first test (step by step) to find the static/shared variable or logic that makes the problem.
In my case, I had an Environment variable set with:
Environment.SetEnvironmentVariable("KEY", "value");
The implementation code for a few other tests was assuming a default value, and if the test(s) with the above line were executed first, those failed. The solution is to clean up with the following (at the end of each unit test, or in a special method for the same purpose - TestCleanup in MS/VSTest):
Environment.SetEnvironmentVariable("KEY", null);
Though redundant, it is best practice to also set the environment variable to any value previously assumed by (the failing) tests to be default. Do this at the top of those unit tests (the Arrange step).
I would like to unit test a method on a class I have made, but this method requires another method to be called first. Example:
// This would work
MyClass myClass1 = new MyClass(mockDevice);
myClass1.Run(myDatastructure);
myClass1.Stop();
// This would throw an InvalidOperationException
MyClass myClass2 = new MyClass(mockDevice);
myClass2.Stop();
Run is starting an operation on a hardware device, and Stop is of course trying to stop that operation (sending a reset-command and starting a timeout-timer).
Anyway I would like to test various post-conditions of calling Stop, but I would like NOT to have to call Run, because I am testing Stop - not Run! I would like something like this:
MyClass myClass = new MyClass(mockDevice);
myClass.Stop();
Assert.IsTrue(mockDevice.ResetCalled);
So far I only see one possible solution, and that is to create a TestableMyClass that inherits from MyClass, that makes it possible to set the right internal state of the MyClass instance before calling Stop. The problem with this solution is that I have to change my MyClass-implementation to have protected members instead of private members, and I don't like the idea of having to change the implementation in order to test it!
Should I use this solution, is there an error in my design, or is there a smarter way of doing this?
As far as I see it, you are already testing Stop in the two ways in which it can be used (with and without a running operation). As long as the mockDevice is doing its job, it seem to me that you're testing it reasonably. Ideally you should be able to verify the commands sent to the device etc (which most mock frameworks will make simple).
In this situation, personally, I would have two tests for this:
Test without Run() being called first. I would test if it really throws the exception. I would also test if the post conditions are what I expect them to be.
Test with Run() being called first. I would test only the post conditions, that I expect.
Those are the only two important uses of the method that have different behaviors - therefor I would test them both.
EDIT:
I understand, why you don't want to call run, before stop - you think that if run fails, the test, that is supposed to only test stop method will most likely fail as well.
However, I would assume, that you also have test for the run method. This means, that when the tests, that test the behavior of run method pass - stop method tests must pass as well. If the run method tests fail, then the results of run method tests are undefined - they may or may not fail.
So, I'd say, don't be afraid to call other dependent methods in your tests, but make sure you test those dependent methods in separate tests.
It won't help your immediate problem, but I tend to favour mapping state to type rather than having objects with different modes.
IdleDevice idle = new IdleDevice(mockDevice);
RunningDevice running = idle.Run(myDatastructure);
running.Stop();
// This would not compile, as IdleDevice has no Stop method
IdleDevice idle = new IdleDevice(mockDevice);
idle.Stop();
If you can't compile impossible situations, you don't need to test them.