According 8th step of this post I wrote following simple unit test to sure my Test class doesn't cause memory leak:
private class TestClass
{
}
[TestMethod]
public void MemoryLeakTest()
{
vat testObj = new TestClass();
var weakRef = new WeakReference(testObj)
testObj = null;
GC.Collect();
GC.WaitForPendingFinalizers();
GC.Collect();
Assert.IsFalse(weakRef.IsAlive);
}
The test pass in the master branch of my code repository, but when I run the test in another feature branch, the test fails. So I have two questions:
Is this method reliable to detect memory leak of my classes?
What conditions can cause this test pass in one branch and fail in another branch?
There are very few possibilities where this test would do something useful. All of them would involve that the constructor of Test does some kind of registration on a global variable (either register itself on a global event or store the this pointer in a static member). Since that is something that's very rarely done, doing the above test for all classes is overshooting. Also, the test does not cover the far more typical cases for a memory leak in C#: Building up a data structure (e.g. a List) and never clean it up.
This test may fail for a number of reasons: GC.Collect() does not necessarily force all garbage to be cleaned. It should, but there's no guarantee that it will always happen. Also, since testObj is a local variable, it is not yet out of scope when you call GC.Collect(), so depending on how the code is optimized, the variable cannot be considered garbage yet, which makes the test fail.
Related
I have 2 projects, one is the "Main" project and the other is the "Test" project.
The policy is that all methods in the Main project must have at least one accompanying test in the test project.
What I want is a new unit test in the Test project that verifies this remains the case. If there are any methods that do not have corresponding tests (and this includes method overloads) then I want this test to fail.
I can sort out appropriate messaging when the test fails.
My guess is that I can get every method (using reflection??) but I'm not sure how to then verify that there is a reference to each method in this Test project (and ignore references in projects)
You can use any existing software to measure code coverage but...
Don't do it!!! Seriously. The aim should be not to have 100% coverage but to have software that can easily evolve. From your test project you can invoke by reflection every single existing method and swallow all the exceptions. That will make your coverage around 100% but what good would it be?
Read about TDD. Start creating testable software that has meaningful tests that will save you when something goes wrong. It's not about coverage, it's about being safe.
This is an example of a meta-test that sounds good in principle, but once you get to the detail you should quickly realise is a bad idea. As has already been suggested, the correct approach is to encourage whoever owns the policy to amend it. As you’ve quoted the policy, it is sufficiently specific that people can satisfy the requirement without really achieving anything of value.
Consider:
public void TestMethod1Exists()
{
try
{
var classUnderTest = new ClassToTest();
classUnderTest.Method1();
}
catch (Exception)
{
}
}
The test contains a call to Method1 on the ClassToTest so the requirement of having a test for that method is satisfied, but nothing useful is being tested. As long as the method exists (which is must if the code compiled) the test will pass.
The intent of the policy is presumably to try to ensure that written code is being tested. Looking at some very basic code:
public string IsSet(bool flag)
{
if (flag)
{
return "YES";
}
return "NO";
}
As methods go, this is pretty simple (it could easily be changed to one line), but even so it contains two routes through the method. Having a test to ensure that this method is being called gives you a false sense of security. You would know it is being called but you would not know if all of the code paths are being tested.
An alternative that has been suggested is that you could just use code coverage tools. These can be useful and give a much better idea as to how well exercised your code is, but again they only give an indication of the coverage, not the quality of that coverage. So, let’s say I had some tests for the IsSet method above:
public void TestWhenTrue()
{
var classUnderTest = new ClassToTest();
Assert.IsString(classUnderTest.IsSet(true));
}
public void TestWhenFalse()
{
var classUnderTest = new ClassToTest();
Assert.IsString(classUnderTest.IsSet(false));
}
I’m passing sufficient parameters to exercise both code paths, so the coverage for the IsSet method should be 100%. But all I am testing is that the method returns a string. I’m not testing the value of the string, so the tests themselves don’t really add much (if any) value.
Code coverage is a useful metric, but only as part of a larger picture of code quality. Having peer reviews and sharing best practice around how to effectively test the code you are writing within your team, whilst it is less concretely measurable will have a more significant impact on the quality of your test code.
I just delivered my first C# WebAPI application to the first customer. Under normal load, performance initially is even better than I expected. Initially.
Everything worked fine until, at some point, memory was up and garbage collection started running riot (as in "It collects objects that are not yet garbage"). At that point, there were multiple W3WP threads with some ten gigs of ram altogether, and single-digit gigs per worker. After a restart of the IIS everything was back to normal, but of course the memory usage is rising again.
Please correct me if I am wrong, but
Shouldn't C# have automatic garbage collection?
Shouldn't it be easy for GC to collect the garbage of a WebAPI application?
And please help me out:
How can I explicitly state what GC should collect, thus preventing memory leaks? Is someBigList = null; the way to go?
How can I detect where the memory leaks are?
EDIT: Let me clarify some things.
My .NET WebAPI application is mostly a bunch of
public class MyApiController:ApiController
{
[HttpGet]
public MyObjectClass[] MyApi(string someParam) {
List<MyObjectClass> list = new List<MyObjectClass>();
...
for/while/foreach {
MyObjectClass obj = new MyObjectClass();
obj.firstStringAttribute = xyz;
...
list.Add(obj);
}
return list.ToArray();
}
}
Under such conditions, GC should be easy: after "return", all local variables should be garbage. Yet with every single call the used memory increases.
I initially thought that C# WebAPI programs behave similar to (pre-compiled) PHP: IIS calls the program, it is executed, returns the value and is then completely disposed off.
But this is not the case. For instance, I found static variables to keep their data between runs, and now I disposed of all static variables.
Because I found static variables to be a problem for GC:
internal class Helper
{
private static List<string> someVar = new List<string>();
internal Helper() {
someVar=new List<string>();
}
internal void someFunc(string str) {
someVar.Add(str);
}
internal string[] someOtherFunc(string str) {
string[] s = someVar.ToArray();
someVar=new List<string>();
return s;
}
}
Here, under low-memory conditions, someVar threw a null pointer error, which in my opinion can only be caused by GC, since I did not find any code where someVar is actively nullified by me.
I think the memory increase slowed down since I actively set the biggest array variables in the most often used Controllers to null, but this is only a gut feeling and not even nearly a complete solution.
I will now do some profiling using the link you provided, and get back with some results.
Shouldn't C# have automatic garbage collection?
C# is a programming language for the .NET runtime, and .NET brings the automatic garbage collection to the table. So, yes, although technically C# isn't the piece that brings it.
Shouldn't it be easy for GC to collect the garbage of a WebAPI application?
Sure, it should be just as easy as for any other type of .NET application.
The common theme here is garbage. How does .NET determine that something is garbage? By verifying that there are no more live references to the object. To be honest I think it is far more likely that you have verified one of your assumptions wrongly, compared to there being a serious bug in the garbage collector in such a way that "It collects objects that are not yet garbage".
To find leaks, you need to figure out what objects are currently held in memory, make a determination whether that is correct or not, and if not, figure out what is holding them there. A memory profiler application would help with that, there are numerous available, such as the Red-Gate ANTS Memory Profiler.
For your other questions, how to make something eligible for garbage collection? By turning it into garbage (see definition above). Note that setting a local variable to null may not necessarily help or be needed. Setting a static variable to null, however, might. But the correct way to determine that is to use a profiler.
Here are some shot-in-the-dark type of tips you might look into:
Look at static classes, static fields, and static properties. Are you storing data there that is accumulating?
How about static events? Do you have this? Do you remember to unsubscribe the event when you no longer need it?
And by "static fields, properties, and events", I also mean normal instance fields, properties and events that are held in objects that directly or indirectly are stored in static fields or properties. Basically, anything that will keep the objects in memory.
Are you remembering to Dispose of all your IDisposable objects? If not, then the memory being used could be unmanaged. Typically, however, when the garbage collector collects the managed object, the finalizer of that object should clean up the unmanaged memory as well, however you might allocate memory that the GC algorithm isn't aware of, and thus thinks it isn't a big problem to wait with collection. See the GC.AddMemoryPressure method for more on this.
I wanted to detect whether or not my code generates garbage. So I created the following unit test.
[TestClass]
public class AllocationTest
{
int[] generationCollections = new int[3];
[TestMethod]
public void TestGarbageGeneration()
{
generationCollections[0] = GC.CollectionCount(0);
generationCollections[1] = GC.CollectionCount(1);
generationCollections[2] = GC.CollectionCount(2);
// Test for garbage here
for (int generation = 0; generation < generationCollections.Length; generation++)
{
Assert.AreEqual(GC.CollectionCount(generation), generationCollections[generation]);
}
}
}
I put the code in question where the "Test for garbage here" comment is and the results are unpredictable. My understanding is that this is due to the fact that GC runs on a separate thread and can be triggered by code other than my test at any time.
I tried GC.Collect to forcefully run collections before and after the test code but then realized that that always increment the collection count, so that test always fails.
Is there a meaningful way to test for garbage in a unit test?
You can use WMemoryProfiler to find out how many additional types were created. If you profile your own process you will get all addtional created types + some instances used by WMemoryProfiler to generate the report.
You can work around by using a separate process to monitor your managaed heap or by limiting yourself to only your types. If you leak memory you will see it normally in addtional instances created by you.
using (var dumper = new InProcessMemoryDumper(false,false))
{
var statOld = dumper.GetMemoryStatistics();
// allocaton code here
var diff = dumper.GetMemoryStatisticsDiff(statOld);
foreach (var diffinst in diff.Where(d => d.InstanceCountDiff > 1))
{
Console.WriteLine("Added {0} {1}", diffinst.TypeName, diffinst.InstanceCountDiff);
}
}
If you are after how much memory temporary objects did use you you will need to use some profiling Api or tools like PerfView which does use ETL traces generaeted by the CLR. For GC you would need to programatically enable specific stuff like his. I think the GCAllocationTick_V1 event would be interesting in your case as well.
If you do keep a reference to your object before you try to get the diff you would get a pretty good understanding how much memory your object graph will consume.
What you can try to do is to use exactly the same logic to dump GC state before actually asserting like
// do some logic
// GC.Collect, Thread.Sleep, ...
currentCollections[0] = GC.CollectionCount(0);
currentCollections[1] = GC.CollectionCount(1);
currentCollections[2] = GC.CollectionCount(2);
and after that do asserts with these dumped values (BTW in assertions first parameter is expected, and second one is actual)
for (int generation = 0; generation < generationCollections.Length; generation++)
{
Assert.AreEqual(generationCollections[generation], currentCollections(generation));
}
So this may work for most cases, but there is no way to make GC do something - you may just ask it do to something, and then wait in belief...
I have stumbled across a situation where garbage collection seems to be behaving differently between the same code running written as a Unit Test vs written in the Main method of a Console Application. I am wondering the reason behind this difference.
In this situation, a co-worker and I were in disagreement over the effects of registering an event handler on garbage collection. I thought that a demonstration would be better accepted than simply sending him a link to a highly rated SO answer. As such I wrote a simple demonstration as a unit test.
My unit test showed things worked as I said they should. However, my coworker wrote a console application that showed things working his way, which meant that GC was not occurring as I expected on the local objects in the Main method. I was able to reproduce the behavior he saw simply by moving the code from my test into the Main method of a Console Application project.
What I would like to know is why GC does not seem to collecting objects as expected when running in the Main method of a Console Application. By extracting methods so that the call to GC.Collect and the object going out of scope occurred in different methods, the expected behavior was restored.
These are the objects I used to define my test. There is simply an object with an event and an object providing a suitable method for an event handler. Both have finalizers setting a global variable so that you can tell when they have been collected.
private static string Log;
public const string EventedObjectDisposed = "EventedObject disposed";
public const string HandlingObjectDisposed = "HandlingObject disposed";
private class EventedObject
{
public event Action DoIt;
~EventedObject()
{
Log = EventedObjectDisposed;
}
protected virtual void OnDoIt()
{
Action handler = DoIt;
if (handler != null) handler();
}
}
private class HandlingObject
{
~HandlingObject()
{
Log = HandlingObjectDisposed;
}
public void Yeah()
{
}
}
This is my test (NUnit), which passes:
[Test]
public void TestReference()
{
{
HandlingObject subscriber = new HandlingObject();
{
{
EventedObject publisher = new EventedObject();
publisher.DoIt += subscriber.Yeah;
}
GC.Collect(GC.MaxGeneration);
GC.WaitForPendingFinalizers();
Thread.MemoryBarrier();
Assert.That(Log, Is.EqualTo(EventedObjectDisposed));
}
//Assertion needed for foo reference, else optimization causes it to already be collected.
Assert.IsNotNull(subscriber);
}
GC.Collect(GC.MaxGeneration);
GC.WaitForPendingFinalizers();
Thread.MemoryBarrier();
Assert.That(Log, Is.EqualTo(HandlingObjectDisposed));
}
I pasted the body above in to the Main method of a new console application, and converted the Assert calls to Trace.Assert invocations. Both equality asserts fail then fail. Code of resulting Main method is here if you want it.
I do recognize that when GC occurs should be treated as non-deterministic and that generally an application should not be concerning itself with when exactly it occurs.
In all cases the code was compiled in Release mode and targeting .NET 4.5.
Edit: Other things I tried
Making the test method static since NUnit supports that; test still worked.
I also tried extracting the whole Main method into an instance method on program and calling that. Both asserts still failed.
Attributing Main with [STAThread] or [MTAThread] in case this made a difference. Both asserts still failed.
Based on #Moo-Juice's suggestions:
I referenced NUnit to the Console app so that I could use the NUnit asserts, they failed.
I tried various changes to visibility to the both the test, test's class, Main method, and the class containing the Main method static. No change.
I tried making the Test class static and the class containing the Main method static. No change.
If the following code was extracted to a separate method, the test would be more likely to behave as you expected. Edit: Note that the wording of the C# language specification does not require this test to pass, even if you extract the code to a separate method.
{
EventedObject publisher = new EventedObject();
publisher.DoIt += subscriber.Yeah;
}
The specification allows but does not require that publisher be eligible for GC immediately at the end of this block, so you should not write code in such a way that you are assuming it can be collected here.
Edit: from ECMA-334 (C# language specification) §10.9 Automatic memory management (emphasis mine)
If no part of the object can be accessed by any possible continuation of execution, other than the running of finalizers, the object is considered no longer in use and it becomes eligible for finalization. [Note: Implementations might choose to analyze code to determine which references to an object can be used in the future. For instance, if a local variable that is in scope is the only existing reference to an object, but that local variable is never referred to in any possible continuation of execution from the current execution point in the procedure, an implementation might (but is not required to) treat the object as no longer in use. end note]
The problem isn't that it's a console application - the problem is that you're likely running it through Visual Studio - with a debugger attached! And/or you're compiling the console app as a Debug build.
Make sure you're compiling a Release build. Then go to Debug -> Start Without Debugging, or press Ctrl+F5, or run your console application from a command line. The Garbage Collector should now behave as expected.
This is also why Eric Lippert reminds you not to run any perf benchmarks in a debugger in C# Performance Benchmark Mistakes, Part One.
The jit compiler knows that a debugger is attached, and it deliberately de-optimizes the code it generates to make it easier to debug. The garbage collector knows that a debugger is attached; it works with the jit compiler to ensure that memory is cleaned up less aggressively, which can greatly affect performance in some scenarios.
Lots of the reminders in Eric's series of articles apply to your scenario. If you're interested in reading more, here are the links for parts two, three and four.
When I run my tests in Visual Studio individually, they all pass without a problem. However, when I run all of them at once some pass and some fail. I tried putting in a pause of 1 second in between each test method with no success.
Any ideas? Thanks in advance for your help...
It's possible that you have some shared data. Check for static member variables in the classes in use that means one test sets a value that causes a subsequent test to fail.
You can also debug unit tests. Depending on the framework you're using, you should be able to run the framework tool as a debug start application passing the path to the compiled assembly as a parameter.
It's very possible that some modifications/instantiations done in one test affect the others. That indicates poor test design and lack of proper isolation.
Everyone is probably right, some shared date is being modified between tests. But note the order of MS Test execution. Simply pausing between tests is not a solution. Each test is executed in it's own instance of the test class on a separate thread.
as per the other responses. It sounds like you have a singleton or a global variable that is causing the interaction.
Other unit test frameworks that I have used work hard to ensure that a test produces identical results whether the test is run individually or is run as part of the 'run them all' alternative. The goal is to prevent one test from having an effect on another due to side effects such as (for example) having one test leave the static state of a class in a configuration that another test is not expecting. The VS unit test framework does not appear to provide this isolation. I have 2 suggestions for minimizing the kinds of problems that the question implies. First, use non-static class in preference to a static class if the class has state (has anything other than static methods). Create a single instance of this class and have it keep the state info that was being kept in the static class. Second, if you do elect to have static class(es) with static state, have a static method that sets the static state back to 'empty' (e.g., a method that sets all static properties to null/zero/etc.). Call this at the end of each unit test to undo any effects that the test has imposed on the static state. (This is admittedly less than elegant but can be workable if done in moderation). Or do what I plan to do - find a unit test framework that provides isolation across tests.
I also ran into this problem although my issue ended up being a threading problem. In my case I was faking the HttpContext object since the tests relied on it's existence. However, I was setting this in the ClassInitialize method thinking this would be used for each method like below:
[ClassInitialize]
public static void ClassInit(TestContext testContext)
{
HttpContext.Current = new HttpContext(new HttpRequest(null, "http://tempuri.org", null), new HttpResponse(null));
}
However, it turns out that each test method in the class runs in a separate thread. So I had to add this code to every test method that relied upon it to fix the issue.
[TestMethod]
public void TestMethod1()
{
HttpContext.Current = new HttpContext(new HttpRequest(null, "http://tempuri.org", null), new HttpResponse(null));
...
}
[TestMethod]
public void TestMethod2()
{
HttpContext.Current = new HttpContext(new HttpRequest(null, "http://tempuri.org", null), new HttpResponse(null));
...
}
See link for more information on this.
http://blogs.msdn.com/b/nnaderi/archive/2007/02/17/explaining-execution-order.aspx
I faced a similar issue here how I solved it:
Copy the code of the second test inside the first test (after it).
Try to test the first test. The first test will probably fails, and then you can debug the first test (step by step) to find the static/shared variable or logic that makes the problem.
In my case, I had an Environment variable set with:
Environment.SetEnvironmentVariable("KEY", "value");
The implementation code for a few other tests was assuming a default value, and if the test(s) with the above line were executed first, those failed. The solution is to clean up with the following (at the end of each unit test, or in a special method for the same purpose - TestCleanup in MS/VSTest):
Environment.SetEnvironmentVariable("KEY", null);
Though redundant, it is best practice to also set the environment variable to any value previously assumed by (the failing) tests to be default. Do this at the top of those unit tests (the Arrange step).