I have a very long (over 3 hour, and somewhat manual) pre-processing method to obtain all the data I need to run an analysis. I am running this in debug mode and the pre-processing works great and I get all the data I want correctly, however once I start processing the data, I discover a bug. If I stop the process I will have to re-run the pre-processing again, only to discover another possible bug. Is there a way to save this pre-processed data, so I can just dump it into memory without having to pre-process everytime without stopping the process?
I am break pointed just after the pre-process and before the processing, and would kind of like a save point without having to STOP the process and add code.
If the data takes long to generate, but there's not actually that much of it, then you could use serialization to write your data into a file.
Probably the simplest option would be to use BinaryFormatter: you just need to mark all the types that you want to save as [Serializable] and it will work automatically.
Not sure I fully understand your requirement but how about creating a memory dump file? Then you can continue execution as many times as you want from a known save point. See on MSDN: Use Dump Files to Debug App Crashes and Hangs in Visual Studio.
I have a computationally-expensive multi-threaded C# app that seems to crash consistently after 30-90 minutes of running. The error it gives is
The runtime has encountered a fatal error. The address of the error was at 0xec37ebae, on thread 0xbcc. The error code is 0xc0000005. This error may be a bug in the CLR or in the unsafe or non-verifiable portions of user code. Common sources of this bug include user marshaling errors for COM-interop or PInvoke, which may corrupt the stack.
(0xc0000005 is the error-code for Access Violation)
My app does not invoke any native code, or use any unsafe blocks, or even any non-CLS compliant types like uint. In fact, the line of code that the debugger says caused the crash is
overallLength += distanceTravelled;
Where both values are of type double
Given all this, I believe the crash must be due to a bug in the compiler or CLR or JIT. I'd like to figure out what causes it, or at the very least write a smaller reproduction to send into Microsoft, but I have no idea where to even begin. I've never had to view the CIL-binary, or the compiled JIT output, or the native stacktrace (there is no managed stacktrace at the time of the crash), so I'm not sure how. I can't even figure out how to view the state of all the variables at the time of the crash (VS unfortunately won't tell me like it does after managed-exceptions, and outputting them to console/a file would slow down the app 1000-fold, which is obviously not an option).
So, how do I go about debugging this?
[Edit] Compiled under VS 2010 SP1, running latest version of .Net 4.0 Client Profile. Apparently it's ".Net 4.0C/.Net 4.0E, .Net CLR 1.1.4322"
I'd like to figure out what causes it, or at the very least write a smaller reproduction to send into Microsoft, but I have no idea where to even begin.
"Smaller reproduction" definitely sounds like a great idea here... even if "smaller" won't mean "quicker to reproduce".
Before you even start, try to reproduce the error on another machine. If you can't reproduce it on another machine, that suggests a whole different set of tests to do - hardware, installation etc.
Also, check you're on the latest version of everything. It would be annoying to spend days debugging this (which is likely, I'm afraid) and then end up with a response of "Yes, we know about this - it was a bug in .NET 4 which was fixed in .NET 4.5" for example. If you can reproduce it on a variety of framework versions, that would be even better :)
Next, cut out everything you can in the program:
Does it have a user interface at all? If possible, remove that.
Does it use a database? See if you can remove all database access: definitely any output which isn't used later, and ideally input too. If you can hard code the input within the app, that would be ideal - but if not, files are simpler for reproductions than database access.
Is it data-sensitive? Again, without knowing much about the app it's hard to know whether this is useful, but assuming it's processing a lot of data, can you use a binary search to find a relatively small amount of data which causes the problem?
Does it have to be multi-threaded? If you can remove all the threading, obviously that may well then take much longer to reproduce the problem - but does it still happen at all?
Try removing bits of business logic: if your app is componentized appropriately, you can probably fake out whole significant components by first creating a stub implementation, and then simply removing the calls.
All of this will gradually reduce the size of the app until it's more manageable. At each step, you'll need to run the app again until it either crashes or you're convinced it won't crash. If you have a lot of machines available to you, that should help...
tl;dr Make sure you're compiling to .Net 4.5
This sounds suspiciously like the same error found here. From the MSDN page:
This bug can be encountered when the Garbage Collector is freeing and compacting memory. The error can happen when the Concurrent Garbage Collection is enabled and a certain combination of foreground Garbage Collection and background Garbage Collection occurs. When this situation happens you will see the same call stack over and over. On the heap you will see one free object and before it ends you will see another free object corrupting the heap.
The fix is to compile to .Net 4.5. If for some reason you can't do this, you can also disable concurrent garbage collection by disabling gcConcurrent in the app.config file:
<configuration>
<runtime>
<gcConcurrent enabled="false"/>
</runtime>
</configuration>
Or just compile to x86.
WinDbg is your friend:
http://blogs.msdn.com/b/tess/archive/2006/02/09/net-crash-managed-heap-corruption-calling-unmanaged-code.aspx
http://www.codeproject.com/Articles/23589/Get-Started-Debugging-Memory-Related-Issues-in-Net
http://www.codeproject.com/Articles/22245/Quick-start-to-using-WinDbg
Download Debug Diagnostic Tool v1.2
Run program
Add Rule "Crash"
Select "Specific Process"
on page Advanced Configuration set your exception if you know on which exception it fails or just leave this page as is
Set userdump location
Now wait for process to crash, log file is created by DebugDiag. Now activate tab Advanced Analysis, select Crash/Hang Analyzers in top list and dump file in lower list and hit Start Analysis. This will generate html report for you. Hopes you found usefull info in that report. If you have problem with analyze, upload html report somewhere and place url here so we can focus on it.
My app does not invoke any native code, or use any unsafe blocks, or
even any non-CLS compliant types like uint
You may think this, but threading, synchronization via semaphore, mutex it any handles all are native. .net is a layer over operating system, .net itself does not support pure clr code for multithreading apps, this is because OS already does it.
Most likely this is thread synchronization error. Probably multiple threads are trying to access shared resource like file etc that is outside clr boundary.
You may think you aren't accessing com etc, but when you call certain API like get desktop folder path etc it is called through shell com API.
You have following two options,
Publish your code so that we can review the bottleneck
Redesign your app using .net parallel threading framework, which includes variety of algorithms requiring CPU intensive operations.
Most likely programs fail after certain period of time as collections grow up and operations fail to execute before other thread interfere. For example, producer consumer problem, you will not notice any problem till producer will become slower or fail to finish its operation before consumer kicks in.
Bug in clr is rare, because clr is very stable. But poorly written code may lead error to appear as bug in clr. Clr can not and will never detect whether the bug is in your code or in clr itself.
Did you run a memory test for your machine as the one time I had comparable symptoms one of my dimms turned out to be faulty (a very good memorytester is included in Win7; http://www.tomstricks.com/how-to-test-your-ram-or-memory-with-windows-memory-diagnostic-tool-in-windows-7/)
It might also be a heating/throttling issue if your CPU gets too hot after this period of time. Although that would happen sooner imho.
There should be a dumpfile that you can analyze. If you never did this find someone who did, or send that to microsoft
I will suggest you open a support case via http://support.microsoft.com immediately, as the support guys can show you how to collect the necessary information.
Generally speaking, like #paulsm4 and #psulek said, you can utilize WinDbg or Debug Diag to capture crash dumps of the process, and within it, all necessary information is embedded. However, if this is the very first time you use those tools, you might be puzzled. Microsoft support team can provide you step by step guidance on them, or they can even set up a Live Meeting session with you to capture the data, as the program crashes so often.
Once you are familiar with the tools, in the future you can perform similar troubleshooting more easily,
http://blogs.msdn.com/b/lexli/archive/2009/08/23/when-the-application-program-crashes-on-windows.aspx
BTW, it is too early to say "I've found a bug". Though you cannot obviously find in your program a dependency on native code, it might still have a dependency on native code. We should not draw a conclusion before debugging further into the issue.
I am thinking about adding a diagnostics mode build into an app I am writing to count method usage and execution time, similar to what many code profilers like dotTrace do.
I'm having some trouble finding resources through google on how to accomplish this though; obviously it is possible, but can anyone point me to some resources on how I can implement something like method call counts in .NET?
The Code Project article Creating a Custom .NET Profiler describes the process of creating a profiler using the CLR profiler hooks.
This involves creating a COM object that implements the ICorProfilerCallback2 interface and then using environment variables to indicate to the CLR that we wish to profile by using this class:
When the CLR begins a process, it looks for two environment variables:
COR_ENABLE_PROFILING: This environment variable is set to either 1 or 0. 1 indicates that the CLR should use a profiler. 0 (or the non-existence of this environment variable) indicates that it should not use a profiler.
COR_PROFILER: Now that we've told the CLR that we want to profile, we have to tell it which profiler to use. Because profilers are implemented as COM objects, this environment variable will be set to the GUID of the coclass that implements the ICorProfilerCallback2 interface.
Perhaps I am being too simple here, but my solution to this would be logging. Using entlib or log4net and log debug level messages. Then you can just write a little script/program to analyse the log file and give you the method count. There might even be other log diagnostic tools.
Unless you need rich visualization or real time complex relationship mapping etc. Would you need a profiler? For method count and execution time, wouldn't a log file suffice? Once you are in production or don't care about instrumentation you turn your logging level up and forget about those debug messages.
Is there any tool such that it can get a heap dump from a running application and determine/group objects by where in source code they were created?
With no changes to source code and ideally something free.
What about .NET Memory Profiler from ANTS for example.
Maybe CLR Profiler.
The information is not available if you create a memory dump. In order to gather this, you have to monitor the process as it is running. You could launch the application via WinDbg and set breakpoints on all the constructors you're interested in (hopefully you don't want to look at each and every object).
If you create the breakpoint, so it dumps the stack you will have the point of creation for the object. However, keep in mind that the objects may move around during GC, which will make parring objects with stacks difficult (or even impossible in some cases).
Since your question is tagged with performance and profiling, I gather that you want to reduce memory allocations. Why not just look at the numbers of objects created (or possibly look at the largest objects created) by looking at the heap. Then go through the source code and figure out where such instances are created.
As others suggested memory profilers, Memprofiler is definitely the most advanced one (I've tried all existing .NET profilers). It has a 14 day trial.
You need a .NET memory profiler. These tools allow you to follow the object graphs on the garbage collected heap and can be very useful in identifying the sources of memory leaks. While they may not necessarily tell you the method where an object was created they will tell which instances of which classes are holding on to the objects and allow you to take differences of snap shots of the gc heap. They don't require modifications to source code. You may want to have a look at
What Are Some Good .NET Profilers?
Our QA teams use http://www.jetbrains.com/profiler/ for this kind of thing here when we run into bottlenecks. I'm pretty sure it will give you a list of allocations by method call. I'll go install it and check :)
Good old windbg + sos + pdb will make the dumping.
As for the "where in source code they were created" part - is impossible without instrumentation or injection.
The SOS Debugging Extension
How to use : http://msdn.microsoft.com/en-us/library/yy6d2sxs.aspx
is there a way to get, for debuging purpose, a list of needed RAM for all my objects incl. sub-objects and so on, written to a text-file?
I need to get these values while running in production mode without visual studio running. For example every x minutes. Is this possible? How can I do that?
Thank you very much in advance,
Chris
Load windbg and load sos.dll, and attach to process.
log the output to a file
then do a !DumpHeap -stat (and variants)
psscor2.dll can dump the heap to an xml file. (or one for use with ClrProfiler)
You can use JetBrains DotTrace to do the memory profiling. Its only 30 days trial but what a great tool to navigate through specific in-memory objects.
You can create a tool, that will monitor size of process.
Or do this in seperate thread of running application.
Process.GetProcessesByName()...
For value types, you should be able to use sizeof(). I don't think there is an equivalent for reference types, though. Note you can only use sizeof() inside an unsafe block.