I am using sos.dll and windbg to anayze a w3wp.exe dump. There is a high number of .Net CLR exceptions thrown per/sec shown in perfmon and i am trying to investigate this. I tried doing a !dumpheap -stat -type Exception. But does this show the exceptions that were thrown at the instance i took the dump or does this show all the exception object instances that were created? Exception object instances may be created without being thrown.
Is there a way to just get the exceptions that were thrown?
You use the wrong tools. Install Windows Performance Toolkit which is part of the Windows 10 SDK. The 1607 SDK can be used for Win8/10 systems, the older 1511 SDK can be used for Windows 7/2008R2.
Now use the WPRP profile that I posted here to capture the activity of your application by opening a cmd.exe as admin
"C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\wpr.exe" -start C:\DotNetRuntime.wprp
After captured some activity of your tool, run this command to stop the capturing:
"C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\wpr.exe" -stop C:\Result.etl
Now make a double click on the Result.etl to open it in Windows Performance Analyzer and load debug symbols.
Now drag & drop the Generic Event graph to the analysis pane, order the colums for Provider, process, Taskname, Field 1, Time, Opcode Name and Stack. Now filter for the Microsoft-Windows-DotNETRuntime provider and expand your process name entry and next expand the entry for Taskname Exception:
Here in this demo, the VS Addon Resharper caused a JetBrains.Application.Progress.ProcessCancelledException . Check which excceptions you see for your process and check the stack where the exceptions are raised.
Exceptions that are thrown are first chance exceptions. Since your program does not crash, this means they are caught and handled.
In addition to #magicandre1981's approach, I see two other options:
ProcDump
ProcDump can create crash dumps on first chance exceptions with the -e 1 command line switch. Also define -n to specify the maximum number of dumps you want to take. Once you became aware of an exception and no longer want it to be reported, use -f to filter it.
Advantage: you do not only have the exception, you also have a call stack and all the heap information which you can analyze later.
Disadvantage: this will significantly slow down your process and take a lot of disk space.
WinDbg exception commands
You could attach WinDbg to a process and use the sxe command with the -c switch to analyze first chance exceptions. Include g in the command to continue execution. It's helpful to write all output into a log file (use .logopen).
Example:
.symfix
.reload
.logopen c:\debug\logs\firstchance.log
.loadby sos clr
ld *
sxe -c "!pe;!clrstack;g" clr
g
.symfix and .reload may not be necessary. Just make sure your symbols are correct, otherwise all the analysis may be useless. The ld * will just pre-load things to make the analysis faster later on.
Advantage: you can capture as many information as you want without creating huge crash dumps.
Disadvantage: The execution of commands may significantly slow down the process. WinDbg may become unstable when you do this with hundreds of exceptions. (I never did this for a long time, this warning is given based on my experience with WinDbg 6.12 somewhen in 2010)
Related
This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
How to Debug .net applications without Visual Studio
Hi i made a small program it works for me and for a few people
but its being crashed at a friend
how can i debug and learn which lines exactly causes error?
without installing visual studio
Best you can do is (and is a good coding practice) in a new buid add try catch exception block in methods with logging facility that logs(in catch block) in lets say a text file
Method Name (Find using new StackTrace()..GetFrame(1).GetMethod().Name or System.Reflection.MethodBase.GetCurrentMethod().Name),
its parameters and
Exception message.
If you don't mean "without installing a debugger" you could try using Mono.
You can use WinDbg which can be run portable (xcopy from an existing installation), and with the SoS add-on it can help a lot in debugging C# applications - even in post mortem. It might be difficult to grasp at first, but it's worth learning - it will come very useful in several situations.
SoS reference here : http://msdn.microsoft.com/en-us/library/bb190764.aspx
I have this quick cheat-sheet with what I found most useful (found somewhere on the internet and then enriched with time) :
Starting, Attaching, Executing and Exiting
Start -> All Programs -> Debugging Tools for Windows -> WinDbg
F6
attach to process
Ctrl-Break
interrupt debugee
.detach
detach from a process
g
continue debugee execution
q
exit WinDbg
Getting Help
?
help on commands that affect the debugee
.help
help on commands that affect the debugger
.hh command
view the on line help file
!help
help on the extension dll at the top of the chain (e. g., SOS)
Issuing Commands
up arrow, down arrow, enter
scroll through command history
Right mouse button
paste into command window
Examining the Unmanaged Environment
lmf
list loaded modules with full path
lmt
list loaded modules with last modified timestamp
~
list unmanaged threads
~thread s
select a thread for thread specific commands
!token -n
view thread permissions
k
view the unmanaged call stack
!runaway
view thread CPU consumption
bp
set a breakpoint
.dump path
dump small memory image
.dump /ma path
dump complete memory image
Working with Extension DLLs (e. g., SOS)
.chain
list extensions dlls
.load clr10\sos
load SOS for debugging framework 1.0 / 1.1 (use .unload to unload)
.loadby sos mscorwks
load SOS for debugging framework 2.0
.loadby sos clr
load SOS for debugging framework 4.0
SOS Commands
!threads
view managed threads
!clrstack
view the managed call stack
!dumpstack
view combined unmanaged & managed call stack
!clrstack -p
view function call arguments
!clrstack –l
view stack (local) variables
!name2ee module class
view addresses associated with a class or method
!dumpmt –md address
view the method table & methods for a class
!dumpmd address
view detailed information about a method
!do address
view information about an object
!dumpheap –stat
view memory consumption by type
!dumpheap –min size
view memory consumption by object when at least size
!dumpheap –type type
view memory consumption for all objects of type type
!gcroot address
view which object are holding a reference to address
!syncblk
view information about managed locks
SOS 2.0 Commands
!bpmd module method
set breakpoint
!DumpArray address
view contents of an array
!PrintException
view information about most recent exception
If you don't want to install any other debugger,
Then you should simply implement trace and log on each section of code carefully and then you can view the traces or errors on the text file..
Ways to implement trace and log:
use can use TraceListener class on system.diaognosis namespace
or use
Microsoft.Practices.EnterpriseLibrary.Logging namespace's class `Logger` to implement tracing:
syntax:
Logger.Write(message, "Trace", 0, 0, System.Diagnostics.TraceEventType.Information);
My application is crashing constantly while I was trying run it from the release folder.
I put logs inside try catch blocks and capture them but they all pointed to one method. The deatiled problem is in my previous post.
Then I decided to use WinDbg and attach the executable to check what exactly is crashing my application. Now the info from WinDbg seems cryptic.
(13e4.1444): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
*** ERROR: Symbol file could not be found. Defaulted to export symbols for E:\VCS\DeskconWSP\Deskcon\bin\Release\tinyWRAP.dll -
eax=0e7e1c00 ebx=0d83d918 ecx=0d835b70 edx=0cce8ce0 esi=0d835b70 edi=ffffffff
eip=00000000 esp=0e4dfa4c ebp=0e4dfa58 iopl=0 nv up ei pl nz na po nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00010202
00000000 ?? ???
Any reference or pointers on how to use this debug info?
I mean this in the best way possible, but you need to read-up on WinDbg (see WinDbg A-Z). It has a huge learning curve, but it's really useful once you get used to it.
You need to configure WinDbg to load in the debug information for tinyWRAP.dll. There should be a file called tinyWRAP.PDB, assuming you're the developer for this file. Open File->Symbol Path and add as needed.
Microsoft supports a symbolserver (i.e., PDB files) for their own binaries. Add this "path" to the WinDbg symbol server path and WinDbg will download whatever it can find from MS: SRV*C:\SymbolServer\symserver*http://msdl.microsoft.com/download/symbols
Access violation just means the program is trying to access heap memory that is shouldn't; i.e., memory allocated for another process.
e.g., if you're doing arithmetic on a pointer to an integer without dereferencing it first, you'll end up pointing the variable to some other location which the process may not have access to.
You'll almost never see this in a purely managed program, but if you're interacting with native DLLs or code then this may give you a hint of what's going on.
Back in the Java world I was able to specify that the JVM should create a heap dump file on the first OutOfMemoryError. Is there anything equivalent in C#?
A heap dump would be ideal but I'd settle for a histogram.
If there's no way to do this automatically is there a way to hook this exception and then walk the heap manually?
You can use the MS Debugging tools ADPlus utility to get a process dump that includes the all the heaps.
Then using WinDBG.exe (also in debugging tools) you can use the !DumpHeap command to get all the data you need.
a little old but useful HOWTO link
I would use procdump because you can easily set it up before the crash to dump out the memory when the crash occurs. Then use windbg to look through the memory.
I would add that if it is not immediately obvious where the memory is going (using sos.dll tools command !dumpheap -stat), then you can use procdump to take a number of crash dumps at particular intervals so that you can track what memory is growing.
If you are not familiar with windbg and sos, you might want to check this out.
My .net application has a global exception handler by subscribing to AppDomain.Current.Domain UnhandledException event. On a few occassions i have seen that my application crashes but this global exception handler never gets hit. Not sure if its help but application is doing some COM interop.
My understanding is that as long as I don't have any local catch blocks swallowing the exception, this global exception handler should always be hit. Any ideas on what I might be missing causing this handler never been invoked?
Is this the cause of your problem?
AppDomain.CurrentDomain.UnhandledException not firing without debugging
The CLR is not all-powerful to catch every exception that unmanaged code can cause. Typically an AccessViolationException btw. It can only catch them when the unmanaged code is called from managed code. The scenario that's not supported is the unmanaged code spinning up its own thread and this thread causing a crash. Not terribly unlikely when you work with a COM component.
Since .NET 4.0, a Fatal Execution Engine exception no longer causes the UnhandledException event to fire. The exception was deemed too nasty to allow any more managed code to run. It is. And traditionally, a StackOverflowException causes an immediate abort.
You can diagnose this somewhat from the ExitCode of the process. It contains the exception code of the exception that terminated the process. 0x8013yyyy is an exception caused by managed code. 0xc0000005 is an access violation. Etcetera. You can use adplus, available from the Debugging Tools For Windows download to capture a minidump of the process. Since this is likely to be caused by the COM component, working with the vendor is likely to be important to get this resolved.
Since you are doing COM interop I do strongly suspect that some unmanaged code was running in another thread which did cause an unhandled exception. This will lead to application exit without a call to your unhandled exception handler.
Besides this with .NET 4.0 the policy did get stronger when the application is shut down without further notice.
Under the following conditions your application is shut down without further notice (Environmnt.FailFast).
Pre .NET 4:
StackOverFlowException
.NET 4:
StackoverFlowException
AccessViolationException
You can override the behaviour in .NET 4 by decorating a method with the HandleProcessCorruptedStateExceptionsAttribute or you can add the legacyCorruptedStateExceptionsPolicy tag to your App.config.
If your problem is an uncatched exception in unmanaged code you can either run your application under a debugger or you let it crash und collect a memory dump for post mortem debugging. Debugging crash dumps is usualy done with WindDbg.
After you have downloaded Windbg you have adplus (a vbs script located under Programm Files\Debugging Tools for Windows) which you can attach to your running process to trigger a crash dump when the process terminates due to an exception.
adplus -crash -p yourprocessid
Then you have a much better chance to find out what was going on when your process did terminate. Windows can also be configured to take a crash dump for you via DrWatson on older Windows Versions (Windows Error Reporting)
Crash Dump Generation
Hard core programmers will insist to create their own dump generation tool which basically uses the AEDebug registry key. When this key has a value which points to an existing executable it will be called when an application crashes which can e.g. show the Visual Studio Debugger Chooser Dialog or it can trigger the dump generation for your process.
Suspend Threads
An often overlooked thing is when you create a crash dump with an external tool (it is better to rely on external tools since you do not know how bad your process is corrupted and if it is out memory you are already in a bad situation) that you should suspend all threads from the crashed process before you take the dump.
When you take a big full memory dump it can take several minutes depending on the allocated memory of the faulted process. During this time the application threads can continue to wreak havoc on your application state leaving you with a dump which contains an inconsistent process state which did change during dump generation.
This would happen if your handler throws an exception.
It would also happen if you call Environment.FailFast or if you Abort the UI thread.
I have a Windows Forms application (.NET 4) that runs fine on my development machine but crashes on two other test machines. I can load the minidump that it creates in VS2010.
Choosing to "Debug with Mixed" leads to apparently endless (I killed devenv after about 20 minutes) abuse of the CPU by Visual Studio.
When I "Debug with Native Only", it can't find the source (even though I have mirrored the source in the same folder as on the test machine). It simply says:
Unhandled exception at 0x793f5b8c in
YourWinApp.exe.hdmp: 0xC0000409: 0xc0000409.
And then shows me
Call stack location: clr.dll!793f5b8c()
How would I find out what's causing the application to crash? Can I take a full crashdump whilst the "Notify Microsoft" dialog is being displayed, and would that help?
Minidump debugging was supposed to be majorly improved in VS2010. Haven't seen a lot of evidence for it myself yet, mixed-mode debugging looks as awkward as it was before when I did some quick tests. Don't take my word for it though. Native-only is however never going to show you a managed call stack.
Tackle this at the source. Write an event handler for AppDomain.CurrentDomain.UnhandledException and register it in your Main() method. Let it display the value of e.ExceptionObject.ToString() in, say, a message box. That gets you the managed stack trace of the exception. While that message box is displayed you could also snap the minidump, ought to get you closer to the crash location.
The particular exception you are getting is however definitely pointing to native C/C++ code. A buffer overflow that is corrupting the stack. Make sure you have the .pdb files for any native code your app uses. And setup the Microsoft symbol server so you get a good native stack trace from the minidump.
Edit: the fact that you don't get UnhandledException raised definitely points to stack integrity checking in the CRT. It was designed to not raise an exception but terminate the program immediately. Necessary behavior because the stack is compromised, the code cannot assume that it can be unwound safely. Given the crash location, it is likely that this check is actually done in the CLR code. I know this wasn't done in previous CLR versions but that might be different in the CLR version included with .NET 4.0
This is going to make it quite difficult to get a managed stack trace. There's a lot you can reverse-engineer from the unmanaged stack trace, as long as you setup the symbol server so that you'll get identifier names from the CLR stack frames. Post that stack trace in your question if you want help interpreting it. A bug in the CLR code is not unlikely btw, you may want to consider calling Microsoft Support. They will however need a consistent repro. They may make do with that all important stack trace if the repro is hard to come by. Setup the symbol server to get a good unmanaged stack trace. Easy in VS2010: Tools + Options, Debugging, Symbols, tick "Microsoft Symbol Servers".
You configure procdump to get full memory dump if the application has unhandled exception ,which you can debug it in VS or Windbg
And the minidump has call-stack information as watson buckets, here is one from CLR team and I wrote about the same
A brief explanation on the watson bucket information that you see in event viewer for unhandled exception
ExeFileName
Exe Assembly Version
Exe Assembly Timestamp
Full Name
Faulting Assembly version
Faulting assembly timestamp
Faulting assembly method def
Faulting method IL instruction that caused the exception
Exception type