Develop Reference

Sos.dll and windbg Get exceptions thrown

I am using sos.dll and windbg to anayze a w3wp.exe dump. There is a high number of .Net CLR exceptions thrown per/sec shown in perfmon and i am trying to investigate this. I tried doing a !dumpheap -stat -type Exception. But does this show the exceptions that were thrown at the instance i took the dump or does this show all the exception object instances that were created? Exception object instances may be created without being thrown.
Is there a way to just get the exceptions that were thrown?

You use the wrong tools. Install Windows Performance Toolkit which is part of the Windows 10 SDK. The 1607 SDK can be used for Win8/10 systems, the older 1511 SDK can be used for Windows 7/2008R2.
Now use the WPRP profile that I posted here to capture the activity of your application by opening a cmd.exe as admin
"C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\wpr.exe" -start C:\DotNetRuntime.wprp
After captured some activity of your tool, run this command to stop the capturing:
"C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\wpr.exe" -stop C:\Result.etl
Now make a double click on the Result.etl to open it in Windows Performance Analyzer and load debug symbols.
Now drag & drop the Generic Event graph to the analysis pane, order the colums for Provider, process, Taskname, Field 1, Time, Opcode Name and Stack. Now filter for the Microsoft-Windows-DotNETRuntime provider and expand your process name entry and next expand the entry for Taskname Exception:
Here in this demo, the VS Addon Resharper caused a JetBrains.Application.Progress.ProcessCancelledException . Check which excceptions you see for your process and check the stack where the exceptions are raised.

Exceptions that are thrown are first chance exceptions. Since your program does not crash, this means they are caught and handled.
In addition to #magicandre1981's approach, I see two other options:
ProcDump
ProcDump can create crash dumps on first chance exceptions with the -e 1 command line switch. Also define -n to specify the maximum number of dumps you want to take. Once you became aware of an exception and no longer want it to be reported, use -f to filter it.
Advantage: you do not only have the exception, you also have a call stack and all the heap information which you can analyze later.
Disadvantage: this will significantly slow down your process and take a lot of disk space.
WinDbg exception commands
You could attach WinDbg to a process and use the sxe command with the -c switch to analyze first chance exceptions. Include g in the command to continue execution. It's helpful to write all output into a log file (use .logopen).
Example:
.symfix
.reload
.logopen c:\debug\logs\firstchance.log
.loadby sos clr
ld *
sxe -c "!pe;!clrstack;g" clr
g
.symfix and .reload may not be necessary. Just make sure your symbols are correct, otherwise all the analysis may be useless. The ld * will just pre-load things to make the analysis faster later on.
Advantage: you can capture as many information as you want without creating huge crash dumps.
Disadvantage: The execution of commands may significantly slow down the process. WinDbg may become unstable when you do this with hundreds of exceptions. (I never did this for a long time, this warning is given based on my experience with WinDbg 6.12 somewhen in 2010)

View/Diagnose logical .NET threads in memory dump using Visual Studio debugger

I have a service that is reporting a large amount of logical threads. From PerfMon:
.NET CLR LocksAndThreads -> # of current logical threads: 663
.NET CLR LocksAndThreads -> # of current physical threads: 659
Process -> Thread Count: 15
This is too high, so I captured a memory dump (via sysinternals procdump.exe) and opened it from Visual Studio (Debug with Mixed). Once everything is loaded up, I looked in the threads window, and it only shows the 15 OS threads, not the .net physical or .net logical. The service itself is a windows service that hosts 4 WCF services (System.ServiceModel.ServiceHost).
How do I find out what these threads are, so that I can fix the code and get rid of them?
How do I get the logical threads to be recognized and displayed by visual studio?
Is it a problem with Visual Studio, or a problem with the dump itself?

First, you need to obtain a memory dump. There are a variety of methods to do this. The easiest one that I've found is procdump.exe, part of SysInternals, available with documentation here.
Next, you need to download and install WinDbg and get SOS working (SOS is the module that let's you view .net managed processes). More information on setting that up is available here. If you got your dump from a server (as was my case), then the .net versions might be slightly off, and SOS won't be able to work correctly with the dump file. If that happens, you'll need to copy a few relevant files from the source .net version to your windbg installation as per this comment.
Once everything is setup and you have the dump loaded, run the command:
!threads
This should give you a list of .net logical threads. Also of note is that before the list of threads, it will give you a summary. Of importance for this question, DeadThread was extremely high (600+). This indicates that the threads have completed, but the memory they are holding (the stack) can't be released.
In my specific case, I found that the threads that were getting hung were threads from the WCF thread pool that couldn't GC because other threads were holding references to it. I changed the other threads to null out the parent thread when they didn't need it anymore, , and that allowed it to get GC'd correctly.

How do I see the c# code that caused a crashdump in clr.dll?

I have a Windows Forms application (.NET 4) that runs fine on my development machine but crashes on two other test machines. I can load the minidump that it creates in VS2010.
Choosing to "Debug with Mixed" leads to apparently endless (I killed devenv after about 20 minutes) abuse of the CPU by Visual Studio.
When I "Debug with Native Only", it can't find the source (even though I have mirrored the source in the same folder as on the test machine). It simply says:
Unhandled exception at 0x793f5b8c in
YourWinApp.exe.hdmp: 0xC0000409: 0xc0000409.
And then shows me
Call stack location: clr.dll!793f5b8c()
How would I find out what's causing the application to crash? Can I take a full crashdump whilst the "Notify Microsoft" dialog is being displayed, and would that help?

Minidump debugging was supposed to be majorly improved in VS2010. Haven't seen a lot of evidence for it myself yet, mixed-mode debugging looks as awkward as it was before when I did some quick tests. Don't take my word for it though. Native-only is however never going to show you a managed call stack.
Tackle this at the source. Write an event handler for AppDomain.CurrentDomain.UnhandledException and register it in your Main() method. Let it display the value of e.ExceptionObject.ToString() in, say, a message box. That gets you the managed stack trace of the exception. While that message box is displayed you could also snap the minidump, ought to get you closer to the crash location.
The particular exception you are getting is however definitely pointing to native C/C++ code. A buffer overflow that is corrupting the stack. Make sure you have the .pdb files for any native code your app uses. And setup the Microsoft symbol server so you get a good native stack trace from the minidump.
Edit: the fact that you don't get UnhandledException raised definitely points to stack integrity checking in the CRT. It was designed to not raise an exception but terminate the program immediately. Necessary behavior because the stack is compromised, the code cannot assume that it can be unwound safely. Given the crash location, it is likely that this check is actually done in the CLR code. I know this wasn't done in previous CLR versions but that might be different in the CLR version included with .NET 4.0
This is going to make it quite difficult to get a managed stack trace. There's a lot you can reverse-engineer from the unmanaged stack trace, as long as you setup the symbol server so that you'll get identifier names from the CLR stack frames. Post that stack trace in your question if you want help interpreting it. A bug in the CLR code is not unlikely btw, you may want to consider calling Microsoft Support. They will however need a consistent repro. They may make do with that all important stack trace if the repro is hard to come by. Setup the symbol server to get a good unmanaged stack trace. Easy in VS2010: Tools + Options, Debugging, Symbols, tick "Microsoft Symbol Servers".

You configure procdump to get full memory dump if the application has unhandled exception ,which you can debug it in VS or Windbg
And the minidump has call-stack information as watson buckets, here is one from CLR team and I wrote about the same
A brief explanation on the watson bucket information that you see in event viewer for unhandled exception
ExeFileName
Exe Assembly Version
Exe Assembly Timestamp
Full Name
Faulting Assembly version
Faulting assembly timestamp
Faulting assembly method def
Faulting method IL instruction that caused the exception
Exception type

Visual Studio and Windbg: which one is better to debug large scale managed application

Visual Studio seems is very native to debug the C# project but when the scale is up seems it is not very convenient to debug, for example when the project contains some code that is calling in/out of native code...
I wanna to know if there is any evidence that windbg is better over VS.Net when we are debugging large/serious project.
Another question is without the SOS extension can windbg set breakpoint on C# source file?

WinDbg is at least an order of magnitude faster than VS when debugging unmanaged applications.
It is faster when debugging managed applications as well; however, it does not offer the level of support for managed debugging VS offers.
WinDbg can't be used for managed debugging without SOS.
On a separate note, sometimes it is possible to get the best of both worlds by running VS inside WinDbg. However, you need to:
Have a screaming fast machine with tons of memory. Don't even dream of doing this in a VM.
Don't forget to set the option on WinDbg to attach to child processes as well.
Avoid building in VS while running under WinDbg.

This hanselminutes podcast goes into some detail on windbg:
Scott's in Sweden this week and he sat down with master debugger and ASP.NET Escalation Engineer Tess Ferrandez. She explains .NET Debugging 101. What's a dump file? Do you need PDBs? How do you use WinDBG and what are the best ways to debug memory issues, perf problems and hangs.

I use WinDbg on a regular basis for debugging a very large mixed mode application. I find that the options for digging into the runtime details, the heaps, threads, and so forth are very useful. Several of the options are hardly supported in VS, so WinDbg is really useful in that context.
The biggest problem with WinDbg is probably its long and steep learning curve, but once you have a grasp of the basics I find WinDbg a lot easier to use than VS for many tasks. I have used it for some years now and I still learn new stuff on a regular basis.
The current version of WinDbg + SOS does not support source level debugging for managed code. If you really need this try to locate version 6.7.5. For additional details see this post: http://voneinem-windbg.blogspot.com/2007/04/windbg-6750-released.html
The next version of SOS (for .NET 4) does support some source level debugging, as it is able to extract source file and line info for managed call stacks and disassembly (using the !u command).

C# How to dump all variables & current values during runtime

Are there any in-built or 3rd party libraries that allow you to simply dump all variables in memory during run time? What I would like is to be able to view variables & current values similarly to viewing them by hitting a break point and hovering over variables, but without actually having to halt the program execution (i.e. just get a snapshot). Would be good if it could dump them to a file which can then be opened later in a program to get a nice GUI interface to view them, but simple text file dump would be good enough.

I can't think of an easy way to do this in a generic fashion. What could work is programmatically creating a dump file of your running process. You could either do this with P/Invoke to the dbghelp.dll routines or spawn a cdb.exe process to create the dump file. Once you have the file, you could open it up in a debugger for later analysis using SOS.dll with cdb.exe/windbg.exe, or even write a debugger script to dump the data you want (mostly) automatically.

I believe some sort of logging framework would help you to do that...
Check out:
http://www.dotnetlogging.com/
At my workplace we use log4net which works pretty well for us.
So how come you're wanting to dump out all the variables for later analysis? Have you considered writing your code test first so that you can reduce your reliance on the debugger and have a suite of automated test checking the values for you?

In the past I've used the YourKit .Net profiler in order to profile .Net applications.
While I've only ever used it to connect to running applications personally the Snapshot documentation does state that they have a Profiler API that can be used to programmatically dump snapshots for later review.
Code wise this looks to be as simple as the following:
Controller c = new Controller();
String snapshotPath = c.CaptureSnapshot();
I believe you can then load the snapshot files into the YourKit GUI at a later date to review them.
I would not be surprised if some of the other popular profilers like JetBrains dotTrace Performance and RedGates ANTS Performance Profiler have similar programmatic APIs but I couldn't quickly find obvious documentation on their websites (and I didn't want to watch their webinars to find out if this feature existed!)

For this you can use WMemoryProfiler to
Get all objects in all appdomains as an object array
Create a memory dump of your own process
Serialize specific objects to disc
To make this happen you need Windbg of course but the Api of WMemoryProfiler is fully managed and you can basically self debug your process. The library takes care of the usual debugger oddities since it does wrap Windbg in a nice accessible library.
The code below does get all instances of System.Threading.Thread objects into an object array. This way you can write a visualizer for your own application objects at runtime. The other overload does simply give you all objects in all AppDomains.
using (var debugger = new MdbEng())
{
var dummy = new Thread(() => {});
dummy.Name = "Dummy Thread";
// Get all thread objects in all AppDomains
var threads = debugger.GetObjects("System.Threading.Thread", true);
foreach (Thread t in threads)
{
Console.WriteLine("Managed thread {0} has Name {1}", t.ManagedThreadId, t.Name);
}
GC.KeepAlive(dummy);
}
Since it is a wrapper around Windbg you can also create a memory dump on the fly and later load a memory dump from your process to extract object data for visualization from the dump. Commerical Memory Profilers (e.g. MemoryProfiler from Scitech) employ this technique since years but it is quite slow when you have a huge memory dump since they are using also Windbg as dump analyzer.

You can try Intellitrace tool provided with ultimate version of visual studio. It is what you describe - it records what is happening in your app and allows you to debug it without executing your program with hovering over variables and all other debug windows to help you.

You can use PostSharp . I found it very useful to record debug times because of the environment application was deployed. And instrumented/recorded many things.
But obviously you'll need to specify all variables you need to record.
Check more details here.