Develop Reference

What is LogHelp_TerminateOnAssert?

There is a similar question from a decade ago, but there was no good answer - hopefully things have changed since then.
I have a fairly multithreaded Winforms app based on .NET 4.72. I am looking at it with Process Explorer Threads view and it has a lot of clr.dll!LogHelp_TerminateOnAssert+0x6835 type calls. I've setup the Symbols path but it didn't really clear anything up for me.
I took a dump of the application and ran it through DebugDiag and WinDbg and didn't see anything suspicious that stood out.
So my questions:
Should I be concerned with the large number of LogHelp_TerminateOnAssert calls?
Is the application leaking memory?
Does it have an excessive number of exceptions that don't filter down when I am running the app in Visual Studio?
The only entry from my code here is !get_FrameReceived and the stack for that thread is as follows:
The stack for the thread with the most cycles is like this:

Large offsets
clr.dll!LogHelp_TerminateOnAssert+0x6835
means that the actual execution in that method is 0x6835 = 26661 bytes away from its beginning. It's unlikely that a method is that big. (As #blabb points out, it's a 1 byte method).
Usually you see that when you have not set up the symbols correctly (like in the linked original question), but you have that fixed.
Chances are that Microsoft has only release the public symbols of clr.dll and not the private ones. In that case, you'll only see the last known public method.
Start address
Please note that the column is named "Start address". Process Explorer will show the first entry on the stack.
So this is where everything starts. You seem to be concerned that this is where everything ends.
Note: some known internal methods like RtlUserThreadStart and BaseThreadInitThunk will be skipped when displaying the start address. Otherwise they'd probably all look the same.
What the thread is really doing is on the top of the list, i.e. ZwRemoveIoCompletion, so it seems to do some IO operation.
Your questions
Should I be concerned with the large number of LogHelp_TerminateOnAssert calls?
No. These are just the starting point for something good. The GetQueuedCompletionStatus() looks like there's some IO going on and .NET uses IO Completion Ports (IOCP) for you.
Is the application leaking memory?
You don't tell that from a look at call stacks. You tell that by looking at the memory over time.
If you have too much network IO going on and the network can't keep up with it, .NET may have more and more items in the queue, so it may look like a memory leak.
Does it have an excessive number of exceptions that don't filter down when I am running the app in Visual Studio?
You would also not tell that from the call stack. You would attach a debugger (e.g. WinDbg) and check for exceptions (like sxe clr), if you don't trust Visual Studio.

on release build all these asserts are compiled into a simple ret something similar to
ifdef ( debug ) { function body here } elseif { ret } endif
so the symbols with such great offset are bogus
so you may need to load the actual symbols for that address for a sensible callstack
you can see the size of function in clr 4.0.30319 clr.dll is just 1 byte
0:000> x /v /t clr!LogHelp_TerminateOnAssert
pub func 100115a0 0 <NoType> clr!LogHelp_TerminateOnAssert (<no parameter info>)
0:000> .fnent clr!LogHelp_TerminateOnAssert
Debugger function entry 01bad5e0 for:
(100115a0) clr!RtlUnwindCallback | (100115a1) clr!memset
Exact matches:
clr!RtlUnwindCallback (void)
clr!_TlgDefineProvider_annotation__Tlgg_hClrProviderProv (void)
OffStart: 000115a0
ProcSize: 0x1
Prologue: 0x0
Params: 0n0 (0x0 bytes)
Locals: 0n0 (0x0 bytes)
Registers: 0n0
0:000> u clr!LogHelp_TerminateOnAssert l1
clr!RtlUnwindCallback:
100115a0 c3 ret

Sos.dll and windbg Get exceptions thrown

I am using sos.dll and windbg to anayze a w3wp.exe dump. There is a high number of .Net CLR exceptions thrown per/sec shown in perfmon and i am trying to investigate this. I tried doing a !dumpheap -stat -type Exception. But does this show the exceptions that were thrown at the instance i took the dump or does this show all the exception object instances that were created? Exception object instances may be created without being thrown.
Is there a way to just get the exceptions that were thrown?

You use the wrong tools. Install Windows Performance Toolkit which is part of the Windows 10 SDK. The 1607 SDK can be used for Win8/10 systems, the older 1511 SDK can be used for Windows 7/2008R2.
Now use the WPRP profile that I posted here to capture the activity of your application by opening a cmd.exe as admin
"C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\wpr.exe" -start C:\DotNetRuntime.wprp
After captured some activity of your tool, run this command to stop the capturing:
"C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\wpr.exe" -stop C:\Result.etl
Now make a double click on the Result.etl to open it in Windows Performance Analyzer and load debug symbols.
Now drag & drop the Generic Event graph to the analysis pane, order the colums for Provider, process, Taskname, Field 1, Time, Opcode Name and Stack. Now filter for the Microsoft-Windows-DotNETRuntime provider and expand your process name entry and next expand the entry for Taskname Exception:
Here in this demo, the VS Addon Resharper caused a JetBrains.Application.Progress.ProcessCancelledException . Check which excceptions you see for your process and check the stack where the exceptions are raised.

Exceptions that are thrown are first chance exceptions. Since your program does not crash, this means they are caught and handled.
In addition to #magicandre1981's approach, I see two other options:
ProcDump
ProcDump can create crash dumps on first chance exceptions with the -e 1 command line switch. Also define -n to specify the maximum number of dumps you want to take. Once you became aware of an exception and no longer want it to be reported, use -f to filter it.
Advantage: you do not only have the exception, you also have a call stack and all the heap information which you can analyze later.
Disadvantage: this will significantly slow down your process and take a lot of disk space.
WinDbg exception commands
You could attach WinDbg to a process and use the sxe command with the -c switch to analyze first chance exceptions. Include g in the command to continue execution. It's helpful to write all output into a log file (use .logopen).
Example:
.symfix
.reload
.logopen c:\debug\logs\firstchance.log
.loadby sos clr
ld *
sxe -c "!pe;!clrstack;g" clr
g
.symfix and .reload may not be necessary. Just make sure your symbols are correct, otherwise all the analysis may be useless. The ld * will just pre-load things to make the analysis faster later on.
Advantage: you can capture as many information as you want without creating huge crash dumps.
Disadvantage: The execution of commands may significantly slow down the process. WinDbg may become unstable when you do this with hundreds of exceptions. (I never did this for a long time, this warning is given based on my experience with WinDbg 6.12 somewhen in 2010)

How to interpret this stack trace

I recently released a Windows phone 8 app.
The app sometimes seem to crash randomly but the problem is it crash without breaking and the only info I get is a message on output that tells me there were an Access violation without giving any details.
So after releasing, from the crash reports I was able to obtain some more information, but they're kinda cryptical to me.
The info are:
Problem function: unknown //not very useful
Exception type: c0000005 //this is the code for Access violation exception
Stack trace:
Frame Image Function Offset
0 qcdx9um8960 0x00035426
1 qcdx9um8960 0x000227e2
I'm not used to work with memory pointer et similia and I'm not used to see a stack trace like that.
So I have those question:
How should I interpret/read those information, what's the meaning of every piece of information?
Is there a way to leverage those information to target my search for the problem?
Is there a way to get those information while debugging in VS2012
Notes:
I'm not asking what an Access Violation is
I tagged this as c# and c++ because my code is in c# but the exception is generated (I'm semi-guessing) by c++ implementation for the WebBrowser component
edit:
I tried setting the Debug type to Native only, this let me obtain the same info I had in the crash report on the dev center. This way the debugger break when the exception is thrown and let me see the disassebled code, unfortunately there's no qcdx9um8960 .pdb file (even on Microsoft Symbol Server), so I don't know the function name that caused the error.

Curiously, a search on the web for the image name "qcdx9um8960" returns several results referencing Windows Phone 8 and the WebBrowser control. Gathering the answers and replies (some even by MSFT), here is what you should possibly look into:
If you upgraded your application from Windows Phone 6/7 to 8, make sure you are not still referencing any 6/7 DLLs. 1
Make sure you aren't testing or publishing your software in Debug mode. There is a "qcdx9um8960.pdb" file that might be missing, causing the access violation. 1
"...there is a possible race condition known issue if the app has multiple copies of WebBrowser open. See if your code perhaps inadvertently makes more than one instance." 1
That image, "qcdx9um8960" is referencing a Qualcomm DirectX driver DLL. Perhaps it's not the WebBrowser component's fault, but the DirectX driver it might be using to render the web pages. 2
The name of the image suggests that the crash is happening on devices powered by a Qualcomm Snapdragon S4 Plus with model number MSM8960. 3
Assuming the processor above, and cross referencing Windows phones that use that chip, you're likely looking at the issue occurring on the Nokia Lumia 920T. 3 That's not to say that the driver doesn't work on several processor architectures or phones.
There are several other hits regarding crashes and issues debugging in the presence of that DLL, so unfortunately for you, I think you might be at the mercy of some third party software that has a few unresolved issues.
References
1 Access Violation since updated to WP8
2 [Toolkit][WP8] Performance issues with DepthStencilBuffer
3 Snapdragon (system on chip)

This kind of crash "should" never be caused by managed code, so you could go looking for a case where your app invokes some system or library API incorrectly. That's tedious. And the problem might have nothing to do with your app, it might be entirely internal to someone else's code. E.g, maybe WebBrowser crashes when user browses to some evil page. Or the failing code could be running on a thread that never even runs your code. From your observation that the debugger doesn't show any message before the access violation, and the fact that there are only 2 frames on the call stack, I suspect that's most likely.
So you should focus first on getting a (fairly) reliable repro scenario: the (minimal) set of steps that will (often or usually) produce the crash. This may involve interviewing the users who experienced the crash, or maybe some test automation on your part to try to accelerate the failure rate.
Once you have that, Microsoft (or another 3rd party) will accept responsibility -- managed code is never supposed to be able to cause an unhandled exception like access violation. And the scenario might give you a hint about how you can change your app's behavior to avoid the problem, because a real fix might take a long time to be released and distributed.

application level global exception handler didn't get hit

My .net application has a global exception handler by subscribing to AppDomain.Current.Domain UnhandledException event. On a few occassions i have seen that my application crashes but this global exception handler never gets hit. Not sure if its help but application is doing some COM interop.
My understanding is that as long as I don't have any local catch blocks swallowing the exception, this global exception handler should always be hit. Any ideas on what I might be missing causing this handler never been invoked?

Is this the cause of your problem?
AppDomain.CurrentDomain.UnhandledException not firing without debugging

The CLR is not all-powerful to catch every exception that unmanaged code can cause. Typically an AccessViolationException btw. It can only catch them when the unmanaged code is called from managed code. The scenario that's not supported is the unmanaged code spinning up its own thread and this thread causing a crash. Not terribly unlikely when you work with a COM component.
Since .NET 4.0, a Fatal Execution Engine exception no longer causes the UnhandledException event to fire. The exception was deemed too nasty to allow any more managed code to run. It is. And traditionally, a StackOverflowException causes an immediate abort.
You can diagnose this somewhat from the ExitCode of the process. It contains the exception code of the exception that terminated the process. 0x8013yyyy is an exception caused by managed code. 0xc0000005 is an access violation. Etcetera. You can use adplus, available from the Debugging Tools For Windows download to capture a minidump of the process. Since this is likely to be caused by the COM component, working with the vendor is likely to be important to get this resolved.

Since you are doing COM interop I do strongly suspect that some unmanaged code was running in another thread which did cause an unhandled exception. This will lead to application exit without a call to your unhandled exception handler.
Besides this with .NET 4.0 the policy did get stronger when the application is shut down without further notice.
Under the following conditions your application is shut down without further notice (Environmnt.FailFast).
Pre .NET 4:
StackOverFlowException
.NET 4:
StackoverFlowException
AccessViolationException
You can override the behaviour in .NET 4 by decorating a method with the HandleProcessCorruptedStateExceptionsAttribute or you can add the legacyCorruptedStateExceptionsPolicy tag to your App.config.
If your problem is an uncatched exception in unmanaged code you can either run your application under a debugger or you let it crash und collect a memory dump for post mortem debugging. Debugging crash dumps is usualy done with WindDbg.
After you have downloaded Windbg you have adplus (a vbs script located under Programm Files\Debugging Tools for Windows) which you can attach to your running process to trigger a crash dump when the process terminates due to an exception.
adplus -crash -p yourprocessid
Then you have a much better chance to find out what was going on when your process did terminate. Windows can also be configured to take a crash dump for you via DrWatson on older Windows Versions (Windows Error Reporting)
Crash Dump Generation
Hard core programmers will insist to create their own dump generation tool which basically uses the AEDebug registry key. When this key has a value which points to an existing executable it will be called when an application crashes which can e.g. show the Visual Studio Debugger Chooser Dialog or it can trigger the dump generation for your process.
Suspend Threads
An often overlooked thing is when you create a crash dump with an external tool (it is better to rely on external tools since you do not know how bad your process is corrupted and if it is out memory you are already in a bad situation) that you should suspend all threads from the crashed process before you take the dump.
When you take a big full memory dump it can take several minutes depending on the allocated memory of the faulted process. During this time the application threads can continue to wreak havoc on your application state leaving you with a dump which contains an inconsistent process state which did change during dump generation.

This would happen if your handler throws an exception.
It would also happen if you call Environment.FailFast or if you Abort the UI thread.

How do I see the c# code that caused a crashdump in clr.dll?

I have a Windows Forms application (.NET 4) that runs fine on my development machine but crashes on two other test machines. I can load the minidump that it creates in VS2010.
Choosing to "Debug with Mixed" leads to apparently endless (I killed devenv after about 20 minutes) abuse of the CPU by Visual Studio.
When I "Debug with Native Only", it can't find the source (even though I have mirrored the source in the same folder as on the test machine). It simply says:
Unhandled exception at 0x793f5b8c in
YourWinApp.exe.hdmp: 0xC0000409: 0xc0000409.
And then shows me
Call stack location: clr.dll!793f5b8c()
How would I find out what's causing the application to crash? Can I take a full crashdump whilst the "Notify Microsoft" dialog is being displayed, and would that help?

Minidump debugging was supposed to be majorly improved in VS2010. Haven't seen a lot of evidence for it myself yet, mixed-mode debugging looks as awkward as it was before when I did some quick tests. Don't take my word for it though. Native-only is however never going to show you a managed call stack.
Tackle this at the source. Write an event handler for AppDomain.CurrentDomain.UnhandledException and register it in your Main() method. Let it display the value of e.ExceptionObject.ToString() in, say, a message box. That gets you the managed stack trace of the exception. While that message box is displayed you could also snap the minidump, ought to get you closer to the crash location.
The particular exception you are getting is however definitely pointing to native C/C++ code. A buffer overflow that is corrupting the stack. Make sure you have the .pdb files for any native code your app uses. And setup the Microsoft symbol server so you get a good native stack trace from the minidump.
Edit: the fact that you don't get UnhandledException raised definitely points to stack integrity checking in the CRT. It was designed to not raise an exception but terminate the program immediately. Necessary behavior because the stack is compromised, the code cannot assume that it can be unwound safely. Given the crash location, it is likely that this check is actually done in the CLR code. I know this wasn't done in previous CLR versions but that might be different in the CLR version included with .NET 4.0
This is going to make it quite difficult to get a managed stack trace. There's a lot you can reverse-engineer from the unmanaged stack trace, as long as you setup the symbol server so that you'll get identifier names from the CLR stack frames. Post that stack trace in your question if you want help interpreting it. A bug in the CLR code is not unlikely btw, you may want to consider calling Microsoft Support. They will however need a consistent repro. They may make do with that all important stack trace if the repro is hard to come by. Setup the symbol server to get a good unmanaged stack trace. Easy in VS2010: Tools + Options, Debugging, Symbols, tick "Microsoft Symbol Servers".

You configure procdump to get full memory dump if the application has unhandled exception ,which you can debug it in VS or Windbg
And the minidump has call-stack information as watson buckets, here is one from CLR team and I wrote about the same
A brief explanation on the watson bucket information that you see in event viewer for unhandled exception
ExeFileName
Exe Assembly Version
Exe Assembly Timestamp
Full Name
Faulting Assembly version
Faulting assembly timestamp
Faulting assembly method def
Faulting method IL instruction that caused the exception
Exception type