Develop Reference

Application hang: interpret WinDbg commands' output

My .NET application uses BlockingCollection to process data received from RabbitMQ. Sometimes on production it stops processing messages. I made dump using procdump util and tried to inspect the dump using WinDbg. !threads command shows me one thread with LockCount=1. ~*e!clrstack command gave me this output for another thread:
Child SP IP Call Site
0f3bf068 7760c8ac [GCFrame: 0f3bf068]
0f3bf118 7760c8ac [HelperMethodFrame_1OBJ: 0f3bf118] System.Threading.Monitor.ObjWait(Boolean, Int32, System.Object)
0f3bf1a4 70ac5fb7 System.Threading.Monitor.Wait(System.Object, Int32, Boolean)
0f3bf1b4 70ad76a7 System.Threading.SemaphoreSlim.WaitUntilCountOrTimeout(Int32, UInt32, System.Threading.CancellationToken)
0f3bf1d0 70ad75f0 System.Threading.SemaphoreSlim.Wait(Int32, System.Threading.CancellationToken)
0f3bf230 703dc110 System.Collections.Concurrent.BlockingCollection`1[[System.__Canon, mscorlib]].TryAddWithNoTimeValidation(System.__Canon, Int32, System.Threading.CancellationToken)
I tried to find more information about locks. !locks command gave me only this Scanned 40 critical sections. !SyncBlock shows this:
Index SyncBlock MonitorHeld Recursion Owning Thread Info SyncBlock Owner
-----------------------------
Total 1364
CCW 1
RCW 1
ComClassFactory 0
Free 713
What isGCFrame? What is CCW and RCW? I can't find it in documentation.
Also is it a way to find the thread which owns the lock?

The process was terminated due to an unhandled exception

In the code for my windows service I have a try catch block.
When I run a unit test on my development machine on a the database , the unit test passes.
When the service runs on the production server with the same database, an error occurs and two events appear in the Windows Event log.
EventType clr20r3, P1 MKG.XPREFLIGHT.servicehost.exe, P2 2.0.62.0,
P3 574a1457, P4 mscorlib, P5 4.0.0.0, P6 53b50a71, P7 3e5, P8 10d,
P9 system.formatexception, P10 NIL.
and
Event Type: Error
Event Source: .NET Runtime
Event Category: None
Event ID: 1026
Date: 29/05/2016
Time: 8:03:54 AM
User: N/A
Computer: SERVERX1
Description:
Application: MKG.XPREFLIGHT.ServiceHost.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.Reflection.TargetInvocationException
Stack:
at System.RuntimeMethodHandle._InvokeMethodFast(System.IRuntimeMethodInfo, System.Object, System.Object[], System.SignatureStruct ByRef, System.Reflection.MethodAttributes, System.RuntimeType)
at System.RuntimeMethodHandle.InvokeMethodFast(System.IRuntimeMethodInfo, System.Object, System.Object[], System.Signature, System.Reflection.MethodAttributes, System.RuntimeType)
at System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo, Boolean)
at System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)
at System.Reflection.MethodBase.Invoke(System.Object, System.Object[])
at MKG.XPREFLIGHT.Engine.FlightEngine.ParseCommand(MKG.ServiceData.EngineCommandQueue)
at MKG.XPREFLIGHT.Engine.FlightEngine.TakeCommands()
at MKG.XPREFLIGHT.Engine.FlightEngine.PollingProc(System.Object)
at System.Threading._TimerCallback.TimerCallback_Context(System.Object)
at System.Threading.ExecutionContext.runTryCode(System.Object)
at System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode, CleanupCode, System.Object)
at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading._TimerCallback.PerformTimerCallback(System.Object)
I isolated the area of code that the error occurs in, but since the unit test passes I cannot see why the error occurs.
The error is repeatable although the fault does not occur in the same place the first time the process runs after a server reboot.
The code is within a Try Catch block, however it does not get caught. Instead the service stops with errors in the windows event log.
How can I troubleshoot this?
[update]
the code that fails is
public Job GetDeepJobWithUploadInfo(int jobId)
{
// if I throw an error here it is caught
Job job = null;
using (var connect = MakeConnect())
{
job =
connect.job.AsNoTracking()
.Include(
j =>
j.VivUpload.Select(
u =>
u.VivSplitUpload.Select(
s =>
s.VivSplitProfile.Select(
p => p.VivSplitLog.Select(l => l.VivSplitLogType)))))
.Include(j => j.VivUpload.Select(u => u.VivSplitUpload.Select(s => s.VivSplitProfile)))
.Include(j => j.JobTag)
.SingleOrDefault(j => j.JobID == jobId);
}
// never gets to here in production, but does get here in unit test
return job;
}
I have tried running the custom tool on the edmx to regenerate the Entity Framework classes.

Get object details from clrstack output in windbg

I have a multi-threaded .NET application that hangs on an OnUserPreferenceChanged event. This is typically caused by a UI control or message loop started on a background thread (see e.g. http://www.ikriv.com/en/prog/info/dotnet/MysteriousHang.html), but as far as I can tell that isn't the case here. I verified this by setting a breakpoint in the WindowsFormsSynchronizationContext (as suggested here http://www.aaronlerch.com/blog/2008/12/15/debugging-ui/) and it is only constructed once, in the main UI thread.
Here's the output from !clrstack in windbg:
0013eea8 7c90e514 [HelperMethodFrame_1OBJ: 0013eea8]
System.Threading.WaitHandle.WaitOneNative(Microsoft.Win32.SafeHandles.SafeWaitHandle,
UInt32, Boolean, Boolean) 0013ef54 792b68af
System.Threading.WaitHandle.WaitOne(Int64, Boolean) 0013ef70 792b6865
System.Threading.WaitHandle.WaitOne(Int32, Boolean) 0013ef84 7b6f1a4f
System.Windows.Forms.Control.WaitForWaitHandle(System.Threading.WaitHandle)
0013ef98 7ba2d68b
System.Windows.Forms.Control.MarshaledInvoke(System.Windows.Forms.Control,
System.Delegate, System.Object[], Boolean) 0013f038 7b6f33ac
System.Windows.Forms.Control.Invoke(System.Delegate, System.Object[])
0013f06c 7b920bd7
System.Windows.Forms.WindowsFormsSynchronizationContext.Send(System.Threading.SendOrPostCallback,
System.Object) 0013f084 7a92ed62
Microsoft.Win32.SystemEvents+SystemEventInvokeInfo.Invoke(Boolean,
System.Object[]) 0013f0b8 7a92dc8f
Microsoft.Win32.SystemEvents.RaiseEvent(Boolean, System.Object,
System.Object[]) 0013f104 7a92e227
Microsoft.Win32.SystemEvents.OnUserPreferenceChanged(Int32, IntPtr,
IntPtr) 0013f124 7aaa06ec
Microsoft.Win32.SystemEvents.WindowProc(IntPtr, Int32, IntPtr, IntPtr)
The last method I can get param info on is:
0013f084 7a92ed62
Microsoft.Win32.SystemEvents+SystemEventInvokeInfo.Invoke(Boolean,
System.Object[])
PARAMETERS:
this = 0x01404420
checkFinalization = 0x00000001
args = 0x0144a298
Here's my question: How can I get more information here? Ultimately, I'd like to know which objects and/or threads this Invoke is for. Something like "!do 0x01404420" or "!do 0x0144a298" but I don't know where to go from there.

Search for exceptions in the heap by using !dumpheap -type Exception.
Also you can see the value of variables in a class,which will be useful to understand the state of the class. Use !dumpheap -type ClassName. You will get a MT(Method Table) address. From MT address see the Object address. Use !do address to dump the class.
Use !syncblk to see the locked threads

Regarding problems caused by SystemEvents class and in particular OnUserPreferenceChanged event try to use the CheckSystemEventsHandlersForFreeze() function in this answer which can help to find root cause, i.e. which controls was created on wrong thread and thus causing freeze.

How to deal with this C# hang invloving SystemEvents.OnUserPreferenceChanged

My WinForm application having a hang problem. What happen is that client sometime leave the application running overnight and when they comeback in morning application is typically in a hang state. This is what I see in dump file on the main thread. What I don't understand is what could make SystemEvents.OnUserPreferenceChanged event to be invoked, although I don't think I am doing anything that is invoking this event.
0024e480 770496f4 System.Threading.WaitHandle.WaitOneNative(Microsoft.Win32.SafeHandles.SafeWaitHandle, UInt32, Boolean, Boolean)
0024e52c 702c68af System.Threading.WaitHandle.WaitOne(Int64, Boolean)
0024e548 702c6865 System.Threading.WaitHandle.WaitOne(Int32, Boolean)
0024e55c 6e891a6f System.Windows.Forms.Control.WaitForWaitHandle(System.Threading.WaitHandle)
0024e570 6ebcd6eb System.Windows.Forms.Control.MarshaledInvoke(System.Windows.Forms.Control, System.Delegate, System.Object[], Boolean)
0024e610 6e8933cc System.Windows.Forms.Control.Invoke(System.Delegate, System.Object[])
0024e644 6eac0c83 System.Windows.Forms.WindowsFormsSynchronizationContext.Send(System.Threading.SendOrPostCallback, System.Object)
0024e65c 6fe1eed2 Microsoft.Win32.SystemEvents+SystemEventInvokeInfo.Invoke(Boolean, System.Object[])
0024e690 6fe1d07f Microsoft.Win32.SystemEvents.RaiseEvent(Boolean, System.Object, System.Object[])
0024e6dc 6fe1e38f Microsoft.Win32.SystemEvents.OnUserPreferenceChanged(Int32, IntPtr, IntPtr)
0024e6fc 6fa64c29 Microsoft.Win32.SystemEvents.WindowProc(IntPtr, Int32, IntPtr, IntPtr)
0024e700 000a1104 [InlinedCallFrame: 0024e700]
0024e8d8 6e378d5e System.Windows.Forms.Application+ComponentManager.System.Windows.Forms.UnsafeNativeMethods.IMsoComponentManager.FPushMessageLoop(Int32, Int32, Int32)
0024e974 6e3789c7 System.Windows.Forms.Application+ThreadContext.RunMessageLoopInner(Int32, System.Windows.Forms.ApplicationContext)
0024e9c8 6e378811 System.Windows.Forms.Application+ThreadContext.RunMessageLoop(Int32, System.Windows.Forms.ApplicationContext)
0024e9f8 6e88de47 System.Windows.Forms.Application.RunDialog(System.Windows.Forms.Form)
0024ea0c 6e8c25cb System.Windows.Forms.Form.ShowDialog(System.Windows.Forms.IWin32Window)
0024ea98 6e8c27e3 System.Windows.Forms.Form.ShowDialog()
0024ea9c 56c26e76 MyNameSpace.MyForm.MyMethod2(Object, Boolean, Boolean, System.Guid, Boolean)
0024eba0 56c26c47 MyNameSpace.MyForm.MyMethod1(System.Guid, System.Guid, System.Guid, Boolean)
0024ecf8 56c91f4c MyNameSpace.MyForm.MyButton_Click(System.Object, System.EventArgs)
0024ee88 6e334180 System.Windows.Forms.Control.OnClick(System.EventArgs)

Controls subscribe this event so that they will redraw themselves when the user changed the theme or system colors. This event also gets fired when you're not close to the machine and Windows automatically locks the workstation. Which explains the morning-after hangover.
The deadlock is caused by a threading problem, the SystemEvents class fires the event on the wrong thread. Which is caused by an initialization problem in your program. The typical trigger is not creating the first window on the main thread, that confuzzles SystemEvents. It tries to fire an event on that same thread again but it isn't around anymore. Or it copied SynchronizationContext.Current before it got initialized by Winforms. Either way, the event will fire on a threadpool thread instead of the main UI thread. That's lethal.
Common when you implement your own splash screen for example. Use the built-in support instead.

Need help deciphering a C# stack trace

I have an application that is locking on the GUI thread, and I've used WinDbg, along with the "!clrstack" command to get this stack trace, but I can't figure out where the issue is. All of these methods look like framework methods, and none are mine. Any help would be much appreciated. I apologize for the long lines
OS Thread Id: 0x724 (0)
ESP EIP
0012ec88 7c90e4f4 [HelperMethodFrame_1OBJ: 0012ec88] System.Threading.WaitHandle.WaitOneNative(Microsoft.Win32.SafeHandles.SafeWaitHandle, UInt32, Boolean, Boolean)
0012ed34 792b687f System.Threading.WaitHandle.WaitOne(Int64, Boolean)
0012ed50 792b6835 System.Threading.WaitHandle.WaitOne(Int32, Boolean)
0012ed64 7b6f192f System.Windows.Forms.Control.WaitForWaitHandle(System.Threading.WaitHandle)
0012ed78 7ba2d0bb System.Windows.Forms.Control.MarshaledInvoke(System.Windows.Forms.Control, System.Delegate, System.Object[], Boolean)
0012ee18 7b6f328c System.Windows.Forms.Control.Invoke(System.Delegate, System.Object[])
0012ee4c 7b920717 System.Windows.Forms.WindowsFormsSynchronizationContext.Send(System.Threading.SendOrPostCallback, System.Object)
0012ee64 7a924102 Microsoft.Win32.SystemEvents+SystemEventInvokeInfo.Invoke(Boolean, System.Object[])
0012ee98 7a922833 Microsoft.Win32.SystemEvents.RaiseEvent(Boolean, System.Object, System.Object[])
0012eee4 7a923d2f Microsoft.Win32.SystemEvents.OnUserPreferenceChanged(Int32, IntPtr, IntPtr)
0012ef04 7aa8f184 Microsoft.Win32.SystemEvents.WindowProc(IntPtr, Int32, IntPtr, IntPtr)
0012ef08 003620a4 [InlinedCallFrame: 0012ef08]
0012f0cc 7b1d8cce System.Windows.Forms.Application+ComponentManager.System.Windows.Forms.UnsafeNativeMethods.IMsoComponentManager.FPushMessageLoop(Int32, Int32, Int32)
0012f168 7b1d8937 System.Windows.Forms.Application+ThreadContext.RunMessageLoopInner(Int32, System.Windows.Forms.ApplicationContext)
0012f1bc 7b1d8781 System.Windows.Forms.Application+ThreadContext.RunMessageLoop(Int32, System.Windows.Forms.ApplicationContext)
0012f1ec 7b195911 System.Windows.Forms.Application.Run(System.Windows.Forms.Form)
0012f200 00eb0ebb WinForms1.Program.Main()
0012f69c 79e71b4c [GCFrame: 0012f69c]

It looks like this code isn't the true cause of the problem. See if one of these pages helps:
Mysterious Hang or The Great Deception of InvokeRequired
OnUserPreferenceChanged Hang
Update: Fixed url to first page.

The other answer comes from this Aaron Lerch. I really like the "filtered breakpoint on someone else's code" part. I think it would've saved me a couple of days.
http://www.aaronlerch.com/blog/2008/12/15/debugging-ui/

I had similar issue, and found the culprit in our code.
Technique used:
0. Make sure in Spy++ that there are only two unmanaged threads with controls on them (WinForms and GDI+):
public static class ThreadingHelper_NativeMethods
{
[DllImport("user32.dll")]
public static extern bool IsGUIThread(bool bConvert);
}
call this from UI thread upon init:
// This code forces initialization of .NET BroadcastEventWindow to the UI thread.
// http://social.msdn.microsoft.com/Forums/en-US/netfxbcl/thread/fb267827-1765-4bd9-ae2f-0abbd5a2ae22
if (ThreadingHelper_NativeMethods.IsGUIThread(false))
{
Microsoft.Win32.SystemEvents.InvokeOnEventsThread(new MethodInvoker(delegate()
{
int x = 0;
}));
}
Remember managed id of Ui Thread in a singleton class.
Search for all UserControls defined in our code. In each control's constructor, before calling InitializeComponent(), i placed code that checks current thread id against the main thread id. If they aren't equal, Assert(false).
Subscribe to SystemEvents.UserPreferencesChanging. Debug.Assert(false) in handler: this happens before SystemEvents.UserPreferencesChanged, so debugger will hopefully pause here.
Inspect list of subscribers to SystemEvents in the debugger. Find a subscriber in _handles dictionary of lists. Opening up SynchronizationContext of the each callback should reveal the problem: same thread id as for the control created on non-UI thread. SystemEvents will will execute event handler on that thread, deadlocking against UI thread.

The function CheckSystemEventsHandlersForFreeze() in this answer can help you to find root cause of why your GUI thread becomes locked.