. NET Debugging for the Production Environment Part 4: Common Debugging Commands Brad Linscott Premier Field Engineeringing

Agenda Symbols Commands Common debugging commands: miscellaneous Common debugging commands for hangs Common debugging commands for exceptions Common debugging commands for memory pressure

3 Not needed to resolve managed call stacks Needed to resolve native call stacks Translation: Not always mandatory, but in general, a good idea to ensure symbols are lined up for managed issues File, Symbol File Path;.sympath;.symfix Symbols

4 Commands Meta commands (.) Extension commands(!) Breakdown of Debugger Commands

5 | - pipe command displays command line of process 0:029> |. 0id: 5ecexaminename: c:\WINDOWS\system32\inetsrv\w3wp.exe vertarget shows dump time, OS version, process’ lifetime, and more 0:029> vertarget Windows Server 2003 Version 3790 (Service Pack 2) MP (2 procs) Free x86 compatible Product: Server, suite: TerminalServer SingleUserTS kernel32.dll version: (srv03_sp2_gdr ) Machine Name: Debug session time: Tue Jul 29 10:51: (GMT-5) System Uptime: 0 days 6:19: Process Uptime: 0 days 4:34: Kernel time: 0 days 0:24: User time: 0 days 0:05: Common Miscellaneous commands

6.load loads a debugging extension.unload unloads a debugging extension.chain shows loaded extensions, most recently-loaded is listed first. 0:000>.chain Extension DLL search Path: C:\debuggersx86\WINXP;C:\debuggersx86\winext;C:\debuggersx86\winext\arcade psscor2.dll: image , API 1.0.0, built Wed Oct 29 12:48: [path: C:\debuggersx86\psscor2.dll] dbghelp: image , API 6.2.6, built Tue May 17 18:15: [path: C:\debuggersx86\dbghelp.dll] ext: image , API 1.0.0, built Tue May 17 18:16: [path: C:\debuggersx86\winext\ext.dll] exts: image , API 1.0.0, built Tue May 17 18:16: [path: C:\debuggersx86\WINXP\exts.dll] uext: image , API 1.0.0, built Tue May 17 18:16: [path: C:\debuggersx86\winext\uext.dll] Common Miscellaneous commands (cont’d)

7 !help displays help for top-most extension !psscorX.help displays detailed help for.hh opens debugger.chm to the topic. can’t be from an extension outside the Debugging Windows for Tools package !peb shows loaded modules, environment variables, command line arg, and more Common Miscellaneous commands (cont’d)

8 !runaway shows quanta of each thread 0:029> !runaway User Mode Time Thread Time 17:828 0 days 0:04: :380 0 days 0:04: :288 0 days 0:04: :4a0 0 days 0:03: :13e4 0 days 0:01: ~ displays PID and TID; ~#s switches the active thread to thread # 0:000> ~. 0 Id: 2c Suspend: 0 Teb: 7ffde000 Unfrozen 1 Id: 2c14.35b8 Suspend: 0 Teb: 7ffdd000 Unfrozen 2 Id: 2c14.32f8 Suspend: 0 Teb: 7ffdc000 Unfrozen 3 Id: 2c Suspend: 0 Teb: 7ffd9000 Unfrozen Red – logical Thread IDBlue – Hex Thread ID Purple – Process IDGreen – beginning address of Thread Environment Block Common Miscellaneous commands (cont’d)

9 k, kn, kb, kP, etc displays the native stack in various ways 0:006> kbnL # ChildEBP RetAddr Args to Child fcec 7c827b89 7c83c5fe c 0114fd34 ntdll!KiFastSystemCallRet fcf0 7c83c5fe c 0114fd ntdll!ZwWaitForMultipleObjects+0xc ffb8 77e6482f ntdll!RtlpWaitThread+0x ffec c83c4b kernel32!BaseThreadStart+0x34 !clrstack is from psscorX; displays managed call stack 0:055> !clrstack OS Thread Id: 0x1ea0 (55) ESP EIP 386fe4a4 7c82847c [HelperMethodFrame_1OBJ: 386fe4a4] System.Threading.WaitHandle.WaitMultiple(System.Threading.WaitHandle[], Int32, Boolean, Boolean) 386fe b3ff System.Threading.WaitHandle.WaitAny(System.Threading.WaitHandle[], Int32, Boolean) 386fe dd8cb System.Net.TimerThread.ThreadProc() 386fe5dc 792d6e46 System.Threading.ThreadHelper.ThreadStart_Context(System.Object) 386fe5e8 792f5781 System.Threading.ExecutionContext.runTryCode(System.Object) Common Miscellaneous commands (cont’d)

10 ~*k; ~*e!clrstack displays all the stacks !syncblk finds outstanding syncblocks on which other threads are waiting ||0:0:000> !syncblk Index SyncBlock MonitorHeld Recursion Thread ThreadID Object Waiting x06c7b13c x101dc0 0x120c 14 0x1c2643f8 System.String x06c7b13c x101dc0 0x120c 14 0x1c2643f8 System.String Waiting threads: x02b382a x6d5a4b8 0x x140159ec System.String x02b382a x6d5a4b8 0x x140159ec System.String Waiting threads: Total 3942 ComCallWrapper 7 ComPlusWrapper 3 ComClassFactory 0 Free 234 Common commands for hangs

11 !finalizequeue from psscorX lists objects registered for finalization. 0:000> !finalizequeue SyncBlocks to be cleaned up: 524 MTA Interfaces to be released: 0 STA Interfaces to be released: generation 0 has 577 finalizable objects (0e >0e481244) generation 1 has 41 finalizable objects (0e48089c->0e480940) generation 2 has 543 finalizable objects (0e >0e48089c) Ready for finalization objects (0e >0e5150d0)  A number > 0 indicates possible problems. A number this big indicates a definite problem. Statistics: If this is the case, investigate the Finalizer thread (!threads is one way to find it) to determine cause of blockage. Common commands for hangs, cont’d

12 !dae from psscorX displays all managed exceptions in the GC heaps Double-check single OutOfMemoryException, StackOverflowException, and ExecutionEngineException instances !pe from psscorX provides more detailed information on a specific managed exception. Common commands for exceptions

13 To a large extent, debugging crash dumps of a managed & native app don’t differ by much Upon opening dump, active thread is faulting thread.. !u for deeper analysis of the disassembly Native debugging skills as well commonly needed to determine root cause Common commands for exceptions, cont’d

14 !address shows information about a particular address (type, protection, state, usage) 0:000> !address -summary Usage SUMMARY TotSize ( KB) Pct(Tots) Pct(Busy) Usage 2ec53000 ( ) : 36.54% 72.26% : RegionUsageIsVAD 3f ( ) : 49.43% 00.00% : RegionUsageFree 83fb000 ( ) : 06.44% 12.74% : RegionUsageImage Tot: 7fff0000 ( KB) Busy: 40b9f000 ( KB) Type SUMMARY TotSize ( KB) Pct(Tots) Usage 3f ( ) : 49.43% : 8cb5000 ( ) : 06.87% : MEM_IMAGE 1aa64000 ( ) : 20.82% : MEM_MAPPED 1d ( ) : 22.88% : MEM_PRIVATE State SUMMARY TotSize ( KB) Pct(Tots) Usage ( ) : 39.36% : MEM_COMMIT 3f ( ) : 49.43% : MEM_FREE e ( ) : 11.20% : MEM_RESERVE Largest free region: Base 6e0e Size (58816 KB)  1GB free and your largest contiguous FREE is 58MB is a good sign of VA fragmentation Common commands for memory pressure

15 A managed leak/bloat is usually in one of 2 places: GC Heaps (!eeheap –gc) Loader heaps (!eeheap –loader) Common commands for memory pressure

16 Loader heaps: Contain type definitions and other internal structures ~20MB is usually considered high for a busy w3wp If !dda results in too many assemblies (generally, more than a few hundred means this is contributor to high mem), find why they’re being created !dumpdomain lists assemblies loaded in each appdomain In w3wp, !finddebugtrue tells you if, which is a root cause of high memory Q886385, Q are common root causes Common commands for memory pressure, cont’d

17 !eeheap –gc Breaks down mem usage among Gens within GC heaps. Helps you to see if memory issue is concentrated in a specific generation or the LOH. !dumpheap –stat may help. Focus on objects with high Count and/or high Total Memory. !gcroot may help find root cause !clrusage may also give helpful insight ||0:0:000> !help clrusage !CLRUsage [-v] !CLRUsage dumps out the memory that.NET is using. This is different from !EEHeap -gc in that it gives how much memory is committed and reserved and also gives the initial allocation that we made. Common commands for memory pressure, cont’d

18 Many commands are used regularly, this session has covered just a few Be curious – use !help frequently Practice, Practice, Practice Conclusion