1. SQL Server Crash Dump Analysis A brief tour with WinDbg and other ugly tools Pablo Álvarez Doval Debugging & Optimization Team Lead pablod@plainconcepts.com

2. Who am I?

5. Session Objectives  What is this session about?  What isn’t this session about?

7. Who are you?

8. Agenda  Tools of the Trade  Brief Windows Architecture Refresher  SQL Server Post-mortem Debugging  Handling SQL Server dumps  Analyzing SQL Server dumps  Debugging.NET Applications with SOS

9. Debugging Tools for Windows  Free download:  http://www.microsoft.com/whdc/devtools/debugging http://www.microsoft.com/whdc/devtools/debugging  Updated several times a year  Debuggers, extensions, tools and a great help file:  windbg.exe, kd.exe, cdb.exe  gflags.exe, tlist.exe, etc  debugger.chm  Can be installed via xcopy

10. Demo 0: … is it really so ugly?

11. Thesaurus  Just to keep with the forensics analogy:  Corpse  Dump file  Forensic Lab  WinDbg  Forensic Scientist  You!  Gray’s Anathomy  Windows Internals 5 th Ed.  We are not going to get into details, but we will do a little refresher of some key concepts

12. User mode vs. Kernel mode User Mode Kernel Mode Hardware Abstraction Layer (HAL) Device Drivers Microkernel Graphics Controller Object Manager Executive Services FS I/OIPC Memory Processes Security WMPNP UNIX LSA Shell Lsass.exe Client/Server csrss.exe Notepad notepad.exe Windows on Windows wowexec.exe Virtual DOS Machine ntvdm.exe Win32Interix

13. Application, Processes and Threads  An application is formed by one or more processes  A process is an in-memory executable, which is made up of one or more threads and its resources  A thread is the basic unit of execution and scheduling in the OS.

14. … is it really worth it?

16. Other good reasons…

17. Win32 Virtual Memory Addressing (I) KernelKernel Process 1 Thread 1 Thread 2 Thread n : Process 2 Thread 1 Thread 2 Thread n : sqlsrv.exe Thread 1 Thread 2 Thread n : Process n Thread 1 Thread 2 Thread n : … 4 Gb 2 Gb

18. Win32 Virtual Memory Addressing(II)

19. Thread Call Stacks  Shows part of the history of the function calls of the thread  Each thread has its own Call Stack  i.e: ntdll!KiFastSystemCallRet USER32!NtUserGetMessage+0xc notepad!WinMain+0xe5 notepad!WinMainCRTStartup+0x174 kernel32!BaseProcessStart+0x23

20. Call Stacks (I)  Each thread of the process has its own call stack:

21. Call Stacks (II)  Each frame has the following structure: Frame Parameters Return Address Frame Pointer Exception Handler Local Variables Registros

22. Symbols  Symbols make the call stack useful:  Without Symbols:  With Symbols: kernel32!+136aa kernel32!CreateFileW+0x35f

23. Symbol formats  Current format:.PDB  Old Format:.DBG  Retail vs. Debug (Free vs. Checked) builds  Private symbols vs. public symbols

24. Symbol Servers  Uses the File System as a Symbol’s database:  Organized by name and a unique identifier  Folder structure: \\SymSrv\file_name.pdb\unique_number\____  i.e: \\Symbols\ntdll.pdb\3B5EDCA52\ntdll.pdb \\Symbols\ntdll.pdb\380FCC4F2\ntdll.pdb

25. Demo 1: Scheduler Non-Yielding

26. Scenario  A customer’s SQL Server 2000 is hanging, showing 17883 errors in SQL Server’s ErrorLog  When these errores ocurr, SQL Server automatically triggers the creation of a dump … 2007-02-12 11:17:14.10 server Error: 17883, Severity: 1, State: 0 2007-02-12 11:17:14.10 server Process 59:0 (834) UMS Context 0x125ABD80 appears to be non-yielding on Scheduler 1. … 2007-02-12 11:17:14.10 server Error: 17883, Severity: 1, State: 0 2007-02-12 11:17:14.10 server Process 59:0 (834) UMS Context 0x125ABD80 appears to be non-yielding on Scheduler 1. …

27. Demo 2: DBCC CHECKDB

28. Demo 3: Cluster Resources

29. Managed Debugging with.NET  WinDbg is a native debugger  In order to debug.NET code we need to use debugger extensions:  SOS.dll (until framework.NET 3.5)  CLR.dll (framework 4.0)  Why all this? Is it worth it?

30. Demo 4: Managed Debugging with SOS

31. Some cool tips…  Did we really get to this slide in time?!  Well.. enjoy some free tips!  Using SOS from VS.NET  Memory dump analysis from inside VS2010

32. Resources  pablod@plainconcepts.com  @Plain Concepts  http://www.geeks.ms/blogs/palvarez http://www.geeks.ms/blogs/palvarez  http://www.geeks.ms/blogs/rcorral http://www.geeks.ms/blogs/rcorral  http://www.geeks.ms/blogs/luisguerrero http://www.geeks.ms/blogs/luisguerrero  @MSDN:  http://blogs.msdn.com/tess/ http://blogs.msdn.com/tess/  Books:  Microsoft Windows Internals, 5th Ed. [Mark E. Russinovich and David A. Solomon] Microsoft Press.  Debugging Applications for Microsoft.NET and Microsoft Windows [John Robbins] Microsoft Press.

33. Any Questions? Thanks!