SQL Server Crash Dump Analysis A brief tour with WinDbg and other ugly tools Pablo Álvarez Doval Debugging & Optimization Team Lead

Who am I?

Session Objectives  What is this session about?  What isn’t this session about?

Who are you?

Agenda  Tools of the Trade  Brief Windows Architecture Refresher  SQL Server Post-mortem Debugging  Handling SQL Server dumps  Analyzing SQL Server dumps  Debugging.NET Applications with SOS

Debugging Tools for Windows  Free download:   Updated several times a year  Debuggers, extensions, tools and a great help file:  windbg.exe, kd.exe, cdb.exe  gflags.exe, tlist.exe, etc  debugger.chm  Can be installed via xcopy

Demo 0: … is it really so ugly?

Thesaurus  Just to keep with the forensics analogy:  Corpse  Dump file  Forensic Lab  WinDbg  Forensic Scientist  You!  Gray’s Anathomy  Windows Internals 5 th Ed.  We are not going to get into details, but we will do a little refresher of some key concepts

User mode vs. Kernel mode User Mode Kernel Mode Hardware Abstraction Layer (HAL) Device Drivers Microkernel Graphics Controller Object Manager Executive Services FS I/OIPC Memory Processes Security WMPNP UNIX LSA Shell Lsass.exe Client/Server csrss.exe Notepad notepad.exe Windows on Windows wowexec.exe Virtual DOS Machine ntvdm.exe Win32Interix

Application, Processes and Threads  An application is formed by one or more processes  A process is an in-memory executable, which is made up of one or more threads and its resources  A thread is the basic unit of execution and scheduling in the OS.

… is it really worth it?

Other good reasons…

Win32 Virtual Memory Addressing (I) KernelKernel Process 1 Thread 1 Thread 2 Thread n : Process 2 Thread 1 Thread 2 Thread n : sqlsrv.exe Thread 1 Thread 2 Thread n : Process n Thread 1 Thread 2 Thread n : … 4 Gb 2 Gb

Win32 Virtual Memory Addressing(II)

Thread Call Stacks  Shows part of the history of the function calls of the thread  Each thread has its own Call Stack  i.e: ntdll!KiFastSystemCallRet USER32!NtUserGetMessage+0xc notepad!WinMain+0xe5 notepad!WinMainCRTStartup+0x174 kernel32!BaseProcessStart+0x23

Call Stacks (I)  Each thread of the process has its own call stack:

Call Stacks (II)  Each frame has the following structure: Frame Parameters Return Address Frame Pointer Exception Handler Local Variables Registros

Symbols  Symbols make the call stack useful:  Without Symbols:  With Symbols: kernel32!+136aa kernel32!CreateFileW+0x35f

Symbol formats  Current format:.PDB  Old Format:.DBG  Retail vs. Debug (Free vs. Checked) builds  Private symbols vs. public symbols

Symbol Servers  Uses the File System as a Symbol’s database:  Organized by name and a unique identifier  Folder structure: \\SymSrv\file_name.pdb\unique_number\____  i.e: \\Symbols\ntdll.pdb\3B5EDCA52\ntdll.pdb \\Symbols\ntdll.pdb\380FCC4F2\ntdll.pdb

Demo 1: Scheduler Non-Yielding

Scenario  A customer’s SQL Server 2000 is hanging, showing errors in SQL Server’s ErrorLog  When these errores ocurr, SQL Server automatically triggers the creation of a dump … :17:14.10 server Error: 17883, Severity: 1, State: :17:14.10 server Process 59:0 (834) UMS Context 0x125ABD80 appears to be non-yielding on Scheduler 1. … :17:14.10 server Error: 17883, Severity: 1, State: :17:14.10 server Process 59:0 (834) UMS Context 0x125ABD80 appears to be non-yielding on Scheduler 1. …

Demo 2: DBCC CHECKDB

Demo 3: Cluster Resources

Managed Debugging with.NET  WinDbg is a native debugger  In order to debug.NET code we need to use debugger extensions:  SOS.dll (until framework.NET 3.5)  CLR.dll (framework 4.0)  Why all this? Is it worth it?

Demo 4: Managed Debugging with SOS

Some cool tips…  Did we really get to this slide in time?!  Well.. enjoy some free tips!  Using SOS from VS.NET  Memory dump analysis from inside VS2010

Resources  Concepts      Books:  Microsoft Windows Internals, 5th Ed. [Mark E. Russinovich and David A. Solomon] Microsoft Press.  Debugging Applications for Microsoft.NET and Microsoft Windows [John Robbins] Microsoft Press.

Any Questions? Thanks!