End-to-End Troubleshooting CHECKLIST for Microsoft SQL Server Kevin Kline Technical Evangelist, SQL Sentry SQL Server MVP since 2003 Twitter, FB, LI, KEKline Blog: http://blogs.sqlsentry.com/KevinKline

FREEBIES! Articles: Videos: Downloads: Services: Community: SQL Performance: www.sqlperformance.com Team Blogs: blogs.sentryone.com Articles: Educational and Product Videos at sqlsentry.tv Videos: Trial Downloads at www.sentryone.com/trial Free Plan Explorer at www.sentryone.com/Plan-Explorer Downloads: Free query tuning advice at Answers.SQLPerformance.com Services: Twitter: @SQLSentry | Facebook: Facebook.com/SQLSentry | LinkedIn: SQL Sentry, LLC Community: Or email us at SalesEngineering@sqlsentry.com

Tuning blog: www.sqlperformance.com/ Normally $10. Get them free at SentryOne.com/ebooks.

Book a one-on-one demonstration at www.sentryone.com/BookADemo! FREE!

Agenda Methodology for troubleshooting Troubleshooting tools and techniques using the native SQL Server tool kit: Wait Stats Windows Performance Monitor (PerfMon) SQL Profiler, Server-Side Traces, and XEvents SQL Server DMVs Execution Plans [X] Summary, Resources, and Q&A © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Where to Begin? 9/11/2018 4:46 AM Error Logs PerfMon DMVs Profiler & Trace SSMS (Execution Plan) There’s not a “right” or “wrong” place to start. You can start at any of the points shown above. Start with the information source that provides the actionable information most quickly © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Identification & Resolution Methodology Errors in the log Effective troubleshooting is like a funnel or series of continuously more refined sieves. Each successive sieve filters out smaller “chunks”; that is, harder and more transient errors and problems More work is required… Resource issues Specific SQL Cmds Identification & Resolution

Troubleshooting checklist Shortcut! Has anything changed? Inside or outside of SQL Server? Is the issue caused by a SQL Server error? Are there excessive wait stats? Correlate wait stats against other metrics. Follow-up 1. 2. 3. 4. Has anything changed? Inside or outside of SQL Server Check Windows Server error logs. Resolve errors that correlate to the issue. Hand off non-SQL Server errors to appropriate teams. Is the issue caused by a SQL Server error? Check SQL Server and SQL Agent logs. Resolve any errors that correlate to the issue. Identify if other SQL Server components are having problems. Are there excessive wait stats? Assess wait statistics to categorize the bottleneck. Answer these questions: "Has anything changed recently?" and "Is the current behavior consistent with the baseline?" Cross-check wait stats and PerfMon queues against specific user sessions or processes. Tune problematic user transactions, jobs or processes. Assess/correlate other performance metrics and system config settings. Follow-up Document the resolution. Update alerting systems to either catch or ignore the issue, as appropriate, in the future. 5. 6.

Your best shortcut Remember: Change = Risk Your best shortcut? Know what has changed! Sp_Configure or sys.configurations Sp_Dboption (pre-SQL2012) or sys.databases (SQL2012 +) DDL triggers for meta-data changes: Developers? Unfettered access to databases?

Error Logs Windows Application helps eliminate non-SQL Server problems 9/11/2018 4:46 AM Windows Application helps eliminate non-SQL Server problems SQL Server Error Log and SQL Server Agent Log Available both as TXT and through the GUI SQL Server keeps the six most recent, incrementing with each reboot WARNING! Always make sure to enable SQL Server Agent notifications for severity level 18 or greater! © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Acting Upon Error Logs No further research required 9/11/2018 4:46 AM No further research required Error found with easy fix Error found with difficult fix or restore required Further research required Error found, but time or symptoms of error do not correlate to the problem No error found Error 1101 “Could not allocate a new page for database '%.*ls' because of insufficient disk space in filegroup '%.*ls'.” Error 605 “This error may be raised for any table if the allocation unit ID of the page does not match the expected allocation unit ID. In general, this signifies page or allocation corruption.” Error 5243 “SQL Server detected a structural inconsistency in an in-memory storage engine structure.” © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Demo Error Notification 9/11/2018 4:46 AM © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Rocks, Gravel, or Sand We retrieved the top level information, “the big rocks” Now, what’s the best way to go deeper?

Wait Stats: It’s all about Bottlenecks! Anytime a task in SQL Server waits for something: It is reported as a wait type Reveals where the bottlenecks are SQL Server 2005 aggregates wait type information SQL Server 2008 provides new pre-emptive wait stats Post 2008, new wait stats support new features

Demo Wait Statistics 9/11/2018 4:46 AM © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Schedulers & Wait Stats 1 Window = 1 Scheduler Users are assigned to a thread No problem. Step aside… More syrup for the sodas! Uh oh! The out of soda! Yeah! I’m next in line! Goes to the waiting or “suspended” queue

Waits by Task sys.dm_os_waiting_tasks Wait information Task level Very accurate Transient data

Top 10 Waits from the Field CPU PRESSURE CPU pressure: SOS_SCHEDULER_YIELD Parallelism: CXPACKET LOCKING Long term blocking: LCK_X, LCK_M_U, & LCK_M_X MEMORY Data Cache latch: PAGELATCH_X Non-data cache latch: LATCH_X Query memory grants: RESOURCE_SEMAPHORE I/O Buffer I/O latch: PAGEIOLATCH_X Tran log disk subsystem: WRITELOG & LOGBUFFER General I/O issues: ASYNC_IO_COMPLETION & IO_COMPLETION NETWORK PRESSURE Network I/O: ASYNC_NETWORK_IO From Jimmy May: http://blogs.msdn.com/jimmymay/archive/2009/04/27/wait-stats-by-joe-sack.aspx and Joe Sack: Presentation Deck for "Performance Tuning with Wait Statistics" http://blogs.msdn.com/joesack/archive/2009/04/22/presentation-deck-for-performance-tuning-with-wait- statistics.aspx

Correlating PERF Information With wait stats, other older standbys are not as frequently needed. But they still help! PerfMon Xevents and Traces (either Profiler or Server-side) DMVs

9/11/2018 4:46 AM PerfMon Benefits: Shows the rate of resource consumption or activity in a wide variety of areas on the server, for example Disk IO; Memory; Network SQL Server activity - Locking, Blocking, and Deadlocking; Cache Activity; Object Utilization Limitations Very hard to know what to track and what values indicate good or bad performance Doesn’t offer good root-cause analysis, only resource consumption info © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Windows Performance Monitor KK> I usually give a mini-demo that emphasizes what a pain in the rear PerfMon can be. It is definitely useful, since it shows the overall resource consumption of various attributes of SQL Server, the OS, and the hardware. However, I always point out how there are hundreds of Counters in SQL Server with potentially dozens of Objects within each counter. If you don’t know which Counters and Objects to track, as well as what the values mean, then you’ll have a hard time with PerfMon. Also, always emphasize that you should your target server should not be the server you are running PerfMon from, due to high overhead, and that users must be certain to save any log files to a different server than the one they’re tracking lest they have a burdensome IO overhead. Also point out that PerfMon is the 2nd step in our methodology of troubleshooting. First, we look for an error. If we can fix it, great. If not, we need to do more troubleshooting to narrow down possible root-causes.

Profiler / XEvents Monitors SQL Server for the occurrence of events 9/11/2018 4:46 AM Monitors SQL Server for the occurrence of events When an event fires, Profiler logs the event and information about it Useful for: Finding and diagnosing slow-running code. Capturing the series of SQL statements that lead to a problem Replaying and reproducing a problem on a test machine Doesn’t offer resource consumption info, just granular details © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Server-side Traces 9/11/2018 4:46 AM Warning! Profiler can be overwhelmed by a high throughput system! Server-side traces happen entirely on the server (no client GUI) and are controlled using stored procedures Useful for “auto-start” logging and high performance scenarios TIP! Profiler can be used to create a server-side trace. That means no procedures to learn. © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Demo Correlating PerfMon and Profiler Information 9/11/2018 4:46 AM © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Demo of SQL Profiler 1: Invoke SQL Profiler 2: Choose a template, Standard is usually ok. 3: Choose your Events from the Events Selection tab. 4: Click RUN to begin the trace. 5: Click STOP to end the trace and write it to a file. 1st image: As with PerfMon, make sure the audience understands that logging the trace file to the monitored machine can greatly increase overhead. Logging is fine, but make sure that you do it to a machine other than the monitored machine, else the overhead will be onerous. You can further reduce overhead, via filtering, by monitoring less. For example, you might exclude all database except a single production database and tempdb. Or you might choose to exclude some applications and only include applications that have end-user traffic. 2nd image: make sure to always include the Start Time and End Time if you choose to select your own events from the Events Selection tab. If you don’t do this, you won’t be able to correlate PerfMon and Profiler values against one another. 3rd image: Click “RUN” to begin collecting the trace. You’ll begin to see lots of activity from the server. Click the “STOP” button to write the live trace to a file.

Correlating PerfMon and Trace Data After collecting a PerfMon and Profiler trace file, load the Profiler file using File  Open  Trace File. Choose File  Import Performance Data to load in the PerfMon trace file. Choose the PerfMon counters to show on your analysis screen. You’ll then have the overlay of PerfMon & Profiler data as shown on the right. You can click anywhere on the timeline to see what was happening at that point in time. To correlate the data of both PerfMon and Profiler, you must stop both tools from tracing and have their output files handy. Load the Profiler output file first, then File >> Import Performance Data to load the PerfMon data. PerfMon shows when resource utilization peaks, so click just before that point in time to see what SQL statements (which become highlighted in the top Profiler section) were running at that time to cause the excessive resource consumption.

Dynamic Management Views (DMV) Tell exactly what’s happening at present inside of SQL Server Multitude of DMVs, which can tell things like: What are the top 10 most CPU-intensive queries? What are the 5 biggest objects in cache? Which objects get the most IO? Which users consume the most resources? DBCC SQLPERF ('sys.dm_os_wait_stats', CLEAR); © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Why code it yourself? Get Glenn Berry’s latest scripts at Essential DMVs Transactions Sys.dm_tran_lock Sys.dm_db_index_operational_stats Sys.dm_db_index_usage_stats SPID Activity & SQL Statements Sys.dm_os_exec_requests Sys.dm_exec_requests Sys.dm_exec_query_stats Sys.dm_exec{procedure | trigger}_stats Performance & Wait Stats Sys.dm_os_wait_stats Sys.dm_os_performance_counters Sys.dm_os_waiting_tasks I/O Sys.dm_io_virtual_file_stats Sys.dm_io_pending_io_requests Why code it yourself? Get Glenn Berry’s latest scripts at http://sqlserverperformance.wordpress.com/category/diagnostic-queries/

My SQL Server is not Performing Well My SQL Server is “Slow” My query is now “slow” My queries are timing out Overall Server High CPU I/O bottleneck Blocking Latch Waits Other Waits Specific Query Running Waiting RUNNING WAITING Live Perf Dashboard DMVs Port-Mortem PerfMon Profiler/Xevents The reason for post-mortem we use PSSDiag is to do an uber-collection so we don’t have to collect data for yet another instance of the problem to occur. But if you catch it live the 2nd time you may be able to use Perf Dashboard reports and DMVs to solve it.

Check Windows Server logs. Resolve any errors and recheck. 1. Inside or Outside of MSSQL? Check Windows Server logs. Resolve any errors and recheck. 2. Caused by an MSSQL or SQLAgent error? Check SQL Server and SQL Agent logs. Resolve any errors and recheck. 3. Excessive wait stats? Assess wait statistics to categorize the bottleneck using sys.dm_wait_stats. 4. Wait stats correlate to specific sessions or components? Assess session wait stats using sys.dm_waiting_tasks. Resolve problematic user activity or process. Assess other problem areas using other DMVs, like dm.os_performance_counters. Resolve system misconfiguration, design problem, or resource shortage.

remediation: SQL Server Management Studio 9/11/2018 4:46 AM remediation: SQL Server Management Studio Once the root-problem is revealed, you still have to fix it. Common resolutions using SSMS include: Debug a SQL Server procedure or function Tune one or more SQL statements Add or alter indexes Tuning SQL code can be difficult unless you know how to read an execution plan: SQL Sentry Plan Explorer is FREE! © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Execution Plans 9/11/2018 4:46 AM Explain plans tell you all the internal steps that the SQL Server takes to complete the action Read graphic explain plans from right to left. (Textual ones from bottom to top) Graphic plans use icons to represent actions, while arrows represent data flows © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Demo Reading Basic Execution Plans 9/11/2018 4:46 AM © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Demo of SSMS Graphic Execution Plans Take two SQL statements that produce the same result sets, as in Image 1. Let’s compare to see which consumes the least resources. Select Query >> Include Actual Execution Plan Select Query >> Query Options >> Advanced >> and toggle ON both SET STATISTICS TIME and SET STATISTICS IO Under the Execution Plan tab, compare the Query Cost of the two queries. You can see from Image 2, that the second query has a lower cost. Under the Messages tab, you can see that the second query also has a lower elapsed time and that it performs less logical reads.

Fixing Bad Code In SSMS 9/11/2018 4:46 AM Fixing bad code is an exercise in experimentation Lots of tips & tricks to try Check out our tuning content at http://sqlsentry.tv Figure out the work done in the code, then try a new version that might improve it: Different search arguments in the WHERE or JOIN clauses to make better use of indexes Use an alternative pattern Apply a different locking strategy Use a query hint Clear you caches, then rewrite, test, repeat… © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Summary 9/11/2018 4:46 AM When using the native tools in SQL Server to detect, diagnose, and resolve problems, you must use: The SQL Server Error Log Wait Stats PerfMon SQL Profiler and/or server-side traces DMVs (sometimes, the Perf Dashboard) Assess execution plans © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Free Tools for finding performance problems Adam Machanic’s sp_whoisactive Brent Ozar’s sp_askBrent, sp_blitz Extended events – Jonathan Kehayias DMVs – Glenn Berry’s diagnostic queries http://sqlserverperformance.wordpress.com/category/diagnostic-queries/ SQL Sentry’s free Plan Explorer tool http://sqlsentry.com/plan-explorer sp_whoisactive : http://sqlblog.com/files/folders/beta/entry42453.aspx Sp_askbrent: http://www.brentozar.com/askbrent/ Sp_blitz: http://www.brentozar.com/blitz/ Extended events 31-day series: http://www.sqlskills.com/blogs/jonathan/category/xevent-a-day-series/ Glenn Berry’s diagnostic queries: http://sqlserverperformance.wordpress.com/category/diagnostic-queries/

THANK YOU! Slides, video recording and scripts at http://blogs.sqlsentry.com/kevinkline/end-to-end-troubleshooting-checklist-for-microsoft-sql-server/ . I’m online on social media and email at ‘kekline’: Twitter.com/kekline Facebook.com/kekline LinkedIn.com/in/kekline kkline@sentryone.com