© 2012 Microsoft Corporation. All rights reserved. Microsoft Confidential © 2012 Microsoft Corporation Microsoft Confidential

SQL Saturday – Performance Troubleshooting Methodology and Tools Microsoft Confidential © 2012 Microsoft Corporation Microsoft Confidential

Conditions and Terms of Use Microsoft Confidential This training package is proprietary and confidential, and is intended only for uses described in the training materials. Content and software is provided to you under a Non-Disclosure Agreement and cannot be distributed. Copying or disclosing all or any portion of the content and/or software included in such packages is strictly prohibited. The contents of this package are for informational and training purposes only and are provided "as is" without warranty of any kind, whether express or implied, including but not limited to the implied warranties of merchantability, fitness for a particular purpose, and non- infringement. Training package content, including URLs and other Internet Web site references, is subject to change without notice. Because Microsoft must respond to changing market conditions, the content should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented after the date of publication. Unless otherwise noted, the companies, organizations, products, domain names, e-mail addresses, logos, people, places, and events depicted herein are fictitious, and no association with any real company, organization, product, domain name, e-mail address, logo, person, place, or event is intended or should be inferred. Copyright and Trademarks © 2012 Microsoft Corporation. All rights reserved. Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in written license agreement from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property. Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Microsoft Corporation. For more information, see Use of Microsoft Copyrighted Content at http://www.microsoft.com/about/legal/permissions/ Microsoft®, Internet Explorer®, and Windows® are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. Other Microsoft products mentioned herein may be either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. All other trademarks are property of their respective owners. © 2012 Microsoft Corporation Microsoft Confidential

Microsoft Confidential Who am I ? What is PFE? Joe McTaggart Native Texan Working with SQL since 2001 Worked with SQL 6.5 - 2012 Premier Field Engineering Reactive and Proactive support for Premier customers Architecture/Project Guidance Team Mentoring Deliver workshops Troubleshooting Microsoft Confidential © 2012 Microsoft Corporation Microsoft Confidential

Microsoft Confidential Overview Methodology – Breaking down the problem What is the problem? Isolate the problem What are my bottlenecks Tools SQLDiag Perfmon Profiler and extended events DMV’s Microsoft Confidential © 2012 Microsoft Corporation Microsoft Confidential

Troubleshooting Methodology Microsoft Confidential

What’s your problem? “MyApp is slow today.” “MyWebPage is slow when I click on some button.” “MyApp is slow because of blocking in the database.” Users reporting problems are often nebulas and biased. What changed? When did the problem start? Is it always slow or only at certain times of day? Is anything else behaving differently? What does “slow” mean? Ask Questions A problem not defined is impossible to solve Smaller scope means less data to look at Narrow the problem Microsoft Confidential © 2012 Microsoft Corporation Microsoft Confidential

Microsoft Confidential Divide and Conquer Isolate the issue as much as possible Ruling out possible causes is as important to finding the actual problem. Play “20 questions” Client or Server problem Windows or SQL All SQL Servers or just one All databases or just one Microsoft Confidential © 2012 Microsoft Corporation Microsoft Confidential

What resource is my bottleneck? System CPU Memory Disk Network SQL Blocking Internal memory Latches Application Consuming results Transaction times App Server and Client machines Microsoft Confidential

SQL Bottleneck Is there high CPU yes Some other process High CPU on sqlservr.exe What are the expensive queries? Are statistics up to date? Missing indexes no Waiting Locks IO Network Other Waits Microsoft Confidential

Microsoft Confidential High CPU User or Privileged time? Privileged time represents time in the kernel AntiVirus IO Filter drivers Other Xperf utility can help identify cause of kernel time User time represents normal process activity spent in application code. Sqlservr.exe high CPU Find expensive queries Look for expensive operations like sorts, hashes, or scans CPU and IO tend to go hand in hand. Find high IO queries Common Causes Poor estimates Stale Statistics Poor Query Design Missing Indexes Microsoft Confidential

Microsoft Confidential IO Bottleneck PAGEIOLATCH_EX and PAGEIOLATCH_SH Latches that indicate a query is waiting on an IO to complete Is SQL asking for too much? Identify high read/write queries Sys.dm_io_virtual_file_stats – Identify which db files are busiest Profiler Common solutions Adding Indexes Updating statistics Query Redesign Are the disks performance correctly? Perfmon counters can help analyze Throughput IOPS Latency Microsoft Confidential

Microsoft Confidential Memory Bottleneck External Pressure Ensure there is free memory available to the OS Is there paging? Causes sql server to reduce it’s memory caches Internal Pressure Under pressure, data/index pages and query plans are aged out of memory Less page reuse May see more query compilations Many times memory pressure manifests as IO issues Microsoft Confidential

Microsoft Confidential Blocking Find the lead blocker Sys.dm_exec_requests Sys.dm_tran_locks Is it running, idle, or waiting on some resource Running Tune the query Idle Investigate the client Orphaned transaction Waiting What resource is it waiting on Too high Isolation Level? Snapshot? Normal concurrency? Microsoft Confidential

Microsoft Confidential Deadlock Two or more processes holding resources in a circular locking chain SQL Server will detect deadlocks (not just lock deadlocks) How to identify Client error message Profiler trace Xevent Trace Flag 1222 Most likely a design issue Different processes accessing resources in different orders Unnecessarily long transactions Poorly tuned queries Microsoft Confidential

Network waits, must be a problem with the network. Async_Network_IO waits Sql is sending result sets to the client Slow Network Packets are taking a long time to get to the destination Network issues causing lots of resends Client not consuming results Far more common Perf issue on client Poor design Microsoft Confidential

Microsoft Confidential Data Collection Microsoft Confidential

Microsoft Confidential Performance Monitor Perfmon is particularly useful for: Collecting information on system resource utilization Collecting performance information over a long period of time. Base lining server and application performance. Microsoft Confidential

General Perfmon Counters The following counters are useful for just about any server and any application. All Counters for: LogicalDisk Memory Network Interface (optional) Process Processor These counters cover your basic system resource monitoring. Microsoft Confidential

Microsoft Confidential Perfmon Sql Counters Buffer Manager -- These counters work to give you an idea of the overall memory pressure within the sql server process. Free Pages Lazy Writes/Sec Page Life Expectancy In general this should be above 300 sec (5 minutes) General Statistics User Connections Locks Number of Deadlocks/sec Quick way to identify if deadlocks are occurring. Requires follow up with Profiler and DMV information to troubleshoot. Microsoft Confidential

Perfmon SQL Counters (cont.) Memory Manager Target Server Memory Total Server Memory If target is below total this is an indication that the OS is under memory pressure and sql server is trying to give memory back to the OS. At startup, Target will be above Total as sql server is acquiring memory as needed. Sql Statistics Batch Request/sec How busy is the server Sql Compilations and Sql Re-Compilations Is sql server spending a lot of time compiling queries. Microsoft Confidential

Microsoft Confidential Profiler Profiler is a performance tuning and troubleshooting tool. Profiler is good at: Capturing queries responsible for periodic performance events Auditing security and activity (beware overhead). Benchmarking, stress testing, baselining activity and throughput. (Replay.) Visualizing deadlock behavior Viewing SHOWPLAN results for each query captured Capturing events for Database Engine Tuning Advisor Microsoft Confidential

General Events to collect Errors And Warning Group All events Tend to have very few hits and typically very informative Locks Deadlock Graph Lock Escalation Security Audit Audit Login Audit Logout Sessions Existing Connections Stored Procedures RPC:Completed TSQL SQL:BatchCompleted Transactions SQL Transaction DTC Transaction Microsoft Confidential

Microsoft Confidential Detailed Events Use with caution as these may impact performance. SP:StmtCompleted and SQL:Stmtcompleted Only use if you have isolated the problem down to a batch and/or stored procedure and need to determine which query in the batch is the root cause. Many times there are only one statement per batch or the problem query in a batch is obvious. Stmt level events will typically generate many events which can cause trace level waits on the sql server as processes have to wait to write to the trace file before moving on. Performance: Any of the Showplan events The Showplans will generate statement level query plans. Expensive to produce and large events Starting Events (SQL and RPC) Useful for replay traces Microsoft Confidential

Profiler Best Practices Run Server side Trace always If possible avoid the GUI when capturing. If not, then run GUI with server side trace Trace to a file not to a table Never Trace to a UNC Path Capture Events Judiciously XML_SHOWPLAN Events are expensive Statement level events may be expensive depending on how busy a server Default to a Batch Level Trace Monitor running traces with sys.traces Trace filtering reduces Trace size but increases CPU burden Capture on a drive different that Data drive for the databases Microsoft Confidential

Microsoft Confidential Trace Analysis Tools RML ReadTrace is a utility inside of the RML tools that can parse and aggregate sql trace files. RML can also be used to replay traces and do stress testing. http://support.microsoft.com/kb/944837 SQL Nexus Contains the ReadTrace piece of RML but also has parsers for Perfmon and the perf stats script from PSSDiag. http://sqlnexus.codeplex.com/ Microsoft Confidential