0. © 2012 Microsoft Corporation. All rights reserved.
Microsoft Confidential
© 2012 Microsoft Corporation Microsoft Confidential

1. SQL Saturday – Performance Troubleshooting
Methodology and Tools
Microsoft Confidential
© 2012 Microsoft Corporation Microsoft Confidential

2. Conditions and Terms of Use
Microsoft Confidential
This training package is proprietary and confidential, and is intended only for uses described in the training materials. Content and software is provided to you under a Non-Disclosure Agreement and cannot be distributed. Copying or disclosing all or any portion of the content and/or software included in such packages is strictly prohibited.
The contents of this package are for informational and training purposes only and are provided "as is" without warranty of any kind, whether express or implied, including but not limited to the implied warranties of merchantability, fitness for a particular purpose, and non- infringement.
Training package content, including URLs and other Internet Web site references, is subject to change without notice. Because Microsoft must respond to changing market conditions, the content should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented after the date of publication. Unless otherwise noted, the companies, organizations, products, domain names, addresses, logos, people, places, and events depicted herein are fictitious, and no association with any real company, organization, product, domain name, address, logo, person, place, or event is intended or should be inferred.
Copyright and Trademarks
© 2012 Microsoft Corporation. All rights reserved.
Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in written license agreement from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property.
Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Microsoft Corporation.
For more information, see Use of Microsoft Copyrighted Content at
Microsoft®, Internet Explorer®, and Windows® are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. Other Microsoft products mentioned herein may be either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. All other trademarks are property of their respective owners.
© 2012 Microsoft Corporation Microsoft Confidential

3. Microsoft Confidential
Who am I ? What is PFE?
Joe McTaggart
Native Texan
Working with SQL since 2001
Worked with SQL
Premier Field Engineering
Reactive and Proactive support for Premier customers
Architecture/Project Guidance
Team Mentoring
Deliver workshops
Troubleshooting
Microsoft Confidential
© 2012 Microsoft Corporation Microsoft Confidential

4. Microsoft Confidential
Overview
Methodology – Breaking down the problem
What is the problem?
Isolate the problem
What are my bottlenecks
Tools
SQLDiag
Perfmon
Profiler and extended events
DMV’s
Microsoft Confidential
© 2012 Microsoft Corporation Microsoft Confidential

5. Troubleshooting Methodology
Microsoft Confidential

6. What’s your problem?
“MyApp is slow today.”
“MyWebPage is slow when I click on some button.”
“MyApp is slow because of blocking in the database.”
Users reporting problems are often nebulas and biased.
What changed?
When did the problem start?
Is it always slow or only at certain times of day?
Is anything else behaving differently?
What does “slow” mean?
Ask Questions
A problem not defined is impossible to solve
Smaller scope means less data to look at
Narrow the problem
Microsoft Confidential
© 2012 Microsoft Corporation Microsoft Confidential

7. Microsoft Confidential
Divide and Conquer
Isolate the issue as much as possible
Ruling out possible causes is as important to finding the actual problem.
Play “20 questions”
Client or Server problem
Windows or SQL
All SQL Servers or just one
All databases or just one
Microsoft Confidential
© 2012 Microsoft Corporation Microsoft Confidential

8. What resource is my bottleneck?
System
CPU
Memory
Disk
Network
SQL
Blocking
Internal memory
Latches
Application
Consuming results
Transaction times
App Server and Client machines
Microsoft Confidential

9. SQL Bottleneck Is there high CPU yes Some other process
High CPU on sqlservr.exe
What are the expensive queries?
Are statistics up to date?
Missing indexes no
Waiting
Locks IO
Network
Other Waits
Microsoft Confidential

10. Microsoft Confidential
High CPU
User or Privileged time?
Privileged time represents time in the kernel
AntiVirus
IO Filter drivers
Other
Xperf utility can help identify cause of kernel time
User time represents normal process activity spent in application code.
Sqlservr.exe high CPU
Find expensive queries
Look for expensive operations like sorts, hashes, or scans
CPU and IO tend to go hand in hand. Find high IO queries
Common Causes
Poor estimates
Stale Statistics
Poor Query Design
Missing Indexes
Microsoft Confidential

11. Microsoft Confidential
IO Bottleneck
PAGEIOLATCH_EX and PAGEIOLATCH_SH
Latches that indicate a query is waiting on an IO to complete
Is SQL asking for too much?
Identify high read/write queries
Sys.dm_io_virtual_file_stats – Identify which db files are busiest
Profiler
Common solutions
Adding Indexes
Updating statistics
Query Redesign
Are the disks performance correctly?
Perfmon counters can help analyze
Throughput
IOPS
Latency
Microsoft Confidential

12. Microsoft Confidential
Memory Bottleneck
External Pressure
Ensure there is free memory available to the OS
Is there paging?
Causes sql server to reduce it’s memory caches
Internal Pressure
Under pressure, data/index pages and query plans are aged out of memory
Less page reuse
May see more query compilations
Many times memory pressure manifests as IO issues
Microsoft Confidential

13. Microsoft Confidential
Blocking
Find the lead blocker
Sys.dm_exec_requests
Sys.dm_tran_locks
Is it running, idle, or waiting on some resource
Running
Tune the query
Idle
Investigate the client
Orphaned transaction
Waiting
What resource is it waiting on
Too high Isolation Level? Snapshot?
Normal concurrency?
Microsoft Confidential

14. Microsoft Confidential
Deadlock
Two or more processes holding resources in a circular locking chain
SQL Server will detect deadlocks (not just lock deadlocks)
How to identify
Client error message
Profiler trace
Xevent
Trace Flag 1222
Most likely a design issue
Different processes accessing resources in different orders
Unnecessarily long transactions
Poorly tuned queries
Microsoft Confidential

15. Network waits, must be a problem with the network.
Async_Network_IO waits
Sql is sending result sets to the client
Slow Network
Packets are taking a long time to get to the destination
Network issues causing lots of resends
Client not consuming results
Far more common
Perf issue on client
Poor design
Microsoft Confidential

16. Microsoft Confidential
Data Collection
Microsoft Confidential

17. Microsoft Confidential
Performance Monitor
Perfmon is particularly useful for:
Collecting information on system resource utilization
Collecting performance information over a long period of time.
Base lining server and application performance.
Microsoft Confidential

18. General Perfmon Counters
The following counters are useful for just about any server and any application.
All Counters for:
LogicalDisk
Memory
Network Interface (optional)
Process
Processor
These counters cover your basic system resource monitoring.
Microsoft Confidential

19. Microsoft Confidential
Perfmon Sql Counters
Buffer Manager -- These counters work to give you an idea of the overall memory pressure within the sql server process.
Free Pages
Lazy Writes/Sec
Page Life Expectancy
In general this should be above 300 sec (5 minutes)
General Statistics
User Connections
Locks
Number of Deadlocks/sec
Quick way to identify if deadlocks are occurring. Requires follow up with Profiler and DMV information to troubleshoot.
Microsoft Confidential

20. Perfmon SQL Counters (cont.)
Memory Manager
Target Server Memory
Total Server Memory
If target is below total this is an indication that the OS is under memory pressure and sql server is trying to give memory back to the OS.
At startup, Target will be above Total as sql server is acquiring memory as needed.
Sql Statistics
Batch Request/sec
How busy is the server
Sql Compilations and Sql Re-Compilations
Is sql server spending a lot of time compiling queries.
Microsoft Confidential

21. Microsoft Confidential
Profiler
Profiler is a performance tuning and troubleshooting tool.
Profiler is good at:
Capturing queries responsible for periodic performance events
Auditing security and activity (beware overhead).
Benchmarking, stress testing, baselining activity and throughput. (Replay.)
Visualizing deadlock behavior
Viewing SHOWPLAN results for each query captured
Capturing events for Database Engine Tuning Advisor
Microsoft Confidential

22. General Events to collect
Errors And Warning Group
All events
Tend to have very few hits and typically very informative
Locks
Deadlock Graph
Lock Escalation
Security Audit
Audit Login
Audit Logout
Sessions
Existing Connections
Stored Procedures
RPC:Completed
TSQL
SQL:BatchCompleted
Transactions
SQL Transaction
DTC Transaction
Microsoft Confidential

23. Microsoft Confidential
Detailed Events
Use with caution as these may impact performance.
SP:StmtCompleted and SQL:Stmtcompleted
Only use if you have isolated the problem down to a batch and/or stored procedure and need to determine which query in the batch is the root cause.
Many times there are only one statement per batch or the problem query in a batch is obvious.
Stmt level events will typically generate many events which can cause trace level waits on the sql server as processes have to wait to write to the trace file before moving on.
Performance: Any of the Showplan events
The Showplans will generate statement level query plans.
Expensive to produce and large events
Starting Events (SQL and RPC)
Useful for replay traces
Microsoft Confidential

24. Profiler Best Practices
Run Server side Trace always
If possible avoid the GUI when capturing.
If not, then run GUI with server side trace
Trace to a file not to a table
Never Trace to a UNC Path
Capture Events Judiciously
XML_SHOWPLAN Events are expensive
Statement level events may be expensive depending on how busy a server
Default to a Batch Level Trace
Monitor running traces with sys.traces
Trace filtering reduces Trace size but increases CPU burden
Capture on a drive different that Data drive for the databases
Microsoft Confidential

25. Microsoft Confidential
Trace Analysis Tools
RML
ReadTrace is a utility inside of the RML tools that can parse and aggregate sql trace files.
RML can also be used to replay traces and do stress testing.
SQL Nexus
Contains the ReadTrace piece of RML but also has parsers for Perfmon and the perf stats script from PSSDiag.
Microsoft Confidential