1. Monitor SQL Server Efficiently
Remus Rusanu
#sqlsaturday565
24 Sept 2016, Bucharest
24 Sept 2017
SQLSaturday 565 Bucharest

2. whoami Worked in the SQL Server team with Microsoft since 2001
Now founder with DBHistory.com
Spent many hours investigating performance
On-call performance engineer for Azure SQL DB
Troubleshooting based on telemetry alone
24 Sept 2017
SQLSaturday 565 Bucharest

3. Why do we monitor? Troubleshooting Post Mortem analysis Trending
We need to investigate incidents as they occur and we need to measure right now
On demand, high resolution, short duration, discardable
Requires access to the system being monitored
Post Mortem analysis
After an incident we need to look back to understand why it occurred
Contiguous, medium resolution, discardable after a grace period
Missing data feedback loop
Trending
Understand long term direction, estimate capacity needs, justify spending requests
Contiguous, low resolution, long retention
Baselining
Periodically collect same data we would collect in troubleshooting so we can compare an incident with normal activity
On schedule, high resolution, medium duration, long retention
Alerting
Detect incidents, notify on-call team and trigger investigation
Alerting coupled with automation: mitigation bots for know issues
24 Sept 2017
SQLSaturday 565 Bucharest

4. What do we monitor? Activity Capacity Availability Recoverability
What is running
Capacity
Free Disk, space used
CPU utilization
IO utilization
Memory use, paging
Network use, bandwidth, latency
Availability
Uptime, SLA
Errors
Recoverability
Backups
Availability Groups
Specific features
Replication
24 Sept 2017
SQLSaturday 565 Bucharest

5. How do we monitor? Performance Counters DMVs XEvents ETW Logs
The Golden Standard when it comes to measurement
Easy to collect, low impact, rich toolset, cheap to store
DMVs
Difficult to collect, many require snapshot-store-and-compare
Some have significant impact
XEvents
Abundant information
Easy to filter at source
Difficult to collect
ETW
Logs
Event Notifications
Query Store
24 Sept 2017
SQLSaturday 565 Bucharest

6. USE methodology http://www.brendangregg.com/usemethod.html
Utilization, Saturation and Errors
Identify resources in the system
For each resource, identify metrics that represent utilization (in use vs. idle) and saturation (queueing, blocking, waiting). Identify errors indicators (events, logs etc)
When investigating, iterate through resources
Look at error indicators
Look for saturation indicators
Look for high utilization percentage
Generic methodology, the trick is identifying resources and collecting/finding the associated metrics
Can be applied at host level (CPU, IO, network) but also at SQL Server internals level
24 Sept 2017
SQLSaturday 565 Bucharest

7. SQL Server Query Execution http://rusanu
24 Sept 2017
SQLSaturday 565 Bucharest

8. Performance Counters
Extremely cheap for a process to produce performance counters
Increment a memory location in a shared memory area
Extremely cheap for monitoring to read a value
Read the value via shared memory
Low impact
Rich toolset
SDK: .Net, PDH native
OS service for collecting them (Data Collection Sets)
perfmon.exe, logman.exe, typeperf.exe
Can write directly to SQL (and this is supported by Data Collection Sets)
PowerShell supports direct counter querying Get-Counter
Data Collector Sets must go through COM object Pla.DataCollectorSet
24 Sept 2017
SQLSaturday 565 Bucharest

9. Performance Counters tools
perfmon.exe
Interactive GUI for Data Collector Sets management
Interactive GUI for on-demand counters collection
Graphic visualization for both real time and historic logs
logman.exe
CLI for Data Collector Sets (counters, ETW, alerts)
Countless 3rd party tools
DMV sys.dm_os_performance_counters
I advice against it’s use: expensive to query, difficult to read correctly
Has the advantage of being available over TDS
24 Sept 2017
SQLSaturday 565 Bucharest

10. Performance Counters SQL logging option
Read and Write counter values into an ODBC defined destination
Define an ODBC 64bit System DSN, specify database name explicitly
A feature of PDH, available for all PDH consumers
SDK: PdhOpenLog (…, PDH_LOG_TYPE_SQL, …)
perfmon, logman, typeperf
On first connect it will deploy the SQL tables
Schema is documented at
CounterData table is a perfect columnstore candidate
24 Sept 2017
SQLSaturday 565 Bucharest

11. SQL Logging table schema
24 Sept 2017
SQLSaturday 565 Bucharest

12. Interpreting the SQL logged data
24 Sept 2017
12 |
SQLSaturday 565 Bucharest

13. Reading SQL logged counters using Perfmon
24 Sept 2017
SQLSaturday 565 Bucharest

14. What to collect: CPU Processor (_Total)\% Processor Time
Processor Object(*) seldom justifies the overhead
Processor (_Total)\% Priviledged Time
Process(*)\% Processor Time
Can help track down when other processes starve CPU
Expensive to collect, each process is a separate instance
Collect Process(sqlservr)\% Processor Time and Process(_Total)\% Processor Time can at least shift the blame, but not pinpoint the culprit.
Buffer Manager\Page lookups/sec
Indicative of scans, it helps explain high CPU
24 Sept 2017
SQLSaturday 565 Bucharest

15. What to collect: memory (OS)
Memory\Page Reads/sec
This are hard page faults. Page Faults\sec is soft faults.
Memory\% Committed Bytes In Use
Memory\Available Bytes
Memory\Commit Limit
Process(sqlservr)\Private Bytes
24 Sept 2017
SQLSaturday 565 Bucharest

16. What to collect: memory (SQL)
Memory Manager\*
Really: collect every counter if you can afford it.
No magic formula for ‘good’ vs. ‘bad’ values, but can be compared with baseline
Memory Grants Outstanding
Memory Grants Pending
Buffer Manager\Buffer cache hit ratio
Buffer Node(*)\Page Life Expectancy
24 Sept 2017
SQLSaturday 565 Bucharest

17. What to collect: IO (OS)
Process(sqlservr)\
IO Read Operations/sec
IO Read Bytes/sec
IO Write Operations/sec
IO Write Bytes/sec
The counters measure all IO (disk, network, devices)
Physical Disk\
What to capture and how to interpret it is a black art
Windows Performance Monitor Disk Counters Explained
‘Good’ vs. ‘Bad’ values are highly dependent on hardware
Memory\Pages/sec
24 Sept 2017
SQLSaturday 565 Bucharest

18. What to collect: IO (SQL)
Buffer Manager/
Page reads/sec
Readahead pages/sec
Page writes/sec
Background writer pages/sec
Checkpoint pages/sec
Extension page reads/sec
Extension pages writes/sec
Lazy writes/sec
Database(*)/
Log Bytes Flushed/sec
Backup/Restore Throughput/sec
24 Sept 2017
SQLSaturday 565 Bucharest

19. Understanding how SQL Server executes a query http://rusanu
TL/DR:
CPU
CPU
CPU
CPU
Wait
Wait
Wait
Time
24 Sept 2017
SQLSaturday 565 Bucharest

20. What to collect: blocking (the aspiration slide)
sys.dm_os_wait_stats
sys.dm_os_latch_stats
sys.dm_os_spinlock_stats
TODO: fill up when decent monitoring possible
Ever Increasing values
Require snapshot-store-and-compare
Can be reset, difficult to detect in processing logic
No differentiation between idle wait and busy wait resulting in tribal knowledge ‘benign waits’
However, still important to collect… somehow
sys.dm_session_wait_stats
24 Sept 2017
SQLSaturday 565 Bucharest

21. What to collect: blocking (the pragmatic slide)
Wait Statistics(*)\Average wait time (ms)
A performance counter, easy to collect and analyze
There is an instance per wait type
but only few selected wait types are represented
Latches\Average Latch Wait Time (ms)
Latches\Total Latch Wait Time (ms)
Locks\Average Wait Time (ms)
Locks\Number of Deadlocks/sec
General Statistics\Processes blocked
24 Sept 2017
SQLSaturday 565 Bucharest

22. What to collect: workload
SQL Statistics\Batch Requests/sec
SQL Statistics\SQL Attention rate
‘Attention’ is the TDS jargon for client command timeout
Transactions\Transactions
General Statistics\User Connections
SQL Errors(_Total)\Errors/sec
24 Sept 2017
SQLSaturday 565 Bucharest

23. What to collect: miscelaneous
LogicalDisk(*)\% Free Space
Databases(*)\Data File(s) Size (KB)
Databases(*)\Log Growths
Plan Cache(*)\Cache Objects Count
General Statistics\Temp tables creation rate
Access Methods
Full scans/sec, Probe Scans/sec, Range Scans/sec, Index Searches/sec
Page Splits/sec
Skipped Ghosted Records/sec, Forwarded Records/sec
24 Sept 2017
SQLSaturday 565 Bucharest

24. Measure workload response distribution
Batch Resp Time(*)\*
“one latency distribution plot is worth a thousand throughput measurements”
Typical SQL Server has a heterogeneous workload
X batches complete in .01s Y in 1s and Z in 100.0
Is it one query over a latency distribution? Or is it 3 very different queries?
Latency for specific queries better measured in app
It is trivial to expose new counters from apps
24 Sept 2017
SQLSaturday 565 Bucharest

25. Collecting via querying
Collecting data by periodically querying catalog views/DMVs
Expensive, some DMVs have serious performance overhead
Snapshot-store-and-compare
Many DMVs arbitrarily reset internally, unreliable for monitoring
Some information, still, hard to discover any other way
sys.dm_io_virtual_file_stats
sys.dm_db_index_usage_stats
24 Sept 2017
SQLSaturday 565 Bucharest

26. Collecting query execution stats
Don’t. Use Query Store instead. But if you must:
sys.dm_exec_query_stats
sys.dm_exec_procedure_stats
Avoid the join with sys.dm_exec_sql_text and sys.dm_exec_sql_plan
Very expensive
Honor query_hash and plan_hash
24 Sept 2017
SQLSaturday 565 Bucharest

27. Using Event Notifications
create queue notifications; go create service notifications on queue notifications ([ create event notification [sqlsaturday] on server for ddl_events to service N'notifications', N'current database'; waitfor(receive cast(message_body as xml) from notifications);
24 Sept 2017
SQLSaturday 565 Bucharest

28. DDL_EVENTS Captures any DDL, in any database, for any type
CREATE/ALTER/DROP
GRANT/DENY/REVOKE
sp_rename, sp_tableoptions etc
Captures any configuration change
sp_configure
Database scoped options
It can be made more granular if desired
Capture only specific database
Capture only specific events:
select * from sys.server_events
24 Sept 2017
SQLSaturday 565 Bucharest

29. Event Notifications for Profiler events
Bridges Profiler events as Event Notification messages
select * from sys.trace_events
It can do everything administrative trace can do plus:
Deliver the message remotely
Trigger activated procedure
24 Sept 2017
SQLSaturday 565 Bucharest

30. Collect all warnings? File Growth? Deadlocks? Logins?
create event notification [trace_events] on server for Hash_Warning, Execution_Warnings, Sort_Warnings, Bitmap_Warning, Log_File_Auto_Grow, Deadlock_graph, Audit_Login_Failed, Audit_Login to service …
24 Sept 2017
SQLSaturday 565 Bucharest

31. Collecting Event Notification
Events can be delivered remotely
All events look the same, an XML payload
Must shred the XML and discover object types, object names and event types
Some events apply to multiple objects eg. CREATE INDEX
Coupled with Activation can trigger warnings or mitigation
Reliable delivery, can survive disconnects
Also means that events always refer to past, automated mitigation must check current state before proceeding
No toolset whatsoever
24 Sept 2017
SQLSaturday 565 Bucharest

32. Query Data Store
Absolutely best option for monitoring query performance
Optimized, hooked deep into execution
Cannot be simulated with DMV snapshotting
Excellent troubleshooting tool
Runtime stats, Compilation Stats
24 Sept 2017
SQLSaturday 565 Bucharest

33. Query Store missing features
Collection of wait info
Centralized aggregation of multiple sources
Collect info from readable secondaries
24 Sept 2017
SQLSaturday 565 Bucharest

34. XEvents Overwhelmingly rich information Cheap to produce
Built in analytical capabilities
Counter, Histogram, Pair Matching
Difficult to collect
Require ETL through a SQL Server
Poor toolset
24 Sept 2017
SQLSaturday 565 Bucharest

35. ETW, Windows Performance Analyzer
Very powerful for analyzing the entire stack
A must when the issue is not SQL Server
Quick Start Guide: WPA Basics
Bruce Dawson blog:
24 Sept 2017
SQLSaturday 565 Bucharest

36. Sampling Profiling Requires Visual Studio toolset vsperf.exe
Captures execution stacks on all threads, about 10k/sec
Only as an on-demand troubleshooting option
Difficult to setup
Can have impact
Incredibly rich insight into what the server is doing
Provided you manage to resolve the symbols…
Requires code understanding
Educated guess of execution role from function names
24 Sept 2017
SQLSaturday 565 Bucharest

37. Sqlservr sample profile
24 Sept 2017
SQLSaturday 565 Bucharest

38. Tanks, Q&A and please review
24 Sept 2017
SQLSaturday 565 Bucharest