1. Analyzing Performance Problems Using XEvents, DMVs & Query Store
Ola Hallengren, Principal Database Architect, Saxo Bank

2. Principal Database Architect, Saxo Bank
Ola Hallengren
Principal Database Architect, Saxo Bank
Microsoft MVP – Data Platform
Author of the popular maintenance scripts at
Database Consultant

3. Thanks to our sponsors

4. Agenda Extended Events, DMVs and Query Store
How are they working in different scenarios?
How can you use them together?
Comparing performance data
Troubleshooting performance problems
Experiences from the bank and when working with customers

5. Extended Events in a query or stored procedure execution
Level
Reads
Writes
CPU time
Dura-tion
Result
sql_batch_completed
Outer
Yes
OK/Abort/Error
rpc_completed
module_end
Module * -
sp_statement_completed
Statement
Only fires if completed successfully
sql_statement_completed
* A module could be a stored procedure or a function

6. A stored procedure was timing out – What statement was executing?
You are looking at a rpc_completed event with result = Abort, and a module_end event. What statement was executing when the timeout happened?
module_end tsql_frame Successful execution: <frame … line="0" offsetStart="-1" offsetEnd="-1“ /> Aborted execution: <frame … line="6" offsetStart="74" offsetEnd="202“/>
You can pass the sql_handle to sys.dm_exec_sql_text to retrieve the sql text, and then use the offsets to calculate the statement
Demo

7. Using Extended Events Histograms for troubleshooting connection pooling
Real-world case: You have recently started to get more connection timeouts, expecially from one application. The developers are saying that they are using connection pooling, but how can you check it?
The Extended Event login has a column is_cached. If you filter on is_cached = 0 you get connections where a real physical connection is established (so a non-pooled connection).
Using a histogram target you can find out about which applications and host names that have the most non-pooled connections.
Demo

8. Query Store – The basics
The typical use case for Query Store is to look at changes to query plans.
Automatically stores queries, query plans, run_time statistics and wait statistics.
Run_time statistics and wait statistics are aggregated for one hour intervals. Stored by execution type: Regular, Aborted, and Exception
Query Store data stored in the user database

9. Query_id – The starting point for Query Store analysis
Real-world case: A critical report timed out last night. You have the application name, the host name, and the timestamp. Unfortunately the error message in the application was not very clear, and you are not sure what the query was.
Limitation: Query Store does not have any information about application name, host name or login name. How can you find the query in Query Store? You need some additional data to find the query:
sys.dm_exec_requests: statement_sql_handle and statement_context_id
Extended Events: sql_handle, start_offset, and end_offset in tsql_frame Unfortunately there is no statement_context_id Extended Events.
Demo

10. Using context_setting_id to find the correct query_id for your business application
Real-world case: Some days later another critical report times out. This time you know the name of the stored procedure. You are thinking that it should be easy to find the query in Query Store and maybe force a good plan.
To your surprise there are several query_id’s for the same object_id and query_text_id in sys.query_store_query. Only the context_setting_id is different.
It turns out that some developers have been executing the stored procedure in SSMS to try to find out about the problem. SSMS has different set_options than the application.
Again you need statement_context_id in sys.dm_exec_requests.
Demo

11. Killed sessions not in Query Store runtime and wait statistics
Real-world case: A query has been running for a long time. There is no command timeout in the application, so eventually you have to kill it. You look in the Query Store to compare the plans, but you can’t find any Aborted executions.
Limitation: Executions are not captured in the Query Store sys.query_store_runtime_stats and sys.query_store_wait_stats, if a session is killed or if the client application or host is restarted or crash.
The plan is still captured in sys.query_store_plan.
Demo

12. Navigating XEvents, DMVs and Query Store
Look at DMV snapshots for the same session_id, within the interval for the Extended Event XEvent start time <= DMV snapshot time <= XEvent end time
Extended Events: rpc_completed sql_batch_completed module_end wait_completed
DMV snapshots sys.dm_exec_sessions sys.dm_exec_requests sys.dm_exec_session_wait_stats sys.dm_os_waiting_tasks sys.sysprocesses
Start time = timestamp – duration End time = timestamp
statement_sql_handle and statement_context_id
sql_handle, start_offset, and end_offset in tsql_frame (statement_context_id is missing)
Query Store

13. A parallel query is running – Where are the reads in sys
A parallel query is running – Where are the reads in sys.dm_exec_requests?
Real-world case: A parallel query has been running for a long time. You suspect that it is doing a lot of reads, but the reads and logical_reads in sys.dm_exec_requests are 0. What is going on?
Limitation: When a parallel query is running, the reads, logical_reads, and writes in sys.dm_exec_requests, are only updated for the coordinator thread.
sys.sysprocesses.physical_io can be used as a workaround. That is updated for all child threads.
Demo

14. A serial query is running – When are the DMVs updated?
elapsed_time
cpu_time
reads logical_reads
writes
sys.dm_exec_requests
Continuously
sys.dm_exec_sessions
When the request ends

15. A parallel query is running – When are the DMVs updated?
elapsed_time
cpu_time
reads logical_reads
writes
sys.dm_exec_requests
Continuously
Continuously for the coordinator thread
When the statement ends for the child threads
sys.dm_exec_sessions
When the request ends
If the statement, request and session end at the same time, you will never see the reads, logical_reads and writes

16. Looking at log writes and availability group synchronizations in Extended Events, DMVs and Query Store
Real-world case: You are troubleshooting an issue with slow transactions. You are collecting data from Extended Events, DMVs, and you are also looking in Query Store. The durations just do not add up.
Limitation: Query Store, sys.dm_exec_query_stats, and the statement - level Extended Events do not include the hardening of the log writes to disk, or the synchronizations with the secondary replica in availability groups.
Demo

17. Durations – Log writes and AG synchronizations
Duration includes transaction log writes to disk
Duration includes AG synchronizations *
sql_batch_completed
Yes
rpc_completed
module_end
sp_statement_completed No
sql_statement_completed
Only if outside a module_end
Query Store
sys.dm_exec_query_stats
sys.dm_exec_procedure_stats
* Availability groups with synchronous-commit

18. A comparison between Query Store and DMVs
Level
Includes
Where is it stored?
Interval
Query Store
Statement
All executions, except killed
In memory  User database
1 hour (default)
sys.dm_exec_query_stats
Successful executions only
In memory
No interval
sys.dm_exec_procedure_stats
Stored procedure
All executions
Bonus slide

19. How recompiles and FREEPROCCACHE affects Query Store and DMVs
Recompile triggered by statistics update or index rebuild
sp_recompile
DBCC FREEPROC
CACHE
Query Store
Not affected
sys.dm_exec_query_stats
Stats reset plan_generation_num updated
Entry removed
All entries removed
sys.dm_exec_procedure_stats
Bonus slide

20. How recompile hints affects Query Store and DMVs
Are stored procedures WITH RECOMPILE included?
Are queries with
OPTION (RECOMPILE)
included?
Query Store
Yes
sys.dm_exec_query_stats No
Only last execution
sys.dm_exec_procedure_stats
Bonus slide

21. Looking at problems back in time
How do you answer questions like “Why did this stored procedure time out last night?” When you are investigating problems back in time, the key thing is to have the data.
For Query Store the data collection happens automatically in the user database.
We have seen that in many cases the data collected in Query Store is not enough. Often you also need data from Extended Events and DMVs.
For Extended Events and DMVs, you need to do the collection of data, or use a monitoring tool. You are welcome to use my monitoring scripts:

22. Questions?
The presentation and demo code is available at
Monitoring scripts:
You can contact me at