1. Become a SQL Server Performance Detective
Danette Dineen Riviello
Magellan Health December 5, 2015

2. Speaker Over 30 years working with Databases
Working with SQL Server since 1997
Manage an environment with over 270 SQL Servers

3. Goal
To learn ways to collect and interpret the data available in SQL Server 2008 and above to determine the culprit in chronic or emergent performance issues.

4. Game Plan What triggers an Investigation? Emergent Performance Issues
Chronic Performance Problems
Solving the Case

5. Open a case Increase in User Complaints Application Timeouts
Long-running queries
Chain of blocking

6. Look for Clues Look at all running processes
sp_who2 active
Look for one login or one database:
SELECT spid, [status],
loginame [Login],hostname,
blocked BlkBy,
Db_name(dbid) DBName,
Open_tran,
cmd Command,
cpu CPUTime,
physical_io DiskIO,
last_batch LastBatch,
[program_name] ProgramName
FROM master.dbo.sysprocesses
where [status] not in ('sleeping')
and loginame like '%login%'
And Db_name(dbid) = 'DBName'
ORDER BY dbname

7. Find Lead Blocker Look for the lead of a blocking chain
SELECT spid,sp.STATUS
,loginame = SUBSTRING(loginame, 1, 12)
,hostname = SUBSTRING(hostname, 1, 12)
,blk = CONVERT(CHAR(3), blocked)
,open_tran ,dbname = SUBSTRING(DB_NAME(sp.dbid),1,10)
,cmd,waittype,program_name
,waittime ,last_batch
,SQLStatement =SUBSTRING (
qt.text,
er.statement_start_offset/2,
(CASE WHEN er.statement_end_offset = -1
THEN LEN(CONVERT(nvarchar(MAX), qt.text)) * 2
ELSE er.statement_end_offset
END - er.statement_start_offset)/2 )
FROM master.dbo.sysprocesses sp
LEFT JOIN sys.dm_exec_requests er
ON er.session_id = sp.spid
OUTER APPLY sys.dm_exec_sql_text(er.sql_handle) AS qt
WHERE spid IN (SELECT blocked FROM master.dbo.sysprocesses)
AND blocked = 0 or blocked =spid

8. Gather Evidence - Locks
Look at object locks
SELECT resource_type,
db_name(resource_database_id) "DatabaseName",
object_name(resource_associated_entity_id) "ObjectName",
request_status,
request_mode,request_session_id,
resource_description
FROM sys.dm_tran_locks sl
JOIN sys.objects so
ON SO.object_id = sl.resource_associated_entity_id
WHERE resource_type = 'OBJECT'

9. Gather Evidence - Results

10. Decipher Lock Modes Sch-S Schema stability. Sch-M Schema modification.
Shared. U
Update. X
Exclusive. IU
Intent Update. IX
Intent Exclusive. IS
Intent Shared.
SIU
Shared Intent Update.
SIX
Shared Intent Exclusive.
UIX
Update Intent Exclusive. BU
Bulk Update.

11. Demo 1
Demo – blocking chain and locks

12. Collect further clues Look for open transactions
SELECT spid, [status],
loginame [Login],hostname,
blocked BlkBy,
Db_name(dbid) DBName,
cmd Command,
Open_tran,
cpu CPUTime,
physical_io DiskIO,
last_batch LastBatch,
[program_name] ProgramName
FROM master.dbo.sysprocesses
WHERE open_tran>0
ORDER BY spid

13. Investigate Further DBCC OPENTRAN Oldest active transaction:
SPID (server process ID): 52
UID (user ID) : -1
Name : user_transaction
LSN : (46:472:1)
Start time : Nov :59:43:213PM
SID : 0x f9b6e0b adf672e
DBCC execution completed. If DBCC printed error messages, contact your system administrator.

14. Search for the “Smoking Gun”
When a stored procedure is performing poorly, run the following query to figure out what line of code it is running:
    SELECT [Spid] = session_Id
      , ecid
      , [Database] = DB_NAME(sp.dbid)
      , [User] = nt_username
      , [Status] = er.status
      , [Wait] = wait_type
      , [Individual Query] = SUBSTRING (qt.text,
             er.statement_start_offset/2,
      (CASE WHEN er.statement_end_offset = -1
             THEN LEN(CONVERT(NVARCHAR(MAX), qt.text)) * 2
            ELSE er.statement_end_offset END -
                                er.statement_start_offset)/2)
      ,[Parent Query] = qt.text
      , Program = program_name
      , Hostname
      , nt_domain
      , start_time
    FROM sys.dm_exec_requests er
    INNER JOIN sys.sysprocesses sp ON er.session_id = sp.spid
    CROSS APPLY sys.dm_exec_sql_text(er.sql_handle) as qt
    WHERE session_Id -- is the one in Question
    ORDER BY 1, 2

15. Investigate the environment
What has changed?
Look for DDL Changes in the default system trace:
varchar (500)
/* Fetch default trace file path */
= PATH FROM sys .traces WHERE is_default = 1
select ObjectName, DatabaseName
, StartTime, EventClass
, Case eventclass
when 46 then 'CREATE'
when 47 then 'DROP'
when 164 then 'ALTER'
end "DDLOperation"
, EventSubClass, ObjectType
, ServerName, LoginName
, ApplicationName, spid,
HostName,e.name "Eventname"
FROM DEFAULT ) evi
JOIN sys.trace_events e ON evi.eventclass = e.trace_event_id
WHERE
objecttype not in (21587)
AND EventClass in (46,47,164)
AND EventSubclass = 0

16. Inspect System Trace Data
Look for recent changes
Look at the Log directory for prior files

17. Identify Chronic Offenders
To find most expensive stored procedures:
SELECT TOP 25 d.object_id, d.database_id,
OBJECT_NAME(object_id, database_id) 'proc name',
d.cached_time, d.last_execution_time, d.total_elapsed_time,
d.total_elapsed_time/d.execution_count AS [avg_elapsed_time],
d.last_elapsed_time, d.execution_count
FROM sys.dm_exec_procedure_stats AS d
ORDER BY [total_worker_time] DESC;

18. Prioritize Worst Offenders
Most expensive Stored procedure runs

19. Find Costliest Procedures
Number of Days from cache time and most recent execution is about 15 – 16 days
Number of times executed during time period
Avg elapsed time multiplied by number of executions
Time procedure plan was cached

20. Find Costliest Queries
To find most expensive queries:
SELECT top 25 total_worker_time/execution_count AS AvgCPU
, total_worker_time AS TotalCPU
, total_elapsed_time/execution_count AS AvgDuration
, total_elapsed_time AS TotalDuration
, (total_logical_reads+total_physical_reads)/execution_count AS AvgReads
, (total_logical_reads+total_physical_reads) AS TotalReads
, execution_count
, SUBSTRING(st.TEXT, (qs.statement_start_offset/2)+1
, ((CASE qs.statement_end_offset WHEN -1 THEN datalength(st.TEXT)
ELSE qs.statement_end_offset
END - qs.statement_start_offset)/2) + 1) AS txt
, query_plan
FROM sys.dm_exec_query_stats AS qs
cross apply sys.dm_exec_sql_text(qs.sql_handle) AS st
cross apply sys.dm_exec_query_plan (qs.plan_handle) AS qp
ORDER BY 2 DESC

21. Analyze Costliest Queries
Number of times executed during time period
Total cost over all executions

22. Demo 2
Demo looking at Trace file and prior Trace file - most expensive stored procedures and queries

23. Setup More Extended Events
Blocking and Deadlocking
CREATE EVENT SESSION [blocked_process_demo] ON SERVER
ADD EVENT sqlserver.blocked_process_report(
ACTION(sqlserver.client_app_name,
sqlserver.client_hostname,
sqlserver.database_name)) ,
ADD EVENT sqlserver.xml_deadlock_report (
sqlserver.database_name))
ADD TARGET package0.asynchronous_file_target
(SET filename = N'c:\temp\XEventSessions\blocked_process_demo.xel',
metadatafile = N'c:\temp\XEventSessions\blocked_process_demo.xem',
max_file_size=(65536),
max_rollover_files=5)
WITH (MAX_DISPATCH_LATENCY = 5SECONDS)
/* Start the Extended Events session */
ALTER EVENT SESSION [blocked_process_demo] ON SERVER
STATE = START;

24. Look at Results To stop trace and look at the file :
ALTER EVENT SESSION [blocked_process_demo] ON SERVER
STATE = STOP;
SELECT *, CAST(event_data as XML) AS 'event_data_XML'
FROM sys.fn_xe_file_target_read_file
('C:\temp\Xeventsessions\blocked_process_demo*.xel', NULL,NULL, NULL);

25. Demo 3
Look at the blocked process trace

26. Set up a Profiler trace Job - Why
Less impact than an interactive trace
Can load trace data on an alternate server
Can load trace data at a different time of day
Capture specific parameters passed
Compare same time of day on different days
Works on all versions of SQL Server

27. Set up a Profiler trace Job - How

28. Set up a Profiler trace Job - How

29. Inspect the Profiler Trace File
Load the trace file to another server
select * into dbo.tmp_loadtraceFile_ServerA_201502'01_8
FROM ::fn_trace_gettable('d:\trace_ _8.trc', 1)
Query trace file to find commands that are calling the suspected stored procedure
select top 25 textdata, loginname, spid, duration, starttime, endtime, reads, cpu
From dbo.tmp_loadtraceFile_ServerA_ _8
Where textdata like '%offendingproc%'
Order by duration desc

30. Inspect Profiler Trace Data

31. Examine long running procedures
Look at the query plan
Missing index or wrong index chosen?
Look at the parameters sent in
Check for other runs that perform better
Could it be a parameter sniffing issue?

32. Parameter Sniffing - Defined
Query plan developed based on the first values passed to the procedure
Pros:
Saves time: only one compile needed
Cons:
Wrong query plan chose

33. Parameter Sniffing - Detection
Look at Query plans
If one procedure performs well in one case and not others
Do the index choices make sense?

34. Parameter Sniffing – Solutions
Do Nothing
Force Recompile each run (expensive!)
Query Hints (OPTIMIZE FOR)
Break down stored procedures to handle specific cases
Education users on best parameter choices

35. Rule out other culprits
Check for table scans caused by:
Missing index
Broad “where” clause
Check for improper join (many-to-many)
Check for too many tables in one join
Use of a function in a large query result set

36. Search for missing Indexes
SELECT
migs.avg_total_user_cost * (migs.avg_user_impact / 100.0) * (migs.user_seeks + migs.user_scans)
AS improvement_measure,
'CREATE INDEX [missing_index_' + CONVERT (varchar, mig.index_group_handle) + '_' +
CONVERT (varchar, mid.index_handle)
+ '_' + LEFT (PARSENAME(mid.statement, 1), 32) + ']'
+ ' ON ' + mid.statement
+ ' (' + ISNULL (mid.equality_columns,'')
+ CASE WHEN mid.equality_columns IS NOT NULL AND mid.inequality_columns IS NOT NULL THEN ','
ELSE ''
END
+ ISNULL (mid.inequality_columns, '')
+ ')'
+ ISNULL (' INCLUDE (' + mid.included_columns + ')', '') AS create_index_statement,
migs.*, mid.database_id,
mid.[object_id]
FROM sys.dm_db_missing_index_groups mig
INNER JOIN sys.dm_db_missing_index_group_stats migs ON migs.group_handle = mig.index_group_handle
INNER JOIN sys.dm_db_missing_index_details mid ON mig.index_handle = mid.index_handle
WHERE migs.avg_total_user_cost * (migs.avg_user_impact / 100.0) * (migs.user_seeks + migs.user_scans) > 10
ORDER BY migs.avg_total_user_cost * migs.avg_user_impact * (migs.user_seeks + migs.user_scans) DESC

37. Solve the Case! Solution may change over time
Tables grow
Statistics out of date
Parameter Sniffing
Some problems result from multiple issues
Do least disruptive changes first:
Add an index
Close open connections

38. Questions? Thank you for attending! Further questions: