1. Comprehensive Indexing via Automated Execution Plan Analysis (ExecStats) Joe Chang jchang6 @ yahoo www.qdpma.com Slide deck here

2. About Joe SQL Server consultant since 1999 Query Optimizer execution plan cost formulas (2002) True cost structure of SQL plan operations (2003?) Database with distribution statistics only, – no data 2004 Decoding statblob-stats_stream – writing your own statistics Disk IO cost structure Tools for system monitoring, execution plan analysis See ExecStats http://www.qdpma.com/ExecStats/SQLExecStats.htmlhttp://www.qdpma.com/ExecStats/SQLExecStats.html Download: http://www.qdpma.com/ExecStatsZip.htmlhttp://www.qdpma.com/ExecStatsZip.html Blog: http://sqlblog.com/blogs/joe_chang/default.aspxhttp://sqlblog.com/blogs/joe_chang/default.aspx

3. Objectives Complexities & depth SQL performance – Cause and Effect Focus on the execution plan – Inefficient plans – missing indexes – very large estimate/actual row discrepancies Comprehensive Index Strategy – few good indexes, but no more than necessary

4. Non-Objective List of rules to be followed blindly without consideration for the underlying reason and whether rule actually applies in the current circumstance DBA skill: cause and effect analysis & assessment

5. Preliminary: Correct Results Normalization – Data stored once, avoid anomalies Unique Keys – Avoid duplicate rows Foreign Keys – Avoid orphaned rows Incorrect architecture requires use of SELECT DISTINCT etc. to correct architecture deficiencies Which may cause performance problems as well Correct action is to address the architecture mistakes before the performance issue.

6. Performance Big Picture Natural keys with unique indexes, not SQL The Execution Plan links all the elements of performance Index tuning alone has limited value Over indexing can cause problems as well Index and Statistics maintenance policy 1 Logic may need more than one execution plan? Compile cost versus execution cost? Tables and SQL combined implement business logic Plan cache bloat? SQL Tables natural keys Indexes Execution Plan Statistics & Compile parameters Compile Row estimate propagation errors Storage Engine Hardware DOP Memory Parallel plans Recompile temp table / table variable Query Optimizer Index & Stats Maintenance API Server Cursors: open, prepare, execute, close? SET NO COUNT Information messages

7. Indexing Principles Good cluster key choice – Grouping + unique, not too wide Good nonclustered indexes – For key queries, not necessarily every query – Covered indexes where practical – Create and drop custom indexes for maintenance ops/special circum. No more indexes than necessary – Update overhead – Compile overhead – May tolerate occasional scans to avoid update maintenance Note emphasis on good, not perfect

8. SQL – Plan – Index Usage dm_exec_query_stats dm_db_index_usage_stats Execution Plan

9. Using DMVs – Execution Plan dm_exec_query_stats dm_exec_sql_text dm_exec_query_plan dm_exec_text_query_plan dm_db_index_usage_stats dm_db_index_operational_stats dm_db_index_physical_stats DBCC SHOW_STATISTICS STATS_DATE (object_id, stats_id) dm_db_stats_properties Execution Plan Indexes, joins Compile parameters System views Indexes, key columns, Include list, filter, XML, Columns store etc. sys.dm_db_stats_properties, is available in SQL Server 2012 starting with Service Pack 1 and in SQL Server 2008 R2 starting with SP2. last_updated, rows, rows_sampled, steps, unfiltered_rows, modification_counter dm_exec_query_profiles 2014 Real time query progress?

10. Performance Analysis Getting Top SQL from dm_exec_query_stats – Manually examining top execution plans Index Reduction – dm_db_index_usage_stats – Drop unused indexes (based on long period) – Consolidating indexes with similar keys – Infrequently used indexes? Must hunt down SQL, possibly low item in query stats Can it use another index?

11. Real World Example July 2014 3.6B rows, 3.4TB, 1.5TB data1.8TB indexes Key tables have 21, 8, 14, 4 and 14 nonclustered indexes

12. Largest Table – 21 (NC) Indexes

13. Index Reduction Nonclustered indexes reduced from 21 to 6 Dec 2014 4.6B rows, 2.3TB, 1.7TB data0.5TB indexes Key tables have 6, 4, 3, 2 and 4 nonclustered indexes

14. Index Reduction + Compression 1 index added, 1 index awaiting removal, possibly 2 Feb 2015 4.9B rows, 0.9TB, 0.55TB data0.34TB indexes Key tables have 7, 5, 3, 2 and 4 nonclustered indexes

15. Compression Notes Very high compression was achieved – Because all keys were 16-byte GUID – Even on dimensions, when natural key would have been 1, 2 or 4 bytes! Core data + indexes – 648GB data230GB indexes w/o compression – 174GB data185GB indexes w/compression Reduction in I/O even with SSD storage far outweigh compression overhead! System memory: 256GB (220) Storage: Violin (NAND Flash)

16. Violin Compression & statistics Index reduction HDD Storage GDC

17. Database view

18. File IO view

19. Table view columns

20. Indexes - continued Number of execution plans that reference the index in Seeks, Scans, Lookups, Insert/Updates and Deletes Literal identifying the execution plans that reference the index in Seeks, Scans, Lookups, Insert/Upd & Deletes

21. Query Execution Stats - 1

22. Query Execution Stats - more

23. Dataspace – Partition Scheme view Partition View

24. Procedure and Functions Columns Dbid, schema, object, object_id, type, Create date, modify date, Number of references (NumRef) (literal) plan reference (from QExec Stats) Caller reference (Functions only)

25. Volumes

28. Slides not used

29. Performance Strategy Tables – support business logic – Normalization, uniqueness etc. SQL – clear SARG, Query optimizer interpretable – 1 Logic maps to X Execution plans Indexes – good cluster key choice – Good nonclustered indexes, no more than necessary Statistics – sample strategy & update frequency Compile parameter strategy Temp table / Table variable strategy: Recompile & Row est. prop. error Parallel execution plans: DOP and CTOP strategy Identity key / alternative: large & small customers

30. Identify (weight) important SQL statements – stored procedure: parameter values & code path Recompile impact for temp tables Execution plan cross references SQL & indexes – Actual plan is better than estimate plan – Compile parameters & skewed statistics Temp tables - Recompile impact Automate Execution Plan analysis to fully cross-reference SQL to index usage

31. SQL & Execution Plan Sources Estimated Execution Plan – dm_exec_query_stats Contents of plan cache + execution statistics – List of stored procedures SELECT name FROM sys.procedures Any SQL list – Plans not in cache, to be generated – Can also execute SQL for actual plans

32. sys.dm_exec_query_stats sql_handle – token for batch or stored procedure statement_start_offset – sql_handle + offset = SQL statement plan_handle – SQL (batch) can have multiple plans on recompile query_hash – identify queries with similar logic, – differing only by literal values

33. sys.procedures Get list of stored procedures in database – functions are called from procedure? Generate estimated execution plan for each – Default parameters Full map of index usage to stored procedure No trigger details in estimated plan

34. SQL List Configuration file has SQL to retrieve SQL list – Can be explicit SQL or stored procedures with parameters – Same procedure, multiple parameter set To expose different code path (actual plan) EXEC proc WITH @P1 RECOMPILE (estimated plan)

35. About ExecStats General information Execution plan sources 1.dm_exec_query_stats 2.list of all stored procedures (estimated) 3.List of SQL in table (estimated or actual plan) 4.Trace file Correlates execution plans to index usage Procedures, functions and triggers Rollup file IO stats by DB, filegroup, disk/vol, data/log Distribution Statistics Output to Excel, sqlplan file, (sql in txt file)

36. ExecStats Output Files Txt – runtime info Log – abbreviated SQL error logs Excel – Missing Indexes DMV SQL plan directory This can be sent to someone who can identify and fix your problem

37. Important Items Query cost – plan efficiency? Recompiles? – Compile parameters – skewed statistics CPU versus Duration (worker – elapsed time) – Disk IO, network transmission, parallel plan? Execution count – network roundtrip? Plan cost – Parallelism – High volume of quick queries is bad, so is excessive DOP Index – current rows, rows at time stats generated, sample rows & date

38. Execution Plans estimate - actual Actual: estimated cost, actual rows, DOP – Compile parameters – Actual rows/executions versus estimated Execute stored procedure once for each possible code path – with appropriate parameters

39. Execution Plans Analysis Predicate – index key columns does not matching full SARG – SQL has function on SARG, data type mismatch – Compile parameters & statistics – Actual and Estimated rows/execution mismatch – Large table scans: how many rows output? – Rebinds and Rewinds – key lookup – Parallelism

40. Execution Plans Pay attention to: – Compile parameters – Large table scans: how many rows output? – Predicate search condition without suitable index – Rebinds and Rewinds – key lookup – Parallelism

41. Index Usage – missing IX, excess IX? Index usage – seek, scan, lookup & update – Unused indexes (infrequent code?) can be dropped – Infrequent usage: check plan references – Similar indexes (leading keys) Same keys, different order Check plan reference – consolidate if possible Scans to large tables or even nonclustered IX – Is it real (SELECT TOP 1 may not be a real scan) Lookups – can these be reduced?

43. SQL Server Skills & Roles Developers SQL code Architect Table structure, unique keys Data Architect normalization DBA Index + Statistics Maintenance Hardware & Storage Performance.

44. SQL Server Performance History Before DMVs (SQL Server 2000) – Profiler/Trace to get top SQL – Execution plans – not really exportable – Which indexes are actually used? Today – Trace/Extended Events sometimes not necessary If the dm_exec_query_stats content is good – Execution plans are exportable – Index Usage Stats

45. How much can be automated? Data collectionall, of course – Top resource consumers, etc. Assessment sometimes – Is there a problem – Can it be fixed or improved Fix/Changesometimes – Indexes – SQL – sometimes – Table structure, architectureno If problems could be solved by pushing a button, what would be the skill requirements to be a DBA? Great accomplishments – 99% perspiration 1% inspiration

46. Performance Approaches Check against list of “Best Practices” Manual DMV scripts approach – Find Top 5 or 10 SQL – Fix it if/when there is a problem All Indexes and procedures/SQL – Examine the complete set of stored procedures – Or the full list of SQL statements – Good indexes for all SQL, no more indexes than

47. Why bother when there are no problems? No problems for over 1 year – Never bothered to collect performance baseline Problem Today – Find it with DMV, fix it – the problem was xxx – but why did it occur today & not before? Probably statistics or compile parameters, but prove it? Why ExecStats – SQL scripts? – too much manual work – Third party tools? – only find problem

48. Rigorous Optimization Table structure, SQL, Client-side Cluster Key Good (nonclustered) Indexes – All indexes are actually used No more indexes than necessary – Consolidate similar indexes same keys, same order, or reverse order? – What SQL is impacted? Statistics update Index maintenance Must consider the full set of SQL/procedures in removing indexes?

49. SQL versus programming languages SQL – great for data access – Not good for everything else – When SQL becomes horribly complicated – What would the code looks like in VB/Java/Cxx Client-side program C#

50. Performance Information Server, Storage OS & SQL Server Settings SQL Server – SQL, query execution statistics, execution plan – Compile parameters – Indexes and index usage statistics – Statistics sampling – when? percentage? skew?