1. Statistics That Need Special Attention Joe Chang www.qdpma.com Jchang6 @ yahoo http://www.qdpma.com/CBO/Statistics.html

2. About Joe SQL Server consultant since 1999 Query Optimizer execution plan cost formulas (2002) True cost structure of SQL plan operations (2003?) Database with distribution statistics only, no data 2004 Decoding statblob/stats_stream – writing your own statistics Disk IO cost structure Tools for system monitoring, execution plan analysis See http://www.qdpma.com/http://www.qdpma.com/ Download: http://www.qdpma.com/ExecStatsZip.htmlhttp://www.qdpma.com/ExecStatsZip.html Blog: http://sqlblog.com/blogs/joe_chang/default.aspxhttp://sqlblog.com/blogs/joe_chang/default.aspx

3. Statistics – Special Attention What works automatically? – No need for special attention What does not work by default – or simple adjustments, trace flags etc. – What could cause spectacular failures Ok, now what do we do? – scheduled jobs, triggered jobs, in-procedure, optimize for “fake data” Other?

4. Topics Quick Statistics Overview – Details available elsewhere Weak point of SQL Server Statistics – Recompute set points – Sampling – Compile parameter etc. Options – Trace 2371, 4136 Workarounds

5. SQL Performance Natural keys with unique indexes, not SQL The Execution Plan links all the elements of performance Index tuning alone has limited value Over indexing can cause problems as well Index and Statistics maintenance policy 1 Logic may need more than one execution plan? Compile cost versus execution cost? Tables and SQL combined implement business logic Plan cache bloat? SQL Tables natural keys Indexes Execution Plan Statistics & Compile parameters Compile Row estimate propagation errors Storage Engine Hardware DOP Memory Parallel plans Recompile temp table / table variable Query Optimizer Index & Stats Maintenance API Server Cursors: open, prepare, execute, close? SET NO COUNT Information messages

6. Factors to Consider SQLTablesIndexes Query Optimizer Statistics Compile Parameters Storage Engine Hardware DOP memory

7. STATISTICS

8. SQL Server Statistics Principles: it should just work Statistics are automatically – created and updated Indexes and columns Statistics Used by the Query Optimizer in Microsoft SQL Server 2008 http://technet.microsoft.com/en-us/library/dd535534(v=sql.100).aspx Eric N. Hanson and Yavor Angelov Contributor Lubor Kollar

9. DBCC SHOW_STATISTICS DBCC SHOW_STATISTICS('LINEITEM',L_SHIPDATE_CLUIDX) header Density vector Histogram Options: WITH STAT_HEADER, DENSITY VECTOR, HISTOGRAM, STATS_STREAM binary

10. Density Vector Statistics binary storage structure allows for 30 rows in density vector Full nonclustered index key consists of explicitly declared columns plus cluster key columns (not in NC key) Limit of 15 columns in clustered index key And 15 columns in nonclustered index key

11. Histogram Up to 200 steps Captures low and high bounds Attempts to capture skewed distribution – frequent values equal rows, others in range rows

12. UPDATE STATISTICS – STATS_STREAM UPDATE STATISTICS LINEITEM(L_SHIPDATE_CLUIDX) WITH STATS_STREAM = 0x010000000300000000000000000000000792A163000000009A128038F3..., ROWCOUNT = 59986052, PAGECOUNT = 574971 DBCC SHOW_STATISTICS('LINEITEM',L_SHIPDATE_CLUIDX) WITH STATS_STREAM

13. Statistics Structure Stored (mostly) in binary field Scalar values Density Vector – limit 30, half in NC, half Cluster key Histogram Up to 200 steps Consider not blindly using IDENTITY on critical tables Example: Large customers get low ID values Small customers get high ID values http://sqlblog.com/blogs/joe_chang/archive/2012/05/05/decoding-stats-stream.aspx

14. Statistics Auto - re-compute at 20% (first 6, 500 rows) Sampling strategy – How much to sample - theory? – Random pages versus random rows – Histogram Equal and Range Rows – Out of bounds, value does not exist – etc. Statistics Used by the Query Optimizer in SQL Server 2008 Eric N. Hanson and Yavor Angelov, Contributor: Lubor Kollar Optimizing Your Query Plans with the SQL Server 2014 Cardinality Estimator Joseph Sack http://msdn.microsoft.com/en-us/library/dd535534.aspx

15. System tables, views, functions ;WITH k AS ( SELECT k.object_id, k.stats_id, k.stats_column_id, c.column_id, c.name FROM sys.stats_columns k INNER JOIN sys.columns c ON c.object_id = k.object_id AND c.column_id = k.column_id ) SELECT s.name, o.name, d.name, d.stats_id, d.auto_created, ISNULL(STUFF(( SELECT ', ' + name FROM k WHERE k.object_id = d.object_id AND k.stats_id = d.stats_id ORDER BY k.stats_column_id FOR XML PATH(''), TYPE, ROOT).value('root[1]','nvarchar(max)'),1,1,''),'') as Keys, p.rows, p.rows_sampled, p.steps, p.modification_counter, p.last_updated FROM sys.objects o JOIN sys.schemas s ON s.schema_id = o.schema_id JOIN sys.stats d ON d.object_id = o.object_id OUTER APPLY sys.dm_db_stats_properties(d.object_id, d.stats_id) p WHERE o.is_ms_shipped = 0 ORDER BY s.name, o.name, d.stats_id sys.stats sys.stats_columns sys.dm_db_stats_properties (2008R2 sp2, 2012 sp1, 2014 RTM)

16. ;WITH b AS ( SELECT d.object_id, d.index_id, part = COUNT(*), reserved = 8*SUM(d.reserved_page_count), used = 8*SUM(d.used_page_count ), in_row_data = 8*SUM(d.in_row_data_page_count), lob_used = 8*SUM(d.lob_used_page_count), overflow = 8*SUM( d.row_overflow_used_page_count), row_count = SUM(row_count), notcompressed = SUM(CASE data_compression WHEN 0 THEN 1 ELSE 0 END), compressed = SUM(CASE data_compression WHEN 0 THEN 0 ELSE 1 END) -- change to 0 for SQL Server 2005 FROM sys.dm_db_partition_stats d WITH(NOLOCK) INNER JOIN sys.partitions r WITH(NOLOCK) ON r.partition_id = d.partition_id GROUP BY d.object_id, d.index_id ), j AS ( SELECT j.object_id, j.index_id, j.key_ordinal, c.column_id, c.name,is_descending_key FROM sys.index_columns j INNER JOIN sys.columns c ON c.object_id = j.object_id AND c.column_id = j.column_id ) SELECT t.name, o.name, ISNULL(i.name, '') [index], ISNULL(STUFF(( SELECT ', ' + name + CASE is_descending_key WHEN 1 THEN '-' ELSE '' END FROM j WHERE j.object_id = i.object_id AND j.index_id = i.index_id AND j.key_ordinal >0 ORDER BY j.key_ordinal FOR XML PATH(''), TYPE, ROOT).value('root[1]','nvarchar(max)'),1,1,''),'') as Keys, ISNULL(STUFF(( SELECT ', ' + name FROM j WHERE j.object_id = i.object_id AND j.index_id = i.index_id AND j.key_ordinal = 0 ORDER BY j.column_id FOR XML PATH(''), TYPE, ROOT).value('root[1]','nvarchar(max)'),1,1,''),'') as Incl, i.index_id, CASE WHEN i.is_primary_key = 1 THEN 'PK' WHEN i.is_unique_constraint = 1 THEN 'UC' WHEN i.is_unique = 1 THEN 'U' WHEN i.type = 0 THEN 'heap' WHEN i.type = 3 THEN 'X' WHEN i.type = 4 THEN 'S' ELSE CONVERT(char, i.type) END typ, i.data_space_id dsi, b.row_count, b.in_row_data in_row, b.overflow ovf, b.lob_used lob, b.reserved - b.in_row_data - b.overflow -b.lob_used unu, 'ABR' = CASE row_count WHEN 0 THEN 0 ELSE 1024*used/row_count END, y.user_seeks, y.user_scans u_scan, y.user_lookups u_look, y.user_updates u_upd, b.notcompressed ncm, b.compressed cmp, rw_delta = b.row_count - s.rows, s.rows_sampled --, s.unfiltered_rows, s.modification_counter mod_ctr, s.steps, CONVERT(varchar, s.last_updated,120) updated, i.is_disabled dis, i.is_hypothetical hyp, ISNULL(i.filter_definition, '') filt FROM sys.objects o JOIN sys.schemas t ON t.schema_id = o.schema_id JOIN sys.indexes i ON i.object_id = o.object_id LEFT JOIN b ON b.object_id = i.object_id AND b.index_id = i.index_id LEFT JOIN sys.dm_db_index_usage_stats y ON y.object_id = i.object_id AND y.index_id = i.index_id AND y.database_id = DB_ID() OUTER APPLY sys.dm_db_stats_properties(i.object_id, i.index_id) s WHERE o.is_ms_shipped = 0

18. Statistics Auto/Re-Compute Automatically generated on query compile Recompute at 6 rows, 500, every 20%? Has this changed? 2008 R2 Trace 2371 – lower threshold auto recomputed for large tables http://support.microsoft.com/kb/2754171 http://www.sqlskills.com/blogs/erin/understanding-when-statistics-will-automatically-update/ Understanding When Statistics Will Automatically Update

19. Statistics Sampling Sampling theory – True random sample – Sample error - square root N Relative error 1/ N SQL Server sampling – Random pages But always first and last page??? – All rows in selected pages

20. Query Optimizer Assumptions Statistics has captured lower and upper bounds If there are no range rows between range hi keys, then it knows there are no rows in between The page sampling method compensates for lack of true random row sample

21. Hypothesis If we insert/update rows, then dm_db_stats_properties modification_counter > 0 Should not we assume that there is new data?

22. Row Estimate Problems (at source) Skewed data distribution – Does query compile with low, medium or high skew parameter value? – What about subsequent executes? Errors – due to random page sample Out of bounds Value does not exist Row estimate errors at source – is classified under statistics topic

23. Problems in Statistics Scenario – update all statistics with default Insert rows, new (int / identity) values that are higher than upper bound at time of stats if plan compiles with previously existing value – Then fine If plan compiles with new value, – Then optimizer knows value “does not exist” – But plan will show 1 row

24. SELECT * FROM PART WHERE P_PARTKEY = 1999999 SELECT * FROM PART WHERE P_PARTKEY = 2000001

25. SELECT * FROM LINEITEM WHERE L_PARTKEY = 2000001

26. SELECT * FROM PART JOIN LINEITEM ON L_PARTKEY = P_PARTKEY WHERE P_PARTKEY = 2000001 SELECT * FROM PART JOIN LINEITEM ON L_PARTKEY = P_PARTKEY WHERE P_PARTKEY = 1999999

27. Loop Join - Table Scan on Inner Source Estimated out from first 2 tabes (at right) is zero or 1 rows. Most efficient join to third table (without index on join column) is a loop join with scan. If row count is 2 or more, then a fullscan is performed for each row from outer source Default statistics rules may lead to serious ETL issues Consider custom strategy

28. Limited distinct values Column has less than 200 distinct values – Statistics via histogram knows the exact values – i.e., it also knows what values do not exist Now insert rows with different values – If >= 20%, then statistics update is triggered – Next query w/explicit SARG on new value good Ex WHERE bIsProcessed = 0 – If too few row to trigger stats, then bad news

29. Compile Parameter Not Exists Main procedure has cursor around view_Servers First server in view_Servers is ’CAESIUM’ Cursor executes sub-procedure for each Server sql: SELECT MAX(ID) FROM TReplWS WHERE Hostname = @ServerName But CAESIUM does not exist in TReplWS!

30. Good and Bad Plan?

31. SqlPlan Compile Parameters

32. <StmtSimple StatementText="@ServerName varchar(50) SELECT @maxid = ISNULL(MAX(id),0) FROM TReplWS WHERE Hostname = @ServerName" StatementId="1" StatementCompId="43" StatementType="SELECT" StatementSubTreeCost="0.0032843" StatementEstRows="1" StatementOptmLevel="FULL" QueryHash="0x671D2B3E17E538F1" QueryPlanHash="0xEB64FB22C47E1CF2" StatementOptmEarlyAbortReason="GoodEnoughPlanFound"> <StatementSetOptions QUOTED_IDENTIFIER="true" ARITHABORT="false" CONCAT_NULL_YIELDS_NULL="true" ANSI_NULLS="true" ANSI_PADDING="true" ANSI_WARNINGS="true" NUMERIC_ROUNDABORT="false" /> <RelOp NodeId="0" PhysicalOp="Compute Scalar" LogicalOp="Compute Scalar" EstimateRows="1" EstimateIO="0" EstimateCPU="1e-007“ AvgRowSize="15" EstimatedTotalSubtreeCost="0.0032843" Parallel="0" EstimateRebinds="0" EstimateRewinds="0"> Compile parameter values at bottom of sqlplan file

33. Microsoft Responds (Empire Strikes Back) http://blogs.msdn.com/b/saponsqlserver/archive/2011/09/07/changes-to- automatic-update-statistics-in-sql-server-traceflag-2371.aspx Changes to automatic update statistics in SQL Server – traceflag 2371 Trace flag 4136 http://blogs.msdn.com/b/axinthefield/archive/2010/11/04/sql-server-trace- flags-for-dynamics-ax.aspx QL Server Trace Flags for Dynamics AX Disables use of the histogram

34. 2371 http://blogs.msdn.com/b/saponsqlserver/archive/2011/09/07/changes-to-automatic-update-statistics-in-sql-server-traceflag-2371.aspx

35. 2014 The New and Improved Cardinality Estimator in SQL Server 2014 http://blogs.technet.com/b/dataplatforminsider/archive/2014/03/17/th e-new-and-improved-cardinality-estimator-in-sql-server-2014.aspx http://blogs.technet.com/b/dataplatforminsider/archive/2014/03/17/th e-new-and-improved-cardinality-estimator-in-sql-server-2014.aspx Filtered Stats and CE Model Variation http://www.queryprocessor.com/ce_filteredstats/ http://www.somewheresomehow.ru/ Dima Puligin

36. Upper Bound Problem Insert dummy row with maximum value – SET IDENTITY_INSERT table ON – int 2,147,483,647, datetime Dec 31, 9999 OPTION (OPTIMIZE FOR (@P1 UNKNOWN)) SQL Server 2014 – not a problem

40. Temp Table and Table Variable Forget what other people have said – Most is cr@p Temp Tables – subject to statistics auto/re-compile Table variable – no statistics, assumes 1 row Question: In each specific case: does the statistics and recompile help or not? – Yes: temp table – No: table variable Is this still true?

41. Multiple joins to one table SELECT xxx FROM Cession.RetroTransaction rt INNER JOIN Common.Code c1 ON c1.CodeId = rt.TransactionTypeId INNER JOIN Cession.NARSplit ns ON ns.NARSplitId = rt.NARSplitId AND ns.CessionId = rt.CessionId INNER JOIN CRM.Company comp ON comp.CompanyId = ns.CompanyId INNER JOIN Cession.RetroTransactionPeriod rtp ON rt.RetroTransactionPeriodId = rtp.RetroTransactionPeriodId LEFT OUTER JOIN Cession.RetroTransactionAllocation BasePremium WITH ( NOLOCK ) ON BasePremium.RetroTransactionId = rt.RetroTransactionId AND BasePremium.AllocationTypeId = '02C48EDA-57ED-4704-99CD-6122AE372B3D' AND BasePremium.AllocationGrossNetTypeId = '09E95F88-4546-49B9-B4FB-A068CDBDBFDF' LEFT OUTER JOIN Cession.RetroTransactionAllocation BaseAllowance WITH ( NOLOCK ) ON BaseAllowance.RetroTransactionId = rt.RetroTransactionId AND BaseAllowance.AllocationTypeId = 'CA5047A3-43CD-496B-8F71-3E95E209806E' LEFT OUTER JOIN Cession.RetroTransactionAllocation PolicyFee WITH ( NOLOCK ) ON PolicyFee.RetroTransactionId = rt.RetroTransactionId AND PolicyFee.AllocationTypeId = 'FCD31932-840F-4BCA-99DC-F55BD46E46AF' AND PolicyFee.AllocationGrossNetTypeId = '09E95F88-4546-49B9-B4FB-A068CDBDBFDF' LEFT OUTER JOIN Cession.RetroTransactionAllocation PolicyFeeAllowance ON PolicyFeeAllowance.RetroTransactionId = rt.RetroTransactionId AND PolicyFeeAllowance.AllocationTypeId = 'E570DA71-6B6C-4742-AB8B-DDFDB5286938' LEFT OUTER JOIN Cession.RetroTransactionAllocation TempFlatExtra WITH ( NOLOCK ) ON TempFlatExtra.RetroTransactionId = rt.RetroTransactionId AND TempFlatExtra.AllocationTypeId = '82CB2D45-904F-4B50-8C36-A39CD2F33D9D' AND TempFlatExtra.AllocationGrossNetTypeId = '09E95F88-4546-49B9-B4FB-A068CDBDBFDF' LEFT OUTER JOIN Cession.RetroTransactionAllocation TempFlatExtraAllowance ON TempFlatExtraAllowance.RetroTransactionId = rt.RetroTransactionId AND TempFlatExtraAllowance.AllocationTypeId = '43E49878-9AC3-4A1F-A13F-19A1BA7D3AFA' LEFT OUTER JOIN Cession.RetroTransactionAllocation PermFlatExtra WITH ( NOLOCK ) ON PermFlatExtra.RetroTransactionId = rt.RetroTransactionId AND PermFlatExtra.AllocationTypeId = '889639AB-620C-4234-9B9E-2970A4500B57' AND PermFlatExtra.AllocationGrossNetTypeId = '09E95F88-4546-49B9-B4FB-A068CDBDBFDF' LEFT OUTER JOIN Cession.RetroTransactionAllocation PermFlatExtraAllowance ON PermFlatExtraAllowance.RetroTransactionId = rt.RetroTransactionId AND PermFlatExtraAllowance.AllocationTypeId = '43DFA681-44FB-4025-B75A-1E12498A679B' LEFT OUTER JOIN Cession.RetroTransactionAllocation CashSurrender WITH ( NOLOCK ) ON CashSurrender.RetroTransactionId = rt.RetroTransactionId AND CashSurrender.AllocationTypeId = 'BCBDF86C-DCC5-49DA-BD2E-CD42CE402E99' LEFT OUTER JOIN Common.Code splittype ON splittype.CodeId = ns.SplitTypeId WHERE rt.CessionId = N'86037aac-465a-4e47-8bf1-6f8868852af5'

42. <ShowPlanXML xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" Version="1.1" Build="10.50.2500.0" xmlns="http://schemas.microsoft.com/sqlserver/2004/07/showplan"> <StmtSimple StatementCompId="1“ StatementEstRows="5004190000“ StatementId="1" StatementOptmLevel="FULL“ StatementOptmEarlyAbortReason="TimeOut" StatementSubTreeCost="44029" StatementText="SELECT rt.RetroTransactionId, FROM Cession.vwRetroTransaction AS rt WHERE rt.CessionId = N'86037aac-465a-4e47-8bf1-6f8868852af5'" StatementType="SELECT" ParameterizedText="(@1 nvarchar(4000)) " QueryHash="0x19274C9794934C69" QueryPlanHash="0x80E6A4E14DF1DC0A"> <RelOp AvgRowSize="365" EstimateCPU="133.341" EstimateIO="0" EstimateRebinds="0" EstimateRewinds="0“ EstimateRows="5004190000" LogicalOp="Gather Streams" NodeId="0" Parallel="true" PhysicalOp="Parallelism“ EstimatedTotalSubtreeCost="44029" >

43. <ShowPlanXML xmlns:xsi=http://www.w3.org/2001/XMLSchema-instancehttp://www.w3.org/2001/XMLSchema-instance xmlns:xsd="http://www.w3.org/2001/XMLSchema" Version="1.1" Build="10.50.2500.0“ xmlns="http://schemas.microsoft.com/sqlserver/2004/07/showplan"> <StmtSimple StatementCompId="1“ StatementEstRows="24.2411“ StatementId="1" StatementOptmLevel="FULL“ StatementOptmEarlyAbortReason="TimeOut" StatementSubTreeCost="0.683036“ StatementText="SELECT rt.RetroTransactionId FROM Cession.vwRetroTransaction rt WHERE rt.CessionId=@1“ StatementType="SELECT" QueryHash="0x19274C9794934C69" QueryPlanHash="0xD5FF47FBFB2AD6AE">

44. plan