Blazing-Fast Performance: A Technical Best Practices Tour with ColumnStore Index Susan Price Senior Program Manager

Is this your data warehouse experience? Waiting ….

Columnstore indexes Waiting ….

Agenda Benefits Demo Overview How to use ColumnStore Indexes Batch mode query execution How to use ColumnStore Indexes Experiences with customer data Best practices

BenEfits of columnstore indexes Faster, interactive query response time Easier data exploration Better decision Reduced physical DB design effort Fewer indexes Reduced need for summary aggregates and indexed views May eliminate need for OLAP cubes Transparent to the application Less hardware needed for given workload

Demo

What is a columnstore index? … C1 C2 C3 C4 Stores data column-wise Each page stores data from a single column Highly compressed About 2x better than PAGE compression More data fits in memory Each column can be accessed independently Fetch only needed columns Can dramatically reduce IO

Batch Mode Query Execution But Wait, there’s more Would you rather process your data like this … Batch Mode Query Execution Vector-oriented processing Compact data representation Highly efficient algorithms Better parallelism … or like this?

Easy to use No need to rewrite application Create a ColumnStore Index after you create and load the table CREATE NONCLUSTERED COLUMNSTORE INDEX ncci ON myTable(OrderDate, ProductID, SaleAmount) The optimizer makes a cost-based decision When to use the ColumnStore Index When to use batch mode processing Memory management is automatic New object cache for ColumnStore structures New memory broker manages memory allocation Best performance when data is in memory Data not required to fit in memory Options for loading data Use partition switching Build ColumnStore Index on staging table Disable ColumnStore Index, load data, rebuild index

Observed compression ratios Data Set Uncompressed Table Size (MB) ColumnStore Index Size Compression Ratio Cosmetics 1,302 88.5 14.7 SQL 1,431 166 8.6 Xbox 1,045 202 5.2 MS Sales 642,000 126,000 5.1 Web Analytics 2,560 553 4.6 Telecom 2,905 727 4.0 1.8X better compression than SQL Server PAGE compression

real customer experiences Customer Type Industry segment/ Application Measure Without ColumnStore Index (sec) With ColumnStore Index Improvement External Online services Query 1020 3 340x Retail 1080 63 17.1x Healthcare Set of 6 Queries 73894 12782 5.8x Internal HR reporting Avg. response time on production system 220 66 3.3x Financial reporting 3 Queries Each > 50x Queries taking longer than 10 min 90% reduction

Best Practices: When to build a Columnstore index Workload Read-mostly Mostly append new data Star join queries Queries to scan and aggregate large data volumes Workflow Permits partitioning (or disable/rebuild) to handle new data Typically a nightly load window Data selection Build ColumnStore Indexes on large fact tables Consider ColumnStore Indexes for large dimension tables

Best Practices: building a Columnstore index Include all columns in the columnstore index Don’t use to seek into the row Order of listed columns not important Convert decimal/numeric to precision <= 18 If possible Use integer types when possible More compact representation More opportunity for early filtering Ensure sufficient memory to build the columnstore index Create the columnstore index from a clustered index Better segment elimination if filter is on a key column

Best Practices: Writing queries Consider modifying queries to hit “sweet spot” Star joins Inner joins Aggregations Keep statistics up to date Remember that ColumnStore Index is not sorted Consider using hints if needed Pay attention to fundamentals No Magic Bullet Query performance still affected by Schema design Query design Cardinality estimates

Columnstore indexes in a nutshell ColumnStore technology + Advanced query processing Astonishing speedup for DW queries Great compression

Thank You! ColumnStore Indexes white paper Resources ColumnStore Indexes white paper http://download.microsoft.com/download/8/C/1/8C1CE06B-DE2F-40D1-9C5C-3EE521C25CE9/Columnstore%20Indexes%20for%20Fast%20DW%20QP%20SQL%20Server%2011.pdf ColumnStore Index FAQs (search for technet columnstore) http://social.technet.microsoft.com/wiki/contents/articles/sql-server-columnstore-index-faq.aspx http://social.technet.microsoft.com/wiki/contents/articles/sql-server-columnstore-performance-tuning.aspx ColumnStore Index Blog http://blogs.technet.com/b/dataplatforminsider/archive/2011/08/04/columnstore-indexes-a-new-feature-in-sql-server-known-as-project-apollo.aspx (with link to YouTube video) http://sqlapollo.com YouTube videos http://www.youtube.com/watch?v=C2LY_FhMnfc http://www.youtube.com/watch?v=ySOXC18bg3I

Further. Forward. Faster. Learn more Visit the Microsoft Virtual Academy Free technical SQL Server training courses Get more Download SQL Server 2012 Trial Software 180-day evaluation of SQL Server 2012 Do more Participate in virtual launch activities and collect points for prizes The more points you earn, the bigger your prize could be