Columnstore Indexing: From SQL Server 2012 to SQL Server 2014

Slides:



Advertisements
Similar presentations
Tuning: overview Rewrite SQL (Leccotech)Leccotech Create Index Redefine Main memory structures (SGA in Oracle) Change the Block Size Materialized Views,
Advertisements

Dos and don’ts of Columnstore indexes The basis of xVelocity in-memory technology What’s it all about The compression methods (RLE / Dictionary encoding)
Project Management Database and SQL Server Katmai New Features Qingsong Yao
Module 6 Implementing Table Structures in SQL Server ®2008 R2.
CS 345: Topics in Data Warehousing Tuesday, October 19, 2004.
Physical Database Design & Performance. Optimizing for Query Performance For DBs with high retrieval traffic as compared to maintenance traffic, optimizing.
Chapter 6 1 © Prentice Hall, 2002 The Physical Design Stage of SDLC (figures 2.4, 2.5 revisited) Project Identification and Selection Project Initiation.
Module 5 Planning for SQL Server® 2008 R2 Indexing.
Indexes / Session 2/ 1 of 36 Session 2 Module 3: Types of Indexes Module 4: Maintaining Indexes.
Data Types Lesson 4. Skills Matrix Table A table stores your data. Tables are relational in that they are organized as rows and columns (a matrix). Each.
SQL Server 2005 Implementation and Maintenance Chapter 3: Tables and Views.
SQL/Lesson 7/Slide 1 of 32 Implementing Indexes Objectives In this lesson, you will learn to: * Create a clustered index * Create a nonclustered index.
Chapter 4 Indexes. Index Architecture  By default data is inserted on a first-come, first-serve basis  Indexes bring order to this chaos  Once you.
Physical Database Design Purpose- translate the logical description of data into the technical specifications for storing and retrieving data Goal - create.
Chapter 5 Index and Clustering
Session 1 Module 1: Introduction to Data Integrity
INTRODUCING SQL SERVER 2012 COLUMNSTORE INDEXES Exploring and Managing SQL Server 2012 Database Engine Improvements.
2012 © Trivadis BASEL BERN LAUSANNE ZÜRICH DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MÜNCHEN STUTTGART WIEN Welcome November 2012 Columnstore Indexes.
October 15-18, 2013 Charlotte, NC Accelerating Database Performance Using Compression Joseph D’Antoni, Solutions Architect Anexinet.
--A Gem of SQL Server 2012, particularly for Data Warehousing-- Present By Steven Wang.
How to kill SQL Server Performance Håkan Winther.
SQLUG.be Case study: Redesign CDR archiving on SQL Server 2012 By Ludo Bernaerts April 16,2012.
Diving into Query Execution Plans ED POLLACK AUTOTASK CORPORATION DATABASE OPTIMIZATION ENGINEER.
Views / Session 3/ 1 of 40 Session 3 Module 5: Implementing Views Module 6: Managing Views.
Doing fast! Optimizing Query performance with ColumnStore Indexes in SQL Server 2012 Margarita Naumova | SQL Master Academy.
Best Practices for Columnstore Indexes Warner Chaves SQL MCM / MVP SQLTurbo.com Pythian.com.
Data Integrity & Indexes / Session 1/ 1 of 37 Session 1 Module 1: Introduction to Data Integrity Module 2: Introduction to Indexes.
SQL IMPLEMENTATION & ADMINISTRATION Indexing & Views.
Introduction to columnstore indexes Taras Bobrovytskyi SQL wincor nixdorf.
Creating Database Objects
Tuning Transact-SQL Queries
Managing Tables, Data Integrity, Constraints by Adrienne Watt
Power BI Performance Tips & Tricks
Indexes By Adrienne Watt.
Indexing Structures for Files and Physical Database Design
Module 2: Creating Data Types and Tables
SQL Implementation & Administration
Informatica PowerCenter Performance Tuning Tips
Physical Database Design for Relational Databases Step 3 – Step 8
Software Architecture in Practice
Presented by: Warren Sifre
Database Performance Tuning and Query Optimization
Four Rules For Columnstore Query Performance
Blazing-Fast Performance:
Power BI Performance …Tips and Techniques.
Migrating a Disk-based Table to a Memory-optimized one in SQL Server
ColumnStore Index Primer
Azure SQL Data Warehouse Performance Tuning
Introduction to columnstore indexes
Statistics: What are they and How do I use them
Steve Hood SimpleSQLServer.com
Practical Database Design and Tuning
Azure SQL DWH: Optimization
Microsoft SQL Server 2014 for Oracle DBAs Module 7
The Five Ws of Columnstore Indexes
The Physical Design Stage of SDLC (figures 2.4, 2.5 revisited)
Sunil Agarwal | Principal Program Manager
Four Rules For Columnstore Query Performance
Chapter 11 Database Performance Tuning and Query Optimization
Diving into Query Execution Plans
A – Pre Join Indexes.
Introduction to Execution Plans
Using Columnstore indexes in Azure DevOps Services. Lessons learned
Creating Database Objects
Using Columnstore indexes in Azure DevOps Services. Lessons learned
Introduction to Execution Plans
Reinhard Flügel Possiblities and Limitations of System-Versioned Temporal Tables beyond the Basics.
SQL Server Columnar Storage
Reinhard Flügel Possiblities and Limitations of System-Versioned Temporal Tables beyond the Basics.
Reinhard Flügel Possiblities and Limitations of System-Versioned Temporal Tables beyond the Basics.
Presentation transcript:

Columnstore Indexing: From SQL Server 2012 to SQL Server 2014 Presented by: Sifiso Ndlovu

A Tale of Two Queries TSQL Script Without Columnstore Index TSQL Script That Uses Columnstore Index

Speaker BIO Sifiso W Ndlovu Senior BI Tech at Karabina Member of Johannesburg SQL User Group Mcom: IT Management with UJ Microsoft Certifications MCSE: SQL Server 2012 BI; Data Platform MCSD: ALM MCSA: SQL Server 2012, SQL Server 2008 MCITP: DBA 2008, BI 2008 MCTS: TFS 2010 Blog/Writing SQLShack.com BIDN.com SQLServerCentral.com Social Network @mafiswana sifiso.w.ndlovu facebook.com/sifiso.w.ndlovu Sifiso W Ndlovu Website www.selectsifiso.net

SQL Saturday Audience: Assumptions Developers DBAs Architects Technical Oriented People Managers Business Analysts Product Owners Business Minded People

SQL Saturday Audience: Assumptions Intermediate Level Clustered Index SQL Server Statistics Understanding Execution Plans Non-Clustered Index Database Diagrams SQL Server Data Storage Data Warehouse Design Principles

Columnstore Index: SQL Server 2012 Agenda Introduction Columnstore Index: SQL Server 2012 What Is a Columnstore Index How to Create Columnstore Index Columnstore Index Internals Advantages to Columnstore Index Usage Limitations to Columnstore Index Usage Tips, Tricks and Best Practice Case Study Columnstore Index: SQL Server 2014

Session Objectives & Takeaway Understand what SQL Server 2012 Columnstore indexes are Understand ways of creating SQL Server 2012 Columnstore Index Understand the Internals of SQL Server 2012 Columnstore Index Understand the advantages of Columnstore indexes in SQL Server 2012 Understand the limitations of Columnstore indexes in SQL Server 2012 Understand Best Practices to Columnstore indexes Implementation in SQL Server 2012 Understand the significant improvements available in SQL Server 2014

Introduction

Introduction http://www.sqlshack.com

Columnstore Index: SQL Server 2012 What Is a Columnstore Index Several Explanations Enterprise feature that is intended to improve the query response on T-SQL queries against Analytical/Data Warehouse environments Data warehouse workloads typically scan, aggregate, and join large amounts of data Introduced in SQL Server 2012 as an alternative to the traditional row-based indexing (in Data Warehouse environments) that has been dominant in SQL Server versions prior to SQL Server 2012

Columnstore Index: SQL Server 2012 What Is a Columnstore Index SQL Server 2008 R2 Express

Columnstore Index: SQL Server 2012 What Is a Columnstore Index Several Explanations Column-based non-clustered index geared toward increasing query performance for workloads that involve large amounts of data, typically found in data warehouse fact tables Column store index stores data in columnar fashion and uses compression aggressively to reduce the disk I/O needed to serve the query request

Columnstore Index: SQL Server 2012 What Is a Columnstore Index Several Explanations The defining characteristic of columnar storage is the ability to read the values of a particular column of a table without having to read the values of all the other columns In row-oriented storage this is impossible because the individual column values are physically stored grouped in rows on the pages and reading a page in order to read a column value must fetch the entire page in memory, thus automatically reading all the other columns in the row

Columnstore Index: SQL Server 2012 What Is a Columnstore Index Several Explanations Columnstores are much more like the row store heaps and supports only end-to-end scans. They do not have a key and do not offer any order of rows. Yet, on large data sets, they can offer exceptional performance for analytical workloads due to the factors mentioned above: read only for the columns referenced by the query, high compression, segment elimination and fast batch mode processing

Columnstore Index: SQL Server 2012 What Is a Columnstore Index Row-based storage

Columnstore Index: SQL Server 2012 What Is a Columnstore Index Column-based storage

Session Objectives & Takeaway Understand what SQL Server 2012 Columnstore indexes are Understand ways of creating SQL Server 2012 Columnstore Index Understand the Internals of SQL Server 2012 Columnstore Index Understand the advantages of Columnstore indexes in SQL Server 2012 Understand the limitations of Columnstore indexes in SQL Server 2012 Understand Best Practices to Columnstore indexes Implementation in SQL Server 2012 Understand the significant improvements available in SQL Server 2014

Questions…

Columnstore Index: SQL Server 2012 How to Create Columnstore Index The T-SQL Syntax

Columnstore Index: SQL Server 2012 How to Create Columnstore Index Create Columnstore Index: GUI SQL Server 2008 R2 SQL Server 2012

Columnstore Index: SQL Server 2012 How to Create Columnstore Index Create Columnstore Index: GUI

Columnstore Index: SQL Server 2012 How to Create Columnstore Index

Columnstore Index: SQL Server 2012 How to Create Columnstore Index Create Columnstore Index: GUI Potential Issue of Creating Index via GUI I tried to create a columnstore index with SQL Server Management Studio using the Indexes->New Index menu and it timed out after 20 minutes. How can I work around this? Run a CREATE NONCLUSTERED COLUMNSTORE INDEX statement manually in a T-SQL window instead of using the graphical interface. This will avoid the timeout imposed by the Management Studio graphical user interface.

Columnstore Index: SQL Server 2012 How to Create Columnstore Index Create Columnstore Index: Script

Columnstore Index: SQL Server 2012 How to Create Columnstore Index Create Columnstore Index: Script CREATE NONCLUSTERED COLUMNSTORE INDEX ix_cs_MyDWTable ON dbo.MyDWTable (col1 , col2 , ... , coln); DROP INDEX dbo.MyDWTable.ix_cs_MyTable; ALTER INDEX dbo.MyDWTable.ix_cs_MyTable DISABLE; ALTER INDEX dbo.MyDWTable.ix_cs_MyTable ENABLE;

Columnstore Index: SQL Server 2012 How to Create Columnstore Index

Columnstore Index: SQL Server 2012 How to Create Columnstore Index Supported Data Types Character Strings Unicode Character Strings Exact Numerics Approximate Numerics Date and Time CHAR NCHAR INT FLOAT DATE VARCHAR – except for MAX values NVARCHAR – except for MAX values BIGINT REAL DATETIME SMALLINT DATETIME2 TINYINT SMALLDATETIME BIT TIME MONEY DATETIMEOFFSET – with a scale less than 2 SMALLMONEY DECIMAL NUMERIC

Columnstore Index: SQL Server 2012 How to Create Columnstore Index

Session Objectives & Takeaway Understand what SQL Server 2012 Columnstore indexes are Understand ways of creating SQL Server 2012 Columnstore Index Understand the Internals of SQL Server 2012 Columnstore Index Understand the advantages of Columnstore indexes in SQL Server 2012 Understand the limitations of Columnstore indexes in SQL Server 2012 Understand Best Practices to Columnstore indexes Implementation in SQL Server 2012 Understand the significant improvements available in SQL Server 2014

Questions…

Columnstore Index: SQL Server 2012 Columnstore Index Internals Segment, Segmentation, and Segment Limitation

Columnstore Index: SQL Server 2012 Columnstore Index Internals Segment, Segmentation, and Segment Limitation Columnstore data is physically stored in one or more segments A segment is a highly compressed Large Object (LOB) that can contain up to one million rows The data within each column’s segment matches row-by-row so that the rows can always be assembled correctly Segment can contain values from one column only, which allows each column’s data to be accessed independently Segments are compressed on disk and remain compressed when cached in memory

Columnstore Index: SQL Server 2012 Columnstore Index Internals Segment, Segmentation, and Segment Limitation Of course, a column’s data won’t always fit into a single segment, given the one-million-row limitation. In such cases, multiple segments are created for each column and grouped into multiple row groups, one for each set of segments A column can span multiple segments, and each segment can be made up of multiple data pages Data is transferred from the disk to memory by segment, not by page

Columnstore Index: SQL Server 2012 Columnstore Index Internals Segment, Segmentation, and Segment Limitation The underlying storage mechanism is still the good old 8K pages Segment elimination refers to the skipping of large chunks of data to speed up scans Each segment has metadata that stores the minimum and maximum value of each column for the segment. The storage engine checks filter conditions against the metadata. If it can detect that no rows will qualify then it skips the entire segment without even reading it from disk

Columnstore Index: SQL Server 2012 Columnstore Index Internals Segment, Segmentation, and Segment Limitation Segment Limitation Row Group Segment

Columnstore Index: SQL Server 2012 Columnstore Index Internals xVelocity, Vertipac, and Compression

Columnstore Index: SQL Server 2012 Columnstore Index Internals xVelocity, Vertipac, and Compression formerly known as Vertipac, was first introduced to the BI analytical line of products: Power Pivot and Tabular Mode for Analysis Services enables large amounts of data to be compressed in-memory reduces the number of disk reads and increases buffer cache hit ratios because only the smaller column-based data pages that need to satisfy a query are moved into memory columnstore indexes are possible by leveraging the xVelocity compression technology

Columnstore Index: SQL Server 2012 Columnstore Index Internals xVelocity, Vertipac, and Compression Because column oriented storage groups values from the same column together there is a new beneficial side effect: data becomes more compressible Compression can be deployed on row-oriented storage too (i.e. Page Compression), however with column oriented storage the data is more homogenous, as it contains only values from a single column

Columnstore Index: SQL Server 2012 Columnstore Index Internals xVelocity, Vertipac, and Compression Since compression ratio is subject to the data entropy (how homogenous the data is) it follows that columnar storage format is more compressible than the same data represented in row oriented storage format Unlike base tables whereby user can specify page or row compression, columnstore index compression cannot be specified by the user

Columnstore Index: SQL Server 2012 Columnstore Index Internals Batch, Batches, and Processing

Columnstore Index: SQL Server 2012 Columnstore Index Internals Batch, Batches, and Processing new query execution engine optimized for BI queries highly efficient, vector-based query execution method A "batch" is an object that contains about 1000 rows. Each column within the batch is represented internally as a vector. Batch processing can reduce CPU consumption 7X to 40X compared to the older, row-based query execution methods. Efficient vector-based algorithms allow this by dramatically reducing the CPU overhead of basic filter, expression evaluation, projection, and join operations

Columnstore Index: SQL Server 2012 Columnstore Index Internals Batch, Batches, and Processing When the query uses at least one columnstore index, batch mode processing can speed up joins, aggregations, and filtering During batch mode processing, columnar data is organized in vectors during query execution Sets of data are processed a-batch-at-a-time instead of a-row-at-a-time, using highly efficient algorithms designed to take advantage of modern hardware The query optimizer takes care of choosing when to use batch mode processing and when to use traditional row mode query processing

Columnstore Index: SQL Server 2012 Columnstore Index Internals Batch, Batches, and Processing

Columnstore Index: SQL Server 2012 Columnstore Index Internals Batch, Batches, and Processing For best performance a query plan has to run in batch mode from top to bottom batch mode execution requires everybody to play the same tune: all operators have to implement the batch mode execution Row mode operators cannot interact with batch mode operators nor the other way around

Columnstore Index: SQL Server 2012 Columnstore Index Internals Batch, Batches, and Processing The Query Optimizer can resort to a conversion operator from batch mode to row mode and it can create plans that have sub trees in batch mode even if the upper part of the tree is in row mode This mixed mode can still yield significant performance improvements, provided that most data processing still occurs in batch mode (eg. aggregation occurs in batch mode and the result of aggregation is consumed in row mode)

Columnstore Index: SQL Server 2012 Columnstore Index Internals Batch, Batches, and Processing List of plan operators support batch mode operation Columnstore Index Scan Hash Inner Join Batch Hash Table Build Bitmap Filter Hash Aggregate (not scalar aggregates) Filter Compute Scalar (for projection and expression evaluation)

Columnstore Index: SQL Server 2012 Columnstore Index Internals Batch, Batches, and Processing Most tricks rely on splitting the query in two, a subquery that does the bulk of the work in batch mode and then project aggregated results into a an outer query that uses some non batch mode friendly operator like OUTER JOIN, NOT IT, IN, EXISTS, Scalar Aggregates or DISTINCT aggregates

Session Objectives & Takeaway Understand what SQL Server 2012 Columnstore indexes are Understand ways of creating SQL Server 2012 Columnstore Index Understand the Internals of SQL Server 2012 Columnstore Index Understand the advantages of Columnstore indexes in SQL Server 2012 Understand the limitations of Columnstore indexes in SQL Server 2012 Understand Best Practices to Columnstore indexes Implementation in SQL Server 2012 Understand the significant improvements available in SQL Server 2014

Questions…

Columnstore Index: SQL Server 2012 Advantages to Columnstore Index Usage

Columnstore Index: SQL Server 2012 Advantages to Columnstore Index Usage Read only for the columns referenced by the query High compression Segment elimination Fast batch mode processing Buffer hit rates are improved because data is highly compressed, and frequently accessed parts of commonly used columns remain in memory, while infrequently used parts are paged out

Session Objectives & Takeaway Understand what SQL Server 2012 Columnstore indexes are Understand ways of creating SQL Server 2012 Columnstore Index Understand the Internals of SQL Server 2012 Columnstore Index Understand the advantages of Columnstore indexes in SQL Server 2012 Understand the limitations of Columnstore indexes in SQL Server 2012 Understand Best Practices to Columnstore indexes Implementation in SQL Server 2012 Understand the significant improvements available in SQL Server 2014

Questions…

Columnstore Index: SQL Server 2012 Limitations to Columnstore Index Usage

Columnstore Index: SQL Server 2012 Limitations to Columnstore Index Usage Restrictions on Column-Store Indexes Columnsstore index is not allowed to be created against a sparse column Columnsstore index is not allowed to have more than 1024 columns Columnsstore index is not allowed to be used as a primary key in a given object Columnsstore index is not allowed to be used as a foreign key in a given object

Columnstore Index: SQL Server 2012 Limitations to Columnstore Index Usage Restrictions on Column-Store Indexes During a create columnstore index statement, you are not allowed to specify sorting keywords (i.e. ASC, DESC) on the columns that are used as part of the columnstore index Columnsstore index is not allowed to be created against a view or indexed view Columnsstore create index statement is not allowed to contain the INCLUDE keyword You are not allowed to modify the columnsstore index using the ALTER INDEX statement

Columnstore Index: SQL Server 2012 Limitations to Columnstore Index Usage Restrictions on Column-Store Indexes A columnstore index can only be created in a table that currently has no data You are restricted to creating a single columnstore index per given object You are not allowed to update the data of an object that uses columnstore indexing Although you are allowed to add columns to a table that uses columnstore index, you are not allowed to drop any of the columns that are referenced in a columnstore index

Columnstore Index: SQL Server 2012 Limitations to Columnstore Index Usage SQL Server 2012 Features that do not support Column-Store Indexes SQL Server Replication Change Tracking Change Data Capture Filestream Compression Page Row SQL Server Vardecimal Storage Format

Columnstore Index: SQL Server 2012 Limitations to Columnstore Index Usage SQL Server 2012 Data Types that do not support Column-Store Indexes Character Strings Unicode Character Strings Exact Numerics Binary Strings Date and Time Other Data Types VARCHAR (MAX) NVARCHAR (MAX) DECIMAL - with precision greater than 18 digits BINARY DATETIMEOFFSET - with scale greater than 2 XML NUMERIC - with precision greater than 18 digits VARBINARY HIERARCHYID IMAGE SPATIALTYPES TIMESTAMP SQLVARIANT UNIQUEIDENTIFIER ROWVERSION

Columnstore Index: SQL Server 2012 Limitations to Columnstore Index Usage

Session Objectives & Takeaway Understand what SQL Server 2012 Columnstore indexes are Understand ways of creating SQL Server 2012 Columnstore Index Understand the Internals of SQL Server 2012 Columnstore Index Understand the advantages of Columnstore indexes in SQL Server 2012 Understand the limitations of Columnstore indexes in SQL Server 2012 Understand Best Practices to Columnstore indexes Implementation in SQL Server 2012 Understand the significant improvements available in SQL Server 2014

Questions…

Columnstore Index: SQL Server 2012 Tips, Tricks and Best Practice

Columnstore Index: SQL Server 2012 Tips, Tricks and Best Practice Comparing with traditional row store indexing strategies, the columnstore indexing rules are straight forward: pick a large table that is usually the fact table in a star schema data warehouse and add a columnstore index that includes every column. Without key order, ASCENDING or DESCENDING clauses, fill factors, filtered indexes, INCLUDE columns or any other of the whistles and bells of row store indexes, the decision is much simpler. Consider the columnstore index as an alternative to the base table

Columnstore Index: SQL Server 2012 Tips, Tricks and Best Practice The key to getting the best performance is to make sure your queries process the large majority of data in batch mode To tell if the main part of your query is running in batch mode, look at the graphical showplan, hover the mouse pointer over the most expensive scan operator (usually a scan of a large fact table) and check the tooltip. It will say whether the estimated and actual execution mode was Row or Batch Columnstore indexes are not (yt) available in SQL Azure

Columnstore Index: SQL Server 2012 Tips, Tricks and Best Practice Avoid use of OUTER JOIN on columnstore-indexed tables. Outer joins don't benefit from batch processing instead, SQL Server 2012 reverts to row-at-a-time processing Avoid use of NOT IN on columnstore-indexed tables. NOT IN (<subquery>) (which internally uses an operator called "anti-semi-join") can prevent batch processing and cause the system to revert to row mode NOT IN (<list of constants>) typically works fine though If you frequently update your dimension tables, and they are not too large, you may find the maintenance effort outweighs the benefit of a columnstore index

Columnstore Index: SQL Server 2012 Tips, Tricks and Best Practice Avoid use of UNION ALL to directly combine columnstore-indexed tables with other tables. Batch processing doesn't get pushed down over UNION ALL. So, for example, creating a view vFact that does a UNION ALL of two tables, one with a columnstore indexes and one without, and then querying vFact in a star join query, will not use batch processing Columnstore indexes are designed to accelerate data warehouse queries, not OLTP workloads.  Use columnstore indexes when your query workload entails scanning and aggregating large amounts of data or joining multiple tables, especially in a star join pattern

Columnstore Index: SQL Server 2012 Tips, Tricks and Best Practice Be sure to populate the table with data before you create the columnstore index Microsoft recommends that columnstore indexes are used for fact tables in datawarehouses, for large dimensions (say with more than 10 millions of records), and any large tables designated to be used as read-only Microsoft recommends that you place columnstore indexes on all of the columns in a given table Disable or drop the columnstore index. You will then be able to update the table and then rebuild the columnstore index

Columnstore Index: SQL Server 2012 Tips, Tricks and Best Practice Use partition switching. If your table is partitioned, you can put new data into a staging table, build a columnstore index on the staging table, and switch the staging table into an empty partition of your main table. Similarly, you could modify existing data by first switching a partition from the main table into a staging table, disable the columnstore index on the staging table, perform your updates, rebuild the columnstore index, and switch the partition back into the main table To build columnstore indexes as fast a possible, make sure to provide plenty of memory

Columnstore Index: SQL Server 2012 Tips, Tricks and Best Practice Creating a columnstore index is a parallel operation, subject to the limitations on the number of CPUs available and any restrictions set on MaxDOP. Creating a columnstore index takes on the order of 1.5 times as long as building a B-tree on the same columns Use best practices for statistics management and query design. This is independent of columnstore technology. Use good statistics and avoid query design pitfalls to get the best performance

Columnstore Index: SQL Server 2012 Tips, Tricks and Best Practice Statistics are not created automatically when you create a columnstore index.  The optimizer will use statistics that exist on the base table, whether those statistics were created automatically or manually.  Hence, you will generally want to create statistics on all the columns you expect to query or leave auto create statistics turned on When you create a columnstore index, a statistics object is created, but the values reported by DBCC SHOW_STATISTICS are all NULL.  The statistics object associated with a columnstore index is used only for database cloning.

Columnstore Index: SQL Server 2012 Tips, Tricks and Best Practice Avoid joins and string filters directly on columns of columnstore-indexed tables. String filters don't get pushed down into scans on columnstore indexes, and join processing on strings is less efficient than on integers. Filters on number and date types are pushed down. Consider using integer codes (or surrogate keys) instead of strings in columnstore indexed fact tables. You can move the string values to a dimension table. Joins on the integer columns normally will be processed very efficiently

Columnstore Index: SQL Server 2012 Tips, Tricks and Best Practice If your table has less than one million rows, SQL Server will use only one thread to create the columnstore index.  Creating the index in parallel requires more memory than creating the index serially.  If your table has more than one million rows, but SQL Server cannot get a large enough memory grant to create the index using MAXDOP, SQL Server will automatically decrease DOP as needed to fit into the available memory grant.  In some cases, DOP must be decreased to one in order to build the index under constrained memory

Columnstore Index: SQL Server 2012 Tips, Tricks and Best Practice The memory required for creating a columnstore index depends on the number of columns, the number of string columns, the degree of parallelism (DOP), and the characteristics of the data. SQL Server will request a memory grant before trying to create the index. If not enough memory is available to create the index in parallel with the current max DOP, SQL Server will reduce the DOP as needed to get an adequate memory grant. If SQL Server cannot get a memory grant to build the index with DOP = 1, the index creation will fail

Columnstore Index: SQL Server 2012 Tips, Tricks and Best Practice A rule of thumb for estimating the memory grant that will be requested for creating a columnstore index is: Memory grant request in MB = [(4.2 *Number of columns in the CS index) + 68]*DOP + (Number of string cols * 34) It's possible for creation of a columnstore index to fail either at the very beginning of execution if it can't get the necessary initial memory grant, or later during execution if supplemental grants can't be obtained If the initial grant fails, you'll see error 8657 or 8658 You may get error 701 or 802 if memory runs out later during execution

Columnstore Index: SQL Server 2012 Tips, Tricks and Best Practice If out-of-memory error 8657 or 8658 occur at the beginning of columnstore index creation, first, check your resource governor settings. The default setting for resource governor limits a query in the default pool to 25% of available memory even if the server is otherwise inactive. This is true even if you have not enabled resource governor.  Consider changing the resource governor settings to allow the create index statement to access more memory

Columnstore Index: SQL Server 2012 Tips, Tricks and Best Practice Another way to deal with out-of-memory conditions during columnstore index build is to vertically partition a wide table into two or more tables so that each table has fewer columns. If a query touches both tables, the table will have to be joined, which will affect query performance. If you use this option, you will want to allocate columns to the different tables carefully so that queries will usually touch only one of the tables. This option would also affect any existing queries and loading scripts. Another option is to omit some columns from the columnstore index. Good candidates are columns that are infrequently touched by queries that require scanning large amounts of data

Session Objectives & Takeaway Understand what SQL Server 2012 Columnstore indexes are Understand ways of creating SQL Server 2012 Columnstore Index Understand the Internals of SQL Server 2012 Columnstore Index Understand the advantages of Columnstore indexes in SQL Server 2012 Understand the limitations of Columnstore indexes in SQL Server 2012 Understand Best Practices to Columnstore indexes Implementation in SQL Server 2012 Understand the significant improvements available in SQL Server 2014

Questions…

AdventureWorks Cycles Case Study AdventureWorks Cycles Fictitious company on which the AdventureWorks sample databases are based Manufactures and sells metal and composite bicycles to North American, European and Asian commercial markets http://www.adventure-works.com/ Actually redirects to Microsoft.com Requirement/Product Backlog Item/User Story “As a Procurement Manager, I would like a report that shows Products by their Units In versus Units Out”

Case Study AdventureWorksDW2012

Case Study AdventureWorksDW2012

Case Study AdventureWorksDW2012

Case Study AdventureWorksDW2012

Case Study AdventureWorksDW2012

Case Study AdventureWorksDW2012

A Tale of Two Queries AdventureWorksDW2012 TSQL Script Without ColumnStore Index TSQL Script That Uses ColumnStore Index

Case Study

Case Study

Case Study AdventureWorksDW2012

Questions…

Columnstore Index: SQL Server 2014 Fully Read/Write Clustered Columnstore No need for “trickle-loading” or other workarounds Yet Partition switching & BULK INSERT remain best practices Data type support expanded All data types except: (n)varchar(max), varbinary(max), XML, Spatial, CLR Basically, SQL14 columnstore is compatible with all non-blob datatypes Note: Non-clustered columnstore is still supported & is still a read-only structure Dependency on conventional b-tree structures has been removed

Columnstore Index: SQL Server 2014 Batch mode Query Processor vector-based (L1 cache resident) operations expanded, improved New support for: All joins (including OUTER, HASH, SEMI (NOT IN, IN), UNION ALL, Scalar aggregates, “Mixed mode” plans, Hash join enhancements Columnstore joins on string columns is better in 2014 than 2012, yet still slower! Compression Adds an additional layer of  compression on top of the inherent compression used by ColumnStore Note: Non-clustered columnstore is still supported & is still a read-only structure Dependency on conventional b-tree structures has been removed

Columnstore Index: SQL Server 2014 How to Create Columnstore Index Create Columnstore Index: Script CREATE CLUSTERED COLUMNSTORE INDEX ix_cs_MyDWTable ON dbo.MyDWTable; --no column list! DROP INDEX dbo.MyDWTable.ix_cs_MyTable; ALTER INDEX dbo.MyDWTable.ix_cs_MyTable DISABLE; ALTER INDEX dbo.MyDWTable.ix_cs_MyTable ENABLE;

Session Objectives & Takeaway Understand what SQL Server 2012 Columnstore indexes are Understand ways of creating SQL Server 2012 Columnstore Index Understand the Internals of SQL Server 2012 Columnstore Index Understand the advantages of Columnstore indexes in SQL Server 2012 Understand the limitations of Columnstore indexes in SQL Server 2012 Understand Best Practices to Columnstore indexes Implementation in SQL Server 2012 Understand the significant improvements available in SQL Server 2014

Questions…