Enable Operational Analytics (HTAP) in SQL Server 2016 and Azure SQL Database Sunil Agarwal Principal Program Manager, SQL Server Product Tiger Team sunila@microsoft.com

Session Objectives What/Why Real-Time Analytics? Tech Ready 15 12/2/2017 Session Objectives What/Why Real-Time Analytics? How does Columnstore Index help? Rich set of configurations to minimize impact of Analytics © 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Outline Motivation Real-Time analytics with disk-based tables Real-Time analytics with in-memory OLTP

Traditional Real-time/Analytics Architecture Microsoft Ignite 2015 12/2/2017 1:51 AM Traditional Real-time/Analytics Architecture Select ProduceName, ExpiryDate, SUM (inventory – item_sold) From <transactions> where [Date] >= DATEADD(day, -1, GETDATE()) Group by ProductName, ExpiryDate, DATEPART(HOUR, [Date]) Insert into <transactions> values (‘<upc-code>, ‘fish’, $20.00) SQL Server Database Application Tier Presentation Layer IIS Server BI and analytics Dashboards Reporting SQL Server Analysis Server Key Issues Complex Implementation Requires two Servers (CapEx and OpEx) Data Latency in Analytics More businesses demand/require real-time Analytics AS – (a) data model variance from budget, forecasting using MDX and DAX (b) caching Insert into CreditCardTrans values (‘123-123-1234’, ‘Surface’, ‘Seatte’, $450.00) Select count(*) as transacitons, max(price) from creditCardTrans Where CreditCardID = ‘123-123-123’) Group By City SQL Server Relational DW Database ETL Hourly, Daily, Weekly © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

DEMO: Power Of Columnstore Index

Minimizing Data Latency for Analytics Microsoft Ignite 2015 12/2/2017 1:51 AM Minimizing Data Latency for Analytics Select ProduceName, ExpiryDate, SUM (inventory – item_sold) From <transactions> where [Date] >= DATEADD(day, -1, GETDATE()) Group by ProductName, ExpiryDate, DATEPART(HOUR, [Date]) Insert into <transactions> values (‘<upc-code>, ‘fish’, $20.00) Benefits No Data Latency No ETL No Separate DW Challenges Minimizing Impact on OLTP workload Delivering Performant Analytics SQL Server Database Application Tier Presentation Layer IIS Server BI and analytics Dashboards Reporting This is Real-Time ANALYTICS SQL Server Analysis Server Add columnstore index © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Real-Time Analytics – What it is NOT for 12/2/2017 Real-Time Analytics – What it is NOT for OLTP Operational Data Coming from multiple sources Extreme Analytics Needs pre-aggregated cubes Star-Schema Challenge with OLTP schema Data is normalized Queries require multi-table joins © 2015 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Columnstore Index: Overview 12/2/2017 Columnstore Index: Overview Data stored as columns C5 C1 C2 C3 C4 rowgroup segment Improved Performance: More data fits in memory Optimized for CPU utilization Ideal for DW Workload Improved compression: Data from same domain compress better Reduced I/O: Fetch only columns needed © 2015 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Outline Motivation Real-Time analytics with disk-based tables Real-Time analytics with in-memory OLTP

Real-time Analytics: Nonclustered Columnstore Index (NCCI) Relational Table (disk-based) (Clustered Index/Heap) Btree Index Nonclustered columnstore index (NCCI) Delta rowgroups Delete Bitmap Delete Buffer Key Points Create an updateable non-clustered columnstore index (NCCI) for analytics queries Drop all other indexes that were created for analytics. No OLTP Application changes. ColumnStore index automatically keeps up with DML operations Query Optimizer will choose columnstore index where needed

Real-Time Analytics: Columnstore Index Overhead DML Operations on OLTP workload Operation BTREE (NCI) Non Clustered ColumnStore Index (NCCI) Insert Insert row into btree Insert row into btree (delta store) Delete Seek row(s) to be deleted Delete the row Seek for the row in the delta stores (there can be multiple) If row found, then delete Else insert the key into delete row buffer Update Seek the row(s) Delete the row (steps same as above) Insert the updated row into delta store Improvement - No PAGE compression Solution – Exponentially Increasing sized Delta RGs Reclaiming Deleted Rows: Run ALTER Index <> REORGANIZE (it is ONLINE)

Real-time Analytics: Minimizing Columnstore Index overhead Relational Table (Clustered Index/Heap) Btree Index OLTP Workload HOT Nonclustered columnstore index (NCCI) Delta rowgroups Delete Bitmap Key Points Create Columnstore only on cold data – using filtered predicate to minimize maintenance Analytics query accesses both columnstore and ‘hot’ data transparently Example – Order Management Application – create nonclustered columnstore index ….. where order_status = ‘SHIPPED’

Real-time Analytics: Minimizing Columnstore Index overhead Relational Table (Clustered Index/Heap) Btree Index 30 mins Syntax: Create nonclustered columnstore index <name> on <table> (<columns>) with (compression_delay = 30 Minutes) HOT Nonclustered columnstore index (NCCI) Delete Bitmap Delete Buffer Nonclustered columnstore index (NCCI) Delta rowgroups Delete Bitmap Delete Buffer Key Points Delta RG is only compressed after ‘Compression_Delay’ duration Minimizes/Eliminates index fragmentation

Real-time Analytics: Minimizing Columnstore overhead AlwaysOn Availability Group Analytics workload Real-time workload Primary Replica Secondary Secondary Replica Insert into <transactions> values (‘<upc-code>, ‘fish’, $20.00) Select ProduceName, ExpiryDate, SUM (inventory – item_sold) From <transactions> where [Date] >= DATEADD(day, -1, GETDATE()) Group by ProductName, ExpiryDate, DATEPART(HOUR, [Date]) Key Points Mission Critical Real-time Workloads typically configured for High Availability using AlwaysOn Availability Groups You can offload analytics to readable secondary replica

Customer Success Story – M-Files Environment 77 million rows per table (5 tables involved in a query) Total size of database tested: 5 TB without compression Performance with NCCI in SQL Server 2016 OLTP performance impact < 5% OLAP Query performance Query Before (sec) After (sec) Contracts by Pricing Type and Agency 6.5 0.5 Contracts by year and month 2.6 0.05

M-Files – Customer Quote “Lightning fast analytical queries as well as online transactions are a critical success factor in M-Files implementations. The addition of updatable non-clustered columnstore indexes in Azure SQL Database and in Microsoft SQL Server 2016 helped us to achieve over 10x faster queries in critical workloads at our customers without significant performance cost on online transaction processing. The change required no code changes in our application and provided instant performance boost both to our clients on-premises as well as to our M-Files Cloud Vault users in Azure” Antti Nivala - Founder and CTO

Customer Success Stories - FISGlobal Microsoft Ignite 2016 Customer Success Stories - FISGlobal 12/2/2017 1:51 AM Status - Application Live in Production FIS Case Study © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

DEMO: NonClustered Columnstore Index

Outline Motivation Real-Time analytics with disk-based tables Real-Time analytics with in-memory OLTP

Real-time Analytics: Columnstore on In-Memory Tables 12/2/2017 In-Memory OLTP Table Range Index Hash Index Deleted Rows Table (DRT) – Tracks deleted rows No explicit delta rowgroup Rows (tail) not in columnstore stay in in-memory OLTP table No columnstore index overhead when operating on tail Background task migrates rows from tail to columnstore in chunks of 1 million rows Compression Delay Option for Hot Data Storage Columnstore data fully resident in memory Persisted together with OLTP data in Memory-Optimized FG No OLTP application changes required. Deleted Rows Table Hot Delta RG Hot Delta RG Columnstore Index © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

DEMO: In-Memory OLTP and ColumnStore

Session Objectives And Takeaways Tech Ready 15 12/2/2017 Session Objectives And Takeaways Session Objective(s): What/Why Real-Time Analytics? How does Columnstore Index help? Rich set of configurations to minimize impact of Analytics Key Takeaways I can do real-time analytics with SQL Server with no application changes I have multiple options to minimize/eliminate impact on my OLTP workload © 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.