SQL Server 2016: Real-time operational analytics Sunil Agarwal Principal Program Manager SQL Server

Outline Motivation Real-Time analytics with disk-based tables Real-Time analytics with in-memory OLTP

Traditional Real-time/Analytics Architecture Microsoft Ignite 2015 Traditional Real-time/Analytics Architecture 4/28/2017 4:15 PM Select ProduceName, ExpiryDate, Quantity, Sum(Sales) From <transactions> Where ProduceType =‘perishables’ Group By ProduceName, ExpiryDate Insert into <transactions> values (‘<upc-code>, ‘flowers’, $20.00) SQL Server Database Application Tier Presentation Layer IIS Server BI and analytics Dashboards Reporting SQL Server Analysis Server Key Issues Complex Implementation Requires two Servers (CapEx and OpEx) Data Latency in Analytics More businesses demand/require real-time Analytics AS – (a) data model variance from budget, forecasting using MDX and DAX (b) caching Insert into CreditCardTrans values (‘123-123-1234’, ‘Surface’, ‘Seatte’, $450.00) Select count(*) as transacitons, max(price) from creditCardTrans Where CreditCardID = ‘123-123-123’) Group By City SQL Server Relational DW Database ETL Hourly, Daily, Weekly © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Minimizing Data Latency for Analytics Microsoft Ignite 2015 Minimizing Data Latency for Analytics 4/28/2017 4:15 PM Select ProduceName, ExpiryDate, Quantity, sum (sales) From <transactions> Where ProduceType =‘perishables’ Group By ProduceName, ExpiryDate Insert into <transactions> values (‘<upc-code>, ‘flowers’, $20.00) Benefits No Data Latency No ETL No Separate DW Challenges Minimizing Impact on OLTP workload Delivering Performant Analytics SQL Server Database Application Tier Presentation Layer IIS Server BI and analytics Dashboards Reporting This is Real-Time ANALYTICS SQL Server Analysis Server Add columnstore index © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Real-Time Analytics – What it is NOT for 4/28/2017 Real-Time Analytics – What it is NOT for OLTP Operational Data Coming from multiple sources Extreme Analytics Needs pre-aggregated cubes Star-Schema Challenge with OLTP schema Data is normalized Queries require multi-table joins © 2015 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Outline Motivation Real-Time analytics with disk-based tables Real-Time analytics with in-memory OLTP

Real-time Analytics: Nonclustered Columnstore Index (NCCI) Dynamics AX 7 Production Preview in Azure SQLDB Relational Table (disk-based) (Clustered Index/Heap) Btree Index Nonclustered columnstore index (NCCI) Delta rowgroups Delete Bitmap Delete Buffer Key Points Create an updateable non-clustered columnstore index (NCCI) for analytics queries Drop all other indexes that were created for analytics. No OLTP Application changes. ColumnStore index automatically keeps up with DML operations Query Optimizer will choose columnstore index where needed

Real-Time Analytics: Columnstore Index Overhead DML Operations on OLTP workload Operation BTREE (NCI) Non Clustered ColumnStore Index (NCCI) Insert Insert row into btree Insert row into btree (delta store) Delete Seek row(s) to be deleted Delete the row Seek for the row in the delta stores (there can be multiple) If row found, then delete Else insert the key into delete row buffer Update Seek the row(s) Delete the row (steps same as above) Insert the updated row into delta store Improvement - No PAGE compression Solution – Exponentially Increasing sized Delta RGs Reclaiming Deleted Rows: Run ALTER Index <> REORGANIZE (it is ONLINE)

Real-time Analytics: Minimizing Columnstore Index overhead OLTP Workload Relational Table (Clustered Index/Heap) Btree Index HOT Nonclustered columnstore index (NCCI) Delta rowgroups Delete Bitmap Key Points Create Columnstore only on cold data – using filtered predicate to minimize maintenance Analytics query accesses both columnstore and ‘hot’ data transparently Example – Order Management Application – create nonclustered columnstore index ….. where order_status = ‘SHIPPED’

Real-time Analytics: Minimizing Columnstore Index overhead Relational Table (Clustered Index/Heap) Btree Index 30 mins Syntax: Create nonclustered columnstore index <name> on <table> (<columns>) with (compression_delay = 30 Minutes) HOT Nonclustered columnstore index (NCCI) Delete Bitmap Delete Buffer Nonclustered columnstore index (NCCI) Delta rowgroups Delete Bitmap Delete Buffer Key Points Delta RG is only compressed after ‘Compression_Delay’ duration Minimizes/Eliminates index fragmentation

Real-time Analytics: Minimizing Columnstore overhead AlwaysOn Availability Group Analytics workload Real-time workload Primary Replica Secondary Secondary Replica Insert into <transactions> values (‘<upc-code>, ‘flowers’, $20.00) Select ProduceName, ExpiryDate, Quantity, sum (sales) From <transactions> Where ProduceType =‘perishables’ Group By ProduceName, ExpiryDate Key Points Mission Critical Real-time Workloads typically configured for High Availability using AlwaysOn Availability Groups You can offload analytics to readable secondary replica

Outline Motivation Real-Time analytics with disk-based tables Real-Time analytics with in-memory OLTP

Real-time Analytics: Columnstore on In-Memory Tables 4/28/2017 In-Memory OLTP Table Range Index Hash Index Deleted Rows Table (DRT) – Tracks deleted rows No explicit delta rowgroup Rows (tail) not in columnstore stay in in-memory OLTP table No columnstore index overhead when operating on tail Background task migrates rows from tail to columnstore in chunks of 1 million rows Compression Delay Option for Hot Data Storage Columnstore data fully resident in memory Persisted together with OLTP data in Memory-Optimized FG No OLTP application changes required. Deleted Rows Table Hot Delta RG Hot Delta RG Columnstore Index © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Session Objectives And Takeaways Tech Ready 15 4/28/2017 Session Objectives And Takeaways Session Objective(s): What/Why Real-Time Analytics? Rich set of configurations for Real-Time Analytics First leading commercial database combining in-memory OLTP and in-memory DW Key Takeaways I can do real-time analytics with SQL Server with no application changes I have multiple options to minimize/eliminate impact on my OLTP workload Superior Real-Time analytics over Oracle12C (In-Memory) © 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

4/28/2017 Q&A Note - It is important for attendees to use the Q&A microphone located in the session room: TechReadyTV recordings cannot capture Q&A unless it is spoken using the microphone Attendees in the back of the room may not be able to hear a question from someone in the front of the room © 2015 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.