1. Enable Operational Analytics (HTAP) in SQL Server 2016 and Azure SQL Database
Sunil Agarwal
Principal Program Manager,
SQL Server Product Tiger Team

2. Session Objectives What/Why Real-Time Analytics?
Tech Ready 15
12/2/2017
Session Objectives
What/Why Real-Time Analytics?
How does Columnstore Index help?
Rich set of configurations to minimize impact of Analytics
© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

3. Outline Motivation Real-Time analytics with disk-based tables
Real-Time analytics with in-memory OLTP

4. Traditional Real-time/Analytics Architecture
Microsoft Ignite 2015
12/2/2017 1:51 AM
Traditional Real-time/Analytics Architecture
Select ProduceName, ExpiryDate, SUM (inventory – item_sold)
From <transactions>
where [Date] >= DATEADD(day, -1, GETDATE())
Group by ProductName, ExpiryDate, DATEPART(HOUR, [Date])
Insert into <transactions>
values (‘<upc-code>, ‘fish’, $20.00)
SQL Server
Database
Application Tier
Presentation Layer
IIS Server
BI and analytics
Dashboards
Reporting
SQL Server
Analysis Server
Key Issues
Complex Implementation
Requires two Servers (CapEx and OpEx)
Data Latency in Analytics
More businesses demand/require real-time Analytics
AS – (a) data model variance from budget, forecasting using MDX and DAX (b) caching
Insert into CreditCardTrans values (‘ ’, ‘Surface’, ‘Seatte’, $450.00)
Select count(*) as transacitons, max(price)
from creditCardTrans
Where CreditCardID = ‘ ’)
Group By City
SQL Server
Relational DW
Database
ETL
Hourly, Daily, Weekly
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

5. DEMO: Power Of Columnstore Index

6. Minimizing Data Latency for Analytics
Microsoft Ignite 2015
12/2/2017 1:51 AM
Minimizing Data Latency for Analytics
Select ProduceName, ExpiryDate, SUM (inventory – item_sold)
From <transactions>
where [Date] >= DATEADD(day, -1, GETDATE())
Group by ProductName, ExpiryDate, DATEPART(HOUR, [Date])
Insert into <transactions>
values (‘<upc-code>, ‘fish’, $20.00)
Benefits
No Data Latency
No ETL
No Separate DW
Challenges
Minimizing Impact on OLTP workload
Delivering Performant Analytics
SQL Server
Database
Application Tier
Presentation Layer
IIS Server
BI and analytics
Dashboards
Reporting
This is Real-Time ANALYTICS
SQL Server
Analysis Server
Add columnstore index
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

7. Real-Time Analytics – What it is NOT for
12/2/2017
Real-Time Analytics – What it is NOT for
OLTP
Operational Data Coming from multiple sources
Extreme Analytics
Needs pre-aggregated cubes
Star-Schema
Challenge with OLTP schema
Data is normalized
Queries require multi-table joins
© 2015 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

8. Columnstore Index: Overview
12/2/2017
Columnstore Index: Overview
Data stored as columns C5 C1 C2 C3 C4
rowgroup
segment
Improved Performance:
More data fits in memory
Optimized for CPU utilization
Ideal for DW Workload
Improved compression:
Data from same domain compress better
Reduced I/O:
Fetch only columns needed
© 2015 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

9. Outline Motivation Real-Time analytics with disk-based tables
Real-Time analytics with in-memory OLTP

10. Real-time Analytics: Nonclustered Columnstore Index (NCCI)
Relational Table (disk-based)
(Clustered Index/Heap)
Btree Index
Nonclustered columnstore index (NCCI)
Delta rowgroups
Delete Bitmap
Delete Buffer
Key Points
Create an updateable non-clustered columnstore index (NCCI) for analytics queries
Drop all other indexes that were created for analytics.
No OLTP Application changes.
ColumnStore index automatically keeps up with DML operations
Query Optimizer will choose columnstore index where needed

11. Real-Time Analytics: Columnstore Index Overhead
DML Operations on OLTP workload
Operation
BTREE (NCI)
Non Clustered ColumnStore Index (NCCI)
Insert
Insert row into btree
Insert row into btree (delta store)
Delete
Seek row(s) to be deleted
Delete the row
Seek for the row in the delta stores
(there can be multiple)
If row found, then delete
Else insert the key into delete row buffer
Update
Seek the row(s)
Delete the row (steps same as above)
Insert the updated row into delta store
Improvement - No PAGE compression
Solution – Exponentially Increasing sized Delta RGs
Reclaiming Deleted Rows: Run ALTER Index <> REORGANIZE (it is ONLINE)

12. Real-time Analytics: Minimizing Columnstore Index overhead
Relational Table
(Clustered Index/Heap)
Btree Index
OLTP Workload
HOT
Nonclustered columnstore index (NCCI)
Delta rowgroups
Delete Bitmap
Key Points
Create Columnstore only on cold data – using filtered predicate to minimize maintenance
Analytics query accesses both columnstore and ‘hot’ data transparently
Example – Order Management Application –
create nonclustered columnstore index ….. where order_status = ‘SHIPPED’

13. Real-time Analytics: Minimizing Columnstore Index overhead
Relational Table
(Clustered Index/Heap)
Btree Index
30 mins
Syntax:
Create nonclustered columnstore index <name> on <table> (<columns>) with (compression_delay = 30 Minutes)
HOT
Nonclustered columnstore index (NCCI)
Delete Bitmap
Delete Buffer
Nonclustered columnstore index (NCCI)
Delta rowgroups
Delete Bitmap
Delete Buffer
Key Points
Delta RG is only compressed after ‘Compression_Delay’ duration
Minimizes/Eliminates index fragmentation

14. Real-time Analytics: Minimizing Columnstore overhead
AlwaysOn Availability Group
Analytics workload
Real-time workload
Primary
Replica
Secondary
Secondary
Replica
Insert into <transactions>
values (‘<upc-code>, ‘fish’, $20.00)
Select ProduceName, ExpiryDate, SUM (inventory – item_sold)
From <transactions>
where [Date] >= DATEADD(day, -1, GETDATE())
Group by ProductName, ExpiryDate, DATEPART(HOUR, [Date])
Key Points
Mission Critical Real-time Workloads typically configured for High Availability using AlwaysOn Availability Groups
You can offload analytics to readable secondary replica

15. Customer Success Story – M-Files
Environment
77 million rows per table (5 tables involved in a query)
Total size of database tested: 5 TB without compression
Performance with NCCI in SQL Server 2016
OLTP performance impact < 5%
OLAP Query performance
Query
Before (sec)
After (sec)
Contracts by Pricing Type and Agency
6.5
0.5
Contracts by year and month
2.6
0.05

16. M-Files – Customer Quote
“Lightning fast analytical queries as well as online transactions are a critical success factor in M-Files implementations. The addition of updatable non-clustered columnstore indexes in Azure SQL Database and in Microsoft SQL Server 2016 helped us to achieve over 10x faster queries in critical workloads at our customers without significant performance cost on online transaction processing. The change required no code changes in our application and provided instant performance boost both to our clients on-premises as well as to our M-Files Cloud Vault users in Azure” Antti Nivala - Founder and CTO

17. Customer Success Stories - FISGlobal
Microsoft Ignite 2016
Customer Success Stories - FISGlobal
12/2/2017 1:51 AM
Status - Application Live in Production
FIS Case Study
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

18. DEMO: NonClustered Columnstore Index

19. Outline Motivation Real-Time analytics with disk-based tables
Real-Time analytics with in-memory OLTP

20. Real-time Analytics: Columnstore on In-Memory Tables
12/2/2017
In-Memory OLTP Table
Range Index
Hash Index
Deleted Rows Table (DRT) – Tracks deleted rows
No explicit delta rowgroup
Rows (tail) not in columnstore stay in in-memory OLTP table
No columnstore index overhead when operating on tail
Background task migrates rows from tail to columnstore in chunks of 1 million rows
Compression Delay Option for Hot Data
Storage
Columnstore data fully resident in memory
Persisted together with OLTP data in Memory-Optimized FG
No OLTP application changes required.
Deleted Rows Table
Hot
Delta RG
Hot
Delta RG
Columnstore Index
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

21. DEMO: In-Memory OLTP and ColumnStore

22. Session Objectives And Takeaways
Tech Ready 15
12/2/2017
Session Objectives And Takeaways
Session Objective(s):
What/Why Real-Time Analytics?
How does Columnstore Index help?
Rich set of configurations to minimize impact of Analytics
Key Takeaways
I can do real-time analytics with SQL Server with no application changes
I have multiple options to minimize/eliminate impact on my OLTP workload
© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.