1. Operational Analytics in SQL Server 2016 and Azure SQL Database
Microsoft Ignite 2016
5/27/ :11 PM
BRK3071
Operational Analytics in SQL Server 2016 and Azure SQL Database
Kevin Farlee
Senior Program Manager, Microsoft
Sunil Agarwal
Principal Program Manager, Microsoft
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

2. Outline Motivation Real-Time analytics with disk-based tables
Real-Time analytics with in-memory OLTP

3. Traditional Analytics Architecture
Microsoft Ignite 2015
5/27/ :11 PM
Select ProduceName, ExpiryDate, SUM (inventory – item_sold)
From <transactions>
where [Date] >= DATEADD(day, -1, GETDATE())
Group by ProductName, ExpiryDate, DATEPART(HOUR, [Date])
Insert into <transactions>
values (‘<upc-code>, ‘flowers’, $20.00)
SQL Server
Database
Application Tier
Presentation Layer
IIS Server
BI and analytics
Dashboards
Reporting
SQL Server
Analysis Server
Key Issues
Complex Implementation
Requires two Servers (CapEx and OpEx)
Data Latency in Analytics
More businesses demand/require real-time Analytics
SQL Server
Relational DW
Database
ETL
Hourly, Daily, Weekly
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

4. Minimizing Data Latency for Analytics
Microsoft Ignite 2015
Minimizing Data Latency for Analytics
5/27/ :11 PM
Select ProduceName, ExpiryDate, SUM (inventory – item_sold)
From <transactions>
where [Date] >= DATEADD(day, -1, GETDATE())
Group by ProductName, ExpiryDate, DATEPART(HOUR, [Date])
Insert into <transactions>
values (‘<upc-code>, ‘flowers’, $20.00)
SQL Server
Database
Application Tier
Presentation Layer
IIS Server
Benefits
No Data Latency
No ETL
No Separate DW
Challenges
Minimizing Impact on OLTP workload
Delivering Performant Analytics
BI and analytics
Dashboards
Reporting
This is Real-Time ANALYTICS
SQL Server
Analysis Server
Add columnstore index
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

5. Real-Time Analytics – What it is NOT for
5/27/2018
Real-Time Analytics – What it is NOT for
OLTP
Operational Data Coming from multiple sources
Extreme Analytics
Needs pre-aggregated cubes
Star-Schema
Challenge with OLTP schema
Data is normalized
Queries require multi-table joins
© 2015 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

6. Outline Motivation Real-Time analytics with disk-based tables
Real-Time analytics with in-memory OLTP

7. Real-time Analytics: Nonclustered Columnstore Index (NCCI)
Relational Table (disk-based)
(Clustered Index/Heap)
Btree Index
Delete bitmap
Delta rowgroups
Delete Buffer
Nonclustered columnstore index (NCCI)
Key Points
Create an updateable non-clustered columnstore index (NCCI) for analytics queries
Drop all other indexes that were created for analytics.
No OLTP Application changes.
ColumnStore index automatically keeps up with DML operations
Query Optimizer will choose columnstore index where needed

8. Real-Time Analytics: Columnstore Index Overhead
DML Operations on OLTP workload
Operation
BTREE (NCI)
Non Clustered ColumnStore Index (NCCI)
Insert
Insert row into btree
Insert row into btree (delta store)
Delete
Seek row(s) to be deleted
Delete the row
Seek for the row in the delta stores
(there can be multiple)
If row found, then delete
Else insert the key into delete row buffer
Update
Seek the row(s)
Delete the row (steps same as above)
Insert the updated row into delta store
Improvement - No PAGE compression
Solution – Exponentially Increasing sized Delta RGs
Reclaiming Deleted Rows: Run ALTER Index <> REORGANIZE (it is ONLINE)

9. Real-time Analytics: Minimizing Columnstore Index overhead
OLTP Workload
Relational Table
(Clustered Index/Heap)
Btree Index
HOT
Delete bitmap
Delete Buffer
Delta rowgroups
Nonclustered columnstore index (NCCI) – filtered index
Key Points
Create Columnstore only on cold data – using filtered predicate to minimize maintenance
Analytics query accesses both columnstore and ‘hot’ data transparently
Example – Order Management Application –
create nonclustered columnstore index ….. where order_status = ‘SHIPPED’

10. Real-time Analytics: Minimizing Columnstore Index overhead
Relational Table
(Clustered Index/Heap)
Btree Index
Syntax:
Create nonclustered columnstore index <name> on <table> (<columns>) with (compression_delay = 30 Minutes)
HOT
Delete bitmap
Nonclustered columnstore index (NCCI) – Compression Delay
Delete Buffer
Delta rowgroups
Compression Delay
Key Points
Delta RG is only compressed after ‘Compression_Delay’ duration
Minimizes/Eliminates index fragmentation

11. Real-time Analytics: Minimizing Columnstore overhead
AlwaysOn Availability Group
Analytics workload
Real-time workload
Primary
Replica
Secondary Replica
Secondary
Insert into <transactions>
values (‘<upc-code>, ‘flowers’, $20.00)
Select ProduceName, ExpiryDate, SUM (inventory – item_sold)
From <transactions>
where [Date] >= DATEADD(day, -1, GETDATE())
Group by ProductName, ExpiryDate, DATEPART(HOUR, [Date])
Key Points
Mission Critical Real-time Workloads typically configured for High Availability using AlwaysOn Availability Groups
You can offload analytics to readable secondary replica

12. Demo: Real Time Analytics - NCCI
Kevin Farlee

13. Outline Motivation Real-Time analytics with disk-based tables
Real-Time analytics with in-memory OLTP

14. SQL In-Memory Technologies
5/27/2018
SQL In-Memory Technologies
SQL Real-Time Analytics
In 2016 and Azure DB
Faster Transactions
Faster Analytics +
IN-MEMORY OLTP +
IN-MEMORY DW
Up to 30x faster transaction processing with In-Memory OLTP
Over 100x analytics query speed and significant data compression with In-Memory ColumnStore
Using the same tables
© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

15. Real-time Analytics: Columnstore on In-Memory Tables
5/27/2018
In-Memory OLTP Table
Range Index
Hash Index
Deleted Rows Table (DRT) – Tracks deleted rows
No explicit delta rowgroup
Rows (tail) not in columnstore stay in in-memory OLTP table
No columnstore index overhead when operating on tail
Background task migrates rows from tail to columnstore in chunks of 1 million rows
Compression Delay Option for Hot Data
Storage
Columnstore data fully resident in memory
Persisted together with OLTP data in Memory-Optimized FG
No OLTP application changes required.
Deleted Rows Table
Hot
Delta RG
Hot
Delta RG
Columnstore Index
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

16. Demo – In-Memory OLTP and ColumnStore
Kevin Farlee

17. Free IT Pro resources To advance your career in cloud technology
Microsoft Ignite 2016
5/27/ :11 PM
Free IT Pro resources To advance your career in cloud technology
Plan your career path
Microsoft IT Pro Career Center
Cloud role mapping
Expert advice on skills needed
Self-paced curriculum by cloud role
$300 Azure credits and extended trials
Pluralsight 3 month subscription (10 courses)
Phone support incident
Weekly short videos and insights from Microsoft’s leaders and engineers
Connect with community of peers and Microsoft experts
Get started with Azure
Microsoft IT Pro Cloud Essentials
Demos and how-to videos
Microsoft Mechanics
Connect with peers and experts
Microsoft Tech Community
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

18. Please evaluate this session
5/27/ :11 PM
Please evaluate this session
Your feedback is important to us!
From your PC or Tablet visit MyIgnite at
From your phone download and use the Ignite Mobile App by scanning the QR code above or visiting
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

19. 5/27/ :11 PM
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.