Download presentation
Presentation is loading. Please wait.
Published byAugusta Robinson Modified over 6 years ago
1
Operational Analytics in SQL Server 2016 and Azure SQL Database
Microsoft Ignite 2016 5/27/ :11 PM BRK3071 Operational Analytics in SQL Server 2016 and Azure SQL Database Kevin Farlee Senior Program Manager, Microsoft Sunil Agarwal Principal Program Manager, Microsoft © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
2
Outline Motivation Real-Time analytics with disk-based tables
Real-Time analytics with in-memory OLTP
3
Traditional Analytics Architecture
Microsoft Ignite 2015 5/27/ :11 PM Select ProduceName, ExpiryDate, SUM (inventory – item_sold) From <transactions> where [Date] >= DATEADD(day, -1, GETDATE()) Group by ProductName, ExpiryDate, DATEPART(HOUR, [Date]) Insert into <transactions> values (‘<upc-code>, ‘flowers’, $20.00) SQL Server Database Application Tier Presentation Layer IIS Server BI and analytics Dashboards Reporting SQL Server Analysis Server Key Issues Complex Implementation Requires two Servers (CapEx and OpEx) Data Latency in Analytics More businesses demand/require real-time Analytics SQL Server Relational DW Database ETL Hourly, Daily, Weekly © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
4
Minimizing Data Latency for Analytics
Microsoft Ignite 2015 Minimizing Data Latency for Analytics 5/27/ :11 PM Select ProduceName, ExpiryDate, SUM (inventory – item_sold) From <transactions> where [Date] >= DATEADD(day, -1, GETDATE()) Group by ProductName, ExpiryDate, DATEPART(HOUR, [Date]) Insert into <transactions> values (‘<upc-code>, ‘flowers’, $20.00) SQL Server Database Application Tier Presentation Layer IIS Server Benefits No Data Latency No ETL No Separate DW Challenges Minimizing Impact on OLTP workload Delivering Performant Analytics BI and analytics Dashboards Reporting This is Real-Time ANALYTICS SQL Server Analysis Server Add columnstore index © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
5
Real-Time Analytics – What it is NOT for
5/27/2018 Real-Time Analytics – What it is NOT for OLTP Operational Data Coming from multiple sources Extreme Analytics Needs pre-aggregated cubes Star-Schema Challenge with OLTP schema Data is normalized Queries require multi-table joins © 2015 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
6
Outline Motivation Real-Time analytics with disk-based tables
Real-Time analytics with in-memory OLTP
7
Real-time Analytics: Nonclustered Columnstore Index (NCCI)
Relational Table (disk-based) (Clustered Index/Heap) Btree Index Delete bitmap Delta rowgroups Delete Buffer Nonclustered columnstore index (NCCI) Key Points Create an updateable non-clustered columnstore index (NCCI) for analytics queries Drop all other indexes that were created for analytics. No OLTP Application changes. ColumnStore index automatically keeps up with DML operations Query Optimizer will choose columnstore index where needed
8
Real-Time Analytics: Columnstore Index Overhead
DML Operations on OLTP workload Operation BTREE (NCI) Non Clustered ColumnStore Index (NCCI) Insert Insert row into btree Insert row into btree (delta store) Delete Seek row(s) to be deleted Delete the row Seek for the row in the delta stores (there can be multiple) If row found, then delete Else insert the key into delete row buffer Update Seek the row(s) Delete the row (steps same as above) Insert the updated row into delta store Improvement - No PAGE compression Solution – Exponentially Increasing sized Delta RGs Reclaiming Deleted Rows: Run ALTER Index <> REORGANIZE (it is ONLINE)
9
Real-time Analytics: Minimizing Columnstore Index overhead
OLTP Workload Relational Table (Clustered Index/Heap) Btree Index HOT Delete bitmap Delete Buffer Delta rowgroups Nonclustered columnstore index (NCCI) – filtered index Key Points Create Columnstore only on cold data – using filtered predicate to minimize maintenance Analytics query accesses both columnstore and ‘hot’ data transparently Example – Order Management Application – create nonclustered columnstore index ….. where order_status = ‘SHIPPED’
10
Real-time Analytics: Minimizing Columnstore Index overhead
Relational Table (Clustered Index/Heap) Btree Index Syntax: Create nonclustered columnstore index <name> on <table> (<columns>) with (compression_delay = 30 Minutes) HOT Delete bitmap Nonclustered columnstore index (NCCI) – Compression Delay Delete Buffer Delta rowgroups Compression Delay Key Points Delta RG is only compressed after ‘Compression_Delay’ duration Minimizes/Eliminates index fragmentation
11
Real-time Analytics: Minimizing Columnstore overhead
AlwaysOn Availability Group Analytics workload Real-time workload Primary Replica Secondary Replica Secondary Insert into <transactions> values (‘<upc-code>, ‘flowers’, $20.00) Select ProduceName, ExpiryDate, SUM (inventory – item_sold) From <transactions> where [Date] >= DATEADD(day, -1, GETDATE()) Group by ProductName, ExpiryDate, DATEPART(HOUR, [Date]) Key Points Mission Critical Real-time Workloads typically configured for High Availability using AlwaysOn Availability Groups You can offload analytics to readable secondary replica
12
Demo: Real Time Analytics - NCCI
Kevin Farlee
13
Outline Motivation Real-Time analytics with disk-based tables
Real-Time analytics with in-memory OLTP
14
SQL In-Memory Technologies
5/27/2018 SQL In-Memory Technologies SQL Real-Time Analytics In 2016 and Azure DB Faster Transactions Faster Analytics + IN-MEMORY OLTP + IN-MEMORY DW Up to 30x faster transaction processing with In-Memory OLTP Over 100x analytics query speed and significant data compression with In-Memory ColumnStore Using the same tables © 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
15
Real-time Analytics: Columnstore on In-Memory Tables
5/27/2018 In-Memory OLTP Table Range Index Hash Index Deleted Rows Table (DRT) – Tracks deleted rows No explicit delta rowgroup Rows (tail) not in columnstore stay in in-memory OLTP table No columnstore index overhead when operating on tail Background task migrates rows from tail to columnstore in chunks of 1 million rows Compression Delay Option for Hot Data Storage Columnstore data fully resident in memory Persisted together with OLTP data in Memory-Optimized FG No OLTP application changes required. Deleted Rows Table Hot Delta RG Hot Delta RG Columnstore Index © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
16
Demo – In-Memory OLTP and ColumnStore
Kevin Farlee
17
Free IT Pro resources To advance your career in cloud technology
Microsoft Ignite 2016 5/27/ :11 PM Free IT Pro resources To advance your career in cloud technology Plan your career path Microsoft IT Pro Career Center Cloud role mapping Expert advice on skills needed Self-paced curriculum by cloud role $300 Azure credits and extended trials Pluralsight 3 month subscription (10 courses) Phone support incident Weekly short videos and insights from Microsoft’s leaders and engineers Connect with community of peers and Microsoft experts Get started with Azure Microsoft IT Pro Cloud Essentials Demos and how-to videos Microsoft Mechanics Connect with peers and experts Microsoft Tech Community © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
18
Please evaluate this session
5/27/ :11 PM Please evaluate this session Your feedback is important to us! From your PC or Tablet visit MyIgnite at From your phone download and use the Ignite Mobile App by scanning the QR code above or visiting © 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
19
5/27/ :11 PM © 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.