IIS Server ETL Key Issues  Complex Implementation  Requires two Servers (CapEx and OpEx)  Data Latency in Analytics  More businesses demand/require real-time Analytics Insert into values (‘, ‘flowers’, $20.00 ) Select ProduceName, ExpiryDate, Quantity, Sum(Sales) From Where ProduceType =‘perishables’ Group By ProduceName, ExpiryDate

IIS Server Insert into values (‘, ‘flowers’, $20.00 ) Select ProduceName, ExpiryDate, Quantity, sum (sales) From Where ProduceType =‘perishables’ Group By ProduceName, ExpiryDate This is Real-Time ANALYTICS

OLTP Operational Data Coming from multiple sources Extreme Analytics – Needs pre-aggregated cubes – Star-Schema Challenge with OLTP schema – Data is normalized – Queries require multi-table joins

8 Relational Table (disk-based) (Clustered Index/Heap ) Btree Index Delete bitmap Delta rowgroups Delete Buffer

9 Real-Time Analytics: Columnstore Index Overhead DML Operations on OLTP workload OperationBTREE (NCI)Non Clustered ColumnStore Index (NCCI) InsertInsert row into btreeInsert row into btree (delta store) Delete(a)Seek row(s) to be deleted (b)Delete the row (a)Seek for the row in the delta stores (there can be multiple) (b)If row found, then delete Else insert the key into delete row buffer Update(a)Seek the row(s) (b)Update (a)Delete the row (steps same as above) (b)Insert the updated row into delta store Reclaiming Deleted Rows: Run ALTER Index <> REORGANIZE (it is ONLINE) Solution – Exponentially Increasing sized Delta RGs Improvement - No PAGE compression

Real-time Analytics: Minimizing Columnstore Index overhead Nonclustered Columnstore Index (NCCI) Relational Table (disk-based) (Clustered Index/Heap ) Btree Index Nonclustered columnstore index (NCCI) delta filtered HOT

SQL In-Memory Technologies Over 100x analytics query speed and significant data compression with In-Memory ColumnStore Up to 30x faster transaction processing with In-Memory OLTP Faster Analytics Faster Transactions + IN-MEMORY OLTP IN-MEMORY DW + Using the same tables In 2016 and Azure DB SQL Real-Time Analytics

14 Real-time Analytics: Columnstore on In-Memory Tables In-Memory OLTP Table Range Index Hash Index Deleted Rows Table Columnstore Index Delta RG

15 Primary Replica Secondary Replica Secondary Replica Secondary Replica Insert into values (‘, ‘flowers’, $20.00 ) Select ProduceName, ExpiryDate, Quantity, sum (sales) From Where ProduceType =‘perishables’ Group By ProduceName, ExpiryDate