1. SQL Server 2016 Operational Analytics

2. Sponsorzy strategiczni
Sponsorzy srebrni

3. Łukasz Grala Microsoft MVP Data Platform | MCT | MCSE
Architect - Mentor Data Platform & Business Intelligence Solutions
Trainer Data Platform and Business Intelligence
University Lecturer
Author Webcasts and Publications
Microsoft MVP Data Platform
Leader PLSSUG Poznań
Phd Student on Poznan University of Technology, Faculty of Computing Science (topics – database and datawarehouse architecture, data mining, machine learning)

4. Marcin Szeliga Data Philosopher BI Expert and Consultant
Data Platform Architect
20 years of experience with SQL Server
Ph.D. Candidate at Politechnika Śląska

5. Microsoft platform leads the way on-premises and cloud
4/26/2017
Leader in 2014 for Gartner Magic Quadrants
Microsoft platform leads the way on-premises and cloud
Operational Database Management Systems
Data Warehouse Database Management Systems
Business Intelligence and Analytics Platforms
x86 Server Virtualization
Cloud Infrastructure as a Service
Enterprise Application Platform as a Service
Public Cloud Storage
Magic Quadrant leader in Operational Database Management Systems
(paywall)
Magic Quadrant leader in Data Warehouse Database Management Systems
(paywall)
Magic Quadrant leader in Business Intelligence and Analytics Platforms
(paywall)
Magic Quadrant for x86 Server Virtualization
Magic Quadrant for Cloud Infrastructure as a Service
Magic Quadrant for Enterprise Application Platform as a Service
Gartner Magic Quadrant for Public Cloud Storage
© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

6. Do more. Achieve more. Mission-critical performance
Deeper insights across data
Hyperscale cloud

7. SQL Server 2016 improvements
Performance
Security
Availability
Scalability
Operational analytics
Insights on operational data; Works with in-memory OLTP and disk-based OLTP
In-memory OLTP enhancements
Greater T-SQL surface area, terabytes of memory supported, and greater number of parallel CPUs
Query data store
Monitor and optimize query plans
Native JSON
Expanded support for JSON data
Temporal database support
Query data as points in time
Always encrypted
Sensitive data remains encrypted at all times with ability to query
Row-level security
Apply fine-grained access control to table rows
Dynamic data masking
Real-time obfuscation of data to prevent unauthorized access
Other enhancements
Audit success/failure of database operations
TDE support for storage of in- memory OLTP tables
Enhanced auditing for OLTP with ability to track history of record changes
Enhanced AlwaysOn
Three synchronous replicas for auto failover across domains
Round robin load balancing of replicas
Automatic failover based on database health
DTC for transactional integrity across database instances with AlwaysOn
Support for SSIS with AlwaysOn
Enhanced database caching
Cache data with automatic, multiple TempDB files per instance in multi- core environments
Performance
Enhanced in-memory performance with upto 30xfaster transactions, more than 100x faster queries than disk based relational databases and real-time operational analytics.
Security Upgrades
Always Encrypted technology helps protect your data at rest and in motion, on-premises and in the cloud, with master keys sitting with the application, without any application changes.
High Availability
Even higher availability and performance than SQL Server 2014 of your AlwaysOn secondaries with the ability to have up to 3 synchronous replicas, DTC support and round-robin load balancing of the secondaries.
Scalability
Enhanced database caching across multiple cores & support for Windows Server 2016 that efficiently scale compute, networking and storage in both physical and virtual environments.

8. Mission-critical performance
Security
Availability
Scalability
Operational analytics
Insights on operational data; Works with in-memory OLTP and disk-based OLTP
In-memory OLTP enhancements
Greater T-SQL surface area, terabytes of memory supported, and greater number of parallel CPUs
Query data store
Monitor and optimize query plans
Native JSON
Expanded support for JSON data
Temporal database support
Query data as points in time
Always encrypted
Sensitive data remains encrypted at all times with ability to query
Row-level security
Apply fine-grained access control to table rows
Dynamic data masking
Real-time obfuscation of data to prevent unauthorized access
Other enhancements
Audit success/failure of database operations
TDE support for storage of in- memory OLTP tables
Enhanced auditing for OLTP with ability to track history of record changes
Enhanced AlwaysOn
Three synchronous replicas for auto failover across domains
Round robin load balancing of replicas
Automatic failover based on database health
DTC for transactional integrity across database instances with AlwaysOn
Support for SSIS with AlwaysOn
Enhanced database caching
Cache data with automatic, multiple TempDB files per instance in multi- core environments

9. What does operational mean?
Refers to Operational Workload (i.e. OLTP)
Examples:
Enterprise Resource Planning (ERP) – Inventory, Order, Sales,
Machine Data – Data from machine operations on factory floor
Online Stores (e.g. Amazon, Expedia)
Stock/Security trades
Mission Critical
No downtime (High Availability) – impact on revenue
Low latency and high transaction throughput

10. What does analytics mean?
Studying past data (e.g. operational, social media) to identify potential trends
To analyze the effects of certain decisions or events (e.g. Ad campaign)
Analyze past/current data to predict outcomes (e.g. credit score)
Goals
Enhance the business by gaining knowledge to make improvements or changes
Source – MIT/SLOAN Management Review

11. Traditional BI architecture
Microsoft Ignite 2015
4/26/2017 6:49 AM
Traditional BI architecture
SQL Server
Database
Application Tier
Presentation Layer
IIS Server
BI and analytics
Dashboards
Reporting
Key Issues
Complex Implementation
Requires two Servers (CapEx and OpEx)
Data Latency in Analytics
More businesses demand/require real-time Analytics
SQL Server
Analysis Server
SQL Server
Relational DW
Database
ETL
Hourly, Daily, Weekly
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

12. Minimizing data latency for analytics
Microsoft Ignite 2015
4/26/2017 6:49 AM
Minimizing data latency for analytics
Benefits
No Data Latency
No ETL
No Separate DW
Challenges
Analytics queries are resource intensive and can cause blocking
How to minimize Impact on Operational workload
Sub-optimal execution of Analytics on relational schema
SQL Server
Database
Application Tier
Presentation Layer
IIS Server
BI and analytics
Dashboards
Reporting
SQL Server
Analysis Server
This is OPERATIONAL ANALYTICS
Add analytics specific indexes
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

13. SQL Server 2016

14. Operational Analytics
4/26/2017
Operational Analytics
Ability to run analytics queries concurrently with operational workload using the same schema
Not a replacement for
Extreme analytics queries performance possible using schemas customized (e.g. star/snowflake) and pre-aggregated cubes
Data coming from non-relational sources
Data coming from multiple relational sources requiring integrated analytic
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

15. Achieved using columnstore Index
Operational Analytics with SQL Server 2016
Goals
Minimal impact on operational workload with concurrent analytics
Performant analytics on operational schema
Achieved using columnstore Index

16. Quick Recap: Columnstore Index
Data stored as columns
Data stored as rows C1 C2 C3 C5 C4 …
Ideal for OLTP
Efficient operation on small set of rows
Improved compression:
Data from same domain compress better
Reduced I/O:
Fetch only columns needed
Improved performance:
More data fits in memory
Optimized for CPU utilization
Ideal for DW workload

17. Clustered Columnstore Performance: TPC-H

18. Operational analytics disk-based tables
4/26/2017 6:49 AM
Operational analytics disk-based tables
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

19. (Clustered Index/Heap)
Operational Analytics with columnstore index
Relational Table
(Clustered Index/Heap)
Btree Index
Nonclustered columnstore index (NCCI)
Delete bitmap
Delta rowgroups
Key Points
Create an updateable non-clustered columnstore index (NCCI) for analytics queries
Drop all other indexes that were created for analytics
No application changes
ColumnStore index is maintained just like any other index
Query Optimizer will choose columnstore index where needed

20. (Clustered Index/Heap)
Minimizing CSI overhead
DML Operations
Relational Table
(Clustered Index/Heap)
Btree Index
HOT
Delete bitmap
Nonclustered columnstore index (NCCI) – filtered index
Delta rowgroups
Key Points
Create Columnstore only on cold data – using filtered predicate to minimize maintenance
Analytics query accesses both columnstore and ‘hot’ data transparently
Example – Order Management Application –
CREATE NONCLUSTERED COLUMNSTORE INDEX ….. WHERE order_status = ‘SHIPPED’

21. Operational analytics for in-memory tables
4/26/2017 6:49 AM
Operational analytics for in-memory tables
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

22. Operational Analytics with columnstore on In-Memory Tables
4/26/2017
Operational Analytics with columnstore on In-Memory Tables
Hash Index
No explicit delta rowgroup
Rows (tail) not in columnstore stay in in-memory OLTP table
No columnstore index overhead when operating on tail
Background task migrates rows from tail to columnstore in chunks of 1 million rows not changed in last 1 hour
Deleted Rows Table (DRT) – tracks deleted rows
Columnstore data fully resident in memory
Persisted together with operational data
No application changes required
Range Index
Updateable CCI
DRT
Tail
In-Memory OLTP Table
Hot
Like
Delta rowgroup
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

23. Columnstore index on in-memory table overhead
DML Operations on In-Memory OLTP
Operation
Hash or Range Index
HK-CCI
Insert
Insert row into HK
Delete
Seek row(s) to be deleted
Delete the row
Delete the row in HK
If row in TAIL then return
else insert <colstore-RID> into DRT
Update
Seek the row(s)
Update (delete/insert)
Update (delete/insert) in HK

24. Minimizing this overhead
Microsoft Ignite 2015
4/26/2017 6:49 AM
Minimizing this overhead
DML Operations
In-Memory OLTP Table
Updateable CCI
DRT
Tail
Range Index
Hash Index
Like
Delta rowgroup
Hot
Keep hot data only in in-memory tables
Example – data stays hot for 1 day, 1 week…
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

25. Query processing

26. Demo time

27. Performance improvments
Scan type
Elapsed time (s)
Speedup
Row store scan, interop
44.441
Row store scan, native
28.445
1.6x
CSI scan, interop
0.802
55.4x

28. Insert, Update, Delete costs and query time
Operation
Elasped time (s) with CSI
Elasped time (s) No CSI
Increase % Update
Increase % Query
CSI scan, interop
0.802
BASE
Insert rows
53.5
47.8
11.9%
0.869
8.4%
Update rows
42.4
28.9
46.7%
1.181
47.3%
Delete rows
38.3
30.5
25.6%
1.231
53.5%

29. Single thread insert and update
Operation
Rows affected
Row store (s)
Secondary CSI (s)
Primary CSI (s)
1000 updates
10 000
0.893
1.400
6.866
10% insert
18M
233.9
566
291.4
2% update
3.96M
123.2
314.3
275.9

30. Single thread scan Millions of rows Row store Secondary CSI
Primary CSI
New built
180
99.1
4.7
1.71
After 1000 updates
99.4
5.4
1.75
After 10% inserts
198
108.7
14.5
9.5
After 2% updates
109.5
16.8
10.0

32. Elapsed time per lookup in a B-Tree index
Columns projected
B-tree over CSI (ms) without mapping
B-tree over CSI with mapping
B-tree over B-tree (ms) No compresion
B-tree over B-tree (ms) Page compresion 4
3.92
5.28
2.41
3.65 8
4.33
5.73
2.32
3.85 20
6.67
8.07
2.55
4.44

33. Overhead on data loading of having an NCI
New partition size (milion rows)
Time for bulk load into CSI (ms)
Time to create B-tree (ms)
Ratio (index creation/bulk load) 1
5 327 66
1.24% 5
5 335 77
1.45% 10
5 354 83
1.55%

34. Comparing performance
Operation
Billions of value per second
No SIMD
SIMD
Speedup
Bit unpacking 6bits
2.08
11.55
5.55x
Bit unpacking 12 bits
1.91
9.76
5.11x
Bit unpacking 21 bits
1.96
5.29
2.70x
Compaction 32 bits
1.24
6.70
5.40x
Range predicate 16 bits
0.94
11.42
5.06x
Sum 16 bit values
2.86
14.46
128-bit bitmap filter
0.97
11.77x
64KB bitmap filter
1.01
2.37
2.35x

35. Query performance (1) Predicate or aggregation Duration SQL2014 (ms)
Speedup
Billion of rows per s
Q1-Q4: select count(*) from LINEITEM where <predicate>
L_ORDERKEY =
220
140
1.57x
12.9
L_QUANTITY = 1900
664 68
9.76x
26.5
L_SHIPMODE='AIR'
694
147
4.72x
12.2
L_SHIPDATE between ' ' and ' '
512 87
5.89x
20.7

36. Query performance (2) Predicate or aggregation Duration SQL2014 (ms)
Speedup
Billion of rows per s
Q5-Q6: select count(*) from PARTSUPP where <predicate>
PS_AVAILQTY < 10 50 27
1.85x
8.9
PS_AVAILQTY = 10 45 15
3.00x 16
Q7-Q8: select <aggregates> from LINEITEM
avg(L_DISCOUNT)
1272
196
6.49x
9.1
avg(L_DISCOUNT), min(L_ORDERKEY), max(L_ORDERKEY)
1978
356
5.56x
5.1

37. Columnstore index overhead
DML operations on OLTP workload
Operation
BTREE (NCI)
Non Clustered ColumnStore Index (NCCI)
Insert
Insert row into btree
Insert row into btree (delta store)
Delete
Seek row(s) to be deleted
Delete the row
Seek for the row in the delta stores
(there can be multiple)
If row found, then delete
Otherwise insert the key into delete row buffer
Update
Seek the row(s)
Delete the row (steps same as above)
Insert the updated row into delta store

38. Availability Groups as data warehouse
Always on Availability Group
Key points
Mission Critical Operational Workloads typically configured for High Availability using AlwaysOn Availability Groups
You can offload analytics to readable secondary replica
Secondary Replica
Primary Replica
Source:
To configure an AlwaysOn availability group to support read-only routing in SQL Server 2016, you can use either Transact-SQL or PowerShell. Read-only routing refers to the ability of SQL Server to route qualifying read-only connection requests to an available AlwaysOn readable secondary replica (that is, a replica that is configured to allow read-only workloads when running under the secondary role). To support read-only routing, the availability group must possess an availability group listener. Read-only clients must direct their connection requests to this listener, and the client's connection strings must specify the application intent as "read-only." That is, they must be read-intent connection requests.

39. Minimizing data latency for analytics
Microsoft Ignite 2015
4/26/2017 6:49 AM
Minimizing data latency for analytics
SQL Server
Database
Application Tier
Presentation Layer
IIS Server
BI and analytics
Dashboards
Reporting
SQL Server
Analysis Server
Add analytics specific indexes
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

40. High-end Server Hardware
SSAS Enterprise Readiness: Tabular
New DirectQuery
DirectQuery for Oracle, Teradata, ASP
DirectQuery support for MDX query(Excel Tools)
High-end Server Hardware
Source:

41. Summary – OA with SQL Server 2016
Analytics in real-time with no data latency
Rich set of options to control impact on Operational workload
Industry leading solution Integrating in-memory OLTP with in-memory Analytics
No Application changes required

42. Sponsorzy strategiczni
Sponsorzy srebrni