1. Microsoft Ignite 2016
6/12/2018 5:05 AM
BRK3094
Accelerate SQL Server 2016 to the max Lessons learned from customer engagements
Denzil
Arvind
Program Managers, Microsoft SQLCAT
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

2. SQL Customer Advisory Team (SQLCAT)
Our mission: Be the “voice of the customer” inside the product development organization
Help customers adopt new SQL technologies: both in Azure and on premise
Engage on challenging “push the envelope” solutions
Architecture Reviews
SQLCAT Customer Lab engagements (in Redmond, WA)
Improve the product and service through customer engagements
Deliver Best Practices to our community
Current Focus: Azure SQL DB, Azure SQL DW, SQL Server on Linux / Windows
Blog:

3. Today’s menu Upgrading to SQL Server 2016 Columnstore Indexes
Key Themes
Compatibility Level considerations
Columnstore Indexes
Data Loading
Converting Existing tables to Columnstore
In-Memory OLTP
Methodology
Indexing
Temporal
Hot/cold Pattern
History table archiving

4. Upgrading to SQL Server 2016
Related content
BRK3043: Upgrade to Microsoft SQL Server 2016: it just runs faster
BRK3282: Dive deep into application performance using Query Store in SQL Server 2016 and Azure SQL Database
BRK2007: Accelerate SQL Server 2016 HTAP performance with Windows and HPE
Upgrading to SQL Server 2016

5. New Features only in Compatibility Level 130
Microsoft Ignite 2016
6/12/2018 5:05 AM
New Features only in Compatibility Level 130
Parallelism
Parallelism in some operators for memory optimized tables
Parallel INSERT…SELECT
Batch mode enhancements for Columnstore tables
Batch Mode Sort
Batch Window Aggregates
Batch Mode even with serial plans
Some cardinality estimation changes
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

6. Compatibility Level Considerations
Microsoft Ignite 2016
6/12/2018 5:05 AM
Compatibility Level Considerations
Upgrade using older DB compatibility level 110/120
Enable Query Store
Execute a test workload to establish baseline
Change the compatibility level to 130
Re-run workload to compare with baseline
Use Query Store to detect regressed Plans
Optionally, force query plans to restore performance
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

7. Columnstore Indexes: Learnings
Related content
BRK3030: Review Columnstore Index in SQL Server 2016 and Azure SQL DB
BRK2083: Hear customer success stories with Columnstore index in SQL Server 2016
BRK3071: Enable operational analytics in SQL Server and Azure SQL DB
Columnstore Indexes: Learnings

8. Columnstore Indexes in SQL 2016
Microsoft Ignite 2016
6/12/2018 5:05 AM
Columnstore Indexes in SQL 2016
Widely adopted in production during Preview Program.
Additional B-tree indexes for fast lookups
Primary/Foreign Key support
Granular Locking with NCI
CCI accessible on readable secondary
Performance Improvements
String predicate / Aggregate pushdown
Batch mode for many operators
Parallel insert-select
B-tree (NCI)
Mapping Index(internal)
Delete bitmap
Compressed RGs
Delta RGs
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

9. Columnstore Lessons: Converting Existing Table (1)
Microsoft Ignite 2016
6/12/2018 5:05 AM
Columnstore Lessons: Converting Existing Table (1)
Staging tables created with CCI
Fastest way to convert an existing table to minimize downtime.
Converting few partitions at a time
Using Staging Tables for parallel insert select
Convert via staging tables
Direct convert by partition
Insert select WITH (TABLOCK)
Fact Table
2010
2011
2012
2013
2014
2015
2016
New Table
2010
2011
2012
2013
2014
2015
2016
Fact Table
2010
2011
2012
2013
2014
2015
2016
New Table
2010
2011
2012
2013
2014
2015
2016
Insert select
SWITCH
Stg_2010
Stg_2011
Stg_2012
Stg_2013
Stg_2014
Stg_2015
Stg_2016
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

10. Converting Existing Table (2)
Microsoft Ignite 2016
6/12/2018 5:05 AM
Converting Existing Table (2)
Sample 3 billion row table conversion
Concurrent sessions loading data
Each Load is serial without staging table
Parallel insert select with Staging
Used TABLOCK
Bulk Logged operation
Each session is serial
Each session has parallel Plan with TABLOCK hint
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

11. Columnstore Data Loading: Lessons learned
Microsoft Ignite 2016
6/12/2018 5:05 AM
Columnstore Data Loading: Lessons learned
Concurrent loading is the way to go for large Loads
Computed columns need to be ‘moved’ to ETL as they are unsupported with Columnstore.
Concurrent TSQL MERGE statements: may need to hint the NCI to ensure key Locking
Primary Key constraints or Unique Indexes
ETL with SSIS - use SQL 2016 Data Flow Buffer Auto Sizing
Refer to Data Loading Guide, CCI data load considerations, Parallel Insert-Select for details
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

12. Demo: Columnstore Indexes
Parallel INSERT
SSIS: Automatic Buffer Size Adjustment

13. Spinlock Troubleshooting
Symptoms
High CPU usage not directly attributed to query workload
Workload not scaling (or even degrading!) with higher number of processors
SQL requests waiting on CPU time (‘runnable’ status)
Tools of the trade
Extended Events
Windows Performance Toolkit
Debugging symbols for SQL Server and Windows

14. Walkthrough: Scalability Troubleshooting
Microsoft Ignite 2016
6/12/2018 5:05 AM
Walkthrough: Scalability Troubleshooting
CMED_HASH_SET and SECURITY_CACHE spinlocks
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

15. Post-RTM scalability fixes
Microsoft Ignite 2016
6/12/2018 5:05 AM
Post-RTM scalability fixes
Description
KB Article
Available In
Contention for CMED_HASH_SET and SECURITY_CACHE spinlocks
CU2
SECURITY_CRYPTO_CONTEXT_MUTEX waits cause SQL Server 2016 to stop responding to client requests in SQL Server 2016
CU1
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

16. In-memory OLTP: Learnings
Related content
BRK3097: Review In-Memory OLTP in SQL Server 2016 and Azure SQL DB
BRK3103: Explore In- Memory OLTP architectures and customer case studies
BRK2231: Understand how ChannelAdvisor is using SQL Server 2016
In-memory OLTP: Learnings

17. In-Memory OLTP: Target Scenarios
OLTP apps with high throughput and low latency requirements
Extreme OLTP performance due to:
High scalability through lock- and latch-free architecture enables linear scaling
Low latency through efficient data access and query processing: memory-optimized data structures and native compilation
Integrated into SQL Server for low TCO
Same database can have in-memory and on-disk
Integrated developer and DBA experience: same T-SQL,
client stack, tooling, backup/restore, Always On Availabilty Groups

18. “The surgical strike”
Identify bottleneck tables
Validate schema is suitable and migrate data
Test with memory optimized tables
Identify performance critical TSQL code
Validate surface area limitations and migrate to Natively compiled
Staged approach in identifying candidate tables and procedures:
Tables should be identified based on bottlenecks solved by in-memory OLTP
Convert performance critical TSQL to natively compiled procedures.
Migrate existing (large) table data in chunks.
Test at high scale to see performance differences
IO Latency ( Log) & Network round trips do matter

19. In-Memory OLTP Lessons Learned: Indexing
Index type choice can make significant difference in performance.
Non-clustered indexes used for range lookups and non-selective columns
Are not bi-directional
Hash indexes for point lookups, Size bucket count appropriately
Full key required for composite key indexes in order to seek.
DML operations on in-memory tables results in a serial Plan.
Index Foreign Key columns, they are used for validation as well – details in Blog Post

20. In-Memory OLTP: Log Truncation difference
Automatic checkpoint and impact on clearing the log on a backup
Size based checkpoint (if 1.5GB log is used)
Manual checkpoint can be run at any time
Large Checkpoints:
Were on by default in RTM; OFF by default in CU1
Auto-detected based on hardware configuration
Data / Delta File pairs change from 128MB/16MB to 1GB/128MB
With Large checkpoints, size based log truncation limit was 12 GB ( 8* 1.5GB)

21. Demo: In-Memory OLTP Indexing Considerations Checkpoints
Natively compiled UDFs

22. Temporal: learnings Related content
BRK3151: Make it real with Always Encrypted, Stretch Database and Temporal Tables
BRK3152: Achieve developer productivity with SQL Server and Azure SQL Database
BRK3223: Understand how Attom Data Solutions is using SQL Server 2016 to accelerate their business
Temporal: learnings

23. Temporal tables temporal table (current data) history table
Microsoft Ignite 2015
6/12/2018 5:05 AM
Temporal tables
temporal table (current data)
history table
* Old versions
Insert / Bulk Insert / Select
Update */ Delete *
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

24. * Include historical versions
Microsoft Ignite 2015
6/12/2018 5:05 AM
Temporal queries
temporal table (current data)
history table
* Include historical versions
Temporal queries*: SELECT … AS OF time
Regular queries: SELECT …
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

25. Temporal tables ( With In-memory Tables)
Microsoft Ignite 2015
6/12/2018 5:05 AM
Temporal tables ( With In-memory Tables)
temporal table (current data)
history table
In-Memory
In-Memory - Internal
* Old versions
Async Data migration
Disk Based History table
Update */ Delete *
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

26. Temporal Tables: Hot/cold pattern
Fairly popular scenarios:
Trading Order Management: Active orders are in-memory and Closed orders are in history
Manufacturing Plant type telemetry: In-memory serves as Shock absorber
In-Memory tables with Temporal enable Hot/cold data management
Enables memory optimized tables to be smaller.
Delete on current table transparently moves deleted row to history table
Low impact on transactions (as long as Log drive I/O can keep up)

27. Temporal Tables: Indexing history table
Index and compress the On-disk history table based on query patterns.
Page compression if access pattern on history table is retrieving few rows or the entire row.
Evaluate Columnstore indexes if query pattern is primarily aggregate queries.
In-memory internal ‘history’ table cannot have additional indexes as of CU2
Default index is based on SysEnd/SysStart time and Primary Key
Data Flush interval cannot be automatically controlled currently
Can be flushed using sys.sp_xtp_flush_temporal_history

28. Temporal Tables: Archival
Partition the history table to aid in History archival
Delete of data does require setting system versioning to be turned OFF
Switch out of a partition does not require setting system versioning OFF
Can use managed lock priority when swapping out for On-disk tables

29. Demo: Temporal Tables
In-Memory OLTP + Temporal
History archival

30. In Closing
SQL Server 2016 is the most extensively validated release of SQL – ever!
You should consider upgrading sooner than later, don’t wait for Service Pack N!
Real-world customers had rich benefits from the significant improvements in SQL Server 2016
In-memory OLTP, Columnstore, R Services, Security features have seen extensive preview participation
Your feedback and inputs are critical for us to keep improving SQL Server moving ahead
Engage with us: and
Use Connect to log feedback:

31. Please evaluate this session
6/12/2018 5:05 AM
Please evaluate this session
Your feedback is important to us!
From your PC or Tablet visit MyIgnite at
From your phone download and use the Ignite Mobile App by scanning the QR code above or visiting
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

32. Free IT Pro resources To advance your career in cloud technology
Microsoft Ignite 2016
6/12/2018 5:05 AM
Free IT Pro resources To advance your career in cloud technology
Plan your career path
Microsoft IT Pro Career Center
Cloud role mapping
Expert advice on skills needed
Self-paced curriculum by cloud role
$300 Azure credits and extended trials
Pluralsight 3 month subscription (10 courses)
Phone support incident
Weekly short videos and insights from Microsoft’s leaders and engineers
Connect with community of peers and Microsoft experts
Get started with Azure
Microsoft IT Pro Cloud Essentials
Demos and how-to videos
Microsoft Mechanics
Connect with peers and experts
Microsoft Tech Community
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

33. 6/12/2018 5:05 AM
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.