Working with Very Large Tables Like a Pro in SQL Server 2017 Guy Glantser Image courtesy of marin / FreeDigitalPhotos.net

Copyright © 2019 Madeira Ltd All Rights Reserved Full rights, including copyrights, belong exclusively to Madeira Ltd No use of the materials, in any form, is allowed, unless receiving a prior written permission from Madeira Ltd

A Few Words about Me… Name: Guy Glantser Email Address: guy@madeiradata.com Twitter: @guy_glantser Blog: www.linkedin.com/in/glantser/detail/recent-activity/posts Podcast: www.sqlserverradio.com Image courtesy of Mister GC / FreeDigitalPhotos.net

SPONSORS Loyalty Sponsors Gold Sponsors Silver Sponsors Bronze Sponsors

http://bit.ly/2Q9t3ku http://bit.ly/2QePNzE Evaluations EVENT EVALUATION LINK SESSIONS EVALUATION LINK Evaluations http://bit.ly/2Q9t3ku http://bit.ly/2QePNzE

Working with Very Large Tables Like a Pro in SQL Server 2017 Agenda Working with Very Large Tables Like a Pro in SQL Server 2017 Image courtesy of marin / FreeDigitalPhotos.net

Working with Very Large Tables Like a Pro in SQL Server 2017 Agenda Working with Very Large Tables Like a Pro in SQL Server 2017 Image courtesy of marin / FreeDigitalPhotos.net

Working with Very Large Tables Like a Pro in SQL Server 2017 Agenda Working with Very Large Tables Like a Pro in SQL Server 2017 Image courtesy of marin / FreeDigitalPhotos.net

Working with Very Large Tables Like a Pro in SQL Server 2017 Agenda Working with Very Large Tables Like a Pro in SQL Server 2017 Image courtesy of marin / FreeDigitalPhotos.net

Working with Very Large Tables Like a Pro in SQL Server 2017 Agenda Working with Very Large Tables Like a Pro in SQL Server 2017 Image courtesy of marin / FreeDigitalPhotos.net

So What is a Very Large Table? 5GB 5TB Images courtesy of satit_srihin & John Kasawa / FreeDigitalPhotos.net

So What is a Very Large Table? It depends… Image courtesy of David Castillo Dominici / FreeDigitalPhotos.net

So What is a Very Large Table? is One that Requires Special Care Image courtesy of stockimages / FreeDigitalPhotos.net

So What is a Very Large Table? File.Events Web.PageViews Audit.Messages Monitor.Measures Billing.Transactions

So What is a Very Large Table? Web.PageViews

Working with Very Large Tables

Working with Very Large Tables Manage the Table Image courtesy of xedos4 / FreeDigitalPhotos.net

Working with Very Large Tables Manage the Table Query the Table Image courtesy of Master isolated images / FreeDigitalPhotos.net

Working with Very Large Tables Manage the Table Query the Table Update the Table Image courtesy of Stuart Miles / FreeDigitalPhotos.net

Working with Very Large Tables Manage the Table Query the Table Update the Table Maintain the Table Image courtesy of Stuart Miles / FreeDigitalPhotos.net

How do We Manage the Table? Image courtesy of xedos4 / FreeDigitalPhotos.net

Table Partitioning Facilitates Data Archiving Scenarios Optionally, Spread Partitions Between Storage Tiers Improves Table Maintenance (More of That Later…) Does not Necessarily Improve Query Performance Image courtesy of xedos4 / FreeDigitalPhotos.net

Stretching the Table to the Cloud

Stretching the Table to the Cloud Run the Data Migration Assistant Enable Remote Data Archive at the Instance Level Choose/Create an Azure Server Create a Firewall Rule for Your On-Prem SQL Server

Stretching the Table to the Cloud Create a Database Master Key Create a Database Scoped Credential Enable Remote Data Archive at the Database Level Optionally Create a Filter Function Enable Remote Data Archive at the Table Level

Now Let’s Query the Table… Image courtesy of Master isolated images / FreeDigitalPhotos.net

The Ascending Key Problem Image courtesy of suphakit73 / FreeDigitalPhotos.net

Problem #1 – Solutions Update Statistics Manually Trace Flag 2371 New Cardinality Estimator Filtered Statistics Incremental Statistics Image courtesy of suphakit73 / FreeDigitalPhotos.net

What about the Data Load? Image courtesy of Stuart Miles / FreeDigitalPhotos.net

Problem #2 The WRITELOG Wait Type Image courtesy of stockimages / FreeDigitalPhotos.net

Problem #2 – Solution Delayed Durability Image courtesy of stockimages / FreeDigitalPhotos.net

Delayed Durability Transaction Properties (ACID): Atomicity Consistency Isolation Durability

Fully-Durable Transactions Delayed Durability Data File Buffer Pool Commit Checkpoint Request Transaction Log Buffer Log File WRITELOG

Delayed Durability Transactions Data File Buffer Pool Commit Checkpoint Request Transaction Log Buffer Log File

Delayed Durability The log buffer is flushed to disk when: A fully durable transaction in the same database makes a change in the database and successfully commits Image courtesy of Keerati / FreeDigitalPhotos.net

Delayed Durability The log buffer is flushed to disk when: The user executes the system stored procedure “sys.sp_flush_log” successfully Image courtesy of Keerati / FreeDigitalPhotos.net

Delayed Durability The log buffer is flushed to disk when: The in-memory transaction log buffer fills up and automatically flushes to disk Image courtesy of Keerati / FreeDigitalPhotos.net

Delayed Durability The log buffer is flushed to disk when: Every few milliseconds Image courtesy of Keerati / FreeDigitalPhotos.net

Delayed Durability Can be controlled by: Database (Disabled | Allowed | Forced) Transaction (Commit) Atomic Block (Natively Compiled Procedure) Image courtesy of digitalart / FreeDigitalPhotos.net

Delayed Durability The following transactions are always fully durable: Transactions that perform write operations against tables that are enabled for either Change Tracking or Change Data Capture Image courtesy of Iamnee / FreeDigitalPhotos.net

Delayed Durability The following transactions are always fully durable: Cross-database transactions or distributed transactions Image courtesy of Iamnee / FreeDigitalPhotos.net

Delayed Durability The following features apply only to transactions that have been made durable: Transaction Replication – Transactions are only replicated after they have been made durable Image courtesy of Stuart Miles / FreeDigitalPhotos.net

Delayed Durability The following features apply only to transactions that have been made durable: Log Backup – Only transactions that have been made durable are included in the backup Image courtesy of Stuart Miles / FreeDigitalPhotos.net

Delayed Durability The following features apply only to transactions that have been made durable: Log Shipping – Only transactions that have been made durable are shipped to the secondaries Image courtesy of Stuart Miles / FreeDigitalPhotos.net

Delayed Durability What are the benefits? Less roundtrips to the log file on disk Less WRITELOG waits Shorter transactions Locks are released earlier Better throughput Image courtesy of Ambro / FreeDigitalPhotos.net

Delayed Durability But remember that you might lose data!

Finally, Let’s Talk about Maintenance… Image courtesy of xedos4 / FreeDigitalPhotos.net

Problem #3 Index Rebuild Image courtesy of suphakit73 / FreeDigitalPhotos.net

Index Rebuild Before SQL Server 2014 we could: Rebuild all partitions offline Rebuild all partitions online Rebuild a specific partition offline Image courtesy of David Castillo Dominici / FreeDigitalPhotos.net

Index Rebuild Since SQL Server 2014 we can: Rebuild a specific partition online Wait at low priority when: Rebuilding an index online Switching partitions Image courtesy of David Castillo Dominici / FreeDigitalPhotos.net

Index Rebuild Since SQL Server 2017 we can: Rebuild an index online and also pause and resume it Image courtesy of David Castillo Dominici / FreeDigitalPhotos.net

Wait at Low Priority Wait in a separate low priority queue Wait for a specified number of minutes Image courtesy of ddpavumba / FreeDigitalPhotos.net

Wait at Low Priority After that, choose one of 3 options: Continue to wait at normal priority Abort the operation Kill all blockers and start the operation Image courtesy of ddpavumba / FreeDigitalPhotos.net

Wait at Normal Priority Queue Insert Insert Online Index Rebuild Running Image courtesy of artur84 / FreeDigitalPhotos.net

Wait at Low Priority “Normal” Queue Insert Insert Online Index Rebuild Running Low Priority Queue Image courtesy of artur84 / FreeDigitalPhotos.net

Working with Very Large Tables Like a Pro in SQL Server 2017 Summary Working with Very Large Tables Like a Pro in SQL Server 2017 Image courtesy of marin / FreeDigitalPhotos.net

Working with Very Large Tables Like a Pro in SQL Server 2017 Summary Working with Very Large Tables Like a Pro in SQL Server 2017 Image courtesy of marin / FreeDigitalPhotos.net

Summary A Very Large Table is One that Requires Special Care Image courtesy of stockimages / FreeDigitalPhotos.net

Working with Very Large Tables Like a Pro in SQL Server 2017 Summary Working with Very Large Tables Like a Pro in SQL Server 2017 Image courtesy of marin / FreeDigitalPhotos.net

Summary Manage the Table Query the Table Update the Table Maintain the Table Image courtesy of Master isolated images, Stuart Miles & xedos4 / FreeDigitalPhotos.net

Working with Very Large Tables Like a Pro in SQL Server 2017 Summary Working with Very Large Tables Like a Pro in SQL Server 2017 Image courtesy of marin / FreeDigitalPhotos.net

Summary Solving the Ascending Key problem Update Statistics Manually Trace Flag 2371 New Cardinality Estimator Filtered Statistics Incremental Statistics Image courtesy of suphakit73 / FreeDigitalPhotos.net

Summary Solving the WRITELOG problem with Delayed Durability Less roundtrips to the log file on disk Less WRITELOG waits Shorter transactions & Locks Better throughput Image courtesy of Ambro / FreeDigitalPhotos.net

Summary Solving the Index Rebuild problem Rebuild a specific partition online Resumable online index rebuild Wait at low priority Image courtesy of suphakit73 / FreeDigitalPhotos.net

Working with Very Large Tables Like a Pro in SQL Server 2017 Summary Working with Very Large Tables Like a Pro in SQL Server 2017 Image courtesy of marin / FreeDigitalPhotos.net

Summary New Features Since SQL Server 2014 Stretch Database New Cardinality Estimator Incremental Statistics Delayed Durability Online Indexing Enhancements

Additional Things to Explore Compress the Data Optimize Parallelism Use a Lower Isolation Level Leverage Columnstore Indexes Intelligent Query Processing Image courtesy of kdshutterman / FreeDigitalPhotos.net

Keep In Touch… Name: Guy Glantser Email Address: guy@madeiradata.com Twitter: @guy_glantser Blog: www.linkedin.com/in/glantser/detail/recent-activity/posts Podcast: www.sqlserverradio.com Image courtesy of Nuttapong / FreeDigitalPhotos.net

http://bit.ly/2Q9t3ku http://bit.ly/2QePNzE Evaluations EVENT EVALUATION LINK SESSIONS EVALUATION LINK Evaluations http://bit.ly/2Q9t3ku http://bit.ly/2QePNzE

SPONSORS Loyalty Sponsors Gold Sponsors Silver Sponsors Bronze Sponsors

Image courtesy of David Castillo Dominici / FreeDigitalPhotos.net