1. Working with Very Large Tables Like a Pro in SQL Server 2017
Guy Glantser
Image courtesy of marin / FreeDigitalPhotos.net

2. Copyright © 2019 Madeira Ltd All Rights Reserved
Full rights, including copyrights, belong exclusively to Madeira Ltd
No use of the materials, in any form, is allowed,
unless receiving a prior written permission from Madeira Ltd

3. A Few Words about Me… Name: Guy Glantser
Address:
Twitter: @guy_glantser
Blog:
Podcast:
Image courtesy of Mister GC / FreeDigitalPhotos.net

4. SPONSORS Loyalty Sponsors Gold Sponsors Silver Sponsors
Bronze Sponsors

5. http://bit.ly/2Q9t3ku http://bit.ly/2QePNzE Evaluations
EVENT EVALUATION LINK
SESSIONS EVALUATION LINK
Evaluations

6. Working with Very Large Tables Like a Pro in SQL Server 2017
Agenda
Working with Very Large Tables Like a Pro in SQL Server 2017
Image courtesy of marin / FreeDigitalPhotos.net

7. Working with Very Large Tables Like a Pro in SQL Server 2017
Agenda
Working with Very Large Tables Like a Pro in SQL Server 2017
Image courtesy of marin / FreeDigitalPhotos.net

8. Working with Very Large Tables Like a Pro in SQL Server 2017
Agenda
Working with Very Large Tables Like a Pro in SQL Server 2017
Image courtesy of marin / FreeDigitalPhotos.net

9. Working with Very Large Tables Like a Pro in SQL Server 2017
Agenda
Working with Very Large Tables Like a Pro in SQL Server 2017
Image courtesy of marin / FreeDigitalPhotos.net

10. Working with Very Large Tables Like a Pro in SQL Server 2017
Agenda
Working with Very Large Tables Like a Pro in SQL Server 2017
Image courtesy of marin / FreeDigitalPhotos.net

11. So What is a Very Large Table?
5GB
5TB
Images courtesy of satit_srihin & John Kasawa / FreeDigitalPhotos.net

12. So What is a Very Large Table?
It depends…
Image courtesy of David Castillo Dominici / FreeDigitalPhotos.net

13. So What is a Very Large Table?
is One that Requires
Special Care
Image courtesy of stockimages / FreeDigitalPhotos.net

14. So What is a Very Large Table?
File.Events
Web.PageViews
Audit.Messages
Monitor.Measures
Billing.Transactions

15. So What is a Very Large Table?
Web.PageViews

16. Working with Very Large Tables

17. Working with Very Large Tables
Manage the Table
Image courtesy of xedos4 / FreeDigitalPhotos.net

18. Working with Very Large Tables
Manage the Table
Query the Table
Image courtesy of Master isolated images / FreeDigitalPhotos.net

19. Working with Very Large Tables
Manage the Table
Query the Table
Update the Table
Image courtesy of Stuart Miles / FreeDigitalPhotos.net

20. Working with Very Large Tables
Manage the Table
Query the Table
Update the Table
Maintain the Table
Image courtesy of Stuart Miles / FreeDigitalPhotos.net

21. How do We Manage the Table?
Image courtesy of xedos4 / FreeDigitalPhotos.net

22. Table Partitioning Facilitates Data Archiving Scenarios
Optionally, Spread Partitions Between Storage Tiers
Improves Table Maintenance (More of That Later…)
Does not Necessarily Improve Query Performance
Image courtesy of xedos4 / FreeDigitalPhotos.net

23. Stretching the Table to the Cloud

24. Stretching the Table to the Cloud
Run the Data Migration Assistant
Enable Remote Data Archive at the Instance Level
Choose/Create an Azure Server
Create a Firewall Rule for Your On-Prem SQL Server

25. Stretching the Table to the Cloud
Create a Database Master Key
Create a Database Scoped Credential
Enable Remote Data Archive at the Database Level
Optionally Create a Filter Function
Enable Remote Data Archive at the Table Level

26. Now Let’s Query the Table…
Image courtesy of Master isolated images / FreeDigitalPhotos.net

27. The Ascending Key Problem
Image courtesy of suphakit73 / FreeDigitalPhotos.net

28. Problem #1 – Solutions Update Statistics Manually Trace Flag 2371
New Cardinality Estimator
Filtered Statistics
Incremental Statistics
Image courtesy of suphakit73 / FreeDigitalPhotos.net

29. What about the Data Load?
Image courtesy of Stuart Miles / FreeDigitalPhotos.net

30. Problem #2 The WRITELOG Wait Type
Image courtesy of stockimages / FreeDigitalPhotos.net

31. Problem #2 – Solution Delayed Durability
Image courtesy of stockimages / FreeDigitalPhotos.net

32. Delayed Durability
Transaction Properties (ACID):
Atomicity
Consistency
Isolation
Durability

33. Fully-Durable Transactions
Delayed Durability
Data File
Buffer Pool
Commit
Checkpoint
Request
Transaction Log Buffer
Log File
WRITELOG

34. Delayed Durability Transactions
Data File
Buffer Pool
Commit
Checkpoint
Request
Transaction Log Buffer
Log File

35. Delayed Durability The log buffer is flushed to disk when:
A fully durable transaction in the same database makes a change in the database and successfully commits
Image courtesy of Keerati / FreeDigitalPhotos.net

36. Delayed Durability The log buffer is flushed to disk when:
The user executes the system stored procedure “sys.sp_flush_log” successfully
Image courtesy of Keerati / FreeDigitalPhotos.net

37. Delayed Durability The log buffer is flushed to disk when:
The in-memory transaction log buffer fills up and automatically flushes to disk
Image courtesy of Keerati / FreeDigitalPhotos.net

38. Delayed Durability The log buffer is flushed to disk when:
Every few milliseconds
Image courtesy of Keerati / FreeDigitalPhotos.net

39. Delayed Durability Can be controlled by:
Database (Disabled | Allowed | Forced)
Transaction (Commit)
Atomic Block (Natively Compiled Procedure)
Image courtesy of digitalart / FreeDigitalPhotos.net

40. Delayed Durability
The following transactions are always fully durable:
Transactions that perform write operations against tables that are enabled for either Change Tracking or Change Data Capture
Image courtesy of Iamnee / FreeDigitalPhotos.net

41. Delayed Durability
The following transactions are always fully durable:
Cross-database transactions or distributed transactions
Image courtesy of Iamnee / FreeDigitalPhotos.net

42. Delayed Durability
The following features apply only to transactions that have been made durable:
Transaction Replication – Transactions are only replicated after they have been made durable
Image courtesy of Stuart Miles / FreeDigitalPhotos.net

43. Delayed Durability
The following features apply only to transactions that have been made durable:
Log Backup – Only transactions that have been made durable are included in the backup
Image courtesy of Stuart Miles / FreeDigitalPhotos.net

44. Delayed Durability
The following features apply only to transactions that have been made durable:
Log Shipping – Only transactions that have been made durable are shipped to the secondaries
Image courtesy of Stuart Miles / FreeDigitalPhotos.net

45. Delayed Durability What are the benefits?
Less roundtrips to the log file on disk
Less WRITELOG waits
Shorter transactions
Locks are released earlier
Better throughput
Image courtesy of Ambro / FreeDigitalPhotos.net

46. Delayed Durability
But remember that
you might lose data!

47. Finally, Let’s Talk about Maintenance…
Image courtesy of xedos4 / FreeDigitalPhotos.net

48. Problem #3 Index Rebuild
Image courtesy of suphakit73 / FreeDigitalPhotos.net

49. Index Rebuild Before SQL Server 2014 we could:
Rebuild all partitions offline
Rebuild all partitions online
Rebuild a specific partition offline
Image courtesy of David Castillo Dominici / FreeDigitalPhotos.net

50. Index Rebuild Since SQL Server 2014 we can:
Rebuild a specific partition online
Wait at low priority when:
Rebuilding an index online
Switching partitions
Image courtesy of David Castillo Dominici / FreeDigitalPhotos.net

51. Index Rebuild Since SQL Server 2017 we can:
Rebuild an index online and also pause and resume it
Image courtesy of David Castillo Dominici / FreeDigitalPhotos.net

52. Wait at Low Priority Wait in a separate low priority queue
Wait for a specified number of minutes
Image courtesy of ddpavumba / FreeDigitalPhotos.net

53. Wait at Low Priority After that, choose one of 3 options:
Continue to wait at normal priority
Abort the operation
Kill all blockers and start the operation
Image courtesy of ddpavumba / FreeDigitalPhotos.net

54. Wait at Normal Priority
Queue
Insert
Insert
Online
Index Rebuild
Running
Image courtesy of artur84 / FreeDigitalPhotos.net

55. Wait at Low Priority “Normal” Queue Insert Insert Online Index Rebuild
Running
Low Priority
Queue
Image courtesy of artur84 / FreeDigitalPhotos.net

56. Working with Very Large Tables Like a Pro in SQL Server 2017
Summary
Working with Very Large Tables Like a Pro in SQL Server 2017
Image courtesy of marin / FreeDigitalPhotos.net

57. Working with Very Large Tables Like a Pro in SQL Server 2017
Summary
Working with Very Large Tables Like a Pro in SQL Server 2017
Image courtesy of marin / FreeDigitalPhotos.net

58. Summary A Very Large Table is One that Requires Special Care
Image courtesy of stockimages / FreeDigitalPhotos.net

59. Working with Very Large Tables Like a Pro in SQL Server 2017
Summary
Working with Very Large Tables Like a Pro in SQL Server 2017
Image courtesy of marin / FreeDigitalPhotos.net

60. Summary Manage the Table Query the Table Update the Table
Maintain the Table
Image courtesy of Master isolated images, Stuart Miles & xedos4 / FreeDigitalPhotos.net

61. Working with Very Large Tables Like a Pro in SQL Server 2017
Summary
Working with Very Large Tables Like a Pro in SQL Server 2017
Image courtesy of marin / FreeDigitalPhotos.net

62. Summary Solving the Ascending Key problem Update Statistics Manually
Trace Flag 2371
New Cardinality Estimator
Filtered Statistics
Incremental Statistics
Image courtesy of suphakit73 / FreeDigitalPhotos.net

63. Summary Solving the WRITELOG problem with Delayed Durability
Less roundtrips to the log file on disk
Less WRITELOG waits
Shorter transactions & Locks
Better throughput
Image courtesy of Ambro / FreeDigitalPhotos.net

64. Summary Solving the Index Rebuild problem
Rebuild a specific partition online
Resumable online index rebuild
Wait at low priority
Image courtesy of suphakit73 / FreeDigitalPhotos.net

65. Working with Very Large Tables Like a Pro in SQL Server 2017
Summary
Working with Very Large Tables Like a Pro in SQL Server 2017
Image courtesy of marin / FreeDigitalPhotos.net

66. Summary New Features Since SQL Server 2014 Stretch Database
New Cardinality Estimator
Incremental Statistics
Delayed Durability
Online Indexing Enhancements

67. Additional Things to Explore
Compress the Data
Optimize Parallelism
Use a Lower Isolation Level
Leverage Columnstore Indexes
Intelligent Query Processing
Image courtesy of kdshutterman / FreeDigitalPhotos.net

68. Keep In Touch… Name: Guy Glantser Email Address: guy@madeiradata.com
Twitter: @guy_glantser
Blog:
Podcast:
Image courtesy of Nuttapong / FreeDigitalPhotos.net

69. http://bit.ly/2Q9t3ku http://bit.ly/2QePNzE Evaluations
EVENT EVALUATION LINK
SESSIONS EVALUATION LINK
Evaluations

70. SPONSORS Loyalty Sponsors Gold Sponsors Silver Sponsors
Bronze Sponsors

71. Image courtesy of David Castillo Dominici / FreeDigitalPhotos.net