Download presentation
Presentation is loading. Please wait.
Published byRuby Powers Modified over 6 years ago
1
Parallel Database Maintenance with 24/7 Systems and Huge DBs
SQLBits 2018
2
Grzegorz Łyp MS SQL Architect & Optimizer
MS SQL, .NET & Java Developer Product and Project Manager 15 years in professional IT 10 years in MS SQL Member of Data Community Poland Blog: Mail: LinkedIn: Facebook:
3
Agenda Background Database Maintenance Speedup maintenance tasks
Locking during database maintenance Creating specific statistics
4
DEMOS WorldWideImporters
world-importers-v1.0
5
Background Once upon a time there was… a database
At the moment of start it worked for 8 hours a day… DBA had rest of the time for maintenance
6
Background As the database owner company grew database usage grew
After certain amount of time the maintenace time went down to „0” DBA lost his job as he could not do the database maintenance in such conditions Did he try hard ? Let’s try to satisfy 24/7 system maintenance needs!
7
Database Maintenance Indexes Statistics
8
Indexes Index fragmentation
What is the actual value of fragmentation ? Is fragmentation bad or good ? DEMO 1.sql
9
Indexes
10
Indexes Page splitting does not directly depend on index fragmentation
Read ahead is off when reading data from pages that are not neighbours Data size necessary on physical storage and in memory higher for higher fragmentation avg_fragmentation_in_percent is not enough for saving space - avg_page_space_user_in_percent gives additional details Typically we assume that at 5-20% of fragmentation REORGANIZE index and over 20% we should REBUILD
11
REORGANIZE VS REBUILD REORGANIZE PROS REBUILD PROS Non-blocking
Faster than reorganize No extra space required Rebuilds not only leaf part of index Non-blocking when ONLINE (Enterprise Only) Can run on multiple processors REORGANIZE CONS REBUILD CONS Slow Blocking (ONLINE OFF) Only leaf level part of the index Slow ONLINE mode Needs a lot of space
12
Conslusions Gathering information about fragmentation can be slow
REBUILD can only run on Enterprise systems with ONLINE = ON and need a lot of free space REOGRANIZE is slow but it is not blocking even in standard edition It can block on SQL Server 2014 SP2 (fixed in CU) How to speed up index fragmentation gathering process ? How to speed up REORGANIZE ?
13
Statistics Settings Manual statistics maintenance DEMO
Auto Create Statistics – very dangerous in case of unexpected statistics You can’t see which statistics are used in query plan before SQL 2017 Auto Update Statistics – unpredictible moment of statistics update Causes recompilations and possible changes in execution plans For big tables refresh is too rare – especially for estimator before SQL 2014 Same for Auto Update Statistics Asynchronously Manual statistics maintenance Periodical Updating statistics can take some time Default sample for update can be not enough for rare records DEMO 2.sql
14
Conclusions Auto Create Statistics Off Auto Update Statistics Off
Periodical statistics update (even everyday) How to get proper sample ? How to reduce time of statistics update ?
15
How to get proper update statistics sample ?
Important for seeking with direct values All seek values should be in the sample For optimizer 120 or higher values out of statistics are not of 0 cardinality when no direct value is used, but it might be not enough
16
Speedup Maintenance Tasks
17
Typical maintenance process
Iterate across indexes For each index Get index fragmenation Defragment if necessary (REORGANIZE or REBUILD) Update statistics if necessary
18
Proposed maintenace process
Iterate across indexes and make queue Spread across number of threads each index in queue Single thread task Get index fragmenation Defragment if necessary Update statistics if necessary
19
Proposed maintenace process
1 2 3 4 5 6 7 8 9 10 11 12 13 TIME
20
Proposed maintenace proces - queue
1 2 3 4 5 6 7 8 9 10 11 12 13
21
Proposed maintenace proces 4 threads timing
1 5 9 13 Thread 2 2 6 10 Thread 3 3 7 8 12 Thread 4 4 11 TIME
22
Proposed maintenace proces 4 threads timing vs Single thread
1 5 9 13 Thread 2 2 6 10 Thread 3 3 7 8 12 Thread 4 4 11 TIME Single thread 1 2 3 4 5 6 7 8 9 10 11 12 13 TIME
23
Proposed maintenace proces – queue – order by length
13 4 11 3 5 6 9 10 1 2 12 7 8
24
Proposed maintenace proces 4 threads timing – queue ordered by length
13 10 7 Thread 2 4 9 12 Thread 3 11 6 2 Thread 4 3 5 1 8 TIME
25
Proposed maintenace proces 4 threads timing – comparison
13 10 7 Thread 2 4 9 12 Thread 3 11 6 2 Thread 4 3 5 1 8 TIME Thread 1 1 5 9 13 Thread 2 2 6 10 Thread 3 3 7 8 12 Thread 4 4 11 TIME
26
Proposed maintenace proces
DEMO 3.sql, 4.sql, 5.sql, 6.sql, 7.sql, 8.sql
27
Proposed maintenance proces profit
28
Proposed maintenance proces
Optimal number of threads can be calculated based on real index operation times The shortest possible time is the time of the longest index to maintain More threads are applicable as each index mainenance is uniform to each other
29
Proposed maintenace process – v2
Iterate across indexes and make queue Spread across number of threads each index in queue Single thread perform Get index fragmenation Store index fragmentation in the table Remove indexes below fragmentation limits Spread across numer of treads each index in queue Do index maintenance (fragmentation & statistics)
30
Proposed maintenace process – v2
Have to measure time for index fragmentation gathering and index maintenance separately Should we split index defragmentation and statistics update ? This might be version 3 – try yourself Is it better ?
31
Split maintenance task into parts
Read current fragmentation Do index defragmentation Update statistics 1 2 3 4 5 6 7 8 9 10 11 12 13 TIME
32
Three stages – better decisions
Thread 1 1 5 12 1 6 12 2 9 Thread 2 2 6 10 2 7 9 3 10 Thread 3 3 9 13 4 13 5 13 Thread 4 4 7 8 11 5 11 7 8 11 Fragmentation analysis Index defragmentation Statistics update
33
Proposed maintenace process – summary
Parallel maintenance shortens processing time Each maintenance task split to 2 or 3 stages for better resources use Operations time gather for better thread tasks allocation Process can be fully automated
34
Parallel tasks - implementation
35
Parallel tasks – how to implement ?
Jobs Need to create jobs No performance balance built-in Service broker Need to create service broker objects (queues, services) Number of parallel executions configurable and performance dependent Created requests treated as system requests (cannot kill ) CLR Unsafe assembly with multithread implementation Need to implement breaking method (killing does not work)
36
Service Broker – parallel tasks implementation
T-SQL Source Queue Destination Queue T-SQL T-SQL T-SQL 9.sql DEMO
37
Service Broker - conclusions
Easy to implement Some difficulties in error handling Parallelism controlled by server with MAX_QUEUE_READERS suggestion Faster thread allocation can be obtained with CLR
38
Conclusions
39
Conclusions Database index maintenance can consume time and resources
For 24/7 systems there are no technical breaks and every time we expect plan stability Typical maintenance proces is single-threaded Making maintenance multi-thread reduce time and make better use from resources The final solution can be parametrized according to specific hardware abilities (number of cores, storage type) We have non-blocking, adaptive, multi-threaded maintenace process that can work anytime
40
If we have more time… BONUS DEMO
41
Just like Jimi Hendrix …
We love to get feedback Please complete the session feedback forms
42
SQLBits - It's all about the community...
Please visit Community Corner, we are trying this year to get more people to learn about the SQL Community, equally if you would be happy to visit the community corner we’d really appreciate it.
43
Q&A Thank You for the attention
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.