We Have Found Nirvana with Online Dump and Load (224) Mike Furgal Director Database and Pro2 Services June 28th, 2016 Progress

This session will describe the process of using Table Partitioning to be able to dump and load a table with little to no interruption to the application or their users.

There will also be a Demo

Why Dump and Load Type II storage format has lessened the need to dump and load But did not get rid of it Recommendation is to dump and load every 3 to 5 years Or when you need it 3 Types Fragmentation Record Fragmentation Physical Table Fragmentation Logical Table Fragmentation Type II storage eliminates Physical Table Fragmentation Type II storage may help Record Fragmentation Type II storage does little for Logical Table Fragmentation

Database Blocks in a Type II cluster Index Leaf Block Index Keys Local Scatter K1 K2 K3 K4 K5 K6 K7 K8 Database Blocks in a Type II cluster

Index is fastest when: Index order matches storage order

Database Blocks in a Type II cluster Index Leaf Block Index Keys Local Scatter K1 K2 K3 K4 K5 K6 K7 K8 Database Blocks in a Type II cluster

This is a 4x performance improvement for Record Reads Scatter Matters Dump Test Data Dump Time Logical Scattered Database where > 10% of the rows are not contiguous 44 Minutes, 52 Seconds No Logical Scatter (index matches storage order) 10 Minutes, 17 Seconds This is a 4x performance improvement for Record Reads

Periodocially Dump and Load to maintain the best performance So what can you do? Periodocially Dump and Load to maintain the best performance

Binary dump and load requires downtime But: Binary dump and load requires downtime maybe a whole day

Binary dump and load requires downtime maybe a whole day

You can do it with only a minute or three of downtime

The secret Use Table Partitioning to move data between storage areas while the application is running. This essentially performs a dump and load.

Add an integer field with initial value 0 Details Add an integer field with initial value 0 Add an index on this new field Enable Table Partitioning on the database Make a list partition on this field Value of 0 is in the current storage area Value of 1 is in a different storage area Run a program to change all the values of the new field to 1 to migratte the data

Demo Create a demo database Add 1,000,000 rows to the database Make it fragmented Add an integer field “myarea” Add an index to myarea Enable Table Partitioning Run an application Move the data Check the results

set schema 'pub’; alter table pub set schema 'pub’; alter table pub.customer partition by list myarea using table area "Customer/Order Area” ( partition "Initial" values in (0) using table area "Customer/Order Area” ) using index "myarea"; commit; alter table pub.customer add partition "newpartition" values in (1) using table area "Newcust"; commit; quit;

DEMO

TA-DA! But wait… There’s more

Move was done with no wait time and 1,000 rows per transaction ATM Benchmark Account Table 240,000,000 Rows Database Size 35 GB Build Time 3 Hours, 15 Minutes No partitioning 3,329 TPS With paritioning 3,307 TPS While “table move” running 2,607 TPS (21% performance loss) Table Move Time 8 Hours, 30 Minutes After “table move” complete 3,634 TPS (9% performance gain) Move was done with no wait time and 1,000 rows per transaction

Does the table already need to be in Type 2 Storage to do this? YES! Table Partitioning requires the tables being partitioned to be in Type 2 format

Do I need to recompile the application after I add the field and table? No You can add the field and index and since the application is not using it, it does not need to be recompiled YMMV

Can I control which index is used for the table move? Yes! This is the best part. Since you are writing the program to move the data, you can do it anyway you want

def var n as integer no-undo. find first customer. mainloop: do while (true) transaction: do n = 1 to 1000: if not available (customer) then leave mainloop. customer.myarea = 1. find next customer. end.

How can I tell if all the data is moved? Use the proutil db -C partitionmanage view table customer status command

Can I pause it? Can I abort it? Can I restart it? YES YES YES

Can I Multi Thread the move? Yes, you have total control, but be aware of the impact to the production environment

What is the impact to BI and AI Large – Each row is moved, then the index entry is updated. You get to control the transaction size for BI growth. You have no control over AI growth.

Do I need to Rebaseline OE Replication once it is complete?

Can dump and load without downtime PROS CONS Can dump and load without downtime Can control how much impact the process will have on production Can control how much to do at any time Has little to no impact on the running production system Need to add an index which can be time consuming The result will have 2 large storage areas The old area where the data came from and the new one where it now resides

Online dump and load exists! It requires Table Partitioning Summary Online dump and load exists! It requires Table Partitioning Do this in a test environment first Make sure you get it right Understand the duration and performance impact