1. We Have Found Nirvana with Online Dump and Load (224)
Mike Furgal Director Database and Pro2 Services June 28th, 2016
Progress

2. This session will describe the process of using Table Partitioning to be able to dump and load a table with little to no interruption to the application or their users.

3. There will also be a Demo

4. Why Dump and Load
Type II storage format has lessened the need to dump and load
But did not get rid of it
Recommendation is to dump and load every 3 to 5 years
Or when you need it
3 Types Fragmentation
Record Fragmentation
Physical Table Fragmentation
Logical Table Fragmentation
Type II storage eliminates Physical Table Fragmentation
Type II storage may help Record Fragmentation
Type II storage does little for Logical Table Fragmentation

5. Database Blocks in a Type II cluster
Index Leaf Block
Index Keys
Local Scatter K1 K2 K3 K4 K5 K6 K7 K8
Database Blocks in a Type II cluster

6. Index is fastest when: Index order matches storage order

7. Database Blocks in a Type II cluster
Index Leaf Block
Index Keys
Local Scatter K1 K2 K3 K4 K5 K6 K7 K8
Database Blocks in a Type II cluster

8. This is a 4x performance improvement for Record Reads
Scatter Matters
Dump Test
Data Dump Time
Logical Scattered Database where > 10% of the rows are not contiguous
44 Minutes, 52 Seconds
No Logical Scatter
(index matches storage order)
10 Minutes, 17 Seconds
This is a 4x performance improvement for Record Reads

9. Periodocially Dump and Load to maintain the best performance
So what can you do?
Periodocially Dump and Load to maintain the best performance

10. Binary dump and load requires downtime
But:
Binary dump and load requires downtime
maybe a whole day

11. Binary dump and load requires downtime
maybe a whole day

12. You can do it with only a minute or three of downtime

13. The secret
Use Table Partitioning to move data between storage areas while the application is running. This essentially performs a dump and load.

15. Add an integer field with initial value 0
Details
Add an integer field with initial value 0
Add an index on this new field
Enable Table Partitioning on the database
Make a list partition on this field
Value of 0 is in the current storage area
Value of 1 is in a different storage area
Run a program to change all the values of the new field to 1 to migratte the data

16. Demo
Create a demo database
Add 1,000,000 rows to the database
Make it fragmented
Add an integer field “myarea”
Add an index to myarea
Enable Table Partitioning
Run an application
Move the data
Check the results

17. set schema 'pub’; alter table pub
set schema 'pub’; alter table pub.customer partition by list myarea using table area "Customer/Order Area” ( partition "Initial" values in (0) using table area "Customer/Order Area” ) using index "myarea"; commit;
alter table pub.customer add partition "newpartition" values in (1) using table area "Newcust"; commit;
quit;

18. DEMO

19. TA-DA!
But wait… There’s more

20. Move was done with no wait time and 1,000 rows per transaction
ATM Benchmark
Account Table
240,000,000 Rows
Database Size
35 GB
Build Time
3 Hours, 15 Minutes
No partitioning
3,329 TPS
With paritioning
3,307 TPS
While “table move” running
2,607 TPS (21% performance loss)
Table Move Time
8 Hours, 30 Minutes
After “table move” complete
3,634 TPS (9% performance gain)
Move was done with no wait time and 1,000 rows per transaction

21. Does the table already need to be in Type 2 Storage to do this?
YES! Table Partitioning requires the tables being partitioned to be in Type 2 format

22. Do I need to recompile the application after I add the field and table?
No You can add the field and index and since the application is not using it, it does not need to be recompiled
YMMV

23. Can I control which index is used for the table move?
Yes! This is the best part. Since you are writing the program to move the data, you can do it anyway you want

24. def var n as integer no-undo.
find first customer.
mainloop:
do while (true) transaction:
do n = 1 to 1000:
if not available (customer) then leave mainloop.
customer.myarea = 1.
find next customer.
end.

25. How can I tell if all the data is moved?
Use the proutil db -C partitionmanage view table customer status command

26. Can I pause it? Can I abort it? Can I restart it?
YES YES YES

27. Can I Multi Thread the move?
Yes, you have total control, but be aware of the impact to the production environment

28. What is the impact to BI and AI
Large – Each row is moved, then the index entry is updated. You get to control the transaction size for BI growth. You have no control over AI growth.

30. Do I need to Rebaseline OE Replication once it is complete?

31. Can dump and load without downtime
PROS
CONS
Can dump and load without downtime
Can control how much impact the process will have on production
Can control how much to do at any time
Has little to no impact on the running production system
Need to add an index which can be time consuming
The result will have 2 large storage areas
The old area where the data came from and the new one where it now resides

32. Online dump and load exists! It requires Table Partitioning
Summary
Online dump and load exists!
It requires Table Partitioning
Do this in a test environment first
Make sure you get it right
Understand the duration and performance impact