Partition, Compress, Archive and Purge Keep Your System on the Road David Kurtz Go-Faster Consultancy Ltd.

Who Am I? Oracle Database Specialist –Independent consultant Performance tuning –PeopleSoft ERP –Oracle RDBMS Book – OakTable UKOUG PeopleSoft Partition, Compress, Archive, Purge ©

It depends… The answer to all questions should be deemed to be implicitly prefixed it depends if they are not already explicitly so prefixed. The reasons for this include, but are not limited to: –There may be exceptions where the answer is either false or not completely true –There may be exceptions to the exceptions and so on ad infinitum. –The question may not explicitly scope all the conditions upon which the answer depends © UKOUG PeopleSoft Partition, Compress, Archive, Purge

This happens to be a story about... An ERP System –But it could be about any on-line transaction processing system. A PeopleSoft system –But it could be about any ERP PeopleSoft HR/Time & Labour/Payroll –But it could be about any product UKOUG PeopleSoft Partition, Compress, Archive, Purge ©

PeopleSoft HR, T&L, Morrisons 130,000 employees –100,000 in stores Scheduled hours –Clock in/out –T&L matches reported hours with schedules to determine payable time –Legacy T&L feed from Manufacturing and Distribution. –Payable time passed to Global Payroll. UKOUG PeopleSoft Partition, Compress, Archive, Purge ©

UKOUG PeopleSoft Partition, Compress, Archive, Purge © Tb/yr

Two Challenges Performance –Big Tables (for OLTP) Eg payroll accumulator table 2.4 billion rows! –Poor/Variable Performance Various processes and reports suffering Critical Processes Database Size –>4TB –Growth 1TB/year –Backup Size/Contention –Cloning Dev/Test environments UKOUG PeopleSoft Partition, Compress, Archive, Purge ©

Throw Away When we say the database is slow, often: –It is fetching data from disk, reading it, and deciding it isnt something it wants and throwing it away. –It would be better if we had not spent time and resource fetching it in the first place The fastest way to do anything is not to do it at all. –Cary Millsap UKOUG PeopleSoft Partition, Compress, Archive, Purge ©

4 Techniques Partition –Breaking tables and indexes into managable pieces Compress –Squeezing more data into the same space Archive –Get the data out of the live application tables and put it somewhere else. Purge –Expunge the data from the database UKOUG PeopleSoft Partition, Compress, Archive, Purge ©

PARTITIONING UKOUG PeopleSoft Partition, Compress, Archive, Purge ©

Partitioning UKOUG PeopleSoft Partition, Compress, Archive, Purge ©

Partitioning decisions driven by Application Global Payroll –streamed processing –RANGE on EMPLID to match streaming Financials General Ledger –FISCAL_YEAR & ACCOUNTING_PERIOD –Monthly/Quarterly/Annual partitions. Other Financials Modules? –Less Obvious UKOUG PeopleSoft Partition, Compress, Archive, Purge ©

Partitioning is a Pre-requisite Links logical data value to physical location of data in the database. Time-based partitioning –Future / Current / Historical Partition-wise operations –Compress historical partitions –Archive by partition exchange –Purge by dropping whole partitions. –Less Redo. Faster. UKOUG PeopleSoft Partition, Compress, Archive, Purge ©

Delete rows –v- Drop Partition? Free space is left in data block. Data block remain part of table/index Still scanned by queries. Still backed up by RMAN No longer in a database object. So not referenced. Free space is available in tablespace. –Still backed up If the space at end of the datafile it can be trimmed off by resize. –No longer backed up UKOUG PeopleSoft Partition, Compress, Archive, Purge ©

Managing Partitioning in PeopleSoft Partitioning strategies –Described own meta-data tables PL/SQL Utility package –generate DDL scripts. –Driven by PeopleTools tables and our own meta-data Application Designer from PT8.51 will preserve some existing partitioning –See Managing Oracle Table Partitioning in PeopleSoft Applications with GFC_PSPART Package UKOUG PeopleSoft Partition, Compress, Archive, Purge ©

Partitioning PeopleSoft at Morrisons Global Payroll –46 tables range on EMPLID, matching 32 streams –List sub-partition largest tables by pay period (CAL_RUN_ID). –24 archive tables, range partitioned on pay period into tax-year partitions. –2 custom GP reporting tables Schedules & Time and Labor –10 tables in weekly range partitions –of which Reported Time table hash sub-partitioned Audit Tables –28 monthly range partitions on audit timestamp Security Tables –Separate partitions for different business units UKOUG PeopleSoft Partition, Compress, Archive, Purge ©

Partitioning Trick Timezone Table –Small, but accessed VERY frequently by T&L –1 partition for GMT only Only 1 row –1 partition for CET only Only 1 row –1 partition for the rest –Less CPU to read first row in block than second etc. UKOUG PeopleSoft Partition, Compress, Archive, Purge ©

Licencing of Partitioning Oracle RDBMS Enterprise Edition only –And then separately licenced Also a cost option on other database platforms. No licence implication for PeopleSoft. –Partitioning does not invalidate any aspect of your support. UKOUG PeopleSoft Partition, Compress, Archive, Purge ©

COMPRESSION UKOUG PeopleSoft Partition, Compress, Archive, Purge ©

Oracle RDBMS Compression Basic Compression –Not separately licenced –Only option in 10g Advanced Compression Out of scope of this presentation –Licenced option –Available in 11g Hybrid Columnar Compression (HCC) Also out of scope of this presentation. –Exadata and ZFS filesystem only. UKOUG PeopleSoft Partition, Compress, Archive, Purge ©

Simple Compression Block Dictionary Encoding –Data values stored only stored once in array in data block. –Pointer from row of data to array of values. Shorter Rows, more rows per block, fewer Blocks, less I/O More CPU to read row. UKOUG PeopleSoft Partition, Compress, Archive, Purge ©

Table Compression Specified at table or partition level with DDL –Sub-partition level inherits from partition level Compression only occurs –On creation of the object ALTER TABLE … COMPRESS –Direct path insert into object –CPU overhead during compression and read Decompression occurs –during any other DML UKOUG PeopleSoft Partition, Compress, Archive, Purge ©

Index Compression Apply to all or none of index Index remains compressed –DML does not decompress –Performance overhead We have not compressed many live application indexes. We have compressed all archive tables. Table compression does not affect index size. UKOUG PeopleSoft Partition, Compress, Archive, Purge ©

What do I compress? Depends on application, business, and partitioning strategy. –Audit data partitions: as soon as month ends –Payroll result data: when pay period is closed for 1 month –T&L: after 6 months UKOUG PeopleSoft Partition, Compress, Archive, Purge ©

Compression Gotcha! During –ALTER TABLE PARTITION … MOVE … COMPRESS. Local index partitions can momentarily become unusable before they go back to being usable. This can cause application errors –Choose when to compress carefully, or do it in an outage! UKOUG PeopleSoft Partition, Compress, Archive, Purge ©

When can I compress my data? When it is no longer updated by application. –Because DML uncompresses it –But that may be well be you can archive/purge. Written custom PL/SQL utility package –Meta-data rules to determine when partition no longer updated. –Needs knowledge of business/application. –Incrementally compress static partitions. UKOUG PeopleSoft Partition, Compress, Archive, Purge ©

How much compression? It depends on your data! We have see a factor 2x – 5x on tables –(50%-80% reduction in size) Indexes are less compressible –Indexes with more columns can be more compressible UKOUG PeopleSoft Partition, Compress, Archive, Purge ©

PL/SQL Utility Package GFC_ARCH_MGMT –DBA run management utility –Moves partitions to correct tablespace –Compresses/Decompresses partitions per specification in meta-data. UKOUG PeopleSoft Partition, Compress, Archive, Purge ©

ARCHIVE & PURGE UKOUG PeopleSoft Partition, Compress, Archive, Purge ©

Archive & Purge Archive: –Move data from live application table to another table. –In this case another to table in the same database. –Read-only access provided to archive tables. Purge: –Expunge data from the database. UKOUG PeopleSoft Partition, Compress, Archive, Purge ©

Archive/Purge For each table or set of tables –Define a policy that specifies when data is archived or purged. Compression too, but that has no functional impact. Discuss with business –Check legal requirements PAYE 3 years, Company Accounts 6 years UKOUG PeopleSoft Partition, Compress, Archive, Purge ©

But my data will disappear That may be a good thing –If you keep data you have to look after it responsibly. –Pay for the disk to store it/back it up. Read-only access via PS/Query tool to archive data. A very few components and reports access archive tables –Development cost of customisation UKOUG PeopleSoft Partition, Compress, Archive, Purge ©

Example Archive/Purge Policies –GP: archive 2 rolling years, purge after 8 tax years –Schedules, purge at 56 weeks –T&L, purge at 108 weeks –Audit: various: Archive after 3 months, 1, 3 year Purge 6 months, 1 year, 2 years, never UKOUG PeopleSoft Partition, Compress, Archive, Purge ©

Archive Method Partition Management –Exchange/Drop whole Partitions Time based partitioning –Avoid rewriting data –Avoids redo Faster –Compression must match Some archive table indexes not compressed. Row-by-Row –Non-time based partitioning GP –Can be done on-line –Can compress data during rewrite –Different partitioning strategy in archive and live. UKOUG PeopleSoft Partition, Compress, Archive, Purge ©

PeopleTools Archive Utility Row-by-Row processing only –Slow and cant reclaim space –Cant compress data during copy Additional key columns in archive table –Enlarges archive tables –Causes performance problems with index access Still need to drop empty partitions. Generic –Meta-data driven Sets of tables to archive Corresponding archive tables GP set delivered UKOUG PeopleSoft Partition, Compress, Archive, Purge ©

PL/SQL Archive/Purge Utility Custom PL/SQL Package –Driven by same meta-data as partitioning utility –Archive by partition exchange –Purge by dropping partition. –Called from custom Application Engine So archiving can be done via application. –Writes To PeopleSoft Message Log Various Oracle trace/log files UKOUG PeopleSoft Partition, Compress, Archive, Purge ©

No-Archive Rules UKOUG PeopleSoft Partition, Compress, Archive, Purge ©

Custom Archive Utility Based on delivered PeopleSoft Archive Utility We only use it for GP archiving Copies data in direct path –Faster –Can compress during copy Calls PL/SQL utility to drop partitions to save deleting data. UKOUG PeopleSoft Partition, Compress, Archive, Purge ©

PSARCH Schema Design 2 nd schema to hold archive tables –Only tables archived by partition exchange –Same name as live tables –Same local indexes as live Benefits –SYSADM cannot update/delete from these tables. Drawbacks –Views to make them accessible to PS/Query. –Cant manage indexes through Application Designer. UKOUG PeopleSoft Partition, Compress, Archive, Purge ©

Tablespace Design No impact on performance –Because all on same tier of disk Do not keep objects with dissimilar purge policies in same tablespaces –You can drop empty tablespaces and recover space. Periodic tablespaces make sense without purge policy With a purge policy might just have one tablespace. UKOUG PeopleSoft Partition, Compress, Archive, Purge ©

Wrong Tablespace Design Before archive policy –Weekly T&L and Schedule partitions in same monthly tablespaces Archive Policy –T&L: 108 weeks –Schedules: 56 weeks Schedules populated first, T&L later. –Schedules tend to be nearer start of tablespace –Schedules purged first leaving hole than cannot be trimmed off. UKOUG PeopleSoft Partition, Compress, Archive, Purge ©

Free Space Fragmentation Free Space left in middle of tablespace –Dropping objects –Compressing objects can leave holes Less Severe since Oracle 9i –Uniform extent sizes –Bitmap space map UKOUG PeopleSoft Partition, Compress, Archive, Purge ©

UKOUG PeopleSoft Partition, Compress, Archive, Purge ©

Oracle RMAN Backup Backs up block with SCN –If an object is dropped blocks still have SCN –Empty blocks can still be backed up. Can only reduce backup size by trimming free space from end of data files. UKOUG PeopleSoft Partition, Compress, Archive, Purge ©

Defragmentation of Free Space PL/SQL utility to rebuild objects nearest end of data file into spaces in the middle of the tables –Needs to be done during an application outage. Frees space at end of data file that can be trimmed off. –Thus reducing size of database and backup. UKOUG PeopleSoft Partition, Compress, Archive, Purge ©

Oracle Flashback Database SQL Rollback –We are used to the idea of rolling back transaction that have not been committed. Flashback database –Generates additional flashback logging –Rollback committed transaction and DDL but not through a drop/resize data file –Whole database option Highly effective in testing archive/purge UKOUG PeopleSoft Partition, Compress, Archive, Purge ©

Future: Oracle 11g Automatic Partition Creation Range-Range partitioning Advanced Compression –Licenced Option –Supports DML Active Data-Guard –Off-load query activity to standby database UKOUG PeopleSoft Partition, Compress, Archive, Purge ©

PROGRESS UKOUG PeopleSoft Partition, Compress, Archive, Purge ©

UKOUG PeopleSoft Partition, Compress, Archive, Purge ©

UKOUG PeopleSoft Partition, Compress, Archive, Purge © Tb/yr 0.48Tb/yr

UKOUG PeopleSoft Partition, Compress, Archive, Purge ©

Achievements Database Size –4.5TB => <2.25TB (with GP purge) Still have to purge old schedule and T&L data Expected Performance Improvements –GP calculation and other processing –T&L processing –Financial Reports UKOUG PeopleSoft Partition, Compress, Archive, Purge ©

Conclusion It depends… UKOUG PeopleSoft Partition, Compress, Archive, Purge ©

QUESTIONS? Nullius in verba UKOUG PeopleSoft Partition, Compress, Archive, Purge ©

You know my methods, apply them. A Study in Scarlet, Arthur Conan-Doyle UKOUG PeopleSoft Partition, Compress, Archive, Purge ©

Partition, Compress, Archive and Purge Keep Your System on the Road David Kurtz Go-Faster Consultancy Ltd.