prligence Empowering Intelligence A Multi-Source Time-Variant Datawarehouse Case Study Session# by Arup Nanda Proligence, Inc. Norwalk, CT

prligence Empowering Intelligence Objectives Exploring DW Techniques in Oracle Case Study Oracle 10 G Additions

prligence Empowering Intelligence Datawarehouse Cust1 Cust2 Cust3 Cust4 Cust10 Cust9 Cust8 Cust7 Cust11? Cust5Cust6 DB3 DB6 DB4DB5 DB1DB2

prligence Empowering Intelligence A Real Life Case Claims Datawarehouse Several Customers/Sources Several Quarters Data Volume Was High Irregular Frequency Data Comes Often Late Near Real Time Requirements

prligence Empowering Intelligence Detail Table Detail Table Summary Table Summary Table Detail Table Detail Table CUST2 Problem of Irregular Data DBMS_MVIEW.REFRESH (…)

prligence Empowering Intelligence Problems Incoming Data Irregular Summary Tables Need Refreshing Quarters Added Continuously Archival Requirements Vary Across Customers Quick Retrieval of Archival Needed

prligence Empowering Intelligence Problems contd. Summary on Summary Tables as Materialized Views Need Refresh Whenever New Data Arrives Or When Data is Purged/Reinstated Customers Added and Deleted Frequently

prligence Empowering Intelligence Objective To Minimize Downtime for Refreshes –Incrementally Refresh –Partitioning Techniques To Add Customers Easily To Add Quarters Easily To Archive Off and Purge Easily and Atomically To Restore Archives Quickly

prligence Empowering Intelligence Objective contd. To have an ETL Setup for Easy Addition of Objects Such As Tables, Indexes, Mat Views. Use Only Available Oracle and Unix Tools –PL/SQL –Unix Shell Scripts –SQL*Plus

prligence Empowering Intelligence Design Varying Dimensions – –Customer –Quarter Composite Partitioning –Range (for Quarters) –List (for Customers) Local Indexes

prligence Empowering Intelligence Partitioning Partitioned on CLAIM_DATE –RANGE –Partitioned named Y yy Q q –Storage Clauses Not Defined Supartitioned on CUST_NAME –LIST –Named Y yy Q q_CustName, e.g. Y03Q3_CUST1

prligence Empowering Intelligence Indexing All Indexes Local CREATE INDEX IN_CLAIM_SUM_01 LOCAL ON SUMTAB1 (COL1, COL2)… No Indexes UNIQUE and GLOBAL

prligence Empowering Intelligence Storage Each Subpartition – of Index or Table is kept in separate tablespaces named in the format Y Q _ _DATA e.g. Y02Q2_CUST1_DATA Y02Q2_CUST2_DATA Y03Q3_CUST1_DATA

prligence Empowering Intelligence QuarterCustomers Cust3 Y03 Q3 Table In Tablespace Y03Q3_CUST3_DATA Cust3 Y03 Q3 Index In Tablespace Y03Q3_CUST3_INDX

prligence Empowering Intelligence Tablespace create tablespace y03q3_cust1_data datafile ‘/oradata/y03q3_cust1_data_01.dbf’ size 500m autoextend on next 500m extent management local segment space management auto

prligence Empowering Intelligence Table DDL CREATE TABLE TAB1 ( … ) PARTITION BY RANGE (CLAIM_DATE) SUBPARTITION BY LIST (CUST_NAME) ( PARTITION Y03Q1 VALUES LESS THAN (TO_DATE(‘2003/04/01’,’YYYY/MM/DD’)), ( SUBPARTITION Y03Q1_CUST1 VALUES (‘CUST1’) TABLESPACE Y03Q1_CUST1_DATA, SUBPARTITION Y03Q1_CUST2 VALUES (‘CUST2’) TABLESPACE Y03Q1_CUST2_DATA, … and so on for all subpartitions … SUBPARTITION Y03Q1_DEF VALUES (DEFAULT) TABLESPACE USER_DATA ), PARTITION Y03Q2 VALUES LESS THAN (TO_DATE(‘2003/07/01’,’YYYY/MM/DD’)), ( SUBPARTITION Y03Q2_CUST1 VALUES (‘CUST1’) TABLESPACE Y03Q2_CUST1_DATA, SUBPARTITION Y03Q2_CUST2 VALUES (‘CUST2’) TABLESPACE Y03Q2_CUST2_DATA, … and so on for all subpartitions … SUBPARTITION Y03Q2_DEF VALUES (DEFAULT) TABLESPACE USER_DATA ), … and so on for all the partitions … PARTITION DEF VALUES LESS THAN (MAXVALUE), ( SUBPARTITION DEF_CUST1 VALUES (‘CUST1’) TABLESPACE USER_DATA, SUBPARTITION DEF_CUST2 VALUES (‘CUST2’) TABLESPACE USER_DATA, … and so on for all subpartitions … SUBPARTITION DEF_DEF VALUES (DEFAULT) TABLESPACE USER_DATA )

prligence Empowering Intelligence Index DDL CREATE INDEX IN_TAB1_01 ON TAB1 (COL1) LOCAL NOLOGGING ( PARTITION Y03Q1 ( SUBPARTITION Y03Q1_CUST1 TABLESPACE Y03Q1_CUST1_INDX, SUBPARTITION Y03Q1_CUST2 TABLESPACE Y03Q1_CUST2_INDX, … and so on for all subpartitions … SUBPARTITION Y03Q1_DEF TABLESPACE USER_DATA ), PARTITION Y03Q2 ( SUBPARTITION Y03Q2_CUST1 TABLESPACE Y03Q2_CUST1_INDX, SUBPARTITION Y03Q2_CUST2 TABLESPACE Y03Q2_CUST2_INDX, … and so on for all subpartitions … SUBPARTITION Y03Q2_DEF TABLESPACE USER_DATA ), … and so on for all the partitions … PARTITION DEF ( SUBPARTITION DEF_CUST1 TABLESPACE USER_DATA, SUBPARTITION DEF_CUST2 TABLESPACE USER_DATA, … and so on for all subpartitions … SUBPARTITION DEF_DEF TABLESPACE USER_DATA )

prligence Empowering Intelligence Creating DDLs Static Part Static Part Variable Part Variable Part create table tab1 (………) partition y03q1 ( subpartition y03q1_cust1 tablespace …) DDL to Create Table DDL to Create Table

prligence Empowering Intelligence Constraints Constraints defined as DISABLE NOVALIDATE RELY ALTER TABLE … ADD CONSTRAINT … RELY DISABLE NOVALIDATE;

prligence Empowering Intelligence Constraint VALIDATE/NOVALIDATE –Table TAB1 (Column: STATUS) –Current Values A, I, F –Check Constraint: STATUS IN (‘A’,’I’) ENABLE/DISABLE –New Value ‘F’ RELY

prligence Empowering Intelligence RELY Reasons To Include Relation Information to the Metadata To Enable Query Rewrite

prligence Empowering Intelligence Summary Tab and View Summary Table CUST_NAME CLAIM_DATE PROVIDER_ID NUM_CLAIMS NUM_LINES View SELECT ‘CUST1’ AS CUST_NAME, CLAIM_DATE, PROVIDER_ID, COUNT(DISTINCT CLAIM_ID) AS NUM_CLAIMS, COUNT(*) AS NUM_LINES FROM …. GROUP BY … On Source On DW

prligence Empowering Intelligence Casting SELECT CAST (CUST_NAME AS VARCHAR2(20)) AS CUST_NAME FROM CAST ( column_name AS datatype ( precision ))

prligence Empowering Intelligence cust1 Summary Table Summary Table DW View Owned by Cust Schema Temporary Table Index of Temporary Table Massaging Analyzing Filter: Where CLAIM_DATE is in that quarter INDEX TABLE For Customer Cust1 and Quarter Q1

prligence Empowering Intelligence cust DW View Old Sub Partition INDEX TABLE ALTER TABLE … EXCHANGE SUBPARTITION subpartname WITH TEMPTABLE INCLUDING INDEXES

prligence Empowering Intelligence Technique Not Using DBMS_MVIEW.REFRESH MV is always STALE

prligence Empowering Intelligence Temp Table CREATE TABLE T1_Y03Q1_CUST1 TABLESPACE Y03Q1_CUST1_DATA PARALLEL 8 NOLOGGING AS SELECT … FROM WHERE CLAIM_DATE >= add_months(trunc(to_date(‘03','RR'),'YYYY'), 3*(to_number(‘1')-1)) and batch_date < last_day(add_months(trunc( to_date(‘03','RR'),'YYYY'), 3*(to_number(‘1')) - 1 )) + 1

prligence Empowering Intelligence Script CREATE TABLE T1_Y&&YY.Q&&Q._&&CUST TABLESPACE Y&&YY.Q&&Q._&&CUST._DATA PARALLEL 8 NOLOGGING AS SELECT … FROM WHERE CLAIM_DATE >= ADD_MONTHS(TRUNC(TO_DATE('&&YY','RR'),'YYYY'), 3*(TO_NUMBER('&&Q')-1)) AND BATCH_DATE < LAST_DAY(ADD_MONTHS(TRUNC( TO_DATE('&&YY','RR'),'YYYY'), 3*(TO_NUMBER('&&Q')) -1 )) + 1

prligence Empowering Intelligence External Table Reason Source is a non-Oracle DB, e.g. DB2 Source is External, no DB Link Allowed Fixed Format –vs- Delimited Fixed Format Faster, Easier More Space Delimited Less Space Slower, Slightly More Complex

prligence Empowering Intelligence Massaging Removing NOT NULL Constraints Making Datatypes Consistent –The CAST operation converts NUMBER(m,n) to NUMBER –cast(col1 as number(10,2)) as col1_m –COL1 NUMBER(5,2) –COL1_M NUMBER

prligence Empowering Intelligence Analyzing Using DBMS_STATS.GATHER_TABLE_STATS PARALLEL Degree dbms_stats.gather_table_stats ( ownname => ‘DWOWNER', tabname => '&&TABNAME', estimate_percent => dbms_stats.auto_sample_size, method_opt => 'FOR ALL INDEXED COLUMNS SIZE AUTO', degree => dbms_stats.default_degree, cascade => TRUE );

prligence Empowering Intelligence Mat Views MVs Created as Tables CREATE TABLE MV_SUMMTAB1 Storage clauses just like the underlying table CREATE MATERIALIZED VIEW MV_SUMMTAB1 ON PREBUILT TABLE AS SELECT ……

prligence Empowering Intelligence Query Rewrite Table SUM_CLAIMS PROVIDER_ID, STATE, TYPE, TOT_AMT Table MV_SUM_CLAIMS PROVIDER_ID, STATE, SUM(TOT_AMT) TOT_AMT GROUP BY PROVIDER_ID, STATE SELECT SUM(TOT_AMT) FROM SUM_CLAIMS SELECT SUM(TOT_AMT) FROM MV_SUM_CLAIMS

prligence Empowering Intelligence Query Rewrite Init.ora Parameters query_rewrite_enabled='TRUE' query_rewrite_integrity='STALE_TOLERATED‘ ENFORCED – Rewrite only if guaranteed TRUSTED – Uses only if RELY STALE_TOLERATED – Even if not RELY

prligence Empowering Intelligence Checking QR dbms_mview.explain_rewrite ( ‘select cust_name, count(*) from summtab1 group by cust_name’ ); select message from rewrite_table; QSM-01033: query rewritten with materialized view, MV_SUMMTAB1 QSM-01101: rollup(s) took place on mv, MV_SUMMTAB1

prligence Empowering Intelligence Design … MV_* subpartitions are on the same tablespace as the parents. Subparts of MV_SUMMTAB1_0? are in the same TS as SUMMTAB1 Subparts of MV_SUMMTAB2_0? in SUMMTAB2

prligence Empowering Intelligence Customer name Quarter PARENT MV1 MV2 TableSpace1TableSpace2

prligence Empowering Intelligence MV and Parents Partition Pruning Partition-wise Joins Partition Independence

prligence Empowering Intelligence Adding Quarters/Customers Partition –Default Partition – VALUES LESS THAN (MAXVALUE) Subpartition –Default Subpartition – VALUES (DEFAULT)

prligence Empowering Intelligence Qtr1 Qtr2Qtr3 DEF Cust1 Cust2 Cust3 DEF

prligence Empowering Intelligence Qtr1 Qtr2Qtr3 DEF Cust1 Cust2 Cust3 DEF

prligence Empowering Intelligence Qtr1 Qtr2Qtr3 DEF Cust1 Cust2 Cust3 DEF Cust4 alter table … split subpartition

prligence Empowering Intelligence Qtr1 Qtr2Qtr3 DEF Cust1 Cust2 Cust3 DEF

prligence Empowering Intelligence Qtr1 Qtr2Qtr3Qtr4 Cust1 Cust2 Cust3 DEF alter table … split partition

prligence Empowering Intelligence Backup/Restore Backup –ALTER TABLESPACE READ ONLY –Copy the files to tape/CD. Restore –Copy the file back into the directory –ALTER TABLESPACE RECOVER

prligence Empowering Intelligence Archival/Purge SP1SP2SP3 Table SP4 SP1SP2SP3 Table Table4 SP1SP2SP3SP4 Table Table4

prligence Empowering Intelligence Archival/Purge CREATE TABLE S1_Y Q _ TABLESPACE Y Q _ _ AS SELECT * FROM SUMMTAB1 WHERE 1=2 / CREATE INDEXES, CONSTRAINTS, etc. / ALTER TABLE SUMMTAB1 EXCHANGE SUBPARTITION Y Q _ WITH TABLE Y Q _ INCLUDING INDEXES /

prligence Empowering Intelligence Check TTS ALTER TABLESPACE Y Q _ _ READ ONLY; DBMS_TTS.TRANSPORT_SET_CHECK (, ) ; SELECT * FROM TRANSPORT_SET_VIOLATIONS;

prligence Empowering Intelligence Transport TS Export Parameter File TRANSPORT_TABLESPACE=y TTS_FULLCHECK=Y FILE=‘ /exp.dmp ’ TABLESPACES=(, ) Copy the exp.dmp and Datafiles to tape/CD.

prligence Empowering Intelligence Purge Drop Subpartition Drop the Tablespace DROP TABLESPACE INCLUDING CONTENTS AND DATAFILES;

prligence Empowering Intelligence Restore ALTER TABLE SPLIT SUBPARTITION Copy Datafiles & Export Dump Files from CD/Tape Import Parameter File TRANSPORT_TABLESPACES=Y TABLESPACES=(, ) DATAFILES=(…)

prligence Empowering Intelligence Minimizing Refresh Unit Months – instead of quarters refreshed at a time. Last Quarter Split into a Subpartition per Month Naming Convention –Y yy Q q M mm –Y03Q3M09 Merge Subpartition

prligence Empowering Intelligence Merging Subpartitions Index Subpartitions Created in User’s Default Tablespace Subpartition Template ALTER TABLE SUMTAB1 ADD SUBPARTITION TEMPLATE

prligence Empowering Intelligence Resumable Statement When? –Running Large Report Jobs –Creating Large Indexes ALTER SESSION ENABLE RESUMABLE NAME ‘Job1’; View DBA_RESUMABLE –NAME – Name specified in ALTER SESSION –COORD_SESSION_ID – Coord Session in PQ –SQL_TEXT – The text of the SQL –STATUS - RUNNING, SUSPENDED, ABORTED, ABORTING, TIMEOUT –ERROR_NUMBER/ERROR_MSG

prligence Empowering Intelligence Objectives Revisited To Minimize Downtime for Refreshes –Incrementally Refresh –Partitioning Techniques To Add Customers Easily To Add Quarters Easily To Archive Off and Purge Easily and Atomically To Restore Archives Quickly

prligence Empowering Intelligence Oracle 10G Transportable Tablespaces Can Be Reinstated At a Different Operating System –Can be used for Restoring to a Different OS Tablespaces Can Be Renamed –Restoring Tablespace of the Same Name Multiple Temporary Tablespace –For Large Index Creation, Sorting, etc.

prligence Empowering Intelligence Oracle 10G contd. Partition Change Tracking Support for List Partitioning Query Rewrites Can Use Multiple MVs OEM Shows All Partitioning Features Data Pump –Export/Import on Steroids –Parallel Operation

prligence Empowering Intelligence Oracle 10G contd. External Table Download –A Utility to Create File from Table Data CREATE TABLE … ORGANIZATION EXTERNAL AS SELECT * FROM –Platform Independent File –Can Be Used In External Tables

prligence Empowering Intelligence Thank You! Please Submit Reviews Session#