Optimizing Data Warehouse Ad-Hoc Queries against "Star Schemas " By Bert Scalzo, Ph.D.

Learning Objectives Just one major objective – to make the majority of unknown/unplanned ad-hoc end-user queries (e.g. BI tool reports’ many “SELECT” statements) execute as fast as is possible – and very efficiently, thus using the least amount of database server resources We cannot allow big databases to have queries that simply take too long …

Speaker’s Qualifications Oracle Solutions Product Architect for Quest Software Chief architect for Quest’s popular “TOAD” product Oracle DBA for 20+ years, versions 4 through 10g Worked for Oracle Education & Consulting Holds several Oracle Masters (DBA & CASE) BS, MS, PhD in Computer Science and also an MBA LOMA insurance industry designations: FLMI and ACS Books –The TOAD Handbook (Feb 2003) –Oracle DBA Guide to Data Warehousing and Star Schemas (Mar 2003) –The TOAD Pocket Reference 2 nd edition (June 2005) Articles –Oracle Magazine –Oracle Technology Network (OTN) –Oracle Informant –PC Week (now E-Magazine) –Linux Journal –

New 2 nd Edition – June 2005

About Quest Software Quest Software (NASDAQ: QSFT) Founded: 1987 More than 2000 employees in 40 offices: North America, South America, Europe, Asia, Australia Application management leader: 75% of Fortune 500 Develop, deploy, manage and maintain enterprise applications without downtime or business interruption Best known in the Oracle community for TOAD, Spotlight, Quest Central, Shareplex, etc.

Know Your Application … What type of application are you building: On Line Transaction Processing (OLTP) Operational Data Store (ODS) On Line Analytical Processing (OLAP) Data Mart / Data Warehouse (DM/DW)

OLTPODSOLAPDM/DW Business FocusOperationalOperational Tactical TacticalTactical Strategic End User ToolsClient Server Web Client ServerClient Server Web DB TechnologyRelational CubicRelational Trans CountLargeMediumSmall Trans SizeSmallMedium Large Trans TimeShortMediumLong Size in Gigs10 – – Normalization3NF N/A0NF Data Modeling Traditional ER N/ADimensional Application Natures…

Embrace New Concepts “Teach Old Dog New Tricks” Throw out any OLTP baggage Forget OLTP “Golden Rules” X

Star Schema Design “Star schema” approach to dimensional data modeling was pioneered by Ralph Kimball Dimensions: smaller, de-normalized tables containing business descriptive columns that end-users query on Facts: very large tables with primary keys formed from the concatenation of related dimension table foreign key columns, and possessing numerically additive, non-key columns used for calculations during end-user queries

Dimensions Facts

10 8th th 10 3rd -10 5th

The Ad-Hoc Challenge How much data would a data miner mine, if a data miner could mine data? Dimensions: generally queried selectively to find lookup value matches that are used to query against the fact table Facts: must be selectively queried, since they generally have hundreds of millions to billions of rows – even full table scans utilizing parallel are too big for most systems

Hardware Not Compensate Often, people have expectation that using expensive hardware is only way to obtain optimal performance for a data warehouse CPU SMP MPP Disk 15,000 RPM RAID (EMC) OS UNIX 64-bit Oracle OPS / PQO 64-bit

DB Design Paramount In reality, the database design is the key factor to optimal query performance for a data warehouse built as a “Star Schema” There are certain minimum hardware and software requirements that once met, play a very subordinate role to tuning the database Golden Rule: get the serial explain plan correct before doing partitioning or PQ

Key Tuning Requirements 1.Right version of Oracle 2.Properly tuned Init.Ora 3.Lots of Bitmap Indexes!!! 4.Cost Based Optimizer 5.Statistics and Histograms 6.“Star Transformation” explain plan

Oracle Version Problems Oracle 7.X lacks: True Partitioning of Tables and Indexes Reliable and Efficient Bitmap Indexes Histograms for Cost-Based Optimization “Star Transformation” explain plan Oracle 8.X lacks: List Partitioning Scalable Bitmap Indexes (500+ million) Scalable Histograms (500+ million) Bitmap Join Index Numerous new GORUP & OLAP Functions ANSI Syntax Support (e.g. ANSI 99 Join)

Oracle 8.X Init.Ora always_anti_join=HASH always_semi_join=HASH bitmap_merge_area_size=>16M buffer_pool_keep=xM (for bitmap indexes) buffer_pool_recycle=xM (for fact tables) compatible= create_bitmap_area_size>=16M db_block_buffers= (1.7G / DB_BLOCK_SIZE) – shared_pool_size db_block_size>=16K db_file_direct_io_count= MAX_IO_SIZE / DB_BLOCK_SIZE db_file_multiblock_read_count= MAX_IO_SIZE / DB_BLOCK_SIZE db_writer_processes=10 disk_asynch_io=TRUE hash_area_size>=16M hash_join_enabled=TRUE hash_multiblock_io_count= MAX_IO_SIZE / DB_BLOCK_SIZE optimizer_features_enable=8.1.7 optimizer_percent_paralell=0 (favor best serial plan – explained later) parallel_automatic_tuning=TRUE parallel_broadcast_enabled=TRUE parallel_max_servers=4 * CPU_COUNT if you have fast disks parallel_min_percent=0 query_rewrite_enabled=TRUE sort_area_size>=4M sort_multiblock_read_count= MAX_IO_SIZE / DB_BLOCK_SIZE star_transformation_enabled=TEMP_DISABLE

Oracle 9.X Init.Ora compatible= db_block_size>=16K db_cache_size=(1.7G / DB_BLOCK_SIZE) – shared_pool_size db_file_multiblock_read_count= MAX_IO_SIZE / DB_BLOCK_SIZE db_keep_cache_size=xM (for bitmap indexes) db_recycle_cache_size=xM (for fact tables) db_writer_processes=20 disk_asynch_io=TRUE filesystemio_options=SETALL (directIO + asynch) hash_join_enabled=TRUE log_parallelism=8 optimizer_features_enable=9.2.0 parallel_automatic_tuning=TRUE parallel_max_servers=4 * CPU_COUNT if you have fast disks parallel_min_percent=0 pga_aggregate_size>=64M query_rewrite_enabled=TRUE star_transformation_enabled=TRUE or TEMP_DISABLE undo_management=AUTO workarea_size_policy=AUTO

Oracle 10.X Init.Ora compatible= db_block_size>=16K db_file_multiblock_read_count= MAX_IO_SIZE / DB_BLOCK_SIZE db_keep_cache_size=xM (for bitmap indexes) db_recycle_cache_size=xM (for fact tables) db_writer_processes=20 disk_asynch_io=TRUE filesystemio_options=SETALL (directIO + asynch) optimizer_features_enable= parallel_automatic_tuning=TRUE parallel_max_servers=4 * CPU_COUNT if you have fast disks parallel_min_percent=0 pga_aggregate_size>=64M query_rewrite_enabled=TRUE sga_target_size=1.7G (for 32-bit systems) star_transformation_enabled=TRUE or TEMP_DISABLE undo_management=AUTO workarea_size_policy=AUTO

Bitmap Indexes All fact table, foreign key columns must have individual bitmap indexes on them If using Oracle 8.1, then ALTER TABLE MINIMIZE RECORDS_PER_BLOCK to optimize the mapping of bitmaps to rowids All dimension table, non-key columns should have individual bitmap indexes

Nighttime - 10 B-Tree Indexes

Daytime - 48 Bitmap Indexes!!!

Bit-map indexes –Contrary to widespread belief, can be effective when there are many distinct column values –Not suitable for OLTP however

Fact Table Indexes Table Index Type Uniq Column Null DW_ORDER DW_POS_DAY_B1 BITMAP N PERIOD_ID N DW_POS_DAY_B2 BITMAP N LOCATION_ID N DW_POS_DAY_B3 BITMAP N PRODUCT_ID N DW_POS_DAY_B4 BITMAP N VENDOR_ID N DW_POS_DAY_PK NORMAL Y PERIOD_ID N NORMAL Y LOCATION_ID N NORMAL Y PRODUCT_ID N NORMAL Y VENDOR_ID N NORMAL Y RECMD_FLAG N NORMAL Y NEW_ITEM_FLAG N

Dimension Table Indexes Table Index Type Uniq Column Null DW_VENDOR DW_VENDOR_B02 BITMAP N LEVELX N DW_VENDOR_B03 BITMAP N VENDOR_CODE N DW_VENDOR_B05 BITMAP N CITY Y DW_VENDOR_B06 BITMAP N STATE Y DW_VENDOR_B07 BITMAP N ZIP N DW_VENDOR_PK NORMAL Y VENDOR_ID N DW_VENDOR_U1 NORMAL Y VENDOR_NAME N

Cost Based Optimizer Statistics and Histograms: 1.First, analyze table delete statistics 2.For dimension tables, estimate statistics for table for all indexes 3.For fact tables, estimate statistics for table for all indexes for all indexed columns size 254 sample rows

Ready to Rock and Roll With all our database tuning pre-requisites addressed, we can now rely on Oracle to formulate reasonable explain plans for “Star Schema” data warehouse queries In fact if we’ve done everything correctly, then table/index partitioning and parallel query will merely eliminate partitions or perform parallel operations on our serial explain plans (i.e. 95%+ identical plans)

Sample Tuning Cases Oracle / HP-UX HP-9000, V2200, 16 CPU, 16 GB RAM Avg Fact table row count = 600 million Oracle / HP-UX HP-9000, K460, 4 CPU, 4 GB RAM Avg Fact table row count = 3 million Oracle / Windows 2003 Server Quad Xeon, 4 GB RAM Avg Fact table row count = 3 million Oracle / Redhat Linux AS 4.0 Quad Xeon, 4 GB RAM Avg Fact table row count = 3 million Same Explain Plans!!!

Query: beer and coffee sales for November of 98 in Dallas

Star Transformation

Star join performance 3 orders of magnitude difference between best and worst plan

| Operation | Name | Rows | Bytes| Cost | Pstart| Pstop | | SELECT STATEMENT | | 1 | 154 | 34 | | | | SORT GROUP BY | | 1 | 154 | 34 | | | | HASH JOIN | | 1 | 154 | 29 | | | | TABLE ACCESS BY INDEX ROWID | DW_ORDER | 1 | 95 | 17 | | | | BITMAP CONVERSION TO ROWIDS | | | | | | | | BITMAP AND | | | | | | | | BITMAP MERGE | | | | | | | | BITMAP KEY ITERATION | | | | | | | | TABLE ACCESS BY INDEX ROWID | DW_PERIOD | 1 | 51 | 2 | | | | BITMAP CONVERSION TO ROWIDS | | | | | | | | BITMAP AND | | | | | | | | BITMAP INDEX SINGLE VALUE | DW_PERIOD_B03 | | | | | | | BITMAP INDEX SINGLE VALUE | DW_PERIOD_B12 | | | | | | | BITMAP INDEX RANGE SCAN | DW_ORDER_B1 | | | | | | | BITMAP MERGE | | | | | | | | BITMAP KEY ITERATION | | | | | | | | TABLE ACCESS BY INDEX ROWID | DW_LOCATION | 1 | 46 | 2 | | | | BITMAP CONVERSION TO ROWIDS | | | | | | | | BITMAP AND | | | | | | | | BITMAP INDEX SINGLE VALUE | DW_LOCATION_B03 | | | | | | | BITMAP INDEX SINGLE VALUE | DW_LOCATION_B41 | | | | | | | BITMAP INDEX RANGE SCAN | DW_ORDER_B2 | | | | | | | BITMAP MERGE | | | | | | | | BITMAP KEY ITERATION | | | | | | | | TABLE ACCESS BY INDEX ROWID | DW_PRODUCT | 17 | 1K| 10 | | | | BITMAP CONVERSION TO ROWIDS | | | | | | | | BITMAP AND | | | | | | | | BITMAP INDEX SINGLE VALUE | DW_PRODUCT_B03 | | | | | | | BITMAP OR | | | | | | | | BITMAP INDEX SINGLE VALUE | DW_PRODUCT_B14 | | | | | | | BITMAP INDEX RANGE SCAN | DW_ORDER_B3 | | | | | | | TABLE ACCESS BY INDEX ROWID | DW_PRODUCT | 17 | 1K| 10 | | | | BITMAP CONVERSION TO ROWIDS | | | | | | | | BITMAP AND | | | | | | | | BITMAP INDEX SINGLE VALUE | DW_PRODUCT_B03 | | | | | | | BITMAP OR | | | | | | | | BITMAP INDEX SINGLE VALUE | DW_PRODUCT_B14 | | | | | | UTLXPLS: Serial Plan, No Partitioning

| Operation | Name | Rows | Bytes| Cost | Pstart| Pstop | | SELECT STATEMENT | | 1 | 154 | 35 | | | | SORT GROUP BY | | 1 | 154 | 35 | | | | HASH JOIN | | 1 | 154 | 30 | | | | PARTITION RANGE ALL | | | | | 1 | 10 | | TABLE ACCESS BY LOCAL INDEX ROWID| DW_ORDER_PART | 0 | 20 | 18 | 1 | 10 | | BITMAP CONVERSION TO ROWIDS | | | | | | | | BITMAP AND | | | | | | | | BITMAP MERGE | | | | | | | | BITMAP KEY ITERATION | | | | | | | | SORT BUFFER | | | | | | | | TABLE ACCESS BY INDEX ROWID| DW_PERIOD | 1 | 51 | 2 | | | | BITMAP CONVERSION TO ROWID| | | | | | | | BITMAP AND | | | | | | | | BITMAP INDEX SINGLE VALU| DW_PERIOD_B03 | | | | | | | BITMAP INDEX SINGLE VALU| DW_PERIOD_B12 | | | | | | | BITMAP INDEX RANGE SCAN | DW_ORDER_PART_B1 | | | | 1 | 10 | | BITMAP MERGE | | | | | | | | BITMAP KEY ITERATION | | | | | | | | SORT BUFFER | | | | | | | | TABLE ACCESS BY INDEX ROWID| DW_LOCATION | 1 | 46 | 2 | | | | BITMAP CONVERSION TO ROWID| | | | | | | | BITMAP AND | | | | | | | | BITMAP INDEX SINGLE VALU| DW_LOCATION_B03 | | | | | | | BITMAP INDEX SINGLE VALU| DW_LOCATION_B41 | | | | | | | BITMAP INDEX RANGE SCAN | DW_ORDER_PART_B2 | | | | 1 | 10 | | BITMAP MERGE | | | | | | | | BITMAP KEY ITERATION | | | | | | | | SORT BUFFER | | | | | | | | TABLE ACCESS BY INDEX ROWID| DW_PRODUCT | 17 | 1K| 10 | | | | BITMAP CONVERSION TO ROWID| | | | | | | | BITMAP AND | | | | | | | | BITMAP INDEX SINGLE VALU| DW_PRODUCT_B03 | | | | | | | BITMAP OR | | | | | | | | BITMAP INDEX SINGLE VAL| DW_PRODUCT_B14 | | | | | | | BITMAP INDEX RANGE SCAN | DW_ORDER_PART_B3 | | | | 1 | 10 | | TABLE ACCESS BY INDEX ROWID | DW_PRODUCT | 17 | 1K| 10 | | | | BITMAP CONVERSION TO ROWIDS | | | | | | | | BITMAP AND | | | | | | | | BITMAP INDEX SINGLE VALUE | DW_PRODUCT_B03 | | | | | | | BITMAP OR | | | | | | | | BITMAP INDEX SINGLE VALUE | DW_PRODUCT_B14 | | | | | | UTLXPLS: Serial Plan, with Partitioning

| Operation | Name | Rows | Bytes| Cost | TQ |IN-OUT| PQ Distrib | Pstart| Pstop | | SELECT STATEMENT | | 1 | 154 | 34 | | | | | | | SORT GROUP BY | | 1 | 154 | 34 | 2,03 | P->S | QC (RANDOM)| | | | SORT GROUP BY | | 1 | 154 | 34 | 2,02 | P->P | HASH | | | | HASH JOIN | | 1 | 154 | 29 | 2,02 | PCWP | | | | | TABLE ACCESS BY INDEX ROWID | DW_ORDER | 1 | 95 | 17 | 2,01 | P->P | HASH | | | | BITMAP CONVERSION TO ROWIDS | | | | | | | | | | | BITMAP AND | | | | | | | | | | | BITMAP MERGE | | | | | | | | | | | BITMAP KEY ITERATION | | | | | | | | | | | TABLE ACCESS BY INDEX ROWID | DW_PERIOD | 1 | 51 | 2 | | | | | | | BITMAP CONVERSION TO ROWIDS | | | | | | | | | | | BITMAP AND | | | | | | | | | | | BITMAP INDEX SINGLE VALUE | DW_PERIOD_B03 | | | | | | | | | | BITMAP INDEX SINGLE VALUE | DW_PERIOD_B12 | | | | | | | | | | BITMAP INDEX RANGE SCAN | DW_ORDER_B1 | | | | | | | | | | BITMAP MERGE | | | | | | | | | | | BITMAP KEY ITERATION | | | | | | | | | | | TABLE ACCESS BY INDEX ROWID | DW_LOCATION | 1 | 46 | 2 | | | | | | | BITMAP CONVERSION TO ROWIDS | | | | | | | | | | | BITMAP AND | | | | | | | | | | | BITMAP INDEX SINGLE VALUE | DW_LOCATION_B03 | | | | | | | | | | BITMAP INDEX SINGLE VALUE | DW_LOCATION_B41 | | | | | | | | | | BITMAP INDEX RANGE SCAN | DW_ORDER_B2 | | | | | | | | | | BITMAP MERGE | | | | | | | | | | | BITMAP KEY ITERATION | | | | | | | | | | | TABLE ACCESS BY INDEX ROWID | DW_PRODUCT | 17 | 1K| 10 | | | | | | | BITMAP CONVERSION TO ROWIDS | | | | | | | | | | | BITMAP AND | | | | | | | | | | | BITMAP INDEX SINGLE VALUE | DW_PRODUCT_B03 | | | | | | | | | | BITMAP OR | | | | | | | | | | | BITMAP INDEX SINGLE VALUE| DW_PRODUCT_B14 | | | | | | | | | | BITMAP INDEX RANGE SCAN | DW_ORDER_B3 | | | | | | | | | | TABLE ACCESS BY INDEX ROWID | DW_PRODUCT | 17 | 1K| 10 | 2,00 | S->P | HASH | | | | BITMAP CONVERSION TO ROWIDS | | | | | | | | | | | BITMAP AND | | | | | | | | | | | BITMAP INDEX SINGLE VALUE | DW_PRODUCT_B03 | | | | | | | | | | BITMAP OR | | | | | | | | | | | BITMAP INDEX SINGLE VALUE | DW_PRODUCT_B14 | | | | | | | | | UTLXPLP: Parallel Plan, No Partitioning

| Operation | Name | Rows | Bytes| Cost | TQ |IN-OUT| PQ Distrib | Pstart| Pstop | | SELECT STATEMENT | | 1 | 154 | 34 | | | | | | | SORT GROUP BY | | 1 | 154 | 34 | 5,03 | P->S | QC (RANDOM)| | | | SORT GROUP BY | | 1 | 154 | 34 | 5,02 | P->P | HASH | | | | HASH JOIN | | 1 | 154 | 29 | 5,02 | PCWP | | | | | PARTITION RANGE ALL | | | | | 5,02 | PCWP | | 1 | 10 | | TABLE ACCESS BY LOCAL INDEX ROWID| DW_ORDER_PART | 0 | 20 | 18 | 5,01 | P->P | HASH | 1 | 10 | | BITMAP CONVERSION TO ROWIDS | | | | | | | | | | | BITMAP AND | | | | | | | | | | | BITMAP MERGE | | | | | | | | | | | BITMAP KEY ITERATION | | | | | | | | | | | SORT BUFFER | | | | | | | | | | | TABLE ACCESS BY INDEX ROWID| DW_PERIOD | 1 | 51 | 2 | | | | | | | BITMAP CONVERSION TO ROWID| | | | | | | | | | | BITMAP AND | | | | | | | | | | | BITMAP INDEX SINGLE VALU| DW_PERIOD_B03 | | | | | | | | | | BITMAP INDEX SINGLE VALU| DW_PERIOD_B12 | | | | | | | | | | BITMAP INDEX RANGE SCAN | DW_ORDER_PART_B1 | | | | | | | 1 | 10 | | BITMAP MERGE | | | | | | | | | | | BITMAP KEY ITERATION | | | | | | | | | | | SORT BUFFER | | | | | | | | | | | TABLE ACCESS BY INDEX ROWID| DW_LOCATION | 1 | 46 | 2 | | | | | | | BITMAP CONVERSION TO ROWID| | | | | | | | | | | BITMAP AND | | | | | | | | | | | BITMAP INDEX SINGLE VALU| DW_LOCATION_B03 | | | | | | | | | | BITMAP INDEX SINGLE VALU| DW_LOCATION_B41 | | | | | | | | | | BITMAP INDEX RANGE SCAN | DW_ORDER_PART_B2 | | | | | | | 1 | 10 | | BITMAP MERGE | | | | | | | | | | | BITMAP KEY ITERATION | | | | | | | | | | | SORT BUFFER | | | | | | | | | | | TABLE ACCESS BY INDEX ROWID| DW_PRODUCT | 17 | 1K| 10 | | | | | | | BITMAP CONVERSION TO ROWID| | | | | | | | | | | BITMAP AND | | | | | | | | | | | BITMAP INDEX SINGLE VALU| DW_PRODUCT_B03 | | | | | | | | | | BITMAP OR | | | | | | | | | | | BITMAP INDEX SINGLE VAL| DW_PRODUCT_B14 | | | | | | | | | | BITMAP INDEX RANGE SCAN | DW_ORDER_PART_B3 | | | | | | | 1 | 10 | | TABLE ACCESS BY INDEX ROWID | DW_PRODUCT | 17 | 1K| 10 | 5,00 | S->P | HASH | | | | BITMAP CONVERSION TO ROWIDS | | | | | | | | | | | BITMAP AND | | | | | | | | | | | BITMAP INDEX SINGLE VALUE | DW_PRODUCT_B03 | | | | | | | | | | BITMAP OR | | | | | | | | | | | BITMAP INDEX SINGLE VALUE | DW_PRODUCT_B14 | | | | | | | | | UTLXPLP: Parallel Plan, with Partitioning

SQL Plus Timing Results Fact ImplementationUNIX2003 Serial, No Partition9,68822,344 Serial, with Partition5,57811,625 Parallel, No Partition7,75017,875 Parallel, with Partition11,14025,454 NOTE: specific to my data and user queries

Plethora Table Design Options Fact ImplementationUNIX Regular “Heap” Table9,293 Single Column Partition4,747 Multi Column Partition4,987 Composite Partition6,319 Index Organized12,508 Partition Index Organized14,902 NOTE: specific to my data and user queries

Conclusions Remember: should use Oracle latest Oracle version for best results (optimizer has matured for DW and specifically star queries) proper init.ora (i.e. spfile) parameters lots and lots of bitmap indexes must use statistics and histograms shoot for the “Star Transformation” Get non-parallel, non-partitioned explain plan right first, then improve upon that with others highly recommend SQL tuning tools – like “TOAD Xpert”