Informix Formation Chetana Mehta PSPL, Pune

Outline b Overview of Formation b PSPL’s role b Future work

Data Warehouse Architecture Data Warehouse Engine Optimized Loader Extraction Cleansing Analyze Query Metadata Repository Relational Databases Legacy Data Purchased Data ERP Systems

What is ETL? b Extract data from existing operational and legacy data, transform and load the warehouse. b Issues: Sources of data for the warehouseSources of data for the warehouse Data quality at the sourcesData quality at the sources Merging different data sourcesMerging different data sources Data TransformationData Transformation How to propagate updates (on the sources) to the warehouseHow to propagate updates (on the sources) to the warehouse Terabytes of data to be loadedTerabytes of data to be loaded

Overview of Formation b ETL Tool b User-friendly b Scalable

Operators b Join - Hash, Non-equi, Nested loop, Sort- merge b Aggregate/GroupBy b Sort b Deduplicate b Surrogate Key

Performance Subsystem b Periodic statistics b Summary statistics Operator summaryOperator summary Group summaryGroup summary Performance hintsPerformance hints

Periodic Statistics b No. of records pushed/pulled b Memory used b Disk reads/writes b Temporary space used

Summary Statistics b No. of records pulled/pushed b Record size b Time when first/last record sent/received b No. of unique keys/groups b Ratio of output size to input size b Selectivity

Performance Hints b Ideal memory size b Suggested memory size b Parallelizing

Future work b Memory cognizant optimization b Parametric query optimization b Operator ordering b XML extensions