Data Warehouse Chapter 11

Multiple Files Problem Added complexity of multiple source files Start simple Multiple Source files Extracted data Logic to detect Correct source

Transforming Data from Multiple files File

Missing Values Problem Solution Ignore Wait Mark rows Extract when time-stamped A If NULL then Field=‘A’

Duplicate Value Problem Solution SQL self-join techniques RDMBS constrains utilities SELECT… FROM table_a, table_b WHERE table_a.key(+)=table_b.key UNION SELECT… FROM table_a, table_b WHERE table_a.key=table_b.key(+) ACME Inc

Element Names Problem Solution CTAS SQL*Loader Customer Client Contact Name Customer

Element Meaning Problem Avoid misinterpretation Complex solution Document meaning in metadata Customer’s name All customer details All details Except name

Input Format Problem EBCDICASCII “123-73” 12373

Referential Integrity Problem Solution SQL anti-join Server constraints Dedicated tools Department Emp Name Department 1099 Smith Jones Doe Harris 60

Name and Address Problem No unique key Missing values Personal and commercial names mixed Different addresses for same member Different names and spelling for same number Many names on one line One name on two lines

Name and Address Problem Single-field format Multiple-field format Mr.J.Smith, 100 Main St., Bigtown, County Luth, Name Street Town County Code Mr.J.Smith 100 Main St. Bigtown County Luth 23565

Clean and Organize 1. Create atomic values. 2. Standardize formats. 3. Verify data accuracy. 4. Match with other records. 5. Identify private and commercial addresses and inhabitants. 6. Document in metadata. Requires sophisticated tools and techniques

Merging Data Operational transactions do not usually map one-to-one with warehouse data Data for the warehouse is merged to provide information for analysis Sale 1/2/98 12:00:01 Ham Pizza $10.00 Sale 1/2/98 12:00:02 Cheese Pizza $15.00 Sale 1/2/98 12:00:02 Anchovy Pizza $12.00 Return 1/2/98 12:00:03 Ham Pizza -$12.00 Sale 1/2/98 12:00:04 Sausage Pizza $11.00 Pizza sales/return by day, hour, seconds

Merging Data Sale 1/2/98 12:00:01 Ham Pizza $10.00 Sale 1/2/98 12:00:02 Cheese Pizza $15.00 Sale 1/2/98 12:00:02 Anchovy Pizza $12.00 Return 1/2/98 12:00:03 Ham Pizza -$12.00 Sale 1/2/98 12:00:01 Ham Pizza $10.00 Sale 1/2/98 12:00:02 Cheese Pizza $10.00 Sale 1/2/98 12:00:04 Sausage Pizza $11.00

Adding a Date Stamp Enables time analysis Label loaded data with a date stamp Add time to fact and dimension data

Adding a Date Stamp Sales Fact Table Item_id Store_id Time_key Sales_dollars Sales_units Store Table Store_id District_id Time_key Item_Table Item_id Dept_id Time_key Time Table Week_id Period_id Year_id Time_key Product Table Product_id Time_key Product_desc

Adding a Date Stamp Fact table - Add triggers - Recode applications - Compare tables Dimension table Time representation - Point in time - Time span

Adding Keys to Data #1 Sale 1/2/98 12:00:01 Ham Pizza $10.00 #2 Sale 1/2/98 12:00:02 Cheese Pizza $15.00 #3 Sale 1/2/98 12:00:02 Anchovy Pizza $12.00 #4 Sale 1/2/98 12:00:03 Ham Pizza -$12.00 #dw1 Sale 1/2/98 12:00:01 Ham Pizza $10.00 #dw2 Sale 1/2/98 12:00:02 Cheese Pizza $10.00 #dw3 Sale 1/2/98 12:00:04 Sausage Pizza $11.00 #5 Sale 1/2/98 12:00:04 Sausage Pizza $11.00 Data values or artificial keys

Summarizing Data During extraction on staging area After loading onto the warehouse server Operational databases Staging area Warehouse database

Maintaining Transformation Metadata Contains transformation rules, algorithms, and routines Sources Stages Rules Publish Extract Transform Load Query

Transformation Timing and Location Transformation is performed: - Before load - In parallel May be initiated at different points UnlikelyProbablePossible

Choosing a Transformation Point * Workload * Network bandwidth * Environment * Parallel execution * CPU use * Load window time * Disk space * User information needs

Monitoring and Tracking Transformations should: Be self-documenting Provides summary statistics Handle process exceptions

Designing Transformation Processes Analysis: - Sources and target mappings, business rules - Key users, metadata, grain Design options: PL/SQL, replication, custom, third-party tools Design issues: - Performance - Size of the staging area - Exception handling, integrity maintenance

Transformation Tools Purchased SQL*Loader In-house developed

Data Management, Quality, and Auditing Tools Data management: - Innovative Systems - Postalsoft - Vality Technology Data quality and auditing: - Innovative Systems - Vality Technology

Summary This lesson discussed the following topics: Importance of data quality Transformation processes Data transformation issuess Data anomalies Name and address management Tools