MIS 451 Building Business Intelligence Systems Data Staging

2 Project Planning Requirements Analysis Physical Design Logical Design Data Staging Data Analysis (OLAP)

3 What has been done? Project plan User requirements Dimentional Model (Star Schema) Index plan

4 Data Staging Data Warehouse (Oracle) DB2 Access Excel Legacy System Data Staging

5 Extraction Data Cleansing Data Integration Transformation Transportation (Loading) Maintenance

6 Extraction During extraction step, source data are extracted from legacy systems and placed in a staging area To keep the performance of legacy systems, source data are only extracted without any cleansing, integration and transformation operations.

7 Extraction A variety formats of data exist in legacy systems Relational database: DB2, Oracle, SQL Server, Informix, Access … Flat file: Excel file, text file Commercial data extraction tools are very helpful in data extraction. Ex: Oracle Data Mart Builder

8 Data Cleansing Data cleansing is to clean errors in source data. Data entry error Missing value

9 Data Cleansing Data Entry Error Reason: typing error, wrong deletion How to correct: Pre-extraction checking: domain constraints, referential integrity constraints, value dependency checking Post-extraction checking

10 Data Cleansing Missing value Reason: information not available, typing error How to correct: Replace with the attribute mean value Replace with a constant value Replace with the most probable value

11 Data Integration Data from different data sources with different formats need to be integrated into one data warehouse Ex: 3 customer table in sales department, marketing department and an acquired company Customer (cid, cname, city …) Customer (customerid, customername,city…) Customer (custid, custname, cname,…)

12 Data Integration Same attribute with different name: cid, customerid, custid Different attribute with same name: cname -> customer name cname -> city name Same attribute with different format

13 Data Integration How to integrate Get the schemas of all data sources Get the schema of the data warehouse Integrate source schemas with the help of commercial tools

14 Transformation Prepare data ready to load into the data warehouse Change the format of data Create derived attributes and tables Aggregate Create warehouse keys

15 Transportation Using bulk load tools, such as Oracle SQL Loader, instead of SQL command Create indexes

16 Maintenance Maintenance frequency: daily, weekly, monthly Identify change records and new records in legacy systems Create timestamp for change and new records in legacy systems Compare data between legacy systems and DW Load change and new records into DW