Data Staging Data Staging Legacy System Data Warehouse SQL Server Access Excel Legacy System

Data Staging Extraction Data Cleansing Data Integration Transformation Transportation (Loading) Maintenance

Data Staging Area The construction site for the warehouse Required by most scenarios Connected to wide variety of sources Clean / aggregate / compute / validate data Operational system Data staging area Warehouse Extract Transport (Load) Transform

Extraction Extract source data from legacy systems and place it in a staging area. To reduce the impact on the performance of legacy systems, source data is extracted without any cleansing, integration and transformation operations.

In Data Warehouse terms, a data staging area is an intermediate storage area between the sources of information and the data warehouse (DW) . It is usually of temporary nature and its contents can be erased after the DW/DM has been loaded successfully. A staging area can be used for any of the following purposes, among others: To gather data from different sources that will be ready to process at different times. To quickly load information from the operational database, To find changes against current DW/DM values. For data cleansing To pre-calculate aggregates.

Data Mart A subset of a data warehouse that supports the requirements of a particular department or business function. Characteristics include: Do not normally contain detailed operational data unlike data warehouses. May contain certain levels of aggregation

Extraction A variety of file formats exist in legacy systems Relational database: DB, Oracle, SQL Server, Access … Flat file: Excel file, text file Commercial data extraction tools are very helpful in data extraction.

Data Preparation (Cleansing) It’s all about data quality!!!

Outline Measures for Data Quality Causes for data errors Common types of data errors Common error checks Correcting missing values Timing for error checks and corrections Steps of data preparation

Measures (Dealing)for Data Quality Correctness/Accuracy - w.r.t. the real data Consistency/Uniqueness – data values, references, measures and interpretations Completeness - scope of data & values Relevancy – w.r.t. the requirements Current data – relevant to the requirements

Causes for Data Errors Data entry errors Correct data not available at the time of data entries By different users same time or same users overtime Inconsistent or incorrect use of “codes” Inconsistent or incorrect interpretation of “fields” Transaction processing errors System and recovery errors Data extract/transformation errors

Common Data Errors Missing (null) values Incorrect use of default values (e.g., zero) Data value (dependency) integrity violation Data referential integrity violation (e.g., a customer’s order record cannot exist unless the customer record already exists)

Common Data Errors, Cont’d Data retention integrity violation (e.g., old inventory snapshots should not be stored) Data Derivation/Transformation/Aggregation Integrity Violation Inconsistent data values of the same data (M versus m for male) Inconsistent use of the same data value (DM for Data Mining and Data Marts)

Error Checks Referential integrity validation Identify missing-value or default-value records Identify outliers (Exception) Process validation

Data Cleaning: Missing Values Data cleansing or data cleaning is the process of detecting and correcting (or removing) corrupt or inaccurate records from a record set, table or database, and refers to identifying incomplete, incorrect, parts of the data and then replacing, modifying, or deleting the dirty data .

Data Quality and Cleansing To get quality data into in the warehouse the data gathering process must be well designed. Clean data comes from two processes: entering clean data and cleaning up problems once the data are entered.

Data Quality and Cleansing Characteristics of quality data : Accurate –data in Warehouse matches with the system of record. Complete – the data in the warehouse represents the entire set of relevant data Consistent: the data in the warehouse is free from contradiction (Uniqueness) Timely: data must be updated on a scheduled that is useful to the business users

Data Improvement - The actual content of the data presents a substantial(large) challenge to the warehouse: Inconsistent or incorrect use of codes and special characters: Overloaded codes: Evolving data: Missing, incorrect or duplicate values

Approach to improving data Following steps helps to deal with data cleansing issues: Identify the highest quality source system Examine the code how bad it is. Find the minor variations in spelling during scanning lists. Fix the problem with the source if at all possible Fix some problem during data staging. Use data cleansing tools against the data and use trusted sources for correct values. Work with the source system owners for regular examination and cleansing of the source system Make the source system team responsible for a clean extract

Timing for Error Checking During Data Staging During Data Loading Others Before data extraction (data entries, transaction processing, recovery, audits, etc.) After data loading

Steps of Data Preparation Identify data sources Extract and analyze source data Standardize data Correct and complete data Match and consolidate data Transform and enhance data into target Calculate derivations and summary data Audit and control data extract, transformation and loading

Data Integration Data from different data sources with different formats need to be integrated into one data warehouse Ex: 3 customer table in sales department, marketing department and an acquired company Customer (cid, cname, city …) Customer (customerid, customername,city…) Customer (custid, custname, cname,…)

Data Integration Same attribute with different name: cid, customerid, custid Different attribute with same name: cname -> customer name cname -> city name Same attribute with different formats

Data Integration How to integrate Get the schemas of all data sources A database schema is the skeleton structure that represents the logical view of the entire database. It defines how the data is organized and how the relations among them are associated. Get the schema of the data warehouse Integrate source schemas with the help from commercial tools and domain experts

Stepwise plan for creating data staging application… High Level Plan Data Staging Tools Detailed Plan

Step 1: Plan effectively Planning phase starts out with the high-level plan. Step 1: High-Level Plan Start the design process. Keep it very high-level (that highlights the data coming from and challenges we know). Data staging applications perform 3 major steps : Extract from source Transform it Load into warehouse

Step 2: Data staging tools.. Data staging tools are system code generators. The data staging tool is used instead of hand-coding the extracts. Transformation engines are designed to improve scalability. Ex: Tools like Information Builders - Data Migrator, Oracle - Data Integrator

Organize the data staging area. Step 3: Detailed plan.. Drill down on each of the flows & phase of ETL Process. Cleaning, Integration, Combined data from different sources Plan which table to work on in which order. Organize the data staging area. It is the place where the raw data is loaded, cleaned and combined and exported.

Transformation Prepare data for loading into the data warehouse Change the data format (One source format to other)

Maintenance Maintenance frequency: daily, weekly, monthly Identify change records and new records in legacy systems Create timestamps for changes and new records in legacy systems Compare data between legacy systems and DW Load changes and new records into DW

Data Warehouse Architecture Overview

Data Mining Applications Data mining covers wide field and diverse applications Some application domains Financial data analysis (Finance and Banking) Retail and Telecommunication industries Data Mining in Science & Engineering

Data Mining for Financial Data Analysis Financial data collected in banks and financial institutions are often relatively complete, reliable, and of high quality Data Mining for Retail Industry Retail industry: huge amounts of data on sales, customer shopping history, etc. Applications of retail data mining Identify customer buying behaviors Discover customer shopping patterns and trends Improve the quality of customer service Achieve better customer retention and satisfaction Design more effective goods transportation and distribution policies

Data Mining for Telecomm. Industry A rapidly expanding and highly competitive industry and a great demand for data mining Understand the business involved Identify telecommunication patterns Catch fake activities Make better use of resources Improve the quality of service

Tools for Data Mining Rapid Miner (This is very popular since it is a ready made, open source, no-coding required software) WEKA (This is a JAVA based customization tool, which is free to use) R-Programming Tool (This is written in C and FORTRAN, and allows the data miners to write scripts just like a programming language/platform.) Python based Orange (Python is very popular due to ease of use and its powerful features. Orange is an open source tool that is written in Python with useful data analytic)