Database Management System Prepared by Dr. Ahmed El-Ragal Reviewed & Presented By Mr. Mahmoud Rafeek Alfarra College Of Science & Technology- Khan younis Information Technology & Computer Science Dep. Part 8 Data Warehousing

Info sources for organizations 16 November 2015 Data Warehouse 2 HR Financial/ Accounting ERP CRM and eCRM Internet Procurement Call Center Inventory Islands of information

Applications reporting systems? 16 November 2015Data Warehouse 3 Ad- hoc Standard Reports ParameterizedReports

Data Warehousing 16 November 2015 Data Warehouse 4 Enterprise HR Financial/ Accounting ERP CRM and eCRM Internet Procurement Call Center Inventory

16 November 2015 Data Warehouse 5

16 November 2015 Data Warehouse 6

16 November 2015 Data Warehouse 7 Data Warehouse-- Defined Collection of Data in Support of Management Reporting Needs and Decision-Making Processes. Organized by subject areas (known as data marts) and structured for query. Integrated across subject areas. Associated with a point in time, such as pay period, fiscal year, semester. Query only, that is, the data does not change.

16 November 2015 Data Warehouse 8 A Data Warehouse Designer’s View of the Business MARKETMARKET P R O D U C T T I M E The three key words become the dimensions of the cube. The points inside the cube store the measurements of the business – a combination of Product, Market, and Time. These points are the business facts.

16 November 2015 Data Warehouse 9 Goals of the Data Warehouse Provide Access to various scattered Data. Include Business Descriptions of Data. Promote Consistency Across Subject Areas. Assure Data Quality. Support User Directed Analysis.

16 November 2015 Data Warehouse 10 Runs every night External Data Operational Data ExtractTransformLoad The Data Loading Process

16 November 2015 Data Warehouse 11 External Data ExtractTransformLoad Ready for Access and Query Operational Data Data Warehouse Business Descriptions The Data Loading Process

Characteristics of a Data Warehouse 16 November 2015 Data Warehouse 12 Subject oriented – organized based on use (on subject not on application). Integrated – inconsistencies removed. Nonvolatile – stored in read-only format (historical ). Time variant – data are normally time series. Summarized – in decision-usable format. Large volume – data sets are quite large. Non normalized – often redundant. Metadata – data about data are stored. Data sources – comes from nonintegrated sources.

Data Warehousing Architecture 16 November 2015 Data Warehouse 13 Data Marts ExtractTransformLoad External Sources Operational db systems Serve OLAP servers Analysis Query/ Reporting Data Mining

16 November 2015 Data Warehouse 14 Data Warehouse vs. Data Mart Source: adapted from Strange (1997).

Data Extraction 16 November 2015 Data Warehouse 15 Often performed by COBOL routines (not recommended because of high program maintenance and no automatically generated meta data). Sometimes source data is copied to the target database using the replication capabilities of standard RDMS (not recommended because of “dirty data” in the source systems). Increasing performed by specialized ETL software.

Reasons for “Dirty” Data 16 November 2015 Data Warehouse 16  Dummy Values.  Absence of Data.  Multipurpose Fields.  Cryptic Data.  Contradicting Data.  Inappropriate Use of Address Lines.  Violation of Business Rules.  Reused Primary Keys.  Non-Unique Identifiers.  Data Integration Problems.

Data Cleansing 16 November 2015 Data Warehouse 17 Source systems contain “dirty data” that must be cleansed. ETL software contains rudimentary data cleansing capabilities. Specialized data cleansing software is often used. Leading data cleansing vendors include Vality (Integrity), Harte-Hanks (Trillium), and Firstlogic (i.d.Centric)

Steps in Data Cleansing 16 November 2015 Data Warehouse 18  Parsing  Correcting  Standardizing  Matching  Consolidating

Parsing 16 November 2015 Data Warehouse 19 Parsing locates and identifies individual data elements in the source files and then isolates these data elements in the target files. Examples include parsing the first, middle, and last name; street number and street name; and city and state.

Correcting 16 November 2015 Data Warehouse 20 Corrects parsed individual data components using sophisticated data algorithms and secondary data sources. Example include replacing an incomplete address and adding a zip code.

Standardizing 16 November 2015 Data Warehouse 21 Standardizing applies conversion routines to transform data into its preferred (and consistent) format using both standard and custom business rules. Examples include adding a pre name, replacing a nickname, and using a preferred telephone.

Matching 16 November 2015 Data Warehouse 22 Searching and matching records within and across the parsed, corrected and standardized data based on predefined business rules to eliminate duplications. Examples include identifying similar names and addresses.

Consolidating 16 November 2015 Data Warehouse 23  Analyzing and identifying relationships between matched records and consolidating/merging them into ONE representation.

Data Staging 16 November 2015 Data Warehouse 24 Often used as an interim step between data extraction and later steps. Accumulates data from asynchronous sources using flat files, FTP sessions, or other processes. At a predefined cutoff time, data in the staging file is transformed and loaded to the warehouse. There is usually no end user access to the staging file. An operational data store may be used for data staging.

Data Transformation 16 November 2015 Data Warehouse 25 Transforms the data in accordance with the business rules and standards that have been established. Example include: format changes, deduplication, splitting up fields, replacement of codes, derived values, and aggregates.

Building The Warehouse Building The Warehouse Transforming Data 16 November 2015 Data Warehouse 26 Transform Change Combine Calculate buyer_name Barr, Adam Chai, Sean O’Melia, Erin... reg_id total_sales buyer_name Barr, Adam Chai, Sean O’Melia, Erin... reg_id total_sales buyer_name Barr, Adam Chai, Sean O’Melia, Erin... price qty buyer_name Barr, Adam Chai, Sean O’Melia, Erin... reg_id II IV VI... total_sales buyer_name Barr, Adam Chai, Sean O’Melia, Erin... price qty total_sales buyer_first Adam Sean Erin... buyer_last Barr Chai O’Melia... reg_id total_sales

Data Loading 16 November 2015 Data Warehouse 27 Data are physically moved to the data warehouse. The loading takes place within a “load window”. The trend is to near real time updates of the data warehouse as the warehouse is increasingly used for decisional activities that affect operations.

Meta Data 16 November 2015 Data Warehouse 28 Data about data. Needed by both information technology personnel and users. IT personnel need to know data sources and targets; database, table and column names; refresh schedules; data usage measures; etc. Users need to know entity/attribute definitions; reports/query tools available; report distribution information; help desk contact information, etc.

Reviewed By Mr. Mahmoud Rafeek Alfarra