09. Data Warehouse (DW) & On-line Analytic Processing (OLAP) Rev: Feb, 2013 Euiho (David) Suh, Ph.D. POSTECH Strategic Management of Information and Technology Laboratory (POSMIT: Dept. of Industrial & Management Engineering POSTECH

Contents 1Data Warehouse 1)Introduction of Data Warehouse 2)Concepts for Data Warehouse 3)Difficulties and Trends 2On-line Analytic Processing (OLAP) 1)Introduction of OLAP 2)Concepts for OLAP 3 Case Study

3 ■Data Warehouse –Stores static data that has been extracted from other databases in an organization –Central source of data that has been cleaned, transformed, and cataloged –Data is used for data mining, analytical processing, analysis, research, decision support Definition of Data Warehouse 1. Data Warehouse 1) Introduction of Data Warehouse Integrated Non-volatile Time variant A data warehouse is a collection of data in support of management’s decisions Scattered Information Cleaned Data WarehouseQuery & Distribute to End User Sales HR Cost Finance Bond Customer

4 ■Data Warehouse architecture Data Warehouse Architecture 1. Data Warehouse 1) Introduction of Data Warehouse * Building the Data Warehouse *Use of Data Warehouse Data Warehouse External file OLTP System Back up file Enterprise server Workgroup server Query, Reporting tool OLAP tool Datamining Application EIS/DSS Application Web browser Slice/Dice SQL Data Mart Source Data MDB RDB Infra, Data integration and Administration Application development, Data access & Use

5 ■Technical architecture for a data warehousing system Data Warehouse Architecture Data Acquisition Component Design Component Data Manager Component Information Directory Component Data Delivery Component Middleware Component Data Access Component warehouse data warehouse metadata external data external metadata source data Management Component 1. Data Warehouse 1) Introduction of Data Warehouse

6 ■Definition of database –Integrated collection of logically related data elements ■Common Database Structures (Types) –Hierarchical Early DBMS structure Records arranged in tree-like structure Relationships are one-to-many –Network Used in some mainframe DBMS packages Many-to-many relationships –Relational Most widely used structure Data elements are stored in tables Row represents a record; column is a field Can relate data in one file with data in another, if both files share a common data element –Multidimensional Variation of relational model Uses multidimensional structures to organize data Data elements are viewed as being in cubes Popular for analytical databases that support Online Analytical Processing (OLAP) –Object-Oriented Store data together with the appropriate methods for accessing it i.e. encapsulation Information is represented in the form of objects as used in object-oriented programming Introduction of Database 1. Data Warehouse 2) Concepts for Data Warehouse Relational Structure Object-Oriented Structure

7 ■Metadata –Data about data (similar to catalog card in library) –Define the data in the data warehouse –Enable to find the data in data warehouse, more easily and fast ■Data Marts –Collection of database –Comparing with Data Warehouse, data marts are usually smaller and focus on a particular subject or department. –Data marts are subsets of larger Data Warehouse ■Data Warehouse vs. Data Mart –Data in Data Warehouse The data needs to be gathered from all the relevant transactional systems that produce it, cleansed and validated, and made available from a system-of-record that ensures the referential integrity of the data –Data in Data Mart The data needs to be presented in a structure that is intuitive to the users and facilitates their ability to query the data that is relevant to their needs Metadata and Data Marts 1. Data Warehouse 2) Concepts for Data Warehouse

8 ■Data Warehouse built on top of DB Information Flow 1. Data Warehouse 2) Concepts for Data Warehouse Internal / External Database Data Warehouse Metadata Repository Internal / External Database Data Marts Finance Management Reporting Accounting Sales Marketing

9 ■Data Warehouse Components Data Warehouse Components 1. Data Warehouse 2) Concepts for Data Warehouse

10 ■Applications and Data Marts Applications and Data Marts 1. Data Warehouse 2) Concepts for Data Warehouse

11 Difficulties in implementing DW ■Complete Alignment –Make sure you have full involvement and buy -in from those that represent your users - the consumers of your data warehouse. ■Iterative & Frequent Update –Consider all aspects of the process of researching your data sources, capturing and transmitting that data to the data warehouse, transforming and loading it into the data warehouse and accounting for its lineage. ■Risk –Make sure you develop a proper risk management plan. 1. Data Warehouse 3) Difficulties and Trends

12 Future Trends ■Enterprise Data Warehouse –The enterprise data warehouse, whether a single store or integrated data marts across a variety of platforms, yields a view of the operation previously unattainable by Don Hatcher, SAS ■Real-time –Organization move to more real-time data transformation and seek to better leverage common metadata across applications by Allan Houpt, CA ■Capacity –The future of data warehousing is all about ever larger data warehouses - in fact I just read about a U.S. Government effort to create petabyte repositories by Roman Bukary, SAP Director of Market Strategy 1. Data Warehouse 3) Difficulties and Trends

13 Definition of OLAP ■OLAP (On-Line Analytical Processing) –The dynamic enterprise analysis required to create, manipulate, animate and synthesis information from Enterprise Data Models * Providing OLAP: An IT Mandate E.F. Codd (1993) –FASMI (Fast Analysis of Shared Multidimensional Information) This definition was first used in early 1995, and has not needed revision since Pendse & Greeth (1995) 2. OLAP 1) Introduction of OLAP FAST ANALYSIS SHARED MULTIDIMENSIONAL INFORMATION

14 OLAP Architecture ■OLAP Architecture 2. OLAP 1) Introduction of OLAP

15 From OLTP to OLAP ■Data used in OLAP –Sales data of June? (OLTP) –Multi-dimensional data (having many features) (OLAP) ■ Direct Access: EUC Environment ■From What to Why –OLTP: Storing primitive data, supporting routine business operation (What) –OLAP: Storing cumulative data, supporting business goal (Why) 2. OLAP 2) Concepts for OLAP Information Source Information Broker Information Consumer

16 OLTP vs. OLAP ■OLTP vs. OLAP 2. OLAP 2) Concepts for OLAP OLTPOLAP DefinitionOn-Line Transaction ProcessingOn-Line Analytical Processing ObjectiveOperationalAnalytical FocusDaily repetitious workDecision support in organization DeveloperComputer expertEnd-user UserSimple operatorSpecial analyst StoringCurrent valueSummarized and Consolidated data UseRepetitiveUnstructured ResponseImmediateDelayed DataUpdatedSummarized UpdateFieldRecomputation Amount of DataSmallMuch Data StructureComplexSimple DatabaseRDBMDB Data periodPast, CurrentPast, Current, Future Query typeRegularIrregular, Analytical

17 Enterprise IT Architecture ■OLTP/OLAP Enterprise IT Architecture 2. OLAP 2) Concepts for OLAP

18 Data Warehouse vs. OLAP Server ■Data Warehouse vs. OLAP Server 2. OLAP 2) Concepts for OLAP Data WarehouseOLAP Server ObjectiveReady to all kinds of retrievalSpecialized retrieval CharacteristicsData StorageComputation Engine Query TypeRead onlyRead/Write ResponseFlexibleConsistent, rapid ContentHistorical, presentHistorical, present, Future Data StructurePlainMulti-dimensional Amount of DataHuge, much detailMuch, detail Development periodA few month, yrsA few weeks, months

19 Two types of OLAP ■MOLAP ■ROLAP 2. OLAP 2) Concepts for OLAP Clients MDBMS RDBMSMD Processing QuerySQL Respond MD Processing Query Respond

20 From RDB to MDB ■Basic Data Structure of MDB & RDB –RDB: OLTP, Data Warehouse ■RDB as OLAP Server –Cannot handle and represent Multi-dimensional relationship well –Cannot summarize data well ■ MDB as OLAP Server – Gives many managerial viewpoints – EUC – Supports analysis functionality Table Field, Row Record, Column Cube Dimension Hierarchy –MDB: OLAP 2. OLAP 2) Concepts for OLAP

21 Reference ■Euiho Suh, “EIS_DSS_OLAP_DW (PPT Slide)”, POSMIT Lab. (POSTECH Strategic Management of Information and Technology Laboratory)Euiho Suh, “EIS_DSS_OLAP_DW (PPT Slide)”, POSMIT Lab. (POSTECH Strategic Management of Information and Technology Laboratory) ■Euiho Suh, “OLAP (PPT Slide)”, POSMIT Lab. (POSTECH Strategic Management of Information and Technology Laboratory)Euiho Suh, “OLAP (PPT Slide)”, POSMIT Lab. (POSTECH Strategic Management of Information and Technology Laboratory) ■O’Brien & Marakas, “Introduction to Information Systems – Sixteenth Edition”, McGraw – Hill, Chapter 5O’Brien & Marakas, “Introduction to Information Systems – Sixteenth Edition”, McGraw – Hill, Chapter 5