Han: Dataware Houses and OLAP 1 What is Data Warehouse? Defined in many different ways, but not rigorously. A decision support database that is maintained.

Slides:



Advertisements
Similar presentations
Outline What is a data warehouse? A multi-dimensional data model Data warehouse architecture Data warehouse implementation Further development of data.
Advertisements

April 30, Data Warehousing and OLAP Technology: An Overview  What is a data warehouse?  Data warehouse architecture  From data warehousing to.
Data Warehousing.
Data Warehousing Willem Visser RW334. Somebody is watching! Everybody seems to be recording your every move Loyalty cards Cookies – Facebook, Twitter,…
Data Warehouses and Data Cubes
Introduction to Data Warehousing CPS Notes 6.
1 Lecture 09: OLAP
ICS 421 Spring 2010 Data Warehousing (1) Asst. Prof. Lipyeow Lim Information & Computer Science Department University of Hawaii at Manoa 3/18/20101Lipyeow.
The Role of Data Warehousing and OLAP Technologies CS 536 – Data Mining These slides are adapted from J. Han and M. Kamber’s book slides (
1 Data Warehouse Min Song IS Data Warehousing and OLAP Technology for Data Mining  What is a data warehouse?  A multi-dimensional data model 
Instructor: Pedro Domingos
Data Warehouses and OLAP
Data Warehousing Xintao Wu. Evolution of Database Technology (See Fig. 1.1) 1960s: Data collection, database creation, IMS and network DBMS 1970s: Relational.
Data Warehousing.
Dr. M. Sulaiman Khan Dept. of Computer Science University of Liverpool 2010 COMP207: Data Mining Data Warehousing COMP207: Data Mining.
1 Lecture 10: More OLAP - Dimensional modeling
© Tan,Steinbach, Kumar Introduction to Data Mining 8/05/ Data Warehouse and Data Cube Lecture Notes for Chapter 3 Introduction to Data Mining By.
Ch3 Data Warehouse part2 Dr. Bernard Chen Ph.D. University of Central Arkansas Fall 2009.
1 Data Warehousing and OLAP. 2 Data Warehousing & OLAP Defined in many different ways, but not rigorously.  A decision support database that is maintained.
CS346: Advanced Databases
Ch3 Data Warehouse Dr. Bernard Chen Ph.D. University of Central Arkansas Fall 2010.
1 Data Warehouses C hapter 2. 2 Chapter 2 Outline Chapter 2 Outline – Introduction –Data Warehouses –Data Warehouse in Organisation – OLTP vs. OLAP –Why.
August 14, 2015Data Mining: Concepts and Techniques 1 Chapter 3: Data Warehousing and OLAP Technology: An Overview What is a data warehouse? Data warehouse.
Dr. Bernard Chen Ph.D. University of Central Arkansas
8/20/ Data Warehousing and OLAP. 2 Data Warehousing & OLAP Defined in many different ways, but not rigorously. Defined in many different ways, but.
Data Warehousing: and OLAP —BIS 541— — Chapter 3 —
August 25, 2015Data Mining: Concepts and Techniques 1 Data Warehousing What is a data warehouse? A multi-dimensional data model Data warehouse architecture.
Data Warehouses and OLAP — Slides for Textbook — — Chapter 2 —
An overview of Data Warehousing and OLAP Technology
August 25, 2015Data Mining: Concepts and Techniques 1 Data Mining: Concepts and Techniques — Chapter 3 —
School of Management, HUST
©Jiawei Han and Micheline Kamber
Data Warehouses and OLAP — Slides for Textbook — — Chapter 2 —
1 Data Warehousing and Decision Support. 2 Data Warehousing and OLAP Technology What is a data warehouse? A multi-dimensional data model Data warehouse.
Data warehousing and online analytical processing- Ref Chap 4) By Asst Prof. Muhammad Amir Alam.
Data Warehousing Xintao Wu. Can You Easily Answer These Questions? What are Personnel Services costs across all departments for all funding sources? What.
OLAP & DSS SUPPORT IN DATA WAREHOUSE By - Pooja Sinha Kaushalya Bakde.
Roadmap 1.What is the data warehouse, data mart 2.Multi-dimensional data modeling 3.Data warehouse design – schemas, indices 4.The Data Cube operator –
October 28, Data Warehouse Architecture Data Sources Operational DBs other sources Analysis Query Reports Data mining Front-End Tools OLAP Engine.
Dr. N. MamoulisAdvanced Database Technologies1 Topic 6: Data Warehousing & OLAP Defined in many different ways, but not rigorously. A decision support.
Ch3 Data Warehouse Dr. Bernard Chen Ph.D. University of Central Arkansas Fall 2009.
1 CSE 592 Data Mining Instructor: Pedro Domingos.
Data Mining Data Warehouses.
Han: Dataware Houses and OLAP 1 Data Warehouses and OLAP — Slides for Textbook — — Chapter 2 — ©Jiawei Han and Micheline Kamber Intelligent Database Systems.
Managing Data for DSS II. Managing Data for DS Data Warehouse Common characteristics : –Database designed to meet analytical tasks comprising of data.
M. Sulaiman Khan Dept. of Computer Science University of Liverpool 2009 This is the full course notes, but not quite complete. You.
January 21, 2016Data Mining: Concepts and Techniques 1 Chapter 3: Data Warehousing and OLAP Technology: An Overview What is a data warehouse? A multi-dimensional.
Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke1 Data Warehousing and Decision Support Chapter 25.
Datawarehousing and OLAP C.Eng 714 Spring
Introduction to OLAP and Data Warehouse Assoc. Professor Bela Stantic September 2014 Database Systems.
CPT-S Advanced Databases 1 Yinghui Wu EME 49 ADB (ln27)
Data Warehouses and OLAP. Data Warehousing and OLAP Technology for Data Mining  What is a data warehouse?  A multi-dimensional data model  Data warehouse.
Data Mining and Data Warehousing: Concepts and Techniques What is a Data Warehouse? Data Warehouse vs. other systems, OLTP vs. OLAP Conceptual Modeling.
Data Mining: Data Warehousing
Introduction to Data Warehousing
Information Management course
Data warehouse and OLAP
Information Management course
Data Warehouse—Subject‐Oriented
OLAP Concepts and Techniques
Data Warehouse.
Data Warehousing and OLAP Technology for Data Mining
Chapter 2: Data Warehousing and OLAP Technology for Data Mining
Data Warehouse and OLAP
Lecture 4: From Data Cubes to ML
Overview of Data Warehousing and OLAP
Data Warehousing and Decision Support Chapter 25
Dr. Bernard Chen Ph.D. University of Central Arkansas Fall 2009
Data Mining: Concepts and Techniques
Data Warehouse and OLAP
Presentation transcript:

Han: Dataware Houses and OLAP 1 What is Data Warehouse? Defined in many different ways, but not rigorously. A decision support database that is maintained separately from the organization’s operational database Support information processing by providing a solid platform of consolidated, historical data for analysis. “A data warehouse is a subject-oriented, integrated, time-variant, and nonvolatile collection of data in support of management’s decision-making process.”—W. H. Inmon Data warehousing: The process of constructing and using data warehouses

Han: Dataware Houses and OLAP 2 Data Warehousing and OLAP/Multi- dimensional Data Model 1. What is a data warehouse? 2. Specific Software for Data Warehousing: OLAP (Online Analytical Processing) / The Multi- Dimensional Data Model / Data Cubes

Han: Dataware Houses and OLAP 3 Data Warehouse—Subject-Oriented Organized around major subjects, such as customer, product, sales. Focusing on the modeling and analysis of data for decision makers, not on daily operations or transaction processing. Provide a simple and concise view around particular subject issues by excluding data that are not useful in the decision support process.

Han: Dataware Houses and OLAP 4 Data Warehouse—Integrated Constructed by integrating multiple, heterogeneous data sources relational databases, flat files, on-line transaction records Data cleaning and data integration techniques are applied. Ensure consistency in naming conventions, encoding structures, attribute measures, etc. among different data sources E.g., Hotel price: currency, tax, breakfast covered, etc. When data is moved to the warehouse, it is converted.

Han: Dataware Houses and OLAP 5 Data Warehouse—Time Variant The time horizon for the data warehouse is significantly longer than that of operational systems. Operational database: current value data. Data warehouse data: provide information from a historical perspective (e.g., past 5-10 years) Every key structure in the data warehouse Contains an element of time, explicitly or implicitly But the key of operational data may or may not contain “time element”.

Han: Dataware Houses and OLAP 6 Data Warehouse—Non-Volatile A physically separate store of data transformed from the operational environment. Operational update of data does not occur in the data warehouse environment. Does not require transaction processing, recovery, and concurrency control mechanisms Requires only two operations in data accessing: initial loading of data and access of data.

Han: Dataware Houses and OLAP 7 Data Warehouse vs. Heterogeneous DBMS Traditional heterogeneous DB integration: Build wrappers/mediators on top of heterogeneous databases Query driven approach When a query is posed to a client site, a meta-dictionary is used to translate the query into queries appropriate for individual heterogeneous sites involved, and the results are integrated into a global answer set Complex information filtering, compete for resources Data warehouse: update-driven, high performance Information from heterogeneous sources is integrated in advance and stored in warehouses for direct query and analysis

Han: Dataware Houses and OLAP 8 OLTP vs. OLAP (Online Analytical Processing )

Han: Dataware Houses and OLAP 9 Why Separate Data Warehouse? High performance for both systems DBMS— tuned for OLTP: access methods, indexing, concurrency control, recovery Warehouse—tuned for OLAP: complex OLAP queries, multidimensional view, consolidation. Different functions and different data: missing data: Decision support requires historical data which operational DBs do not typically maintain data consolidation: DS requires consolidation (aggregation, summarization) of data from heterogeneous sources data quality: different sources typically use inconsistent data representations, codes and formats which have to be reconciled

Han: Dataware Houses and OLAP 10 From Tables and Spreadsheets to Data Cubes A data warehouse is based on a multidimensional data model which views data in the form of a data cube A data cube, such as sales, allows data to be modeled and viewed in multiple dimensions Dimension tables, such as item (item_name, brand, type), or time(day, week, month, quarter, year) Fact table contains measures (such as dollars_sold) and keys to each of the related dimension tables In data warehousing literature, an n-D base cube is called a base cuboid. The top most 0-D cuboid, which holds the highest-level of summarization, is called the apex cuboid. The lattice of cuboids forms a data cube.

Han: Dataware Houses and OLAP 11 OLAP Terminology A data cube supports viewing/modelling of a variable (a set of variables) of interest. Measures are used to report the values of the particular variable with respect to a given set of dimensions. A fact table stores measures as well as keys representing relationships to various dimensions. Dimensions are perspectives with respect to which an organization wants to keep record. A star schema defines a fact table and its associated dimensions.

Han: Dataware Houses and OLAP 12 Conceptual Modeling of Data Warehouses Modeling data warehouses: dimensions & measures Star schema: A fact table in the middle connected to a set of dimension tables Snowflake schema: A refinement of star schema where some dimensional hierarchy is normalized into a set of smaller dimension tables, forming a shape similar to snowflake Fact constellations: Multiple fact tables share dimension tables, viewed as a collection of stars, therefore called galaxy schema or fact constellation

Han: Dataware Houses and OLAP 13 Example of Star Schema time_key day day_of_the_week month quarter year time location_key street city province_or_street country location Sales Fact Table time_key item_key branch_key location_key units_sold dollars_sold avg_sales Measures item_key item_name brand type supplier_type item branch_key branch_name branch_type branch

Han: Dataware Houses and OLAP 14 A Concept Hierarchy: Dimension (location) all EuropeNorth_America MexicoCanadaSpainGermany Vancouver M. WindL. Chan... all region office country TorontoFrankfurtcity

Han: Dataware Houses and OLAP 15 View of Warehouses and Hierarchies Specification of hierarchies Schema hierarchy day < {month < quarter; week} < year Set_grouping hierarchy {1..10} < inexpensive

Han: Dataware Houses and OLAP 16 Multidimensional Data Sales volume as a function of product, month, and region Product Region Month Dimensions: Product, Location, Time Hierarchical summarization paths Industry Region Year Category Country Quarter Product City Month Week Office Day

Han: Dataware Houses and OLAP 17 A Sample Data Cube Total annual sales of TV in U.S.A. Date Product Country sum TV VCR PC 1Qtr 2Qtr 3Qtr 4Qtr U.S.A Canada Mexico sum

Han: Dataware Houses and OLAP 18 Browsing a Data Cube Visualization OLAP capabilities Interactive manipulation

Han: Dataware Houses and OLAP 19 Typical OLAP Operations Roll up (drill-up): summarize data by climbing up hierarchy or by dimension reduction Drill down (roll down): reverse of roll-up from higher level summary to lower level summary or detailed data, or introducing new dimensions Slice and dice: project and select Pivot (rotate): reorient the cube, visualization, 3D to series of 2D planes. Other operations drill across: involving (across) more than one fact table drill through: through the bottom level of the cube to its back- end relational tables (using SQL)

Han: Dataware Houses and OLAP 20 A Star-Net Query Model Shipping Method AIR-EXPRESS TRUCK ORDER Customer Orders CONTRACTS Customer Product PRODUCT GROUP PRODUCT LINE PRODUCT ITEM SALES PERSON DISTRICT DIVISION OrganizationPromotion CITY COUNTRY REGION Location DAILYQTRLYANNUALY Time Each circle is called a footprint

Views and Decision Support OLAP queries are typically aggregate queries. Precomputation is essential for interactive response times. The CUBE is in fact a collection of aggregate queries, and precomputation is especially important: lots of work on what is best to precompute given a limited amount of space to store precomputed results. Warehouses can be thought of as a collection of asynchronously replicated tables and periodically maintained views. Has renewed interest in view maintenance!

Issues in View Materialization What views should we materialize, and what indexes should we build on the precomputed results? Given a query and a set of materialized views, can we use the materialized views to answer the query? How frequently should we refresh materialized views to make them consistent with the underlying tables? (And how can we do this incrementally?)

Han: Dataware Houses and OLAP 23 Discovery-Driven Exploration of Data Cubes Hypothesis-driven: exploration by user, huge search space Discovery-driven (Sarawagi et al.’98) pre-compute measures indicating exceptions, guide user in the data analysis, at all levels of aggregation Exception: significantly different from the value anticipated, based on a statistical model Visual cues such as background color are used to reflect the degree of exception of each cell Computation of exception indicator (modeling fitting and computing SelfExp, InExp, and PathExp values) can be overlapped with cube construction

Han: Dataware Houses and OLAP 24 Examples: Discovery-Driven Data Cubes

Han: Dataware Houses and OLAP 25 Data Warehouse Usage Three kinds of data warehouse applications Information processing supports querying, basic statistical analysis, and reporting using crosstabs, tables, charts and graphs Analytical processing and Interactive Analysis multidimensional analysis of data warehouse data supports basic OLAP operations, slice-dice, drilling, pivoting Data mining knowledge discovery from hidden patterns supports associations, constructing analytical models, performing classification and prediction, and presenting the mining results using visualization tools. Differences among the three tasks

Han: Dataware Houses and OLAP 26 Software to Work with Data Cubes

Han: Dataware Houses and OLAP 27 Summary Data warehouse A subject-oriented, integrated, time-variant, and nonvolatile collection of data in support of management’s decision-making process A multi-dimensional model of a data warehouse Star schema, snowflake schema, fact constellations A data cube consists of dimensions & measures OLAP operations: drilling, rolling, slicing, dicing and pivoting Efficient computation of data cubes Partial vs. full vs. no materialization Special index structures (not discussed) Further development of data cube technology Discovery-drive and multi-feature cubes From OLAP to OLAM (on-line analytical mining)

Han: Dataware Houses and OLAP 28 References (I) S. Agarwal, R. Agrawal, P. M. Deshpande, A. Gupta, J. F. Naughton, R. Ramakrishnan, and S. Sarawagi. On the computation of multidimensional aggregates. In Proc Int. Conf. Very Large Data Bases, , Bombay, India, Sept D. Agrawal, A. E. Abbadi, A. Singh, and T. Yurek. Efficient view maintenance in data warehouses. In Proc ACM-SIGMOD Int. Conf. Management of Data, , Tucson, Arizona, May R. Agrawal, J. Gehrke, D. Gunopulos, and P. Raghavan. Automatic subspace clustering of high dimensional data for data mining applications. In Proc ACM-SIGMOD Int. Conf. Management of Data, , Seattle, Washington, June R. Agrawal, A. Gupta, and S. Sarawagi. Modeling multidimensional databases. In Proc Int. Conf. Data Engineering, , Birmingham, England, April K. Beyer and R. Ramakrishnan. Bottom-Up Computation of Sparse and Iceberg CUBEs. In Proc ACM-SIGMOD Int. Conf. Management of Data (SIGMOD'99), , Philadelphia, PA, June S. Chaudhuri and U. Dayal. An overview of data warehousing and OLAP technology. ACM SIGMOD Record, 26:65-74, OLAP council. MDAPI specification version 2.0. In J. Gray, S. Chaudhuri, A. Bosworth, A. Layman, D. Reichart, M. Venkatrao, F. Pellow, and H. Pirahesh. Data cube: A relational aggregation operator generalizing group-by, cross-tab and sub- totals. Data Mining and Knowledge Discovery, 1:29-54, 1997.

Han: Dataware Houses and OLAP 29 References (II) V. Harinarayan, A. Rajaraman, and J. D. Ullman. Implementing data cubes efficiently. In Proc ACM-SIGMOD Int. Conf. Management of Data, pages , Montreal, Canada, June Microsoft. OLEDB for OLAP programmer's reference version 1.0. In K. Ross and D. Srivastava. Fast computation of sparse datacubes. In Proc Int. Conf. Very Large Data Bases, , Athens, Greece, Aug K. A. Ross, D. Srivastava, and D. Chatziantoniou. Complex aggregation at multiple granularities. In Proc. Int. Conf. of Extending Database Technology (EDBT'98), , Valencia, Spain, March S. Sarawagi, R. Agrawal, and N. Megiddo. Discovery-driven exploration of OLAP data cubes. In Proc. Int. Conf. of Extending Database Technology (EDBT'98), pages , Valencia, Spain, March E. Thomsen. OLAP Solutions: Building Multidimensional Information Systems. John Wiley & Sons, Y. Zhao, P. M. Deshpande, and J. F. Naughton. An array-based algorithm for simultaneous multidimensional aggregates. In Proc ACM-SIGMOD Int. Conf. Management of Data, , Tucson, Arizona, May 1997.