Data Warehousing and Data Mining J. G. Zheng May 20 th 2008 MIS Chapter 3

2 Types of Information Processing Transactional Processing Focus on data collection, update and simple calculation Analytical Processing Focus on data analysis and decision support

3 Data Warehouse Data warehouse is a special kind of database that stores data from many operational (or transactional) databases supports analytical processing and decision making

4 Why Data Warehouse? Traditional database facilitates data management and transaction processing Two limitations with databases in practice They are transaction oriented and not optimized for complex data analysis Individual databases usually manage data in very different ways, even in the same organization (heterogeneity)

5 Data Warehouse for OLAP Data warehousing approach to satisfy the need for knowledge generation Transaction Processing Analytical Processing

6 Data Warehousing: a Complete View Figure 3.1 on page 127 Should we invest more on our e-business? (fuzzy question need high level analysis for decision making) How do advertising activities affect sales of different products bought by different type of customers, in different regions? (synthesizing) What is the reason for a decrease of total sales this year? (reasoning)

7 Whats the Difference? Data warehouse is (often) multi-dimensional Figure 3.10 on page 145

8 Multi-dimensionality in Depth Star structure Time Sales Data Customer Product Location Fact Table Dimensions

9 An Example in Relational Model Time TimeKey Hour Date Week Month Quarter Year Product ProductKey Product Brand Category Manufacturer Category Location LocationKey Store City State Region Country Customer Customer key Customer AgeGroup Gender CareerGroup Sales TimeKey CustomerKey ProductKey LocationKey Amount Quantity AveUnitPrice

10 Data Mining Data mining (also called knowledge discovery in database, KDD): p rocess and techniques for seeking knowledge (relationship, trends, patterns, etc) from a large amount of data non-trivial, non-obvious implicit knowledge Extremely large datasets

11 Data Mining Tasks What does data mining do? Estimation/prediction Classification/clustering Association/Affinity grouping Market basket analysis in retail

12 Data Mining Techniques Multidimensional analysis (MDA) tools OLAP (online analytic processing) Slice-and-dice Statistical tools Apply mathematical and statistical models, for example, time serials analysis for trend Artificial Intelligence (more in chapter 4)

13 Summary Business intelligence/knowledge comes from data and information Data warehousing is a popular approach to support OLAP and data mining Data mining is a concept of seeking knowledge from large amount of data

14 Good Resources A practitioner's views on data warehousing