1. Data Warehouse and the Star Schema CSCI 242 ©Copyright 2016, David C. Roberts, all rights reserved

2. 2 Agenda Definition Why data warehouse Product History Processing Star queries Data warehouse in the enterprise Data warehouse design Relevance of normalization Star schema Processing the star schema

3. 3 Definition Data warehouse: A repository of integrated information, available for queries and analysis. Data and information are extracted from heterogeneous sources as they are generated The point is that it’s not used for transaction processing; that is, it’s read-only. And the data can come from heterogeneous sources and it can all be queried in one database.

4. 4 Why Data Warehouse A read lock on a table will prevent any updating of a table A long-running analytic operation on all rows of a table will prevent any updates Analysis (a.k.a. decision support) can seriously interfere with updates Using a duplicate table for analysis, recopied once a day, allows unlimited analysis and doesn’t interfere with OLTP.

5. 5 Data Warehouse vs. OLTP OLTPDW PurposeAutomate day-to-day operations Analysis StructureRDBMSRMBMS Data ModelNormalizedDimensional AccessSQLSQL and business analysis programs DataData that runs the businessCurrent and historical information Condition of dataChanging, incompleteHistorical, complete, descriptive

6. Red Brick Invented data warehouse; they sold a hardware product with a star schema database You loaded the Red Brick Warehouse and then queried it for OLTP It featured new optimizations for star schemas, was very fast 6

7. Enter Sybase Sybase learned the optimization and developed their own product. The Sybase product was a stand-alone software data warehouse product It couldn’t do general-purpose database work, was just a data warehouse They appear to have copied the Red Brick idea, without selling hardware 7

8. Enter Oracle Oracle, later, also copied the same optimization They added a bitmap index to their database product, and added the star schema optimization Now their product could do data warehouse as well as database 8

9. Status Today Oracle dominates the field today IBM eventually bought Red Brick so still offers some sort of Red Brick product Sybase offers their OLTP product, now as an offering of SAP 9

10. PROCESSING STAR QUERIES So what is this algorithm that is so copied? 10

11. Optimizing Star Queries Build a bitmap index on each foreign key column of the fact table Index is a 2-dimensional array, one column for each row being indexed, one row per value of that column Bitmap indexes are typically much smaller than b-tree indexes, that can be larger than the data itself 11

12. Bitmap Index Example 12

13. Query Processing The typical query is a join of foreign keys of dimension tables to the fact table This is processed in two phases: 1. From the fact table, retrieve all rows that are part of the result, using bitmap indexes 2. Join the result of the step above to the dimension tables 13

14. Example Query Find sales and profits from the grocery departments of stores in the West and Southwest districts over the last three quarters 14

15. Example Query SELECT store.sales_district, time.fiscal_period, SUM(sales.dollar_sales) revenue, SUM(dollar_sales) - SUM(dollar_cost) income FROM sales, store, time, product WHERE sales.store_key = store.store_key AND sales.time_key = time.time_key AND sales.product_key = product.product_key AND time.fiscal_period IN ('3Q95', '4Q95', '1Q96') and product.department = 'Grocery' AND store.sales_district IN ('San Francisco', 'Los Angeles') GROUP BY store.sales_district, time.fiscal_period; 15

16. Phase 1 Finding the rows in the SALES table (using bitmap indexes): SELECT... FROM sales WHERE store_key IN (SELECT store_key FROM store WHERE sales_district IN ('WEST', 'SOUTHWEST')) AND time_key IN (SELECT time_key FROM time WHERE quarter IN ('3Q96', '4Q96', '1Q97')) AND product_key IN (SELECT product_key FROM product WHERE department = 'GROCERY'); 16

17. Phase 2 Now the fact table is joined to dimension tables. For dimension tables of small cardinality, a full-table scan may be used. For large cardinality, a hash join could be used. 17

18. The Star Transformation Use bitmap indexes to retrieve all relevant rows from the fact table, based on foreign key values – This happens very fast Join this result set to the dimension tables – If there are many values, a hash join may be used – If there are fewer values, a b-tree driven join may be used 18

19. 19 How DW Fits into the Enterprise OLTP3 Data Mart Data Warehouse Data Mart Data Mart Data Mart Application A Application B Application C User Extract, Transform And Load OLTP2 OLTP1 Integration

20. 20 Data Warehouse Database Design A conventional database design for data warehouse would lead to joins on large amounts of data that would run slowly The star schema allows for fast processing of very large quantities of data in the data warehouse It also allows for very compact representation of events that occur many times

21. 21 A Sample OLTP Schema orders products order items customers

22. 22 Transformed to a Star Schema products customers sales channels times fact table dimension table dimension table dimension table dimension table

23. 23 Star Schema Fact Table Customer Item Supplier Time Location

24. 24 Fact Table The fact table contains the actual business process measurements or metrics for a specific event, called facts, usually numbers. A fact table represents facts by foreign keys from other tables, called “dimension” tables These foreign keys are usually generated keys, in order to save fact table space If you are building a DW of monthly sales in dollars, your fact table will contain monthly sales, one row per month. If you are building a DW of retail sales, each row of the fact table might have one row for each item sold.

25. 25 Fact Table Design A fact table may contain one or more facts. Usually you create one fact table per business event. For example if you want to analyze the sales numbers and also advertising spending, they are two separate business processes. So you will create two separate fact tables, one for sales data and one for advertising cost data. On the other hand if you want to track the sales tax in addition to the sales number, you simply create one more fact column in the Sales fact table called Tax.

26. 26 Dimension Table Dimension tables have a small number of rows (compared to fact tables) but a large number of columns For the lowest level of granularity of a fact in the fact table, a dimension table will have one row that gives all the categories for each value The dimension table is often all key, so a generated key is used so that the fact table reference to the dimension table can be small

27. 27

28. 28 Time Dimension Schema Column NameType Dim_IdINTEGER (4) MonthSMALL INTEGER (2) Month_NameVARCHAR (3) QuarterSMALL INTEGER (4) Quarter_NameVARCHAR (2) YearSMALL INTEGER (2)

29. 29 Time Dimension Data TM _Dim_IdTM _MonthTM_Month_NameTM _Quarter TM_Quarter_N ame TM_Year 1001 1Jan1Q12003 10022Feb1Q12003 10033Mar1Q12003 10044Apr2Q22003 10055May2Q22003

30. 30 Location Dimension Schema Column NameType Dim_IdINTEGER (4) Loc_CodeVARCHAR (4) NameVARCHAR (50) State_NameVARCHAR (20) Country_NameVARCHAR (20)

31. 31 Location Dimension Data Dim_IdLoc_CodeNameState_NameCountry_Name 1001 IL01Chicago LoopIllinoisUSA 1002 IL02Arlington HtsIllinoisUSA 1003NY01BrooklynNew YorkUSA 1004TO01TorontoOntarioCanada 1005MX01Mexico CityDistrito FederalMexico

32. 32 Product Data Schema Column NameType Dim_IdINTEGER (4) SKUVARCHAR (10) NameVARCHAR (30) CategoryVARCHAR (30)

33. 33 Product Data Dim_IdSKUNameCategory 1001DOVE6KDove Soap 6PkSanitary 1002MLK66F#Skim Milk 1 GalDairy 1003SMKSAL55Smoked Salmon 6ozMeat

34. Categories in Dimension Tables Categories may or may not be hierarchical; or can be both Categories provide canned values that can be given to users for queries 34

35. 35 Granularity (Grain) of the Fact Table The level of detail of the fact table is known as the grain of the fact table. In this example the grain of the fact table is monthly sales number per location per product.

36. Note about Granularity There may be multiple star schemas at different levels of granularity, especially for very large data warehouses The first could be the finest—say, each transaction such as a sale The next could be an aggregation, like the previous example There could be more levels of aggregation 36

37. 37 Design Approach 1. Identify the business process. In this step you will determine what is your business process that your data warehouse represents. This process will be the source of your metrics or measurements. 2. Identify the Grain You will determine what does one row of fact table mean. In the previous example you have decided that your grain is 'monthly sales per location per product'. It might be daily sales or even each sale could be one row. 3. Identify the Dimensions Your dimensions should be descriptive (SQL VARCHAR or CHARACTER) as much as possible and conform to your grain. 4. Finally Identify the facts In this step you will identify what are your measurements (or metrics or facts). The facts should be numeric and should confirm to the grain defined in step 2.

38. 38 Monthly Sales Fact Table Schema Field NameType TM_Dim_IdINTEGER (4) PR_ Dim_IdINTEGER (4) LOC_ Dim_IdINTEGER (4) SalesINTEGER (4)

39. 39 Monthly Sales Fact Table Data TM_Dim_IdPR_ Dim_IdLOC_ Dim_IdSales 1001 1003435677 1002 1001451121 10031001100398765 10011004100165432

40. 40 Data Mart A data mart is a collection of subject areas organized for decision support based on the needs of a given department. Examples: finance has their data mart, marketing has theirs, sales has theirs and so on. Each department generally runs its own data mart. Ownership of the data mart allows each department to bypass the control that might coordinate the data found in the different departments. Each department's data mart is peculiar to and specific to its own needs. Typically, the database design for a data mart is built around a star-join structure designed for that department. The data mart contains only a modicum of historical information and is granular only to the point that it suits the needs of the department. The data mart may also include data from outside the organization, such as purchased normative salary data that might be purchased by an HR department.

41. 41 About the Data Mart The structure of the data in the data mart may or may not be compatible with the structure of data in the data warehouse. The amount of historical data found in the data mart is different from the history of the data found in the warehouse. Data warehouses contain robust amounts of history, while data marts usually contain modest amounts of history. The subject areas found in the data mart are only faintly related to the subject areas found in the data warehouse. The relationships found in the data mart may not be those relationships that are found in the data warehouse. The types of queries satisfied in the data mart are quite different from those queries found in the data warehouse.

42. Walmart’s Data Warehouse Half a petabyte in capacity (.5 x 10 15 bytes) World’s largest DW Tracks 100 million customers buying billions of products every week Every sale from every store is transmitted to Bentonville every night Walmart has more than 18,000 retail stores, employs 2.2 million, serves 245 million customers every week 42

43. Typical Questions How much orange juice did we sell last year, last month, last week in store X? What internal factors (position in store, advertising campaigns...) influence orange juice sales? How much orange juice are we going to sell next week, next month, next year? 43

44. 44