Dr. M. Sulaiman Khan Dept. of Computer Science University of Liverpool 2010 COMP207: Data Mining Data Warehousing COMP207: Data Mining

Data Warehouses Data Cubes Warehouse Schemas OLAP Materialisation Today's Topics Data Warehousing COMP207: Data Mining

Most common definition: “A data warehouse is a subject-oriented, integrated, time-variant and nonvolatile collection of data in support of management's decision-making process.” - W. H. Inmon  Corporate focused, assumes a lot of data, and typically sales related  Data for “Decision Support System” or “Management Support System”  1996 survey: Return on Investment of 400+% Data Warehousing: Process of constructing (and using) a data warehouse What is a Data Warehouse? Data Warehousing COMP207: Data Mining

 Subject-oriented: Focused on important subjects, not transactions Concise view with only useful data for decision making  Integrated: Constructed from multiple, heterogeneous data sources. Normally distributed relational databases, not necessarily same schema. Cleaning, pre-processing techniques applied for missing data, noisy data, inconsistent data (sounds familiar, I hope)‏ Data Warehouse Data Warehousing COMP207: Data Mining

 Time-variant: Has different values for the same fields over time. Operational database only has current value. Data Warehouse offers historical values.  Nonvolatile: Physically separate store Updates not online, but in offline batch mode only Read only access required, so no concurrency issues Data Warehouse Data Warehousing COMP207: Data Mining

Data Warehouses are distinct from: Distributed DB: Integrated via wrappers/mediators. Far too slow, semantic integration much more complicated. Integration done before loading, not at run time. Operational DB: Only records current value, lots of extra non useful information. Different schemas/models, access patterns, users, functions, even though the data is derived from an operational db. Data Warehouse Data Warehousing COMP207: Data Mining

OLAP: Online Analytical Processing (Data Warehouse)‏ OLTP: Online Transaction Processing (Traditional DBMS)‏ OLAP data typically: historical, consolidated, and multi- dimensional (eg: product, time, location). Involves lots of full database scans, across terabytes or more of data. Typically aggregation and summarisation functions. Distinctly different uses to OLTP on the operational database. OLAP vs OLTP Data Warehousing COMP207: Data Mining

Data is normally Multi-Dimensional, and can be thought of as a cube. Often: 3 dimensions of time, location and product. No need to have just 3 dimensions -- could have one for cars with make, colour, price, location, and time for example. Image courtesy of IBM OLAP Miner documentation Data Cubes Data Warehousing COMP207: Data Mining

 Can construct many 'cuboids' from the full cube by excluding dimensions.  In an N dimensional data cube, the cuboid with N dimensions is the 'base cuboid'. A 0 dimensional cuboid (other than non existent!) is called the 'apex cuboid'.  Can think of this as a lattice of cuboids... (Following lattice courtesy of Han & Kamber) Data Cubes Data Warehousing COMP207: Data Mining

Lattice of Cuboids Data Warehousing COMP207: Data Mining all timeitemlocationsupplier time,itemtime,location time,supplier item,location item,supplier location,supplier time,item,location time,item,supplier time,location,supplier item,location,supplier time, item, location, supplier 0-D(apex) cuboid 1-D cuboids 2-D cuboids 3-D cuboids 4-D(base) cuboid

Each dimension can also be thought of in terms of different units.  Time: decade, year, quarter, month, day, hour (and week, which isn't strictly hierarchical with the others!)‏  Location: continent, country, state, city, store  Product: electronics, computer, laptop, dell, inspiron This is called a “Star-Net” model in data warehousing, and allows for various operations on the dimensions and the resulting cuboids. Multi-dimensional Units Data Warehousing COMP207: Data Mining

Star-Net Model Data Warehousing COMP207: Data Mining Shipping Method AIR-EXPRESS TRUCK ORDER Customer Orders CONTRACTS Customer Product PRODUCT GROUP PRODUCT LINE PRODUCT ITEM SALES PERSON DISTRICT DIVISION OrganizationPromotion DISTRICT REGION COUNTRY Geography DAILY QTRLY ANNUALY Time

 Roll Up: Summarise data by climbing up hierarchy. Eg. From monthly to quarterly, from Liverpool to England  Drill Down: Opposite of Roll Up Eg. From computer to laptop, from £ to £  Slice: Remove a dimension by setting a value for it Eg. location/product where time is Q1,2007  Dice: Restrict cube by setting values for multiple dimensions Eg. Q1,Q2 / North American cities / 3 products sub cube  Pivot: Rotate the cube (mostly for visualisation) Data Cube Operations Data Warehousing COMP207: Data Mining

 Star Schema: Single fact table in the middle, with connected set of dimension tables (Hence a star)‏  Snowflake Schema: Some of the dimension tables further refined into smaller dimension tables (Hence looks like a snow flake)‏  Fact Constellation: Multiple fact tables can share dimension tables (Hence looks like a collection of star schemas. Also called Galaxy Schema)‏ Data Cube Schemas Data Warehousing COMP207: Data Mining

Star Schema Data Warehousing COMP207: Data Mining Sales Fact Table time_key item_key location_key units_sold Time Dimension time_key day day_of_week month quarter year Item Dimension item_key name brand type supplier_type Loc.n Dimension location_key street city state country continent Measure (value)‏

Snowflake Schema Data Warehousing COMP207: Data Mining Sales Fact Table time_key item_key location_key units_sold Time Dimension time_key day day_of_week month quarter year Item Dimension item_key name brand type supplier_key Loc Dimension location_key street city_key Measure (value)‏ City Dimension city_key city state country

Fact Constellation Data Warehousing COMP207: Data Mining Sales Fact Table time_key item_key location_key units_sold Time Dimension time_key day day_of_week month quarter year Item Dimension item_key name brand type supplier_key Loc Dimension location_key street city_key Measure (value)‏ City Dimension city_key city state country Shipping Table time_key item_key from_key units_shipped

ROLAP: Relational OLAP Uses relational DBMS to store and manage the warehouse data Optimised for non traditional access patterns Lots of research into RDBMS to make use of! MOLAP: Multidimensional OLAP Sparse array based storage engine Fast access to precomputed data HOLAP: Hybrid OLAP Mixture of both MOLAP and ROLAP OLAP Server Types Data Warehousing COMP207: Data Mining

Data Warehouse Architecture Data Warehousing COMP207: Data Mining Data Warehouse Extract Transform Load Refresh OLAP Engine Analysis Query Reports Data mining Monitor & Integrator Metadata Data Sources Front-End Tools Serve Data Marts Operational DBs Other sources Data Storage OLAP Server (also courtesy of Han & Kamber)‏

In order to compute OLAP queries efficiently, need to materialise some of the cuboids from the data. None: Very slow, as need to compute entire cube at run time Full: Very fast, but requires a LOT of storage space and time to compute all possible cuboids Partial: But which ones to materialise? Called an 'iceberg cube', as only partially materialised and the rest is "below water". Many cells in a cuboid will be empty, only materialise sections that contain more values than a minimum threshold. Materialisation Data Warehousing COMP207: Data Mining

and subsequent links Further Reading Data Warehousing COMP207: Data Mining