1. Advanced Database Systems: DBS CB, 2nd Edition
Data Warehouse, OLAP, Data Mining
Ch , Ch. 22

2. Outline What is a data warehouse? Why a data warehouse?
Data Warehouse Models & operations
Query & Analysis Tools
Implementing a data warehouse
Future directions

3. What is a Data Warehouse3 3 3

4. What is a Data Warehouse
Collection of diverse data
subject oriented
aimed at executive, decision maker
often a copy of operational data
with value-added data (e.g., summaries, history)
integrated
time-varying
non-volatile
Collection of tools
gathering data
cleansing, integrating, ...
querying, reporting, analysis
data mining
monitoring, administering warehouse
more

5. What is a Data Warehouse
Data Warehouse Architecture
Client
Warehouse
Source
Query & Analysis
Integration
Metadata

6. What is a Data Warehouse
Motivation Examples:
Forecasting
Comparing performance of units
Monitoring, detecting fraud
Visualization

7. Why a Data Warehouse 7 7 7

8. Why a Data Warehouse
Two Approaches to Extract Information in the Enterprise:
Query-Driven (Lazy)
Warehouse (Eager)

9. Why a Data Warehouse Query Driven Approach (Query Federation): Client
Wrapper
Mediator
Source

10. Why a Data Warehouse Warehouse Advantages
Queries not visible outside warehouse
Local processing at sources are unaffected
Can operate when sources unavailable
Can query data not stored in a DBMS
Extra information at warehouse
Modify, summarize (store aggregates)
Add historical information

11. Why a Data Warehouse Query-driven (Federation) Advantages
No need to copy data
less storage
no need to purchase data
More up-to-date data – real-time
Query needs can be unknown
Only query interface needed at sources
May be less draining on sources

12. What is a Data Warehouse
OLTP vs. OLAP
OLTP: On Line Transaction Processing
Describes processing at operational sites
OLAP: On Line Analytical Processing
Describes processing at warehouse

13. What is a Data Warehouse
OLTP vs. OLAP
OLTP
OLAP
Mostly updates
Many small transactions
Mb-Gb of data
Raw data
Clerical users
Up-to-date data
Consistency, recoverability critical
Mostly reads
Queries long, complex
GB-TB of data
Summarized, consolidated data
Decision-makers, analysts as users

14. What is a Data Warehouse
Data Marts
Smaller warehouses
Spans part of organization
e.g., marketing (customers, products, sales)
Do not require enterprise-wide consensus
but long term integration problems?

15. Data Warehouse Models & Operations15 15 15

16. Data Warehouse Models & Operations
Data Models
relations
Star & Snowflakes & Fact Constallation/Galaxy
cubes
Operators
slice & dice
roll-up, drill down
pivoting
other

17. Data Warehouse Models & Operations
Star Schema

18. Data Warehouse Models & Operations
Star Schema
Terms:
Fact table: PK is the aggregate of the dimensions’ PK - 2nd NF
Dimension tables: “by” condition - one view for how the data will be analyzed – 3rd NF
Measures: aggregates – sales/day, sales/quarter, etc.
Summary Tables: pre-summarized data to meet 90% of user queries. Use triggers/stored procedures to maintain summary tables current on insert/delete
sale
orderId
date
custId
prodId
storeId
qty
amt
customer
custId
name
address
city
product
prodId
name
price
store
storeId
city

19. Data Warehouse Models & Operations
Star Schema - Dimension Hierarchies
store
sType
city
region

20. Data Warehouse Models & Operations
Snowflakes schema, a dimension table have the hierarchy broken into set of tables – more complicated and slower queries
time_key
day
day_of_the_week
month
quarter
year
time
location_key
street
city_key
location
Sales Fact Table
item_key
branch_key
units_sold
dollars_sold
avg_sales
Measures
item_name
brand
type
supplier_key
item
branch_name
branch_type
branch
supplier_type
supplier
city
state_or_province
country

21. Data Warehouse Models & Operations
Fact Constellations schema, multiple fact tables that share the set of dimensional tables; viewed as a collection of stars
time_key
day
day_of_the_week
month
quarter
year
time
location_key
street
city
province_or_state
country
location
Sales Fact Table
item_key
branch_key
units_sold
dollars_sold
avg_sales
item_name
brand
type
supplier_type
item
branch_name
branch_type
branch
Shipping Fact Table
shipper_key
from_location
to_location
dollars_cost
units_shipped
shipper_name
shipper_type
shipper
Measures

22. Data Warehouse Models & Operations
product
prodId
name
price
customer
custId
address
city
store
storeId
sale
orderId
date
qty
amt
Fact table view:
Multi-dimensional cube:
Star Schema
dimensions = 2

23. Data Warehouse Models & Operations
3D Cube
Fact table view:
Multi-dimensional cube:
day 2
day 1
dimensions = 3

24. Data Warehouse Models & Operations
ROLAP vs. MOLAP
ROLAP: Relational On-Line Analytical Processing
MOLAP: Multi-Dimensional On-Line Analytical Processing
Aggregates
Add up amounts for day 1
In SQL: SELECT sum(amt) FROM SALE WHERE date = 1; 81

25. Data Warehouse Models & Operations
Aggregates (Contd.)
Add up amounts by day
In SQL: SELECT date, sum(amt) FROM SALE GROUP BY date;

26. Data Warehouse Models & Operations
Aggregates (Contd.)
Add up amounts by day, product
In SQL: SELECT date, sum(amt) FROM SALE GROUP BY date, prodId;
drill-down
rollup

27. Data Warehouse Models & Operations
Aggregates (Contd.)
Operators: sum, count, max, min, median, ave
“Having” clause
Using dimension hierarchy
average by region (within store)
maximum by month (within date)

28. Data Warehouse Models & Operations
Cube Aggregations
day 2
day 1
129
. . .
Example: computing sums
drill-down
rollup

29. Data Warehouse Models & Operations
Cube Operators
day 2
day 1
129
. . .
sale(c1,*,*)
sale(*,*,*)
sale(c2,p2,*)

30. Data Warehouse Models & Operations
Extended Cube
day 2
day 1 *
sale(*,p2,*)

31. Data Warehouse Models & Operations
Aggregation Using Hierarchies
day 2
day 1
customer
region
country
(customer c1 in Region A;
customers c2, c3 in Region B)

32. Data Warehouse Models & Operations
Pivoting
day 2
day 1
Multi-dimensional cube:
Fact table view:

33. Query & Analysis Tools 33 33 33

34. Query & Analysis Tools Query Building
Report Writers (comparisons, growth, graphs,…)
Spreadsheet Systems
Web Interfaces
Data Mining

35. Query & Analysis Tools Other Operations Time functions
e.g., time average
Computed Attributes
e.g., commission = sales * rate
Text Queries
e.g., find documents with words X AND B
e.g., rank documents by frequency of words X, Y, Z

36. Query & Analysis Tools Data Mining (i) Decision Tree: Decision Trees
Clustering
Association Rules
(i) Decision Tree:
Conducted survey to see what customers were interested in new model car
Want to select customers for advertising campaign
training
set

37. Query & Analysis Tools One possibility: age<30 Y N city=sf car=van
likely
unlikely
likely
unlikely

38. Query & Analysis Tools Another Possibility: car=taurus Y N city=sf
age<45 Y Y N N
likely
unlikely
likely
unlikely

39. Query & Analysis Tools Decision Tree Issues:
Decision tree cannot be “too deep” ☻
would not have statistically significant amounts of data for lower decisions
Need to select tree that most reliably predicts outcomes

40. Query & Analysis Tools
(ii) Clustering:
age
income
education

41. Query & Analysis Tools Another Example: Text Each document is a vector
e.g., < > contains words 1,4,5,...
Clusters contain “similar” documents
Useful for understanding, searching documents
international
news
sports
business

42. Query & Analysis Tools Clustering Issues:
Given desired number of clusters?
Finding “best” clusters
Are clusters semantically meaningful?
e.g., “yuppies’’ cluster?
Using clusters for disk storage

43. Query & Analysis Tools (iii) Association Rules Mining: products
bought
customer id
sales
records:
market-basket
data
Trend: Products p5, p8 often bought together
Trend: Customer 12 likes product p9

44. Query & Analysis Tools Association Rule Rule: {p1, p3, p8}
Support: number of baskets where these products appear
High-support set: support  threshold s
Problem: find all high support sets

45. Query & Analysis Tools Finding High-Support Pairs
Baskets(basket, item)
SELECT I.item, J.item, COUNT(I.basket) FROM Baskets I, Baskets J WHERE I.basket = J.basket AND I.item < J.item GROUP BY I.item, J.item HAVING COUNT(I.basket) >= s;
WHY?

46. Query & Analysis Tools
Example:
check if
count  s

47. Query & Analysis Tools Association Rule Issues:
Performance for size 2 rules
Performance for size k rules
even
bigger!
big

48. Implementing a Data Warehouse48 48 48

49. Implementing a Data Warehouse
Monitoring: Sending data from sources
Integrating: Loading, cleansing,...
Processing: Query processing, indexing, ...
Managing: Metadata, Design, ...

50. Implementing a Data Warehouse
Monitoring:
Source Types: relational, flat file, IMS, VSAM, IDMS, WWW, news-wire, …
Incremental vs. Refresh
new

51. Implementing a Data Warehouse
Monitoring Techniques:
Periodic snapshots
Database triggers
Log shipping
Data shipping (replication service)
Transaction shipping
Polling (queries to source)
Screen scraping
Application level monitoring
è Advantages & Disadvantages!!

52. Implementing a Data Warehouse
Monitoring Issues:
Frequency
periodic: daily, weekly, …
triggered: on “big” change, lots of changes, ...
Data transformation
convert data to uniform format
remove & add fields (e.g., add date to get history)
Standards (e.g., ODBC)
Gateways

53. Implementing a Data Warehouse
Integration
Data Cleaning
Data Loading
Derived Data
Client
Warehouse
Source
Query & Analysis
Integration
Metadata

54. Implementing a Data Warehouse
Data Cleaning:
Migration (e.g., yen ð dollars)
Scrubbing: use domain-specific knowledge (e.g., social security numbers)
Fusion (e.g., mail list, customer merging)
Auditing: discover rules & relationships (like data mining)
billing DB
service DB
customer1(Joe)
customer2(Joe)
merged_customer(Joe)

55. Implementing a Data Warehouse
Loading Data:
Incremental vs. refresh
Off-line vs. on-line
Frequency of loading
At night, 1x a week/month, continuously
Parallel/Partitioned load

56. Implementing a Data Warehouse
Derived Data:
Derived Warehouse Data
indexes
aggregates
materialized views (next slide)
When to update derived data?
Incremental vs. refresh

57. Implementing a Data Warehouse
Materialized Views:
Define new warehouse relations using SQL expressions
does not exist
at any source

58. Implementing a Data Warehouse
Processing
ROLAP servers vs. MOLAP servers
Index Structures
What to Materialize?
Algorithms
Client
Warehouse
Source
Query & Analysis
Integration
Metadata

59. Implementing a Data Warehouse
ROLAP Server: Relational OLAP Server
tools
utilities
ROLAP
server
Special indices, tuning;
Schema is “denormalized”
relational
DBMS

60. Implementing a Data Warehouse
MOLAP Server: Multi-Dimensional OLAP Server
Product
City
Date
milk
soda
eggs
soap A B
Sales
M.D. tools
utilities
multi-dimensional
server
could also
sit on
relational
DBMS

61. Implementing a Data Warehouse
Index Structures
Traditional Access Methods
B-trees, hash tables, R-trees, grids, …
Popular in Warehouses
inverted lists
bit map indexes
join indexes
text indexes

62. Implementing a Data Warehouse
Inverted Lists
. . .
age
index
inverted
lists
data
records

63. Implementing a Data Warehouse
Using Inverted Lists:
Query:
Get people with age = 20 and name = “fred”
List for age = 20: r4, r18, r34, r35
List for name = “fred”: r18, r52
Answer is intersection of “list for name” & “list for age”: r18

64. Implementing a Data Warehouse
Bit Map Index:
. . .
age
index
bit
maps
data
records

65. Implementing a Data Warehouse
Using Bit Map Index:
Query:
Get people with age = 20 and name = “fred”
List for age = 20:
List for name = “fred”:
Answer is intersection:
Good if domain cardinality small
Bit vectors can be compressed

66. Implementing a Data Warehouse
Join
“Combine” SALE, PRODUCT relations
In SQL: SELECT * FROM SALE, PRODUCT;

67. Implementing a Data Warehouse
Join Index
join index

68. Implementing a Data Warehouse
What to Materialize
Store in warehouse results useful for common queries
Example:
day 2
day 1
129
. . .
total sales
materialize

69. Implementing a Data Warehouse
Materialization Factors:
Type/frequency of queries
Query response time
Storage cost
Update cost

70. Implementing a Data Warehouse
Cube Aggregates Lattice:
129
all
city
product
date
city, product
city, date
product, date
use greedy
algorithm to
decide what
to materialize
day 2
day 1
city, product, date

71. Implementing a Data Warehouse
Dimension Hierarchies:
all
state
city

72. Implementing a Data Warehouse
Dimension Hierarchies:
all
product
city
date
city, product
city, date
product, date
state
city, product, date
state, date
state, product
state, product, date
not all arcs shown...

73. Implementing a Data Warehouse
Interesting Hierarchy:
all
years
weeks
quarters
conceptual
dimension table
months
days

74. Implementing a Data Warehouse
Algorithms:
Query Optimization
Parallel Processing
Data Mining

75. Implementing a Data Warehouse
Managing:
Metadata
Warehouse Design
Tools
Client
Warehouse
Source
Query & Analysis
Integration
Metadata

76. Implementing a Data Warehouse
Metadata:
Administrative
definition of sources, tools, ...
schemas, dimension hierarchies, …
rules for extraction, cleaning, …
refresh, purging policies
user profiles, access control, ...

77. Implementing a Data Warehouse
Metadata (Contd.):
Business
business terms & definition
data ownership, charging
Operational
data lineage
data currency (e.g., active, archived, purged)
use stats, error reports, audit trails

78. Implementing a Data Warehouse
Design:
What data is needed?
Where does it come from?
How to clean data?
How to represent in warehouse (schema)?
What to summarize?
What to materialize?
What to index?

79. Implementing a Data Warehouse
Tools:
Development
design & edit: schemas, views, scripts, rules, queries, reports
Planning & Analysis
what-if scenarios (schema changes, refresh rates), capacity planning
Warehouse Management
performance monitoring, usage patterns, exception reporting
System & Network Management
measure traffic (sources, warehouse, clients)
Workflow Management
“reliable scripts” for cleaning & analyzing data

80. Implementing a Data Warehouse
Current State of Industry:
Extraction and integration done off-line
Usually in large, time-consuming, batches
Everything copied at warehouse
Not selective about what is stored
Query benefit vs. storage & update cost
Query optimization aimed at OLTP
High throughput instead of fast response
Process whole query before displaying anything

81. Future Directions 81 81 81

82. Future Directions Better performance Larger warehouses Easier to use
What are companies & research labs working on?

83. Future Directions Research: Incremental Maintenance Data Consistency
Data Expiration
Recovery
Data Quality
Error Handling (Back Flush)

84. Future Directions Research: Rapid Monitor Construction
Temporal Warehouses
Materialization & Index Selection
Data Fusion
Data Mining
Integration of Text & Relational Data

85. END