Data Warehouse Design Enrico Franconi CS 636

Implementing a Warehouse Monitoring: Sending data from sources Integrating: Loading, cleansing,... Processing: Query processing, indexing, ... Managing: Metadata, Design, ... CS 336

Monitoring Source Types: relational, flat file, IMS, VSAM, IDMS, WWW, news-wire, … How to get data out? Replication tool Dump file Create report ODBC or third-party “wrappers” CS 336

Monitoring Techniques Periodic snapshots Database triggers Log shipping Data shipping (replication service) Transaction shipping Polling (queries to source) Screen scraping Application level monitoring CS 336

Monitoring Issues Frequency Data transformation Standards (e.g., ODBC) periodic: daily, weekly, … triggered: on “big” change, lots of changes, ... Data transformation convert data to uniform format remove & add fields (e.g., add date to get history) Standards (e.g., ODBC) Gateways CS 336

Wrapper Converts data and queries from one data model to another Queries Data Model B Data Model A Data Extends query capabilities for sources with limited capabilities Wrapper Source Queries CS 336

Wrapper Generation Solution 1: Hard code for each source Solution 2: Automatic wrapper generation Wrapper Generator Definition Wrapper CS 336

Integration Data Cleaning Data Loading Derived Data CS 336 Client Warehouse Source Query & Analysis Integration Metadata CS 336

Data Integration Receive data (changes) from multiple wrappers/monitors and integrate into warehouse Rule-based Actions Resolve inconsistencies Eliminate duplicates Integrate into warehouse (may not be empty) Summarize data Fetch more data from sources (wh updates) etc. CS 336

Data Cleaning Find (& remove) duplicate tuples e.g., Jane Doe vs. Jane Q. Doe Detect inconsistent, wrong data Attribute values that don’t match Patch missing, unreadable data Insert default values Notify sources of errors found CS 336

Data Cleaning Migration (e.g., yen to dollars) Scrubbing: use domain-specific knowledge (e.g., social security numbers) Fusion (e.g., mail list, customer merging) billing DB service DB customer1(Joe) customer2(Joe) merged_customer(Joe) CS 336

Loading Data in the Warehouse Incremental vs. refresh Off-line vs. on-line Frequency of loading At night, 1x a week/month, continuously Parallel/Partitioned load CS 336

Warehouse Maintenance Warehouse data  materialized view Initial loading View maintenance Derived Warehouse Data indexes aggregates materialized views CS 336

Materialized Views Define new warehouse relations using SQL expressions does not exist at any source CS 336

Differs from Conventional View Maintenance... Warehouses may be highly aggregated and summarized Warehouse views may be over history of base data Process large batch updates Schema may evolve CS 336

Differs from Conventional View Maintenance... Base data doesn’t participate in view maintenance Simply reports changes Loosely coupled Absence of locking, global transactions May not be queriable CS 336

Warehouse Maintenance Anomalies Materialized view maintenance in loosely coupled, non-transactional environment Simple example Data Warehouse Sold (item,clerk,age) Sold = Sale Emp Integrator Sales Comp. Sale(item,clerk) Emp(clerk,age) CS 336

Warehouse Maintenance Anomalies Data Warehouse Sold (item,clerk,age) Integrator Sales Comp. Sale(item,clerk) Emp(clerk,age) 1. Insert into Emp(Mary,25), notify integrator 2. Insert into Sale (Computer,Mary), notify integrator 3. (1)  integrator adds Sale (Mary,25) 4. (2)  integrator adds (Computer,Mary) Emp 5. View incorrect (duplicate tuple) CS 336

Maintenance Anomaly - Solutions Incremental update algorithms (ECA, Strobe, etc.) Research issues: Self-maintainable views What views are self-maintainable Store auxiliary views so original + auxiliary views are self-maintainable CS 336

Self-Maintainability: Examples Sold(item,clerk,age) = Sale(item,clerk) Emp(clerk,age) Inserts into Emp If Emp.clerk is key and Sale.clerk is foreign key (with ref. int.) then no effect Inserts into Sale Maintain auxiliary view: Emp-clerk,age(Sold) Deletes from Emp Delete from Sold based on clerk CS 336

Self-Maintainability: Examples Deletes from Sale Delete from Sold based on {item,clerk} Unless age at time of sale is relevant Auxiliary views for self-maintainability Must themselves be self-maintainable One solution: all source data But want minimal set CS 336

Partial Self-Maintainability Avoid (but don’t prohibit) going to sources Sold=Sale(item,clerk) Emp(clerk,age) Inserts into Sale Check if clerk already in Sold, go to source if not Or replicate all clerks over age 30 Or ... CS 336

Warehouse Specification (ideally) View Definitions Warehouse Configuration Module Warehouse Integration rules Change Detection Requirements Integrator Metadata Extractor/ Monitor Extractor/ Monitor Extractor/ Monitor ... CS 336

Processing ROLAP servers vs. MOLAP servers Index Structures What to Materialize? Algorithms Client Warehouse Source Query & Analysis Integration Metadata CS 336

ROLAP Server Relational OLAP Server tools ROLAP utilities server Special indices, tuning; Schema is “denormalized” relational DBMS CS 336

MOLAP Server Multi-Dimensional OLAP Server M.D. tools utilities Product City Date 1 2 3 4 milk soda eggs soap A B Sales M.D. tools utilities multi-dimensional server could also sit on relational DBMS CS 336

Index Structures (sketch) Traditional Access Methods B-trees, hash tables, R-trees, grids, … Popular in Warehouses inverted lists bit map indexes join indexes text indexes CS 336

What to Materialize? Store in warehouse results useful for common queries Example: total sales day 2 . . . day 1 129 materialize CS 336

Materialization Factors Type/frequency of queries Query response time Storage cost Update cost CS 336

Cube Aggregates Lattice 129 all city product date city, product city, date product, date use greedy algorithm to decide what to materialize day 2 day 1 city, product, date CS 336

Dimension Hierarchies all state city CS 336

Dimension Hierarchies all city product date city, product city, date product, date state city, product, date state, date state, product state, product, date not all arcs shown... CS 336

Interesting Hierarchy all years weeks quarters conceptual dimension table months days CS 336

Managing Metadata Warehouse Design Tools CS 336 Client Source Query & Analysis Integration Metadata CS 336

Metadata Administrative definition of sources, tools, ... schemas, dimension hierarchies, … rules for extraction, cleaning, … refresh, purging policies user profiles, access control, ... CS 336

Metadata Business Operational business terms & definition data ownership, charging Operational data lineage data currency (e.g., active, archived, purged) use stats, error reports, audit trails CS 336

Design Summary What data is needed? Where does it come from? How to clean data? How to represent in warehouse (schema)? What to summarize? What to materialize? What to index? CS 336

Tools Development Planning & Analysis Warehouse Management design & edit: schemas, views, scripts, rules, queries, reports Planning & Analysis what-if scenarios (schema changes, refresh rates), capacity planning Warehouse Management performance monitoring, usage patterns, exception reporting System & Network Management measure traffic (sources, warehouse, clients) Workflow Management “reliable scripts” for cleaning & analyzing data CS 336

Current State of Industry Extraction and integration done off-line Usually in large, time-consuming, batches Everything copied at warehouse Not selective about what is stored Query benefit vs storage & update cost Query optimization aimed at OLTP High throughput instead of fast response Process whole query before displaying anything CS 336

State of Commercial Practice ... Connectivity to sources Apertus Information Builders Informix Enterprise Gateway Oracle Open Connect CA-Ingres gateway MS ODBC Platinum InfoHub Data extract, clean, transform, refresh CA-Ingres Replicator ETI-Extract IBM Data Joiner, Data Propagator Prism Warehouse manager SAS Access Sybase Replication Server Trinzic InfoPump CS 336

… State of Commercial Practice ... Multidimensional Database Engines Arbor Essbase Oracle RIR Express Comshare Commader SAS System Warehouse Data Servers CA-Ingres Oracle 8 RedBrick Sybase IQ Informix Dynamic Server IBM DB2 ROLAP Servers HP Intelligent Warehouse Informix Metacube MicroStrategy DSS Server Information Advantage Asxys CS 336

… State of Commercial Practice Multidimensional Analysis Kenan Systems Acumate Microsoft Excel Arbor Essbase Analysis server Cognos PowerPlay IQ Software IQ/Vision Lotus 123 SAS OLAP++ Business Objects Query/Reporting Environments IBM DataGuide SAS Access CA Visual Express Platinum Forest&Trees Informix ViewPoint Lots and lots of consulting!! CS 336

Future Directions Better performance Larger warehouses Easier to use What are companies & research labs working on? CS 336

Research (1) Incremental Maintenance Data Consistency Data Expiration Recovery Data Quality Error Handling (Back Flush) CS 336

Research (2) Rapid Monitor Construction Temporal Warehouses Materialization & Index Selection Data Fusion Data Mining Integration of Text & Relational Data Conceptual Modelling CS 336

Conclusions Massive amounts of data and complexity of queries will push limits of current warehouses Need better systems: easier to use provide quality information CS 336