Data warehouse

Overview Traditional database systems are tuned to many, small, simple queries. Some new applications use fewer, more time-consuming, complex queries. New architectures have been developed to handle complex “analytic” queries efficiently.

What is a Data Warehouse? A single, complete and consistent store of data obtained from a variety of different sources made available to end users in a what they can understand and use in a business context.

The Data Warehouse The most common form of data integration. Copy sources into a single DB (warehouse) and try to keep it up-to-date. Usual method: periodic reconstruction of the warehouse, perhaps overnight. Frequently essential for analytic queries.

Data Warehouse A data warehouse is a subject-oriented integrated time-varying non-volatile collection of data that is used primarily in organizational decision making.

Data Warehouse Architecture Relational Databases Legacy Data Purchased Data ERP Systems Analyze Query Data Warehouse Engine Optimized Loader Extraction Cleansing Metadata Repository

OLTP Most database operations involve On-Line Transaction Processing (OTLP). Short, simple, frequent queries and/or modifications, each involving a small number of tuples. Examples: Answering queries from a Web interface, sales at cash registers, selling airline tickets.

OLAP Of increasing importance are On-Line Application Processing (OLAP) queries. Few, but complex queries --- may run for hours. Queries do not depend on having an absolutely up-to-date database.

OLAP Examples Amazon analyzes purchases by its customers to come up with an individual screen with products of likely interest to the customer. Analysts at Wal-Mart look for items with increasing sales in some region.

Common Architecture Databases at store branches handle OLTP. Local store databases copied to a central warehouse overnight. Analysts use the warehouse for OLAP.

Approaches to Building Warehouses ROLAP = “relational OLAP”: Tune a relational DBMS to support star schemas. MOLAP = “multidimensional OLAP”: Use a specialized DBMS with a model such as the “data cube.”

OLTP vs. Data Warehouse OLTP systems are tuned for known transactions and workloads while workload is not known a priori in a data warehouse Special data organization, access methods and implementation methods are needed to support data warehouse queries (typically multidimensional queries) e.g., average amount spent on phone calls between 9AM-5PM in Pune during the month of December

OLTP vs Data Warehouse OLTP Warehouse (DSS) Application Oriented Used to run business Detailed data Current up to date Isolated Data Repetitive access Clerical User Warehouse (DSS) Subject Oriented Used to analyze business Summarized and refined Snapshot data Integrated Data Ad-hoc access Knowledge User (Manager)

OLTP vs Data Warehouse OLTP Data Warehouse Performance Sensitive Few Records accessed at a time (tens) Read/Update Access No data redundancy Database Size 100MB -100 GB Data Warehouse Performance relaxed Large volumes accessed at a time(millions) Mostly Read (Batch Update) Redundancy present Database Size 100 GB - few terabytes

OLTP vs Data Warehouse OLTP Data Warehouse Transaction throughput is the performance metric Thousands of users Managed in entirety Data Warehouse Query throughput is the performance metric Hundreds of users Managed by subsets