Building a Data Warehouse: Understanding Why & How

Overview Slides 1-3: Benefits Slides 4-5: SDLC Slides 6-9: Reasons why Slides 10-12: Environment Slides 13-16: Building Slides 17-20: Architecture Slides 21-22: Iterative development Slides 23-25: Meta data Slides 26-30: Staffing and requirements

Why Companies Build Automated Data Warehouses Improved access to integrated data Consistent reporting structure Speed of development Improved productivity Better data integrity and quality Reduction in maintenance cost $1,872.95 $1,472.95 ???

Development Cycle With an Automated Tool 12 weeks for full production use 6-8 weeks for first production-level implementation 3 weeks to learn automated data warehouse development process weeks 1 2 3 4 5 6 7 8 9 10 11 12

Number of Years of Data in a Typical Data Warehouse

Customers Receive Operational and Strategic Benefits Operational goals Improve profitability Increase sales Leverage procurement activity Maintain competitive advantage Strategic business goals Identify new market opportunities Strengthen customer relationships Expand market share Manage risk over time

The New Business Paradigm Integrating data to produce useful business information is a major challenge in business today! Operational Processing Informational Processing Legacy Environment Islands of Data Consolidated Information with a Historical Perspective

The Business Problem: Mixed Operational Environments Tuned for transactions, not decision support Data spans short time periods No standards for data definition and naming Queries never get same answer twice Dynamic data No “what ifs” possible No summarized data No integrated data

Typical Customer Environments Data stored in many forms Relational databases Hierarchical databases Flat files Heterogeneous mainframe and UNIX-based environments DEC (VMS, UNIX) HP IBM (MVS, UNIX) Sun Tandem Other UNIX environments

Transforming Data Into a Business Intelligence End User Data Warehouse Operational Systems Informational Processing Transform Operational Data to Data Warehouse Data: Extract Integrate Summarize Filter Convert Set default values Restructure Reformat Establish time variance Create consistency

Comparing the Environments Business Decisions Operational Decisions Data Warehouse Operational Systems Operational Systems No summary data No drill down No historical data Not integrated applications Rich supply of summary data Structure for drill down analysis Historical data for trend analysis Integrated data for corporate analysis

Building and Using the Data Warehouse OPERATIONAL PROCESSING INFORMATION PROCESSING Individual Transactions Consolidated Analysis IBM Ascential SAP Oracle

Requirements to Build a Data Warehouse Information architecture to understand the movement of data Extraction of data from legacy systems, operational applications and external sources Transformation of data to integrate, condense and summarize into a historical format Documentation of the development process (meta data); to understand sources of data, transformations and changes over time Ongoing maintenance to capture changes to source data and perform updates for iterative processing

Building the Data Warehouse 1 CREATE DATA WAREHOUSE DATA MODEL Generic Data Models™ Consulting 2 DEFINE SYSTEM OF RECORD Consulting 3 DESIGN DATA WAREHOUSE Consulting Development Methodology Methodology Readiness Assessment Database Design Project Management DataStage Developer 4a CAPTURE META DATA DEFINITIONS MetaStage™ File Definitions Data Dictionaries CASE Tools Business Descriptions 5 CREATE TRANSFORMATION PROGRAMS Extract Filter Integrate Condense Convert Derive Data Create Time Variance Generate Code 4b CAPTURE LOG CHANGES

Building the Data Warehouse 5 CREATE TRANSFORMATION PROGRAMS 6 EXTRACT, INTEGRATE & CONSOLIDATE SOURCE FILES DB Sources Extract Filter Integrate Condense Convert Derive Data Create Time Variance Generate Code 7 POPULATE & MAINTAIN DATA WAREHOUSE DB Targets DataStage 8 CREATE INFORMATION DIRECTORY & POPULATE ACCESS TOOLS MetaStage Directory™

Distributed Data Warehouse Solution DataStage MetaStage Iterations Methodology

A New Kind of Information Architecture An architected approach to information management and delivery Improves data integrity, performance, manageability, and access Individually Structured Departmentally Structured Data Warehouse m/d m/d m/d m/d m/d Organizationally Structured m/d m/d m/d Acquisition, Transformation & Integration Programs Archived Detail Operational Systems (System of Record) m/d = meta data

Architecture Affects Data Movement, System Impact and Development Effort Virtual Data Warehouse Conversion Technology Our Architected Solution

Structured Information Architecture EXTERNAL DATA DATA ACCESS & MULTIDIMENSIONAL ANALYSIS DATA WAREHOUSE OPERATIONAL DATA INFORMATION DIRECTORY Target Modules Transfer Technical Meta Data Source Modules DEVELOPER WORKSTATION

More Rapid Development Process DEFINE & CAPTURE PHYSICAL META DATA (CASE, Dictionaries, Catalogs, COBOL FDs) Transfer Technical Meta Data IMPORT BUSINESS META DATA (CASE Tools, Repositories, Machine Readable Files) SELECT TRANSFORMATIONS (Mapping, Conversion, Selection & Summarization) CREATE INFORMATION DIRECTORY & POPULATE ACCESS TOOLS CAPTURE LOG CHANGES UPDATE WAREHOUSE Developer Workstations

Iterative Processing All domestic raw goods January All domestic raw goods Foreign raw goods Domestic wip February All domestic raw goods * Foreign raw goods * Domestic wip Foreign wip Large customers March All domestic raw goods Foreign raw goods * Domestic wip * Foreign wip * Large customers Domestic finished goods Foreign customers April All domestic raw goods * Foreign raw goods Domestic wip * Foreign wip Large customers * Domestic finished goods * Foreign customers Foreign finished goods New prospects/customers May * being reworked Successful data warehouses are built in small, fast, iterative development efforts that produce measurable results.

Meta Data Exists Throughout the Structure of the Data Warehouse Highly summarized META DATA appl Lightly summarized appl Current detail appl integration/ transformation 5-10 years Older detail appl A data warehouse contains: Integrated data Subject oriented Historical data with time variance Both detailed and summary data

Turning Meta Data Into Information Public Library Corporation Where’s your card catalog for your corporate information?

Meta Data Answers Questions for Users of the Data Warehouse ?? How do I find the data I need? What is the original source of the data? How was this summarization created? What queries are available to access the data? How have business definitions and terms changed? How do product lines vary across organizations? What business assumptions have been made?

Be Ready to Adopt the Rapidly Advancing Data Warehouse Tool Sets Monitoring Security Design automation Transport automation Log tape as a source Meta Data Transformation Extraction, multiple DBMS Platform, DBMS Let’s build a data warehouse That’s a good idea

Skills and Experience To Build a Data Warehouse Can build on vision instead of hard requirements Can perform data warehouse information modeling Can implement a leveled information architecture Can manage a parallel, iterative, time-boxed project Can use/manage data warehouse specific technologies Can tightly coordinate a broad spectrum of resources Can set and continuously manage realistic expectations Can manage huge amounts of data Has “been there, done that” Uncommon Specialized Skill Set

Staff Requirements Project management Warehouse modeling Database design Data administration Component implementation Data access and analysis Data stewardship 5-7 people involved in a typical project 3-4 people using automated tool 1-2 people for ongoing maintenance

Be Ready for the Technological Impact The data warehouse introduces new technologies and taxes old ones Information modeling A new type of information architecture Interdependence with legacy systems Incredible growth in data volume Rapidly advancing data warehousing tool sets

Ready for Information Modeling? DATA WAREHOUSE TRADITIONAL DATABASE Integrated Data Historical Data Organized by Subject Non-Volatile Data Redundant Data Descriptive Data Summarized Data Meta Data Application-specific Data Current Data Organized for Performance Updated Data Normalized Data Encoded Data Raw Data (Just Data)

Be Ready to Manage Incredible Data Volume Growth Extend history Expand subject areas Cultivate data marts for departmental use Increase data volume due to business growth