1. Building a Data Warehouse: Understanding Why & How

2. Overview Slides 1-3: Benefits Slides 4-5: SDLC Slides 6-9: Reasons why
Slides 10-12: Environment
Slides 13-16: Building
Slides 17-20: Architecture
Slides 21-22: Iterative development
Slides 23-25: Meta data
Slides 26-30: Staffing and requirements

3. Why Companies Build Automated Data Warehouses
Improved access to integrated data
Consistent reporting structure
Speed of development
Improved productivity
Better data integrity and quality
Reduction in maintenance cost
$1,872.95
$1,472.95
???

4. Development Cycle With an Automated Tool
12 weeks for full production use
6-8 weeks for first production-level implementation
3 weeks to learn automated data
warehouse development process
weeks 1 2 3 4 5 6 7 8 9 10 11 12

5. Number of Years of Data in a Typical Data Warehouse

6. Customers Receive Operational and Strategic Benefits
Operational goals
Improve profitability
Increase sales
Leverage procurement activity
Maintain competitive advantage
Strategic business goals
Identify new market opportunities
Strengthen customer relationships
Expand market share
Manage risk over time

7. The New Business Paradigm
Integrating data to produce useful business information is a major challenge in business today!
Operational
Processing
Informational
Processing
Legacy Environment
Islands of Data
Consolidated Information
with a Historical Perspective

8. The Business Problem: Mixed Operational Environments
Tuned for transactions, not decision support
Data spans short time periods
No standards for data definition and naming
Queries never get same answer twice
Dynamic data
No “what ifs” possible
No summarized data
No integrated data

9. Typical Customer Environments
Data stored in many forms
Relational databases
Hierarchical databases
Flat files
Heterogeneous mainframe and UNIX-based environments
DEC (VMS, UNIX) HP
IBM (MVS, UNIX)
Sun
Tandem
Other UNIX environments

10. Transforming Data Into a Business Intelligence
End User
Data Warehouse
Operational Systems
Informational Processing
Transform Operational Data to Data Warehouse Data:
Extract
Integrate
Summarize
Filter
Convert
Set default values
Restructure
Reformat
Establish time variance
Create consistency

11. Comparing the Environments
Business
Decisions
Operational
Decisions
Data
Warehouse
Operational
Systems
Operational
Systems
No summary data
No drill down
No historical data
Not integrated applications
Rich supply of summary data
Structure for drill down analysis
Historical data for trend analysis
Integrated data for corporate analysis

12. Building and Using the Data Warehouse
OPERATIONAL PROCESSING
INFORMATION PROCESSING
Individual
Transactions
Consolidated
Analysis
IBM
Ascential
SAP
Oracle

13. Requirements to Build a Data Warehouse
Information architecture to understand the movement of data
Extraction of data from legacy systems, operational applications and external sources
Transformation of data to integrate, condense and summarize into a historical format
Documentation of the development process (meta data); to understand sources of data, transformations and changes over time
Ongoing maintenance to capture changes to source data and perform updates for iterative processing

14. Building the Data Warehouse1
CREATE DATA WAREHOUSE
DATA MODEL
Generic Data Models™
Consulting 2
DEFINE SYSTEM
OF RECORD
Consulting 3
DESIGN DATA WAREHOUSE
Consulting
Development Methodology
Methodology
Readiness Assessment
Database Design
Project Management
DataStage Developer 4a
CAPTURE META DATA
DEFINITIONS
MetaStage™
File Definitions
Data Dictionaries
CASE Tools
Business Descriptions 5
CREATE TRANSFORMATION
PROGRAMS
Extract
Filter
Integrate
Condense
Convert
Derive Data
Create Time Variance
Generate Code 4b
CAPTURE LOG CHANGES

15. Building the Data Warehouse5
CREATE TRANSFORMATION
PROGRAMS 6
EXTRACT, INTEGRATE &
CONSOLIDATE SOURCE FILES
DB Sources
Extract
Filter
Integrate
Condense
Convert
Derive Data
Create Time Variance
Generate Code 7
POPULATE & MAINTAIN
DATA WAREHOUSE
DB Targets
DataStage 8
CREATE INFORMATION
DIRECTORY & POPULATE
ACCESS TOOLS
MetaStage Directory™

16. Distributed Data Warehouse Solution
DataStage
MetaStage
Iterations Methodology

17. A New Kind of Information Architecture
An architected approach to information management and delivery
Improves data integrity, performance, manageability, and access
Individually
Structured
Departmentally
Structured
Data Warehouse
m/d
m/d
m/d
m/d
m/d
Organizationally Structured
m/d
m/d
m/d
Acquisition,
Transformation &
Integration
Programs
Archived Detail
Operational Systems
(System of Record)
m/d = meta data

18. Architecture Affects Data Movement, System Impact and Development Effort
Virtual Data Warehouse
Conversion
Technology
Our Architected Solution

19. Structured Information Architecture
EXTERNAL DATA
DATA ACCESS &
MULTIDIMENSIONAL
ANALYSIS
DATA WAREHOUSE
OPERATIONAL DATA
INFORMATION
DIRECTORY
Target Modules
Transfer Technical
Meta Data
Source
Modules
DEVELOPER
WORKSTATION

20. More Rapid Development Process
DEFINE & CAPTURE
PHYSICAL META DATA
(CASE, Dictionaries,
Catalogs, COBOL FDs)
Transfer Technical
Meta Data
IMPORT
BUSINESS META DATA
(CASE Tools, Repositories,
Machine Readable Files)
SELECT
TRANSFORMATIONS
(Mapping, Conversion, Selection & Summarization)
CREATE INFORMATION
DIRECTORY & POPULATE
ACCESS TOOLS
CAPTURE
LOG CHANGES
UPDATE
WAREHOUSE
Developer
Workstations

21. Iterative Processing
All domestic raw goods
January
All domestic raw goods
Foreign raw goods
Domestic wip
February
All domestic raw goods *
Foreign raw goods *
Domestic wip
Foreign wip
Large customers
March
All domestic raw goods
Foreign raw goods *
Domestic wip *
Foreign wip *
Large customers
Domestic finished goods
Foreign customers
April
All domestic raw goods *
Foreign raw goods
Domestic wip *
Foreign wip
Large customers *
Domestic finished goods *
Foreign customers
Foreign finished goods
New prospects/customers
May
* being reworked
Successful data warehouses are built in small, fast, iterative development efforts that produce measurable results.

22. Meta Data Exists Throughout the Structure of the Data Warehouse
Highly
summarized
META DATA
appl
Lightly
summarized
appl
Current
detail
appl
integration/
transformation
5-10 years
Older
detail
appl
A data warehouse contains:
Integrated data
Subject oriented
Historical data with time variance
Both detailed and summary data

23. Turning Meta Data Into Information
Public
Library
Corporation
Where’s your card catalog for your corporate information?

24. Meta Data Answers Questions for Users of the Data Warehouse??
How do I find the data I need?
What is the original source of the data?
How was this summarization created?
What queries are available to access the data?
How have business definitions and terms changed?
How do product lines vary across organizations?
What business assumptions have been made?

25. Be Ready to Adopt the Rapidly Advancing Data Warehouse Tool Sets
Monitoring
Security
Design automation
Transport automation
Log tape as a source
Meta Data
Transformation
Extraction, multiple DBMS
Platform, DBMS
Let’s build a data warehouse
That’s a good idea

26. Skills and Experience To Build a Data Warehouse
Can build on vision instead of hard requirements
Can perform data warehouse information modeling
Can implement a leveled information architecture
Can manage a parallel, iterative, time-boxed project
Can use/manage data warehouse specific technologies
Can tightly coordinate a broad spectrum of resources
Can set and continuously manage realistic expectations
Can manage huge amounts of data
Has “been there, done that”
Uncommon Specialized
Skill Set

27. Staff Requirements Project management Warehouse modeling
Database design
Data administration
Component implementation
Data access and analysis
Data stewardship
5-7 people involved in a typical project
3-4 people using automated tool
1-2 people for ongoing maintenance

28. Be Ready for the Technological Impact
The data warehouse introduces new technologies and taxes old ones
Information modeling
A new type of information architecture
Interdependence with legacy systems
Incredible growth in data volume
Rapidly advancing data warehousing tool sets

29. Ready for Information Modeling?
DATA WAREHOUSE
TRADITIONAL DATABASE
Integrated Data
Historical Data
Organized by Subject
Non-Volatile Data
Redundant Data
Descriptive Data
Summarized Data
Meta Data
Application-specific Data
Current Data
Organized for Performance
Updated Data
Normalized Data
Encoded Data
Raw Data
(Just Data)

30. Be Ready to Manage Incredible Data Volume Growth
Extend history
Expand subject areas
Cultivate data marts for departmental use
Increase data volume due to business growth