Data warehouse

Definition Data Warehouse A collection of corporate information, derived directly from operational systems and some external data sources. Its specific purpose is to support business decisions, not business operations. Key Concept : DWH getting the data out of the source systems – standardize it, cleanse it, store it in a common spot. BI is transforming this data into a format that is consumable by business people. Heavy lifting

The Purpose of Data Warehousing Realize the value of data Data / information is an asset Methods to realize the value, (Reporting, Analysis, etc.) Make better decisions Turn data into information Create competitive advantage Methods to support the decision making process, (EIS, DSS, etc.)

Data Warehouse Components Staging Area A preparatory repository where transaction data can be transformed for use in the data warehouse Data Mart Traditional dimensionally modeled set of dimension and fact tables Per Kimball, a data warehouse is the union of a set of data marts Operational Data Store (ODS) Modeled to support near real-time reporting needs.

Data Warehouse Functionality Relational Databases Legacy Data Purchased Data ERP Systems Analyze Query Data Warehouse Engine Optimized Loader Extraction Cleansing Metadata Repository Legacy data is historical data The working information of a staff member Working hours or time-off hours within the fiscal period, up to the current date Working Hours = Overtime, etc. Time-Off Hours = Vacation, Sick Leave, etc.

Evolution architecture of data warehouse Top-Down Architecture Bottom-Up Architecture Enterprise Data Mart Architecture Data Stage/Data Mart Architecture GO TO DIAGRAM GO TO DIAGRAM GO TO DIAGRAM DataStage database, tool A tool set for designing, developing, and runnin.g applications that populate one or more tables in a data warehouse GO TO DIAGRAM

Very Large Data Bases Warehouses are Very Large Databases Terabytes -- 10^12 bytes: Petabytes -- 10^15 bytes: Exabytes -- 10^18 bytes: Zettabytes -- 10^21 bytes: Zottabytes -- 10^24 bytes: Wal-Mart -- 24 Terabytes Geographic Information Systems National Medical Records Weather images Intelligence Agency Videos

Complexities of Creating a Data Warehouse Incomplete errors Missing Fields Records or Fields That, by Design, are not Being Recorded Incorrect errors Wrong Calculations, Aggregations Duplicate Records Wrong Information Entered into Source System Quick review

SUCCESS & FUTURE OF DATA WAREHOUSE The Data Warehouse has successfully supported the increased needs over the past many years. The need for growth continues however, as the desire for more integrated data increases. The Data Warehouse has software and tools in place to provide the functionality needed to support new enterprise Data Warehouse projects. The future capabilities of the Data Warehouse can be expanded to include other programs and agencies.

Data Warehouse Pitfalls You are going to spend much time extracting, cleaning, and loading data You are going to find problems with systems feeding the data warehouse You will find the need to store/validate data not being captured/validated by any existing system Large scale data warehousing can become an exercise in data homogenizing

Data Warehouse Pitfalls… The time it takes to load the warehouse will expand to the amount of the time in the available window... and then some You are building a HIGH maintenance system You will fail if you concentrate on resource optimization to the neglect of project, data, and customer management issues and an understanding of what adds value to the customer

Best Practices Complete requirements and design Prototyping is key to business understanding Utilizing proper aggregations and detailed data Training is an on-going process Build data integrity checks into your system. Quick Review

Thank You

Top-Down Architecture BACK TO ARCHITECTURE

Bottom-Up Architecture BACK TO ARCHITECTURE

Enterprise Data Mart Architecture BACK TO ARCHITECTURE

Data Stage/Data Mart Architecture BACK TO ARCHITECTURE