THE ARCHITECTURAL COMPONENTS

CHAPTER OBJECTIVES Understand data warehouse architecture Learn about the architectural components Review the distinguishing characteristics of data warehouse architecture Examine how the architectural framework supports the flow of data Comprehend what technical architecture means Study the functions and services of the architectural components 1/16/2019 Data Warehouse

Architecture: Definitions The structure that brings all the components of a data warehouse together is known as the architecture. 1/16/2019 Data Warehouse

For example Take the case of the architecture of a school building. The architecture of the building is not just the visual style. It includes the various classrooms, offices, library, corridors, gymnasiums, doors, windows, roof, and a large number of other such components. When all of thesec ompo-nents are brought and placed together, the structure that ties all of the components Toge-ther is the architecture of the school building. 1/16/2019 Data Warehouse

Architecture in Three Major Areas As you already know, the three major areas in the data warehouse are: Data acquisition Data storage Information delivery 1/16/2019 Data Warehouse

Architectural components in the three major areas. 1/16/2019 Data Warehouse

DISTINGUISHING CHARACTERISTICS The architecture has distinguishing characteristics worth considering in detail. Different Objectives and Scope Data Content Complex Analysis and Quick Response Flexible and Dynamic Metadata-driven 1/16/2019 Data Warehouse

Different Objectives and Scope There are several sets of factors to consider. you must consider the number and extent of the data sources. How many legacy systems are you going to extract the data from? What are the external sources? Are you planning to include departmental files, spreadsheets, and private databases? What about including the archived data? 1/16/2019 Data Warehouse

Data Content The “read-only” data in the data warehouse sits in the middle as the primary component in the architecture. In an operational system, although the database is important, this importance does not measure up to that of a data warehouse data repository. 1/16/2019 Data Warehouse

Complex Analysis and Quick Response Your data warehouse architecture must support complex analysis of the strategic information by the users. Information retrieval processes in an operational system dwindle in complexity when compared to the use of information from a data warehouse. 1/16/2019 Data Warehouse

Flexible and Dynamic You have to make sure your data warehouse architecture is flexible enough to accommodate additional requirements as and when they surface. 1/16/2019 Data Warehouse

Metadata-driven Metadata surrounds the entire movement as the data moves from the source systems to the end-users as useful, strategic information. 1/16/2019 Data Warehouse

ARCHITECTURAL FRAMEWORK In this section, we grouped the architectural components as : building blocks in the three distinct areas of data acquisition, data storage, and information delivery. 1/16/2019 Data Warehouse

Architecture Supporting Flow of Data 1/16/2019 Data Warehouse

Manajemen dan Kontrol Module This component has two major functions: to constantly monitor all the ongoing operations to step in and recover from problems when things go wrong. 1/16/2019 Data Warehouse

Technical Architecture The technical architecture of a data warehouse is, therefore, the complete set of functions and services provided within its components. 1/16/2019 Data Warehouse

Data Acquisition This area covers the entire process of extracting data from the data sources, moving all the extracted data to the staging area, and preparing the data for loading into the data warehouse repository. 1/16/2019 Data Warehouse

Data Flow Flow. In the data acquisition area, the data flow begins at the data sources and pauses at the staging area. After transformation and integration, the data is ready for loading into the data warehouse repository. Data Sources. For the majority of data warehouses, the primary data source consists of the enterprise’s operational systems. Many of the operational systems at several enterprises are still legacy systems. 1/16/2019 Data Warehouse

Intermediary Data Stores Intermediary Data Stores. As data gets extracted from the data sources, it moves through temporary files. Staging Area. This is the place where all the extracted data is put together and prepared for loading into the data warehouse. 1/16/2019 Data Warehouse

Functions and Services List of Functions and Services : Data Extraction Data Transformation Data Staging 1/16/2019 Data Warehouse

List of Functions and Services : Data Extraction Select data sources and determine the types of filters to be applied to individual sources Generate automatic extract files from operational systems using replication and other techniques Create intermediary files to store selected data to be merged later Transport extracted files from multiple platforms Provide automated job control services for creating extract files Reformat input from outside sources Reformat input from departmental data files, databases, and spreadsheets Generate common application code for data extraction Resolve inconsistencies for common data elements from multiple sources 1/16/2019 Data Warehouse

List of Functions and Services : Data Transformation Map input data to data for data warehouse repository Clean data, deduplicate, and merge/purge Denormalize extracted data structures as required by the dimensional model of the data warehouse Convert data types Calculate and derive attribute values Check for referential integrity Aggregate data as needed Resolve missing values Consolidate and integrate data 1/16/2019 Data Warehouse

Data Staging Provide backup and recovery for staging area repositories List of Functions and Services : Data Staging Provide backup and recovery for staging area repositories Sort and merge files Create files as input to make changes to dimension tables If data staging storage is a relational database, create and populate database Preserve audit trail to relate each data item in the data warehouse to input source Resolve and create primary and foreign keys for load tables Consolidate datasets and create flat files for loading through DBMS utilities If staging area storage is a relational database, extract load files 1/16/2019 Data Warehouse

Data Storage This area covers the process of loading the data from the staging area into the data warehouse repository. All functions for transforming and integrating the data are completed in the data staging area. 1/16/2019 Data Warehouse

Data Flow Flow. For data storage, the data flow begins at the data staging area. The transformedand integrated data is moved from the staging area to the data warehouse repository. Data Groups. Prepared data waiting in the data staging area fall into two groups. Thefirst group is the set of files or tables containing data for a full refresh. The other group of data is the set of files or tables containing ongoing incremental loads. 1/16/2019 Data Warehouse

The Data Repository. Almost all of today’s data warehouse databases are relational databases. All the power, flexibility, and ease of use capabilities of the RDBMS become available for the processing of data. 1/16/2019 Data Warehouse

Functions and Services List of Functions and Services : Load data for full refreshes of data warehouse tables Perform incremental loads at regular prescribed intervals Support loading into multiple tables at the detailed and summarized levels Optimize the loading process Provide automated job control services for loading the data warehouse Provide backup and recovery for the data warehouse database Provide security Monitor and fine-tune the database Periodically archive data from the database according to preset conditions 1/16/2019 Data Warehouse

Information Delivery This area spans a broad spectrum of many different methods of making information available to users. 1/16/2019 Data Warehouse

Data Flow Flow. For information delivery, the data flow begins at the enterprise-wide data warehouse and the dependent data marts when the design is based on the top-down technique. When the design follows the bottom-up method, the data flow starts at the set of conformed data marts. Service Locations. In your information delivery component, you may provide query services from the user desktop, from an application server, or from the database itself. 1/16/2019 Data Warehouse

Data stores for standard reporting Data Stores. For information delivery, you may consider the following intermediary data stores: Proprietary temporary stores to hold results of individual queries and reports for repeated use Data stores for standard reporting Proprietary multidimensional databases 1/16/2019 Data Warehouse

Functions and Services List of Functions and Services : Provide security to control information access Monitor user access to improve service and for future enhancements Allow users to browse data warehouse content Simplify access by hiding internal complexities of data storage from users Automatically reformat queries for optimal execution Enable queries to be aware of aggregate tables for faster results Govern queries and control runaway queries Provide self-service report generation for users, consisting of a variety of flexible options to create, schedule, and run reports 1/16/2019 Data Warehouse

Functions and Services List of Functions and Services : Store result sets of queries and reports for future use Provide multiple levels of data granularity Provide event triggers to monitor data loading Make provision for the users to perform complex analysis through online analytical processing (OLAP) Enable data feeds to downstream, specialized decisions support systems such as EIS and data mining 1/16/2019 Data Warehouse

CHAPTER SUMMARY Architecture is the structure that brings all the components together. Data warehouse architecture consists of distinct components with the read-only data repository as the centerpiece. The architectural components support the functioning of the data warehouse in the three major areas of data acquisition, data storage, and information delivery. Data warehouse architecture is wide, complex, expansive, and has several distiguishing characteristics. The architectural framework enables the flow of data from the data sources at one end and the user’s desktop at the other. The technical architecture of a data warehouse is the complete set of functions and services provided within its components. It includes the procedures and rules needed to perform the functions and to provide the services. It encompasses the data stores needed for each component to provide the services. 1/16/2019 Data Warehouse