DATA WAREHOUSING

Introduction Modern organizations have huge amounts of data but are starving for information – facing information gap! Reasons for information gap:  Fragmented information systems, uncoordinated (sometimes inconsistent) databases /islands of information due to time and resources constraints  Most systems support operational /transaction processing rather than informational processing. Bridging the information gap are data warehouses!

Introduction (Contd.) A data warehouse bridges this information gap as  it consolidates and integrates information from internal and external sources and  arranges it in a meaningful format for making accurate business decisions The two noticeable pioneers in the DWH field are Ralph Kimball and Bill Inmon. The term Data Warehouse was coined by Bill Inmon in 1990

What is a Data Warehouse? Inmon defines a data warehouse as follows:  "a subject-oriented, integrated, time-variant and non-volatile collection of data in support of management's decision making process". Description of the above terms according to Inmon  Subject Oriented: Data that gives information about a particular subject instead of about a company's ongoing operations.  Integrated: Data that is gathered into the data warehouse from a variety of sources and merged into a coherent whole.

 Time-variant: All data in the data warehouse is identified with a particular time dimension. The time factor can be used to study trends and changes.  Non-volatile Data is stable in a data warehouse. More data is added but data is never removed. This enables management to gain a consistent picture of the business.

What is a Data Warehouse? This definition by Inmon remains reasonably accurate almost ten years later. However,  a single-subject data warehouse is typically referred to as a data mart, while data warehouses are generally enterprise in scope.  data warehouses can be volatile. Due to the large amount of storage required for a data warehouse, (multi-terabyte data warehouses are not uncommon), only a certain number of periods of history are kept in the warehouse. For instance, if three years of data are decided on and loaded into the warehouse, every month the oldest month will be "rolled off" the database, and the newest month added.

What is a Data Warehouse? Ralph Kimball provided a much simpler definition of a data warehouse.  a data warehouse is "a copy of transaction data specifically structured for query and analysis". This definition provides less insight and depth than Mr. Inmon's, but is no less accurate.

The Need for Data warehousing Two factors drive the need for data warehousing  Need of company-wide view of data  Need to separate operational* and informational* systems * See slide notes

Data warehousing? It is the process of creating, populating, and then querying a data warehouse Components of data warehousing are  Source Systems Identification  Data warehouse Design and Creation  Data Acquisition  Changed data capture  Data Cleansing  Data Aggregation  Business Intelligence Multidimensional Analysis Tools Query Tools Data Mining Tools Data Visualization Tools  Metadata Management

Components of Data warehousing Source System Identification  Appropriate data must be located to build a data warehouse.  This data comes from Current OLTP systems (providing day-to-day information) `Legacy systems (providing historical data for prior periods) Data Warehouse Design and Creation  The design is often an iterative process  Requires effort to understand database schema and a great deal of user interaction

Components of Data warehousing Data Acquisition  Involves moving company data from source systems into warehouse  Time consuming activity  Performed using ETL (Extract/Transform/ Load) Tools  Data acquisition is an ongoing, scheduled process. Warehouse is refreshed monthly Changed data capture  Periodic update of warehouse from transactional systems is complicated as it is difficult to identify which records in source have changed since last update.  Some technologies used in this area are Replication servers, Publish/Subscribe, Triggers and Stored procedures, and Database Log analysis.

Components of Data warehousing Data Cleansing  Typically performed in conjunction with data acquisition (part of “T” in “ETL”)  A complicated process that validates and if required corrects data before its loaded into warehouse  Example The entries for "Customer Name" may appear differently in various source systems for the same customer. one entered as "IBM", one as "I.B.M.", and one as "International Business Machines". A decision must be made as to which is correct, and then the data cleansing tool will change the others to match the rule. This process is also referred to as "data scrubbing" or "data quality assurance". It can be an extremely complex process, especially if some of the warehouse inputs are from older mainframe file systems (commonly referred to as "flat files" or "sequential files").

Components of Data warehousing Data Aggregation  If required, it is performed during “T” phase of “ETL”  Data warehouse can be designed to store data at detail level (each individual transaction), at some aggregate level (summary data) or a combination of both.  The advantage of summarized data is that typical queries against the warehouse run faster.  The disadvantage is that information, which may be needed to answer a query, is lost during aggregation Business Intelligence (BI)  Contains technologies such as Decision Support Systems (DSS), Executive Information Systems (EIS), On-Line Analytical Processing (OLAP), Relational OLAP (ROLAP), Multi- Dimensional OLAP (MOLAP), Hybrid OLAP (HOLAP, a combination of MOLAP and ROLAP), and more.

Components of Data warehousing  BI can be broken down into four broad fields Multidimensional Analysis Tools  Allow user to look at data from various different angles  Often use a multidimensional database called cube Query Tools  Allow user to issue SQL queries against warehouse and get results Data Mining Tools  Automatically search for patterns in data  Driven by complex statistical formulas Data Visualization Tools  Graphically represent data including 3D data pictures

Components of Data warehousing Metadata management  Throughout the entire process of identifying, acquiring, and querying the data, metadata management takes place  The datatype (e.g., string or integer) of the column is metadata. The name of the column is another. The actual value in the column for a particular row is not metadata - it is data  Metadata is stored in metadata repository  Metadata is useful in almost all components of data warehousing discussed earlier

Data Mining Definition:  Knowledge discovery using a sophisticated blend of techniques from traditional statistics, artificial intelligence and computer graphics Goals  To explain observed events or conditions, such as why sales of a product have increased in a particular area  To confirm hypothesis, such as, whether two-income families are more likely to buy family medical cover than single-income families  To analyze data for new or unexpected relationships, such as what spending patterns are likely to accompany credit card fraud.

Data Mining Techniques Case-based reasoning  Derives rules from real world case examples Rule discovery  Searches for patterns and correlations in large data sets Signal Processing  Identifies clusters of observations with similar characteristics Neural nets  Develops predictive models based on principles modeled after the human brain Fractals  Compresses large databases without losing information

Some Data Mining Applications Analysis of business trends  Identifying markets with above average or below average growth Target marketing  Identifying customers for promotional activity Usage Analysis  Identifying usage patterns of products and services Product Affinity  Identifying products that are purchased concurrently or characteristics of shoppers For further application types - refer to book page 437 table 11.5

Reading Assignment Read these concepts from Book  The ETL process  data warehouse Data mart  Independent data mart  dependent data mart  EDW Star Schema Snowflake Schema OLAP MOLAP ROLAP

Resources tawarehousing.html tawarehousing.html Modern Database Management, 6/e