Typically data is extracted from multiple sources Loading the Warehouse Typically data is extracted from multiple sources To update the warehouse periodically, we must receive the changes that have occurred and load these into the warehouse An initial load of the warehouse ETL Periodically update the warehouse Changes? ETL Refresh? March 2004 91.4904 Ron McFadyen

Data is obtained from source systems Transform ETL Extract Data is obtained from source systems Transform Data is cleansed and transformed Correcting values, detecting incorrect/impossible values, … Complex fields broken down, standardization of values, … Attribute types may vary in source and target Data may be aggregated Load New data is loaded into dimensions and fact tables Indexes rebuilt March 2004 91.4904 Ron McFadyen

Data capture techniques Synchronous Data is obtained as it is created in the source systems – in real time Asynchronous A delay is present between the time the data changes and the time it is captured in the warehouse Total Refresh A complete table in the warehouse is refreshed from its source Incremental Only changes are acquired and loaded into the warehouse March 2004 91.4904 Ron McFadyen

Data capture techniques Static capture Data is acquired by reading the database or files. Subsets may be acquired by filtering Application assisted The application(s) are modified to write changes out to a file/database Triggered capture Triggers in a DBMS are written to capture changes Replication A DBMS replication feature is used to manage changes Log capture The DBMS logging feature is utilized for capturing changes File comparison A prior copy and the current file are comparing to find changes March 2004 91.4904 Ron McFadyen

Data capture process source Extract Cleanse Repair Transform target errors Load March 2004 91.4904 Ron McFadyen

Star Schema Update Load Dim 1 Load Dim n Load Fact table March 2004 91.4904 Ron McFadyen

Dimension table surrogate key management Figure 16.4 on page 360 Going back in time (page 271) Late-arriving fact rows Late-arriving dimension rows From the source systems we receive data that we should have receive some time ago Technical issues having to do with Dimension records being twin-timestamped for contiguous non-overlapping time intervals, placing records in the right partition For dimensions, you also need to adjust the facts Subtle point about the ordering of surrogate keys March 2004 91.4904 Ron McFadyen

Point-in-time balances See pages 208+, SQL on page 209 The fact table is given on page 210 There is just one date-related attribute: transaction date key “the date key is a set of integers running from 1 to N with a meaningful, predictable sequence. We assign consecutive integers to the date surrogate key so that we can physically partition a large fact table based on the date.” “The date dimension is the only dimension whose surrogate keys have any embedded semi-intelligence” March 2004 91.4904 Ron McFadyen