Designing and Implementing an ETL Framework Mike Litchfield Assistant Vice President – IT Architect & Sr. Developer Income Research & Management mlitchfield@incomeresearch.com

Agenda Architecture Considerations Reference Architecture Why build an ETL framework ? Architecture Considerations Reference Architecture What features to include ? Walk through sql and ssis components Walk through web-based components

Why Build a Framework ? Reusability \ Efficiency \ Time-To-Market Definition Reusability \ Efficiency \ Time-To-Market Code Standardization \ Ease-Of-Support \ Minimized Troubleshooting Adherance to best practices Functional Flexibility and Scalability Accessability to Data Flow Status End User Notification

IRM Goals Reliability and Accuracy Implement many routines quickly, with limited developer resources Reliability and Accuracy Ability to introduce \ modify \ remove transformation rules Ability to maintain and apply logical business rules on data Support for scheduled and user-initiated package execution Visibility into ETL Status Error Notification and Ability to Debug Quickly End User Confirmation Support for logical dependencies between Sql Agent steps and SSIS packages Package instance concurrency handling Historical persistence of data, errors, overrides, thresholds

Architecture Considerations Organization of database servers and databases Sql Server 32-bit vs 64-bit Database access methodology How ssis packages will be invoked Checkpoint restart \ database transactions Compartmentalizing ssis functionality Linked Server Excel vs tables

What features to Include ? Ssis package templates – parent and child Multiple ways to invoke the package Package instance concurrency handling Run date \ Override date handling Tracking package progress Error handling and notification Variables definition Standardized flow Concurrency Check Registration Extract Transform \ re-process corrections Rule execution Final destination posting Historical archiving Notification

Sql Components Framework Tables Etl_pkg Etl_pkg_auth Etl_pkg_req Etl_control ETL_batch_override_dates ETL_data_supplier_vendor Ssis_task_failures Ssis_errors Transact_sql_errors sysdtslog90 Framework procs Flow-Specific Tables Work Staging Transformation_Error Etl_control Override Thresholds Logical_Errors Staging_Hist Override_Hist Threshold_Hist Corresponding procs

SSIS Parent and Child Templates Package naming convention Variable namespaces Data access methodology - procs Handling Task failures Handling detail error messages Handling proc failures Logging execution Pre-compiling script Unique system::id Compartmentalizing into many small packages vs one big Common flow Control the flow, let Sql operate on the data

Web Components - Operations

Web Components – Threshold Maintenance

ETL Success Notification

ETL Failure Notification

Web Components - Reporting

Thank You ! Mike Litchfield Assistant Vice President – IT Architect & Sr. Developer Income Research & Management mlitchfield@incomeresearch.com