Presenter: Rueen-Fang Wang, Ph.D., P.E. Fix Implementation Deficiencies of Database Structure Presenter: Rueen-Fang Wang, Ph.D., P.E. 1

2 Topics   Purpose of this task   Deficiencies identified   Steps to resolve deficiencies   Current status   Remaining tasks 2

3   To address issues discussed in the Panel’s subcommittee presentation (June 2007 ): Enforce meaningful referential integrity Enforce meaningful data uniqueness Purpose

4 Deficiencies Identified in the Initial Implementation   Related to referential integrity: Arc relationship Logical deletion List Item reference   Related to meaningful uniqueness Database Uniqueness rules

5 Identified Deficiency 1 – Arc Implementation DeficiencyScale of Problem Arc Usage Rules have not been enforced during the initial implementation => allowing incorrect references between tables 5 (<2%) rule violations out of 261 arc usage rules, involving approximately 15,000 (<0.1%) records out of 17,000,000 Referential integrity of arc relationship has not been enforced during the initial implementation => allowing creation of orphan records About 480,000 (~1.5%) orphan records exist via arc implementation out of 32,000,000

6 Identified Deficiency 2 – DELETED_DATETIME column DeficiencyScale of Problem Application performed incomplete logical deletion of data records by setting DELETED_DATETIME in the parent records only => bypassing FK check and allowing orphan records to be created at logical level More than 300,000 (~4%) orphan records exist out of 69,000,000

7 Identified Deficiency 3 – List Item Implementation DeficiencyScale of Problem List Item Usage Rules have not been enforced during the initial implementation => allowing creation of data records with incorrect lookup values (i.e. Party record refers to Place Type, etc.) 5 (~2%) rule violations out of 234 list item usage rules, involving approximately 26,000 (<0.05%) records out of 52,000,000

8 Identified Deficiency 4 – Meaningful Data Uniqueness DeficiencyScale of Problem Meaningful Data Uniqueness Rules have not been enforced for many tables in the initial implementation => allowing creation of duplicate data records Approximately 114 (80%) tables out of 142 tables being used do not contain rules to enforce meaningful uniqueness

9   Review and develop adequate database rules as necessary   Identify existing problematic data and the associated causes   Develop solutions to clean up the data and fix application   Develop and impose DB constraints and triggers to enforce database rules Steps to Resolve Deficiencies

10 Current Status ArcDeleted Date time List ItemMeaningful Uniqueness Develop Rules completed as of 9/30/07 completed as of 8/27/07 completed as of 8/31/07 to be completed by 3/31/08 Identify Problem completed as of 10/11/07 completed as of 9/20/07 completed as of 9/13/07 begin in Feb, 2008 Enforce Rules completed as of 12/31/07 to be completed by 3/31/08 begin in Feb, 2008 Fix App & Clean Data to be completed by 6/30/08 begin in Feb, 2008

11 Remaining Tasks Arc Fix application and clean up existing data Deleted Date time Clean up existing data marked as deleted, including linked child records List ItemCleanup 113 existing data records that violate a specific list item usage rule Meaningful Uniqueness Implement uniqueness rules for each table (including fixing application, clean up data, and impose rules)

12 Questions?