August 25, 2008 TPTF Nodal Status Report: Quality Center Update Eileen Hall

2 August 25, 2008Nodal Status Report - Quality Center Overview Define strategy moving forward Demonstrate the consequences of insufficient testing durations Quality Center reports

3 August 25, 2008Nodal Status Report - Quality Center Strategy Moving Forward Revise schedules to provide sufficient test durations –Include adequate testing time in appropriate environment to allow for earlier detection of defects to decrease defect resolution time –Include adequate time for patches –Include adequate time for regression testing Deliver consistent defect management across program Increase focus on integration test Adhere to test phase exit criteria

4 August 25, 2008Nodal Status Report - Quality Center Impacts of Insufficient Testing Duration Impact analysis was conducted using early deliverables of the EMS project Compared ratio of test time to development time of overall effort for two releases –2C (lowest percentage of test time) –3C (highest percentage of test time) Reasons for using EMS project as case study –Ramp up time for testers not a factor –Large volume of historical data –Realistic data used during testing –In EDS for several releases

5 August 25, 2008Nodal Status Report - Quality Center Highest % Lowest % Disproportionate ratio of testing time to development EMS 3C: 17 weeks dev + 11 weeks test = 28 weeks effort (additional 11 weeks defect management) EMS 2C: 30 weeks dev + 3 weeks test = 33 weeks effort (additional 26 weeks defect management) Development (91.3%) Development (61.5%) Testing (8.7%) Testing (38.5%) Defect Management ( additional 79.0%) Defect Management (additional 40.0%)

6 August 25, 2008Nodal Status Report - Quality Center Number of weeks of test vs. number of weeks of defect management The longer the test duration, the shorter the defect management time (when number of weeks in test increases defect management time decreases)

7 August 25, 2008Nodal Status Report - Quality Center Overall % of defect management time per EMS Release Majority of overall defect fix time was expended in earlier releases. Defect management time has decreased over time as a result of longer test durations. 66%

8 August 25, 2008Nodal Status Report - Quality Center EMS defect turnaround time in EDS vs. FAT Defect turnaround time is the average time it takes (in days) to resolve a defect This time includes detection, analysis, fix and successful retest Late detection of defects increases defect turnaround time Current defect turnaround time for a defect detected during FAT is 51 days vs. a defect detected during EDS which is 73 days (43% longer)

9 August 25, 2008Nodal Status Report - Quality Center Nodal Active Defects by Severity Trend Severity 2 Severity 3 Severity 1 Severity 3 defect trend is remaining relatively constant Indicates that current defect close rate is equal to defect open rate 28 severity 1, 159 severity 2, and 741 severity 3 defects are currently outstanding

10 August 25, 2008Nodal Status Report - Quality Center Nodal Active Defects by Severity by Project  Number of Severity 3 defects remains high: MMS has 247 Severity 3 defects EMS has 186 Severity 3 defects NMMS has 55 Severity 3 defects OS has 54 Severity 3 defects Note – Includes New, Open, Reopen, Fixed and Test

11 August 25, 2008Nodal Status Report - Quality Center Average Days to Fix FAT Severity (1, 2, 3) Defect by Vendor Defect turnaround time is one key indicator of vendor performance.

12 August 25, 2008Nodal Status Report - Quality Center Questions?