April 1, 2008 TPTF Quality Center Update Eileen Hall

2 2 Quality - Summary Recent Successes –CRR1 has exited FAT –MMS3 has exited FAT –EMS3B and EMS4.1 have exited FAT Areas of Concern –Lengthy defect resolution time delays test phase migration –Late detection of detects exacerbates defect resolution time Mitigation Measures –Passive data center site with live data feed will allow for earlier defect detection –Additional measures (e.g. key vendor resource retention) in progress Nodal Quality effort experienced recent successes but concerns remain

3 3 Nodal FAT Active Defects by Severity by Project  Overall, active defects are being tracked by project and by classification. Problems areas include: Network Model Management System (NMMS) has 12 Severity 1 defects Outage Scheduler (OS) has 6 Severity 2 defects Congestion Revenue Rights (CRR) has 2 Severity 2 Note - Does not include closed and deferred defects The relative low number of Severity 1 and 2 defects in FAT is encouraging

4 4 Nodal Active Defects by Severity Trend Sev 3 Sev 4 Sev 1 Sev 2 Severity 3 trend is rising due to focus on fixing higher priority severity 1 and 2 defects Severity 2 trend indicates “defect closure rate” is slightly higher than “defect open rate”

5 5 Average Days to Fix Severity (1, 2, 3) Defect by Vendor Defect turnaround time is a key indicator of vendor performance; increased emphasis being placed on vendor defect resolution as needed

6 6 Nodal % Reopened Defects by Vendor Indicates what percentage of fixes delivered by a vendor failed retest.  ABB and AREVA are fixing their products with a low reopen rate, a good indication that the EMS and the MMS vendors are sufficiently testing their software defect bundles prior to releasing software to ERCOT  A high reopen rate is an indication of future vendor performance Smaller vendors have experienced the most difficulties in terms of defect reopening

7 7 Defects found in the Early Delivery Systems (EDS) Severity 1 – Data Loss/Critical Error Severity 2 – Loss of Functionality w/o workaround Severity 3 – Loss of functionality with workaround Severity 4 – Partial Loss of a feature set Severity 5 – Cosmetic/Documentation Error Severity 6 – Enhancement Severity 1 Severity 2 Severity 3 Severity 4 Severity 5 Severity 6  Focus of EDS testing effort on core functionality, therefore majority of defects detected are severity  Lesson learned: EMS 2 and EMS 3 were released with insufficient FAT testing, resulting in substantial number of defects showing up in the Early Delivery Systems  Mitigation measures are being put in place for earlier defect detection, e.g. live data feed in integrated test environment Enhanced FAT testing should result in increased defect detection prior to EDS

8 8 1 Data Loss/Critical Error 2 Loss of functionality w/o wo.. 3 Loss of functionality with w.. Number of Defects Notes: 1) Data only includes defects identified in EDS testing (i.e. defects that migrated into EDS from FAT are excluded) 2) Data does not include closed or deferred defects 3) Release 8 (Credit, Settlement and Billing) not yet in EDS; no EDS identified defects in Release 4 at this time EDS Active Defects Per Release by Severity EDS active critical defects (Severity 1 and 2) are concentrated in Releases 3, 5, and 9* * State Estimator (SE) tuning, Security Constrained Economic Dispatch (SCED) and Locational Marginal Price (LMP) posting, and Day Ahead Market (DAM), et.al. respectively Release No  Current testing in EDS represents approximately % of code to be tested in EDS  The number of defects associated with remaining EDS testing is expected to increase significantly due to the increasing complexity associated with systems’ integration  Timely fixes to the currently unknown number of defects in the code remaining to be tested in EDS represents a major project challenge