Page 1 December 2002 Update Summary December, 2002 As of
Page 2 Contents of this Update Update to Recommendations and Next Steps –Communications across the Market –ERCOT Transaction Processing Pipeline Monitoring Performance Scorecard and Data Transparency Workarounds Functionality –Other ERCOT IT issues –Other ERCOT issues –ERCOT’s Role in the Market Key Initiatives currently underway Additional areas of focus
Page 3 Update to Recommendations and Next Steps
Page 4 Communications Across the Market Effective exchange of key information regarding market related transaction processing has improved: –Planned and unplanned events affecting transaction processing within the Market are now effectively communicated by ERCOT via regularly scheduled conference calls and broadcast s as appropriate. –Processing exception information is shared by ERCOT and Market Participants during weekly teleconferences. –Key ERCOT and Market Participant contact information is published and maintained via the WWW. –A project proposal will be developed and prioritized that enables the use of the WWW via the ERCOT Portal for: Communicating all scheduled events with that schedule maintained by ERCOT and updated as needed. Providing current status of ERCOT processing services updated continuously.
Page 5 Communications Across the Market Visibility into ERCOT’s transaction processing pipeline as well as the current and historical status of all retail market transactions continues to increase: –Beginning June 2002, files have been transmitted to market participants daily that provided additional information regarding transactions received by ERCOT from those market participants. Data continues to be reviewed by ERCOT for accuracy and completeness. –Existing standard report delivery failures affecting market participants have been remedied. –PR ERCOT Data Transparency Initiative has been approved and is underway. Initial phases including “dark side” reporting and ESI ID Tracking System phase 1 delivered and further work/deliverables being planned for –ERCOT’s objective continues to be the delivery of full transaction transparency to market participants and the PUCT with the ability to view transaction history and current status information via the WWW. This includes: Full transaction life cycle within ERCOT systems ESI ID related information for ERCOT and Market Participants A Scorecard of performance in transaction processing within protocol for ERCOT Market Metrics supporting data reflecting market, market participants and ERCOT transaction processing success to be reported by the PUCT
Page 6 Retail Market Transaction Processing Pipeline Monitoring Component Health (Up/Down) Performance & Throughput Exceptions & Events Transaction Status
Page 7 Monitoring of Market Transaction Processing Pipeline Monitoring of key points and components within ERCOT’s transaction processing environment is much improved resulting in more stable transaction processing services to the market –IT has put automated monitoring software and/or manual processes in place that provide real time or near real time status of key processing components allowing more timely detection of any component failures that could affect the reliable operation of key transaction processing services provided by ERCOT. This effort will continue until all critical components can be monitored. –Key processing components have been “instrumented” to allow this ongoing monitoring and exception handling continues to be improved when problems occur. –Further automation is being pursued in such projects as PR20111/ PR ERCOT Transaction Pipeline Hardening and PR ERCOT Transaction Processing Dashboard, to reduce the cost of monitoring and increase the reliability of operation of the transaction processing systems. –Routine “care and feeding” of production related data bases within ERCOT has been expanded and improved resulting in improved stability and throughput.
Page 8 Hardening of Market Transaction Processing Pipeline Modification of major transaction processing software components within ERCOT systems along with tightened exception monitoring has increased overall stability: –Further and ongoing analysis of historical transaction failures provided insight needed to enhance or in some cases correct software within the ERCOT transaction processing systems. This has happened in the Market Participants systems as well. –After deeper technical review of key software implemented within ERCOT, specific components were modified resulting in improved performance and stability. –Tighter controls and monitoring within ERCOT on the flow of transactions entering ERCOT systems as well as the systems themselves have resulted in fewer failures and faster recovery when a failure occurs. –ERCOT personnel’s understanding of the transaction processing software developed by Accenture is growing and their ability to troubleshoot, modify and operate those systems is continually improving. –Key projects such as ERCOT Transaction Pipeline Hardening PR20111/PR30047 have been approved and launched and continue to have positive effects on the quality of the services provided the market though work still remains to be done well into 2003.
Page 9 Performance of Market Transaction Processing Pipeline Overall throughput within ERCOT of transactions submitted by market participants has improved with materially fewer major interruptions or failures: –Hardening efforts have contributed significantly to the improved throughput performance as well as the inherent reliability. –Improved management of the transaction flow is now in place and current average transaction flow rate is approximately 8000/hour with current processing rates being captured for future benchmarking and measurement of further improvement. Target is to eventually more than double this processing rate. –In key areas such as the SeeBeyond transaction processors, “multi-lane highways” have been put in place where “single lanes” previously existed providing parallel, concurrent processing and key software components such as the Siebel system have been upgraded. –A new “front end” based on the industry standard GISB specifications used for exchanging data with many market participants is being implemented to replace the original solution used since market opening. This is expected to further increase reliability and therefore throughput. –Software enhancements have resulted in vastly improved performance and uptime of the Retail Portal used by market participants to access ESI ID data within the ERCOT systems.
Page 10 Scorecard and Data Transparency In addition to the focus on increased data transparency discussed in the previous section on communications, the clear need exists to establish performance reporting at all levels. Toward that objective: –Within the Data Transparency Initiative, ETS Phase 1 transaction search/retrieval capability; initial reports on transaction response times, transaction flow reports within ERCOT were implemented in November –ERCOT is developing next phase of Data Transparency requirements to provide market participants with visibility into ERCOT transaction logging and reports. –The initial definitions of Market Performance Measures were provided by the PUCT in Project # ERCOT is determining supporting data requirements and scoping the effort necessary to collect key market measures by January 1 st –The first version of an automated Market Metrics capability is targeted to be in place by March, 2003 allowing analysis of first quarter 2003 data.
Page 11 Market Transaction Processing Functionality & Workarounds Current “safety net” workarounds have been identified and are being addressed: –Effects of these workaround processes have been examined through such efforts as the Market Synchronization project, the Move In/Out Task Force, the Quick Response Effort and the day to day work of ERCOT’s Registration and Data Analysis/Resolution Teams. –Several systems changes/fixes have already been implemented that will help to mitigate the need to continue these workarounds. –Improved execution of transaction exchange throughout the market has also reduced the systemic failures that caused the workarounds to be pursued in the early months of the market operation. –Key new functionality within TX Set 1.5 as well as the ultimate effects of the transaction processing changes coming from the Move In/Out Task Force will further reduce and in several instances eliminate the need for some workarounds that were implemented as a result of early market start up problems. –The cumulative effects of the above mentioned system changes/fixes as well as the planned implementation of TX Set 1.5 and the Move In/Out task force recommendations are being determined. –ERCOT in cooperation with the market participants will develop a complete plan, including conditions and timeframes, for eventual elimination or if appropriate, institutionalization of each workaround. RMS has already established an effort to develop a transition plan for eliminating the workaround known as the Contingency.
Page 12 Other ERCOT IT and related issues Utilization and effectiveness of all ERCOT and contractor resources continues to improve with positive results on development and operation of market supporting systems within ERCOT: –Formal closure to the contract with Accenture was reached including a final list of warranty items identified that are being completed by Accenture. –All major projects requiring ERCOT IT resources were identified and prioritized as part of the development and approval of the 2003 project list supporting the 2003 ERCOT budget submission to the PUCT. –Related efforts were consolidated and organized into fewer, more comprehensive projects such as the ERCOT Data Transparency project. –The foundation of an ongoing process for identifying and prioritizing projects, originating from the PUC, ERCOT and the Market Participants was established and continues to evolve as the organization absorbs and adapts to this framework. –Key processes within a Project Management Office framework are being institutionalized throughout ERCOT and a formal role in the organization is being established to track project progress including timelines, spending and deliverables. –The roles of ERCOT’s IT steering committee and executive committee are still evolving as is the integration/coordination with TAC and the ERCOT board on the prioritization and approval of all major projects. As these processes mature, we expect the pace of project review, approval and ultimately, implementation to improve further although not without some “spirited” debate over priorities and justifications of related project investments.
Page 13 Other ERCOT IT and related issues Testing methodologies continue to improve and responsibilities for testing are being adjusted for the purposes of effectiveness and efficiency: –Building on historical testing efforts, more complete testing methodologies for market wide projects such as TX SET 1.5 and GISB 1.4 are being developed and utilized. The learnings from these efforts will be incorporated into future market wide efforts to improve the quality and timeliness of system capabilities developed going forward. –Over the next year, ERCOT plans to absorb and integrate the currently independent market testing responsibilities into the recently defined testing organization within Market Services. –Appropriate testing organizations within ERCOT’s Information Systems and Market Services organizations have been defined and are being staffed. –As the planned staffing and processes are put in place, ERCOT will assume the primary role as the coordinator of all market-wide testing.
Page 14 Other ERCOT IT issues – Disaster Recovery Disaster Recovery requirements for Retail/Wholesale transaction processing have been established, agreed to, and approved by the ERCOT Board of Directors and included in the project list accompanying the 2003 ERCOT budget request: –Established 48-hour disaster recovery target for Retail Market processing in addition to Technical Operations requirements. –Assuming approval of the 2003 budget and project list, ERCOT’s plan specifies 48 Hour recovery capability for the market transaction processing services and is targeted for implementation in April –TCC will serve as primary processing with the Met Center serving as Disaster Recovery site. –Negotiations with related hardware and communications services vendors are nearly complete. –Processes supporting business continuity and disaster recovery must still be defined to insure effective continuation of required business processes in the event of a disaster.
Page 15 Other ERCOT issues ERCOT IT and Market Services organizations have been restructured and sized consistent with the roles, responsibilities, and service levels required to support the Texas Electricity Market: –The ERCOT Chief Information Officer (CIO) and Chief of Market Operations (CMO) positions have been filled with highly qualified candidates. –Where appropriate, director and manager positions within ERCOT have been filled from within and remaining open positions are being recruited for at this time. –Job descriptions for all positions within those organizations have been developed. –Overall growth of the leadership of the Retail Market Services and IT organizations is a key focus going forward. This will be accomplished by internal development of the existing leadership team as well as the outside hiring of additional personnel. –Additional positions needed were requested consistent with the proposed 2003 budget.
Page 16 Other ERCOT issues The retail market support resources were strengthened and are addressing a major portion of the backlog of ESI ID Problems that resulted from processing problems during the operation of the market since opening in January, 2003: –Over 160,000 ESI IDs were identified by market participants for error analysis and resolution. To date over 130,000 have been resolved and the rest are currently being fixed or analyzed to determine the requirements to resolve. –The original Tiger Team responsibilities were initially absorbed by the Quick Response Team and at this time, QRE is now being absorbed into the day-to-day retail market support groups (Registration Team and Data Analysis/Resolution Team). –Key problem management and resolution processes for these issues have been developed and are being migrated to the ERCOT Registration and Data Analysis/Resolution Team. –Software was developed to capture and track transaction problems and related information from initial recognition through final resolution. Data from this system continues to be used to update the PUCT, ERCOT Board of Directors, Technical Advisory Committee and the various ERCOT market related subcommittees. –Contract resources used to supplement the efforts continue to be scaled back and will be phased out.
Page 17 Other ERCOT issues The role of ERCOT in it’s support of the Texas Electricity Market has been better defined and reflected in the organization and related efforts to provide key services to that market: –The market services group that supports the retail market participants has assumed an expanded role in identifying, analyzing, and leading the efforts needed to address issues affecting the broader market. –Key leadership positions on efforts to address major market problems such as data transparency, market synchronization, TX Set 1.5 and move in/move out processing are being assumed by ERCOT in cooperation with the market participants on various working groups and task forces. –A more accountable and prominent role has been accepted by ERCOT and progress on problem resolution as well as key projects is regularly reported at the various forums such as PUCT, TAC, RMS, ERCOT BOD etc. –ERCOT must complete it’s work on finalizing their approach to staffing to support the ongoing work of TAC and the various subcommittees, working groups and task forces. Early and competent participation as issues/initiatives and ultimately projects emerge in those forums, is critical. –The proposed 2003 ERCOT budget includes headcount and related funding to adequately staff and support the accountabilities of the expanded role being assumed in providing leadership and services to the market.
Page 18 Key Initiatives Currently Underway
Page 19 Key Initiatives Currently Underway ERCOT’s Master Project Plan and Prioritization processes GISB 1.4 implementation TX Set Version 1.5 implementation ERCOT Data Transparency Move-In/Move-Out Issues/Activities Assimilation of Quick Recovery Effort Completion of Market Synchronization Activities Resettlement Update
Page 20 Additional Areas of Focus
Page 21 Additional Areas of Focus Formalization of an IT Architecture and Technology Plan –IT Architecture, Renewal, and Hardening Project Planning, Prioritization, Approval, and Management Process Critical Problem Management Framework Implementation More structured Vendor Management and Negotiation; Partnering ERCOT Personnel Development –Accountability and performance measurement –Market relationship management –Partnering (internal/external) –Communications to the Market Participants and PUCT –Continue lessening dependency on contractors Development of a data retention plan Completion of Disaster Recovery Project and Relocation to Taylor Control Center Incorporation of Business Continuation Planning into Disaster Recovery Efforts Expanded role for ERCOT in monitoring key market software/service providers Continued Cyber Security improvements