Data Extracts & Reporting Recent Issues ERCOT Information Technology Data Extracts Working Group 11/27/07
Agenda TML API Outage 9/29/07-10/1/07 Market Notice Review –M-A –R-A –M-A –M-C –M-A Update regarding the search for a new EIS manager
TML API Issue Issue: Market connectivity issues to TML via the API –Issue Dates: 9/29/07 – 10/1/07 –Market Notice: Issue was not experienced market-wide. –Root Cause: A greater than allowed number of connections to the API caused the Enterprise Application Integration (EAI) layer to refuse some connection requests. –Resolution: Increasing the number of connection threads in the EAI layer allowed all requests to be handled appropriately. Monitoring: ERCOT has added additional monitoring to capture and correct this prior to becoming a market impacting issue.
10/29/ /26/07 Extract Issues Issue: ERCOT experienced delays with posting several data extracts –Notice Dates: 10/31/07-11/9/07 –Market Notice: M-A –Root Cause: ERCOT batch process delays due to Oracle bug when running database statistics. –Resolution: Utilize Oracle recommended workaround for running database statistics. Further information and timeline on next slide
10/29/ /26/07 Extract Issues Market Notice: M-A (cont’d) Incident Timeline 10/21 – Server consolidation performed during maintenance outage 10/27 – Database statistics fail – first time running statistics on new platform, causes batch process to run slowly 10/28 – Daylight Savings Time adjustment issue with batch scheduling application 10/29 – Partial rollback of database statistics –Severity 1 Technical Assistance Request opened with Oracle –Batch process continuing to run slowly 10/30 – Events –0615 – decision made to rerun database statistics, batch held from 0745 until 1030 when statistics estimated to have completed –1030 – reinstated batch while statistics were still refreshing –1915 – terminated statistics refresh, attempted statistics with smaller database sample –1945 – full rollback to old statistics – batch performs at expected levels »Also began making adjustments to batch schedule in attempt to make up time which caused issue described in Market Notice M-C /5 – Estimated catch up date 11/9 – Actual catch up date 11/16 – Oracle confirms issue that caused statistics refresh to run for so long was due to a software bug
10/29/ /26/07 Extract Issues Issue: Additional ESI ID Extract to post from some Market Participants –Notice Date: 11/9/07 –Market Notice: R-A Extracts - Retail –Root Cause: The 727 delete objects (esiidservicehist_delete, lschannelcutheader_delete, esiidusage_delete) only pull rows that have been actually deleted. The extract views explicitly only pull rows that have an ov_row_status of 'ID' or 'UD‘. –Resolution: Removal of the " and ov_row_status in ('ID' or 'UD') " criterion from the delete extract views. To correct the missed rows, a supplemental extract is created.
10/29/ /26/07 Extract Issues Issue: ERCOT is experiencing delays with posting several extracts –Notice Date: 11/12/07-11/13/07 –Market Notice: M-A –Root Cause: Complications with the batch scheduling software encountered while completing an ERCOT requested rerun of the 11/8 batch (10/31 trade date) resulted in the 11/8 batch running on Sunday 11/10 and completing on 11/11. We moved forward with the 11/9 batch and it completed at approximately 5:00 p.m. on Saturday. When Lodestar batch ran out of order, the controls table was not accurately populated. –Resolution: The control table rows were manually entered. A SIR has been requested to modify the control record creation process.
10/29/ /26/07 Extract Issues Issue: Data for operating day 10/31 was not included in the Load Generation, Settlements and Billing extracts posted on 11/12 –Notice Date: 11/14/07 – 11/15/07 –Market Notice: M-C –Root Cause: Complications with the batch scheduling software encountered while completing an ERCOT requested rerun of the 11/8 batch (10/31 trade date) resulted in the 11/8 batch running on Sunday 11/10 and completing on 11/11. We moved forward with the 11/9 batch and it completed at approximately 5:00 p.m. on Saturday. When Lodestar batch ran out of order, the controls table was not accurately populated. –Resolution: Control table rows previously manually entered did not take into account that the 11/8 batch day data was not complete until 11/11. Related to market notice M-A A SIR has been requested to modify the control record creation process.
10/29/ /26/07 Extract Issues Issue: Delays with posting several data extracts/reports –Notice Date: 11/21/07 - present –Market Notice: M-A –Root Cause: Extracts are posting one day behind due to batch run delays. –Resolution: Currently analyzing several key areas including correcting an issue with backups and tuning database performance.