NERC EAS Update WECC OPEAS Meeting 01/26/2016 Rich Hydzik

NERC EAS Update Third annual Monitoring and Situational Awareness Technical Conference –Held on September 29th and 30th at ERCOT Metro Center in Austin, Texas –Fourth Annual conference is being planned Held two industry webinars on Version 3.0 ERO Event Analysis Process and Appendices approved by the NERC OC –The process is effective January 1, 2016 –Finalized and posted all ERO EAP Documents

NERC EAP Update Revised NERC Event Analysis Process -Main goal: Continued improvement -Minor revisions to help improve and clarify the intent of the EAP -Intro: Reinforced EOP-004 is a required standard, EAP is a voluntary process and they have different purposes -It would be a disservice to industry to for the event lists to be the same -Process: Minor clarifying changes to help the end user grasp the high level process

NERC EAP Update Categories: Retired 1f Unplanned evacuation from a control center facility for 30 minutes or more Retired 2b –Complete loss of SCADA and monitoring capabilities for 30 minutes or more Modified 2c to clarify that reporting is based on an event that affects a number of facilities in a TOP’s footprint –Voltage excursion on one or two buses is not the intent

NERC EAP Update Retired Categories: Will not renumber the list Crossed out Added retirement dates and footnotes Example: Unplanned evacuation from a control center facility with BPS SCADA functionality for 30 minutes or more. Retired on January 01, Category 1f was retired since category 1h will cover an unplanned evacuation of a Control Center if the unplanned evacuation significantly affects the entity’s ability to make operating decisions for 30 continuous minutes or more

NERC EAP Appendix A was modified to clarify the timing requirements on the Brief Report and Event Analysis Report reports The timing requirements of Brief Reports were doubled in response to industry requests EAS’s mission is to improve the quality and completeness of reports and not burden the industry with short time requirements

NERC EAP Update Appendix C was modified to clarify expectations Added –Item 1: NCR # –Item 8: A list of relevant sustained forced outages and the bus configuration is requested –Item 11: Description of emergency actions taken (if required) –Item 19: Corrective actions were included (if applicable)

NERC Lessons Learned

Loss of EMS Communications Due to Lack of Validation on EMS Database RTU Configuration Parameter One point in a new RTU was incorrectly configured in the EMS database This terminated the Remote Communication (RCS) process on the primary SCADA server Failover to the alternate SCADA server also terminated the RCS process because the database error propagated to the alternate SCADA server Quality Assurance testing did not reveal the issue as this parameter was not checked in the database editor Result – Loss of communications to some RTU’s

Loss of EMS Communications Due to Lack of Validation on EMS Database RTU Configuration Parameter Lessons Learned –Evaluate parameters checked by the database editor to determine if there are gaps. Establish procedures for manual validation –SCADA software should be generate error messages and not terminate upon an incorrect parameter –Recovery strategies and procedures should be developed to facilitate quick recovery from failed updates

Relay Design and Testing Practices to Prevent Scheme Failures 230kv single phase to ground fault occurred on three terminal line One terminal reclosed, re-establishing the fault Fully redundant protection failed to clear fault –Failures were unrelated Fault evolved to multiphase for 58 seconds Cleared by backup protection on two 500kV lines Breaker failure protection was not initiated

Relay Design and Testing Practices to Prevent Scheme Failures Lessons Learned –Align relay testing with planned transmission outages to facilitate breaker trip testing –Separate contact strings for Relay #1 auxiliary trip relays were configured such that 52a contacts are in parallel rather than series – all three have to open to show open –Relay #2 was replaced with more modern relay (uP for better uP) –Line relays trip breaker directly, auxiliary tripping relays initiate breaker failure separately –Modern uP Relay #2 allows removal of 52a contacts from auxiliary relay tripping circuit – one less thing to fail

Loss of EMS Due to RTU LAN and UPS Failure Temporary rack mounted UPS failed –Installed to facilitate SCADA equipment additions previously –Supplemented undersized permanent UPS –Battery pack was dead – end of life –No internal bypass to fail over to line if battery failed RTU LAN failed Loss of system visibility Loss of ICCP for 50 minutes New UPS had been installed and temporary was not necessary

Loss of EMS Due to RTU LAN and UPS Failure Lessons Learned –UPS should have internal bypass if UPS fails –Automatic transfer switch should be used if no internal bypass –Periodic monitoring and maintenance of UPS

SOL and IROL Monitoring Tool Leads to Unnecessary Manual Load Shedding Line shunt reactor fault clears line IROL is exceeded post-contingency Immediate action is taken to mitigate IROL within 30 minutes –Start gas turbines –Curtail transactions –Emergency imports –Regional voltage reduction –IROL mitigated at 23 minutes, still an SOL –SOL exceedance was acceptable Interface has an SOL and an IROL Confused SOL with IROL in EMS system Load was shed to mitigate “IROL” which was an SOL at the time (28 minutes into event)

SOL and IROL Monitoring Tool Leads to Unnecessary Manual Load Shedding Lessons Learned –Clearly differentiate IROL and SOL in EMS –Review emergency actions for IROL and SOL on regular basis –More simulation training for System Operators is needed with added stressful circumstances to enhance awareness and response during emergency operations

Control Network Communication Path Following a failover exercise for EOP-008-1, router bandwidth saturated between control centers Stopped communication from Active Control Center (ACC) and Active Data Center (ADC) ADC houses active SCADA servers and associated equipment There is a Backup Control Center (BCC) and a Backup Data Center (BDC) ACC lost ability to monitor and control its portion of system for 39 minutes

Control Network Communication Path Functional test of Backup Data Center (BDC) Transfer SCADA from PDC located at PCC to BDC –Operating system from PCC via BDC Following failover exercise, routine maintenance tasks continued on BDC, the in service DC –Work had been done in the past without incident on an active DC During maintenance, SCADA heartbeat flatlined Failed back to PDC to resolve issue ACC lost ability to monitor and control its portion of system for 39 minutes

Control Network Communication Path When operating workstations remote from EMS/SCADA servers –Prioritize situational awareness (EMS data) traffic over other –Utilize Quality of Service tools on network devices to prioritize network traffic Consider prohibiting non-emergency changes when operating remotely form EMS/SCADA servers

Human Error Leads to Evacuation of Primary Control Room Evacuation of primary control center due to smoke/fire caused by maintenance activity Fire occurred in powerhouse adjacent to control center Fire limited to plastic inside metal tank Extensive smoke caused evacuation of powerhouse Smoke traveled up utility tunnel and elevator shaft reaching energy control center Primary control center was evacuated, backup control center was staffed Decommissioned tank was being removed, torch set off plastic lining Maintenance crew failed to follow hot work procedures

Human Error Leads to Evacuation of Primary Control Room Lessons Learned –Properly assess all work conditions before beginning maintenance activities and follow hot work processes –Periodic training on hot work procedures for all maintenance employees –Control center ventilation equipment and fire stops should be assessed on a regular basis to ensure smoke from internal or external fires cannot enter the control center –When control centers are not completely separate from other active facilities, consider the impact of the those facilities on the control center

Essential Reliability Services Task Force ERSTF reports are complete and posted on NERC website Reliability-Services-Task-Force-%28ERSTF%29.aspxhttp:// Reliability-Services-Task-Force-%28ERSTF%29.aspx Or, follow the links –Committees Other –Essential Reliability Services Task Force »Reports are under “Related Files” at bottom of page