NERC Lessons Learned Summary LLs Published in December 2015

NERC Lessons Learned Published in December Three (3) NERC Lessons Learned (LL) will be published in December 2015 Human Error Leads to Evacuation of Primary Control Room Control Network Communication Path SOL and IROL Monitoring Tool Leads to Unnecessary Manual Load Shedding

Human Error Leads to Evacuation of Primary Control Room 3 A fire was caused by maintenance workers removing a decommissioned potable water tank Sparks from a torch ignited the plastic lining of the tank Fire caused extensive smoke which traveled up a utility tunnel and elevator shaft reaching the Energy Control Center (ECC) As a precaution, the personnel evacuated the primary ECC and utilized their back-up ECC When maintenance workers initially assessed the tank prior to starting the removal, they viewed the side section of the tank which had no lining or combustible materials The fire occurred in the center section of the tank which had the flammable lining

Human Error Leads to Evacuation of Primary Control Room 4 The maintenance workers failed to follow proper hot work procedures, which included requirements to fully inspect the area for combustible materials A fire watch was not established, and a hot work permit was not issued Corrective Actions The utility plans to implement the measures necessary to prevent smoke travel into the ECC as appropriate fire stops and ventilation changes All personnel have been retrained on the hot work permit system Workers should properly assess all work conditions before beginning any maintenance activities and follow established permitting processes

Human Error Leads to Evacuation of Primary Control Room 5 Periodic training on hot work procedures should be given to all maintenance employees ECC ventilation equipment and fire stops should be assessed on a regular basis to verify that proper precautions around ECC to ensure smoke prevented from reaching the ECC IF ECC is located adjacent to other facilities, attention should be given to the impact of these facilities on the ECC and means to mitigate such impacts

Control Network Communication Path 6 Entity scheduled a functional test of its Back-up Data Center (BDC) which included transfer from the primary Control Center (PCC) to the backup Control Center (BCC) When SCADA functionality was transferred, the TOP’s router bandwidth for network traffic between control centers (out-of-band router) saturated, preventing communication between the active Control Center (ACC) and the active data center (ADC) The ACC lost the ability to monitor and control its portion of the BES for approximately 39 minutes. SCADA Administrators failed back over to the PDC to resolve the issue

Control Network Communication Path 7 Going forward, the Entity is adding controls to prioritize SCADA communication network traffic on the out-of- band router to reduce the likelihood of saturation Entities should consider if there should be a prohibition on non-emergency changes when operating remotely from EMS/SCADA Servers

SOL and IROL Monitoring Tool Leads to Unnecessary Manual Load Shedding 8 System Operator (SO) ordered the removal of a shunt reactor for voltage control During removal, a fault occurred in the shunt reactor breaker that caused the high voltage transmission line to trip resulting in two IROL exceedances SO took emergency operating control actions to reduce the interface flows below the IROLs Both the SOL and an IROL for the interface were exceeded These limits are represented at the same time on the EMS monitoring displays adjacent to the interface flows During emergency condition (enabled in the EMS by the System Operators), the values of the SOLs on the EMS monitoring displays disappear and all the monitoring and assessment tools focus on the IROL limits

SOL and IROL Monitoring Tool Leads to Unnecessary Manual Load Shedding 9 When the IROL exceedances occurred, the SO didn’t enable the emergency condition in the EMS procedures to hide SOLs In his confusion, he mistook the value of the SOL as the IROL which led to a false assessment of actions to be taken to clear the IROL exceedances That led to the use of more aggressive emergency operation control actions than necessary 28 minutes after the event, the SO manually shed a significant amount of load even though it was not required

SOL and IROL Monitoring Tool Leads to Unnecessary Manual Load Shedding 10 System Operator – Training Enhance SO simulation training by implementing additional stressful conditions; Review SOs understanding of the different types of limits (SOL versus IROL) on a particular interface and how to distinguish and monitor them; Review the SOs understanding of the emergency operations control actions list; Review the SOs training guide regarding the monitoring of SOL and IROL limits to determine if additional clarification is required Monitoring Tools Verify that the EMS monitoring displays clearly differentiate between the SOL and IROL limits; Create a new tool for displaying the SOL and IROL limit timers; Develop a new display for SOs that would help them to quickly select the required control actions according to the type of exceedance

SOL and IROL Monitoring Tool Leads to Unnecessary Manual Load Shedding 11 Operating Procedures Clarify the definition of the SOL and IROL limits in the operating procedures; Clarify the application of the emergency condition in the operating procedures Clearly differentiate the SOL and IROL limits in the EMS displays and the control room timer display for SOL/IROL exceedances Review on a regular basis with the SOs their understanding of the SOL and IROL limits and the emergency operations control actions permitted in each situation More simulation type training for SOs needed with added stressful conditions in order to enhance their situational awareness and response during emergency operations

Link to Lessons Learned 12 Directions to Lessons Learned: Go to > “Program Areas & Departments” tab > “Reliability Risk Management” (left side menu) > “Event Analysis” (left side menu) > “Lessons Learned” (left side menu) NERC’s goal with publishing lessons learned is to provide industry with technical and understandable information that assists them with maintaining the reliability of the bulk power system. NERC requests that industry provide input on lessons learned by taking the short survey. The survey link is provided on each Lesson Learned.