Loss of EMS Events 2015 System Operator Seminar

Training Objectives 2 System Operators will be able to identify the ERO Event Analysis Process Category associated with a loss of EMS event System Operators will be able to demonstrate through a paper exercise the steps to take in response to a loss of Energy Management System (EMS) Event

NERC Event Analysis of EMS Event 3 ERO Event Analysis Process EMS Event Categories 1f - Unplanned evacuation from a control center facility with BPS SCADA functionality for 30 minutes or more. 1h - Loss of monitoring or control, at a control center, such that it significantly affects the entity’s ability to make operating decisions for 30 continuous minutes or more. 2b - Complete loss of SCADA, control or monitoring functionality for 30 minutes or more.

Examples of 1h Events 4 Examples include, but are not limited to the following: i. Loss of operator ability to remotely monitor, control Bulk Electric System (BES) elements, or both. ii. Loss of communications from SCADA RTUs iii. Unavailability of ICCP links reducing BES visibility iv. Loss of the ability to remotely monitor and control generating units via AGC v. Unacceptable State Estimator or Contingency Analysis solutions

Loss EMS Events 5 CategoryFRCCNERC (including FRCC) 1f121 1h275 2b10110 Total13206 CategoryFRCCNERC (including FRCC) Total28563 Category % NERC EMS Event 124% 281% All37% Data for all qualifying events from October 2010 through December 2014.

Loss of EMS Events 6 Are loss of EMS events impactful to System Operators? Have loss of EMS events been impactful to the BES?

Loss of EMS Events 7 Reasons for EMS events not having BES impact Most commonly occur during maintenance Planned for off peak hours and weekends with fairly flat loads EMS staff is nearby Heightened SO awareness Preparation and good SO action Processes and Procedures Training (Practice makes perfect) Interconnected Systems

Loss of EMS Ex. (Hass Power Company HPC) 8 03:00 – HPC automated planned disk management activity began to run. EMS was operating under normal conditions 03:11 - File system error occurred on the primary EMS server which causes a critical process to stop and the server becomes hung. This renders the replication of data to the backup EMS servers unavailable

Loss of EMS Ex. 9 Additional background on the backup and redundant servers The primary EMS server remained enabled so no automatic fail-over took place The gateway server lost it’s connection with the primary server The memory replication services application on the primary EMS server losses it’s connection to the three other servers that are designed to step in and take over The three backup EMS servers become invalid and unavailable for failover (status disabled)

Loss of EMS Ex :15 – EMS-2, EMS-3, and EMS-4 become disabled 03:16 – HPC System Operator (SO) notifies the RC that HPC has lost EMS functionality. 03:16 - HPC SO notifies all adjacent Balancing Authorities (BA) and Transmission Operators (TOP) via state hotline that they have lost EMS functionality. HPC requests all companies monitor their ties lines and flows with HPC and inform HPC and the RC if HPC is off schedule 03:16 - HPC SO posts a message on the online messaging system and logs the event (like FTMS)

Loss of EMS Ex :16 – HPC Balancing(BA) desk notifies EMS Support that the EMS is not functional 03:17 – HPC SO calls “Conservative Operations” via HPC hotline 03:18 – HPC SO informs RC that HPC has initiated Conservative Operations”. HPC SO states this means generation is to stay level and no new maintenance outages will be taken. 03:25 – EMS On-call Support reports back that EMS-1 (primary) server is still enabled and the three backup servers are disabled. EMS On-call Support dispatched to the control center. 03:26 – An open conference line was established to begin resolving the EMS server outage

“Conservative Operations”? 12 Each company may define Conservative Operations differently, so it’s important to understand what it means within your company. To HPC this means: Pause all work on the system leave the work in a safe state Notify SO of the current state and ask what should be done Do not place lines in or out of service Do not move generation Do not start any new maintenance or testing activities

Loss of EMS Ex :27 – George Power (GP) notifies HPC SO that HPC is dragging 75 MWs on their tie lines 03:27 – HPC SO redispatches generation to get back on schedule 03:27 - HPC BA desk contacts Timmy Generating Station and notifies them to increase output from 400 MWs to 475 MWs George Power (GP) is an BA adjacent to HPC

Loss of EMS Ex :40 –HPC SO calls GP SO to verify HPC is on schedule. 03:41 – GP SO verifies that it appears HPC is on schedule with GP 03:50 – RC contacts HPC SO to check on the status of EMS/SCADA. HPC SO reports that EMS support is working on the issue and they have attempted a manual failover but the failover was unsuccessful

Loss of EMS Ex :55 – Orange Power (OP) SO contacts HPC SO. “HPC is dragging 150 MWs on the ties” 03:55 – HPC SO redistaches generation to get back on schedule. HPC BA desk contacts Whoya Generating Station and requests they increase output from 520 MWs to 670 MWs 04:05 – OP SO contacts HPC SO. “HPC is back on schedule”

Loss of EMS Ex :15 – EMS-4 becomes primary enabled server. Shortly after SCADA application becomes available 04:16 – EMS begins to receive telemetry 04:21 – RTCA runs a successful solution. Solution used old data. 04:26 – EMS-3 transitioned from disabled to standby 04:30 – All application are now functional. EMS-4 is primary enabled and EMS-3 is in standby

Loss of EMS Ex :35 – HPC SO contacts RC to update the status of functionality. HPC SO notifies RC that RTCA is functional but the results are suspect 04:37 – GP SO contacts HPC SO. “HPC is dragging 70 MWs” 04:37 – HPC SO notifies GP SO that HPC has regained monitoring and control functionality. HPC SO confirms that he also see the mismatch and will redistach and get back on schedule.

Loss of EMS Ex :38 – HPC SO redistaches generation to get back on schedule. HPC BA desk contact Whatca Generating Station and ask that he increase output by 70 MWs. 04:40 – Successful RTCA solution using fresh data recorded. 04:45 - HPC SO verify no overload over voltage conditions exist. 04:50 – HPC SO checks into the conference line to communicate functionality has been restored and all applications are working properly

Loss of EMS Ex :51 – HPC SO checks into the conference line to communicate functionality has been restored and all applications are working properly 04:55 – HPC SO terminates “Conservative Operations” via HPC hotline. HPC SO requests that all maintenance activates be coordinated through him prior to any action. 04:56 – HPC SO notifies RC that “Conservative Operations” has been terminated and all clearances will be coordinated properly

FRCC RC Conservative Operations 20 Purpose: Implemented to maximize the FRCC’s ability to operate the BES reliably during limited or complete loss of EMS monitoring capabilities Identifies triggers to implement Conservative Operations Identifies actions and communications the RCSO should take during partial or complete loss of EMS/SCADA or RTCA Procedure title: FRCC Reliability Coordinator Conservative Operations Procedure FRCC- MS-OPRC-029

FRCC RC Conservative Operations 21 RCSO should declare Conservative Operations (Ops) if RC’s EMS, Data Acquisition, or RTCA is lost and ETR is >30 minutes or if RCSO has a reliability concern during loss of functionality In this presentation we will refer to this as a loss of functionality RCSO verify on duty supervisor is notified, request ETR, and (if applicable) ask about a plan to failover RCSO notify SE RC of loss of functionality and potential call for Conservative Ops Estimated Time of Repair (ETR) Southeastern (SE)

FRCC RC Conservative Operations 22 RCSO to request SE RC notify RCSO of any new contingencies and schedule discrepancies of >200 MWs until functionality is back RCSO to notify all OEs via hotline and FTMS of loss of functionality and Conservative Ops could be called OEs to acknowledge message over hotline RCSO to instruct all BAs to maintain ACE as close to zero as possible and notify RCSO of deviations >±50MWs OEs to notify RCSO of any new contingencies

FRCC RC Conservative Operations 23 OEs to notify the FRCC RCSO if they get nominal voltages exceed or if they have any real-time overloads on their transmission facilities For nuclear switchyards, provide notification to the RCSO if the local transmission station voltage is outside of established limits. RCSO to notify the MRCOO (or designee) of loss of functionality and Conservative Ops may be called RCSO to periodically check the RTCA internet sharing site to view other TOP’s CA output RCSO should log all pertinent data and OE reports received in a time sequenced manner

FRCC RC Conservative Operations 24 If the loss of functionality ETR is unknown or >30 minutes, the RCSO should declare Conservative Ops. Actions are the similar but the RCSO will notify that Conservative Ops is being implemented Additional steps: In addition to previous notify SE RC that Conservative Ops is implemented RCSO to contact SE RC for updates on actual vs. scheduled net interchange (ANI vs. SNI) in 15 minute intervals RCSO update SE RC on status of functionality every 60 minutes until Conservative Ops is terminated

FRCC RC Conservative Operations 25 RCSO to notify OEs via hotline and FTMS of loss of functionality and that Conservative Ops is being implemented Request acknowledgement OEs to provide ACE on the hotline and review ACE data against the data provided by SE RC RCSO request off schedule BAs to correct ACE and instruct FPL to redispatch to make up remaining discrepancies BAs to maintain ACE and notify RCSO of deviations >±50MWs Request OEs to notify the RCSO of any new contingencies

FRCC RC Conservative Operations 26 OEs to notify RCSO if nominal voltages exceed their alarm limits, or if they have any real-time overloads on their transmission facilities. For nuclear switchyards, provide notification to the RCSO if the local transmission station voltage is outside of established limits. RCSO to update the OEs on the status of the EMS every 60 minutes until the Conservative Operations have been terminated RCSO to notify MRCOO (or designee) of loss of functionality and implementation of Conservative Ops

FRCC RC Conservative Operations 27 RCSO should: Verify with Next-day planner that outaged lines have been returned to service Defer any planned outages that haven’t been taken until Conservative Ops is terminated Assess the need to reduce imports/exports with OEs Coordinate any interface reductions with interface owners Coordinate with OEs and assess the need to return to service any planned outage

FRCC RC Conservative Operations 28 When EMS/SCADA or RTCA is back: RCSO to terminate Conservative Ops and notify SE RC, OEs and MRCOO

Questions? 29