Loss of EMS Events 2015 System Operator Seminar

Training Objectives 2 System Operators will be able to identify the ERO Event Analysis Process Category associated with a loss of EMS event System Operators will be able to demonstrate through a paper exercise the steps to take in response to a loss of Energy Management System (EMS) Event

NERC Event Analysis of EMS Event 3 ERO Event Analysis Process EMS Event Categories 1f - Unplanned evacuation from a control center facility with BPS SCADA functionality for 30 minutes or more. 1h - Loss of monitoring or control, at a control center, such that it significantly affects the entity’s ability to make operating decisions for 30 continuous minutes or more. 2b - Complete loss of SCADA, control or monitoring functionality for 30 minutes or more.

Examples of 1h Events 4 Examples include, but are not limited to the following: i. Loss of operator ability to remotely monitor, control Bulk Electric System (BES) elements, or both. ii. Loss of communications from SCADA RTUs iii. Unavailability of ICCP links reducing BES visibility iv. Loss of the ability to remotely monitor and control generating units via AGC v. Unacceptable State Estimator or Contingency Analysis solutions

Loss EMS Events 5 CategoryFRCCNERC (including FRCC) 1f121 1h275 2b10110 Total13206 CategoryFRCCNERC (including FRCC) Total28563 Category % NERC EMS Event to All Events 124% 281% All37% Data for all qualifying events from October 2010 through December 2014.

Loss of EMS Events 6 Are loss of EMS events impactful to System Operators? Have loss of EMS events been impactful to the BES?

Loss of EMS Events 7 Reasons for EMS events not having BES impact Most commonly occur during maintenance Planned for off peak hours and weekends with fairly flat loads EMS staff is nearby Heightened SO awareness Preparation and good SO action Processes and Procedures Training (Practice makes perfect) Interconnected Systems

FRCC RC Conservative Operations 8 Purpose: Implemented to maximize the FRCC’s ability to operate the BES reliably during limited or complete loss of EMS monitoring capabilities Identifies triggers to implement Conservative Operations Identifies actions and communications the RCSO should take during partial or complete loss of EMS/SCADA or RTCA Procedure title: FRCC Reliability Coordinator Conservative Operations Procedure FRCC-MS-OPRC-029

FRCC RC Conservative Operations 9 RCSO should declare Conservative Operations (Ops) if RC’s EMS, Data Acquisition, or RTCA is lost and ETR is >30 minutes or if RCSO has a reliability concern during loss of functionality In this presentation we will refer to this as a loss of functionality RCSO verify on duty supervisor is notified, request ETR, and (if applicable) ask about a plan to failover RCSO notify SE RC of loss of functionality and potential call for Conservative Ops Estimated Time of Repair (ETR) Southeastern (SE)

FRCC RC Conservative Operations 10 Operating Entities (OEs) as referenced in the RC Conservative Operations Procedure Balancing Authorities (BAs) Distribution Providers (DPs) Generator Owners (GOs) Generator Operators (GOPs) Load Serving Entities (LSEs) Transmission Operators (TOPs) Transmission Owners (TOs)

FRCC RC Conservative Operations 11 RCSO to request SE RC notify RCSO of any new contingencies and schedule discrepancies of >200 MWs until functionality is back RCSO to notify all OEs via hotline and FTMS of loss of functionality and Conservative Ops could be called OEs to acknowledge message over hotline RCSO to instruct all BAs to maintain ACE as close to zero as possible and notify RCSO of deviations >±50MWs

FRCC RC Conservative Operations 12 OEs to notify RCSO of any new contingencies OEs to notify RCSO if they get nominal voltages exceed or if they have any real-time overloads on their transmission facilities For nuclear switchyards, provide notification to the RCSO if the local transmission station voltage is outside of established limits.

FRCC RC Conservative Operations 13 RCSO to notify the MRCOO (or designee) of loss of functionality and Conservative Ops may be called RCSO to periodically check the RTCA internet sharing site to view other TOP’s CA output RCSO should log all pertinent data and OE reports received in a time sequenced manner

FRCC RC Conservative Operations 14 If the loss of functionality ETR is unknown or >30 minutes, the RCSO should declare Conservative Ops. Actions are similar but the RCSO will notify that Conservative Ops is being implemented Additional steps for SE RC: In addition to previous notify SE RC that Conservative Ops is implemented RCSO to contact SE RC for updates on actual vs. scheduled net interchange (ANI vs. SNI) in 15 minute intervals RCSO update SE RC on status of functionality every 60 minutes until Conservative Ops is terminated

FRCC RC Conservative Operations 15 Additional steps for OEs: RCSO to notify OEs via hotline and FTMS of loss of functionality and that Conservative Ops is being implemented Request acknowledgement OEs to provide ACE on the hotline RCSO to review ACE data against the data provided by SE RC RCSO request off schedule BAs to correct ACE and instruct FPL to redispatch to make up remaining discrepancies RCSO to update the OEs on the status of the EMS every 60 minutes until the Conservative Operations have been terminated

FRCC RC Conservative Operations 16 RCSO to notify MRCOO (or designee) of loss of functionality and implementation of Conservative Ops RCSO should: Verify with Next-day planner that outaged lines have been returned to service Defer any planned outages that haven’t been taken until Conservative Ops is terminated

FRCC RC Conservative Operations 17 RCSO should: Assess the need to reduce imports/exports with OEs Coordinate any interface reductions with interface owners Coordinate with OEs and assess the need to return to service any planned outage

FRCC RC Conservative Operations 18 When EMS/SCADA or RTCA is back: RCSO to terminate Conservative Ops and notify SE RC, OEs and MRCOO

“Conservative Operations” 19 All OEs need to understand what it means when the RC calls for Conservative Operations

“Conservative Operations” 20 FRCC RC requests: - OEs acknowledge the message on the hotline - BAs maintain ACE as close to zero as possible and notify FRCC RC of any deviations > 50 MWs - OEs notify FRCC RC of any new contingencies - OEs notify FRCC RC if nominal voltages exceed alarm limits, or if any real time overloads exist. - OEs notify FRCC RC if the local transmission stations on nuclear switchyards is outside of established limits

“Conservative Operations” 21 Internally each company may define Conservative Operations differently It’s also important to understand what it means within your company

Loss of EMS Exercise 22 An automated planned disk management activity began to run on HPC EMS server EMS was operating under normal conditions A file system error occurred on the primary EMS server which causes a critical process to stop and the server becomes hung This causes the replication of data to the backup EMS servers to be unavailable EMS-2, EMS-3, and EMS-4 become disabled

Loss of EMS Exercise 23 Physical Relocation Was not an Option!!! Primary EMS server remained enabled Therefore no automatic fail-over took place Gateway server lost it’s connection with the primary server Memory replication services application on the primary EMS server losses it’s connection to the three other servers that are designed to step in and take over The three backup EMS servers become invalid and unavailable for failover (status disabled)

Loss of EMS Exercise 24 FRCC RC and HPC share an EMS When HPC losses EMS functionality it usually means the RC will too HPC and the RC lost monitoring and control of their EMSs for over 1 hour HPC and the RC both called for Conservative Ops RC worked with Southeaster RC to monitor the state ACE

Loss of EMS Exercise 25 Assessment Instructions Three roles Reliability Coordinator, HPC Balancing Coordinator and HPC System Operator Checklist for each Sequence of Events provided Complete your work package and make sure to get a signature before you break

Questions? 26