Review of Power Blackout on Telecom P. J Review of Power Blackout on Telecom P. J. Aduskevicz, NRSC Kathryn Condello, CTIA Capt. Katharine Burton, NCS/NCC

Power Related Outage Data Voluntary Outage Reporting Trial Data January – July 2003 Power Related Outage Reports 7 of Final Scrubbed Outage Reports 24% of the Total Final Scrubbed Outage Reports Examples of Causes Included: Back-up Generator Failure, Failed Equipment Power Supplies, and Configuration of Uninterruptible Power Supplies Examples of Applicable Best Practices Included: 6-5-0661 on Routine Exercising of Engines and 6-5-0660 on Planning for Providing Portable Generators Outage Reports Associated with the August 14-15, 2003 Power Failure 4 Initial Outage Reports received by the NCS/NCC Mandatory Outage Reporting Data 8 Initial Outage Reports received by the FCC 2 Final Outage Reports received by the FCC

~250 Cyber Security + Disaster Recovery + Public Safety Quarterly Meeting August 27, 2003 Review of Effectiveness of NRIC Power Best Practices Summary Statistics ~750 NRIC Best Practices ~250 Network Reliability ~250 Physical Security ~250 Cyber Security + Disaster Recovery + Public Safety 98 (13%) with “Power” keyword association 72 from NRIC V or earlier 26 new or edited in NRIC VI

California Rolling Blackouts Finding for Attention Quarterly Meeting August 27, 2003 August 2003 Power Outage Event History NRSC Power Studies August 2002 2001 1996 California Rolling Blackouts Finding for Attention Less power expertise identified in NRIC VI December 2002 Report as “Area for Attention”

Quarterly Meeting August 27, 2003 Review of Effectiveness of NRIC Power Best Practices Possible areas for Best Practices Evaluate fuel contractors ability to pump fuel (reports that some pumps at refueling locations did not have back-up power (work into contracts) Assess risk for need to enhancing back-up power at cell towers, cable networks, CLECs, others (implement existing BPs; consider fuel cell technology) Consider outreach of sharing best practices to electric power sector Pre-established plans for coordination between telecommunications service providers and city, state, and local emergency services Consumer education on the benefits of having a corded phone powered from a Central Office Overall: the traditional wired telecommunications performed well

Wireless Performance During Outage The power outage impacted numerous wireless markets. Responding carriers operate nearly 100 switches / 18,500 sites in blackout area. During first three hours, carriers report demand spikes of 150- 500% over normal busy hour. All Switches were operational throughout the incident. More than 2/3 of the network was uninterrupted throughout the incident. Networks were re-configured centrally to provide continued operation while power was replaced and/or augmented. Backhaul outages (Landline T-1 and DS-3 circuit faults) have been identified as root cause for some service interruptions. Restoration activities were coordinated through the Telecom-ISAC. Regular Electrical-ISAC updates allowed strategic deployment of replacement batteries and generators to impacted sites. 77% of Sites were operational at 24 hour mark, and 97% at 36 hours.

8/14 1530hrs 8/15 0330hrs 8/16 0730hrs

National Coordinating Center (NCC) & Telecom ISAC Established and maintained a conference bridge for wireline and wireless partners for the duration of power outage Coordinated problem resolution across telecommunications industry including generator and fueling strategies for New York and Detroit Coordinated with the Electrical ISAC to identify areas impacted by the outage and provided updates on restoration efforts Manned the ESF 2 position at FEMA HQ CWIN used as a transport media to coordinate with Canada SHARES set Operational Level 2 (emergency potential exists) Stations active in the affected areas TSP not required (circuits were not damaged) GETS: 1,813 GETS calls into or out of affected areas 93% Success rate GETS designed for 85% call completion success

Wireless Priority Service (WPS) Performance Many cell towers had only 2-4 hours of backup battery power 48 WPS calls completed 12 in the Washington DC area 36 in New York/New Jersey Operational measurement of WPS call attempts not yet implemented. Only call completion information available, based on billing information Cell tower access was the major cause of blockage for localized congestion WPS IOC capability provides priority to cell tower only offering little improvement during periods of widespread congestion. Unaware of current limitations, users from FRB New York, New York City government (NYPD), and GSA New York expressed concern over WPS performance.

Lessons Learned Power management and restoral practices at the tactical level under review by carriers Need to understand root cause of outage Appropriate mitigating or prevention techniques can be developed addressing both cause and expected probability of this event occurring again. May need modifications to the TESP program to mitigate additional risks Need to NS/EP services across ALL wireless Carriers and ALL platforms. WPS worked, but would have worked better if the NS/EP user could have had access to the top 5-6 carriers, both GSM and CDMA.

Lessons Learned Frequent Electrical ISAC teleconference participation provided excellent situational awareness to the NCC Telecom-ISAC Development of TESP program for cellular networks to address priority restoration of power for critical cellular communications facilities is needed Additional carriers and WPS FOC capabilities will address widespread outage problem Potential training issues on use and capabilities of WPS Review WPS training to better education users on current and projected program capabilities