Focus Group 2 Network Reliability PJ Aduskevicz, AT&T Ross Callon, Juniper Networks Wayne Hall, Comcast Cable Communications

Focus Group Mission Statement Define reliability measurements (units) for commercial communications networks (i.e., wireline and wireless transport networks, including satellite and cable) and for the Internet by March 22, Define reasonable, measurable customer-affecting outage reporting thresholds for commercial communications networks (i.e., wireline and wireless transport networks, including satellite and cable) and for the Internet by March 22, Conduct voluntary outage reporting trial, collect data, analyze results, and report on the validity, usefulness, and timeliness of the process and information obtained, and make recommendations for improvement. Based on trial results (including information on services affected by an outage), evaluate and report on the reliability of public communications network services in the United States. Should the Commission initiate an inquiry or rulemaking with respect to any of the above-mentioned issues, the Focus Group will provide input to the NRIC, which may make formal recommendations as a part of such proceeding(s). Evaluate, and report on, the reliability of public telecommunications network services in the United States.

Voluntary Outage Reporting Trial Results to 9/4/03 MonthReporting Organizations January6 February11 March12 April16 May15 June13 July19 August12 Note: Data collection for August in progress Note: Reporting Organizations are participating companies that have submitted either an outage report or a Positive Report during the specified month

Voluntary Outage Reporting Trial Results to 9/4/03 Note: Data collection for August in progress Note: Positive Reports are reports to the NCS/NCC that the participating organization did not experience an outage during the specified month Note: Failure Location for one outage report in May was undetermined

Voluntary Outage Reporting Trial Results to 9/4/03 Note: Data collection for August in progress

Voluntary Outage Reporting Trial Results to 9/4/03 Causes of Outages Reported Included: Fiber Optic Cable Damage, Procedural Errors, Hardware Failures, Software Design Errors, and Power Equipment Failures Applicable Best Practices Included: on routine testing of generator engines and power alarms, on deployment of security and reliability related software updates, on excavator protection of underground facilities, on Awareness Training field and management personnel, on employing an "Ask Yourself" program to reinforce the responsibility every employee has to ensure flawless network service

Voluntary Outage Reporting Trial Initial Observations Participation in the outage trial improved as a result of direct outreach to technical contacts of the participating organizations. Outage reporting process and guidelines have been significantly upgraded from trial learning's. Multi-industry Subject Matter Experts analysis of reports improved understanding of nature and scope of outage. Root/direct causes of voluntary trial outages generally mirror those now used by NRSC. Existing NRIC Best Practices have sufficed for analysis to date. Useful data can be obtained from a voluntary outage trial in a multi-network environment. Non-disclosure agreements, while difficult to negotiate, were essential to increase participation in the voluntary outage trial.

Reliability Reporting – NRSC 2Q03 Report Voluntary Trial Process Improvements Outage Reporting Guidelines Re-notification to all participating enterprises Technical Contacts Positive Reporting NCS/NCC Process Resources for NCC to assist process implementation Process Flow Enhancement and Clarification of Scrubbed Data Clarification of Service Provider review of scrubbed data before it is is passed from the NCC to the Focus Group, and clarification and understanding of the Data Elements to be passed Confirms enterprise sensitive data is removed Sufficiently protects “security” information Model Outage Reports Outage Report Withdrawal Process Positive Communication with All Technical Contacts Focus Group 2 Final Report Contents Voluntary Outage Reporting Trial Guidelines Voluntary Outage Reporting Trial Data Analysis Results Comparison to CFR 47 § Data Analysis Findings and Lessons Learned Recommendations on Voluntary Outage Reporting Trial Recommendations on CFR 47 § Network Reliability Steering Committee (NRSC) Report on the Public Switched Telecommunications Network (PSTN)

Reliability Reporting Network Reliability Steering Committee (NRSC) Analysis Reports

Reliability Reporting – NRSC 2Q03 Report FCC Reportable Service Outages (by number of events) Total outages (18) were at the lowest level of any quarter, and lower than any four consecutive quarters since the start of the Baseline Period (94).

Reliability Reporting – NRSC 2Q03 Report FCC Reportable Service Outages (by outage index) The aggregated outage index was at the lowest level (1084) of any four consecutive quarters since the start of the Baseline Period.

Reliability Reporting – NRSC 2Q03 Report Analysis of the outages for 2Q03 indicates: Total outages (18) were at the lowest level of any quarter and significantly lower than the Baseline Level. Total outages were lower than in any four consecutive quarters since the start of the Baseline Period (94). Facility outages (7) were significantly lower than the Baseline Level. Facility outages were lower than in any four consecutive quarters since the start of the Baseline Period (36) CO Power outages (8) were the lowest of any four consecutive quarters since 2Q96 to 1Q97. The aggregated outage index was at the lowest level (1084) of any four consecutive quarters since the start of the Baseline Period. Procedural Error as a root cause of outages (5) was at the lowest level of any quarter and significantly lower than its Baseline Level. Procedural Error outages were lower than in than in any four consecutive quarters since the start of the Baseline Period (35). Based upon analysis of all outages reported from 1Q93 through 2Q03, the NRSC notes that: There is a statistically significant decreasing trend in total outages since There is a statistically significant decreasing trend in frequency of Facility outages since 1995 and in the aggregated outage index since the start of the Baseline Period. There is a statistically significant decreasing trend in frequency of Local Switch outages since 1997 and in the aggregated outage index since the start of the Baseline Period. There is a statistically significant decreasing trend in frequency of CO Power outages over the last two years. There is a statistically significant increasing trend in frequency of CCS outages over the last seven years and in the aggregated outage index since The outage index for DCS outages is significantly higher from as compared to Procedural Error as the root cause of outages has exhibited a statistically significant decline in frequency since 1997.

NRSC 2002 Annual Report Table of Contents Introduction Major Findings Background State of the Network Root Cause Analysis “Special” Outages Conclusion Report Due Out October 1, 2003

Reliability Reporting – NRSC 2002 Annual Report 2002 Snapshot F = Outage Frequency I = Aggregated Outage Index

Reliability Reporting – NRSC Reports The NRSC urges all service providers and equipment vendors to review all best practices for application in their operations. These Best Practices may be found at:

Focus Group 2 Back Up

Focus Group Membership PJ Aduskevicz, AT&T Bonnie Amann, Sprint Jay Bennett, Telcordia Johnathan Boynton, SBC Ken Buckley, Federal Reserve Bob Burkhardt, Nextel Ross Callon, Juniper Rick Canaday, AT&T Kevin Cavanagh, AT&T Wireless John Chapa, SBC John Clarke, NCS/NCC Wayne Chiles, Verizon Joe Craig, Qwest Bernie Farrell, NCS David Fears, Cox Lee Fitzsimmons, Nextel Brian Goemmer, Western Wireless Jeff Goldthorp, FCC Wayne Hall, Comcast John Healy, FCC Dean Henderson, Nortel Dennis Pappas, Qwest Gary Pellegrino, CommFlow Resources Christopher Quesada, PAIX.net Karl Rauscher, Lucent Tony Reed, Charter Arthur Reilly, Cisco Systems Ira Richer, The Telesis Group Jim Runyon, Lucent Falguni Sarkar, AT&T Wireless Andy Scott, NCTA Don Smith, NCS Scott Smith, Cox Ron Stear, C&W Sandy Stephens, Focal Dorothy Stout, NCS/NCC Lee Taylor, RoxTel Whitey Thayer, FCC Nate Wann, NCS/NCC Frances Wentworth, NCS/NCC Chris Whyte, Microsoft Doug Williams, Comcast Cable Linna Zile, Cox Michael Hill, Level 3 Bob Holley, Cisco Robin Howard, Verizon Bruce Johnson, Verisign Rick Kemper, CTIA Percy Kimbrough, SBC Bill Klein, ATIS Bernie Ku, MCI Jim Lankford, SBC Greg Larson, Exodus/CWUSA Mike Lecocke, SBC Chris Liljenstolpe, CW Chris MacFarland, Allegiance Spilios Makris, Telcordia Archie McCain, BellSouth Dave McDysan, MCI Brian Micene, AT&T Wireless Denny Miller, Nortel Erick Mogelgaard, Cox Brad Nelson, Marconi Kent Nilsson, FCC Chris Oberg, Verizon Wireless

Inter-Domain Routers PSTN DSL Wireless Cable Dial-Up DNS RADIUS DHCP Core Backbone Distribution Service Aggregation XYZ Corp Internet/IP Service Provider

Outage Reporting Customer Definitions Type of Internet AccessCustomer DefinitionComments CableHouseholdWhether they are actively using it or not at the time of the outage Dial-Up Dial-Up Port Whether or not port is in use at time of outage DSL Household Whether they are actively using it or not at the time of the outage Satellite Household Whether they are actively using it or not at the time of the outage WirelessCustomers or Blocked Calls Historical trends may be used

Service Provider NCS / NCCFocus Group 2NRIC VI & Industry If yes If no NCS/NCC Logs Report Determine if Outage meets Trial Criteria Outage Occurs Create Initial Report Within 3 days Send to NCS / NCC Conduct Root Cause Analysis Identify Best Practice Send Final Report Within 30 Days Approve Scrubbed Report Local Analysis Collect, Validate and Send Data 1 2 initial final Scrub Data per Criteria established By FG2 Scrubbed Data Received by FG2 Analyze Scrubbed Data, Review BP coverage Provide Status at NRIC VI Council meetings Make Recommendations Monitor Progress Concur or Provide Input on Recommendations Develop Final Report to Include Recommendations File Final Report initial final 3 NRIC VI Voluntary Outage Reporting Trial Process

Voluntary Trial Outage Reporting Guidelines Compiled Guidelines for use by Service Providers, which includes: 1.Units and Thresholds – To determine which outages are to be reported; 2.Report Contents – To determine what information will be reported in confidential outage reports to a trusted third party under NDA; 3.Report Sanitizing – To determine what information is to be scrubbed from the confidential report before the report is made available to NRIC participants; 4.Confidential Report Repository – To determine which organization will be responsible for handling and sanitizing the confidential reports; 5.Reporting Process – To determine the process for reporting during the voluntary trial period. The data collected during the voluntary outage reporting trial is intended for use in improving network reliability, such as by providing information useful in order to verify and improve the NRIC best practices, or to create study groups to understand and improve issues identified as a result of the data. Specifically, it is not appropriate for reported data to be used for Marketing nor for Public Relations purposes.

Contents of Confidential Outage Reports Reporting carrier / service provider Contact person Telephone number of contact person Start date Start time of impact Geographic area affected Estimated number of customers affected Types of services affected (if applicable) Duration of outage (hours and minutes) Apparent or known cause Name of equipment involved [OPTIONAL] Type of equipment involved [OPTIONAL] Specific part of network involved Methods used to restore service [OPTIONAL] Steps taken to prevent recurrence Root cause and trouble found [OPTIONAL] Applicable best practice [OPTIONAL]

Scrubbing of Confidential Reports Scrubbing of Outage Reports Reporting carrier / service provider is deleted. This is replaced by a unique numerical identifier for the outage. Contact person (name, telephone number, address if present) is deleted. Date of incident is left unchanged. Time of incident is left unchanged. Geographic Area affected is made less specific. Only the city or general geographic area is maintained in the scrubbed report. The reporting service provider can work with the NCC/NCS to determine how general the geographic area should be after the scrubbing operation. Name and type of equipment involved is deleted. Other fields are left unchanged in the scrubbed report. Geographic Area affected is made less specific. Only the city or general geographic area is maintained in the scrubbed report. The reporting service provider can work with the NCC/NCS to determine how general the geographic area should be after the scrubbing operation. Other fields are left unchanged in the scrubbed report. Scrubbing of Positive Reports For positive reports, the only information which is maintained after the scrubbing function is the industry segment, and the month.

Voluntary Outage Report Withdrawal Process Initial Outage Reports are filed within 3 days of the event. If the Service Provider determines, upon further investigation, that the outage did not meet the Voluntary Outage Reporting Trial criteria, then the Service provider should notify the NCS/NCC that the Initial Outage Report is being Withdrawn. The NCS/NCC will track the number of Initial Outage Reports that have been Withdrawn for inclusion in analysis reports. Otherwise, the Final Outage Report is submitted within 30 days of the event.

Participating Organizations Providing Technical Contacts Allegiance AT&T AT&T Wireless BellSouth C&W CenturyTel Charter Communications Cingular Comcast Cable Communications Cox Communications EarthLink Focal Communications Intelsat Level 3 MCI McLeod Nextel NSF PanAmSat Qwest Sprint SBC T-Mobile Time Warner Cable Verisign Verizon Verizon Wireless Western Wireless

NRSC Back Up

Reliability Reporting – NRSC 2Q03 Report Incidents by Failure Category (Facility) There is a statistically significant decreasing trend in the number of Facility outages since 1995.

Reliability Reporting – NRSC 2Q03 Report Incidents by Failure Category (Local Switch) There is a statistically significant decreasing trend in the number of Local Switch outages since 1997.

Reliability Reporting – NRSC 2Q03 Report Incidents by Failure Category (CO Power) There is a statistically significant decreasing trend in the number of CO Power outages over the last two years.

Reliability Reporting – NRSC 2Q03 Report Incidents by Failure Category (Common Channel Signaling) There is a statistically significant increasing trend in the number of CCS outages over the last seven years.

Reliability Reporting – NRSC 2Q03 Report Procedural Error Attributed Outages (by number of events) Procedural Error as the root cause of outages has exhibited a statistically significant delcine in frequency since 1997.

Reliability Reporting – NRSC 2002 Annual Report Outage Frequency by Year

Reliability Reporting – NRSC 2002 Annual Report Aggregated Outage Index by Year

Reliability Reporting – NRSC 2002 Annual Report Outage Frequency Failure Category Distribution by Year

Reliability Reporting – NRSC 2002 Annual Report Failure Category Distribution 2002 Versus Baseline

Reliability Reporting – NRSC 2002 Annual Report Root Cause Category Distribution 2002 Versus Baseline

Reliability Reporting – NRSC 2002 Annual Report Summary Overall –Outage frequency and aggregated outage index in Green region Below network growth rates since 1993 Lowest to date –Outage frequency significantly lower than in Baseline Years Facility –Outage frequency in Green region Lowest to date Significantly lower than in Baseline Years Below baseline level for 3 rd consecutive year –Aggregated outage index in Green region Lowest to date Significantly lower than in Baseline Years Below baseline level for 4 th consecutive year Local Switch –Outage frequency in Green region Lowest to date Significantly lower than in Baseline Years Below baseline level for 4 th consecutive year –Aggregated outage index in Green region Significantly lower than in Baseline Years Below baseline level for 5 th consecutive year Tandem Switch –Outage frequency in Green region Lowest to date Below baseline level for 2 nd consecutive year –Aggregated outage index in Green region below baseline level for first year since 1999

Reliability Reporting – NRSC 2002 Annual Report Summary CCS –Outage frequency in Green region Above baseline level for 3 rd consecutive year Declined for 2 nd consecutive year –Aggregated outage index in Yellow region Highest to date Above baseline level for 3 rd consecutive year CO Power –Outage frequency in Green region Below baseline level for first year since 1996 –Aggregated outage index in Green region Below baseline level for first year since 1997 DCS –Outage frequency in Green region Above baseline level for 5 th year out of last 6 years –Aggregated outage index in Green region Below baseline level for 2 nd year out of last 6 years Other –Outage frequency and aggregated outage index in Green region Both below baseline levels for first year since 1999 –Outage frequency matched lowest value to date Procedural Errors –Outage frequency in Green region Lowest to date Significantly lower than in Baseline Years Below baseline level for first year since 1996 –Aggregated outage index in Green region Below baseline level for first year since 1999