1. Verizon Contingency Planning for COOP
“Ensuring Mission Critical Operations in an Unpredictable World…”

2. The Plans Explained Continuity of Operations Plan (COOP) (IAW FPC 65)
Viable, executable plans for leadership, succession, and key personnel to ensure that a department/agency’s essential functions continue to function as needed
Contingency Plan (IAW NIST special PUB )
Planning, development, Implementation, and Testing of the enterprise to support and sustain mission-critical operations with reliable network, communications, and applications solutions during crisis situations

3. The Impact: How Downtime Affects the Mission
Disruptions of the Mission can Impact a Wide Range of Constituents and Customers
External (Customers/General Public/Media)
Services
Salaries and Benefits
Informational Communications
Critical Communications
Internal (Employees)
Benefits
Instructional Communications
Intergovernmental (Federal/State/Local)
Agency-to-Agency Communications and Policy Coordination
Federal-to-State grants Communications and Policy Coordination
Federal-to-Local Communications

4. Verizon’s Experience in Business Continuity
On September 11, 2001, Verizon lost the 140 West Street building that included a major switching center for the area.
On Sept 11, Verizon was impacted in many ways. Not only were we the predominant infrastructure provide to lower Manhattan, but we also had employees both in the World Trade Center as well as buildings close by.

5. Verizon’s Experience in Business Continuity
Entrance to the West Street building that was heavily damaged by the collapse of the WTC. West Street housed a major Central Office and Switching Center, as well as facilities for operations and support staff.
In the upper left corner is a picture of the 140 West street building, located next to the World Trade Center. This building housed the major switching for lower Manhattan as well as Network Engineering, Monitoring and support systems.
The lower right shows the view from inside one of the raised floor areas in the building. The computer system in the picture’s corner is the E911 system for Manhattan. It is interesting to note that because Verizon builds it’s systems with Business Continuity as an objective, this system continued to run on emergency battery and generator power until it was ordered to switch over to the fail-over system in another part of New York city. The E911 system continued to process calls throughout the entire period.
View of damaged New York City E911 switching system. The system was replaced by a new switching system in an alternate location in Southern Manhattan.

6. Verizon’s Experience in Business Continuity
Assessing the WTC Damage
Access lines out of service (voice) 200,000
PBX / Centrex lines out of service 150,000
Data circuits out of service ~4 Million
Cell sites out of service 10
Major Computer Systems 3
Number of customers affected:
Business ~ 14,000
Residential ~ 20,000
The response to our customers in lower Manhattan was tremendous. We deployed 18 new Sonet rings for customer data traffic and 21 temporary cell towers were erected to handle the traffic of the emergency workers, civilians and businesses.
Our computer systems that needed to be moved or recovered were done so within 24 hours and we deployed 3000 technicians and managers from throughout all of Verizon in the U.S. to work on the immediate recovery. Being able to bring in immediate, trained resources from elsewhere in the U.S. provide Verizon with the skills and manpower to provide round the clock support.
All 4 million data circuits were repaired, recovered or re-routed and to help those customers that could not return to lower Manhattan to work, we aggressively deployed our Remote Office solutions such as Centrex, ISDN, DSL and video/audio conferencing to allow them to continue operating their business.
Even though our first and foremost priority was to support the First Responders such as Police/Fire/EMS and critical government agencies, our successful operations allowed Verizon to bring up the SIDN and allow the NYSE to open only 6 days after the tragedy.
Landline and wireless network call completions were at least 2 times higher than normal levels on Sept. 11.

7. Verizon Expertise in Business Recovery & Continuity Services
Verizon’s Response to our Customers
Network Capacity Added:
18 New SONET Rings Built
21 Temporary Cellular Towers Erected
Computer systems recovered within 24 hrs at alternate sites
3,000 technicians and managers deployed in southern Manhattan.
4,000,000 voice and data circuits constructed, repaired, or rerouted within first week
Aggressive deployment of Centrex, ISDN, DSL, video and audio conferencing
The response to our customers in lower Manhattan was tremendous. We deployed 18 new Sonet rings for customer data traffic and 21 temporary cell towers were erected to handle the traffic of the emergency workers, civilians and businesses.
Our computer systems that needed to be moved or recovered were done so within 24 hours and we deployed 3000 technicians and managers from throughout all of Verizon in the U.S. to work on the immediate recovery. Being able to bring in immediate, trained resources from elsewhere in the U.S. provide Verizon with the skills and manpower to provide round the clock support.
All 4 million data circuits were repaired, recovered or re-routed and to help those customers that could not return to lower Manhattan to work, we aggressively deployed our Remote Office solutions such as Centrex, ISDN, DSL and video/audio conferencing to allow them to continue operating their business.
Even though our first and foremost priority was to support the First Responders such as Police/Fire/EMS and critical government agencies, our successful operations allowed Verizon to bring up the SIDN and allow the NYSE to open only 6 days after the tragedy.
Verizon recovered and rerouted the Securities Industry Data Network to allow the NYSE to open on 9/17.

8. Lessons Learned - Network
SONET Worked
Diversity Worked
Entrances, Routes, Risers
Vendor Relationships
Strong Partnerships Pay Off
Remote Access Critical
Assume Non-Local Operations
Dial-In, Vmail, , I-M
Back Up/Standby Service
Dedicated or Shared Service?
Will it be Available?
Practice Contingency Plans
What did we and all of the other carriers learn about the Network
1) SONET worked well. Customers on SONET rings did not have the impact of those who weren’t on SONET.
2) Diversity also worked. Diversity of your building entrances and network routes is a primary step, however, depending on the level of diversity you require and the ROI, diversity of risers and network within campuses and building will provide an even higher level of protection
3) Without strong vendor relationships, we could not have been as successful as we were. Our key vendors such as Nortel/Lucent etc joined in with us side by side to provide the equipment and service we needed to respond to our customers. This is important as you (the customers) look towards your Business Continuity plans. The vendors you select as you partners will be critical to your success.
4) Remote access became the standard method, not the occasional method for thousands of people to continue their work. Assume non-local operations and have backup methods in place.
5) Check you vendor backup services. Some services use shared facilities such as floorspace and systems. In an emergency, the first person there gets the service and everyone else is out of luck.
6) Practice, Practice, Practice…. We in Vz practice our contingency operations in the network continuously and test our 8 major data centers on a yearly basis. This played a key role in having the processes and expertise when needed.

9. Lessons Learned - Data Center
Watch Out for Water!
Back Up Systems are Mandatory
Power & Fuel
Back Up Services (Voice/Data/ISP/Cell)
Back Up Data (Directories, Documentation, PC’s)
Back Up Documentation available in multiple places
Document Procedures
Up to Date (with process to keep current)
Conflicts can be eliminated
Prepare for Employee Downtime
People will need to contact Family
Plan for Family needs at your location
What did we learn about our Data Center preparedness
1) Water is the arch enemy of Computers…. Fire companies have only one goal and that is to put as much water in a fire area to smother it through oxygen deprivation. Unfortunately that takes out your power and systems in a relatively short period of time. Also, locating critical systems in a basement vault needs to be looked at carefully to make sure it is waterproof or it turns into a very expensive swimming pool.
2) Without backup systems in place, recovery timeframes are considerable lengthened.
3) Procedures are the main source of information when people disruption is encountered. If the person who knows how to do it wasn’t available, it wouldn’t get done and it also keeps two very good Subject matter experts from accidentally overwriting each others work. Vz Data Centers are ISO 9000 certified which means that every process and procedure is documented and available for everyone in the organization to use. These procedures were key to providing remote Systems Administrators the ability to begin the recovery of systems, even while the primary admins were traveling from NYC to an alternate facility.
4) Prepare for Employee downtime. Employees will need to contact their family and let them know they are OK and check on them. You also need to consider that there may be a need to provide services to employee’s family’s at your site. On own experiences on 9/11 showed that employees who have their family needs cared for, can focus much more clearly on the tasks at hand.

10. Lessons Learned - Customer Comments and Directions
Top Six Focus Points of Verizon’s Customers Today
Assess Requirements, Plan and Document
Infrastructure Diversification and Hardening
Remote Access Strategies need Defined
Security Updated (Firewalls, Log Reviews, ID’s
Capacity Planning Strategy needs Updated
Contingency “War Games”
What have our customers been saying to us about their experiences and plans ahead?
In order or priority and action:
1) Planning and Assessment is ongoing everywhere.
2) Infrastructure Diversification and hardening has moved to the top of the list or many corporations
3) Remote Access proved itself and many companies found that they did not have adequate access or capacity.
4) Security systems need to be more robust and stronger to handle these occurrences.
5)With the constantly changing landscape of many organizations network and systems, better capacity planning needs to be done on a regular basis to account for future disruptions.
6) “War Games” are becoming the norm to test Business Continuity. As I mentioned, Vz practices this rigorously in our network and Data Centers.

11. Business Continuity Scope
Many layers to Business Continuity, not just the “Building Catastrophe” scenario !
Security
Operational
Capacity Requirements
Application/System Failure
Facilities loss
-> Web Attacks, Virus
-> Staff and Process Issues
-> Unexpected Need
-> System/Data Recovery
-> Complete recovery
A common misperception about Business Continuity is that is is required in the case of an emergency. Not exactly, Disaster Recovery is the reactive nature or preparing for adversity. Business Continuity takes into account a wide variety of situations that disrupt the business, but may not involve a complete “Disaster” scenario. Also,if done properly, Business Continuity planning will provide a wide variety of benefits to every day Production and Operation.
Take for example Security. By addressing the contingency of web attacks and virus’s in every day business, you have built the framework of your Security Business Continuity Plan of a Building Catastrophe.
And by addressing each of the critical areas independently and in a day to day framework, he overall “Facilities Loss” plan is easier to build and maintain.
By planning for the day to day issues in a Business Continuity focus, your overall effort to plan for the “Building Catastrophe” is minimized!

12. Verizon Standard Products for BRCS
Network Diversity and Recoverability
Optical Technology (SONET, DWDM, Frame Relay, ATM)
Alternate/Dual Wire Centers and Central Offices
Dual Routing
Transparent Lan (TLS)
Gigabit Ethernet
VoIP / IPT as backup
Standard Products
Managed Services
Consulting and Assessments
Data Center Hosting and Storage Services
Primary Customer Premise
PBX
Telephone
Router
VOIP
Demarc 1
Demarc 2
Diverse Secondary C.O.
Primary C.O.
Redundant Fiber Backbone with Diverse Entry
AIN Services
Mainframe
Workstation
Hub
Centrex Fail-over

13. Verizon Standard Products for BRCS
Personnel Communications
Video and Audio conferencing
Decreases travel costs
Call Forwarding
Reroute your calls to an alternate phone number
Standby Centrex
Provides backup services without the need to manage the equipment
Remote Access Services
Critical for travel interruptions during crisis
Meets the demands of today’s more flexible workforce
Virtual Office
Capabilities of the corporate office are replicated in the home office
Access Security Solutions
Access security is no longer “optional”
Data Center Hosting and Storage Services
Consulting and Assessments
Managed Services
Standard Products

14. Verizon Managed Services
Network Services
Custom Network Design and Diversity
Virtual Private Network (VPN). Secure, business- quality wide-area communications service
Managed Network Solutions. Outsource the operational management of your network
Site Patrol. Network and Intranet Security
Design Services. Adapt your network design on an on-going basis
Custom Redirect and Portability
Planned redirection of incoming calls for blocks of phone numbers to alternate sites
Built into the Verizon Network, not just your local CO
Local, long distance, and international adaptability
Data Center Hosting and Storage Services
Consulting and Assessments
Managed Services
Standard Products

15. Verizon Managed Services
Virtual Office
Managed Video and Audio Conferencing
Customer-controlled, secure, turn-key solution
Reservation-less Conference Connections enable voice conferencing from any phone, any time, any place
Remote Access Solutions
Central point of coordination for all aspects of a remote access program
Eliminates program management
Control and predictability over access-related costs
Flexibility & scalability – as much as you need but not more than you want
Ability to disperse your company’s human resources
Data Center Hosting and Storage Services
Consulting and Assessments
Managed Services
Standard Products

16. Verizon BRCS Professional Services
Vulnerability Assessments
Business Recovery and Continuity Gap Analysis
Network Survivability Assessment
Security
Complex vulnerability assessments
Cyber attack response
Contingency Planning
Business Impact Analysis
Recoverability Assessment and Health check
Continuity Strategy
Data Center Recovery Plan Development
Corporate and Departmental Continuity Plan Development
Alliances with IBM, EMC and Accenture
Leverage Resources of all Companies
Data Center Hosting and Storage Services
Consulting and Assessments
Managed Services
Standard Products

17. Verizon Data Center Hosting and Storage Services
Remote Data Center Services
Hotsite Services
Backup & Recovery Services
Storage Services
Backup / Archival Solutions
Remote Storage
Storage Area Networks
Hosting Services
Application Hosting
eBusiness, ERP, Application Continuity Services
Total I/S Management Services
Data Center Hosting and Storage Services
Consulting and Assessments
Managed Services
Standard Products

18. Verizon’s Commitment to Business Continuity
Verizon Network Engineering and IS Operations have a legacy of providing extremely high availability.
Depth and Breadth of Services
6300 Central Offices and Raised Floor space
1.5M square miles of service area
50 Mainframes, Unix systems and 100K PC’s
Mission Critical Recovery times of 24 hours or less
ISO 9001:2000 Certifications for IS Operations and Business Continuity
Presence in 41 international locations with operations in 21
Verizon’s commitment to business continuity is and integral part of our Network Engineering and IS Operations legacy. With 6300 C.O.’s and raised floor spaces, 1.5 million square miles of service area, 50 Mainframes, over 5000 Unix-based servers and 100K plus PC’s to account for with our Mission Critical systems targeted with recovery is 24 hours or less, we have continued to keep our ISO certification for our IS Operations and Business Continuity Processes.
And we have extended Verizon’s abilities to 41 international locations with Operations in over half.
Our reputation for service and reliability is part of the heritage that is integral to our customer commitment
“Our reputation for service and reliability is part of our heritage and integral to our customer commitment”

19. BRCS Summary Think Business Continuity...
Verizon’s Business Recovery and Continuity Services (BRCS):
Standard and Customized Solutions for Communications and Network Infrastructures Critical to Continuity of Operations
Professional Assessment and Consultative Services
Remote Data Center and Storage Services
Global Network
Verizon’s Strength and Stability
In summary, disaster recovery only touches the surface for ensuring that your business continues to function during any type of outage (minor or catastrophic). The paradigm has shifted in not only defining a “disaster” but how to address them. Additionally, the reliance of IT services to run a business has necessitated this new shift towards “business continuity” - for even a minor outage now can be catastrophic to a business resulting from loss of revenues.
Verizon’s Business Recovery and Continuity Services has been designed to ensure that operations continue while addressing the needs of a distributed workforce. We will also continue to add more services to the suite of services to ensure continuity of business.

20. BRCS Contacts Earl B. Cohen
Assistant Vice President and General Manager – Verizon Federal
Mark Burg
Senior Manager - Continuity of Operations – Verizon Federal
(202)
William Hummel
Director - Business Recovery & Continuity Services
(703)

21. Verizon Contingency Planning for COOP
“Ensuring Mission Critical Operations in an Unpredictable World…”

22. The Plans Compared
From “What Can We Learn From The September 11th Attacks? Are You Prepared In The Event Of A Disaster?” by Mark T. Edmead  This article was originally published in the Insight Newsletter of The Internet Security Conference ( and has been posted with permission by TISC, LLC.
Mr. Edmead reports on the business loss of computer hardware and software and communications.
Many business leaders are unaware of the amount of downtime their company experiences every week. This leads to a belief that I don’t need to consider Business Continuity in every I/T decision because I can’t justify the cost. However,we all experience issues related to Business Continuity today. On the average most business experience 2hrs of downtime per week.
SOURCE: NIST Special Publication , Contingency Planning Guide for Information Technology Systems, June 2002