I C S Integrated Communication Services The Network During the Crisis - A New York City Perspective Doug Carlson, Director of Network Services Tim Lance, President and Board Chair, NYSERNet

N ew York University

N  Largest private university in US  Located in Greenwich Village area of New York City  Concentrated North of Houston (NoHo) up through 14 th Street  School of Medicine and NYU Hospital near 21 st Street  Residence Halls extend down to blocks away from Ground Zero ew York University

Abilene CA*Net Buffalo Rochester Syracuse Albany New York Topology of the NYSERNet Network. OC-3 OC-12 legend NYSERNet 2000 PoP DS-3 Gateway Network Abilene vBNS+

R esearch Network  Connected Institutions (R&E Network)  Amer. Museum of Natural History  Columbia University  Cornell University  Hauptman-Woodward Institute *  Marist College *  New York University  Pace University *  Rensselaer Polytechnic Institute  Rochester Institute of Technology  Rockefeller University  SUNY Albany  SUNY Binghamton  SUNY Buffalo  SUNY Stony Brook  Syracuse University  University of Rochester  Weill Medical College * Approved for Funding.

Manhattan Project Participants – Geographic Map

B efore September 11th Photograph by Doug Carlson – 7/2001

I mpact of September 11 th Photograph by Doug Carlson – 9/23/2001

Relationship between 7 World Trade Center and 140 West Street prior to September 11 th.

60 Hudson Street 140 West Street World Trade Center Images courtesy Image of Manhattan from IKONOS Satellite taken Summer 2000.

140 West Street 60 Hudson Street World Trade Center Images courtesy Image of Manhattan from IKONOS Satellite taken shortly after the attack on the World Trade Center.

World Trade Center complex showing relationship between 7 World Trade Center and 140 West Street.

Exterior view of 140 West Street showing debris and damage from collapse of 7 World Trade Center. Photographs Copyright 2001 Verizon Communications. All rights reserved.

An interior view of 140 West Street showing damage from collapse of 7 World Trade Center. Photographs Copyright 2001 Verizon Communications. All rights reserved.

N  As a result of the attack…  CO at 140 West Street severely damaged.  NYSERNet OC-48 SONET ring operational but no longer a ring.  NYSERNet R&E Network operational.  NYSERNet POP at 60 Hudson Street operational.  Abilene connections operational.  vBNS+ connection down.  R&E service for SUNY Stony Brook and AMNH is down. etwork Status - NYSERNet

N  As a result of the attack…  CO at 140 West Street severely damaged.  AppliedTheory equipment largely undamaged.  AppliedTheory Network operational.  AppliedTheory POP at 60 Hudson Street operational.  Customers connecting to 60 Hudson operational.  Customers connecting at 140 West are down.  Customers connecting via Garden City, Deer Park, and White Plains are down. etwork Status - Commodity

R Restoring Service to Mt. Sinai & School of Medicine Prepared by Doug Carlson, New York University estoring Service Photo of 140 West Street. Copyright 2001 Verizon Communications. All rights reserved.

R estoring Service  Post Attack  Phone systems immediately get overloaded  Lines by all payphones in NYU area  Manhattan phone and cell systems overwhelmed  Access to Lower Manhattan impossible  Pedestrian access limited below 14 th Steet (Union Square)  Pedestrians required to have reason to go beyond checkpoint and be able to show ID  Some students and others left ID behind

R estoring Service  Post Attack  Initially, no traffic below 14 th Street unless emergency or government vehicle  NYU worked with Mayor’s office to get some critical deliveries through to NYU locations  Most businesses and restaurants closed below 14th Street  NYU Administration establishes a Crisis Command Center  Initially, did not have sufficient phone or data service

R estoring Service  Post Attack  Campus phone system becomes overloaded for a period of time (no dial-tone)  Internet connections provide only reliable information links to family, friends, co-workers:   Web  Instant Messaging  VoIP (Cisco) phones used to communicate with staff unable to come into Manhattan and unable to get through on traditional telephone/cell systems

R estoring Service  Post Attack  Information Technology Services (ITS) takes point in keeping NYU community informed via Web, mass-mailings via , voice mail announcements and Help Desk services.  Mt. Sinai Hospital traffic routed through NYU.  Phone banks set up to allow students to call home for free.  Temporary shelter set up for students and staff who were evacuated.

R estoring Service  Post Attack  Evacuated students and staff are relocated to hotels around the city:  Students given some money for essentials.  Arrangements are made to get new sets of books for students.  Loans of laptops and desktop computers offered.  Preparations made to increase dial-in capacity via ISP. (Eventually not needed due to students returning to their residence halls.)

R Columbia, IP Telephony & The New York Academy of Medicine. Prepared by Alan Crosswell, Columbia University estoring Service Photo of 140 West Street and WTC 7 courtesy of the Westchester Emergency Communications Association,

R estoring Service  Early Morning, September 11th  Dealing with start of semester file server performance issues after a failed Summer file server upgrade project.  Trying to get a handle on filtering Code Red with our Catalyst 6509’s.  Turning off many user network ports for compromised hosts day in and day out.

R estoring Service  Post Attack, September 11th  Trying to find out where all missing staff are.  Discover phone trunks are overloaded.  Send to friends asking them to call our families and let them know we are OK.

R estoring Service  Post Attack, September 11th  Administration establishes response team. They request that we:  Set up large lecture halls with CNN; Students are crowding around TV monitors in lounge areas.  Establish net2phone-like functionality so students can phone home.  Send a mass to the Columbia community from the President.

R estoring Service  Getting CNN Out There, September 11th  Start planning to drag an IPTV encoder over to a cable box in a dorm.  Staff drag some coax from a lounge TV monitor to a large lecture hall.  Read on wg-multicast list that Northwestern has started multicasting CNN, but find our multicast connectivity is broken.

R estoring Service  Getting Phones Up, September 11th  One staff member tests net2phone….  Send to Videnet & others + CS SIP group requesting use of PSTN gateways to route around local congestion. ~11am.  End up configuring H.323 Polycoms via UNC Chapel Hill, and  Cisco IP phones via Penn State, and  SIP phones via 4 sites (Yale, Dynamicsoft, Nortel, Clarent).

R estoring Service  Getting Phones Up, September 11th  Net2phone works but it’s not easy & requires a credit card….  Polycoms work but also not so easy….  CS puts 4 SIP phones in CS conference room. 2:20pm  We put 2 Cisco IP phones via Penn State in student center lobby. Just dial 8 and get a PSU dial tone and call home! 4:00pm.

R estoring Service  Getting Phones Up, September 11th  By the time we had the IP phones in place trunk congestion had declined.  We had many many other Videnet sites offer their H.323 – H.320 PSTN gateways.  Thanks to all of you!

R estoring Service  September 11th Onward  One of our dialup modem pools is out (and still out). A major expensive carrier.  Our free T1’s from a small carrier are up on our main pool.  Weekly volume increased by 40% over prior week.  Daily volume on Thursday 9/13 was 800,000 messages: double last year’s.

R estoring Service  September 11th Onward  Filtering SIRCAM, nimda, Code Red, WTC viruses.  Turning off lots of ports of compromised hosts and attempting to deal with helping people reformat and reinstall their systems.  Established outbound P2P traffic shaping.

R estoring Service  September 18 th  After a week, NYAM’s T1 service via 140 West St is still down and low priority for restoration.  Set up an b link across Central Park and turn it up Wednesday 9/19.

R Restoring Service to Rockefeller, Weill, & HSS. Prepared by Armand Gazes, The Rockefeller University estoring Service Photo of new cable being placed on West Street. Copyright 2001 Verizon Communications. All rights reserved.

Hospital for Special Surgery The Rockefeller University Weill Medical Center of CU AppliedTheory AppliedTheory Router Internet NYSERNet Switch NYSERNetInternet2 Network Connectivity Prior to September 11 th. Rockefeller Router AppliedTheory Router Rockefeller Commodity Path R&E Path key

Hospital for Special Surgery The Rockefeller University Weill Medical Center of CU AppliedTheory AppliedTheory Router Internet NYSERNet Switch NYSERNetInternet2 Network Connectivity Subsequent to September 11 th. Rockefeller Router AppliedTheory Router Rockefeller Commodity Path R&E Path key

Hospital for Special Surgery The Rockefeller University Weill Medical Center of CU AppliedTheory AppliedTheory Router Internet NYSERNet Switch NYSERNetInternet2 Commodity Service Restored. NYSERNet Router AppliedTheory Router Buffalo Rockefeller Router AppliedTheory Router Rockefeller Commodity Path R&E Path key

R  Restoring commodity service…  To Rockefeller University, Weill Medical Center, and the Hospital for Special Surgery.  Service restored through Rockefeller’s R&E connection and emergency installation of a jumper between NYSERNet and AppliedTheory in Buffalo.  Key - cooperation among campus personnel, commercial service providers and NYSERNet.  Service restored by Wednesday afternoon, the 12 th. estoring Service

R Restoring Commodity and R&E Services. Tim Lance, NYSERNet. New York Stock Exchange – Photo by Doug Carlson

AppliedTheory NYSERNet AppliedTheory Long Island NYSERNet Metro Ring & AppliedTheory Prior to September 11 th. NYSERNet Verizon 60 Hudson St. Applied Theory Verizon 140 West St. Verizon Broad St. NYSERNet Metropolitan OC-48 SONET Ring

AppliedTheoryNYSERNet AppliedTheory Long Island NYSERNet Metro Ring & AppliedTheory After September 11 th. NYSERNet 60 Hudson St. Applied Thory Verizon NYSERNet Metropolitan OC-48 SONET Ring Verizon Broad St. Verizon Applied Theory 140 West St.

AppliedTheoryNYSERNet AppliedTheory Long Island Service Restored Utilizing NYSERNet SONET Ring. NYSERNet 60 Hudson St. Applied Thory Verizon NYSERNet Metropolitan OC-48 SONET Ring Verizon Broad St. Verizon Applied Theory 140 West St.

N  Restoring commodity service…  Polytechnic University - commodity service restored on 10/1.  Long Island – commodity traffic still transiting NYSERNet SONET Ring.  White Plains – connected to Garden City and then onto NYSERNet Ring.  Rockefeller Solution - evolving into standby service for all customers having both NYSERNet R&E service and AppliedTheory commodity service.  West Street Service – a few commodity T1’s still down. etwork Status - Commodity

N etwork Status – NYSERNet  Restoring R&E service…  American Museum of Natural History – service restored on 9/20.  SUNY Stony Brook – service restored on 9/25.  vBNS+ – service restored almost immediately.

O  New York University extended service to Mt.Sinai and School of Medicine.  Hofstra University offered NYSERNet bandwidth on its OC3 if it could be utilized to serve any impacted institutions.  City University of New York hosted the New York City Board of Education Web site to ensure that parents could remain informed on the status of their children’s schools.  American Museum of Natural History staff offered to serve as remote hands for any institution needing onsite service.  Columbia University provided New York Academy of Medicine with access via wireless service.  NYSERNet offered to allow AppliedTheory to utilize its Manhattan SONET ring to restore commodity service, if feasible. ther Stories

C onclusions

C  Lessons learned…  The value of networks in sustaining communications.  The survivability of the technology.  How much can be accomplished quickly when the urgency is clear.  The value in multiple independent paths to the Internet.  The value of government investment in science and the often unanticipated nature of the return on that investment.  The value of developing an Emergency Response Plan onclusions

C hallenges Ahead  Networks are necessities, not luxuries…  Networks are now mission critical.  Diverse capabilities must back each other up.  Our constituents are flexible, we must be too.  Merging technologies can lead to redundancy.

C hallenges Ahead  Industry Contraction and Consolidation  Mergers, acquisitions, business failures.  Continued geographic concentration.  Reduced capital investment.

C hallenges Ahead  Shifting Public Policy  H.R. 3162, The Uniting and Strengthening America by Providing Appropriate Tools Required to Intercept and Obstruct Terrorism (USA PATRIOT) Act of  Lack of controlling authority viewed as a deficiency.  House Science Committee considering Cyber security bill.  Senate Commerce Committee considering expanded government role.

C hallenges Ahead  Prioritization of Limited Resources  Attracting, retaining and training staff.  Funding.  Don’t chase the bubble.  Base next generation design on controlling the transport medium, with physical and logical redundancy and ever more intelligence at the edge.

R esponding Brooklyn New Jersey Bronx Redundant Fibers Fiber Trunk Loop 1A Redundant Fibers New School Arts4All Cooper Union NYU (ACF) NYU (MC) CUNY (CIS) NYPL AMNH Rockefeller MSKCC (HQ) MSKCC (DC) Columbia NYPH (CPC) Fordham U. Yeshiva U. Polytechnic 1 15C 15B Fordham U. Tarrytown Fordham U. Marymount Tarrytown 14C 16A NYPH (AP) Yeshiva (Uptown) CUNY (City College) NYSBC NYPH (WMC) 11B 11D MSKCC (64th) 14E NYPH (E61) NYPL (PA) Fordham (LC) 13A MSKCC (53rd) 11A 14A NYPL (Science/DC) 16C Yeshiva (Midtown-1) 16B Yeshiva (Law) 8A7C 16D NYPH (PFS) Yeshiva (Midtown-2) Upstate NY 11C 14B 15A 8B 7A D 7B MSKCC (PCB) NYPL (Annex) Columbia Controller CUNY (Grad. Center) NYPH (DC) MSKCC (Admin) NYSERNet POP NYSERNet’s Manhattan Project. Developing a Dark Fiber Network in the Manhattan area.

M anhattan Project  Advancing Technology for our Members  NYSERNet as catalyst, project manager, business manager.  Seeking to serve needs of the entire community for: redundancy, security, flexibility, performance, cost- control.  Developing a model for future projects.  Urgency has risen since September 11 th.

M anhattan Project  Goals  Implement a network that provides economical, high- performance access to:  the NYSERNet network,  commodity Internet providers,  the public switched telephone network, plus  the ability to implement private intra- and inter- institutional networks, and  unlimited potential for performance upgrades.

M anhattan Project  Participating Institutions (initial list)  Amer. Museum of Natural History  Arts4All  Columbia University  Cooper Union  City University of New York  Fordham University  Memorial Sloan Kettering  New School University  New York Polytechnic Institute  New York Presbyterian Hospital  New York Public Library  New York University  The Rockefeller University  Weill Medical College  Yeshiva University

M anhattan Project  Project Objectives  Survey potential participants about their requirements.  Create network design based upon the survey.  Preliminary costs determination based upon design.  Review costs with participants and secure commitment.  Network and colocation construction commences.  First participant online

A ctions at NYU  In negotiations with vendor for second commodity Internet link with diverse routing within Manhattan  Seeking funding for distributing infrastructure in multiple locations (e.g., network core, key servers, etc.)  Working with FEMA to determine appropriate infrastructure upgrades  Risk assessment and response planning

Washington Square Park – 9/23/2001 Photo by Doug Carlson

NY Harbor – 2/2/2002 Photo by Doug Carlson