Internet2 Abilene Network and Next Generation Optical Networking Steve Corbató Director, Backbone Network Infrastructure Access NovaForum May 28, 2002 This is a general overview presentation about Internet2. Internet2 is a consortium, led by US universities, which is recreating the partnership among academia, industry and government that fostered today’s Internet in its infancy.

This presentation Abilene Network today Optical networking evolution Next generation of Abilene Future national optical initiatives 11/29/2018

Networking hierarchy Internet2 networking is a fundamentally hierarchical and collaborative activity International networking Ad hoc  Global Terabit Research Network (GTRN) National backbones Regional networks GigaPoPs  advanced regional networks Campus networks Much activity now at the metropolitan and regional scales 11/29/2018

Abilene focus Goals Advanced service efforts Enabling innovative applications and advanced services not possible over the commercial Internet Backbone & regional infrastructure provides a vital substrate for the continuing culture of Internet advancement in the university/corporate research sector Advanced service efforts Multicast IPv6 QoS Measurement an open, collaborative approach Security 11/29/2018

Partnership approach The Abilene Network is a UCAID project done in partnership with Cisco Systems (routers, switches, and access) Juniper Networks (routers) Nortel Networks (SONET kit) Qwest Communications (SONET & DWDM circuits) Indiana University (network operations center) Internet2 Test & Evaluation Centers (ITECs) North Carolina and Ohio Partnerships are the foundation of how the Internet developed and they are also a part of the foundation of Internet2. 11/29/2018

11/29/2018

Abilene – May, 2002 IP-over-SONET backbone (OC-48c, 2.5 Gbps) 53 direct connections 4 OC-48c connections 1 Gigabit Ethernet trial 23 will connect via at least OC-12c (622 Mbps) by 1Q02 Number of ATM connections decreasing 215 participants – research universities & labs All 50 states, District of Columbia, & Puerto Rico 15 regional GigaPoPs support ~70% of participants Expanded access 50 sponsored participants New: Smithsonian Institution, Arecibo Radio Telescope 23 state education networks (SEGPs) 11/29/2018

Abilene international connectivity Transoceanic R&E bandwidths growing! GÉANT – 5 Gbps between Europe and New York City now Key international exchange points facilitated by Internet2 membership and the U.S. scientific community STARTAP & STAR LIGHT – Chicago (GigE) AMPATH – Miami (OC-3c  OC-12c) Pacific Wave – Seattle (GigE) MAN LAN - New York City (GigE/10GigE EP soon) CA*NET3/4: Seattle, Chicago, and New York CUDI: CENIC and Univ. of Texas at El Paso International transit service Collaboration with CA*NET3 and STARTAP 11/29/2018

Abilene international connectivity model Abilene is a GTRN partner Already peering with GTRN routers in New York City and Seattle Peering at major int’l EPs in U.S. encouraged Chicago: Star Light (migration from STAR TAP) Seattle: Pacific Wave Miami: AMPATH New York City: Manhattan Landing (MAN LAN) in progress Los Angeles (soon?) Direct BGP peering preferred via Layer-2 EP media or direct connection ATM support generally ends by Sept 2003 No new ATM peers 11/29/2018

Abilene International Peering 09 March 2002 Abilene International Peering STAR TAP/Star Light APAN/TransPAC, Ca*net3, CERN, CERnet, FASTnet, GEMnet, IUCC, KOREN/KREONET2, NORDUnet, RNP2, SURFnet, SingAREN, TAnet2 Pacific Wave AARNET, APAN/TransPAC, CA*net3, TANET2 Sacramento Los Angeles Washington NYCM BELNET, CA*net3, GEANT*, HEANET, JANET, NORDUnet SNVA GEMNET, SINET, SingAREN, WIDE LOSA UNINET OC3-OC12 San Diego (CALREN2) CUDI AMPATH REUNA, RNP2 RETINA, ANSP, (CRNet) El Paso (UACJ-UT El Paso) CUDI * ARNES, CARNET, CESnet, DFN, GRNET, RENATER, RESTENA, SWITCH, HUNGARNET, GARR-B, POL-34, RCST, RedIRIS

Packetized raw High Definition Television (HDTV) Raw HDTV/IP – single UDP flow of 1.5 Gbps Project of USC/ISIe, Tektronix, & U. of Wash (DARPA) 6 Jan 2002: Seattle to Washington DC via Abilene Single flow utilized 60% of backbone bandwidth 18 hours: no packets lost, 15 resequencing episodes End-to-end network performance (includes P/NW & MAX GigaPoPs) Loss: <0.8 ppb (90% c.l.) Reordering: 5 ppb Transcontinental 1-Gbps TCP requires loss of <30 ppb (1.5 KB frames) <1 ppm (9KB jumbo) 11/29/2018

End-to-End Performance: ‘High bandwidth is not enough’ Bulk TCP flows (> 10 Mbytes transfer) Current median flow rate over Abilene: 1.9 Mbps 11/29/2018

True End-to-End Performance requires a system approach User perception Application Operating system Host IP stack Host network card Local Area Network Campus backbone network Campus link to regional network/GigaPoP GigaPoP link to Internet2 national backbones International connections EYEBALL APPLICATION STACK JACK NETWORK . . . 11/29/2018

Jumbo frames supported Default Abilene backbone MTU has been increased from 4.5 to 9 kB We now can support 9 kB MTUs on a per connector basis Motivation: support for HPC computing and large TCP flows 11/29/2018

Abilene traffic characterization information Weekly detailed reports http://netflow.internet2.edu/weekly/ General analysis http://www.itec.oar.net/abilene-netflow/ 11/29/2018

Optical networking technology drivers Aggressive period of fiber construction on the national & metro scales in U.S. Now rapid industry contraction and capital crisis Many university campuses and regional GigaPoPs already use dark fiber Dense Wave Division Multiplexing (DWDM) Allows the provisioning of multiple channels (’s) over distinct wavelengths on the same fiber pair Fiber pair can carry 160 channels (1.6 Tbps!) Optical transport is the current focus Optical switching is still in the realm of experimental networks, but may be nearing practical application

DWDM technology primer DWDM fundamentally is an analog optical technology Combines multiple channels (2-160+ in number) over the same fiber pair Uses slightly displaced wavelengths (’s) of light Generally supports 2.5 or 10 Gbps channels Physical obstacles to long-distance transmission of light Attenuation Solved by amplification (OO) Wavelength dispersion Requires periodic signal regeneration – an electronic process (OEO) 11/29/2018

DWDM system components Base fiber pair (+ right of way & conduit) Multiplexing/demultiplexing terminals OEO equipment at each end of light path Output: SONET or Ethernet (10G/1G) framing Amplifiers All optical (OO) ~100 km spacing Regeneration Electrical (OEO) process – costly (~50% of capital) ~500 km spacing (with Long Haul - LH - DWDM) New technologies (ELH/ULH) can extend this distance Remote huts, operations & maintenance 11/29/2018

Telephony’s recent past (from an IP perspective in the U.S.) 11/29/2018

IP Networking (and telephony) in the not so distant future 11/29/2018

National optical networking options 1 – Provision incremental wavelengths Obtain 10-Gbps ’s as with SONET Exploit smaller incremental cost of additional ’s 1st  costs ~10x than subsequent ’s 2 – Build dim fiber facility Partner with a facilities-based provider Acquire 1-2 fiber pairs on a national scale Outsource operation of inter-city transmission equipment Needs lower-cost optical transmission equipment The classic ‘buy vs. build’ decision in Information Technology 11/29/2018

Future of Abilene Original UCAID/Qwest agreement amended on October 1, 2001 Extension of MoU for another 5 years – until October, 2006 Originally expired March, 2003 Upgrade of Abilene backbone to optical transport capability - ’s (unprotected) x4 increase in the core backbone bandwidth OC-48c SONET (2.5 Gbps) to 10-Gbps DWDM 11/29/2018

Key aspects of next generation Abilene backbone - I Native IPv6 Motivations Resolving IPv4 address exhaustion issues Preservation of the original End-to-End Architecture model p2p collaboration tools, reverse trend to CO-centrism International collaboration Router and host OS capabilities Run natively - concurrent with IPv4 Replicate multicast deployment strategy Close collaboration with Internet2 IPv6 Working Group on regional and campus v6 rollout Addressing architecture 11/29/2018

Key aspects of next generation Abilene backbone - II Network resiliency Abilene ’s will not be ring protected like SONET Increasing use of videoconferencing/VoIP impose tighter restoration requirements (<100 ms) Options: MPLS/TE fast reroute (initially) IP-based IGP fast convergence (preferable) 11/29/2018

Key aspects of next generation Abilene backbone - III New & differentiated measurement capabilities Significant factor in NGA rack design 4 dedicated servers at each nodes Additional provisions for future servers Local data collection to capture data at times of network instability Enhance active probing Now: latency & jitter, loss, reachability (Surveyor) Regular TCP/UDP throughput tests – ~1 Gbps Separate server for E2E performance beacon Enhance passive measurement Now: SNMP (NOC) & traffic matrix/type (Netflow) Routing (BGP & IGP) Optical splitter taps on backbone links at select location(s) 11/29/2018

Abilene Observatories Currently a program outline for better support of computer science research Influenced by discussions with NRLC members 1) Improved & accessible data archive Need coherent database design Unify & correlate 4 separate data types SNMP, active measurement data, routing, Netflow 2) Provision for direct network measurement and experimentation Resources reserved for two additional servers Power (DC), rack space (2RU), router uplink ports (GigE) Need process for identifying meritorious projects Need ‘rules of engagement’ (technical & policy) 11/29/2018

Next generation router selection Extensive router specification and test plan developed Team effort: UCAID staff, NOC, NC and Ohio ITECs Discussions with four router vendors Tests focused on next gen advanced services High performance TCP/IP throughput High performance multicast IPv6 functionality & throughput Classification for QoS and measurement 3 router platforms tested & commercial ISPs referenced  New Juniper T640 platform selected 11/29/2018

Abilene cost recovery model Connection (per connection) Annual fee OC-3 (155 Mbps)* $110,000 OC-12 (622 Mbps) $270,000 Gigabit Ethernet (1 Gbps)** $325,000 OC-48 (2.5 Gbps) $430,000 OC-192/10 GigE (10 Gbps) $490,000 Participation (per university) $20,000 11/29/2018

Abilene program changes 10-Gbps (OC-192c POS) connections  backhaul available wherever needed & possible Only required now for 1 of 4 OC-48c connections 3-year connectivity commitment required Gigabit and 10-Gigabit Ethernet Available when connector has dark fiber access into Abilene router node Backhaul not available ATM connection & peer support TAC recommended ending ATM support by fall 2003 Two major ATM-based GigaPoPs have migrated 2 of 3 NGIXes still are ATM-based NGIX-Chicago @ STAR LIGHT is now GigE Urging phased migration for connectors & peers 11/29/2018

Deployment timing Ongoing – Backbone router procurement Detailed deployment planning July – Rack assembly (Indiana Univ.) Aug/Sep – New rack deployment at all 11 nodes Fall – First Wave ’s commissioned Fall meeting demonstration events iGRID 2002 (Amsterdam) – late Sep. Internet2 Fall Member Meeting (Los Angeles) – late Oct. SC2002 (Baltimore) – mid Nov. Remaining ’s commissioned in 2003

Two leading national initiatives in the U.S. Next Generation Abilene Advanced Internet backbone connects entire campus networks of the research universities 10 Gbps nationally TeraGrid Distributed computing (Grid) backplane connects high performance computing (HPC) machine rooms Illinois: NCSA, Argonne California: SDSC, Caltech 4x10 Gbps: Chicago  Los Angeles Ongoing collaboration between both projects 11/29/2018

TeraGrid: A National Infrastructure For more information: www.teragrid.org

TeraGrid Architecture – 13.6 TF (Source: C. Catlett, ANL) 574p IA-32 Chiba City 256p HP X-Class 32 32 Caltech 32 Nodes 0.5 TF 0.4 TB Memory 86 TB disk Argonne 64 Nodes 1 TF 0.25 TB Memory 25 TB disk 32 32 128p Origin 24 32 128p HP V2500 32 HR Display & VR Facilities 24 8 8 5 92p IA-32 5 HPSS HPSS 24 4 Extreme Black Diamond OC-12 ESnet HSCC MREN/Abilene Starlight OC-48 Calren OC-48 OC-12 NTON OC-12 ATM Juniper M160 GbE SDSC 256 Nodes 4.1 TF, 2 TB Memory 225 TB disk NCSA 500 Nodes 8 TF, 4 TB Memory 240 TB disk Juniper M40 Juniper M40 vBNS Abilene Calren ESnet OC-12 OC-12 OC-3 vBNS Abilene MREN OC-12 2 2 OC-12 OC-3 Myrinet Clos Spine 8 4 HPSS 8 UniTree 2 Sun Starcat 4 Myrinet Clos Spine = 32x 1GbE 1024p IA-32 320p IA-64 1176p IBM SP Blue Horizon 16 = 64x Myrinet 14 4 = 32x Myrinet 1500p Origin Sun E10K = 32x FibreChannel = 8x FibreChannel 10 GbE 32 quad-processor McKinley Servers (128p @ 4GF, 8GB memory/server) 32 quad-processor McKinley Servers (128p @ 4GF, 12GB memory/server) Fibre Channel Switch 16 quad-processor McKinley Servers (64p @ 4GF, 8GB memory/server) Router or Switch/Router IA-32 nodes

Optical networking scaling factors 2 TeraGrid routing nodes 11 Next Generation Abilene routers 53 Abilene connectors 215 Abilene participants (univs & labs) But… 30-60 DWDM access nodes in leading viable carriers’ U.S. networks 11/29/2018

Regional optical fanout Next generation architecture: Regional & state based optical networking projects are critical Three-level hierarchy backbone, GigaPoPs/ARNs, campuses Leading examples CENIC ONI (California), I-WIRE (Illinois), Indiana (I-LIGHT) Collaboration with the Quilt GigaPoPs Regional Optical Networking project U.S. carrier DWDM access is now not nearly as widespread as with SONET circa 1998 30-60 cities for DWDM ~120 cities for SONET 11/29/2018

Optical network project differentiation Distance scale (km) Examples Equipment Metro < 60 UW(SEA), USC/ISI(LA) Dark fiber & end terminals State/ Regional < 500 I-WIRE (IL), CENIC ONI, I-LIGHT (IN) Add OO amplifiers Extended Regional/ National > 500 PLR, TeraGrid Abilene Add OEO regenerators & O&M $’s

Conclusions Backbone upgrade project underway Advanced service foci Partnership with Qwest extended thru 2006 Juniper T640 routers selected for backbone 10-Gbps backbone  deployment starts this fall Incremental, non-disruptive transition Advanced service foci Native, high-performance IPv6 Enhanced, differentiated measurement Network resiliency NSF TeraGrid and Extended Terascale Facility Complementary and collaborative relationship Continue to examine prospects for a fiber optical networking facility – National Light Rail

For more information Web: www.internet2.edu/abilene E-mail: abilene@internet2.edu 11/29/2018

www.internet2.edu