Developing a Hybrid Optical/Packet Infrastructure (HOPI) in the U.S. Steven Corbató Director, Backbone Network Infrastructure Industry Strategy Council Detroit Metro Airport 21 November 2003 This is a general overview presentation about Internet2. Internet2 is a consortium, led by US universities, which is recreating the partnership among academia, industry and government that fostered today’s Internet in its infancy.

Collaborative effort Guy Almes Heather Boyles Steve Corbató Chris Heermann Cheryl Munn-Fremon Rick Summerhill Christian Todorov Doug Van Houweling Steven Wallace (Indiana University) 4/23/2019

Outline Assembling the vital ingredients High-performance national IP network - Abilene Regional Optical Networks (RONs) National optical capabilities - NLR Hybrid networking - next steps Plans for the NLR  dedicated to Internet2 Steps towards developing a Hybrid Optical Packet Infrastructure (HOPI) … 4/23/2019

Abilene Network – second generation 4/23/2019

Abilene Focus Areas – 2003-2004 High performance, native advanced services Multicast IPv6 Large End-to-End Flows Abilene Observatory Supporting Network Research Community Open Measurement & Experimentation Platform Dedicated Capability Experimentation Limited MPLS service Network Security Role of the REN-ISAC Advanced Restoration Techniques Potential backup arrangement with ESnet 4/23/2019

SC2003 Bandwidth Challenge 4/23/2019

One underlying hypothesis The fundamental nature of regional networking is changing The GigaPoP model based on provisioned, high-capacity services steadily is being replaced – on the metro and regional scales A model of facility-based networking built with owned assets – Regional Optical Networks (RONs) – has emerged Notably, the importance of regional networks in the traditional three-level hierarchy of U.S. advanced networking for R&E is not diminshed 4/23/2019

Distance scales for U.S. optical networking Distance scale (km) Examples Equipment Metro < 60 UWash USC/ISI(LA), MAX(DC/MD/VA) Dark fiber & end terminals State/ Regional < 500 I-WIRE (IL), I-LIGHT (IN), CENIC ONI Add OO Amplifiers (or optical TDM) Extended Regional/ National > 500 TeraGrid 2nd Gen Abilene, NLR Add OEO regenerators & O&M $’s 4/23/2019

Leading & Emerging Regional Optical Initiatives California (CALREN) Colorado (FRGP/BRAN) Connecticut (Connecticut Education Network) Florida (Florida LambdaRail) Indiana (I-LIGHT) Illinois (I-WIRE) Maryland, D.C. & northern Virginia (MAX) Michigan Minnesota New York + New England region (NEREN) North Carolina (NC LambdaRail) Ohio (Third Frontier Network) Oregon Rhode Island (OSHEAN) SURA Crossroads (southeastern U.S.) Texas Utah Wisconsin 4/23/2019

FiberCo Designed to support optical initiatives Regional National Not an operational entity – supporting project Will not light any fiber Fiber options Holding company for any future initiatives Assignment vehicle Regional initiatives National initiatives (e.g., NLR) Internet2 took responsibility for LLC formation Idea was spin-off from NLR formation discussions National R&E Fiber Company incorporated in Delaware First acquisition of dark fiber for FiberCo through Level 3 Communication – 3/21/2003 4/23/2019

Available fiber topology

NLR Current Members and Associates CENIC Pacific Northwest Gigapop Pittsburgh SC Duke Univ./NCLR MATP/Va. Tech Cisco Systems Internet2 Florida LambdaRail Georgia Tech CIC Pending: Texas University Consortium 4/23/2019

NLR distinguishing features - I Largest higher-ed owned/managed optical networking & research facility in the world ~10,000 route-miles of dark fiber Four 10-Gbps ’s provisioned at outset One allocated to Internet2 First & foremost, an experimental platform for research Optical, switching & IP capabilities (layers 1, 2 & 3) Research committee integral in NLR governance Advance reservation of  capacity for research Experimental support center 4/23/2019

NLR distinguishing features - II Use of high speed Ethernet for WAN transport 1O Gigabit Ethernet LAN PHY is primary interface Traditional OC-192 SONET available, too ‘Sparse backbone’ topology Each participant/node typically commits $5M Concurrent responsibility for developing optical networking capabilities and sustaining performance in nodal region 4/23/2019

NLR’s ‘Virtuous Circles’ and the Vital Role of Dark Fiber 4/23/2019

Next steps for Internet2 and U.S. R&E optical networking development

Overview of U.S. advanced networking environment Despite last two years of debate, an incredible amount has been achieved: A novel, national, facilities-based optical network - NLR – has gained critical mass and is being deployed. A major corporate partner - Cisco - has made a substantial reinvestment and has encouraged us to reengage the researchers. USA Waves is on the verge of a major fiber donation and has given us a conceptual model for carrier-based, incremental pricing for ’s Over 10,000 miles of dark fiber have been acquired by the community for national and regional optical networks by CENIC, FiberCo, and others. The Abilene Network has been upgraded to a 10-Gbps backbone and supports the research university community through initiatives such as IPv6 deployment and the Observatory. 4/23/2019

Global Lambda Integration Facility (GLIF) Ongoing effort to build dedicated lightpaths (circuit switched sub- ’s) between HPC resources internationally StarLight (Chicago), CA*Net (Canada), SURFnet (The Netherlands) are established leaders NORDUnet (Scandinavia), Czech Republic, Japan active Internet2 now proposes to enter this effort Now placing an optical Ethernet/SONET multiplexer (Cisco 15454) in the MAN LAN facility in NYC Planning to move the 10-Gbps IEEAF/Tyco  (NYC-Amsterdam/SURFnet) from Abilene NYC router to MAN LAN optical TDM Will interconnect with CA*Net GLIF effort in NYC Also will interconnect with Abilene and MAN LAN Ethernet-based int’l R&E exchange 4/23/2019

The current state of the GLIF Reykjavik, August 2003 4/23/2019

Near-term options for provisioning dedicated capabilities in the U.S. Alternatives to the IP common bearer service model… Abilene MPLS tunnels over 10-Gbps IPv4/v6 backbone NLR Gigabit Ethernet service over one 10-Gbps  I2-over-NLR (HOPI) time-multiplexed channels over one 10-Gbps  individual 10-Gbps ’s FiberCo and other sources dark fiber 4/23/2019

Towards a hybrid optical/packet infrastructure Need to converge the two worlds of packet (Abilene) and optical circuit (GLIF) networking Concern about IP packet network scaling (lack of >10 Gbps transport) and drawbacks (security, large data flows) Widely recognized that the current GLIF optical networking model does not scale beyond a limited number of sites However, dedicated capabilities allow a vehicle for advanced testing (transport technologies beyond TCP) and very high-end requirements Potential end-states: Hybrid model – with centrally and perhaps ‘user’ driven allocation of dedicated capabilities Impact of hybrid experience on packet network design 4/23/2019

HOPI: raw materials  new service models Abilene high-performance IP network Capabilities for MPLS tunnels 10-Gbps  over full NLR footprint Details 5-year commitment Likely 10 GigE framing (in lieu of OC192c SONET) Expect some type of ‘TDM’ infrastructure to be provisioned by Internet2 in collaboration with NLR Additional capital investment required Tie points for international collaboration MAN LAN (NYC) and IEEAF  (NYC-Amsterdam) StarLight collaboration (Chicago) Expect similar capabilities to emerge in Seattle (Pacific Wave) 4/23/2019

One view of the NLR – Internet2 relationship 4/23/2019

Internet2 Today Motivate Enable Applications Middleware Services End-to-end Performance Motivate Security Middleware Enable The next slide uses animation to build to this much more complex picture with many more elements. This is what the community is doing today and it is what the Internet2 organization is helping to support. There was some discussion of including more granularity in the visual—like IPv6 and multicast in engineering, or “spot” or “general” instances of application. My feeling is that this slide is busy already and we might want to just make sure the script refers to these specific things. IPv6, FasTCP, Multicast, [need to have specific examples in each of the areas: Enabling new work mainly for subsets of our membership: Haystack observatory New world symphony masters classes Telemedicine instructions Reiterate commitment to shared applications, building the scaffolding, need to acknowledge the difficulty we’ve had in getting folks to converge around shared and common applications We expect Applications Strategy Council will provide more advice in this regard. Our view of engineering has changed: end-to-end IPv6 Multicast Services Networks 4/23/2019

Abilene-NLR Topology Mileage indicates distance between Abilene/Qwest PoPs and NLR/Level3 PoPs, where they are located in the same area These are the three primary goals of Internet2. 4/23/2019

HOPI Network Architecture Characteristics Requires Inter-AS involvement/integration - includes Campus, GigaPoP, RONs, and Backbone Granular network hierarchy User:Lab:Department:Campus:GigaPoP:RON:Backbone Bandwidth can be shared and/or dedicated Shared - IP routing Dedicated - Wavelength, TDM Shared and dedicated - MPLS Shared because MPLS uses the same physical resource as IP Dedicated because it provides a connection oriented service Dark fiber enables wavelength interconnections and the ability to control data rate independent of carrier New operational model - requires experience using optronics in long-haul and regional areas These are the three primary goals of Internet2. 4/23/2019

HOPI - next steps Convening a HOPI design team Focused on both architecture and implementation Leverage the set of available ‘raw materials’ Objective: Short term trials  scalable long-term hybrid architecture Assembling a team of advanced practicioners and experienced standards body participants (IETF, OIF) Academic and industrial representations Target for deliverables: early spring 2004 Can be viewed as a prelude to the process for the 3rd generation Internet2 network architecture 2005-2006 time frame for implementation In the interim, we will observe the SURFnet process carefully 4/23/2019

4/23/2019