Abilene and Internet2 Engineering Update Guy Almes Terena Networking Conference 2002 Limerick, Ireland Guy Almes Terena Networking Conference 2002 Limerick, Ireland

6 May Outline Abilene Update Engineering Update »Multicast »IPv6 »QoS »End-to-End Measurements »Transport for Bulk Data Flows

6 May Internet2 Engineering Objectives Provide our universities with superlative networking: »Performance »Functionality »Understanding Make superlative networking strategic for university research and education

6 May Abilene Update Current 2.5 Gb/s Abilene Network Plans for 10 Gb/s Upgrade

6 May Abilene is a Partnership To build/operate Abilene, Internet2 partners with: »Cisco Systems (routers, switches, and access) »Juniper Networks (routers) »Nortel Networks (SONET kit) »Qwest Communications (circuits and collocation) »Indiana University (network operations center) »Internet2 Test & Evaluation Centers (ITECs) –North Carolina –Ohio

6 May Current Abilene Status IP-over-SONET backbone (2.5 Gb/s) »53 direct connections »4 2.5 Gb/s connections »1 Gigabit Ethernet trial »23 will connect via at least 622 Mb/s by 1Q02 »Number of ATM-based connections decreasing 215 participants: universities and labs »All 50 states, District of Columbia, and Puerto Rico »15 regional gigaPoPs support ~70% of participants Expanded access »50 sponsored participants »23 state education networks (SEGPs)

6 May Abilene international connectivity Transoceanic R&E bandwidths growing !? »GÉANT: 5 Gb/s between Europe and New York City Key international exchange points: »StarTap and StarLight: Chicago (GigE) »AmPath: Miami (155 Mb/s ATM) »Pacific Wave: Seattle (GigE) »MAN LAN: New York City (GigE/10GigE planned) »CA*net3/4: Seattle, Chicago, and New York »CUDI: CENIC and Univ Texas El Paso International transit service »Collaboration with CA*net3 and StarTap

Sacramento Los Angeles Washington Abilene International Peering STAR TAP/Star Light APAN/TransPAC, Ca*net3, CERN, CERnet, FASTnet, GEMnet, IUCC, KOREN/KREONET2, NORDUnet, RNP2, SURFnet, SingAREN, TAnet2 NYCM BELNET, CA*net3, GEANT*, HEANET, JANET, NORDUnet Pacific Wave AARNET, APAN/TransPAC, CA*net3, TANET2 SNVA GEMNET, SINET, SingAREN, WIDE LOSA UNINET AMPATH REUNA, RNP2 RETINA, ANSP, (CRNet) OC3-OC12 El Paso (UACJ-UT El Paso) CUDI San Diego (CALREN2) CUDI * ARNES, CARNET, CESnet, DFN, GRNET, RENATER, RESTENA, SWITCH, HUNGARNET, GARR-B, POL-34, RCST, RedIRIS 09 March 2002

6 May Packetized Raw HDTV Raw HDTV/IP: single 1.5 Gb/s UDP flow DARPA-funded project of USC/ISIe, Tektronix, and Univ Washington »6 Jan 2002: Seattle to Washington DC via Abilene »18 hours: no packets lost, 15 resequencing episodes »End-to-end network performance (includes P/NW & MAX) – Loss: <0.8 ppb (90% c.l.) – Reordering: 5 ppb »Transcontinental 1-Gb/s TCP requires loss of – <30 ppb (1.5 KB frames) – <1 ppm (9 KB jumbo)

6 May End-to-End Performance: High bandwidth is not enough Bulk TCP flows »Current median flow over Abilene: 1.9 Mb/s –95th percentile: 7.0 Mb/s

6 May Future of Abilene Internet2/Qwest agreement amended, and extended to Oct-06 Upgrade now underway to shift »from OC-48c (using Nortel OC-192 Sonet) »to 10-Gb/s lambda (unprotected) »x4 increase in core backbone bandwidth

6 May Next generation Abilene router selection Extensive router specification and testing Tests focused on next gen advanced services »High performance TCP/IP throughput »High performance multicast »IPv6 functionality and throughput »Classification for QoS and measurement 3 router platforms tested and commercial ISPs referenced Juniper T640 platform selected

6 May Deployment timing Ongoing: Backbone router procurement, detailed deployment planning July: Rack assembly (Indiana Univ.) Aug/Sep: New rack deployment Fall: First Wave lambdas commissioned Fall meeting demonstration events »Internet2 Fall Member Meeting (Los Angeles): late Oct. »SC2002 (Baltimore): mid Nov. 2003: Remaining lambdas commissioned

6 May Key next-gen Abilene emphases: I Native IPv6 »Motivations –Resolving IPv4 address exhaustion issues –Preserving original End-to-End Architecture model –International collaboration –Router and host OS capabilities »Run native IPv6, concurrent with IPv4 »Replicate Abilenes 1999 multicast deployment strategy »Close collaboration with Internet2 IPv6 Working Group

6 May Key next-gen Abilene emphases: II Network resiliency »Abilene lambdas will not be ring protected »Increasing use of videoconferencing/VoIP impose tighter restoration requirements (<100 ms) »Options: –MPLS/TE fast reroute (initially) –IP-based IGP fast convergence (preferable)

6 May Key next-gen Abilene emphases: III Deeper measurement capabilities »Significant factor in NGA rack design –4 dedicated servers at each nodes –Additional provisions for future servers –Local data collection to capture data at times of network instability »Enhance active probing –Now: Latency and jitter, loss, reachability (Surveyor) –Regular TCP/UDP throughput tests: ~1 Gbps Separate server for E2E performance beacon »Enhance passive measurement –Now: SNMP (NOC) and traffic matrix/type (Netflow) –Routing (BGP and IGP) –Optical splitter taps on backbone links at select location(s)

6 May Abilene Observatories Currently a sketch of a program for better support of computer science research 1) Improved and accessible data archive »Need coherent database design »Unify & correlate 4 separate data types –SNMP, active measurement data, routing, Netflow 2) Provision for direct network measurement and experimentation »Resources reserved for two additional servers –Power (DC), rack space (2RU), router uplink ports (GigE) »archive/measurement/experiment

6 May Abilene Upgrade Summary Backbone upgrade project underway »Partnership with Qwest extended through 2006 »Juniper T640 routers selected for backbone »10-Gb/s backbone lambda deployment starts this fall Advanced service foci »Native, high-performance IPv6 »Enhanced, differentiated measurement »Network resiliency Incremental, non-disruptive transition

6 May Multicast By 1998, »Routing protocols existed »Deployment of native IP multicast quite rare »Early MBone no longer scalable »Considered key to new conferencing and streaming applications Current native multicast support »PIM-Sparse, MBGP, and MSDP Emphases on »Deployment and support for operations »Applications »Working to make it scalable

6 May

6 May Current Multicast Emphases Pressing ahead on Deployment »What are the current inhibitors to progress? Applications / Content »Make it useful for your campuses »Explore the role of multicast in the future Internet Improve Scalability »Press deployment of SSM »Explore the role of SSM

6 May Could SSM be Enough? 'Classic' Multicast »Group has global significance »A user creates, joins, sends to g »Others can join, then send to and/or listen to g »MBGP, PIM-SM, MSDP triad Source Specific Multicast »Group has local significance »A user 's' creates, sends to »Others can subscribe to, then listen to »No need for MSDP (or allocation of values)

6 May Implications of SSM Simplify Multicast Routing / Addressing »No need for global class-D address allocation »No need for source discovery Complicates 'few-to-few' applications »Define all the members of the application-level group »Both a burden and an opportunity Allows better Security, Scalability Requires new version of IGMP

6 May IPv6 Clarify motivation for IPv6 »End-to-end transparency and global addressability »Supports application innovation, e.g., peer-to-peer Support deployment and engineering expertise on networks, especially on campus Anticipate need for first-class support »E.g., 10 Gb/s Abilene upgrade »E.g., Linux, Windows XP

6 May

6 May Current IPv6 Emphases IPv6 Training Workshops »About 8-10 workshops this year »First: in Los Angeles, hosted by CENIC, in February Get some IPv6 on each campus/gigaPoP Prepare for native peering »Abilene to gigaPoP »gigaPoP to campus »continue within campuses to key departmental LANs Explore applications, DNS, operational stability

6 May QoS Premium Service Retrospective »Inter-AS Premium Service proved too ambitious »Too great a demand on all routers to be able to police and to shape »Too great a demand on ability of peering networks to coordinate Implicit claims »Over-provisioning plus removal of non-congestive loss »Adaptive applications

6 May Non-Elevated Services Scavenger »Less than best effort »Easy to deploy »Applications: –Massive file transfers –Marking non-performance-sensitive applications Alternative Best Efforts »Active area of research »Avoid gaming by users, while avoiding need for policing

6 May Localized Elevated Services Nature of Congestion as a threat »Less on national/international backbones »More at hard-to-upgrade local/metro networks Decentralized Experimentation Coordinates with some Scavenger deployment Active area of study by the working group

6 May The Current Situation Our universities have access to an infrastructure of considerable capacity »examples of multi-hour 1.6 Gb/s flows with no loss and very little reordering End-to-end performance varies widely »but 40 Mb/s flows not always predictable »users don't know what their expectations should be A well-known mismatch

6 May What are our Aspirations? Candidate Answer #1: Switched 100BaseT + Well-provisioned Internet2 networking at 80 Mb/s But user expectations and experiences vary widely

6 May What are our Aspirations? Candidate Answer #2: Lower user expectations and minimize complaining phone calls There is a certain appeal I suppose...

6 May What are our Aspirations? Candidate Answer #3: Raise expectations, encourage aggressive use, deliver on performance/functionality to key constituencies. Not the easy way, but necessary for success

6 May Threats to End to End Performance Fiber problems »dirty fiber »dim lighting »'not quite right' connectors

6 May Threats to End to End Performance Fiber problems Switches »horsepower »full vs half-duplex »head-of-line blocking

6 May Threats to End to End Performance Fiber problems Switches Inadvertently stingy provisioning »mostly communication »happens also in international settings

6 May Threats to End to End Performance Fiber problems Switches Inadvertently stingy provisioning Wrong Routing »asymmetric »best use of Internet2 »distance

6 May Threats to End to End Performance Fiber problems Switches Inadvertently stingy provisioning Wrong Routing Host issues »NIC »OS / TCP stack »CPU

6 May Perverse Result 'Users' think the network is congested or that the Internet2 infrastructure cannot help them 'Planners' think the network is underutilized, no further investment needed, or that users don't need high performance networks

6 May Transport Protocol Issues Improved TCP Implementation »Web100 Project »SACK, Window Scaling »ECN But, still subject to fundamental limits »Mathis et al. Theoretical result »TCP-throughput = (C x MTU) / (RTT x sqrt(loss)) »Prospects for raising MTU, reducing RTT, loss

6 May Need to supplement TCP Bulk Data Transfers »Absolute need to include congestion control »But include flow control in a more aggressive way Related Ideas »Transport-level gateways? »Known-Capacity pipes vs Groping for available Capacity