1. Internet2 Network of the Future
Steve Corbató
Director, Backbone Network Infrastructure
CUDI
Tampico
25 de abril 2002
This is a general overview presentation about Internet2. Internet2 is a consortium, led by US universities, which is recreating the partnership among academia, industry and government that fostered today’s Internet in its infancy.

2. This presentation Abilene Network today
Emergence and evolution of optical networking
Next phase of Abilene
11/30/2018

3. Networking hierarchy
Internet2 networking is a fundamentally hierarchical and collaborative activity
International networking
Ad hoc  Global Terabit Research Network (GTRN)
National backbones
Regional networks
GigaPoPs  advanced regional networks
Campus networks
Much activity now at the metropolitan and regional scales
11/30/2018

4. Abilene background & milestones
Abilene is a UCAID project in partnership with
Qwest Communications (SONET & DWDM service)
Cisco Systems (routers, switches & access)
Juniper Networks (routers)
Nortel Networks (SONET kit)
Indiana University (network operations)
ITECs in North Carolina and Ohio (test and evaluation)
Timeline
Apr 1998: Project announced at White House
Jan 1999: Production status for network
Oct 1999: IP version of HDTV (215 Mbps) over Abilene
Nov 2001: Raw HDTV/IP (1.5 Gbps) over Abilene
Partnerships are the foundation of how the Internet developed and they are also a part of the foundation of Internet2.
11/30/2018

5. Abilene – April, 2002
IP-over-SONET backbone (OC-48c, 2.5 Gbps) 53 direct connections
4 OC-48c connections
1 Gigabit Ethernet trial
23 will connect via at least OC-12c (622 Mbps) by 1Q02
Number of ATM connections decreasing
211 participants – research universities & labs
All 50 states, District of Columbia, & Puerto Rico
15 regional GigaPoPs support ~70% of participants
Expanded access
46 sponsored participants
21 state education networks (SEGPs)
11/30/2018

6. 11/30/2018

7. Abilene international connectivity
Transoceanic R&E bandwidths growing!
GÉANT – 5 Gbps between Europe and New York City
Key international exchange points facilitated by Internet2 membership and the U.S. scientific community
STARTAP & STAR LIGHT – Chicago (GigE)
AMPATH – Miami (OC-3c  OC-12c)
Pacific Wave – Seattle (GigE)
MAN LAN - New York City – GigE/10GigE EP soon
CA*NET3: Seattle, Chicago, and New York
CUDI: CENIC and Univ. of Texas at El Paso
International transit service
Collaboration with CA*NET3 and STARTAP
11/30/2018

8. Abilene International Peering
09 March 2002
Abilene International Peering
STAR TAP/Star Light
APAN/TransPAC, Ca*net3, CERN, CERnet, FASTnet, GEMnet, IUCC, KOREN/KREONET2, NORDUnet, RNP2, SURFnet, SingAREN, TAnet2
Pacific Wave
AARNET, APAN/TransPAC, CA*net3, TANET2
Sacramento
Los Angeles
Washington
NYCM
BELNET, CA*net3,
GEANT*,
HEANET, JANET,
NORDUnet
SNVA
GEMNET, SINET, SingAREN, WIDE
LOSA
UNINET
OC3-OC12
San Diego (CALREN2)
CUDI
AMPATH
REUNA, RNP2 RETINA, ANSP, (CRNet)
El Paso (UACJ-UT El Paso)
CUDI
* ARNES, CARNET, CESnet, DFN, GRNET, RENATER, RESTENA, SWITCH, HUNGARNET, GARR-B, POL-34, RCST, RedIRIS

9. Packetized raw High Definition Television (HDTV)
Raw HDTV/IP – single UDP flow of 1.5 Gbps
Project of USC/ISIe, Tektronix, & UWash (DARPA support)
6 Jan 2002: Seattle to Washington DC via Abilene
Single flow consumed 60% of backbone bandwidth
18 hours: no packets lost, 15 resequencing episodes
End-to-end network performance (includes P/NW & MAX GigaPoPs)
Loss: <0.8 ppb (90% c.l.)
Reordering: 5 ppb
Transcontinental 1-Gbps TCP
requires loss of
<30 ppb (1.5 KB frames)
<1 ppm (9KB jumbo)
11/30/2018

10. End-to-End Performance: ‘High bandwidth is not enough’
Bulk TCP flows (> 10 Mbytes transfer)
Current median flow rate over Abilene: 1.9 Mbps
11/30/2018

11. End-to-End Performance Initiative
To enable the researchers, faculty, students and staff who use high performance networks to obtain optimal performance from the current infrastructure on a consistent basis.
Raw Connectivity
Applications Performance
11/30/2018

12. True End-to-End Performance requires a system approach
User perception
Application
Operating system
Host IP stack
Host network card
Local Area Network
Campus backbone network
Campus link to regional network/GigaPoP
GigaPoP link to Internet2 national backbones
International connections
EYEBALL
APPLICATION
STACK
JACK
NETWORK
. . .
11/30/2018

13. Optical networking technology drivers
Aggressive period of fiber construction on the national & metro scales in U.S.
Many university campuses and regional GigaPoPs with dark fiber
Dense Wave Division Multiplexing (DWDM)
Allows the provisioning of multiple channels (’s) over distinct wavelengths on the same fiber pair
Fiber pair can carry 160 channels (1.6 Tbps!)
Optical transport is the current focus
Optical switching is still in the realm of experimental networks, but may be nearing practical application
11/30/2018

14. DWDM technology primer
DWDM fundamentally is an analog optical technology
Combines multiple channels ( in number) over the same fiber pair
Uses slightly displaced wavelengths (’s) of light
Generally supports 2.5 or 10 Gbps channels
Physical obstacles to long-distance transmission of light
Attenuation
Solved by amplification (OO)
Wavelength dispersion
Requires periodic signal regeneration – an electronic process (OEO)
11/30/2018

15. DWDM system components
Fiber pair
Multiplexing/demultiplexing terminals
OEO equipment at each end of light path
Output: SONET or Ethernet (10G/1G) framing
Amplifiers
All optical (OO)
~100 km spacing
Regeneration
Electrical (OEO) process – costly (~50% of capital)
~500 km spacing (with Long Haul - LH - DWDM)
New technologies can lengthen this distance
Remote huts, operations & maintenance
11/30/2018

16. Telephony’s recent past (from an IP perspective in the U.S.)
11/30/2018

17. IP Networking (and telephony) in the not so distant future
11/30/2018

18. National optical networking options
1 – Provision incremental wavelengths
Obtain 10-Gbps ’s as with SONET
Exploit smaller incremental cost of additional ’s
1st  costs ~10x than subsequent ’s
2 – Build dim fiber facility
Partner with a facilities-based provider
Acquire 1-2 fiber pairs on a national scale
Outsource operation of inter-city transmission equipment
Needs lower-cost optical transmission equipment
The classic ‘buy vs. build’ decision in Information Technology
11/30/2018

19. Future of Abilene
Original UCAID/Qwest agreement amended on October 1, 2001
Extension of for another 5 years – until October, 2006
Originally expired March, 2003
Upgrade of Abilene backbone to optical transport capability - ’s (unprotected)
x4 increase in the core backbone bandwidth
OC-48c SONET (2.5 Gbps) to 10-Gbps DWDM
11/30/2018

20. Key aspects of next generation Abilene backbone - I
Native IPv6
Motivations
Resolving IPv4 address exhaustion issues
Preservation of the original End-to-End Architecture model
p2p collaboration tools, reverse trend to CO-centrism
International collaboration
Router and host OS capabilities
Run natively - concurrent with IPv4
Replicate multicast deployment strategy
Close collaboration with Internet2 IPv6 Working Group on regional and campus v6 rollout
Addressing architecture
11/30/2018

21. Key aspects of next generation Abilene backbone - II
Network resiliency
Abilene ’s will not be protected like SONET
Increasing use of videoconferencing/VoIP impose tighter restoration requirements (<100 ms)
Options:
Currently: MPLS/TE fast reroute
IP-based IGP fast convergence (preferable)
Addition of new measurement capabilities
Enhance active probing (Surveyor)
Latency & jitter, loss, TCP throughput
Add passive measurement taps
Support for computer science research – “Abilene Observatories”
Support of Internet2 End-to-End Performance Initiative
Intermediate performance beacons
11/30/2018

23. Next generation router selection
Extensive router specification and test plan developed
Team effort: UCAID staff, NOC, NC and Ohio ITECs
Discussions with four router vendors
Tests focused on next gen advanced services
High performance TCP/IP throughput
High performance multicast
IPv6 functionality & throughput
Classification for QoS and measurement
3 router platforms tested & commercial ISPs referenced
 New Juniper T640 platform selected
11/30/2018

24. Two leading national initiatives in the U.S.
Next Generation Abilene
Advanced Internet backbone
connects entire campus networks of the research universities
10 Gbps nationally
TeraGrid
Distributed computing (Grid) backplane
connects high performance computing (HPC) machine rooms
Illinois: NCSA, Argonne
California: SDSC, Caltech
4x10 Gbps: Chicago  Los Angeles
Ongoing collaboration between both projects
11/30/2018

25. TeraGrid Architecture – 13.6 TF (Source: C. Catlett, ANL)
574p IA-32 Chiba City
256p HP
X-Class 32 32
Caltech
32 Nodes
0.5 TF
0.4 TB Memory
86 TB disk
Argonne
64 Nodes
1 TF
0.25 TB Memory
25 TB disk 32 32
128p Origin 24 32
128p HP
V2500 32
HR Display & VR Facilities 24 8 8 5
92p IA-32 5
HPSS
HPSS 24 4
Extreme
Black Diamond
OC-12
ESnet
HSCC
MREN/Abilene
Starlight
OC-48
Calren
OC-48
OC-12
NTON
OC-12 ATM
Juniper M160
GbE
SDSC
256 Nodes
4.1 TF, 2 TB Memory
225 TB disk
NCSA
500 Nodes
8 TF, 4 TB Memory
240 TB disk
Juniper M40
Juniper M40
vBNS
Abilene
Calren
ESnet
OC-12
OC-12
OC-3
vBNS
Abilene
MREN
OC-12 2 2
OC-12
OC-3
Myrinet Clos Spine 8 4
HPSS 8
UniTree 2
Sun
Starcat 4
Myrinet Clos Spine
= 32x 1GbE
1024p IA-32
320p IA-64
1176p IBM SP
Blue Horizon 16
= 64x Myrinet 14 4
= 32x Myrinet
1500p Origin
Sun E10K
= 32x FibreChannel
= 8x FibreChannel
10 GbE
32 quad-processor McKinley Servers
4GF, 8GB memory/server)
32 quad-processor McKinley Servers
4GF, 12GB memory/server)
Fibre Channel Switch
16 quad-processor McKinley Servers
4GF, 8GB memory/server)
Router or Switch/Router
IA-32 nodes

26. Regional optical fanout
Next generation architecture: Regional & state based optical networking projects are critical
Three-level hierarchy: backbone, GigaPoPs/ARNs, campuses
Leading examples
CENIC ONI (California), I-WIRE (Illinois),
SURA Crossroads (Southeastern U.S), Indiana, Ohio
Collaboration with the Quilt
Regional Optical Networking project
U.S. carrier DWDM access is now not nearly as widespread as with SONET circa 1998
30-60 cities for DWDM
~120 cities for SONET
11/30/2018

27. Optical network project differentiation
Distance scale (km)
Examples
Equipment
Metro
< 60
UW(SEA),
USC/ISI(LA)
Dark fiber & end terminals
State/
Regional
< 500
I-WIRE (IL),
CENIC ONI,
I-LIGHT (IN)
Add OO
amplifiers
Extended
Regional/
National
> 500
PLR,
TeraGrid
Abilene
Add OEO
regenerators
& O&M $’s
11/30/2018

28. 11/30/2018

29. Conclusions – Abilene future
Partnership with Qwest extended thru 2006
Juniper T640 routers selected for backbone
Backbone upgrade to 10-Gbps in three phases
New foci:
Native, high-performance IPv6
Enhanced, differentiated measurement
Increased network resiliency
Technical design and business model provides for an incremental, non-disruptive transition
Complementary to and collaborative with NSF’s TeraGrid
11/30/2018

30. For more information
Web:
11/30/2018