1. Internet2 Abilene Network and Next Generation Optical Networking
Steve Corbató
Director, Backbone Network Infrastructure
Access NovaForum
May 28, 2002
This is a general overview presentation about Internet2. Internet2 is a consortium, led by US universities, which is recreating the partnership among academia, industry and government that fostered today’s Internet in its infancy.

2. This presentation Abilene Network today Optical networking evolution
Next generation of Abilene
Future national optical initiatives
11/29/2018

3. Networking hierarchy
Internet2 networking is a fundamentally hierarchical and collaborative activity
International networking
Ad hoc  Global Terabit Research Network (GTRN)
National backbones
Regional networks
GigaPoPs  advanced regional networks
Campus networks
Much activity now at the metropolitan and regional scales
11/29/2018

4. Abilene focus Goals Advanced service efforts
Enabling innovative applications and advanced services not possible over the commercial Internet
Backbone & regional infrastructure provides a vital substrate for the continuing culture of Internet advancement in the university/corporate research sector
Advanced service efforts
Multicast
IPv6
QoS
Measurement
an open, collaborative approach
Security
11/29/2018

5. Partnership approach
The Abilene Network is a UCAID project done in partnership with
Cisco Systems (routers, switches, and access)
Juniper Networks (routers)
Nortel Networks (SONET kit)
Qwest Communications (SONET & DWDM circuits)
Indiana University (network operations center)
Internet2 Test & Evaluation Centers (ITECs)
North Carolina and Ohio
Partnerships are the foundation of how the Internet developed and they are also a part of the foundation of Internet2.
11/29/2018

6. 11/29/2018

7. Abilene – May, 2002
IP-over-SONET backbone (OC-48c, 2.5 Gbps) 53 direct connections
4 OC-48c connections
1 Gigabit Ethernet trial
23 will connect via at least OC-12c (622 Mbps) by 1Q02
Number of ATM connections decreasing
215 participants – research universities & labs
All 50 states, District of Columbia, & Puerto Rico
15 regional GigaPoPs support ~70% of participants
Expanded access
50 sponsored participants
New: Smithsonian Institution, Arecibo Radio Telescope
23 state education networks (SEGPs)
11/29/2018

8. Abilene international connectivity
Transoceanic R&E bandwidths growing!
GÉANT – 5 Gbps between Europe and New York City now
Key international exchange points facilitated by Internet2 membership and the U.S. scientific community
STARTAP & STAR LIGHT – Chicago (GigE)
AMPATH – Miami (OC-3c  OC-12c)
Pacific Wave – Seattle (GigE)
MAN LAN - New York City (GigE/10GigE EP soon)
CA*NET3/4: Seattle, Chicago, and New York
CUDI: CENIC and Univ. of Texas at El Paso
International transit service
Collaboration with CA*NET3 and STARTAP
11/29/2018

9. Abilene international connectivity model
Abilene is a GTRN partner
Already peering with GTRN routers in New York City and Seattle
Peering at major int’l EPs in U.S. encouraged
Chicago: Star Light (migration from STAR TAP)
Seattle: Pacific Wave
Miami: AMPATH
New York City: Manhattan Landing (MAN LAN) in progress
Los Angeles (soon?)
Direct BGP peering preferred
via Layer-2 EP media or direct connection
ATM support generally ends by Sept 2003
No new ATM peers
11/29/2018

10. Abilene International Peering
09 March 2002
Abilene International Peering
STAR TAP/Star Light
APAN/TransPAC, Ca*net3, CERN, CERnet, FASTnet, GEMnet, IUCC, KOREN/KREONET2, NORDUnet, RNP2, SURFnet, SingAREN, TAnet2
Pacific Wave
AARNET, APAN/TransPAC, CA*net3, TANET2
Sacramento
Los Angeles
Washington
NYCM
BELNET, CA*net3,
GEANT*,
HEANET, JANET,
NORDUnet
SNVA
GEMNET, SINET, SingAREN, WIDE
LOSA
UNINET
OC3-OC12
San Diego (CALREN2)
CUDI
AMPATH
REUNA, RNP2 RETINA, ANSP, (CRNet)
El Paso (UACJ-UT El Paso)
CUDI
* ARNES, CARNET, CESnet, DFN, GRNET, RENATER, RESTENA, SWITCH, HUNGARNET, GARR-B, POL-34, RCST, RedIRIS

11. Packetized raw High Definition Television (HDTV)
Raw HDTV/IP – single UDP flow of 1.5 Gbps
Project of USC/ISIe, Tektronix, & U. of Wash (DARPA)
6 Jan 2002: Seattle to Washington DC via Abilene
Single flow utilized 60% of backbone bandwidth
18 hours: no packets lost, 15 resequencing episodes
End-to-end network performance (includes P/NW & MAX GigaPoPs)
Loss: <0.8 ppb (90% c.l.)
Reordering: 5 ppb
Transcontinental 1-Gbps TCP
requires loss of
<30 ppb (1.5 KB frames)
<1 ppm (9KB jumbo)
11/29/2018

12. End-to-End Performance: ‘High bandwidth is not enough’
Bulk TCP flows (> 10 Mbytes transfer)
Current median flow rate over Abilene: 1.9 Mbps
11/29/2018

13. True End-to-End Performance requires a system approach
User perception
Application
Operating system
Host IP stack
Host network card
Local Area Network
Campus backbone network
Campus link to regional network/GigaPoP
GigaPoP link to Internet2 national backbones
International connections
EYEBALL
APPLICATION
STACK
JACK
NETWORK
. . .
11/29/2018

14. Jumbo frames supported
Default Abilene backbone MTU has been increased from 4.5 to 9 kB
We now can support 9 kB MTUs on a per connector basis
Motivation: support for HPC computing and large TCP flows
11/29/2018

15. Abilene traffic characterization information
Weekly detailed reports
General analysis
11/29/2018

16. Optical networking technology drivers
Aggressive period of fiber construction on the national & metro scales in U.S.
Now rapid industry contraction and capital crisis
Many university campuses and regional GigaPoPs already use dark fiber
Dense Wave Division Multiplexing (DWDM)
Allows the provisioning of multiple channels (’s) over distinct wavelengths on the same fiber pair
Fiber pair can carry 160 channels (1.6 Tbps!)
Optical transport is the current focus
Optical switching is still in the realm of experimental networks, but may be nearing practical application

17. DWDM technology primer
DWDM fundamentally is an analog optical technology
Combines multiple channels ( in number) over the same fiber pair
Uses slightly displaced wavelengths (’s) of light
Generally supports 2.5 or 10 Gbps channels
Physical obstacles to long-distance transmission of light
Attenuation
Solved by amplification (OO)
Wavelength dispersion
Requires periodic signal regeneration – an electronic process (OEO)
11/29/2018

18. DWDM system components
Base fiber pair (+ right of way & conduit)
Multiplexing/demultiplexing terminals
OEO equipment at each end of light path
Output: SONET or Ethernet (10G/1G) framing
Amplifiers
All optical (OO)
~100 km spacing
Regeneration
Electrical (OEO) process – costly (~50% of capital)
~500 km spacing (with Long Haul - LH - DWDM)
New technologies (ELH/ULH) can extend this distance
Remote huts, operations & maintenance
11/29/2018

19. Telephony’s recent past (from an IP perspective in the U.S.)
11/29/2018

20. IP Networking (and telephony) in the not so distant future
11/29/2018

21. National optical networking options
1 – Provision incremental wavelengths
Obtain 10-Gbps ’s as with SONET
Exploit smaller incremental cost of additional ’s
1st  costs ~10x than subsequent ’s
2 – Build dim fiber facility
Partner with a facilities-based provider
Acquire 1-2 fiber pairs on a national scale
Outsource operation of inter-city transmission equipment
Needs lower-cost optical transmission equipment
The classic ‘buy vs. build’ decision in Information Technology
11/29/2018

22. Future of Abilene
Original UCAID/Qwest agreement amended on October 1, 2001
Extension of MoU for another 5 years – until October, 2006
Originally expired March, 2003
Upgrade of Abilene backbone to optical transport capability - ’s (unprotected)
x4 increase in the core backbone bandwidth
OC-48c SONET (2.5 Gbps) to 10-Gbps DWDM
11/29/2018

23. Key aspects of next generation Abilene backbone - I
Native IPv6
Motivations
Resolving IPv4 address exhaustion issues
Preservation of the original End-to-End Architecture model
p2p collaboration tools, reverse trend to CO-centrism
International collaboration
Router and host OS capabilities
Run natively - concurrent with IPv4
Replicate multicast deployment strategy
Close collaboration with Internet2 IPv6 Working Group on regional and campus v6 rollout
Addressing architecture
11/29/2018

24. Key aspects of next generation Abilene backbone - II
Network resiliency
Abilene ’s will not be ring protected like SONET
Increasing use of videoconferencing/VoIP impose tighter restoration requirements (<100 ms)
Options:
MPLS/TE fast reroute (initially)
IP-based IGP fast convergence (preferable)
11/29/2018

25. Key aspects of next generation Abilene backbone - III
New & differentiated measurement capabilities
Significant factor in NGA rack design
4 dedicated servers at each nodes
Additional provisions for future servers
Local data collection to capture data at times of network instability
Enhance active probing
Now: latency & jitter, loss, reachability (Surveyor)
Regular TCP/UDP throughput tests – ~1 Gbps
Separate server for E2E performance beacon
Enhance passive measurement
Now: SNMP (NOC) & traffic matrix/type (Netflow)
Routing (BGP & IGP)
Optical splitter taps on backbone links at select location(s)
11/29/2018

26. Abilene Observatories
Currently a program outline for better support of computer science research
Influenced by discussions with NRLC members
1) Improved & accessible data archive
Need coherent database design
Unify & correlate 4 separate data types
SNMP, active measurement data, routing, Netflow
2) Provision for direct network measurement and experimentation
Resources reserved for two additional servers
Power (DC), rack space (2RU), router uplink ports (GigE)
Need process for identifying meritorious projects
Need ‘rules of engagement’ (technical & policy)
11/29/2018

28. Next generation router selection
Extensive router specification and test plan developed
Team effort: UCAID staff, NOC, NC and Ohio ITECs
Discussions with four router vendors
Tests focused on next gen advanced services
High performance TCP/IP throughput
High performance multicast
IPv6 functionality & throughput
Classification for QoS and measurement
3 router platforms tested & commercial ISPs referenced
 New Juniper T640 platform selected
11/29/2018

29. Abilene cost recovery model
Connection (per connection)
Annual fee
OC-3 (155 Mbps)*
$110,000
OC-12 (622 Mbps)
$270,000
Gigabit Ethernet (1 Gbps)**
$325,000
OC-48 (2.5 Gbps)
$430,000
OC-192/10 GigE (10 Gbps)
$490,000
Participation (per university)
$20,000
11/29/2018

30. Abilene program changes
10-Gbps (OC-192c POS) connections
 backhaul available wherever needed & possible
Only required now for 1 of 4 OC-48c connections
3-year connectivity commitment required
Gigabit and 10-Gigabit Ethernet
Available when connector has dark fiber access into Abilene router node
Backhaul not available
ATM connection & peer support
TAC recommended ending ATM support by fall 2003
Two major ATM-based GigaPoPs have migrated
2 of 3 NGIXes still are ATM-based
STAR LIGHT is now GigE
Urging phased migration for connectors & peers
11/29/2018

31. Deployment timing Ongoing – Backbone router procurement
Detailed deployment planning
July – Rack assembly (Indiana Univ.)
Aug/Sep – New rack deployment at all 11 nodes
Fall – First Wave ’s commissioned
Fall meeting demonstration events
iGRID 2002 (Amsterdam) – late Sep.
Internet2 Fall Member Meeting (Los Angeles) – late Oct.
SC2002 (Baltimore) – mid Nov.
Remaining ’s commissioned in 2003

32. Two leading national initiatives in the U.S.
Next Generation Abilene
Advanced Internet backbone
connects entire campus networks of the research universities
10 Gbps nationally
TeraGrid
Distributed computing (Grid) backplane
connects high performance computing (HPC) machine rooms
Illinois: NCSA, Argonne
California: SDSC, Caltech
4x10 Gbps: Chicago  Los Angeles
Ongoing collaboration between both projects
11/29/2018

33. TeraGrid: A National Infrastructure
For more information:

34. TeraGrid Architecture – 13.6 TF (Source: C. Catlett, ANL)
574p IA-32 Chiba City
256p HP
X-Class 32 32
Caltech
32 Nodes
0.5 TF
0.4 TB Memory
86 TB disk
Argonne
64 Nodes
1 TF
0.25 TB Memory
25 TB disk 32 32
128p Origin 24 32
128p HP
V2500 32
HR Display & VR Facilities 24 8 8 5
92p IA-32 5
HPSS
HPSS 24 4
Extreme
Black Diamond
OC-12
ESnet
HSCC
MREN/Abilene
Starlight
OC-48
Calren
OC-48
OC-12
NTON
OC-12 ATM
Juniper M160
GbE
SDSC
256 Nodes
4.1 TF, 2 TB Memory
225 TB disk
NCSA
500 Nodes
8 TF, 4 TB Memory
240 TB disk
Juniper M40
Juniper M40
vBNS
Abilene
Calren
ESnet
OC-12
OC-12
OC-3
vBNS
Abilene
MREN
OC-12 2 2
OC-12
OC-3
Myrinet Clos Spine 8 4
HPSS 8
UniTree 2
Sun
Starcat 4
Myrinet Clos Spine
= 32x 1GbE
1024p IA-32
320p IA-64
1176p IBM SP
Blue Horizon 16
= 64x Myrinet 14 4
= 32x Myrinet
1500p Origin
Sun E10K
= 32x FibreChannel
= 8x FibreChannel
10 GbE
32 quad-processor McKinley Servers
4GF, 8GB memory/server)
32 quad-processor McKinley Servers
4GF, 12GB memory/server)
Fibre Channel Switch
16 quad-processor McKinley Servers
4GF, 8GB memory/server)
Router or Switch/Router
IA-32 nodes

35. Optical networking scaling factors
2 TeraGrid routing nodes
11 Next Generation Abilene routers
53 Abilene connectors
215 Abilene participants (univs & labs)
But…
30-60 DWDM access nodes in leading viable carriers’ U.S. networks
11/29/2018

36. Regional optical fanout
Next generation architecture: Regional & state based optical networking projects are critical
Three-level hierarchy
backbone, GigaPoPs/ARNs, campuses
Leading examples
CENIC ONI (California), I-WIRE (Illinois), Indiana (I-LIGHT)
Collaboration with the Quilt GigaPoPs
Regional Optical Networking project
U.S. carrier DWDM access is now not nearly as widespread as with SONET circa 1998
30-60 cities for DWDM
~120 cities for SONET
11/29/2018

37. Optical network project differentiation
Distance scale (km)
Examples
Equipment
Metro
< 60
UW(SEA),
USC/ISI(LA)
Dark fiber & end terminals
State/
Regional
< 500
I-WIRE (IL),
CENIC ONI,
I-LIGHT (IN)
Add OO
amplifiers
Extended
Regional/
National
> 500
PLR,
TeraGrid
Abilene
Add OEO
regenerators
& O&M $’s

38. Conclusions Backbone upgrade project underway Advanced service foci
Partnership with Qwest extended thru 2006
Juniper T640 routers selected for backbone
10-Gbps backbone  deployment starts this fall
Incremental, non-disruptive transition
Advanced service foci
Native, high-performance IPv6
Enhanced, differentiated measurement
Network resiliency
NSF TeraGrid and Extended Terascale Facility
Complementary and collaborative relationship
Continue to examine prospects for a fiber optical networking facility – National Light Rail

39. For more information
Web:
11/29/2018