1. Internet2: A Tutorial Part 2 of 4 17 th Brazilian Symposium on Computer Networks Paul Love, Internet2 Chair, I2 Topology WG epl@internet2.edu

2. Abilene

3. Overview Abilene Background Review of Abilene Architecture 1999 Schedule Launched Network First Half 1999 Expectations Engineering Issues Additional information http://www.internet2.edu/abilene/

4. 25-28 May 99 SBRC99 Salvador, Bahia, Brazil Abilene: Objectives Provide high-quality, widely available Interconnect among participating gigaPoPs/universities IP as the bearer service Goals Reliability Functionality Latency Capacity

5. 25-28 May 99 SBRC99 Salvador, Bahia, Brazil Abilene: Objectives cont Additional, separate networks to Test advanced network capabilities Final proposals being evaluated this month For network research - still TBD Support advanced functionality as it evolves Multicast, QoS, IPv6

6. 25-28 May 99 SBRC99 Salvador, Bahia, Brazil Abilene Team Overall direction by Internet2 Qwest Cisco Nortel (Northern Telecom) Indiana University National Center for Network Engineering Open to other contributors Collaborate with related efforts in network or applications research

7. 25-28 May 99 SBRC99 Salvador, Bahia, Brazil Key Attributes IP over Sonet initially Qwest OC-48 Sonet capacity and collocation sites Nortel OC-192 Sonet equipment Cisco GSR 12000 routers including support for QoS Native multicast

8. 25-28 May 99 SBRC99 Salvador, Bahia, Brazil Abilene and Other Networks Important for Abilene to interconnect with other high performance networks First I2 backbone: vBNS Federal Agency NGI networks (ESnet, NREN, DREN,...) Non-US Research & Education Networks CANARIE, NORDUnet, SURFnet, TransPac, etc Peering in hand or plans in process vBNS, ESnet, CANARIE, DREN, NREN, NORDUnet, SURFnet, etc Advanced functionality across I2 & our peers Native multicast QoS across multiple interconnected networks *RSN

9. 25-28 May 99 SBRC99 Salvador, Bahia, Brazil Broader Internet2, NGI, and International Advanced Nets Initial NGIX sites Possible CA*net3 peering sites StarTap StarTap Plus

10. 25-28 May 99 SBRC99 Salvador, Bahia, Brazil Notes on I2, NGI, and International Interconnections NGIXes are the foundation for peering I2 backbones & with NGI nets Abilene will connect with OC12 StarTap is the foundation of international interconnections - universal solution New York is first example of StarTap Plus Minimize latency Provide redundancy Abilene doesn’t provide transit

11. 25-28 May 99 SBRC99 Salvador, Bahia, Brazil Example: New York City SURFnet has OC-3 circuits and routers at the Teleglobe POP at 60 Hudson Street NORDUnet the same And each connects from New York to the StarTap with DS3 Details on the next slides

12. 25-28 May 99 SBRC99 Salvador, Bahia, Brazil SURFnet NORDUnet Chicago New York City Europe Teleglobe Qwest Abilene ISP

13. 25-28 May 99 SBRC99 Salvador, Bahia, Brazil SURFnet NORDUnet Chicago New York CityEurope Teleglobe Qwest Abilene ISP

14. 25-28 May 99 SBRC99 Salvador, Bahia, Brazil CA*net3 Charlottetown Vancouver CalgaryRegina Winnipeg Ottawa Montreal Halifax Fredericton Chicago Toronto St. John’s

15. 25-28 May 99 SBRC99 Salvador, Bahia, Brazil Abilene NOC Located at Indiana University Publish statistics & online information resources Coverage can be looked at in three broad areas Administration Engineering Operations

16. 25-28 May 99 SBRC99 Salvador, Bahia, Brazil Administration Security Management Communications Overall Management and Coordination

17. 25-28 May 99 SBRC99 Salvador, Bahia, Brazil Engineering Configuration Testing Data collection Traffic Analysis

18. 25-28 May 99 SBRC99 Salvador, Bahia, Brazil Operations 24x7 Monitoring Reporting Change and Problem management Information base Single point of contact Network problems Abilene connection process

19. Online Traffic Map

20. 25-28 May 99 SBRC99 Salvador, Bahia, Brazil For more information on the Abilene NOC www.abilene.iu.edu Abilene traffic map at http://hydra.uits.iu.edu/~abilene/traffic

21. 25-28 May 99 SBRC99 Salvador, Bahia, Brazil Couple of Notes on Costs Cost recovery on UCAID’s part Set each year for following year Participant institution Connector - based on BW (can be connector and not participant) nb There is also a small Qwest access fee Long-distance part of access line is considered part of the ‘backbone’ Thus, number/location of core nodes does not affect costs borne by connecting sites

22. 25-28 May 99 SBRC99 Salvador, Bahia, Brazil Qwest Fiber Network http//www.qwest.com/network/netmaps.html

23. 25-28 May 99 SBRC99 Salvador, Bahia, Brazil Schedule - part 1 Design work: Mar-98 and ongoing Rack design: May-98 to Jul-98 Initial assembly / testing: Jul/Aug-98 Router Nodes / Interior Lines: Jul/Aug-98 Demo network installed: Sep-98 Remainder installed: Nov/Dec-98 Production begins: 29-Jan-99 Launch Event in DC: 24-Feb-99

24. 25-28 May 99 SBRC99 Salvador, Bahia, Brazil Seattle Kansas City Denver Cleveland New York Atlanta Houston Pittsburgh Minneapolis Columbus Washington Phoenix Raleigh Trent on Salt Lake City Wilmington Dallas New Orleans Lincoln New Haven Detroit Miami Westfield Nashville Philadelp hia Indianapolis Newar k Albuquerque Oklahoma City Access Node Router Node Star Tap Sacramento Oakland Eugene Los Angeles Anaheim Boston Abilene Sep 98 Demo Network

25. 25-28 May 99 SBRC99 Salvador, Bahia, Brazil Abilene Architecture: Core Router Nodes located at Qwest PoPs Cisco 12008 GSR ICS Unix PC: IPPM and Network Mgmt Cisco 3640 Remote Access for NOC 100BaseT LAN and ‘console port’ access Remote 48v DC Power Controllers Initially, ten Router Nodes Seattle, Sacramento, Los Angeles, Denver, Kansas City, Houston, Indianapolis, Cleveland, New York City, Atlanta And, coming this Fall - Washington, DC Interior lines connect core nodes OC-12 and OC-48 Sonet IP-over-Sonet interfaces

26. 25-28 May 99 SBRC99 Salvador, Bahia, Brazil

27. 25-28 May 99 SBRC99 Salvador, Bahia, Brazil Abilene Architecture: Access Access Nodes Located at Qwest PoPs Sonet: Connects Local to Long-distance Currently about 135 Access Nodes: This list grows as the Qwest Sonet plant grows Access lines connect from core node to gigaPoP or member IP-over-Sonet or IP-over-ATM possible OC-3 and OC-12 typical Others as possible & appropriate GigE,...

28. 25-28 May 99 SBRC99 Salvador, Bahia, Brazil Seattle Kansas City Denver Cleveland New York Atlanta Houston Pittsburgh Minneapolis Columbus Washington Phoenix Raleigh Trent on Salt Lake City Wilmington Dallas New Orleans Lincoln New Haven Detroit Miami Westfield Nashville Philadelp hia Indianapolis Newar k Albuquerque Oklahoma City Access Node Router Node Sacramento Oakland Eugene Los Angeles Anaheim Boston Chicago Abilene + some Access Nodes

29. 25-28 May 99 SBRC99 Salvador, Bahia, Brazil Launch Network Some Sonet Rings not completed Mix of OC-12 and OC-48 interior Lines NOC fully operational Some Access Lines not completed Connections at two NGIXes NGIX/Central (OC3) NGIX/West (OC12) OC12 into Union Station for demos

30. 25-28 May 99 SBRC99 Salvador, Bahia, Brazil Schedule, part 2 Production began: 29-Jan-99 Launch Event in DC: 24-Feb-99 Member Meeting: 28-Apr-99 Native Multicast deployed in core: Jun-99 Completion of OC-48 Core: TBD Addition of DC Router Node: 3Q99 Seattle Member Meeting: 11/15-Oct-99

31. 25-28 May 99 SBRC99 Salvador, Bahia, Brazil First Half 1999 Expectations Measurements of Delay, Loss, Utilization Gradual Increase in OC-48 Interior Lines Completion of (Qwest) Sonet Rings Accesss to measurements and other engineering data Participation in QBone Upgrade of NGIX/Central to OC12 Native Multicast

32. 25-28 May 99 SBRC99 Salvador, Bahia, Brazil Second Half 1999 Expectations Gigabit Ethernet connection(s)? OC48 connection? Connection to NGIX/East Somewhere in Washington, DC, area OC12

33. 25-28 May 99 SBRC99 Salvador, Bahia, Brazil Abilene Core: Current Seattle Kansas City Denver Cleveland New York Atlanta Houston Indianapolis Sacramento Los Angeles

34. 25-28 May 99 SBRC99 Salvador, Bahia, Brazil Abilene: Current plus On Order Seattle Kansas City Denver Cleveland New York Atlanta Houston Indianapolis Sacramento Los Angeles OC12 OC3 OC12 still being installed OC12 OC48

35. 25-28 May 99 SBRC99 Salvador, Bahia, Brazil Outline of Engineering Issues Routing: OSPF, BGP4, Routing Arbiter Database Multicast PIM-Sparse Mode, MBGP, MSDP Measurements Surveyor: One-way delay and loss Traffic utilization End to end flows with gigaPoP help OC3MON -- passive measurements QoS - QBone

36. GigaPoP Architecture

37. 25-28 May 99 SBRC99 Salvador, Bahia, Brazil Acknowledgments Steve Corbató - University of Washington & Pacific/Northwest Gigapop Ron Hutchins - Georgia Institute of Technology & Southern CrossRoads Gigapop Ken Lindahl - University of California, Berkeley Rick Summerhill - Executive Director, Great Plains Network David Wasley - CENIC Director of Projects/CalREN-2 Gigapop

38. 25-28 May 99 SBRC99 Salvador, Bahia, Brazil GigaPoP Idea From George Strawn of NSF - Predates Internet2 Idea was one of the catalysts for Internet2 Aggregation point Economies of scale Member sites Value added services shared by members Not limited to university/I2 members (High-speed) local traffic stays local Attached networks

39. 25-28 May 99 SBRC99 Salvador, Bahia, Brazil GigaPoPs Variety of services and styles Technical and organizational differences Mixture of technologies Some things must be the same IP as a common bearer service Inter-GigaPoP routing policy and design Measurement Trouble tickets among network operation centers

40. 25-28 May 99 SBRC99 Salvador, Bahia, Brazil Agency Networks Customer links Backbone nodes Aggregation points/ GigaPoPs Peering connection Robust Model With thanks to Ron Hutchins

41. 25-28 May 99 SBRC99 Salvador, Bahia, Brazil Peachnet GSU GT SoX GigaPoP NCSU UNC Duke MCNC vBNS GCATT Vanderbilt FSU, UF, Miami GigaPoPs can be Hierarchical UA UAH UAB With thanks to Ron Hutchins

42. 25-28 May 99 SBRC99 Salvador, Bahia, Brazil Denotes GigaPoP Internet2 GigaPoPs As of November 1997

43. 25-28 May 99 SBRC99 Salvador, Bahia, Brazil Denotes GigaPoP Internet2 GigaPoPs As of 19 May 1999

44. 25-28 May 99 SBRC99 Salvador, Bahia, Brazil GigaPoPs - What Flavors Do They Come In Two broad sets of choices Level Topology Level Layer 1 (optical) Layer 2 (usually ATM) Layer 3 Topology Star - aka Gigapoint Distributed Combination - figure “8”

45. 25-28 May 99 SBRC99 Salvador, Bahia, Brazil Star or Gigapoint Advantages No internal transit issues Disadvantages Longer backhauls for both members & service providers Member links ISP/HPNSP links

46. 25-28 May 99 SBRC99 Salvador, Bahia, Brazil Distributed Advantages Optimize backhauls Take advantage existing connections Politics Disadvantages BW between nodes can be bottleneck Member links Intra-GP links ISP/HPNSP links

47. 25-28 May 99 SBRC99 Salvador, Bahia, Brazil Combination Advantages Optimize backhauls Take advantage existing connections Politics Minimize internal transit Disadvantages BW between hubs can be bottleneck Longer backhauls Member links Intra-GP link ISP/HPNSP links

48. Example - Non-ATM Level 2 GigaPoP

49. 25-28 May 99 SBRC99 Salvador, Bahia, Brazil P/NW GigaPoP Design With thanks to Steve Corbató

50. Example - Combination GigaPoP

51. 25-28 May 99 SBRC99 Salvador, Bahia, Brazil Great Plains Network Participating States N & S Dakota Nebraska Kansas Oklahoma Missouri Arkansas Topology With thanks to Rick Summerhill

52. Example - Distributed GigaPoP

53. 25-28 May 99 SBRC99 Salvador, Bahia, Brazil CalREN-2 Topology Plan (5/98) With thanks to David Wasley

54. 25-28 May 99 SBRC99 Salvador, Bahia, Brazil CalREN-2 Topology With thanks to Ken Lindahl

55. 25-28 May 99 SBRC99 Salvador, Bahia, Brazil CalREN-2 North: Physical Topology With thanks to Ken Lindahl

56. 25-28 May 99 SBRC99 Salvador, Bahia, Brazil CalREN-2 North: Logical Topology With thanks to Ken Lindahl

57. Example - Star/Gigapoint

58. 25-28 May 99 SBRC99 Salvador, Bahia, Brazil R R R R R R R R R R R R R R vBNSISP R R Georgia GigaPoP Peering using BGP Logical Layer 3 Diagram With thanks to Ron Hutchins SoX: Layer 3 Example

59. 25-28 May 99 SBRC99 Salvador, Bahia, Brazil vBNS Router GigaPOP Switch Georgia GigaPOP B “External Gigapop” A PNNI GT Network Dept. Net ISPvBNS Layer 2 diagram With thanks to Ron Hutchins SoX: Layer 2 Example

60. 25-28 May 99 SBRC99 Salvador, Bahia, Brazil ISP 3 * Campus Network * * * * * ISP 1 HPNSP ISP2 Campus 1 Campus 2 3 4 SoX Gigapop vBNS ISP ATM “Pipe” ABR/UBR PVC Burst to Full OC-3 VBR PVC Preempt to 10Mb/s Combined Services With thanks to Ron Hutchins

61. 25-28 May 99 SBRC99 Salvador, Bahia, Brazil The End