Network Update Steve Cotter Director, Network Services Joint Techs, Fermilab July 16, 2007

Internet2 Network Infrastructure Overview Layer 1: Managed wavelengths from Level(3) Communications Level(3) owns and manages Infinera optical gear: responsible for software upgrades, equipment maintenance, remote hands, sparing, NOC services Internet2 NOC has total provisioning control Layer 2: Internet2 owned and managed Ciena CoreDirectors Using DRAGON GMPLS control plane Layer 3: Internet2 owned and managed Juniper T640s Expanded Observatory Platform for layer 1/3 network performance data collection, collocation, experimentation perfSONAR integration for intra- & inter-network performance analysis International connectivity Layer 1 network extended to international exchange points in Seattle, Chicago and New York City Peering points in Seattle, PAIX, Equinix Chicago, others Reason for going with a managed service: End to end service provided over the life of the service order/contract No replacement costs required for hardware or software over the life of the contract No sparing management or other issues to negotiation or implement with vendors, field support, etc. No costs for software upgrades during the life of the contract No end of life issues for platforms during the life of the contract No incremental funding required if issues develop with platform, performance or other service related items Benefit from expertise and experience from network provider Leverage scale and size of network provider

Seattle 1000 Denny Way Level 3 Pacific Northwest GP 2001 6th Ave Westin Bldg Albany 316 N Pearl Level 3 Portland Oregon GP 707 SW Washington Qwest Rieth Cleveland TFN/MERIT 4000 Chester Chicago CIC/MREN BOREAS Internet2 710 N Lakeshore Starlight Cambridge NOX 300 Bent St Portland 1335 NW Northrop Level 3 Syracuse Boise Buffalo New York 111 8th Level 3 Tionesta Detroit New York NYSERNET 32 A of the A Chicago 600 W Chicago Ogden Rawlins Omaha Indianapolis IU 1902 S East St Philadelphia MAGPI 401 N Broad Reno Eureka Edison Pittsburgh Pittsburgh GP 143 S 25th Sacramento via 1005 N B St Washington MAX 1755 Old Meadow San Francisco Salt Lake Inter-Mountain GP Front Range GP 572 S DeLong Denver 1850 Pearl Sunnyvale 1380 Kifer Kansas City GPN 1100 Walnut St. Louis Louisville Univ Louisville 848 S 8th St San Luis Obispo Raleigh 5301 Departure Dr Charlotte Tulsa 18 W Archer Nashville Univ Memphis 2990 Sidco Dr Raton Los Angeles 818 W 7th Level 3 Albuquerque New Mexico GP 104 Gold Ave SE Atlanta 180 Peachtree St NW Los Angeles CENIC 600 W 7th Equinix Atlanta SLR 345 Courtland Phoenix San Diego Tucson Rancho De La Fe (tentative) Birmingham Dallas Mobile El Paso 501 W Overland Valentine Tallahassee Sanderson Jacksonville FAMU&USF 4814 Phillips Hwy Regen DTN Core Node Drop DTN Other Level 3 San Antonio Baton Rouge LONI 9987 Burbank Houston LEARN 1201 N I-45

Circuit Services Nodes Availability Status July 15, 2007 Seattle Seattle 1000 Denny 1000 Denny Pacific Northwest GP Way Way 2001 6 2001 6 th Ave Level 3 Level 3 Westin Bldg Portland Portland Rieth Rieth Cleveland Cleveland Albany Albany TFN TFN 316 N Pearl 316 N Pearl Cambridge Cambridge Oregon GP Oregon GP Portland Portland 707 SW Washington 707 SW Washington 4000 Chester 4000 Chester Syracuse Syracuse NOX NOX Qwest Qwest 1335 NW Northrop 1335 NW Northrop Chicago Chicago Level 3 Level 3 Level 3 Level 3 300 Bent St 300 Bent St Level 3 Level 3 CIC/MREN CIC/MREN Level 3 Level 3 MERIT MERIT Boise Boise BOREAS BOREAS Buffalo Buffalo Internet2 Internet2 New York New York Tionesta Tionesta 710 N Lakeshore 710 N Lakeshore Detroit Detroit 111 8 111 8 th th New York Starlight Level 3 Level 3 NYSERNET 32 Ave of the Chicago Chicago Americas Ogden Ogden Eureka Eureka Rawlins Rawlins Omaha Omaha 600 W Chicago 600 W Chicago Philadelphia Philadelphia via 1075 Triangle Ct via 1075 Triangle Ct Reno Reno Level 3 MC Level 3 MC Pittsburgh Pittsburgh MAGPI MAGPI Sacramento Sacramento Edison Edison Pittsburgh GP Pittsburgh GP 401 N Broad 401 N Broad Indianapolis Indianapolis 143 S 25th 143 S 25th Level 3 Level 3 via 1005 N B St via 1005 N B St Salt Lake Salt Lake 1902 S East St 1902 S East St Level 3 Level 3 Washington Washington San Francisco San Francisco Inter Inter - - Mountain GP Mountain GP Denver Denver Level 3 Level 3 MAX MAX 572 S DeLong 572 S DeLong Front Range GP Front Range GP 1755 Old Meadow 1755 Old Meadow Sunnyvale Sunnyvale Level 3 Level 3 1850 Pearl 1850 Pearl Kansas City Kansas City St. Louis St. Louis Louisville Louisville Level 3 Level 3 CENIC CENIC Level 3 Level 3 GPN GPN 848 S 8 848 S 8 th th St St 1380 Kifer 1380 Kifer Tulsa Tulsa 1100 Walnut 1100 Walnut Raleigh Raleigh Level 3 Level 3 San Luis Obispo San Luis Obispo OneNet OneNet Level 3 Level 3 Level 3 Level 3 Nashville Nashville Charlotte Charlotte NCREN NCREN 18 W Archer 18 W Archer Tennessee GP Tennessee GP 5301 Departure Dr 5301 Departure Dr Los Angeles Los Angeles Albuquerque Albuquerque Raton Raton Level 3 Level 3 2990 Sidco Dr 2990 Sidco Dr Level 3 Level 3 th th Level 3 Level 3 Atlanta Atlanta 818 W 7 818 W 7 New Mexico GP New Mexico GP Level 3 Level 3 SLR SLR 104 Gold Ave SE 104 Gold Ave SE 345 Courtland 345 Courtland Los Angeles Los Angeles Level 3 Level 3 Atlanta Atlanta CENIC CENIC 180 Peachtree St NW 180 Peachtree St NW 600 W 7th 600 W 7th Equinix Equinix San Diego San Diego Phoenix Phoenix Tucson Tucson Rancho De La Fe Rancho De La Fe Level 3 MC Level 3 MC (tentative) (tentative) Birmingham Birmingham Jacksonville Jacksonville Dallas Dallas FLR FLR Mobile Mobile 4814 Phillips Hwy 4814 Phillips Hwy El Paso Valentine Valentine Level 3 Level 3 501 W Overland Tallahassee Tallahassee Level 3 Sanderson Sanderson Baton Rouge Baton Rouge Fully Operational Installed – not fully turned up Not Installed yet Install Cancelled San Antonio San Antonio Houston Houston LONI LONI LEARN LEARN 9987 Burbank 9987 Burbank 1201 N I 1201 N I - - 45 45 Level 3 Level 3 Level 3 Level 3

Transition Progress PHASE 1: New York, Philadelphia, Washington, Pittsburgh, Cleveland, Chicago, Boston Internet2 and Connector Site Design Discussions Complete Colocation Planning, Connector Coordination Complete Colocation Availability and Equipment Installation Complete Wave Availability and Backbone Turn Up Complete Connector Cutover and Abilene Node Turn Down Complete PHASE 2: Raleigh, Atlanta, Nashville, Louisville, Indianapolis Wave Availability and Backbone Turn Up Complete Connector Cutover and Abilene Node Turn Down Complete except USF PHASE 3: Kansas City, Tulsa, Houston, Baton Rouge, Jacksonville, Albuquerque, Denver Internet2 and Connector Site Design Discussions Complete Connector Cutover and Abilene Node Turn Down Complete except HOU, FAMU, BTR PHASE 4: Salt Lake City, Seattle, Portland, Los Angeles Connector Cutover and Abilene Node Turn Down In progress; CENIC, SLC, Portland outstanding

Internet2 Network Network by the Numbers: 13,500 long haul route miles Deployed and configured over 300 Infinera Network Elements Day 1 capacity of 100Gbps Built 27 custom collocation suites representing 3,365 sqft of space including: 91 Racks - Internet2, ESnet, third-parties 60 Individual bulk cables with 48 & 96 fiber count Deployed 64 metro fiber route miles Enabled connectivity from Houston-to-El Paso & El Paso-to-Denver for L(3) to Internet2/GRNOC and ESnet NOC access Product Summary: Wavelengths, HSIP, Private Lines, Metro Fiber, Collocation, Cross-Connects, Power Controllers, and GPS timing Backbone alone was 569 individual tasks completed over 261 days! Developed the Virtual Network Operations Center – Provisioning and Troubleshooting Dashboard

Internet2 Network Flexible Infrastructure Supporting e-Science, Network Research & Education Best-Effort High-Speed IP Service Enables delivery of advanced content, commodity services, etc. Point-to-Point Wavelength Services Circuit Service for static or on-demand bandwidth Point-to-point Ethernet (VLAN) Framed SONET Circuit Point-to-point SONET Circuit Bandwidth provisioning available in 50 Mbps increments Physical Connection 1 or 10 Gigabit Ethernet OC-192 SONET

Internet2 Network Commodity Peering Service The growing list of members taking advantage of this service includes: Great Plains Network (GPN) ● Indiana GigaPoP MERIT through CIC ● Northern Crossroads (NOX) OSCnet ● Oregon GigaPoP University of Iowa through CIC ● University of Louisville University of New Mexico ● University of Texas – Austin Currently, the CPS offers over 60,000 routes through peering partnerships with commercial networks in Chicago, Seattle, and Palo Alto. A connection to NYC is underway, which will further improve and diversify our commercial network peering structure. Additional peering connections are being investigated.

Internet2 Network Connection oriented services provide for: Wavelength & Circuit Services Connection oriented services provide for: Guaranteed bandwidth and predictable latency (repeatable, dependable performance between collaborating sites) Traffic segregation (support specific policy or traffic engineering requirements) Router bypass: Express links created for high-bandwidth, limited duration long-haul traffic reducing the need for mid-path L3 interfaces Cost efficiency: L3 router blades cost > L2 ports > L1 or L0 interfaces Capability tradeoff but could possibly improve performance

Internet2 Network Wavelength & Circuit Services Automated circuit provisioning enable rapid deployment and efficient utilization of capital investment Establishing end-to-end lightpaths is a non-trivial task: it is resource intensive and error prone Automated reservation, allocation, and provisioning enables co-scheduling of network and non-network resources

Multi-Service/Domain/Layer/Vendor Provisioning Multi-Domain Provisioning Interdomain ENNI (Web Service and OIF/GMPLS) Multi-domain, multi-stage path computation process AAA Scheduling GEANT TDM GUI Internet2 Network Regional Network XML Regional Network AST Dynamic Ethernet Dynamic Ethernet TDM Domain Controller ESNet Data Plane Ctrl Element Control Plane Adjacency Ethernet LSP SONET Switch IP Network (MPLS, L2VPN) Router Slide from Tom Lehman, ISI-East

Internet2 Network Circuit Services Applications Create private networks between key collaborating sites within a science community or on a project-by-project basis. Examples: Radio Astronomy community sharing telescopes and correlation facilities High Energy Physics community distributing large data sets Smaller projects may link data repositories and computational resources among key participants – e.g. grid portals Network engineering and operations community can use these capabilities to support traffic engineering objectives We expect applications will use these circuit services to carry high-bandwidth, limited duration flows and will retain use of IP links for generic Internet access.

Collaborations The DICE group - Dante (GEANT2), Internet2, CANARIE, and ESnet Working closely with ESnet on interfacing OSCARS and HOPI - involves AAA work, using OSCARS interface “Stitching” project to describe data layer interconnections between segments of a P2P path Reporting back progress to the GLIF and other organizations For example, Phosphorus, in coordination with the SURFnet and University of Amsterdam participants Also having discussions with JGN2 Coordinating with OGF on various schema - topology, path computation, signaling Working with the appropriate standards bodies - ITU, IETF, and OIF

For more information: http://www. internet2. edu/network/ http://i2net For more information: http://www.internet2.edu/network/ http://i2net.blogspot.com scotter@internet2.edu Thank you! End Slide