1 ESnet Update Summer 2007 Joint Techs Workshop Joe Burrescia ESnet General Manager July 16,2007 Energy Sciences Network Lawrence Berkeley National Laboratory Networking for the Future of Science

2 TWC SNLL YUCCA MT BECHTEL-NV PNNL LIGO INEEL LANL SNLA Allied Signal PANTEX ARM KCP NOAA OSTI ORAU SRS JLAB PPPL Lab DC Offices MIT ANL BNL FNAL AMES NREL LLNL GA DOE-ALB OSC GTN NNSA International (high speed) 10 Gb/s SDN core 10G/s IP core 2.5 Gb/s IP core MAN rings (≥ 10 G/s) Lab supplied links OC12 ATM (622 Mb/s) OC12 / GigEthernet OC3 (155 Mb/s) 45 Mb/s and less NNSA Sponsored (12) Joint Sponsored (3) Other Sponsored (NSF LIGO, NOAA) Laboratory Sponsored (6) 42 end user sites SINet (Japan) Russia (BINP) CA*net4 France GLORIAD (Russia, China) Korea (Kreonet2 Japan (SINet) Australia (AARNet) Canada (CA*net4 Taiwan (TANet2) Singaren ESnet IP core: Packet over SONET Optical Ring and Hubs ELP DC commercial peering points MAE-E PAIX-PA Equinix, etc. PNWGPoP/ PAcificWave ESnet 3 with Sites and Peers (Early 2007) ESnet core hubs IP Abilene high-speed peering points with Internet2/Abilene Abilene CERN (USLHCnet DOE+CERN funded) GÉANT - France, Germany, Italy, UK, etc NYC Starlight SNV Abilene JGI LBNL SLAC NERSC SNV SDN SDSC Equinix SNV ALB ORNL CHI MREN Netherlands StarTap Taiwan (TANet2, ASCC) NASA Ames AU SEA CHI-SL MAN LAN Abilene Specific R&E network peers Other R&E peering points UNM MAXGPoP AMPATH (S. America) ESnet Science Data Network (SDN) core R&E networks Office Of Science Sponsored (22) ATL NSF/IRNC funded Equinix

3 ESnet 3 Backbone as of January 1, 2007 Sunnyvale 10 Gb/s SDN core (NLR) 10/2.5 Gb/s IP core (QWEST) MAN rings (≥ 10 G/s) Lab supplied links Seattle San Diego Albuquerque El Paso Chicago New York City Washington DC Atlanta Future ESnet Hub ESnet Hub

4 ESnet 4 Backbone as of April 15, 2007 Clev. Boston 10 Gb/s SDN core (NLR) 10/2.5 Gb/s IP core (QWEST) 10 Gb/s IP core (Level3) 10 Gb/s SDN core (Level3) MAN rings (≥ 10 G/s) Lab supplied links Sunnyvale Seattle San Diego Albuquerque El Paso Chicago New York City Washington DC Atlanta Future ESnet Hub ESnet Hub

5 ESnet 4 Backbone as of May 15, 2007 SNV Clev. Boston 10 Gb/s SDN core (NLR) 10/2.5 Gb/s IP core (QWEST) 10 Gb/s IP core (Level3) 10 Gb/s SDN core (Level3) MAN rings (≥ 10 G/s) Lab supplied links Clev. Boston Sunnyvale Seattle San Diego Albuquerque El Paso Chicago New York City Washington DC Atlanta Future ESnet Hub ESnet Hub

6 ESnet 4 Backbone as of June 20, 2007 Clev. Boston Houston Kansas City 10 Gb/s SDN core (NLR) 10/2.5 Gb/s IP core (QWEST) 10 Gb/s IP core (Level3) 10 Gb/s SDN core (Level3) MAN rings (≥ 10 G/s) Lab supplied links Boston Sunnyvale Seattle San Diego Albuquerque El Paso Chicago New York City Washington DC Atlanta Denver Future ESnet Hub ESnet Hub

7 ESnet 4 Backbone Target August 1, 2007 Clev. Boston Houston Los Angeles 10 Gb/s SDN core (NLR) 10/2.5 Gb/s IP core (QWEST) 10 Gb/s IP core (Level3) 10 Gb/s SDN core (Level3) MAN rings (≥ 10 G/s) Lab supplied links Clev. Houston Kansas City Boston Sunnyvale Seattle San Diego Albuquerque El Paso Chicago New York City Washington DC Atlanta Denver Future ESnet Hub ESnet Hub

8 ESnet 4 Backbone Target August 30, 2007 Clev. Boston Houston Boise Los Angeles 10 Gb/s SDN core (NLR) 10/2.5 Gb/s IP core (QWEST) 10 Gb/s IP core (Level3) 10 Gb/s SDN core (Level3) MAN rings (≥ 10 G/s) Lab supplied links Clev. Houston Kansas City Boston Sunnyvale Seattle San Diego Albuquerque El Paso Chicago New York City Washington DC Atlanta Denver Future ESnet Hub ESnet Hub

9 ESnet 4 Backbone Target September 30, 2007 Clev. Boston Houston Boise Los Angeles 10 Gb/s SDN core (NLR) 10/2.5 Gb/s IP core (QWEST) 10 Gb/s IP core (Level3) 10 Gb/s SDN core (Level3) MAN rings (≥ 10 G/s) Lab supplied links Clev. Houston Kansas City Boston Sunnyvale Seattle San Diego Albuquerque El Paso Chicago New York City Washington DC Atlanta Denver Nashville Future ESnet Hub ESnet Hub

10 DC ESnet 4 Backbone Target September 15, 2008 Houston 10 Gb/s SDN core (NLR) 10/2.5 Gb/s IP core (QWEST) 10 Gb/s IP core (Level3) 10 Gb/s SDN core (Level3) MAN rings (≥ 10 G/s) Lab supplied links Clev. Houston Kansas City Boston Sunnyvale Seattle San Diego Albuquerque El Paso Chicago New York City Washington DC Atlanta Denver Los Angeles Nashville Future ESnet Hub ESnet Hub

11 ESnet4 Core networks Gbps by (10Gb/s circuits), Gbps by (100 Gb/s circuits) Cleveland Europe (GEANT) Asia-Pacific New York Chicago Washington DC Atlanta CERN (30+ Gbps) Seattle Albuquerque Australia San Diego LA Denver South America (AMPATH) South America (AMPATH) Canada (CANARIE) CERN (30+ Gbps) Canada (CANARIE) Asia- Pacific Asia Pacific GLORIAD (Russia and China) Boise Houston Jacksonville Tulsa Boston Science Data Network Core IP Core Kansas City Australia Core network fiber path is ~ 14,000 miles / 24,000 km 1625 miles / 2545 km 2700 miles / 4300 km Sunnyvale Production IP core (10Gbps) SDN core ( Gbps) MANs (20-60 Gbps) or backbone loops for site access International connections IP core hubs Primary DOE Labs SDN hubs High speed cross-connects with Ineternet2/Abilene Possible hubs USLHCNet

12 Typical ESnet 4 Hub M320 Router 7609 Switch 10G Performance Tester Peering Router Secure Term Server Power Controllers OWAMP Measurement Device

13 ESnet 4 Factiods as of July 16, 2007 Installation to date: o 10 new 10Gb/s circuits o ~10,000 Route Miles o 6 new hubs o 5 new routers 4 new switches -Total of 70 individual pieces of equipment shipped –Over two and a half tons of electronics o 15 round trip airline tickets for our install team -About 60,000 miles traveled so far…. -6 cities –5 Brazilian Bar-B-Qs/Grills sampled

14 ESnet Traffic Now Exceeding 2 PetaBytes/Month 2.4PBytes in May 2007 TeraBytes ESnet traffic historically has increased 10x every 47 months 1 PBytes in April 2006

15 ESnet Availability “5 nines” (>99.995%) “3 nines” (>99.5%) “4 nines” (>99.95%) Dually connected sites Note: These availability measures are only for ESnet infrastructure, they do not include site-related problems. Some sites, e.g. PNNL and LANL, provide circuits from the site to an ESnet hub, and therefore the ESnet-site demarc is at the ESnet hub (there is no ESnet equipment at the site. In this case, circuit outages between the ESnet equipment and the site are considered site issues and are not included in the ESnet availability metric.

16 OSCARS Overview Path Computation Topology Reachability Contraints Scheduling AAA Availability Provisioning Signalling Security Resiliency/Redundancy OSCARS Guaranteed Bandwidth Virtual Circuit Services On-demand Secure Circuits and Advance Reservation System

17 OSCARS Status Update ESnet Centric Deployment o Prototype layer 3 (IP) guaranteed bandwidth virtual circuit service deployed in ESnet (1Q05) o Layer 2 (Ethernet VLAN) virtual circuit service under development Inter-Domain Collaborative Efforts o Terapaths -Inter-domain interoperability for layer 3 virtual circuits demonstrated (3Q06) -Inter-domain interoperability for layer 2 virtual circuits under development o HOPI/DRAGON -Inter-domain exchange of control messages demonstrated (1Q07) -Initial integration of OSCARS and DRAGON has been successful (1Q07) o DICE -First draft of topology exchange schema has been formalized (in collaboration with NMWG) (2Q07), interoperability test scheduled for 3Q07 -Drafts on reservation and signaling messages under discussion o UVA -Integration of Token based authorization in OSCARS under discussion

18 ESnet Measurement Update perfSONAR Services Deployed o Measurement Archives serving Utilization information (Demo later today) o E2EMP serving end-to-end circuit status information o Visualization tools Bandwidth & Latency Measurement Points o Deployed at ESnet4 hubs as they are configured Working with the LHC community o To define a set of perfSONAR services that can meet the network measurement & monitoring needs of large distributed science application communities.