Translight/PacificWave John Silvester Executive Director of Center for Scholarly Technology Professor or Computer Engineering, USC Chair, CENIC Briefing to NSF-OISE, April 28, 2006 NSF OCI-IRNC Cooperative Agreement [# ]

Translight/PacificWave (TL/PW)  Cooperative Agreement [# ] under NSF IRNC Program  PI John Silvester (USC); Co-PI Ron Johnson (UW)  Other Key Partners CENIC, Jim Dolgonas CENIC, Jim Dolgonas PNWG, Jacqueline Brown PNWG, Jacqueline Brown University of Hawaii, David Lassner University of Hawaii, David Lassner AARnet, George McLaughlin AARnet, George McLaughlin  Approx $1M per year for 5 years  Awarded March 1, 2005

TL/PW Primary Objectives  Facilitate international R&E connections on US Pacific Coast  Assist in the termination (equipment, backhaul) of AARnet SXTransport links to Hawaii, Seattle and Los Angeles  Assist in build-out of Hawaii connectivity to Telescopes  Further develop PacificWave exchange capabilities  Assist in operation of IEEAF link Tokyo-Seattle  Assist in operation and support of other International R&E networks participating in PW  Develop and operate advanced capabilities to support optical interconnect and exchange needs of R&E networks

What is Pacific Wave?  Pacific Wave is a state-of-the-art international peering facility designed to serve advanced research and education.  Primary focus is interconnectivity among US R&E networks and International R&E networks from the Pacific Rim  One of the objectives was the desire for all participants to be able to peer with each other which led to the “distributed exchange” design

Context of Pacific Wave  The US States on the Pacific Coast [Alaska, Washington, Oregon, California, Hawaii] have collaborated in networking for several years. (e.g. US Pacific Consortium is an affiliate of APAN)  Seattle, Los Angeles, and Sunnyvale are “natural” places for exchange points  Many international links enter the US through Washington and California (and to a certain extent Hawaii although they tend to remain local there)  Led to the development of Pacific Wave – a collaboration between CENIC and PNWG to build out exchange capabilities on West Coast.

Who Operates Pacific Wave? A joint project of CENIC and Pacific Northwest Gigapop In collaboration with University of Southern California and University of Washington

Pacific Wave Layer 2 Exchange  Extensible peering exchange to allow layer 3 networks to peer easily over a layer 2 fabric  Nodes (currently) in Seattle, Sunnyvale, and Los Angeles. Connected by a 10GbE wave provisioned over National LambdaRail (2,241 kilometers)  AUP free  Supports IPv4 and IPv6; multicast enabled  Provides 24x7 NOC support.  Priced consistently from node to node  Allows participants to self-select their peering  Allows participants to connect to one-location and access participants at all Pacific Wave nodes  Supports advance applications  Welcomes any research or development network that can meet the minimum network configuration requirements (connect with a router; support BGP; support jumbo-frames)

International Participation

“Layer 3” R&E Networks and Pacific Wave AARNET GEMNET GLORIAD IEEAF KREONET MIMOS (Malaysia) QATAR SINET SINGAREN TANET/TWAREN TRANSPAC2 ABILENE (Internet2) CALREN (HPR) DREN ESNET NASA NLR-Packetnet PNWG Pacific Wave CA*NET4 CLARA CUDI

PacificWave Seattle Seattle Westin CISCO 6509 NLR Lambda to LAX (via Sunnyvale) Abilene AARNet CA*NET4 GEMnet SingAREN TANET2/TWAREN ATTBI/Comcast DREN KREONet2 ESnet Microsoft PNWGP NLR Lambda to STARLIGHT WIDE/TLEX (IEEAF) NLR-Packetnet Does not show “lightpath” connections 1 Gigabit 10 Gigabit

Pacific Wave - Sunnyvale NLR TO SEATTLE PWAVE L3 SVL 6509 CENIC SVL HPR NLR TO LA PWAVE PAIX ESNET Does not show “lightpath” connections 1 Gigabit 10 Gigabit

Pacific Wave - Los Angeles 1 Wilshire W 7th W 7th 6509 CalREN-HPR 10 GE NLR Lambda To Seattle PWave (via Sunnyvale) Abilene TWAREN TRANSPAC2 Los Nettos Qatar Foundation NII/SINET AARNet (3) Singaren Mimos Berhad (Malaysia) Cinegrid NLR Packetnet Does not show “lightpath” connections 1 Gigabit 10 Gigabit

Network Topology

The VLAN Architecture

IRNC TL/PW Pacific Connections CA*Net4 POP TλEX Tokyo PW-Seattle AARnet POP Sydney Hawaii Oahu PW-LA CLARA, CUDI POPs (Tijuana) AARnet-SX Transport IEEAF Link

TL-PW – West Coast Detail AARNet-SX(N) To Kahe Point (HI) Los Osos (CA) SLO (CA) Hillsboro (OR) Portland (OR) PW-Seattle NLR link PW-Los Angeles Pacific Wave N-S link AARNet-SX(S) To Spencer Beach (HI) SEA LAX Santa Barbara (CA) SNV (NLR) (CalREN-XD) Optics to carry new lambda over CalREN-XD from SLO to LA

TL/PW – Hawaii Detail SCCN OC-192 SCCN OC-192 AARNet-SX(S) To Sydney Spencer Beach (Big Island) Mauna Kea MKOCN (Summit) Hilo AARNet-SX(S) To Los Osos Waimea Kahe Point (Oahu) Hawaii Gigapop (Oahu) AARNet- SX(S) To Sydney AARNet-SX(N) To Portland

Mauna Kea Observatories Communication Network

Status of TL/PW as of April 2006  Engineering and technical support through PNWG and CENIC for all International connections into Pacific Wave  Northern AARnet Link Operational and in use Operational and in use  Southern AARnet Link Backhaul from Los Osos through San Luis Obispo to PW-LA over CALREN as OC-192 in place Backhaul from Los Osos through San Luis Obispo to PW-LA over CALREN as OC-192 in place ETE equipment for HI and LA on order (arriving as we speak). ETE equipment for HI and LA on order (arriving as we speak). Local connectivity in Hawaii – in process Local connectivity in Hawaii – in process Expected operation (test mode) end of May Expected operation (test mode) end of May

Changing Requirements for Exchanges  The Exchange points need to provide a wider array of services: Layer 3 Interconnect (routed) Layer 3 Interconnect (routed) Layer 3 Interconnect (over layer 2 switch) Layer 3 Interconnect (over layer 2 switch) Layer 2 Interconnect (Ethernet switching) Layer 2 Interconnect (Ethernet switching) Layer 1 Interconnect (wave switching) Layer 1 Interconnect (wave switching) Layer 0 Interconnect (fiber interconnect) Layer 0 Interconnect (fiber interconnect)  PacificWave is moving in this direction to allow us to continue to participate in the world of Global R&E networking

Pacific Wave Lightpath Interconnect  Primary focus has been on Layer 3 interconnect  There is a lot of interest in Layer 2 (“lightpaths”) and Layer 1 (“waves” and “lambdas”) interconnect. [GLIF, etc]  SXTransport(S) will be configured as 8x1GE  PW is deploying additional OC192 from LA to SEA configured as 8x1GE to allow interconnect for AARnet and other nets as demand grows.  Also plans in place to extend PW connectivity to Starlight (Chicago) to further enhance layer 3 and lightpath interconnect capability

Pacific Wave Lightpath Exchange To Canarie NLR To Chicago and Starlight San Diego NLR To Phoenix IEEAF to TλEX AARnet to AU, via HI HOPI SEA SNY LA

Lightpath Networks and PW Others in the planning stages TRANSPAC (?) AARNET (S) (to TλEX) IEEAF CA*NET4 NLR Waves NLR (Layer 2) CALREN-XD HOPI (Internet2) PNWG Pacific Wave

AARNet S Path PW Lightpath Interconnect Abilene Twaren Singaren PW-L3X:10G LAX-SNY PW-L2X (8x1G) LAX-SNY-SEA AARnet Router PW CalREN Router PW-L3X-LA to SNY, SEA CENIC to SLO to SD CENIC NLR Packetnet

Some Current Projects

Activities  Basically all International R&E traffic to Pacific Rim flows through PW  Recent Highlights iGRID iGRID SC’05 SC’05 PRAGMA PRAGMA APAN APAN INTERNET2 meetings INTERNET2 meetings

Examples  January 2005: Huygens Titan probe data transfer: 3Tbytes of satellite data distributed and collated for analysis to locations in Amsterdam and Australia  January 2005: Tele-microscopy demo between University of Hawaii and UCSD (Lariat network)  January 2005: PTC Honolulu 270Mbps interactive HD video conferencing with Australia and Seattle  January 2005: Seattle—Tokyo, uncompressed HDTV 1.5 Gbps Smarr HD lecture between Seattle and JGN meeting  June 2005: Chicago-Seattle-UCSD OptiPuter HD permanent interactive uncompressed HDTV video- conference link

Examples – iGRID05  4k interactive HD between Tokyo & SD  Concurrent n-way real-time low-latency uncompressed 1080 line (ie >1gb/s per stream) HDTV teleconferencing between Australia, Japan, Korea, Ann Arbor, Seattle and San Diego  100 megapixel remote viz. Illinois-SD  Real-time flat-panel stereo, remotely rendered  Clusters in Chicago & Amsterdam loosely coupled compute demo: performance = to single machine room alternative!!  Neptune/‘Looking’ real-time HDTV from 2km below the surface showing volcanic vents in the northeast pacific

Examples  November 14-17, 2005: SC05 Seattle. About 50 10gb/s waves, petabytes of storage, leading edge visualization systems and teraflops of computing assembled for a week computational sciences etc conference for this.  ‘Persistent’ OptIPuter  Wide Area Storage GRIDs  The Next Disaster Recovery Architectures that really work!?  Neptune Canada ‘in the water’

‘Dry’ Headend – Nodes & Access points “Neptune” Ocean Observatories

Recent Stereo Video  I2 Spring MM from Stanford Medical School (Margaret Krebs, Testbed Mgr, Stanford University Medical Media and Information Technologies) ran a stereo video program test with CSIRO in Sydney on March 30th via Abilene and SX.  I2 Spring MM from Stanford Medical School (Margaret Krebs, Testbed Mgr, Stanford University Medical Media and Information Technologies) ran a stereo video program test with CSIRO in Sydney on March 30th via Abilene and SX.  They ran about a 90Mbps stream for the test. No problems through AARnet N and across AARnet.

iGrid 2005, San Diego Sept  iGrid 2005, Over 10 Gbps bidirectional traffic coexisted with production exchange traffic without detriment 4k line interactive Super-HD between Keio University and UCSD 4k line interactive Super-HD between Keio University and UCSD Live HD from the sea floor 100 miles off the Pacific Coast Live HD from the sea floor 100 miles off the Pacific Coast N-way uncompressed multicast HD video conferencing N-way uncompressed multicast HD video conferencing

ResearchChannel N-Way HD Multicast Video Conferencing

Does it Really Perform in Practice? …Yes!  7.5 Gbps of unicast HDTV over IP traffic ( Mbps flows)  3 Gbps of multicast replicated in Seattle switch node using PIM snooping  Traffic test performed in Cisco POC lab before the event

An Amazing Feat at SC05

47 10 Gbps Lambdas to be Exact

HDTV Conferencing Spanning Two Oceans Enabled by Pacific Wave

Traffic Flows at Supercomputing  Among other things, 5-way multicast replication of 3 Gbps, over 12 Gbps total traffic across the switch backplanes

Thank you John Silvester,