GigaPoP Transport Options: I-WIRE Positioning for the Bandwidth Tsunami Virtual Internet2 Member Meeting Oct 4, 2001 Linda Winkler Argonne National Laboratory

I-WIRE Background State Funded Dark Fiber Optical Infrastructure to support Networking and Applications Research $7.5M Total Funding $4M FY00-01 (in hand) $2.5M FY02 (approved 1-June-01) Additional $0.5M in FY03, FY04 Application Driven Access Grid: Telepresence & Media Computational Grids: Internet Computing Data Grids: Information Analysis New Technologies Proving Ground Optical Switching Dense Wave Division Multiplexing Advanced middleware infrastructure UIC ANL NCSA/UIUC UC NU / Starlight Star Tap IIT For more information see

Dark Fiber Location, location, location Metro Roughly $2 per meter per strand Lateral challenges Regional May require regeneration; regen, reshape, retime (3R) is expensive! Wide area Expensive due to 3R requirements Key is location of carrier runs $100K-$1M/month for 2200 mile OC-192 link

Dark Fiber (cont.) Key is the one time up front cost for the purchase of an IRU Maintenance and management are the buyers problem Obtain fiber characteristics as soon as possible (SMF vs. NZDSF, OTDR shots) Rapid provisioning possible allowing more adaptive networks The fiber industry is immature and underdeveloped, allowing sophisticated customers to negotiate much more attractive deals than would result from a standard RFP pricing exercise.

Indefeasible Right to Use (IRU) Services Terms 10, 15, 20 yrs Alternatives Long term capital lease Short term lease Managed service Considered as a physical asset which can be re-sold, traded or used a collateral. Cost can be amortized over lifetime which results in a monthly cost substantially below traditional telecommunication services. Be sure of contract conditions due to shaky nature of some vendors financial situation.

Dense Wavelength Division Multiplexing To ring or not to ring (ring vs. mesh) Redundancy (at what cost?) Survivability Protection Mesh topology benefits Migration, scaling Deployment speed Capacity Utilization Network Restoration Operating Costs Laser reach-3R issues and OEO Large portion of the cost No standards Number and spacing of lambdas are design variables One transponder per wavelength Beware OC-192/10GbE WAN PHY/10GbE LAN PHY are not all equal

TeraGrid Backplane NCSA/UIUC ANL UIC Multiple Carrier Hubs Starlight / NW Univ Ill Inst of Tech Univ of Chicago Indianapolis (Abilene NOC) St Louis GigaPoP I-WIRE StarLight International Optical Peering Point (see Los Angeles San Diego TeraGrid Backplane Abilene Chicago Indianapolis St. Louis Urbana OC-48 (2.5 Gb/s, Abilene) Multiple 10 GbE (Qwest) Multiple 10 GbE (I-WIRE Dark Fiber) Solid lines in place and/or available by October 2001 Dashed I-WIRE lines planned for Summer 2002 Charlie Catlett – Argonne National Laboratory

TeraGrid Proposed Backplane Architecture ChicagoLos Angeles Site Border Switch Cluster Aggregation Switch One Wilshire (Carrier Fiber Collocation Facility) DTF Backbone Core Switch Qwest San Diego POP Vendor TBD Long-Haul DWDM (Operated by site) Vendor TBD Metro DWDM (operated by site) Ciena CoreStream™ Long-Haul DWDM (Operated by Qwest) Vendor TBD Metro DWDM Vendor TBD Switch/Router* (256 Gb/s crossbar) DTF Local Site Resources and External Network Connections 2200mi 140mi25mi 2mi 115mi15mi 20mi 455 N. Cityfront Plaza (Qwest Fiber Collocation Facility) Vendor POP at JPL CaltechSDSC NCSA ANL Caltech Cluster (64p) SDSC Cluster (250p) NCSA Cluster (2000p) ANL Cluster (128p) 4 x 10 GbE *Initial Phase using IP Switch/Routers. This design will be evaluated beginning October Phase 2 (optical mesh) will also be evaluated prior to full DTF cluster deployment in early Charlie Catlett – Argonne National Laboratory

Wavelength Services OC48, OC192 vs. 10GbE Be sure of handoff specifications Management- determine level required Qwest/Teleglobe/(3)Link Global/Global Crossing service offerings Benefits Lower cost Customer responsible for protection Share cost of electronics-save capital investment Customer empowerment Potential for higher utilization of network Transparency of signal Flexibility

IP Routers and Switches Interoperability Which PHY (LAN vs. WAN) interface between DWDM and CPE? $$$ WAN PHYs tend to be pricey Is 10GbE really 10,000 Mb/s, or 9.3 Gb/s, or maybe 8 Gb/s? What is the largest individual stream you must support? Aggregates vs. large streams Currently no visibility into the optical layer

Next Steps: Optical Mux / Wavelength Router / Optical Wavelength Cross-connect System GigE / 2.5 Gb/s  10 Gb/s FDP Customer Interface x OTU 1 n 2.5 / 10 / 40 Gb/s DWDM Optical Mux/  Router/ Cross-Connect

Optical Switches Current tech is O-E-O Incoming signals are converted from Optical to Electrical Signal is switched electrically Outgoing signals are converted back to optical Up to 64x64 at 2.5 Gb/s Smart but slow Future tech will be O-O-O Operate in the all optical domain Bit rate independent A win at 40 Gb/s Below 10 Gb/s electronic switching will be hard to beat Challenge will be in management Fast but dumb

Optical Internetworking Progress Optical Internetworking Forum (OIF) UNI 1.0 specification in progress Based on domain services model Internet Engineering Task Force (IETF) Generalized Multiprotocol Label Switching (GMPLS) Incorporates Domain and Peer Models OIF and IETF are in sync OIF UNI 1.0 based on GMPLS Start with Domain Model and evolve Stay tuned. The ending of this story has not yet been written.

Wavelengths and the Future Wavelength services are causing a network revolution: Core long distance SONET Rings will be replaced by meshed networks using wavelength cross-connects Re-invention of pre-SONET network architecture Improved transport infrastructure will exist for IP/packet services Electrical/Optical grooming switches will emerge at edges Automated Restoration (algorithm/GMPLS driven) becomes technically feasible. Operational implementation will take beyond 2003

StarLight Optical Peering Point & Co-lo Facility 710 N. Lake Shore Dr. Chicago, IL Northwestern Campus Central downtown location Near carrier services serving Chicago loop Telephone switch room Colo space available Multiple carriers access Ameritech, AT&T, Qwest, Global Crossing, Global Crossing, MCI Worldcom Other builds possible