1. CENIC, PACIFIC WAVE, NLR, IEEAF: Infrastructure for Research and Education John Silvester Chair of the CENIC Board Vice-Provost for Scholarly Technology, University of Southern California LISHEP: Digital Divide and HEPGRID Workshop Rio de Janeiro, Brasil February 19 th, 2004

2. 1.CENIC History 2.CalREN and DCP 3.CalREN ONI 4.CalREN Today 5.National LambdaRail (aka Lightrail) 6.Pacific Wave 7.IEEAF 8.Pointers

3. Education in California – Overview University of California – 9 (10) campuses University of California – 9 (10) campuses 3 Private Research Universities – Caltech, Stanford, University of Southern California 3 Private Research Universities – Caltech, Stanford, University of Southern California California State University – 23 campuses California State University – 23 campuses Community Colleges – over 100 Community Colleges – over 100 Other independent institutions of higher education – over 100 Other independent institutions of higher education – over 100 K-12 schools – over 9000 K-12 schools – over 9000 Various government labs and university affiliated research institutes Various government labs and university affiliated research institutes

4. State of Networking in 1996 4-CNET connected the CSU system with extension out to community colleges 4-CNET connected the CSU system with extension out to community colleges Most institutions had their own commodity internet connections Most institutions had their own commodity internet connections UC operated some private leased lines UC operated some private leased lines No statewide K-12 network No statewide K-12 network Some individual county and school district networks Some individual county and school district networks

5. Impetus to Establish R&E Network in California Expansion of the NSF Connections program to facilitate access to the super-computer centers Expansion of the NSF Connections program to facilitate access to the super-computer centers Broadened to allow access to the high performance backbone (vBNS) for meritorious research Broadened to allow access to the high performance backbone (vBNS) for meritorious research Acted as a stimulus to bring the Research Universities to create a joint proposal Acted as a stimulus to bring the Research Universities to create a joint proposal Two initial goals: Two initial goals: –Reduce costs of access to high performance national networks by a collaborative approach –Facilitate communication and collaboration between institutions by building a California REesearch Network

6. How to Proceed? Decision that the network should be funded by the institutions rather than looking for a handout from the state or relying on Federal Government funding Decision that the network should be funded by the institutions rather than looking for a handout from the state or relying on Federal Government funding Startup funds came from a joint proposal on behalf of the R-1’s to the NSF connections program (12 campuses). This allowed the institutions to ease into the burden of covering the network costs over a 3 year period Startup funds came from a joint proposal on behalf of the R-1’s to the NSF connections program (12 campuses). This allowed the institutions to ease into the burden of covering the network costs over a 3 year period Founded a non-profit public benefit entity – CENIC – creating some separation from publicly funded entities (UC, CSU) Founded a non-profit public benefit entity – CENIC – creating some separation from publicly funded entities (UC, CSU) Lightweight organization (1.5 FTE for first 3 years) Lightweight organization (1.5 FTE for first 3 years)

7. 1.CENIC History 2.CalREN and DCP 3.CalREN ONI 4.CalREN Today 5.National LambdaRail (aka Lightrail) 6.Pacific Wave 7.IEEAF 8.Pointers

8. Phase 1 - 1998

9. CENIC Initial 4-Year Funding Charter Associates$16 M42% CENIC Partners $2 M 5% Corporate Discounts$14 M37% Federal Funding $6 M16%

10. Developments Direct funding from UC for a central valley link to link CalREN-north (SF) and CalREN-south (LA-SD) Direct funding from UC for a central valley link to link CalREN-north (SF) and CalREN-south (LA-SD) ISP – took advantage of the aggregated buying power to leverage ISP contracts with multiple vendors – significant savings (additional savings from Quilt pricing) ISP – took advantage of the aggregated buying power to leverage ISP contracts with multiple vendors – significant savings (additional savings from Quilt pricing) Peering – took advantage of peering opportunities at PAIX and MAE-LA Peering – took advantage of peering opportunities at PAIX and MAE-LA DCP – Digital California Project – extend connectivity to (public) K-12 schools DCP – Digital California Project – extend connectivity to (public) K-12 schools

11. Digital California Project Funded from California State in FY 2000/01 Funded from California State in FY 2000/01 $32M through University of California Implementation through CENIC $32M through University of California Implementation through CENIC Program Steering Committee - Advisory board of involved constituents from K-20 Program Steering Committee - Advisory board of involved constituents from K-20 Subsequent budget reductions -- $26M - $21M - $14M – eliminated as direct funding for 2004-05 Subsequent budget reductions -- $26M - $21M - $14M – eliminated as direct funding for 2004-05

13. 1.CENIC History 2.CalREN and DCP 3.CalREN ONI 4.CalREN Today 5.National LambdaRail (aka Lightrail) 6.Pacific Wave 7.IEEAF 8.Pointers

14. California R&E Networks in 2000 CALREN-2 – advanced services network, owned and operated by CENIC (Corporation for Educational Network Initiatives in California) CALREN-2 – advanced services network, owned and operated by CENIC (Corporation for Educational Network Initiatives in California) 4CNet – owned and operated by Cal. State University 4CNet – owned and operated by Cal. State University DCP – K-12 network- owned and operated by CENIC; being implemented DCP – K-12 network- owned and operated by CENIC; being implemented Los Nettos – dark fiber based CWDM Metro network owned and operated by USC and Caltech - last of the NSFnet regionals. Los Nettos – dark fiber based CWDM Metro network owned and operated by USC and Caltech - last of the NSFnet regionals.

15. Phase 2 - 2000

16. Redesigning CalREN In late 1999, with the approaching end of current SONET contracts (late 2002), CENIC began thinking about the next generation CalREN In late 1999, with the approaching end of current SONET contracts (late 2002), CENIC began thinking about the next generation CalREN User demand: User demand: –Reliable ‘commodity’ network –High bandwidth (IP) network in support of research (Abilene) –Some demand for dedicated resources –Significant demand for experimental and research networks at level 3, level 2 and even level 1 This formed the thinking for an integrated infrastructure built on dark fiber This formed the thinking for an integrated infrastructure built on dark fiber

18. CALREN-DC Digital California IP based network. 2.5-10 GB IP based network. 2.5-10 GB Serves-140 H.E institutions; 8000+ elementary and high schools Serves-140 H.E institutions; 8000+ elementary and high schools 8.0 million+ student, faculty and staff users 8.0 million+ student, faculty and staff users I2 connectivity and commodity ISP services. I2 connectivity and commodity ISP services.

19. CALREN-HPR High Performance Research Network IP network: 10Gb, potentially several wavelengths IP network: 10Gb, potentially several wavelengths 50+ Research institutions, National Laboratories and San Diego Super-computing Center in California 50+ Research institutions, National Laboratories and San Diego Super-computing Center in California California component of Internet2 with 10G and OC- 12 connections California component of Internet2 with 10G and OC- 12 connections Serves hundreds of researchers, demanding applications Serves hundreds of researchers, demanding applications

20. CALREN-XD Experimental/Development Network 10.0 Gb Wavelengths and Dark Fiber 10.0 Gb Wavelengths and Dark Fiber Potential for Wavelength Switching and Special Network Configurations Potential for Wavelength Switching and Special Network Configurations Special applications, e.g. Teragrid Special applications, e.g. Teragrid Serves Network Researchers in California Research Institutions – primarily four UC Institutes; USC’s ISI; Stanford; and Caltech Serves Network Researchers in California Research Institutions – primarily four UC Institutes; USC’s ISI; Stanford; and Caltech

21. 1.CENIC History 2.CalREN and DCP 3.CalREN ONI 4.CalREN Today 5.National LambdaRail (aka Lightrail) 6.Pacific Wave 7.IEEAF 8.Pointers

22. CALREN - today 3 backbones – one commodity, one production, one research oriented, sharing physical resources where applicable 3 backbones – one commodity, one production, one research oriented, sharing physical resources where applicable Integrated at the physical and operations level, separable at the link and network levels Integrated at the physical and operations level, separable at the link and network levels Separate local solution from long-haul solution (due to different possibilities, players) Separate local solution from long-haul solution (due to different possibilities, players) Combination of dark fiber and wavelengths Combination of dark fiber and wavelengths

23. CalREN Backbone Network Phase 3 - 2003

24. CalREN Waves CalREN Waves HPR Teragrid HPR & DC CalREN DC San Diego Tustin Los Angeles Santa Barbara Sunnyvale Oakland San Luis Obispo Soledad Fergus Fresno Bakersfield Sacramento UC Davis Triangle Court

25. Digital California Overlay

26. Calren/DC/HPR/XD POP Architecture Long Haul OC48/OC192/10GigE DWDM Gig E 10 Gig E or OC192 CalRen DC HPR XD Campus or Metro Interconnect 15500 CalREN/DC HPR XD switch/mux

27. Historical Review of CalREN 1996- Initial meetings 1997- NSF proposal for CalREN2 funded; CENIC incorporated 1998- CalREN2 operational, connect to vBNS 1999- Connect to Abilene; ISP service launched 2000- DCP project launched 2001- ONI project launched; First DCP nodes operational 2002- ONI vendors selected; DCP 90% completed 2003- CalREN-DCP (optical edition) operational 2004- CalREN-HPR operational 2004- Pacific Wave linkup 2004- XD links operational as NLR comes up

28. 1.CENIC History 2.CalREN and DCP 3.CalREN ONI 4.CalREN Today 5.National LambdaRail (aka Lightrail) 6.Pacific Wave 7.IEEAF 8.Pointers

29. Expanding the reach? Multi-State interest in a larger geographic fiber based infrastructure Multi-State interest in a larger geographic fiber based infrastructure Fiber vendor very interested in selling National footprint Fiber vendor very interested in selling National footprint

30. National Light Rail - concept Dark Fiber National footprint Dark Fiber National footprint Serves very high-end experimental and research applications including network research Serves very high-end experimental and research applications including network research 4 - 10GB Wavelengths initially 4 - 10GB Wavelengths initially Capable of 40 10Gb wavelengths at build-out Capable of 40 10Gb wavelengths at build-out Partnership model (including corporate partners) Partnership model (including corporate partners)

31. Many different perspectives “NLR… “NLR… … aims to reenergize innovative R&D into next generation networking technologies, protocols, services, and apps.” … is a virtual laboratory.” … will contribute to the (NSF) Cyberinfrastructure that is critical to progress in every field of science & engineering.” … will help fuel the growth of the TeraGrid and computational science in general.” … motivates our planned regional optical network.” … is a hedge timed to hit the trough of the Internet/telecom economy.” … is both an experimental facility and a complex, multi-dimensional experiment in and of itself.” ALL OF THE ABOVE!

33. Distinguishing features - I NLR, Inc. has been established as a non-profit organization (May 2003, 501(c)(3) tax-exempt status being sought) NLR, Inc. has been established as a non-profit organization (May 2003, 501(c)(3) tax-exempt status being sought) Partnership with industry (CISCO is a major partner/contributor with special interest in promoting network research) Partnership with industry (CISCO is a major partner/contributor with special interest in promoting network research) Largest higher-ed owned/managed optical networking & research facility in the world Largest higher-ed owned/managed optical networking & research facility in the world –~10,000 route-miles of dark fiber (Level 3) –Four 10-Gbps ’s provisioned at outset First & foremost, an experimental platform for research First & foremost, an experimental platform for research –Optical, switching & network layers –Research committee (with 2 board seats) –Experimental support center –50% of capacity available for network research

34. Distinguishing features - II Sparse backbone’ topology Sparse backbone’ topology Use of high speed Ethernet for WAN transport Use of high speed Ethernet for WAN transport –1O Gigabit Ethernet LAN PHY is primary interface –Traditional OC-192 SONET available, too –Additional ’s (up to 40 on a given segment) can be provisioned with pricing tied to incremental cost Unprecedented level of self-capitalization for national networking initiative by higher ed participants Unprecedented level of self-capitalization for national networking initiative by higher ed participants –~$80M budget for full national backbone (CapEx + 5-yrs OpEx) –Each contribution (typically $5M) assumed to be ‘sunk cost’ Each participant/node has concurrent responsibility for developing optical networking capabilities and sustaining performance in their region Each participant/node has concurrent responsibility for developing optical networking capabilities and sustaining performance in their region

35. Current Members and Associates CENIC CENIC Pacific Northwest Gigapop Pacific Northwest Gigapop Pittsburgh SC Pittsburgh SC Duke Univ./NCLR Duke Univ./NCLR MATP/Va. Tech MATP/Va. Tech Cisco Systems Cisco Systems Internet2 Internet2 Florida LambdaRail Florida LambdaRail Georgia Tech Georgia Tech CIC CIC Pending: Pending: –Texas Consortium –New York Area Consortium –Mountain region consortium

36. NLR Project Segment Deployment Schedule (approx) Chicago to Pittsburgh11/03 Chicago to Pittsburgh11/03 Sunnyvale to Seattle4/04 Sunnyvale to Seattle4/04 Pittsburgh to Washington DC3/04 Pittsburgh to Washington DC3/04 Washington DC to Atlanta5/04 Washington DC to Atlanta5/04 Atlanta to Jacksonville8/04 Atlanta to Jacksonville8/04 Seattle to Denver7/04 Seattle to Denver7/04 Denver to Chicago8/04 Denver to Chicago8/04

37. Related Developments O(10k) miles of dark fiber have been acquired by the community for national and regional optical networks by CENIC, FiberCo (via Level 3), and others. O(10k) miles of dark fiber have been acquired by the community for national and regional optical networks by CENIC, FiberCo (via Level 3), and others. ATT makes major donation of fiber to USA Waves ATT makes major donation of fiber to USA Waves The Abilene Network has been upgraded to a 10-Gbps backbone and supports the research university community through initiatives such as IPv6 deployment and the Observatory. The Abilene Network has been upgraded to a 10-Gbps backbone and supports the research university community through initiatives such as IPv6 deployment and the Observatory. Internet2 plans to use one NLR wave for development of HOPI (hybrid optical packet interconnect) Internet2 plans to use one NLR wave for development of HOPI (hybrid optical packet interconnect)

38. 1.CENIC History 2.CalREN and DCP 3.CalREN ONI 4.CalREN Today 5.National LambdaRail (aka Lightrail) 6.Pacific Wave 7.IEEAF 8.Pointers

39. U.S. Pacific Coast Peering Collaboration CENIC (Corporation for Education Network Initiatives in California) and PNWGP (Pacific Northwest Gigapop) have combined efforts to create an advanced, extended peering facility on the U.S. West Coast. CENIC (Corporation for Education Network Initiatives in California) and PNWGP (Pacific Northwest Gigapop) have combined efforts to create an advanced, extended peering facility on the U.S. West Coast. Concept: an extensible, geographically dispersed peering fabric Concept: an extensible, geographically dispersed peering fabric Result: you connect at any one location on the fabric and have the option to peer with any other participant, regardless of where they are connected Result: you connect at any one location on the fabric and have the option to peer with any other participant, regardless of where they are connected

40. Current Model

41. Summer 2004 - LA & Seattle

42. Multi-Node Pacific Wave The Pacific Wave International Peering exchange facility will offer connection points initially in Los Angeles and Seattle, proximal to submarine cable landing sites on the U.S. Pacific Coast. The Pacific Wave International Peering exchange facility will offer connection points initially in Los Angeles and Seattle, proximal to submarine cable landing sites on the U.S. Pacific Coast. Connection points to be connected by 10GE link derived from NLR Connection points to be connected by 10GE link derived from NLR Expected operation by Summer 2004 Expected operation by Summer 2004

43. Future?

44. 1.CENIC History 2.CalREN and DCP 3.CalREN ONI 4.CalREN Today 5.National LambdaRail (aka Lightrail) 6.Pacific Wave 7.IEEAF 8.Pointers

45. New Public-Private Partnerships Global telecomm build-out of technical infrastructure provides new possibilities for economic development Global telecomm build-out of technical infrastructure provides new possibilities for economic development Current market conditions have resulted in capacity which is currently going unused -- cannot be sold. Current market conditions have resulted in capacity which is currently going unused -- cannot be sold. As a matter of social responsibility, this unused capacity could be made available for stimulating future applications and markets -- by donation for use by research and education institutions. As a matter of social responsibility, this unused capacity could be made available for stimulating future applications and markets -- by donation for use by research and education institutions.

46. Example: IEEAF The IEEAF represents one such partnership whose goal is to obtain donations of international bandwidth to enable a global collaboration in research and education. The IEEAF represents one such partnership whose goal is to obtain donations of international bandwidth to enable a global collaboration in research and education. Current donations have already linked US and Europe, and are linking US and Asia-Pacific. Current donations have already linked US and Europe, and are linking US and Asia-Pacific. This bandwidth helps enable global collaborations in research and education, in the true spirit of the Global Quilt Initiative. This bandwidth helps enable global collaborations in research and education, in the true spirit of the Global Quilt Initiative.

47. IEEAF Vision: The Global Quilt A Network of Networks, “stitched together” to create a common single fabric, and shared equally by all. This will be achieved through collaboration and community effort, until it covers the globe. A Network of Networks, “stitched together” to create a common single fabric, and shared equally by all. This will be achieved through collaboration and community effort, until it covers the globe. The IEEAF has no boundaries of “home” territory….. The IEEAF has no boundaries of “home” territory…..

48. IEEAF - What is it? U.S. 501.c.3 Not-for-profit corporation U.S. 501.c.3 Not-for-profit corporation Formed from original MOU between GEO and CENIC (Corporation for Educational Networking in California) Formed from original MOU between GEO and CENIC (Corporation for Educational Networking in California) Vision: Accelerate the global growth of advanced networking (Internet2) to achieve "universal educational access” to: Vision: Accelerate the global growth of advanced networking (Internet2) to achieve "universal educational access” to: –Enable and stimulate the rapid expansion of research and educational collaboration in many forms between teaching and learning institutions around the world. –Cultivate and promote practical solutions to delivering scalable, universally available and equitable access to suitable bandwidth and necessary network resources in support of these collaborations.

49. IEEAF - How does it work? Leverage global deregulation and new entrants into telco business Leverage global deregulation and new entrants into telco business Leverage private sector business relationships Leverage private sector business relationships –Geographic Network Affiliates, Inc. (GEO) Build donations into business deals (contracts) such as no-cost IRUs Build donations into business deals (contracts) such as no-cost IRUs

50. IEEAF - What does it do? Gets donated communications related assets Gets donated communications related assets Makes them available to existing institutions and networking organizations to put to work Makes them available to existing institutions and networking organizations to put to work Vehicle: Asset Steward Agreement Vehicle: Asset Steward Agreement

51. Tyco Telecomm Donation Summary Co-location space in NYC for Expanded International Exchange Point Co-location space in NYC for Expanded International Exchange Point Production R&E Bandwidth: 622 Mbps Production R&E Bandwidth: 622 Mbps –NY-London-Groningen (Netherlands) »Connects to IEEAF fiber to Amsterdam and Hamburg –US-Tokyo, Seoul, Shanghai, Taipei, Hong Kong, and Singapore Research 10 Gbps optical wavelength (preemptable) Research 10 Gbps optical wavelength (preemptable) –NY-London-Groningen (Netherlands) –US-Tokyo, Seoul, Shanghai, Taipei, Hong Kong, and Singapore 200sq.ft. Co-location space in each of global facilities 200sq.ft. Co-location space in each of global facilities Additional donations as global build-out continues Additional donations as global build-out continues

52. Tyco Global Network - Donations Connectivity Donations 622 Mbps +10 Gbps

53. Tyco Transpacific Donation Available last December, 622Mbps Debut at Busan! 10G available in March! Donated, Available when lit

54. TYCO Donation Key Elements TYCO Committed Assets TYCO Committed Assets –Production R&E Bandwidth: 622 Mbps –Research 10 Gbps optical wavelength (preemptable) –200 sq ft Co-location space in each of global facilities Trans-Atlantic and trans-Pacific already operational Trans-Atlantic and trans-Pacific already operational Others to be made available as business case (for TYCO) becomes feasible. Others to be made available as business case (for TYCO) becomes feasible.

55. USA Waves In U.S., IEEAF partnership with SURA resulted in 8,000 miles of trans-USA fiber donated by AT&T to the Southeastern Universities Research Association (SURA) through GEO and IEEAF efforts. In U.S., IEEAF partnership with SURA resulted in 8,000 miles of trans-USA fiber donated by AT&T to the Southeastern Universities Research Association (SURA) through GEO and IEEAF efforts.

56. IEEAF 8,000 Mile TransUSA Donation with SURA – Potential Footprint

57. AT&T Donation – Key Elements No-cost lease of 6,000 miles of dark fiber pair on NexGen network No-cost lease of 6,000 miles of dark fiber pair on NexGen network No-cost lease of additional 2,000 miles of NexGen fiber (O&M costs waived) - optical research pilot testbed projects No-cost lease of additional 2,000 miles of NexGen fiber (O&M costs waived) - optical research pilot testbed projects Donation of existing Cisco equipment Donation of existing Cisco equipment Very low cost IRUs (lease) for additional dark fiber Very low cost IRUs (lease) for additional dark fiber

58. IEEAF – other Activities Many other donations in progress Many other donations in progress Key is to look for partnership opportunities Key is to look for partnership opportunities GEO as a key player in IEEAF often has access to opportunities at early stages of development and has succeeded in leveraging opportunities. GEO as a key player in IEEAF often has access to opportunities at early stages of development and has succeeded in leveraging opportunities.

59. Developing a Global Quilt Global patchwork of regional initiatives linked with international initiatives resulting global connectivity for R&E worldwide Global patchwork of regional initiatives linked with international initiatives resulting global connectivity for R&E worldwide Much progress but work still to be done... Much progress but work still to be done...

60. Think Globally – Act Locally StrategicOpportunistic

61. 1.CENIC History 2.CalREN and DCP 3.CalREN ONI 4.CalREN Today 5.National LambdaRail (aka Lightrail) 6.Pacific Wave 7.IEEAF 8.Pointers

62. For More Information silvester@usc.edu CENIC: www.cenic.org www.cenic.org NLR: www.nationallambdarail.org www.nationallambdarail.org Pacific Wave: www.pacificwave.net www.pacificwave.net IEEAF: www.ieeaf.org www.ieeaf.org Acknowledgements Tom West: CENIC, NLR Steve Corbato: Internet2, NLR Don Riley: IEEAF Ed Fantegrossi: GEO, IEEAF