1. Hybrid Packet-Optical Infrastructure Tom DeFanti, Maxine Brown, Joe Mambretti & Linda Winkler

2. The Hard Problems New protocols needed: the Internet is not designed for single large-scale users Circuits are not scalable, but neither are router$ All intelligence has to be on the edge Tuning compute, data, visualization, networking using clusters to get even a simple order of magnitude improvement is non-trivial Security at 10Gb line speed

3. Knowing the User’s Bandwidth Requirements DSLGigE LAN C A B A -> Need full Internet routing B -> Need VPN services on/and full Internet routing C -> Need very fat pipes, limited multiple Virtual Organizations Source: Cees de Laat, UvA Number of users Bandwidth consumed

4. We Can Now Build Lambda Grids! Importance for Applications –To create high-performance trials of new technologies that support application-dictated software toolkits, middleware, computing and networking –To provide known and knowable characteristics with deterministic and repeatable behavior on a persistent basis, while encouraging experimentation with innovative concepts –It isn’t science if you can’t repeat it!

5. Performance Monitoring & Control IIIIIIIIIIIIII IIIIIIIIIIIIII IIIIIIIIIIIIII IIIIIIIIIIIIII IIIIIIIIIIIIII IIIIIIIIIIIIII IIIIIIIIIIIIII IIIIIIIIIIIIII OMNInet Testbed Experiments - MEMS-Based Dynamic Lambda Switching (MREN Used as Out-of-Band Control Channel)

6. Metro Lambda Grid (I-WIRE and OMNInet – An Advanced Photonic Metro Testbed, Joint Project with iCAIR et al) UIC Chicago

7. Illinois I-WIRE State US$7.5M Lambda Grid Source: Charlie Catlett, ANL

8. Illinois’ I-WIRE: Distributed Cluster Computing Research Areas Displays/VR Collaboration Rendering Applications Data Mining NCSA Argonne UIC/EVL Research Areas Latency-Tolerant Algorithms Interaction of SAN/LAN/WAN technologies Clusters UIUC CS StarLight Source: Charlie Catlett

9. TeraGrid @ StarLight TeraGrid, an NSF-funded Major Research Equipment initiative, has its Illinois hub located at StarLight.

10. 20 TF Linux TeraGrid 32 5 5 Router or Switch/Router 32 quad-processor McKinley Servers (128p @ 4GF, 8GB memory/server) Fibre Channel Switch HPSS ESnet HSCC MREN/Abilene Starlight 10 GbE 16 quad-processor McKinley Servers (64p @ 4GF, 8GB memory/server) NCSA 500 Nodes 8 TF, 4 TB Memory 240 TB disk SDSC 256 Nodes 4.1 TF, 2 TB Memory 225 TB disk Caltech 32 Nodes 0.5 TF 0.4 TB Memory 86 TB disk Argonne 64 Nodes 1 TF 0.25 TB Memory 25 TB disk IA-32 nodes 4 Juniper M160 OC-12 OC-48 OC-12 574p IA-32 Chiba City 128p Origin HR Display & VR Facilities = 32x 1GbE = 64x Myrinet = 32x FibreChannel Myrinet Clos Spine = 8x FibreChannel OC-12 OC-3 vBNS Abilene MREN Juniper M40 1176p IBM SP Blue Horizon OC-48 NTON 32 24 8 32 24 8 4 4 Sun E10K 4 1500p Origin UniTree 1024p IA-32 320p IA-64 2 14 8 Juniper M40 vBNS Abilene Calren ESnet OC-12 OC-3 8 Sun Starcat 16 GbE = 32x Myrinet HPSS 256p HP X-Class 128p HP V2500 92p IA-32 24 Extreme Black Diamond 32 quad-processor McKinley Servers (128p @ 4GF, 12GB memory/server) OC-12 ATM Calren 2 2 Source: Rick Stevens 12/2001

11. The US National Lambda Rail The Cost of a Couple of University Buildings Source: John Silvester, Dave Reese, Tom West, CENIC

12. USAWaves Over AT&T’s Next Generation Network

13. CA*net 4 Physical Architecture Vancouver Calgary Regina Winnipeg Ottawa Montreal Toronto Halifax St. John’s Fredericton Charlottetown Chicago Seattle New York Los Angeles Miami Europe Dedicated Wavelength or SONET channel OBGP switches Optional Layer 3 aggregation service Large channel WDM system

14. SURFnet5 Partners BT and Cisco 15 PoPs connected by thirty 10 Gbit/s lambdas Dual stack IPv4 and IPv6 500,000 users 84 institutes connected at Gbit/s level Source: Kees Neggars

15. StarLight in Chicago: A 1GigE and 10GigE Exchange Abbott Hall, Northwestern University’s Chicago downtown campus Operational since summer 2001, StarLight is a 1GigE and 10GigE switch/router facility for high- performance access to participating networks. StarLight is equipped for optical switching facility for wavelengths. www.startap.net/starlight/NETWORKS/

16. StarLight US and International Networks as of August 2003 Abilene 10Gb ESnet (DOE) DREN (DOD) NREN (NASA) AMPATH (South America) CA*net4 (Canada) SURFnet (Netherlands) CERN/DataTAG TransPAC/APAN (Asia) NaukaNET (Russia) ASnet (Taiwan) Others via STAR TAP OC-12 and Abilene transit See http://loadrunner.uits.iu.edu/mrtg-monitors/starlight/ for statistics on usagehttp://loadrunner.uits.iu.edu/mrtg-monitors/starlight/

17. StarLight is StarLight is a Gigabit Ethernet and 10 Gigabit Ethernet exchange for R&E Production Networks (Force 10) And a GigE lambda exchange for US, Canada, Europe, Asia and South America for Experimental Networks And 1&10Gb MEMS-switched Research Network hub And the Chicago host to the NSF DTFnet, a 4x10Gb Network for the TeraGrid and DTF/ETF links to Abilene; NLR, USAWaves, others coming. A colo space: 66 racks for networking and computing, data management and visualization support equipment Using fiber and circuits from SBC, Qwest, AT&T, Global Crossing, T-Systems, Looking Glass, RCN, and I-WIRE

18. CANARIE 2xGigE circuits SURFnet 2xGigE circuits NetherLight StarLight

19. NL SURFnet CERN UK SuperJANET4 Abilene ESNET MREN IT GARR-B GEANT NewYork STAR-TAP STAR-LIGHT DataTAG project Major 2.5 Gbps circuits between Europe & USA FR INRIA ATRIUM /VTHD

20. What is TransLight? TransLight is a global-scale experimental networking initiative to support prototypes of the most aggressive e- science applications. TransLight consists of dozens of provisioned Gigabit Ethernet (GigE) circuits among North America, Europe and Asia via StarLight in Chicago, NetherLight in Amsterdam. Some 10Gb circuits are also available and schedulable.

21. European lambdas to US –8 GigEs Amsterdam—Chicago –8 GigEs London—Chicago Canadian lambdas to US –8 GigEs Chicago-Canada-NYC –8 GigEs Chicago-Canada-Seattle US lambdas to Europe –4 GigEs Chicago—Amsterdam –3 GigEs Chicago—CERN European lambdas –8 GigEs Amsterdam—CERN –2 GigEs Prague—Amsterdam –2 GigEs Stockholm—Amsterdam –8 GigEs London—Amsterdam TransPAC lambda (yellow) –1 GigE Chicago—Tokyo IEEAF lambdas (blue) –8 GigEs NYC—Amsterdam –8 GigEs Seattle—Tokyo TransLight Lambdas

22. TransLight Optical/Electronic Switches at StarLight and NetherLight 16-processor cluster Data plane N E T H E R L I G H T 8-processor cluster Router 8 GigE16 GigE 8 GigE16 GigE Control plane 15454 at StarLight 10 GigE 2 GigE 128x128 Calient MEMS Optical Switch N-processor cluster 10 GigE N GigE 15454 at NetherLight Control plane 2 GigE 2xOC-192 N GigE 64x64 Calient MEMS Optical Switch Router An OptIPuter Prototype

23. What is the Global Lambda Integrated Facility (GLIF)? GLIF is a global-scale experimental facility being designed to support advanced applications and to develop new technologies. The GLIF is a multi-organizational international partnership The GLIF will be based on leading edge optical technologies Last Meeting – August, Iceland after NORDUnet conference, 3 rd year of Global Lamda Workshops

24. Bring Us Your Lambdas! Please bring your lambdas to StarLight, NetherLight, CERN, UKLight, CzechLight, NorthernLight, … Build a hub like StarLight Propose an application or network experiment See www.startap.net/starlight and www.startap.net/translight www.startap.net/starlight www.startap.net/translight

25. Thank You! StarLight planning, research, collaborations, and outreach efforts are made possible, in major part, by funding from: –National Science Foundation (NSF) awards ANI-9980480, ANI-9730202, EIA- 9802090, EIA-9871058, ANI-0225642, and EIA-0115809 –NSF Partnerships for Advanced Computational Infrastructure (PACI) cooperative agreement ACI-9619019 to NCSA –State of Illinois I-WIRE Program, and major UIC cost sharing –Northwestern University for providing space, engineering and management NSF/CISE/ANIR and DoE/Argonne National Laboratory for StarLight and I- WIRE network engineering and design NSF/CISE/ACIR and NCSA/SDSC for DTF/TeraGrid/ETF opportunities UCAID/Abilene for Internet2 and ITN/GTRN transit; IU for the GlobalNOC CA*net4 for North American transport Bill St. Arnaud of CANARIE, Kees Neggers of SURFnet, Olivier Martin of CERN and Harvey Newman of CalTech for networking leadership Larry Smarr of Cal-(IT) 2 for I-WIRE and OptIPuter leadership