"The OptIPuter: Enabling Campus, State, National, and Planetary-Scale LambdaGrids" LONI Forum Baton Rouge, LA September 2, 2004 Dr. Larry Smarr Director, California Institute for Telecommunications and Information Technologies Harry E. Gruber Professor, Dept. of Computer Science and Engineering Jacobs School of Engineering, UCSD

OptIPuter Application Motivation: Interactive Analysis of Remote Multi-Gigabyte Data Objects Hundreds Of Megapixels 2-D Images –Satellite Imaging and Remote Sensing –Telescopes or Microscopy GigaZone 3-D (1k x 1k x 1k) Objects –Supercomputer Simulations –Seismic or Medical Imaging Goal: Remote Interactive Analysis and Visualization –Virtually Impossible Today –New Technology of Dedicated Lightpaths –Bring These Capabilities to Laboratory Researchers

Cosmic Simulator with a Billion Zone and Gigaparticle Resolution AMR or Unigrid 8-64 Times Mass Resolution Can Simulate First Galaxies One Gigazone Run: –Output ~10 TeraByte –Snapshot is 100 GB –Must Visually Analyze Compare with Sloan Survey Source: Mike Norman, UCSD SDSC Blue Horizon (2004) Unigrid

OptIPuter Driver: On-Line Microscopes Creating Very Large Biological Montage Images 2-Photon Laser Confocal Microscope –High Speed On-line Capability Montage Image Sizes Exceed 16x Highest Resolution Monitors –~150 Million Pixels! Use Graphics Cluster with Multiple GigEs to Drive Tiled Displays Source: David Lee, NCMIR, UCSD IBM 9M Pixels

Earth System Enterprise-Data Lives in Distributed Active Archive Centers (DAAC) SEDAC (0.1 TB) Human Interactions in Global Change GES DAAC-GSFC (1334 TB) Upper Atmosphere Atmospheric Dynamics, Ocean Color, Global Biosphere, Hydrology, Radiance Data ASDC-LaRC (340 TB) Radiation Budget,Clouds Aerosols, Tropospheric Chemistry ORNL (1 TB) Biogeochemical Dynamics EOS Land Validation NSIDC (67 TB) Cryosphere Polar Processes LPDAAC-EDC (1143 TB) Land Processes & Features PODAAC-JPL (6 TB) Ocean Circulation Air-Sea Interactions ASF (256 TB) SAR Products Sea Ice Polar Processes GHRC (4TB) Global Hydrology EOS Aura Satellite Has Been Launched Challenge is How to Evolve to New Technologies

However, Average Throughput of NASA Data Products to End User is Only Megabits/s Tested from GSFC-ICESAT August

Landsat7 Imagery 100 Foot Resolution Draped on elevation data High Resolution Aerial Photography Generates Images With 10,000 Times More Data than Landsat7 Shane DeGross, Telesis USGS New USGS Aerial Imagery At 1-Foot Resolution ~350,000x350,000 Pixel Images of 350 US Cities 100 Billion Pixel Images!

States are Acquiring Their Own Dark Fiber Networks -- Illinoiss I-WIRE and Indianas I-LIGHT Source: Charlie Catlett, ANL Today Nearly Two Dozen States are Creating Dark Fiber NetworksIncluding the Most Advanced LONI! Indiana

Regional Network Evolution The Fundamental Nature of Regional Networking is Rapidly Changing –The GigaPoP Model Based on Provisioned, High- Capacity Services Steadily Is Being Replaced –On The Metro and Regional Scales A Model of Facility-Based Networking Built with Owned Assets has Emerged –Regional Optical Networks (RONs) –Notably, this Change Increases the Importance of Regional Networks in the Traditional Three-Level Hierarchy of U.S. R&E Advanced Networking Source: Steve Corbató, Internet2

Leading & Emerging Regional Optical Networks California (CALREN) Colorado (FRGP/BRAN) Connecticut (Conn. Education Network) Florida (Florida LambdaRail) Georgia (Southern Light Rail) Indiana (I-LIGHT) Illinois (I-WIRE) Louisiana (LONI) Maryland, D.C. & northern Virginia (MAX) Michigan (MiLR) Minnesota New York + New England Region (NEREN) North Carolina (NC LambdaRail) Ohio (Third Frontier Network) Oklahoma (OneNet) Oregon Pacific Northwest (Lariat – Supported by NIH) Rhode Island (OSHEAN) SURA Crossroads Southeastern U.S.) Texas (LEARN) Utah Virginia (MATP) Wisconsin (WEROC) Source: Steve Corbató, Internet2

NLR Will Provide an Experimental Network Infrastructure for U.S. Scientists & Researchers Starting Fall 2004 National LambdaRail Partnership Serves Very High-End Experimental and Research Applications 4 x 10Gb Wavelengths Initially Capable of 40 x 10Gb wavelengths at Buildout

Dark Fiber: Gauging Community-Wide Progress Aggregate Dark Fiber Assets Acquired by/for U.S. R&E Optical Initiatives –CENIC (for CalREN & NLR) 6,200 (Route-miles) –FiberCo (via Level 3 for NLR & RONs) 5,650 –SURA (via AT&T) 6,000 –Plus 2,000 Route-Miles for Research –NLR Phase 2 (WilTel & Qwest) 4,000 –OARnet 1,600 –ORNL (via Qwest) 900 –Other Projects (IN,IL,MI,OR, …) 1,500+ Total (conservative estimate) 25,850+ –Over 55% of these Assets are in Regionals –Remainder held by NLR (11,250 Route-Miles) Source: Steve Corbató, Internet2

Global Lambda Integrated Facility (GLIF) Creates Metacomputers on the Scale of Planet Earth DWDM SURFnet 10 Gbit/s SURFnet 10 Gbit/s SURFnet 10 Gbit/s IEEAF 10 Gbit/s Dwingeloo ASTRON/JIVE Dwingeloo ASTRON/JIVE Prague CzechLight Prague CzechLight 2.5 Gbit/s NSF 10 Gbit/s Stockholm NorthernLight Stockholm NorthernLight CA*net4 2.5 Gbit/s New York MANLAN New York MANLAN Tokyo WIDE Tokyo WIDE 10 Gbit/s IEEAF 10 Gbit/s 2.5 Gbit/s Tokyo APAN Tokyo APAN Amsterdam NetherLight Amsterdam NetherLight Geneva CERN Geneva CERN London UKLight London UKLight Chicago StarLight Chicago StarLight Source: Kees Neggers, SURFnet Created in Reykjavik, Iceland 2003

The OptIPuter Project – Removing Bandwidth as an Obstacle In Data Intensive Sciences NSF Large Information Technology Research Proposal –Cal-(IT) 2 and UIC Lead CampusesLarry Smarr PI –USC, SDSU, NW, Texas A&M, Univ. Amsterdam Partnering Campuses Industrial Partners –IBM, Sun, Telcordia/SAIC, Chiaro Networks, Calient, Glimmerglass $13.5 Million Over Five Years Optical IP Streams From Lab Clusters to Large Data Objects NIH Biomedical Informatics NSF EarthScope and ORION siovizcenter.ucsd.edu/library/gallery/shoot1/index.shtml Research Network

OptIPuter Middleware Architecture for Distributed Virtual Computers Layer 4: XCP Node Operating Systems -configuration, Net Management Grid and Web Middleware – (Globus/OGSA/WebServices/J2EE) Physical Resources DVC #1 OptIPuter Applications DVC #2DVC #3 Layer 5: SABUL, RBUDP, Fast, GTP Real-Time Objects Security Models Data Services: DWTP Higher Level Grid Services Visualization DVC/ Middleware High-Speed Transport Optical Signaling/Mgmt Source: Andrew Chien, UCSD OptIPuter Software Systems Architect From Grids to LambdaGrids

Creating a Model for a Campus LambdaGrid

OMNInet The Metro Area OOO Tesbed NTON NTONC C DWDM RAM 10 Gb Lambdas

EVL 10GE OptIPuter CAVEWAVE Will Help Launch the National LambdaRail Next Step: Coupling NASA Centers to NSF OptIPuter Source: Tom DeFanti, OptIPuter co-PI

OptIPuter Scalable Visualization Systems: End User Clusters Connected With Gigabit Flows 3 Megapixels 20 Megapixels Earth Sciences are an OptIPuter Driver Major NASA Goddard Visit This Week Linux Cluster with Nvidia Graphics Cards

High Bandwidth Optical Fibers Will Enable High Definition Global Virtual Teaming In 2005 Cal-(IT)2 will Link Its Two Buildings Creating a Collaboration Laboratory UC Irvine UC San Diego

Ultra-Resolution OptIPuter Displays Utilizing Photonic Multicasting --Scaling to 100 Million Pixels Glimmerglass Switch Used to Multicast and Direct TeraVision Stream from One Tile to Another on the Geowall-2 Glimmerglass Switch Driven by Linux Graphics Clusters 30 Megapixel High-Resolution Visualizations = ~1 Gigapixel at 30fps = ~30Gb/s Bandwidth UIC-EVL

Currently Developing OptIPuter Software to Coherently Drive 100 MegaPixel Displays 55-Panel Display –100 Megapixel –11x5 21 LCDs Developed at EVL Driven by 30 Dual- Opterons (64-bit) 60 TB Disk 30 10GE interfaces Linked to OptIPuter Complementary to NASA ARC HyperWall

Special issue of Communications of the ACM (CACM): Blueprint for the Future of High-Performance Networking Introduction –Maxine Brown (guest editor) TransLight: A Global-scale LambdaGrid for e- Science –Tom DeFanti, Cees de Laat, Joe Mambretti, Kees Neggers, Bill St. Arnaud Transport Protocols for High Performance –Aaron Falk, Ted Faber, Joseph Bannister, Andrew Chien, Bob Grossman, Jason Leigh Data Integration in a Bandwidth-Rich World –Ian Foster, Robert Grossman The OptIPuter –Larry Smarr, Andrew Chien, Tom DeFanti, Jason Leigh, Philip Papadopoulos Data-Intensive e-Science Frontier Research –Harvey Newman, Mark Ellisman, John Orcutt