The OptIPuter – From SuperComputers to SuperNetworks GEON Meeting San Diego Supercomputer Center, UCSD La Jolla, CA November 19, 2002 Dr. Larry Smarr Director, California Institute for Telecommunications and Information Technologies Professor, Dept. of Computer Science and Engineering Jacobs School of Engineering, UCSD

California Has Initiated Four New Institutes for Science and Innovation UCSB UCLA California NanoSystems Institute UCSF UCB California Institute for Bioengineering, Biotechnology, and Quantitative Biomedical Research UCI UCSD California Institute for Telecommunications and Information Technology Center for Information Technology Research in the Interest of Society UCSC UCD UCM

Cal-(IT) 2 An Integrated Approach to the Future Internet UC San Diego & UC Irvine Faculty Working in Multidisciplinary Teams With Students, Industry, and the Community The State’s $100 M Creates Unique Buildings, Equipment, and Laboratories

SDSC and Cal-(IT) 2 Have The Same Knowledge and Data Team by Construction! SDSC Data and Knowledge Systems Program Cal-(IT) 2 Knowledge and Data Engineering Laboratory Storage hardware Networked Storage (SAN) Grid Storage Filesystems, Database Systems Data Mining, Simulation Modeling, Analysis, Data Fusion Applications: Medical informatics, Biosciences, Ecoinformatics,… Knowledge-Based Integration Advanced Query Processing Visualization High speed networking sensornets instruments

The Move to Data-Intensive Science & Engineering- e-Science Community Resources ATLAS Sloan Digital Sky Survey LHC ALMA

Why Optical Networks Are Emerging as the 21 st Century Driver for the Grid Scientific American, January 2001

The Rapid Increase in Bandwidth is Driven by Parallel Lambdas On Single Optical Fibers (WDM) Parallel Lambdas Will Drive This Decade The Way Parallel Processors Drove the 1990s

CONTROLPLANECONTROLPLANE Clusters Dynamically Allocated Lightpaths Switch Fabrics Physical Monitoring Apps Middleware A LambdaGrid Will Be the Backbone for an e-Science Network Source: Joe Mambretti, NU

The Next S-Curves of Networking Exponential Technology Growth 0% 100% Research Experimental/ Early Adopters Production/ Mass Market Time Technology S-Curve Gigabit Testbeds Connections Program Internet2 Abilene DWDM Experimental Networks Lambda Grids ~1990s Networking Technology S-Curves Technology Penetration

Data Intensive Scientific Applications Require Experimental Optical Networks Large Data Challenges in Neuro and Earth Sciences –Each Data Object is 3D and Gigabytes –Data are Generated and Stored in Distributed Archives –Research is Carried Out on Federated Repository Requirements –Computing Requirements  PC Clusters –Communications  Dedicated Lambdas Over Fiber –Data  Large Peer-to-Peer Lambda Attached Storage –Visualization  Collaborative Volume Algorithms Response –OptIPuter Research Project

UIUC/NCSA Starlight (NU-Chicago) Argonne UChicago IIT UIC Illinois Century Network James R. Thompson Ctr City Hall State of IL Bldg 4 pair 12 pair 4 pair 2 pair 4 pair 18 pair 410 pair 12 pair 2 pair Level(3) 111 N. Canal McLeodUSA 151/155 N. Michigan Doral Plaza Qwest 455 N. Cityfront UC Gleacher 450 N. Cityfront Illinois’ I-WIRE The First State Dark Fiber Experimental Network Source: Charlie Catlett 12/2001

CENIC Plans to Create a Dark Fiber Experimental and Research Network The SoCal Component

From Telephone Conference Calls to Access Grid International Video Meetings Access Grid Lead-Argonne NSF STARTAP Lead-UIC’s Elec. Vis. Lab Can We Modify This Technology To Create a GEON Research Virtual Laboratory?

Fifteen Countries/Locations Proposing 28 Demonstrations: Canada, CERN, France, Germany, Greece, Italy, Japan, The Netherlands, Singapore, Spain, Sweden, Taiwan, United Kingdom, United States Applications Demonstrated: Art, Bioinformatics, Chemistry, Cosmology, Cultural Heritage, Education, High-Definition Media Streaming, Manufacturing, Medicine, Neuroscience, Physics, Tele-science Grid Technologies: Grid Middleware, Data Management/ Replication Grids, Visualization Grids, Computational Grids, Access Grids, Grid Portal iGrid 2002 September 24-26, 2002, Amsterdam, The Netherlands UIC Sponsors: HP, IBM, Cisco, Philips, Level (3), Glimmerglass, etc.

iGrid 2002 Was Sustaining 1-3 Gigabits/s Total Available Bandwidth Between Chicago and Amsterdam Was 30 Gigabit/s

The NSF TeraGrid A LambdaGrid of Linux SuperClusters NCSA 8 TF 4 TB Memory 240 TB disk Caltech 0.5 TF 0.4 TB Memory 86 TB disk Argonne 1 TF 0.25 TB Memory 25 TB disk TeraGrid Backbone (40 Gbps) SDSC 4.1 TF 2 TB Memory 250 TB disk This will Become the National Backbone to Support Multiple Large Scale Science and Engineering Projects Data Compute Visualization Applications Intel, IBM, Qwest Myricom, Sun, Oracle $53Million from NSF

From SuperComputers to SuperNetworks-- Changing the Grid Design Point The TeraGrid is Optimized for Computing –1024 IA-64 Nodes Linux Cluster –Assume 1 GigE per Node = 1 Terabit/s I/O –Grid Optical Connection 4x10Gig Lambdas = 40 Gigabit/s –Optical Connections are Only 4% Bisection Bandwidth The OptIPuter is Optimized for Bandwidth –32 IA-64 Node Linux Cluster –Assume 1 GigE per Processor = 32 gigabit/s I/O –Grid Optical Connection 4x10GigE = 40 Gigabit/s –Optical Connections are Over 100% Bisection Bandwidth

The OptIPuter is an Experimental Network Research Project Driven by Large Neuroscience and Earth Science Data –EarthScope and BIRN Multiple Lambdas Linking Clusters and Storage –LambdaGrid Software Stack –Integration with PC Clusters –Interactive Collaborative Volume Visualization –Lambda Peer to Peer Storage With Optimized Storewidth –Enhance Security Mechanisms –Rethink TCP/IP Protocols NSF Large Information Technology Research Proposal –UCSD and UIC Lead Campuses—Larry Smarr PI –USC, UCI, SDSU, NW Partnering Campuses –Industrial Partners: IBM, Telcordia/SAIC, Chiaro Networks, CENIC $13.5 Million Over Five Years

Metro Optically Linked Visualization Walls with Industrial Partners Set Stage for Federal Grant Driven by SensorNets Data –Real Time Seismic –Environmental Monitoring –Distributed Collaboration –Emergency Response Linked UCSD and SDSU –Dedication March 4, 2002 Linking Control Rooms Cox, Panoram, SAIC, SGI, IBM, TeraBurst Networks SD Telecom Council UCSD SDSU 44 Miles of Cox Fiber

OptIPuter NSF Proposal Partnered with National Experts and Infrastructure Vancouver Seattle Portland San Francisco Los Angeles San Diego (SDSC) NCSA SURFnet CERN CA* net4 Asia Pacific Asia Pacific AMPATH PSC Atlanta CA*net4 Source: Tom DeFanti and Maxine Brown, UIC NYC TeraGrid DTFnet CENIC Pacific Light Rail Chicago UIC NU USC UCSD, SDSU UCI

switch Cluster – Disk Disk – Disk Viz – Disk DB – Cluster Cluster – Cluster Medical Imaging and Microscopy Chemistry, Engineering, Arts San Diego Supercomputer Center Scripps Institution of Oceanography Chiaro Enstara OptIPuter LambdaGrid Enabled by Chiaro Networking Router

½ Mile The UCSD OptIPuter Deployment SIO SDSC CRCA Phys. Sci - Keck SOM JSOE Preuss 6 th College Phase I, Fall 02 Phase II, 2003 SDSC Annex To Other OptIPuter Sites Collocation point Node M The UCSD OptIPuter LambdaGrid Testbed Earth Sciences SDSC Arts Chemistry Medicine Engineering High School Undergrad College Phase I, Fall 02 Phase II, 2003 SDSC Annex To Other OptIPuter Sites Collocation point Collocation

Fast polygon and volume rendering with stereographics GeoWall Earth Science GeoFusion GeoMatrix Toolkit Underground Earth Science Rob Mellors and Eric Frost, SDSU SDSC Volume Explorer Dave Nadeau, SDSC, BIRN SDSC Volume Explorer Neuroscience Anatomy Visible Human Project NLM, Brooks AFB, SDSC Volume Explorer 3D APPLICATIONS: + = OptIPuter Transforms Individual Laboratory Visualization, Computation, & Analysis Facilities The Preuss School UCSD OptIPuter Facility