High Performance Cyberinfrastructure Discovery Tools for Data Intensive Research Larry Smarr Prof. Computer Science and Engineering Director, Calit2 (UC.

Slides:

Advertisements

Similar presentations

Three Disruptive Leadership Opportunities for Washington State to Live in the Future Keynote Talk Washington Innovation Summit: New Decade, New Partnerships,

Advertisements

High Performance Cyberinfrastructure Enabling Data-Driven Science Supporting Stem Cell Research Invited Presentation Sanford Consortium for Regenerative.

Sequencing Genomics: The New Big Data Driver IntermezzoTalk SURFnet7, Part of GigaPort3 Utrecht, Netherlands December 7, 2011 Dr. Larry Smarr Director,

A Campus-Scale High Performance Cyberinfrastructure is Required for Data-Intensive Research Seminar Presentation Princeton Institute for Computational.

Calit2: Past, Present, and Future University Librarians Advisory Board Luncheon Seminar UC San Diego Library January 4, 2012 Dr. Larry Smarr Director,

High Performance Cyberinfrastructure Enabling Data-Driven Science in the Biomedical Sciences Joint Presentation UCSD School of Medicine Research Council.

Creating High Performance Lambda Collaboratories" ONR Briefing ACCESS DC Arlington, VA March 25, 2005 Dr. Larry Smarr Director, California Institute for.

High Performance Cyberinfrastructure Required for Data Intensive Scientific Research Invited Presentation National Science Foundation Advisory Committee.

Uses of the OptIPortal Presentation to the Minority Serving Institutions Cyberinfrastructure Empowerment Coalition June 10, 2010 Dr. Larry.

Calit2-Living in the Future " Keynote Sharecase 2006 University of California, San Diego March 29, 2006 Dr. Larry Smarr Director, California Institute.

Supercomputers and Supernetworks are Transforming Research Invited Talk Computing Research that Changed the World: Reflections and Perspectives Washington,

Creating a Community Cyberinfrastructure for Advanced Marine Microbial Ecology Research and Analysis (a.k.a. CAMERA) Invited Talk Honoring David Kingsbury.

High Performance Cyberinfrastructure Enables Data-Driven Science in the Globally Networked World Keynote Presentation Sequencing Data Storage and Management.

High Performance Cyberinfrastructure Discovery Tools for Data Intensive Research Larry Smarr Prof. Computer Science and Engineering Director, Calit2 (UC.

Why Optical Networks Are Emerging as the 21 st Century Driver Scientific American, January 2001.

Project StarGate An End-to-End 10Gbps HPC to User Cyberinfrastructure ANL * Calit2 * LBNL * NICS * ORNL * SDSC Report to the Dept. of Energy Advanced Scientific.

PRISM: High-Capacity Networks that Augment Campus’ General Utility Production Infrastructure Philip Papadopoulos, PhD. Calit2 and SDSC.

1 Cyberinfrastructure Framework for 21st Century Science & Engineering (CIF21) NSF-wide Cyberinfrastructure Vision People, Sustainability, Innovation,

1 Cyberinfrastructure Framework for 21st Century Science & Engineering (CF21) IRNC Kick-Off Workshop July 13,

A High-Performance Campus-Scale Cyberinfrastructure: The Technical, Political, and Economic Presentation by Larry Smarr to the NSF Campus Bridging Workshop.

Computer Supported Cooperative Work: Past, Present, and Future Vision Panel on Computer Supported Cooperative Work 9 th Annual ON*VECTOR International.

Genomics at the Speed of Light: Understanding the Living Ocean The Gordon and Betty Moore Foundation 2nd Annual Marine Microbiology Investigator Symposium.

Cyber UC San Diego Elazar C. Harel May 14, 2008.

SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA; SAN DIEGO IEEE Symposium of Massive Storage Systems, May 3-5, 2010 Data-Intensive Solutions.

“An Integrated West Coast Science DMZ for Data-Intensive Research” Panel CENIC Annual Conference University of California, Irvine Irvine, CA March 9, 2015.

“High Performance Cyberinfrastructure Enables Data-Driven Science in the Globally Networked World” Invited Speaker Grand Challenges in Data-Intensive Discovery.

“How LambdaGrids are Transforming Science" Keynote iGrid2005 La Jolla, CA September 29, 2005 Dr. Larry Smarr Director, California Institute.

Science and Cyberinfrastructure in the Data-Dominated Era Symposium #1610, How Computational Science Is Tackling the Grand Challenges Facing Science and.

Building a Community Cyberinfrastructure to Support Marine Microbial Ecology Metagenomics Center for Earth Observations and Applications Advisory Committee.

“Set My Data Free: High-Performance CI for Data-Intensive Research” KeynoteSpeaker Cyberinfrastructure Days University of Michigan Ann Arbor, MI November.

“Mapping the Human Gut Microbiome in Health and Disease Using Sequencing, Supercomputing, and Data Analysis” Invited Talk Delivered by Mehrdad Yazdani,

CI Days: Planning Your Campus Cyberinfrastructure Strategy Russ Hobby, Internet2 Internet2 Member Meeting 9 October 2007.

Source: Jim Dolgonas, CENIC CENIC is Removing the Inter-Campus Barriers in California ~ $14M Invested in Upgrade Now Campuses Need to Upgrade.

“An Integrated Science Cyberinfrastructure for Data-Intensive Research” Panel CISCO Executive Symposium San Diego, CA June 9, 2015 Dr. Larry Smarr Director,

“Calit2: A UC Experiment for Living in the Future" Talk to UCSD Near You La Jolla, CA April 11, 2006 Dr. Larry Smarr Director, California Institute.

“Creating a High Performance Cyberinfrastructure to Support Analysis of Illumina Metagenomic Data” DNA Day Department of Computer Science and Engineering.

Transformation of Research and Education in the 21 st Century Edward Seidel Director, Office of Cyberinfrastructure National Science Foundation

Developing a North American Global LambdaGrid Dr. Larry Smarr Director, California Institute for Telecommunications and Information Technology Harry E.

“Comparative Human Microbiome Analysis” Remote Video Talk to CICESE Big Data, Big Network Workshop Ensenada, Mexico October 10, 2013 Dr. Larry Smarr Director,

What is Cyberinfrastructure? Russ Hobby, Internet2 Clemson University CI Days 20 May 2008.

Cal-(IT) 2 : A Public-Private Partnership in Southern California U.S. Business Council for Sustainable Development Year-End Meeting December 11, 2003 Institute.

Innovative Research Alliances Invited Talk IUCRP Fellows Seminar UCSD La Jolla, CA July 10, 2006 Dr. Larry Smarr Director, California Institute for Telecommunications.

“Metagenomics Over Lambdas: Update on the CAMERA Project" Invited Talk 6 th Annual ON*VECTOR International Photonics Workshop UCSD February 27,

“Cyberinfrastructure for Ocean Cabled Observatories" Invited Talk NEPTUNE Regional Cabled Ocean Observatory Workshop Seattle, WA November 15, 2005 Dr.

Using Photonics to Prototype the Research Campus Infrastructure of the Future: The UCSD Quartzite Project Philip Papadopoulos Larry Smarr Joseph Ford Shaya.

Building a Community Cyberinfrastructure to Support Marine Microbial Ecology Metagenomics Invited Talk Metagenomics 2006 UCSD La Jolla, CA October.

SoCal Infrastructure OptIPuter Southern California Network Infrastructure Philip Papadopoulos OptIPuter Co-PI University of California, San Diego Program.

A High-Performance Campus-Scale Cyberinfrastructure For Effectively Bridging End-User Laboratories to Data-Intensive Sources Presentation by Larry Smarr.

Information Technology Infrastructure Committee (ITIC) Report to the NAC March 8, 2012 Larry Smarr Chair ITIC.

“Big Data” and Data-Intensive Science (eScience) Ed Lazowska Bill & Melinda Gates Chair in Computer Science & Engineering University of Washington July.

Ocean Sciences Cyberinfrastructure Futures Dr. Larry Smarr Director, California Institute for Telecommunications and Information Technologies Harry E.

The OptIPuter Project Tom DeFanti, Jason Leigh, Maxine Brown, Tom Moher, Oliver Yu, Bob Grossman, Luc Renambot Electronic Visualization Laboratory, Department.

The PRPv1 Architecture Model Panel Presentation Building the Pacific Research Platform Qualcomm Institute, Calit2 UC San Diego October 16, 2015.

“The Pacific Research Platform: a Science-Driven Big-Data Freeway System.” Big Data for Information and Communications Technologies Panel Presentation.

“ Collaborations Between Calit2, SIO, and the Venter Institute—a Beginning " Talk to the UCSD Representative Assembly La Jolla, CA November 29, 2005 Dr.

“ Ultra-Broadband and Peta-Scale Collaboration Opportunities Between UC and Canada Summary Talk Canada - California Strategic Innovation Partnership Summit.

“CAMERA Goes Live!" Presentation with Craig Venter National Press Club Washington, DC March 13, 2007 Dr. Larry Smarr Director, California Institute for.

Cyberinfrastructure Overview Russ Hobby, Internet2 ECSU CI Days 4 January 2008.

Cyberinfrastructure: Many Things to Many People Russ Hobby Program Manager Internet2.

“The UCSD Big Data Freeway System” Invited Short Talk Workshop on “Enriching Human Life and Society” UC San Diego February 6, 2014 Dr. Larry Smarr Director,

“ OptIPuter Year Five: From Research to Adoption " OptIPuter All Hands Meeting La Jolla, CA January 22, 2007 Dr. Larry Smarr Director, California.

Southern California Infrastructure Philip Papadopoulos Greg Hidley.

“Genomics: The CAMERA Project" Invited Talk 5 th Annual ON*VECTOR International Photonics Workshop UCSD February 28, 2006 Dr. Larry Smarr Director,

“Pacific Research Platform Science Drivers” Opening Remarks PRP Science Driver PI Workshop UC Davis March 23, 2016 Dr. Larry Smarr Director, California.

SIU Information Technology April 28, 2015 Research Computing and Cyberinfrastructure.

“OptIPuter: From the End User Lab to Global Digital Assets" Panel UC Research Cyberinfrastructure Meeting October 10, 2005 Dr. Larry Smarr.

“ Building an Information Infrastructure to Support Microbial Metagenomic Sciences " Presentation to the NBCR Research Advisory Committee UCSD La Jolla,

Lennart Johnsson Professor CSC Director, PDC

Project StarGate An End-to-End 10Gbps HPC to User Cyberinfrastructure ANL * Calit2 * LBNL * NICS * ORNL * SDSC Report to the Dept. of Energy Advanced.

The OptIPortal, a Scalable Visualization, Storage, and Computing Termination Device for High Bandwidth Campus Bridging Presentation by Larry Smarr to.

Presentation transcript:

High Performance Cyberinfrastructure Discovery Tools for Data Intensive Research Larry Smarr Prof. Computer Science and Engineering Director, Calit2 (UC San Diego/UC Irvine)

AbstractAbstract High performance cyberinfrastructure (10Gbps dedicated optical channels end-to-end) enable new levels of discovery for data-intensive research projects. I will use several different examples of large data projects drawn from cosmological simulations, ocean observing, and microbial metagenomics. I will discuss why local campus high performance clouds are essential for this sort of work in academia--as a high bandwidth, high I/O fast storage, large RAM compute augmentation to the remote commercial cloud.

Dedicated 10,000Mbps (10Gbps) Supernetworks Enable Remote Visual Analysis of Big Data Also NLR 80 x 10Gb Wavelengths

NSF’s OptIPuter Project: Using Supernetworks to Meet the Needs of Data-Intensive Researchers OptIPortal– Termination Device for the OptIPuter 10Gbps Backplane

Exploring Cosmology With Supercomputers, Supernetworks, and Supervisualization Supercomputer Output –148 TB Movie Output (0.25 TB/file) –80 TB Diagnostic Dumps (8 TB/file) Connected at 10Gbps –Oak Ridge to ANL to SDSC Science: Norman, Harkness, Paschos SDSC Visualization: Insley, ANL; Wagner SDSC ANL * Calit2 * LBNL * NICS * ORNL * SDSC Intergalactic Medium on 2 Billion Light Year Scale

Providing End-to-End 10Gbps Cyberinfrastructure for Petascale End Users Mike Norman, SDSC Analyzing Super Data log of gas temperature log of gas density

Calit2 Microbial Metagenomics Cluster- Next Generation Optically Linked Science Data Server 512 Processors ~5 Teraflops ~ 200 Terabytes Storage Source: Phil Papadopoulos, SDSC, Calit2 Nearly 4000 Users Over 75 Countries

Using 10 Gbps Big Data Access and Analysis- Collaboration Between Calit2 and U Washington Ginger Armbrust’s Diatom Chromosomes Photo Credit: Alan Decker Feb. 29, 2008 iHDTV: 1500 Mbits/sec Calit2 to UW Research Channel Over NLR

MIT’s Ed DeLong & Darwin Project Team Using OptIPortal to Analyze 10km Coupled Ocean Microbial Simulation

The NSF-Funded Ocean Observatory Initiative– a Complex System of Systems Cyberinfrastructure Source: Matthew Arrott, Calit2 Program Manager for OOI CI

Taking Sensornets to the Ocean Floor: Remote Interactive HD Imaging of Deep Sea Vent Source: John Delaney and Research Channel, U Washington 1 cm.

NSF OOI is a $400M Program -OOI CI is $34M Part of OOI Source: Matthew Arrott, Calit2 Program Manager for OOI CI Software Engineers Housed at

OOI CI is Built on National LambdaRail’s and Internet2’s DCN Optical Infrastructure Source: John Orcutt, Matthew Arrott, SIO/Calit2

High Definition Video Connected OptIPortals: Virtual Working Spaces for Data Intensive Research Source: Falko Kuester, Kai Doerr Calit2; Michael Sims, NASA

Analyzing Big Data in 3D Stereo: The NexCAVE OptIPortal Source: Tom DeFanti,

“Blueprint for the Digital University”--Report of the UCSD Research Cyberinfrastructure Design Team research.ucsd.edu/documents/rcidt/RCIDTReportFinal2009.pdf April 24, 2009 DataOasis (Central) Storage OptIPortal Campus Lab Cluster Digital Data Collections Triton – Petadata Analysis Gordon – HPC System Cluster Condo Scientific Instruments N x 10Gbe CENIC, NLR, I2DCN Source: Philip Papadopoulos, SDSC, UCSD

California and Washington Universities Are Testing a 10Gbps Connected Commercial Data Cloud Amazon Experiment for Big Data –Only Available Through CENIC and Pacific NW GigaPOP Private 10Gbps Peering Path –Includes Amazon Computing and Storage Services Amazon Experiment for Big Data –Only Available Through CENIC and Pacific NW GigaPOP Private 10Gbps Peering Path –Includes Amazon Computing and Storage Services

You Can Download This Presentation at lsmarr.calit2.net