The TeraGrid David Hart Indiana University AAAS’09, FEBRUARY 13, 2009

Cyberinfrastructure Computing systems, data storage systems, and data repositories, visualization environments, and people, all linked together by high performance networks. 2

A complex collaboration of over a dozen organizations working together to provide cyberinfrastructure that goes beyond what can be provided by individual institutions, to improve research productivity and enable breakthroughs not otherwise possible. 3

Work by Emad Tajkhorshid and James Gumbart, of University of Illinois Urbana-Champaign. –Mechanics of Force Propagation in TonB- Dependent Outer Membrane Transport. Biophysical Journal 93: (2007). –Results of the simulation may be seen at 2.5Ans.mpg 2.5Ans.mpg Modeled mechanisms for transport of molecules through cell membrane. Used 400,000 CPU hours [45 processor-years] on systems at National Center for Supercomputing Applications, IU, Pittsburgh Supercomputing Center Image courtesy of Emad Tajkhorshid, UIUC What you can do with the TeraGrid: Simulation of cell membrane processes 4

Predicting storms Hurricanes and tornadoes cause massive loss of life and damage to property TeraGrid supported spring 2007 NOAA and University of Oklahoma Hazardous Weather Testbed –Major Goal: assess how well ensemble forecasting predicts thunderstorms, including supercells  tornadoes. –Delivers “better than real time” prediction –Used 675,000 CPU hours for the season –Used 312 TB on HPSS storage at PSC Slide courtesy of Dennis Gannon, IU, and LEAD Collaboration 5

What is the TeraGrid? An instrument that delivers high-end IT resources - computation, storage, visualization, and data/service –A computational facility – over a PetaFLOP in parallel computing capability –A data storage and management facility - over 20 PetaBytes of storage (disk and tape), over 100 scientific data collections –A high-bandwidth national data network A service: help desk and consulting, Advanced Support for TeraGrid Applications (ASTA), education and training events and resources Something you can use without financial cost –Research accounts allocated via peer review –Startup and Education accounts automatic 6

TeraGrid Computing Systems Computational Resources (size approximate - not to scale) Slide Courtesy Tommy Minyard, TACC SDSC TACC UC/ANL NCSA ORNL PU IU PSC NCAR 2007 (504TF) 2009 (~1PF) Tennessee LONI/LS U 7

8 Data storage and management: Tape TeraGrid provides persistent storage on disk and tape Could you benefit from having a spare copy of your data stored someplace removed from your home location? Allocatable tape-based storage systems: –IU (Indiana University) - geographically distributed –NCAR (National Center for Atmospheric Research) - also supports dual copy –NCSA (National Center for Supercomputing Applications) –SDSC (San Diego Supercomputer Center) –Note: most sites have massive data storage systems that provide storage in support of computation Command line usage is reasonably straightforward with GridFTP, very easy with File Manager tool in the TeraGrid User Portal ©Trustees of Indiana University. May be reused so long as IU and TeraGrid logos remain, and any modifications to original are noted. Courtesy Craig A. Stewart, IU

9 Data storage and management: Disk GPFS-WAN (General Parallel File System Wide Area Network). ~ 1 petabyte –Home at San Diego Supercomputer Center; may be accessed as if it were a local file system from NCAR, NCSA, IU, UC/ANL IU Data Capacitor –1 petabyte of spinning disk –Primarily for short term storage of data Long term disk storage allocations –Indiana University, National Center for Supercomputing Applications, San Diego Supercomputer Center ©Trustees of Indiana University. May be reused so long as IU and TeraGrid logos remain, and any modifications to original are noted. Courtesy Craig A. Stewart, IU

TeraGrid Participants 10

TeraGrid Architecture Compute Service Viz Service Data Service Network, Accounting, … RP 1 RP 3 RP 2 TeraGrid Infrastructure (Network, Authorization, Accounting, …) POPS Science Gateways User Portal Command Line 11

12

LEAD (portal.leadproject.org/) Simple enough an undergraduate can use it! National Center for Supercomputing Applications (NCSA) and IU teamed up to support WxChallenge weather forecast competition. 64 teams, 1000 students, ~16,000 CPU hours on Big Red XBaya is available from 13

What is a Science Gateway? A Science Gateway –Enables scientific communities of users with a common scientific goal –Has a common interface –Leverages community investment Three common forms: –Web-based Portals –Application programs running on users' machines but accessing services in TeraGrid –Coordinated access points enabling users to move seamlessly between TeraGrid and other grids. 14

How can a Gateway help? Make science more productive –Researchers use same tools –Complex workflows –Common data formats –Data sharing Bring TeraGrid capabilities to the broad science community –Lots of disk space –Lots of compute resources –Powerful analysis capabilities –A nice interface to information 15

NanoHub Harnesses TeraGrid for Education Nanotechnology education Used in dozens of courses at many universities Teaching materials Collaboration space Research seminars Modeling tools Access to cutting edge research software 16

17 SIDGrid sidgrid.ci.uchicago.edu

CY2007 Usage by Discipline 3.95B SUs delivered in CY2007 Molecular Biosciences 31% Chemistry 17% Physics 17% Astronomical Sciences 12% Materials Research 6% Earth Sciences 3% All 19 Others 4% Advanced Scientific Computing 2% Atmospheric Sciences 3% Chemical, Thermal Systems 5% 18

Usage is Growing.... Source: TeraGrid Central Database 3.95B SUs delivered in CY

TeraGrid Resources and Services Computing – over a petaflop of computing power and growing Data –Data storage facilities –Scientific data collections Over 30 Science Gateways Remote visualization servers and software Technical Support –Central point of contact for support of all systems –Advanced Support for TeraGrid Applications (ASTA) Education and training events and resources –K-12 Education –Pathways –Campus Champions 20

Campus Champions The Campus Champions program supports campus representatives as the local source of knowledge about high-performance computing opportunities and resources. This knowledge and assistance will empower campus researchers, educators, and students to advance scientific discovery. Your campus will benefit by having direct access to the TeraGrid and input to its staff, resource allocations awarded for their use, and assistance in using those resources. TeraGrid will support the Campus Champion. See – –To join the Campus Champions program, contact the TeraGrid Campus Champions Program Coordinator, at 21

Online Resources Online resources at TeraGrid User Portal for managing allocations and job flow Documentation –Knowledge Base for quick answers to FAQ’s –HPC University to increase general HPC knowledge Calendar of events including upcoming workshops and training –Annual conference - TG09 Arlington, VA June 22-26,

TeraGrid: greater than the sum of its parts… Leadership in cyberinfrastructure development, deployment and support Expertise in building national computing and data resources Leveraging extensive resources, expertise, R&D, and EOT –leveraging other activities at participant sites –learning from each other improves expertise of all TG staff Simplified access to high end resources –Single unified allocations process –Single point of contact for problem reporting –Coordinated software environments –Uniform access to heterogeneous resources to solve a single scientific problem 23

Allocations Process Startup allocations: for code development, experimentation with TeraGrid platforms, and application testing. Startup requests may total up to 200,000 service units (SUs) of computation, up to 5TB on disk and 25TB on tape of storage. Education allocations: for use in classroom instruction or training activities, with the same SU and storage limits as Startup allocations. Research allocations: requires a detailed justification of resource usage. Requests are reviewed four times a year by the Resource Allocations Committee. National peer-review process –allocates computational and data resources –makes recommendations on allocation of advanced direct support services –Currently awarding >1B Normalized Units of resources Principal investigator (PI) must be a researcher, educator, or postdoctoral researcher at a US academic or non-profit research institution. 24

Go to the POPS page

Create a POPS Login 26

Indicate that you are “New” to the Teragrid 27

Indicate this is a “Start-up” Request 28

Select Startup or Educational 29

Fill out PI information 30

Skip Co-PIs info 31

Fill out info on your project 32

Fill out info on your funding 33

Make reasonable estimates about your computing 34

when ready Upload your CV and Submit! 35

Acknowledgements This work is made possible by the dedicated efforts of the TeraGrid staff. In particular, slides came from Craig Stewart, John Towns, Dana Skow, Daphne Siefert-Herron, Vickie Lynch and Laura McGinnis (and probably others). The Grid Infrastructure Group management of the TeraGrid is funded by NSF grant IU’s involvement as a TeraGrid Resource Partner is supported in part by the National Science Foundation under Grants No. ACI l, OCI , OCI , and OCI The IU Data Capacitor is supported in part by the National Science Foundation under Grant No. CNS Purdue’s involvement as a TeraGrid Resource Partner is supported in part by the National Science Foundation under Grant No. OCI This research was supported in part by the Pervasive Technology Labs and the Indiana METACyt Initiative. Both Indiana University initiatives are supported by the Lilly Endowment, Inc. This work was supported in part by Shared University Research grants from IBM, Inc. to Indiana University. The LEAD portal is developed under the leadership of IU Professors Dr. Dennis Gannon and Dr. Beth Plale, and supported by NSF grant Marcus Christie and Surresh Marru of the Extreme! Computing Lab contributed the LEAD graphics The ChemBioGrid Portal is developed under the leadership of IU Professor Dr. Geoffrey C. Fox and Dr. Marlon Pierce and funded via the Pervasive Technology Labs (supported by the Lilly Endowment, Inc.) and the National Institutes of Health grant P20 HG Any opinions, findings and conclusions or recommendations expressed in this material are those of the author and do not necessarily reflect the views of the National Science Foundation (NSF), National Institutes of Health (NIH), Lilly Endowment, Inc., or any other funding agency. 36

Thank you! Questions? 37