1. Advancing Scientific Discovery through TeraGrid
June 2006
Advancing Scientific Discovery through TeraGrid
Laura McGinnis
Project Manager, Pathways to Broadening Participation in TeraGrid
Pittsburgh Supercomputing Center, Carnegie Mellon University
THESE SLIDES ARE ©UNIVERSITY OF CHICAGO AND MAY BE USED PROVIDING THE TeraGrid LOGO REMAINS ON THE SLIDES, AND
THAT THE SCIENCE GROUPS ARE ACKNOWLEDGED IN CASES WHERE SCIENTIFIC IMAGES ARE USED.
Charlie Catlett

2. 11 Resource Providers, One Facility
TeraGrid'06
June 2006
Grid Infrastructure Group (UChicago) UW
UC/ANL
PSC
NCAR PU
NCSA IU
UNC/RENCI
Caltech
ORNL
U Tenn.
USC/ISI
SDSC
LSU
TACC
Resource Provider (RP)
Software Integration Partner
Charlie Catlett

3. TeraGrid Objectives DEEP Science: Enabling Petascale Science
June 2006
DEEP Science: Enabling Petascale Science
Make Science More Productive through an integrated set of very-high capability resources
Address key challenges prioritized by users
WIDE Impact: Empowering Communities
Bring TeraGrid capabilities to the broad science community
Partner with science community leaders - “Science Gateways”
OPEN Infrastructure, OPEN Partnership
Provide a coordinated, general purpose, reliable set of services and resources
Partner with campuses and facilities
DEEP- To ensure that we are harnessing the enormous power of the TeraGrid resources as an integrated system we have the Advanced Support for
TeraGrid Applications (ASTA) program that involves our distributed user support team (~20 people across 8 sites, coordinated by GIG). The ASTA
program places 1/4 to 1/2 of a support person in a scientific application team for 4 to 12 months, working with that team
to exploit TeraGrid capabilities. The ASTA program supports roughly a dozen teams at a time, with a goal of teams supported per year.
WIDE- For over 2 decades the NSF high performance computing program, including TeraGrid, has served several thousand users very
effectively. However, NSF alone funds tens of thousands of scientists, most of whom have computational requirements that do not
frequently require supercomputers. The TeraGrid Science Gateways program is a set of partnerships with discipline-specific teams
who are providing computational infrastructure for their science communities. The partnerships involve integrating TeraGrid as a
computational and data management “service provider” embedded in the science-community infrastructure. Over a twenty we
portal style science gateways are part of this program, with more being added continually. In addition, this program includes a
peer-grid interoperation effort with Open Science Grid and a desktop application effort that is leveraging an NIH-funded biomedical
project directed by Rick Stevens at the University of Chicago.
OPEN- TeraGrid began as an infrastructure involving four partner sites, grew to nine sites, and is currently organized as a set of
resource providers and a central “grid infrastructure group” (GIG) providing central services and support as well as management
and operations. This structure allows TeraGrid to grow to dozens of resource provider sites, making it an “open” partnership.
In addition, TeraGrid architecture is service-oriented, stressing open source standards such as are deployed with key software
including the Globus Toolkit GT4, Condor, and other tools. In order to partner with the broader community of universities and
other service and resource providers, TeraGrid is defining a set of “campus partnerships” during The goal of these partnership
programs is to work with campuses (where most TeraGrid users reside) to improve and streamline TeraGrid access from campus
systems, as well as to work with campuses to develop a set of frameworks that can be used to create national-scale
cyberinfrastructure for computation and for data federation. These programs will explicitly reach beyond the R1
institutions to include the broader R&E community.
Charlie Catlett

4. TeraGrid Resources Computing - over 250 Tflops today and growing
June 2006
Computing - over 250 Tflops today and growing
400 Tflop system comes on-line in January at TACC
U Tennesee system to come on-line in 2008
Centralized help desk for all resource providers
Visualization - Remote visualization servers and software
Data
Allocation of data storage facilities
Over 100 Scientific Data Collections
Access
Shibboleth testbed to facilitate campus access
Central allocations mechanism
Human Support
Central point of contact for support of all systems
Advanced Support for TeraGrid Applications (ASTA)
Education and training events and resources
Over 20 Science Gateways
Charlie Catlett

5. Advanced Support for TeraGrid Applications
June 2006
Virtualized Resources, Ensembles:FOAM Climate Model
Liu (UWisc)
FOAM manual ensembles required 75 days to run 160 simulations by hand at NCAR.
Use of VDS to manage the ensemble runs resulted in the ability to run 250 simulations at NCSA and SDSC in 4 days.
This represents a 30x improvement in scientific throughput!
Coupled Simulation:
Full Body Arterial Tree Simulation
Karniadakis (Brown)
Sources: Ian Foster (UC/ANL), Mike Papka (UC/ANL),
George Karniadakis (Brown). Images by UC/ANL.
Charlie Catlett

6. Requesting Allocations of Time
TeraGrid'06
June 2006
TeraGrid resources are provided for free to academic researchers and educators
Development Allocations Committee (DAC) for start-up accounts up to 30,000 hours of time are requests processed in two weeks - start-up and courses
Medium Resource Allocations Committee (MRAC) for requests of up to 500,000 hours of time are reviewed four times a year
Large Resource Allocations Committee (LRAC) for requests of over 500,000 hours of time are reviewed twice a year
Charlie Catlett

7. TeraGrid Usage 200 100 33% Annual Growth Normalized Units (millions)
June 2006
Specific Allocations
Roaming Allocations
33% Annual Growth
200
Normalized Units (millions)
100
TeraGrid users are awarded “allocations” of time based on peer review that takes place on a quarterly basis. New users can apply for
development allocations, allowing them to begin computing within 2-3 weeks of initial request.
This slide shows overall TeraGrid usage from April 2005 through May 2006, including the addition of NCSA and SDSC resources in
April ROAMING usage refers to allocates that allow the user to use the allocation on any TeraGrid resource, where other usage
shown is from allocations on specific resources.
TeraGrid currently delivers an average of 420,000 cpu-hours per day -> ~21,000 CPUs DC
Dave Hart
Charlie Catlett

8. TeraGrid Usage Modes in CY2006
June 2006
Use Modality
Community Size (est. number of people/projects)
Batch Computing on Individual Resources
850
Exploratory and Application Porting
650
Workflow, Ensemble, and Parameter Sweep
160
Science Gateway Access
100
Remote Interactive Steering and Visualization 35
Tightly-Coupled Distributed Computation 10
Grid-y Users
Charlie Catlett

9. Science Gateways Broadening Participation in TeraGrid
June 2006
Increasing investment by communities in their own cyberinfrastructure, but heterogeneous:
Resources
Users – from expert to K-12
Software stacks, policies
Science Gateways
Provide “TeraGrid Inside” capabilities
Leverage community investment
Three common forms:
Web-based Portals
Application programs running on users' machines but accessing services in TeraGrid
Coordinated access points enabling users to move seamlessly between TeraGrid and other grids.
Workflow Composer
TeraGrid leaders created the Science Gateways concept and initiative as a way to adapt the TeraGrid to the environments that users have adopted (e.g. web portals), leveraging the work of these science communities in designing and building their infrastructure. This approach also leverages education efforts of these communities, and allows for an order of magnitude increase in the number of users served by TeraGrid. This increase is possible by changing the model of provision of resources and support, where the TeraGrid serves gateways as a “wholesale” supplier, working with the gateway providers who serve as the “retailers” providing tailored support to communities of users.
Source: Dennis Gannon
Charlie Catlett

10. Gateways are Expanding
TeraGrid'06
Gateways are Expanding
June 2006
10 initial projects as part of TG proposal
>20 Gateway projects today
No limit on how many gateways can use TG resources
Prepare services and documentation so developers can work independently
Open Science Grid (OSG)
Special PRiority and Urgent Computing Environment (SPRUCE)
National Virtual Observatory (NVO)
Linked Environments for Atmospheric Discovery (LEAD)
Computational Chemistry Grid (GridChem)
Computational Science and Engineering Online (CSE-Online)
GEON(GEOsciences Network)
Network for Earthquake Engineering Simulation (NEES)
SCEC Earthworks Project
Network for Computational Nanotechnology and nanoHUB
GIScience Gateway (GISolve)
Biology and Biomedicine Science Gateway
Open Life Sciences Gateway
The Telescience Project
Grid Analysis Environment (GAE)
Neutron Science Instrument Gateway
TeraGrid Visualization Gateway, ANL
BIRN
Gridblast Bioinformatics Gateway
Earth Systems Grid
Astrophysical Data Repository (Cornell)
Charlie Catlett

11. TeraGrid as a Social Network
June 2006
Annual TeraGrid conference - TeraGrid ‘08 - Las Vegas - June
LRAC/MRAC liaisons
SGW community very successful
Transitioning to consulting model
CI Days - campus outreach
OSG/Internet2/NLR/EDUCAUSE/MSI-CIEC partnership
HPC University
OSG, Shodor, Krell, OSC, NCSI, MSI-CIEC partnership
Education and Outreach
Engaging thousands of people
Charlie Catlett

12. SC 07-09 Education Program Goals
TeraGrid'06
SC Education Program Goals
June 2006
Multi-year, year-long, Education Program to provide continuity and broader, sustained impact in education
Increase participation of larger, more diverse communities in the SC Conference
Faculty, students, international, under-represented
Integrate HPC into undergraduate science, technology, engineering and mathematics classrooms
Significantly expanded digital libraries of resources for teaching and learning
Sponsors: ACM, IEEE, NCSI, CSERD, Intel, Wolfram Research, TeraGrid
Charlie Catlett

13. “HPC University” Advance researchers’ HPC skills
TeraGrid'06
June 2006
Advance researchers’ HPC skills
Catalog of live and self-paced training
Schedule series of training courses
Gap analysis of materials to drive development
Work with educators to enhance the curriculum
Search catalog of HPC resources
Schedule workshops for curricular development
Leverage good work of others
Offer Student Research Experiences
Enroll in HPC internship opportunities
Offer Student Competitions
Publish Science and Education Impact
Promote via TeraGrid Science Highlights, iSGTW
Publish education resources to NSDL-CSERD
Charlie Catlett

14. Broadening Participation in TeraGrid
June 2006
Broaden awareness of TeraGrid
Campus Visits (coupled with CI Days)
Professional Society Meetings
Develop promotional materials
Build human capacity for Terascale research
In-depth consulting (5-8 consultants)
TeraGrid Fellowship Program for faculty and students
Mentoring Program
Campus Champions
Enhance the usability and access of TG via SGs
Assess Science Gateway readiness and community requirements
Develop replicable strategies for integrating TeraGrid resources into SGs, with an emphasis on under-served community needs
Charlie Catlett

15. CI Days http://cidays.org
TeraGrid'06
June 2006
CI Days
Working with campuses to take a leadership role applying CI to accelerate scientific discovery
First event held at UC Davis has helped catalyze campus-wide discussions and planning
Collaboration of Open Science Grid, Internet 2, National Lamda Rail, EDUCAUSE, Minority Serving Institution Cyberinfrastructure Empowerment Coalition, TeraGrid, and local and regional organizations
Campus Champions Program
Charlie Catlett

16. www.teragrid.org www.computationalscience.org
For More Information
TeraGrid'06
June 2006
cserd.nsdl.org
Charlie Catlett