Pegasus: Planning for Execution in Grids Ewa Deelman, Carl Kesselman, Gaurang Mehta, Gurmeet Singh, Karan Vahi Information Sciences Institute University of Southern California

Pegasus Acknowledgement l Ewa Deelman, Carl Kesselman, Saurabh Khurana, Gaurang Mehta, Sonal Patil, Gurmeet Singh, Mei-Hui Su, Karan Vahi (ISI) l James Blythe, Yolanda Gil (ISI) l l Research funded as part of the NSF GriPhyN, NVO and SCEC projects.

Virtual Data Concept l Capture and manage information about relationships among u Data (of widely varying representations) u Programs (& their execution needs) u Computations (& execution environments) l Apply this information to, e.g. u Discovery: Data and program discovery u Workflow: Structured paradigm for organizing, locating, specifying, & requesting data u Explanation: provenance l Research part of NSF funded GriPhyN project

Grid Applications l Increasing in the level of complexity l Use of individual application components l Reuse of individual intermediate data products l Description of Data Products using Metadata Attributes l Execution environment is complex and very dynamic u Resources come and go u Data is replicated u Components can be found at various locations or staged in on demand l Separation between u the application description u the actual execution description

Abstract Workflow Generation Concrete Workflow Generation

Pegasus: Planning for Execution in Grids l Maps from abstract to concrete workflow u Algorithmic and AI based techniques l Automatically locates physical locations for both components (transformations) and data u Use Globus RLS and the Transformation Catalog l Finds appropriate resources to execute u via Globus MDS l Reuses existing data products where applicable l Publishes newly derived data products u Chimera virtual data catalog

Chimera is developed at ANL By I. Foster, M. Wilde, and J. Voeckler

LIGO Scientific Collaboration l Continuous gravitational waves are expected to be produced by a variety of celestial objects l Only a small fraction of potential sources are known l Need to perform blind searches, scanning the regions of the sky where we have no a priori information of the presence of a source u Wide area, wide frequency searches l Search is performed for potential sources of continuous periodic waves near the Galactic Center and the galactic core l The search is very compute and data intensive l LSC is using the occasion of SC2003 to initiate a month- long production run with science data collected during 8 weeks in the Spring of 2003

Additional resources used: Grid3 iVDGL resources Thanks to everyone involved in standing up the tested and contributing the resources!

LIGO Acknowledgements l Bruce Allen, Scott Koranda, Brian Moe, Xavier Siemens, University of Wisconsin Milwaukee, USA l Stuart Anderson, Kent Blackburn, Albert Lazzarini, Dan Kozak, Hari Pulapaka, Peter Shawhan, Caltech, USA l Steffen Grunewald, Yousuke Itoh, Maria Alessandra Papa, Albert Einstein Institute, Germany l Many Others involved in the Testbed l l group.phys.uwm.edu/lscdatagrid/ l l LIGO Laboratory operates under NSF cooperative agreement PHY

Montage l Montage (NASA and NVO) u Deliver science-grade custom mosaics on demand u Produce mosaics from a wide range of data sources (possibly in different spectra) u User-specified parameters of projection, coordinates, size, rotation and spatial sampling. Mosaic created by Pegasus based Montage from a run of the M101 galaxy images on the Teragrid.

Small Montage Workflow

Montage Acknowledgments l Bruce Berriman, John Good, Anastasia Laity, Caltech/IPAC l Joseph C. Jacob, Daniel S. Katz, JPL l caltech.edu/ l Testbed for Montage: Condor pools at USC/ISI, UW Madison, and Teragrid resources at NCSA, PSC, and SDSC. Montage is funded by the National Aeronautics and Space Administration's Earth Science Technology Office, Computational Technologies Project, under Cooperative Agreement Number NCC5-626 between NASA and the California Institute of Technology.

Current System

Just In-time planning l Partition Abstract workflow into partial workflows

Meta-DAGMan

Other Applications Using Pegasus l Other GriPhyN applications: u High-energy physics: Atlas, CMS (many) u Astronomy: SDSS (Fermi Lab, ANL) l Astronomy: u Galaxy Morphology (NCSA, JHU, Fermi, many others, NVO-funded) l Biology u BLAST (ANL, PDQ-funded) l Neuroscience u Tomography (SDSC, NIH-funded) l l Funding by NSF GriPhyN, NSF NVO, NIH