The ADAMANT Project: Linking Scientific Workflows and Networks “Adaptive Data-Aware Multi-Domain Application Network Topologies” Ilia Baldine, Charles Schmitt, University of North Carolina at Chapel Hill/RENCI Jeff Chase, Duke University Ewa Deelman, University of Southern California Funded by NSF under the Campus Cyberinfrastructure – Network Infrastructure and Engineering (CC-NIE) Program

The Problem Scientific data is being collected at an ever increasing rate The “old days” -- big, focused experiments– LHC, LIGO, etc.. -- big data archives– SDSS, 2MASS, etc.. Today “cheap” DNA sequencers – and an increasing number of them in individual laboratories The complexity of the computational problems is ever increasing Local compute resources are often not enough (too small, limited availability) The computing infrastructure keeps changing Hardware, software, but also computational models

Computational workflow --managing application complexity Helps express multi-step computations in a declarative way Can support automation, minimize human involvement –Makes analyses easier to run Can be high-level and portable across execution platforms Keeps track of provenance to support reproducibility Fosters collaboration—code and data sharing Gives the opportunity to manage resources underneath

Large-Scale, Data-Intensive Workflows Montage Galactic Plane Workflow –18 million input images (~2.5 TB) –900 output images (2.5 GB each, 2.4 TB total) –10.5 million tasks (34,000 CPU hours) An analysis is composed of a number of related workflows– an ensemble Smart data/network provisioning are important 4 John Good (Caltech)

CyberShake PSHA Workflow 239 Workflows Each site in the input map corresponds to one workflow Each workflow has:  820,000 tasks  Description  Builders ask seismologists: “What will the peak ground motion be at my new building in the next 50 years?”  Seismologists answer this question using Probabilistic Seismic Hazard Analysis (PSHA) Southern California Earthquake Center MPI codes ~ 12,000 CPU hours, Post Processing 2,000 CPU hours Data footprint ~ 800GB Coordination between resources is needed

Environment How to manage complex workloads? Data Storage Campus Cluster XSEDE Open Science Grid Amazon Cloud Work definition Local Resource

Use Given Resources Data Storage Campus Cluster FutureGrid XSEDE Open Science Grid Amazon Cloud Work definition As a WORKFLOW Workflow Management System Local Resource work data

Workflow Management You may want to use different resources within a workflow or over time Need a high-level workflow specification Need a planning capability to map from high-level to executable workflow Need to manage the task dependencies Need to manage the execution of tasks on the remote resources Need to provide scalability, performance, reliability

Pegasus Workflow Management System (est. 2001) Pegasus makes use of available resources, but cannot control them A collaboration between USC and the Condor Team at UW Madison (includes DAGMan) Maps a resource-independent “abstract” workflow onto resources and executes the “concrete” workflow Used by a number of applications in a variety of domains Provides reliability—can retry computations from the point of failure Provides scalability—can handle large data and many computations (kbytes-TB of data, tasks) Infers data transfers, restructures workflows for performance Automatically captures provenance information Can run on resources distributed among institutions, laptop, campus cluster, Grid, Cloud

A way to make it work better Data Storage Work definition Pegasus WMS Local Resource work data Resource Provisioner Virtual Resource Pool Resources requests Resources: compute, data, networks Grids and Clouds

ORCA is a “wrapper” for off-the-shelf cloud and circuit nets etc., enabling federated orchestration: +Resource brokering +VM image distribution +Topology embedding +Stitching +Authorization o Deploys a dynamic collection of controllers o Controller receive user requests and provisions resources Open Resource Control Architecture Jeff Chase, Duke University

Pegasus and Orca, initial implementation

What we would like to do: Expand to workflow ensembles

What is missing Tools and systems that can integrate the operation of workflow-driven science applications on top of dynamic infrastructures that link campus, institutional and national resources Tools to manage workflow ensembles Need to –orchestrate the infrastructure in response to the application –monitor various workflow steps and ensemble elements –expand and shrink resource pools in response to application performance demands –integrate data movement/storage decisions with workflows/resource provisioning to optimize performance

Summary: ADAMANT will Focus on data-intensive applications: astronomy, bioinformatics, earth science Interleave workload management with resource provisioning –Emphasis on storage and network provisioning Monitor the execution and adapt resource provisioning and workload scheduling Experiment on exoGeni – – –