1. Frieda meets Pegasus-WMS
What if I want to define workflows that can flexibly take advantage of different grid resources?
What if I want to register data products in a way that makes them available to others?
What if I want to use the grid without a full Condor installation?

2. Pegasus Workflow Management System
A higher level on top of DAGMan
User creates an abstract workflow
Pegasus maps abstract workflow to executable workflow
DAGMan runs executable workflow
Doesn’t need full Condor (schedd only)

3. Pegasus features
Workflow has inter-job dependencies (similar to DAGMan)
Pegasus can map jobs to grid sites
Pegasus handles discovery and registration of data products
Pegasus handles data transfer to/from grid sites

4. Generating mosaics of the sky (Bruce Berriman, Caltech)
Size of the mosaic in degrees square*
Number of jobs
Number of input data files
Number of intermediate files
Total data footprint
Approx. execution time (20 procs) 1
232 53
588
1.2GB
40 mins 2
1,444
212
3,906
5.5GB
49 mins 4
4,856
747
13,061
20GB
1hr 46 mins 6
8,586
22,850
38GB
2 hrs. 14 mins 10
20,652
3,722
54,434
97GB
6 hours
Kent – fixed small typo
*The full moon is 0.5 deg. sq. when viewed form Earth, Full Sky is ~ 400,000 deg. sq. 4

5. Abstract Workflow (DAX)
Pegasus workflow description—DAX
Workflow “high-level language”
Devoid of resource descriptions
Devoid of data locations
Refers to codes as logical transformations
Refers to data as logical files

6. Basic Workflow Mapping
Select where to run the computations
Change task nodes into nodes with executable descriptions
Select which data to access
Add stage-in and stage-out nodes to move data
Add nodes that register the newly-created data products
Add nodes to create an execution directory on a remote site
Write out the workflow in a form understandable by a workflow engine
Include provenance capture steps
Maps jobs to grid sites partially according to a match between grid site’s platform and the binaries you have. Binaries can be transferred or not.
Note that Pegasus-plan creates the condor submit files for all jobs

7. Pegasus Workflow Mapping1 4
Original workflow: 15 compute nodes
devoid of resource assignment 5 8 9 4 10 12 13 15 8 3
Resulting workflow mapped onto 3 Grid sites:
11 compute nodes (4 reduced based on available intermediate data)
13 data stage-in nodes
8 inter-site data transfers
14 data stage-out nodes to long- term storage
14 data registration nodes (data cataloging) 7 9 12 10 15 13 7

8. Pegasus WMS Workflow Description in XML (DAX) Properties
Replica Catalog
Pegasus
Workflow
Mapper
Site Catalog
Transformation
Catalog
TeraGrid
Open Science Grid
Campus resources
Local machine
Condor
DAGMan
Condor
Schedd
Submit Host
Pegasus WMS restructures and optimizes the workflow, provides reliability 8

9. Mapping a workflow To map a workflow, use the pegasus-plan command:
% pegasus-plan -Dpegasus.user.properties=pegasus-wms/config/properties --dir dags --sites viz --output local --force --nocleanup --dax pegasus-wms/dax/montage.dax
Creates executable workflow

10. Running a workflow To run a workflow, use the pegasus-run command:
% pegasus-run -Dpegasus.user.properties=pegasus-wms/dags/train01/pegasus/montage/run0001/pegasus properties pegasus-wms/dags/train01/pegasus/montage/run0001
Runs condor_submit_dag and other tools
Pegasus-plan gives you the pegasus-run command you need

11. There’s much more…
We’ve only scratched the surface of Pegasus’s capabilities

12. Pegasus links Pegasus home page: pegasus.isi.edu
Tutorial materials available at:
For more questions: