1. Pegasus-a framework for planning for execution in grids Karan Vahi vahi@isi.edu USC Information Sciences Institute May 5 th, 2004

2. 2May 5th, 2004 Karan Vahi, ISI vahi@isi.edu People Involved USC/ISI Advanced Systems: Ewa Deelman, Carl Kesselmann, Gaurang Mehta, Mei-Hui Su, Gurmeet Singh, Karan Vahi.

3. 3May 5th, 2004 Karan Vahi, ISI vahi@isi.edu Outline l Introduction To Planning l DAX l Pegasus l Portal l Demonstration

4. 4May 5th, 2004 Karan Vahi, ISI vahi@isi.edu Planning in Grids l One has various alternatives out on the grid in terms of data and compute resources. l Planning –Select the best available resources and data sets, and schedule them on to the grid to get the best possible execution time. –Plan for the data movements between the sites

5. 5May 5th, 2004 Karan Vahi, ISI vahi@isi.edu Recipe For Planning l Understand the request –Figure out what data product the request refers to, and how to generate it from scratch. l Locations of data products –Final data product –Intermediate data products which can be used to generate the final data product. l Location of Job executables l State of the Grid –Available processors, physical memory available, job queue lengths etc.

6. 6May 5th, 2004 Karan Vahi, ISI vahi@isi.edu Constituents of Planning Domain Knowledge Resource Information Location Information Planner Plan submitted the grid

7. 7May 5th, 2004 Karan Vahi, ISI vahi@isi.edu Terms (1) l Abstract Workflow (DAX) –Expressed in terms of logical entities –Specifies all logical files required to generate the desired data product from scratch –Dependencies between the jobs –Analogous to build style dag l Concrete Workflow –Expressed in terms of physical entities –Specifies the location of the data and executables –Analogous to a make style dag

8. 8May 5th, 2004 Karan Vahi, ISI vahi@isi.edu Outline l Introduction to Planning l DAX l Pegasus l Portal l Demonstration

9. 9May 5th, 2004 Karan Vahi, ISI vahi@isi.edu DAX l The format for specifying the abstract workflow, that identifies the recipe for creating the final data product at a logical level. l In case of montage, the IPAC webservice ends up creating the dax for the user request. Developed at University Of Chicago

10. 10May 5th, 2004 Karan Vahi, ISI vahi@isi.edu DAX Example l l -a top -T60 -i -o l l -a bottom -T60 -i -o l

11. 11May 5th, 2004 Karan Vahi, ISI vahi@isi.edu Outline l Introduction to Planning l DAX l Pegasus l Demonstration l Portal

12. 12May 5th, 2004 Karan Vahi, ISI vahi@isi.edu Pegasus l A configurable system to map and execute complex workflows on the grids. –DAX Driven Configuration –Metadata Driven Configuration l Can do full ahead planning or deferred planning to map the workflows.

13. 13May 5th, 2004 Karan Vahi, ISI vahi@isi.edu Full Ahead Planning l At the time of submission of the workflow, you decide where you want to schedule the jobs in the workflow. l Allows you to perform certain optimizations by looking ahead for bottleneck jobs and then scheduling around them. l However, for large workflows the decision you make at submission time may no longer be valid or optimum at the point the job is actually run.

14. 14May 5th, 2004 Karan Vahi, ISI vahi@isi.edu Deferred Planning l Delay the decision of mapping the job to the site as late as possible. l Involves partitioning of the original dax into smaller daxes each of which refers to a partition on which Pegasus is run. l Construct a Mega DAG that ends up running pegasus automatically on the partition daxes, as each partition is ready to run.

15. 15May 5th, 2004 Karan Vahi, ISI vahi@isi.edu High Level Block Diagram

16. 16May 5th, 2004 Karan Vahi, ISI vahi@isi.edu Replica Discovery l Pegasus needs to know where the input files for the workflow reside. l In Montage case, it should know where the fits files that are required for the mProject jobs reside. l Hence Pegasus needs to discover the files that are required for executing a particular abstract workflow.

17. 17May 5th, 2004 Karan Vahi, ISI vahi@isi.edu RLS RLI LRC A LRC C LRC B Each LRC sends periodic updates to the RLI Each LRC is responsible for one pool Pegasus 1) Pegasus queries RLI with the LFN 2) RLI returns the list of LRC’s that contain the desired mappings. 3) Pegasus queries each LRC in the list to get the PFN’s. Figure (1) RLS Configuration for Pegasus Interfacing to RLS done by Karan Vahi, Shishir

18. 18May 5th, 2004 Karan Vahi, ISI vahi@isi.edu Alternate Replica Mechanisms l Replica Catalog –Pegasus supports the LDAP based Replica Catalog l User defined mechanisms –Pegasus provides the flexibility for the user to specify his own replica mechanism instead of RLS or Replica Catalog –The user just has to implement the concerned interface Design and Implementation done by Karan Vahi

19. 19May 5th, 2004 Karan Vahi, ISI vahi@isi.edu Transformation Catalog l Pegasus needs to access a catalog to determine the pools where it can run a particular piece of code. l If a site does not have the executable, one should be able to ship the executable to the remote site. l Generic TC API for users to implement their own transformation catalog. l Current Implementations –File Based –Database Based

20. 20May 5th, 2004 Karan Vahi, ISI vahi@isi.edu File based Transformation Catalog l Consists of a simple text file. –Contains Mappings of Logical Transformations to Physical Transformations. l Format of the tc.data file #poolname logical tr physical tr env isi preprocess /usr/vds/bin/preprocess VDS_HOME=/usr/vds/; l All the physical transformations are absolute path names. l Environment string contains all the environment variables required in order for the transformation to run on the execution pool.

21. 21May 5th, 2004 Karan Vahi, ISI vahi@isi.edu DB based Transformation Catalog l Presently ported on MySQL. Postgres to be tested. l Adds support for transformations, compiled for different architectures, OS, OS version and glibc combination, that would enable us to transfer transformation to remote sites if the executable does not reside there. l Supports multiple profile namespaces. At present using only the env namespace. l Supports multiple physical transformations for the same logical transformation,pool,type tuple.

22. 22May 5th, 2004 Karan Vahi, ISI vahi@isi.edu Pool Configuration (1) l Pool Config is an XML file which contains information about various pools on which DAGs may execute. l Some of the information contained in the Pool Config file is –Specifies the various job-managers which are available on the pool for the different types of condor universes. –Specifies the GridFtp storage servers associated with each pool. –Specifies the Local Replica Catalogs where data residing in the pool has to be cataloged. –Contains profiles like environment hints which are common site wide. –Contains the working and storage directories to be used on the pool.

23. 23May 5th, 2004 Karan Vahi, ISI vahi@isi.edu Pool Configuration (2) l Two Ways to construct the Pool Config File. –Monitoring and Discovery Service –Local Pool Config File (Text Based) l Client tool to generate Pool Config File –The tool genpoolconfig is used to query the MDS and/or the local pool config file/s to generate the XML Pool Config file.

24. 24May 5th, 2004 Karan Vahi, ISI vahi@isi.edu Pool Configuration (3) l This file is read by the information provider and published into MDS. l Format gvds.pool.id : gvds.pool.lrc : gvds.pool.gridftp : @ gvds.pool.gridftp : gsiftp://sukhna.isi.edu/nfs/asd2/gmehta@2.4.0 gvds.pool.universe : @ @ gvds.pool.universe : transfer@columbus.isi.edu/jobmanager- fork@2.2.4 gvds.pool.gridlaunch : gvds.pool.workdir : gvds.pool.profile : @ @ gvds.pool.profile : env@GLOBUS_LOCATION@/smarty/gt2.2.4 gvds.pool.profile : vds@VDS_HOME@/nfs/asd2/gmehta/vds

25. 25May 5th, 2004 Karan Vahi, ISI vahi@isi.edu DAX Driven Configuration(1) l Pegasus uses IPAC/JPL webservice as an abstract workflow generator l Pegasus takes in this abstract workflow and creates a concrete workflow by consulting the various grid services described before

26. 26May 5th, 2004 Karan Vahi, ISI vahi@isi.edu (5) Full Abstract Dag DAX Driven Configuration(2) IPAC/JPL Service MCS RLS MDS Transformation Catalog Current State Generator Request Manager Concrete Planner Abstract Dag Reduction Abstract and Concrete Planner VDL Generator Submit File Generator DAGMan Submission & Monitoring Condor-G/ DAGMan (1) Abstract Workflow (DAG) (2) Abstract Dag (3) Logical File Names (LFN’s) (4) Physical File Names (PFN’s) (6) Reduced Abstract DAG (7) Logical Transformations (8) Physical Transformations and Execution Environment Information (9) Concrete Dag (10) Concrete Dag (11) DAGMan files (13) DAG(14) Log files (15) Monitoring (16) Results (12) DAGMan files

27. 27May 5th, 2004 Karan Vahi, ISI vahi@isi.edu DAG Reduction l Abstract Dag Reduction –Pegasus queries the RLS with the LFN’s referred to in the Abstract Workflow –If data products are found to be already materialized, Pegasus reuses them and thus reduces the complexity of CW

28. 28May 5th, 2004 Karan Vahi, ISI vahi@isi.edu KEY The original node Pull transfer node Registration node Push transfer node Job e Job gJob h Job d Job a Job c Job f Job i Job b Abstract Dag Reduction Pegasus Queries the RLS and finds the data products of jobs d,e,f already materialized. Hence deletes those jobs On applying the reduction algorithm additional jobs a,b,c are deleted Implemented by Karan Vahi

29. 29May 5th, 2004 Karan Vahi, ISI vahi@isi.edu Pegasus adds replica nodes for each job that materializes data (g, h, i ). These three nodes are for transferring the output files of the leaf job (f) to the output pool, since job f has been deleted by the Reduction Algorithm. KEY The original node Pull transfer node Registration node Push transfer node Node deleted by Reduction algo Inter-pool transfer node Job e Job gJob h Job d Job a Job c Job f Job i Job b Concrete Planner (1) Pegasus schedules job g,h on pool X and job i on pool Y. Hence adding an interpool transfer node Pegasus adds transfer nodes for transferring the input files for the root nodes of the decomposed dag (job g) Implemented by Karan Vahi

30. 30May 5th, 2004 Karan Vahi, ISI vahi@isi.edu Transient Files l Selective Transfer of output files –Data Sets generated by intermediate nodes in DAG are huge –However, user maybe interested only in outputs of selected jobs –Transfer of all the files could severely overload the jobmanagers on the compute sites l Need For Selective Transfer of Files –For each file at the virtual data, user can specify whether it is transient or not. –Pegasus bases it’s decision on whether to transfer the file or not on this. Implemented by Karan Vahi

31. 31May 5th, 2004 Karan Vahi, ISI vahi@isi.edu Outline l Introduction to Planning l DAX l Pegasus l Portal l Demonstration

32. 32May 5th, 2004 Karan Vahi, ISI vahi@isi.edu Portal Architecture

33. 33May 5th, 2004 Karan Vahi, ISI vahi@isi.edu Portal Demonstration

34. 34May 5th, 2004 Karan Vahi, ISI vahi@isi.edu Outline l Introduction to Planning l DAX l Pegasus l Portal l Demonstration

35. 35May 5th, 2004 Karan Vahi, ISI vahi@isi.edu Demonstration l Run a small black diamond dag using both full ahead planning and deferred planning on the isi condor pool. l Show the various configuration files (tc.data and pool.config) and how to generate them (pool.config). l Generate the condor submit files. l Submit the condor dag to condor dagman.

36. 36May 5th, 2004 Karan Vahi, ISI vahi@isi.edu Software Required!! l Submit Host –Condor DAGMAN (to submit the workflows on the grid). –Java 1.4 (to run Pegasus) –Globus 2.4 or higher –Globus RLS (the registration jobs run on the local host). –Xerces, ant, cog etc that come with the VDS distribution l Compute Sites (Machines in the pool) –Globus 2.4 or higher (gridftp server, g-u-c, MDS) –On one machine per pool, an lrc should be running. –Condor daemon running. –Various jobmanagers correctly configured.

37. 37May 5th, 2004 Karan Vahi, ISI vahi@isi.edu TC File l Walk through the editing of TC file. l A command line client is also in the works that allows you to update, add and modify the entries in your transformation catalog regardless of the underlying implementation.

38. 38May 5th, 2004 Karan Vahi, ISI vahi@isi.edu GenPoolConfig (Demo) l genpoolconfig is the client to generate the pool config file required by Pegasus. l It queries the MDS and/or a local pool config file (text based) and generates a XML file. l Am going to generate the pool config file from the text based configuration. l Usage : –genpoolconfig –Dvds.giis.host - Dvds.giis.dn --poolconfig --output

39. 39May 5th, 2004 Karan Vahi, ISI vahi@isi.edu gencdag l The Concrete planner takes the DAX produced by Chimera and converts into a set of condor dag and submit files. l Usage : gencdag –dax|--pdax --p [--dir ] [--o ] [-- force] l You can specify more then one execution pools. Execution will take place on the pools on which the executable exists. If the executable exists on more then one pool then the pool on which the executable will run is selected randomly. l Output pool is the pool where you want all the output products to be transferred to. If not specified the materialized data stays on the execution pool

40. 40May 5th, 2004 Karan Vahi, ISI vahi@isi.edu Mei’s Exploits l Mei has been running the montage code for the past one year, including some huge 6 and 10 degree dags (for the m16 cluster). l The 6 degree runs had about 13,000 compute jobs and the 10 degree run had about 40,000 compute jobs!!! l The final mosaic files can be downloaded from http://www.isi.edu/~griphyn/out_M16_10.fits l http://www.isi.edu/~griphyn/out_M16_6.fits

41. 41May 5th, 2004 Karan Vahi, ISI vahi@isi.edu Questions?