Ewa Deelman, Integrating Existing Scientific Workflow Systems: The Kepler/Pegasus Example Nandita Mangal, Ewa Deelman, Gaurang Mehta, Mei-Hui Su, Karan Vahi USC Information Sciences Institute Marina del Rey, CA pegasus.isi.edu

Ewa Deelman, Motivation Many workflow systems exists today The choice of particular system often dictated by who you know Various workflow system have different capabilities Application components versus services Visual vs. scripting workflow descriptions Performance optimization, etc. Can you combine two separate systems? What are the issues?

Ewa Deelman, Kepler (UCSD and UCDavis) Scientific workflow management system based on Ptolemy II Allows scientists to visually design and execute scientific workflows Actor-oriented model with directors acting as the main workflow engine Enables different models of computation.

Ewa Deelman, Pegasus (USC/ISI) Based on programming language principles Leverages abstraction for workflow description to obtain ease of use, scalability, and portability Provides a compiler to map from high-level descriptions to executable workflows Correct mapping Performance enhanced mapping Relies on a runtime engine to carry out the instructions Scalable manner Reliable manner

Ewa Deelman, Combing Kepler & Pegasus Integration of Kepler visual programming environment with the grid mapping abilities of Pegasus Giving Kepler users the ability to map their large workflows onto the grid Giving Pegasus users a visual workflow composition tool Differences in the level of abstraction of workflow description

Ewa Deelman, Kepler Provenance Challenge Workflow

Ewa Deelman, Concrete Workflow Generation and Mapping

Ewa Deelman, Implementation Strategy Develop Pegasus-specific entities Abstract jobs Directors and actors “Pegasus Director” and “Pegasus Jobs (Actor Entities)” act as the main grid components to execute a given grid computation Focus mainly on abstract jobs in the Kepler environment portable and resources-knowledge independent workflow descriptions

Ewa Deelman, Integration

Ewa Deelman, Pegasus Actor & Director Entities

Ewa Deelman, Visual Abstract Workflow Creation Users can create visual models of abstract workflows and specify logical transformations without specifying grid resources

Ewa Deelman, Job Abstract Configuration --Integration with the Transformation Catalog

Ewa Deelman, Resultant Abstract Job on Kepler Canvas: A Pegasus abstract job can take in multiple input files as can output multiple output file Grid resources information is not expected in such an actor.

Ewa Deelman, Support for Concrete Jobs --- useful for monitoring and debugging A concrete job requires specific grid resources information from the scientist. Allows the scientist to directly execute jobs on the grid

Ewa Deelman, Pegasus Director/ DAX Generator Controls the execution of all the job (actor) entities and creates a resulting directed acyclic graph in XML format Generates a DAX Gives it to DAGMan for execution

Ewa Deelman, Sample DAX Generated :

Ewa Deelman,

Ewa Deelman, Provenance Challenge Workflow in Kepler/Pegasus In Kepler each node needs a unique name, so TC needs many duplicate entries

Ewa Deelman, Integration Benefit for Pegasus users Visualizing/ Debugging Existing Models: Support a scientist trying to redo/visualize or easily re-configure existing DAX Provide option to upload existing DAX files into the workspace Convert the specified DAX file into a MoML (Kepler’s) format by passing it through an XSLT processor and generating the required directors and actors on the canvas Issues of scalability (only small workflows can be visualized) Scoping may need to be applied

Ewa Deelman, Integration Issues Kepler acts a visual programming environment Actors represent single units of computation with data flow among each other Some configuration not intuitive (TC entries) There is no concept of representation of files separately in Kepler Have multiport I/O ports for each job The user is given the option to connect as many files going into and coming out of the port Potential use of integrated environment for debugging Not done Integration with Pegasus data registry No monitoring of execution in Kepler Use of Kepler’s workflow execution engine Support for Kepler actors in Pegasus

Ewa Deelman, Relevant Links Kepler: Pegasus: DAGMan: Provenance challenge: Workshop on Tuesday NSF workshop on Challenges of Scientific Workflows: