June Amsterdam A Workflow Bus for e-Science Applications Dr Zhiming Zhao Faculty of Science, University of Amsterdam VL-e SP 2.5

Outline Introduction A workflow bus and generic e-Science framework Prototype and experiment results Discussion Conclusions Future work

Scientific workflow in e-Science … Grid infrastructure, E-Science and Scientific workflow Step1: designing an experiment Step2: performing the experiment Step3: analyzing the experiment results Discovery Grid

Scientific Workflows in e-Science Experiment processes Abstract workflows Executable (concrete workflows) workflows for administration, e.g., AAA,and other issues. A SWMS is able to: Automate experiment routines Rapid prototype experimental computing systems Hide integration details between resources Manage experiment lifecycle

Insight a Scientific Workflow Management System In our view, a SWMS at least implements: A model for describing workflows; An engine for executing/managi ng workflows; Different levels of support for a user to compose, execute and control a workflow. Workflow (based on certain model) Engine User support resources Composition Engine level control Resource level control A SWMS

Diversity in SWMS Taverna: -Web services based language: Scufl; -FreeFluo: engine -Graphical viz of workflow Kepler: -Actor,director -MoML -Execution models Triana: -Components -Task graph -Data/control flow DAGMan: -Computing tasks -DAG Pegasus: -Based on DAGMan -VDL -DAG …

Research context Virtual Lab Grid Layer Application Layer Different levels of abstraction Workflow services:  Short term  Long term: a generic and effective workflow management service

Mission Effectively reuse existing workflow managements systems, and provide a generic e-Science framework for different application domains. A generic framework can Improve the reuse of workflow components and the workflows for different experiments Reduce the learning cost for different systems Allow application users to work on a consistent environment when underlying infrastructure changed

Abstract approach Extend approach Aggregate approach Possible options SWMS 1 SWMS 2 SWMS 3 SWMS G SWMS 1 SWMS 2 SWMS 3 SWMS G SWMS 1 SWMS 2 SWMS 3 SWMS 1 SWMS 2 SWMS 3 SWMS G

Why we choose an aggregation approach? Abstract approach  Build a perfect system  Difficult to find a set of systems cover all the required generic functionality; it requires re-implementation of existing things Extend approach  Incrementally development  The solution depends on a specific system Aggregate approach  Maximize the reuse of the existing workflow systems  Has to handle interoperability issues; provide customized interface existing workflow system

A workflow bus paradigm Workflow bus TavernaKepler Triana Sub workflow 1 Sub workflow 2 Sub workflow 3 Workflow A workflow bus is a special workflow system for executing meta workflows, in which sub workflows will be executed by different engines.

Architecture Terminology:  The execution of a workflow is one study, and the execution of a sub-workflow is called a sub-study, or a scenario Basic idea  Study manager schedules sub workflows  Scenario managers interface third party workflow engines and reacts to the Study manager  A user interface for composition and execution control. Network Scenario Mnger Study Mnger Taverna Engine Triana Engine User interface

Requirements A distributed framework for study and scenario managers Data input/output of a sub-workflow, description of the workflow can be described and recognized by study and scenario managers Handle the user interactions which are needed in scenarios The engine can be decoupled from a SWMS Be fault tolerant

Considerations From integration point view: study and scenario managers can be coupled by:  Web services  Object oriented middleware (CORBA, HLA, etc.)  Agent based middleware  Or an existing workflow system (Kepler, Taverna, Triana or others) The description of meta workflow The execution model of the meta workflow

A JADE/Ptolemy based prototype Director Actor JADE agent framework Scenario Mnger Study Mnger Taverna Engine Triana Engine Ptolemy User interface

How it works In user front end: a user defines meta workflow, each actor represents a sub workflow At runtime, each actor initiates a scenario agent, and passes the workflow description to the scenario manager A scenario manager controls an engine and execute the sub-workflow

Prototype

Experiment results Message delay

Cont. Overhead 10~20% performance improvement.

Discussion Challenges in supporting scientific workflows  Requirements on domain specific experiments  Generic workflow support and domain specific applications Existing workflow management systems are diverse in functionality, design and user support Related work  Interoperability among workflow systems (sister Link project)  Resource level: e.g., Kepler invokes Taverna’s resources

Applications of workflow bus Use case 1:  A user has workflow in Taverna  Some functionality is missing in Taverna but can be provided by Triana  He can develop the workflow in two systems, and run it via the workflow bus Use case 2:  A user wants to execute a Taverna or Triana workflow in multiple instances with different input data

Conclusions A workflow bus is a feasible approach to realize generic e-Science framework Multi agent technology provides a distributed environment for decomposing and encapsulating control intelligence Ptolemy II provides different computing paradigms which give user freedom to execute workflows

Future work Working on developing a scenario manager for Kepler engine. Synchronized data flow is currently used; more computing modes will be evaluated. Data provenance for workflow bus.

Referneces  Z. Zhao; A. Belloum; H. Yakali; P.M.A. Sloot and L.O. Hertzberger: Dynamic Workflow in a Grid Enabled Problem Solving Environment, in Proceedings of the 5th International Conference on Computer and Information Technology (CIT2005), pp IEEE Computer Society Press, Shanghai, China, September  Z. Zhao; A. Belloum; A. Wibisono; F. Terpstra; P.T. de Boer; P.M.A. Sloot and L.O. Hertzberger: Scientific workflow management: between generality and applicability, in Proceedings of the International Workshop on Grid and Peer-to-Peer based Workflows in conjunction with the 5th International Conference on Quality Software, pp IEEE Computer Society Press, Melbourne, Australia, September 19th-21st  Z. Zhao; A. Belloum; P.M.A. Sloot and L.O. Hertzberger: Agent Technology and Generic Workflow Management in an e-Science Environment, in Hai Zhuge and G.C. Fox, editors, Grid and Cooperative Computing - GCC 2005: 4th International Conference, Beijing, China, in series Lecture Notes in Computer Science, vol. 3795, pp Springer, November ISBN (DOI: / _61)  Z. Zhao; A. Belloum; P.M.A. Sloot and L.O. Hertzberger: Agent technology and scientific workflow management in an e-Science environment, in Proceedings of the 17th IEEE International conference on Tools with Artificial Intelligence (ICTAI05), pp IEEE Computer Society Press, Hongkong, China, November 14th-16th Acknowledgement  Suresh Booms  All the members in VL-e SP2.5