1. June 30 2006Amsterdam A Workflow Bus for e-Science Applications Dr Zhiming Zhao Faculty of Science, University of Amsterdam VL-e SP 2.5

2. Outline Introduction A workflow bus and generic e-Science framework Prototype and experiment results Discussion Conclusions Future work

3. Scientific workflow in e-Science … Grid infrastructure, E-Science and Scientific workflow Step1: designing an experiment Step2: performing the experiment Step3: analyzing the experiment results Discovery Grid

4. Scientific Workflows in e-Science Experiment processes Abstract workflows Executable (concrete workflows) workflows for administration, e.g., AAA,and other issues. A SWMS is able to: Automate experiment routines Rapid prototype experimental computing systems Hide integration details between resources Manage experiment lifecycle

5. Insight a Scientific Workflow Management System In our view, a SWMS at least implements: A model for describing workflows; An engine for executing/managi ng workflows; Different levels of support for a user to compose, execute and control a workflow. Workflow (based on certain model) Engine User support resources Composition Engine level control Resource level control A SWMS

6. Diversity in SWMS Taverna: -Web services based language: Scufl; -FreeFluo: engine -Graphical viz of workflow Kepler: -Actor,director -MoML -Execution models Triana: -Components -Task graph -Data/control flow DAGMan: -Computing tasks -DAG Pegasus: -Based on DAGMan -VDL -DAG …

7. Research context Virtual Lab Grid Layer Application Layer Different levels of abstraction Workflow services:  Short term  Long term: a generic and effective workflow management service

8. Mission Effectively reuse existing workflow managements systems, and provide a generic e-Science framework for different application domains. A generic framework can Improve the reuse of workflow components and the workflows for different experiments Reduce the learning cost for different systems Allow application users to work on a consistent environment when underlying infrastructure changed

9. Abstract approach Extend approach Aggregate approach Possible options SWMS 1 SWMS 2 SWMS 3 SWMS G SWMS 1 SWMS 2 SWMS 3 SWMS G SWMS 1 SWMS 2 SWMS 3 SWMS 1 SWMS 2 SWMS 3 SWMS G

10. Why we choose an aggregation approach? Abstract approach  Build a perfect system  Difficult to find a set of systems cover all the required generic functionality; it requires re-implementation of existing things Extend approach  Incrementally development  The solution depends on a specific system Aggregate approach  Maximize the reuse of the existing workflow systems  Has to handle interoperability issues; provide customized interface existing workflow system

11. A workflow bus paradigm Workflow bus TavernaKepler Triana Sub workflow 1 Sub workflow 2 Sub workflow 3 Workflow A workflow bus is a special workflow system for executing meta workflows, in which sub workflows will be executed by different engines.

12. Architecture Terminology:  The execution of a workflow is one study, and the execution of a sub-workflow is called a sub-study, or a scenario Basic idea  Study manager schedules sub workflows  Scenario managers interface third party workflow engines and reacts to the Study manager  A user interface for composition and execution control. Network Scenario Mnger Study Mnger Taverna Engine Triana Engine User interface

13. Requirements A distributed framework for study and scenario managers Data input/output of a sub-workflow, description of the workflow can be described and recognized by study and scenario managers Handle the user interactions which are needed in scenarios The engine can be decoupled from a SWMS Be fault tolerant

14. Considerations From integration point view: study and scenario managers can be coupled by:  Web services  Object oriented middleware (CORBA, HLA, etc.)  Agent based middleware  Or an existing workflow system (Kepler, Taverna, Triana or others) The description of meta workflow The execution model of the meta workflow

15. A JADE/Ptolemy based prototype Director Actor JADE agent framework Scenario Mnger Study Mnger Taverna Engine Triana Engine Ptolemy User interface

16. How it works In user front end: a user defines meta workflow, each actor represents a sub workflow At runtime, each actor initiates a scenario agent, and passes the workflow description to the scenario manager A scenario manager controls an engine and execute the sub-workflow

17. Prototype

18. Experiment results Message delay

19. Cont. Overhead 10~20% performance improvement.

21. Discussion Challenges in supporting scientific workflows  Requirements on domain specific experiments  Generic workflow support and domain specific applications Existing workflow management systems are diverse in functionality, design and user support Related work  Interoperability among workflow systems (sister Link project)  Resource level: e.g., Kepler invokes Taverna’s resources

22. Applications of workflow bus Use case 1:  A user has workflow in Taverna  Some functionality is missing in Taverna but can be provided by Triana  He can develop the workflow in two systems, and run it via the workflow bus Use case 2:  A user wants to execute a Taverna or Triana workflow in multiple instances with different input data

23. Conclusions A workflow bus is a feasible approach to realize generic e-Science framework Multi agent technology provides a distributed environment for decomposing and encapsulating control intelligence Ptolemy II provides different computing paradigms which give user freedom to execute workflows

24. Future work Working on developing a scenario manager for Kepler engine. Synchronized data flow is currently used; more computing modes will be evaluated. Data provenance for workflow bus.

25. Referneces  Z. Zhao; A. Belloum; H. Yakali; P.M.A. Sloot and L.O. Hertzberger: Dynamic Workflow in a Grid Enabled Problem Solving Environment, in Proceedings of the 5th International Conference on Computer and Information Technology (CIT2005), pp. 339-345. IEEE Computer Society Press, Shanghai, China, September 2005.  Z. Zhao; A. Belloum; A. Wibisono; F. Terpstra; P.T. de Boer; P.M.A. Sloot and L.O. Hertzberger: Scientific workflow management: between generality and applicability, in Proceedings of the International Workshop on Grid and Peer-to-Peer based Workflows in conjunction with the 5th International Conference on Quality Software, pp. 357-364. IEEE Computer Society Press, Melbourne, Australia, September 19th-21st 2005.  Z. Zhao; A. Belloum; P.M.A. Sloot and L.O. Hertzberger: Agent Technology and Generic Workflow Management in an e-Science Environment, in Hai Zhuge and G.C. Fox, editors, Grid and Cooperative Computing - GCC 2005: 4th International Conference, Beijing, China, in series Lecture Notes in Computer Science, vol. 3795, pp. 480-485. Springer, November 2005. ISBN 3-540-30510-6. (DOI: 10.1007/11590354_61)  Z. Zhao; A. Belloum; P.M.A. Sloot and L.O. Hertzberger: Agent technology and scientific workflow management in an e-Science environment, in Proceedings of the 17th IEEE International conference on Tools with Artificial Intelligence (ICTAI05), pp. 19-23. IEEE Computer Society Press, Hongkong, China, November 14th-16th 2005. Acknowledgement  Suresh Booms  All the members in VL-e SP2.5