June 30 2006Amsterdam A Workflow Bus for e-Science Applications Dr Zhiming Zhao Faculty of Science, University of Amsterdam VL-e SP 2.5.

Slides:



Advertisements
Similar presentations
Nimrod/K: Towards Massively Parallel Dynamic Grid Workflows David Abramson, Colin Enticott, Monash Ilkay Altinas, UCSD.
Advertisements

The e-Framework Bill Olivier Director Development, Systems and Technology JISC.
A Workflow Engine with Multi-Level Parallelism Supports Qifeng Huang and Yan Huang School of Computer Science Cardiff University
WS-VLAM Introduction presentation ws-VLAM workflow Composer System and Network Engineering group Institute of informatics University of Amsterdam.
OBJECT ORIENTED PROGRAMMING M Taimoor Khan
Ewa Deelman, Integrating Existing Scientific Workflow Systems: The Kepler/Pegasus Example Nandita Mangal,
WS-VLAM Introduction presentation WS-VLAM Workflow Engine System and Network Engineering group Institute of informatics University of Amsterdam.
WS-VLAM Introduction presentation WS-VLAM Semantic tools Systems, Networking, and Engineering group Institute of informatics University of Amsterdam.
Transparent Robustness in Service Aggregates Onyeka Ezenwoye School of Computing and Information Sciences Florida International University May 2006.
Nadia Ranaldo - Eugenio Zimeo Department of Engineering University of Sannio – Benevento – Italy 2008 ProActive and GCM User Group Orchestrating.
Scientific workflow management in the VL-e framework Sub-program 2.5 Department of Computer Science Universiteit van Amsterdam.
GridFlow: Workflow Management for Grid Computing Kavita Shinde.
Parallel Programming Models and Paradigms
UvA, Amsterdam June 2007WS-VLAM Introduction presentation WS-VLAM Requirements list known as the WS-VLAM wishlist System and Network Engineering group.
WS-VLAM Introduction presentation WS-VLAM Introduction Systems and Network Engineering group Institute of informatics University of Amsterdam.
Reuse Activities Selecting Design Patterns and Components
Community Manager A Dynamic Collaboration Solution on Heterogeneous Environment Hyeonsook Kim  2006 CUS. All rights reserved.
Software Architecture for Mobile Distributed Computing Presented by: Deepak N Lakshminarayanan The University of Texas at Dallas Under the Guidance of.
Workflows Information Flows Prof. Silvia Olabarriaga Dr. Gabriele Pierantoni.
A Semantic Workflow Mechanism to Realise Experimental Goals and Constraints Edoardo Pignotti, Peter Edwards, Alun Preece, Nick Gotts and Gary Polhill School.
Annual SERC Research Review - Student Presentation, October 5-6, Extending Model Based System Engineering to Utilize 3D Virtual Environments Peter.
THE NEXT STEP IN WEB SERVICES By Francisco Curbera,… Memtimin MAHMUT 2012.
Adaptive Services Grid FP6 – IST Develop a prototype of an open development platform for adaptive services registration,
Scientific Workflow reusing and long term big data preservation Salima Benbernou Université Paris Descartes Project.
INFSO-SSA International Collaboration to Extend and Advance Grid Education ICEAGE Forum Meeting at EGEE Conference, Geneva Malcolm Atkinson & David.
Composing Models of Computation in Kepler/Ptolemy II
Enabling Workflow in UPnP Networks Andreas BobekUniversity of Rostock Faculty of Computer Science and Electrical Engineering Andreas Bobek, Hendrik Bohn,
CONTENTS Arrival Characters Definition Merits Chararterstics Workflows Wfms Workflow engine Workflows levels & categories.
Software Architecture Framework for Ubiquitous Computing Divya ChanneGowda Athrey Joshi.
David Abramson & Hoang Anh Nguyen Monash University.
Lecture 9: Chapter 9 Architectural Design
Recording application executions enriched with domain semantics of computations and data Master of Science Thesis Michał Pelczar Krakow,
Privacy issues in integrating R environment in scientific workflows Dr. Zhiming Zhao University of Amsterdam Virtual Laboratory for e-Science Privacy issues.
A General and Scalable Solution of Heterogeneous Workflow Invocation and Nesting Tamas Kukla, Tamas Kiss, Gabor Terstyanszky.
Odyssey A Reuse Environment based on Domain Models Prepared By: Mahmud Gabareen Eliad Cohen.
A Novel Approach to Workflow Management in Grid Environments Frank Berretz*, Sascha Skorupa*, Volker Sander*, Adam Belloum** 15/04/2010 * FH Aachen - University.
© DATAMAT S.p.A. – Giuseppe Avellino, Stefano Beco, Barbara Cantalupo, Andrea Cavallini A Semantic Workflow Authoring Tool for Programming Grids.
Scientific Workflow Scheduling in Computational Grids Report: Wei-Cheng Lee 8th Grid Computing Conference IEEE 2007 – Planning, Reservation,
11 CORE Architecture Mauro Bruno, Monica Scannapieco, Carlo Vaccari, Giulia Vaste Antonino Virgillito, Diego Zardetto (Istat)
PROSA, a reference architecture for holonic manufacturing systems dr. ir. Jo Wyns KULeuven / RealSoftware PMA KULeuven
The Grid System Design Liu Xiangrui Beijing Institute of Technology.
Workflow Project Status Update Luciano Piccoli - Fermilab, IIT Nov
August , Elsevier, Amsterdam Scientific Workflows in e-Science Dr Zhiming Zhao System and Network.
07/09/04 Johan Muskens ( TU/e Computer Science, System Architecture and Networking.
San Diego Supercomputer Center Grid Physics Network (GriPhyN) University of Florida DGL: The Assembly Language for Grid Computing Arun swaran Jagatheesan.
Streamflow - Programming Model for Data Streaming in Scientific Workflows Chathura Herath.
ICCS WSES BOF Discussion. Possible Topics Scientific workflows and Grid infrastructure Utilization of computing resources in scientific workflows; Virtual.
Enabling the Future Service-Oriented Internet (EFSOI 2008) Supporting end-to-end resource virtualization for Web 2.0 applications using Service Oriented.
Virtual Lab for e-Science Towards a new Science Paradigm.
WS-VLAM Tutorial Part I: Hands on the User Graphical Interface Adam Belloum.
Cooperative experiments in VL-e: from scientific workflows to knowledge sharing Z.Zhao (1) V. Guevara( 1) A. Wibisono(1) A. Belloum(1) M. Bubak(1,2) B.
Enabling Grids for E-sciencE Astronomical data processing workflows on a service-oriented Grid architecture Valeria Manna INAF - SI The.
Experiment Management with Microsoft Project Gregor von Laszewski Leor E. Dilmanian Link to presentation on wiki 12:13:33Service Oriented Cyberinfrastructure.
Scientific Workflow systems: Summary and Opportunities for SEEK and e-Science.
Independent Insight for Service Oriented Practice Summary: Service Reference Architecture and Planning David Sprott.
ENABLING ADAPTABILITY IN COMPOSITE SERVICES USING TRANSPARENT SHAPING TECHNIQUES Onyeka Ezenwoye Autonomic Computing Research Laboratory School of Computing.
© Fraunhofer IESE Fraunhofer IESE Multi-Language Development of Embedded Systems Dr. Thomas Kuhn Tel.:
UML - Development Process 1 Software Development Process Using UML.
Satisfying Requirements BPF for DRA shall address: –DAQ Environment (Eclipse RCP): Gumtree ISEE workbench integration; –Design Composing and Configurability,
OGCE Workflow and LEAD Overview Suresh Marru, Marlon Pierce September 2009.
Workflow Management Concepts and Requirements For Scientific Applications.
National Aeronautics and Space Administration Jet Propulsion Laboratory March 17, 2009 Workflow Orchestration: Conducting Science Efficiently on the Grid.
Service Composition Orchestration BPEL Cédric Tedeschi ISI – M2R.
Design and Manufacturing in a Distributed Computer Environment
Model-Driven Analysis Frameworks for Embedded Systems
Tools for Composing and Deploying Grid Middleware Web Services
Automated Analysis and Code Generation for Domain-Specific Models
The ViroLab Virtual Laboratory for Viral Diseases
Scientific Workflows Lecture 15
GGF10 Workflow Workshop Summary
Presentation transcript:

June Amsterdam A Workflow Bus for e-Science Applications Dr Zhiming Zhao Faculty of Science, University of Amsterdam VL-e SP 2.5

Outline Introduction A workflow bus and generic e-Science framework Prototype and experiment results Discussion Conclusions Future work

Scientific workflow in e-Science … Grid infrastructure, E-Science and Scientific workflow Step1: designing an experiment Step2: performing the experiment Step3: analyzing the experiment results Discovery Grid

Scientific Workflows in e-Science Experiment processes Abstract workflows Executable (concrete workflows) workflows for administration, e.g., AAA,and other issues. A SWMS is able to: Automate experiment routines Rapid prototype experimental computing systems Hide integration details between resources Manage experiment lifecycle

Insight a Scientific Workflow Management System In our view, a SWMS at least implements: A model for describing workflows; An engine for executing/managi ng workflows; Different levels of support for a user to compose, execute and control a workflow. Workflow (based on certain model) Engine User support resources Composition Engine level control Resource level control A SWMS

Diversity in SWMS Taverna: -Web services based language: Scufl; -FreeFluo: engine -Graphical viz of workflow Kepler: -Actor,director -MoML -Execution models Triana: -Components -Task graph -Data/control flow DAGMan: -Computing tasks -DAG Pegasus: -Based on DAGMan -VDL -DAG …

Research context Virtual Lab Grid Layer Application Layer Different levels of abstraction Workflow services:  Short term  Long term: a generic and effective workflow management service

Mission Effectively reuse existing workflow managements systems, and provide a generic e-Science framework for different application domains. A generic framework can Improve the reuse of workflow components and the workflows for different experiments Reduce the learning cost for different systems Allow application users to work on a consistent environment when underlying infrastructure changed

Abstract approach Extend approach Aggregate approach Possible options SWMS 1 SWMS 2 SWMS 3 SWMS G SWMS 1 SWMS 2 SWMS 3 SWMS G SWMS 1 SWMS 2 SWMS 3 SWMS 1 SWMS 2 SWMS 3 SWMS G

Why we choose an aggregation approach? Abstract approach  Build a perfect system  Difficult to find a set of systems cover all the required generic functionality; it requires re-implementation of existing things Extend approach  Incrementally development  The solution depends on a specific system Aggregate approach  Maximize the reuse of the existing workflow systems  Has to handle interoperability issues; provide customized interface existing workflow system

A workflow bus paradigm Workflow bus TavernaKepler Triana Sub workflow 1 Sub workflow 2 Sub workflow 3 Workflow A workflow bus is a special workflow system for executing meta workflows, in which sub workflows will be executed by different engines.

Architecture Terminology:  The execution of a workflow is one study, and the execution of a sub-workflow is called a sub-study, or a scenario Basic idea  Study manager schedules sub workflows  Scenario managers interface third party workflow engines and reacts to the Study manager  A user interface for composition and execution control. Network Scenario Mnger Study Mnger Taverna Engine Triana Engine User interface

Requirements A distributed framework for study and scenario managers Data input/output of a sub-workflow, description of the workflow can be described and recognized by study and scenario managers Handle the user interactions which are needed in scenarios The engine can be decoupled from a SWMS Be fault tolerant

Considerations From integration point view: study and scenario managers can be coupled by:  Web services  Object oriented middleware (CORBA, HLA, etc.)  Agent based middleware  Or an existing workflow system (Kepler, Taverna, Triana or others) The description of meta workflow The execution model of the meta workflow

A JADE/Ptolemy based prototype Director Actor JADE agent framework Scenario Mnger Study Mnger Taverna Engine Triana Engine Ptolemy User interface

How it works In user front end: a user defines meta workflow, each actor represents a sub workflow At runtime, each actor initiates a scenario agent, and passes the workflow description to the scenario manager A scenario manager controls an engine and execute the sub-workflow

Prototype

Experiment results Message delay

Cont. Overhead 10~20% performance improvement.

Discussion Challenges in supporting scientific workflows  Requirements on domain specific experiments  Generic workflow support and domain specific applications Existing workflow management systems are diverse in functionality, design and user support Related work  Interoperability among workflow systems (sister Link project)  Resource level: e.g., Kepler invokes Taverna’s resources

Applications of workflow bus Use case 1:  A user has workflow in Taverna  Some functionality is missing in Taverna but can be provided by Triana  He can develop the workflow in two systems, and run it via the workflow bus Use case 2:  A user wants to execute a Taverna or Triana workflow in multiple instances with different input data

Conclusions A workflow bus is a feasible approach to realize generic e-Science framework Multi agent technology provides a distributed environment for decomposing and encapsulating control intelligence Ptolemy II provides different computing paradigms which give user freedom to execute workflows

Future work Working on developing a scenario manager for Kepler engine. Synchronized data flow is currently used; more computing modes will be evaluated. Data provenance for workflow bus.

Referneces  Z. Zhao; A. Belloum; H. Yakali; P.M.A. Sloot and L.O. Hertzberger: Dynamic Workflow in a Grid Enabled Problem Solving Environment, in Proceedings of the 5th International Conference on Computer and Information Technology (CIT2005), pp IEEE Computer Society Press, Shanghai, China, September  Z. Zhao; A. Belloum; A. Wibisono; F. Terpstra; P.T. de Boer; P.M.A. Sloot and L.O. Hertzberger: Scientific workflow management: between generality and applicability, in Proceedings of the International Workshop on Grid and Peer-to-Peer based Workflows in conjunction with the 5th International Conference on Quality Software, pp IEEE Computer Society Press, Melbourne, Australia, September 19th-21st  Z. Zhao; A. Belloum; P.M.A. Sloot and L.O. Hertzberger: Agent Technology and Generic Workflow Management in an e-Science Environment, in Hai Zhuge and G.C. Fox, editors, Grid and Cooperative Computing - GCC 2005: 4th International Conference, Beijing, China, in series Lecture Notes in Computer Science, vol. 3795, pp Springer, November ISBN (DOI: / _61)  Z. Zhao; A. Belloum; P.M.A. Sloot and L.O. Hertzberger: Agent technology and scientific workflow management in an e-Science environment, in Proceedings of the 17th IEEE International conference on Tools with Artificial Intelligence (ICTAI05), pp IEEE Computer Society Press, Hongkong, China, November 14th-16th Acknowledgement  Suresh Booms  All the members in VL-e SP2.5