Scientific workflow management in the VL-e framework Sub-program 2.5 Department of Computer Science Universiteit van Amsterdam

Outline Background –Scientific experiments, Workflow and e-Science framework Workflow management in the VL-e framework –The approach followed review the related work –Application use cases and workflow support Future work

Scientific experiments & e-Science Step1: designing an experiment Step2: performing the experiment Step3: analyzing the experiment results success Complex experiments:  have complex processes  require interdisciplinary expertise  require large scale resources Grid & high level support Scientific workflows

Scientific Workflow Management Systems in an e-Science environment Functionalities: –Automating experiment routines; –Rapid prototyping of experimental computing systems; –Hiding integration details between resources; –Managing experiment lifecycle; Cross different layers of middleware for managing: –Data; –Computing; –Information; –Knowledge. Generic Grid middleware Data management Computing tasks Information Knowledge SWMS High level workflow services Engine User support Domain specific Applications e-Science framework Grid infrastructure Workflow Management system In the VL-e project the targeted e-science framework is …

VL-e workflow wish list Classified in 4 categories: –Functionality and Capability –User interface characteristics –Run time capabilities –Software engineering aspects VL-e SIG Workflow meeting Jan 11th, 2005, 10:00–11:30, H220 (NIKHEF building) Present: Belleman, Belloum, Bouwhuis, Breanndán, Kaletas, Konijnenburg, Marshall, Rauwerda, Sterk, Sluiter, Terpstra, Vasunin, wibisono, Yakali. A list of 36 points was established to characterise the ideal workflow for the VL-e

Prioritize the workflow requirements based on the VL-e Applications Classified in 4 categories: –Application domains Model; –Engineering; –Underlying middleware; –Workflow management system : Composition/ Engine (runtime issues)/User support A list of 12 points was established to characterise the practical workflow for VL-e VL-e sub-program 2.5 in collaboration with SP1.X developers SP1.X contributors: Belleman, Klous, Konijnenburg, Marshall, Rauwerda, Sluiter, Terpstra,

Application use cases and workflow requirements Application use cases –Different rounds: a series of meetings –Distinguish workflow requirement Summary –From the resource perspective: To support legacy tools; standard middleware, e.g., web/grid services; To be able to invoke resources from different systems; Provides a rich library of workflow components; –From the application process perspective: To efficiently manage parallel processes/tasks in an experiment (Job farming); To efficiently explore large parameter space (Parameter sweep); To support knowledge based information processing (semantic level data integration). –From the perspective of using a SWMS: To provide a friendly user interface (preferably a GUI); To support the development of new workflow components ( java, scripts, C++, documentation and support); To be able to execute tasks on distributed resources (clusters or Grid); To be stable at runtime; To be able to interoperate with different workflow management systems.

Workflow management in VL-e First prototype –VLAM-G –Shortcoming (GUI, control flow, monitoring etc. + software engineering) Approach –Collect and analyze application use cases –Review the state of art of workflow systems –Propose workflow systems for the PoC environment –Be active in use case projects –Learn lessons from use cases –Propose a new design Based on the list of 36 items was established to characterize the ideal workflow for the VL-e, the VLAM-G scored: 13 Yes, 5 but need to be reimplementation, 09 No, 02 Partially supported, 6 In progress or Planned

Survey of existing workflow systems Participants: Belloum, De Boer, Guevara-Masis, Korkhov, Mirzadeh, Terpstra, van Hooft, Vasunin, wibisono, Yakali, Zhao.

Survey results Based on the survey and the practical tests on the nine workflow systems, we learn: –All of the systems are still in beta-versions (even in alpha), and have the tendency to crash when we do relatively complex tests. –None of the systems have support for collaboration, data sharing, and information management. –None of the systems enforce best practice or provide support for knowledge capture. –Most of systems are not geared to use Grid based systems, they have been built to work on a single system with some features to submit jobs on a remote host (user still exposed to some Grid related issues like writing RSLs). –We have had some problems when testing some features described in the documentation. Participants: Belloum, De Boer, Korkhov, Terpstra, van Hooft, Vasunin, wibisono, Zhao.

Recommendation for PoC R1 ( Part of the short term solution ) Participants: Belloum, De Boer, Korkhov, Terpstra, van Hooft, Vasunin, wibisono, Zhao.

Use cases and small project teams Use case project teams –Participants from SPs from P1, P2, P3 and P4. –Contributions from workflow team: distinguish reusable components and provide integration solution. –We are also active in project management, such as decomposing the implementation into concrete tasks, and track the progress. Inside SP2.5, we divide the group members –SP1.2  Belloum & Korkhov –SP1.3  Belloum & De Boer –SP1.4  Zhao & Vasunin –SP1.5  Zhao & Wibisono –SP1.6  Belloum & Paul & De Boer

Collaboration with VL-e Applications SP1.2 – AID-Food informatics-IvI –WCFS case: searching in “Research Management System” (Selected by the VLeIT) (ongoing …) SP1.3 – AMC-IvI –High-volume data management in the PoC SRB (Selected by the VLeIT) (ongoing …) SP1.4 - IBED-IvI –Run KansK toolbox in Workflow environment (Master thesis project) ( ongoing …)

Collaboration with VL-e Applications SP1.5 IBU-IvI –Histone code - semantic data integration (Selected by VLeIT) (ongoing …) –Running R scripts on multiple nodes using web service (Finished) –Running R scripts in workflows (ongoing …) –Ridge-O-grammer (ongoing …) SP1.6 AMOLF-IvI –SRB Meta data update from file header (Selected by VLeIT) (ongoing …)

SP1.2: WCFS case: searching in Research Management System” AID tools Lab. Exp In Sample Out Data Analysis In Data Out Data Situation Problem Research question Answer / conclusion LiteratureLit Report Much data in scientific research But: –No reuse: data not available across projects –No context: meaning of data not known –Not reproducible experiments –Only successful experiments traceable Wish: –Research Management System: manage experimental data for WCFS researchers

SP1.3: High-volume data management in the PoC SRB The goal of the use case is to: –Facilitate the data management and analysis for the functional MRI studies bu using PoC resources for computation and resources Matrix cluster SRB FMRI pilot is going to be developed as a first step.

SP1.4: Run KansK toolbox in Workflow environment To be integrated in workflow –VLAM The toolbox main processes are dealing with the data preparation, evaluate, prediction, and display The workflow is about the prediction of the location of the birds

SP1.5: Histone code - semantic data integration Model Alignment / Model Extension Data Acquisition e.g. Dbconnection, API, screen scraper Map e.g. Table -> RDF + model Flat map to RDFRDF to structured RDF Assign LSID’s Scaling problems –Sesame –Jena Data Import UCSC tables  RDF repository Data Exploration Extract overlapping genome locations Knowledge & Data Discovery

Read data Normalization F test Gene data generator R web services Model Raw data Normalized data FILE V plot Matrix FDR Gene data Model Local Grid Activity Data SP1.5: Running R scripts in workflows SP1.5 side (Frans and Han) SP2.5 side (Wibi, Zhiming) Define concrete description Provide UML based analysis diagrams Have a meeting: decompose the task Implement the functionality in the modules (Kepler Actor or VLAM module) Work together and give necessary support. Integrating modules into a workflow (a integration meeting) Refine the modulesRefine workflow Final demonstration

SP1.5: Ridge-O-grammer Input: Tamscriptome map Slide Window Median (SWM) Slide Window Median Probability (SWMP) Histogram of frequencies (HF) Histogram of probabilities (HP) False Discovery Rate (FDR) Output: List of Ridges The outcome of this work is going to be presented at “Netherlands Bioinformatics Conference” - 24 April 2006 identify ridges (regions of increased gene expression)

On going development Activities on the rapid prototyping environment Simple file management tools for SRB, and GridFTP R scripts in workflow system Parameters sharing of workflow components. Service discovery using P2P approach Parameter Sweep and Job farming

Future work By far the most active and rapidly progressing WMS is Kepler Beta-version March Kepler/Ptolomy has two ways of extending the Systems: Actors Directors

Summary Survey results showed that the e-science WMS targeted in VL-e – Does not exist yet – Collaboration with other Workflow project will likely speed up the development process Project teams working on application use case is the only way to progress VLAM is still quite useful for rapid prototyping

References People: Adam Belloum (SP2.5 leader), Zhiming Zhao, Paul van Hooft (post doc), Adianto Wibisono, Dmitry Vasyunin, Vladimir Korkhov, Frank Terpstra (Ph.D students), Piter de Boer (Programmer) VL-e Reports: 1.PoC recommendation report; Publications: 1.Z. Zhao; A. Belloum; H. Yakali; P.M.A. Sloot and L.O. Hertzberger: Dynamic Workflow in a Grid Enabled Problem Solving Environment, in Proceedings of the 5th International Conference on Computer and Information Technology, pp IEEE Computer Society Press, Shanghai, China, September Z. Zhao; A. Belloum; A. Wibisono; F. Terpstra; P.T. de Boer; P.M.A. Sloot and L.O. Hertzberger: Scientific workflow management: between generality and applicability, in Proceedings of the International Workshop on Grid and Peer-to-Peer based Workflows, pp IEEE Computer Society Press, Melbourne, Australia, September 19th-21st Z. Zhao; A. Belloum; P.M.A. Sloot and L.O. Hertzberger: Agent technology and scientific workflow management in an e-Science environment, in Proceedings of the 17th IEEE International conference on Tools with Artificial Intelligence, pp IEEE Computer Society Press, Hongkong, China, November 14th-16th Activity: 1.Int’l workshop on Workflow systems in e-Science, organized by Zhiming Zhao and Adam Belloum, in the context of ICCS06, Reading University, May 28, Workshop on Workflow systems in e-Science, to be held during the next e-Science conference in Amsterdam December 2006.