A Definition and Analysis of the Role of Meta-workflows in Workflow Interoperability Junaid Arshad, Gabor Terstyanszky, Tamas Kiss, Noam Weingarten Center for Parallel Computing, University of Westminster, London, UK (J.Arshad, G.Z.Terstyanszky, T.Kiss,

Scientific Workflows Complex computational experiments involving Analysis of large volumes of data Use of distributed/high performance computing infrastructures (DCIs) Individual tasks performing control and data analytics functions A Definition and Analysis of the Role of Meta-workflows in Workflow Interoperability 2 Collect Data Transform Data Analytics Data Store Flush Outputs External Aggregation

Meta-workflows Composed of one or more entities such as jobs and/or sub- workflows Re-use these existing entities as components Faster and more efficient development A Definition and Analysis of the Role of Meta-workflows in Workflow Interoperability 3 Collect Data Transform Data Data Store Flush Outputs External Aggregation Analytics Join Aggreg ate results Run algorith m

Types of Workflows (1) Single Workflow all the nodes of the workflow are individual jobs that are executed and managed by a single workflow system such as P-GRADE [1], Galaxy [2] or Taverna [3] A Definition and Analysis of the Role of Meta-workflows in Workflow Interoperability 4

5 Single workflow Native meta-workflow Non-native meta-workflow Types of Workflows (2) Native meta-workflow the embedded sub-workflows are all from the host workflow system (WS1) Non-native meta-workflow at least one of the embedded workflows is from external workflow systems

Non-native meta-workflows Two approaches have been established to execute non-native meta- workflows Coarse Grained Interoperability (CGI) Host workflow engine and workflows are considered native whereas all other workflows are considered non-native The non-native workflows and workflow engines are managed as legacy applications Fine Grained Interoperability (FGI) A workflow is considered to have two distinguished parts i.e. abstract and concrete Abstract includes workflow graph and abstract I/O whereas concrete part contains implementation details FGI approach creates a IWIR bundle containing IWIR representation of abstract part and the concrete part. A Definition and Analysis of the Role of Meta-workflows in Workflow Interoperability 6

Meta-workflow Execution Approaches A Definition and Analysis of the Role of Meta-workflows in Workflow Interoperability 7 Coarse-grain Workflow Interoperability usage scenarioTransformation of workflows for FGI approach

Formalizing CGI based Approach Based on existing formalization presented in [4] Objective to improve the usability of existing meta-workflows by highlighting the requirements of CGI based workflow interoperability approach Formalization achieved using the Z-notations A Definition and Analysis of the Role of Meta-workflows in Workflow Interoperability 8

9 WF meta =: {J 1 …J k, WF nat1 …WF natl, WF nnt1,..WF nntm }, where J nath - native job h = 1... k Wf nati - native workflow i = 1... l WF nntj - non-native workflow j = 1... m. Existing formalization Formal description of the CGI approach using Z

Types of CGI-based meta-workflow Focused on CGI-based meta-workflows Aimed to facilitate workflow developers to design new workflows by enabling them to comprehend with attributes and semantics of each type of workflow Following types of meta-workflows considered i.Single job meta-workflow ii.Linear multi-job meta-workflow iii.Parallel multi-job meta-workflow iv.Parameter sweep meta-workflow A Definition and Analysis of the Role of Meta-workflows in Workflow Interoperability 10

Single job meta-workflow A job representing this workflow can be a simple native job, a native workflow or even a non-native workflow A Definition and Analysis of the Role of Meta-workflows in Workflow Interoperability 11 J n = {J abs, J cnf, J cnr, J ennat } where J n : native job, J abs : abstract description of the job, J cnf : configuration of the job, J cnr : implementation of the job J ennat : native workflow engine WF n = {WF abs, WF cnr, WF cnf, WF enn } where WF abs : abstract part of workflow WF cnr : concrete part of workflow WF cnf : workflow configuration WF enn : native workflow engine WF nn = {TASKS, WF abs, WF cnr, WF cnf, WF ennn } where J nn = {J abs, J cnf, J cnr, J ennnat } J nn ∈ TASKS WF nn : non-native workflow WF ennn : non-native workflow engine Single native job Single native workflow Single non-native workflow Single Job Meta-Workflow

Linear multi-job meta-workflow A pipeline of multiple jobs in the native workflow system where any (or even all) of these jobs can be native jobs, native or non-native workflows A Definition and Analysis of the Role of Meta-workflows in Workflow Interoperability 12 WF lmj = {J 1 …, J m }={J n1 …, J nk, WF n1, …,WF ny, WF nn1, …,WF nnz } where J j job of the linear multi-job meta-workflow, j = 1 … m J ni native job ∈ J j, i = 1 … k WF np native workflow ∈ J j, p = 1… y WF nnt non-native workflow, ∈ J j, t = 1… z If J x and J y are two consecutive jobs and J x is not the last job of the workflow, there must be an edge e i = (P xo, P yi ) where P xo : output port of job J x P yi : input port of job J y Linear multi-job meta-workflow

Parallel multi-job meta-workflow A native workflow including parallel branches which can include either single jobs or native and non-native workflows, i.e. composed of several single and/or linear multi-job meta-workflows A Definition and Analysis of the Role of Meta-workflows in Workflow Interoperability 13 WF pmj = {J 1 …, J m }={J n1 …, J nk, WF n1, …,WF ny, WF nn1, …,WF nnz } where J j job of the parallel multi-job meta-workflow, j = 1 … m J ni native job ∈ J j, i = 1 … k WF np native workflow ∈ J j, p = 1… y WF nnt non-native workflow, ∈ J j, t = 1… z A parallel multi-job meta-workflow must contain an edge that connects split and worker jobs e s = (P go, P wxi ), where P so : output port of the split job J wx : worker job and x = 1 … k P wxi : input port of the worker job J wx and, an edge e m such that connects worker and merge jobs e m = (P wxo, P mi ), where P wxo : output port of a worker job J wx and x = 1 … k P mi : input port of the merge job J m and i = 1 … k Parallel multi-job meta-workflow

Parameter-sweep meta-workflow A parameter sweep workflow in the native workflow system that includes one or more non-native workflows A Definition and Analysis of the Role of Meta-workflows in Workflow Interoperability 14 WF ps = = {J 1 …, J m }={J n1 …, J nk, WF n1, …,WF ny, WF nn1, …,WF nnz } where J j job of the parallel sweep meta-workflow, j = 1 … m J ni native job ∈ J j, i = 1 … k WF np native workflow ∈ J j, p = 1… y WF nnt non-native workflow, ∈ J j, t = 1… z Connections between the generator, collector and the worker jobs of the parameter sweep meta-workflow are described by two specific types of edges e g = (P go, P wxi ) where P go : output port of the generator job J g P wxi : input port of the worker job J wx and x = 1 … k and e c = (P wxo, P ci ) where P wxo : output of a worker job J wx and x = 1 … k P ci : input port of collector job J c and i = 1 … k Parameter-sweep meta-workflow

SHIWA Simulation Platform (1) The simulation platform contains science gateway SHIWA Science Gateway a submission service SHIWA Submission Service a workflow repository SHIWA Repository a data transfer service Data Avenue Service Enables data, infrastructure and workflow interoperability at both coarse- and fine-grained level Supports the whole workflow life-cycle addressing the challenges of executing workflows of different workflow systems as non-native workflows combining workflows of different workflow systems into meta-workflows running these workflows on different DCIs The gateway is integrated with the WS-PGRADE Workflow System which is used as native workflow engine in the simulation platform Supports ASKALON, Galaxy, GWES, Kepler, MOTEUR, Pegasus, ProActive, Taverna, Triana A Definition and Analysis of the Role of Meta-workflows in Workflow Interoperability 15

A Definition and Analysis of the Role of Meta-workflows in Workflow Interoperability 16 SHIWA Simulation Platform (2)

Meta-workflows in Heliophysics (1) Simulating complex scenarios such as propagation of Coronal Mass Ejection (CME) Two types of meta-workflows used Linear multi-job meta-workflow Investigation of the fastest CME in a given period of time using an embedded Taverna workflow Parameter sweep meta-workflow finds and propagates the fastest CME over a given time range The WS-PGRADE parameter sweep extension of the Taverna meta-workflow performs the same operation over many time ranges Experiments performed using meta-workflow support in SHIWA Simulation Platform A Definition and Analysis of the Role of Meta-workflows in Workflow Interoperability 17

A Definition and Analysis of the Role of Meta-workflows in Workflow Interoperability 18 Linear multi-job meta-workflow for CME propagation Parameter sweep meta-workflow in WS-PGRADE Meta-workflows in Heliophysics (2)

Conclusions and Further Ahead Meta-workflows represent an exciting but challenging prospect within the context of workflow execution and interoperability Formalization for workflow types and CGI based approach for workflow interoperability has been achieved Significant contribution to facilitate design and development of new and existing workflows and workflow engines An example use case from Heliophysics has been presented demonstrating the effectiveness of the efforts for broader scientific domains Envisage expanding the scope of formalization to fine grained workflow interoperability approach A Definition and Analysis of the Role of Meta-workflows in Workflow Interoperability 19

References [1] A. Kertész, G. Sipos and P. Kacsuk; Brokering multi-grid workflows in the P-GRADE portal, in: Euro-Par 2006: Parallel Processing, vol. 4375, Springer, Berlin, 138–149, [2] Giardine, B., Riemer et al.; Galaxy: A platform for interactive large-scale genome analysis. Genome Research, 15(10):1451-5, [3] Oinn, T., Addis et al.; Taverna: a tool for the composition and enactment of bioinformatics workflows. Bioinformatics, 20(17): , 2004 [4] Terstyanszky, G. et al; Enabling Scientific Workflow Sharing Through Coarse Grained Interoperability in the Future Generation Com[puter Systems Vol 37, 46-59, A Definition and Analysis of the Role of Meta-workflows in Workflow Interoperability 20