SHIWA and Coarse-grained Workflow Interoperability Gabor Terstyanszky, University of Westminster Summer School Budapest July 2012 SHIWA is supported by the FP7 Capacities Programme under contract No. RI

2 Workflow Interoperability Challenge

SSP Objectives and Services Objectives: To create an environment which enables seamless execution of workflows of different workflow systems through workflow interoperability Services: To support the whole workflow lifecycle: editing, uploading, browsing downloading and executing workflows To provide coarse- and fine-grained workflow interoperability solution To offer Distributed Computing Infrastructure interoperability solution To provide desktop computer and portal interfaces to manage workflows Key actors: researchersworkflow engine developers workflow developers 3

4 Workflow description abstract workflow plain text which describes the inputs and outputs and explains what the workflow does Workflow implementations concrete workflow It contains all implementation files or references to these files (via eg. URLs) and also holds other data/metadata necessary to run the workflow on its associated workflow engine. Workflow configurations It contains pre-defined input files and parameters or references to these files and parameters and other data/metadata of a workflow. Workflow engines It contains files and other data/metadata necessary to execute a workflow engine on a grid site or references to them SHIWA Data Structure

5 WF Submission client Submission Service Workflow Engine B Workflow of Workflow Engine B DCI Workflow Engine A Coarse-Grained Interoperability: Single Non-native Workflow

6 J2 WF4 J1 J3 Submission client Submission Service Workflow Engine B Workflow of Workflow Engine B Workflows of Workflow Engine A Coarse-Grained Interoperability: Meta-Workflow DCI native workflows:J1, J2, J3 non-native workflows:WF4 - black boxes which are managed as legacy code applications Workflow Engine A

7 gLite DCI Globus DCI DCIs ARC DCI Unicore DCI Kepler WE GWES WE Taverna WE Triana WE MOTEUR WE ASKALON WE PGRADE WE SHIWA Portal GEMLCA admin SHIWA Science Gateway WE1WEp GEMLCA Repository WF1WFm SHIWA Repository WF1WFn WS-PGRADE Workflow engine WS-PGRADE Workflow editor GEMLCA Service GEMLCA with GIB SHIWA Proxy Server Proxy Server Galaxy WE Pegasus WE ProActive WE Workflow Engines SSP: CGI Infrastructure SHIWA Science GatewayResources native WEWS-PGRADElocal resources: invocation of locally deployed WEs portalWS-PGRADE v3.4.1WE submission to local cluster repositoryGEMLCA + SHIWA repo remote resources: through remotely pre-deployed submitterGEMLCA with GIBWEs to ARC, gLite, Globus Unicore DCIs proxy managementSHIWA Proxy Server

88 SHIWA Portal: Abstract Workflow Jobs Ports

9 SHIWA Portal: Configuring Workflow

10 SHIWA Portal: Executing Workflow

11 SHIWA Repository: Browse View domain search

12 inputs outputs data sets implementation SHIWA Repository: Browse View

13 Browse Workflows Browse Implementations Search Advanced Search Sort SHIWA Repository: Table View

14 SHIWA Repository: Execution Support Execution data

15 Taverna workflow SHIWA Portal: Legacy Code Explorer

16 SHIWA Portal: Repository Portlet Workflow Repository

17 CGI Developer Scenario: Specifying WE

18 CGI Developer Scenario: Specifying WF through portal

19 CGI Developer Scenario: Specifying WF through desktop

20 CGI User Scenario: Native WE SHIWA Repository SHIWA Portal WF1 SHIWA Science Gateway GEMLCA Service WFn WE1WEp GEMLCA Repository WE + WF WF1WFm GEMLCA with GIB WF list WS-PGRADE Workflow engine WS-PGRADE Workflow editor user SHIWA Proxy Server Proxy Server step 1 search WF step 2 edit WF Kepler WE GWES WE Taverna WE Triana WE MOTEUR WE ASKALON WE PGRADE WE Galaxy WE Pegasus WE ProActive WE Workflow Engines gLite DCI Globus DCI DCIs ARC DCI Unicore DCI step 7 run WF step 6 retrieve proxy step 5 retrieve WF step 4 submit WF step 3 retrieve WF data

SHIWA Repository SHIWA Portal WF1 SHIWA Science Gateway GEMLCA Service WFn WE1WEp GEMLCA Repository WF1WFm GEMLCA with GIB WS-PGRADE Workflow Engine WS-PGRADE Workflow Editor WE + WF user GEMLCA Client MOTEUR Workflow Engine MOTEUR Workflow editor GEMLCA UI SHIWA Proxy Server Proxy Server CGI User Scenario: Non-native WE step 1 search WF step 2 edit WF step 3 submit WF step 6 retrieve proxy step 5 retrieve WF step 7 run WF Kepler WE GWES WE Taverna WE Triana WE MOTEUR WE ASKALON WE PGRADE WE Galaxy WE Pegasus WE ProActive WE Workflow Engines gLite DCI Globus DCI DCIs ARC DCI Unicore DCI step 4 forward WF

22 SHIWA Repository SHIWA Portal WF1 SHIWA Science Gateway GEMLCA Service WFn WE1WEp GEMLCA Repository WF1WFm GEMLCA with GIB WS-PGRADE Workflow Engine WS-PGRADE Workflow Editor SHIWA Proxy Server Proxy Server gLite DCI Globus DCI DCIs ARC DCI Unicore DCI SHUWA Desktop WE user step 1 search WF step 2 retrieve WF step 3 forward WF bundle step 4 edit WF step 6 submit WF step 5 ask run WF CGI User Scenario: WF Execution via Desktop

SHIWA Repository SHIWA Portal WF1 GEMLCA admin SHIWA Science Gateway WFn WE1WFp GEMLCA Repository WF1WFm GEMLCA submitter WS-PGRADE Workflow engine WS-PGRADE Workflow editor SSP DCI & FGI Infrastructure SHIWA Science GatewayResources native WEASKALON, MOTEUR, WS-PGRADElocal WEs: Kepler, MOTEUR, Taverna & Triana WEs portalWS-PGRADE v3.4.1 submit to the local cluster repositoryGEMLCA + SHIWA remote WEs: ASKALON, GWES & MOTEUR WEs GEMLCA ARC, gLite, Globus and Unicore DCIs proxy server DCI Bridge Meta Broker Proxy Server JSDL Translator gLite DCI Globus DCI DCIs ARC DCI Unicore DCI Kepler WE GWES WE Taverna WE Triana WE MOTEUR WE ASKALON WE PGRADE WE Galaxy WE Pegasus WE ProActive WE Workflow Engines

24 SHIWA Repository SHIWA Portal WF1 SHIWA Science Gateway WFn WE1WFp GEMLCA Repository WF1WFm gLite DCI Globus DCI DCIs ARC DCI Unicore DCI WS-PGRADE Workflow engine DCI Interoperability Scenario DCI Bridge Meta Broker Proxy Server JSDL Translator WS-PGRADE Workflow editor step 1 search WF step 2 edit WF step 3 retrieve WF data step 4 translate WF step 8 submit WF step 7 retrieve certificate step 5 forward WF step 6 select DCI

25 SHIWA Simulation Platform: Access SHIWA Science Gateway (SHIWA Portal + SHIWA Repository) joint portal and repository account given by the science gateway administrator DCI resources certificate given by the relevant DCI administrator, for example the VO sysadmin Access to the SHIWA Portal ssp.shiwa-workflow.eu Access to the SHIWA Repository repo.shiwa-workflow.eu

26 Conclusions the simulation platform supports the whole lifecycle of workflows, i.e. creating, testing, uploading, browsing, downloading and running workflows users can use workflows of their own and of other workflow systems through the same user interface, i.e. they can re-cycle or share workflows created by other research teams or even other research communities workflow developers (or e-scientists) can create, run, upload workflows through the simulation platform researchers (astro physicists, earth scientists, bio scientists, etc.) can browse the workflow repository, select and run workflows through the simulation platform the simulation platform provides access to major European e- infrastructure, the European Grid Initiative (EGI) infrastructure which offers large pool of compute and storage resources