1. Sharing Workflows through Coarse-Grained Workflow Interoperability : Sharing Workflows through Coarse-Grained Workflow Interoperability G. Terstyanszky, T. Kukla, T. Kiss, S. Winter, J.: Centre for Parallel Computing University of Westminster, London, United Kingdom P. Kacsuk, A. Balasko MTA-SZTAKI, Budapest Hungary SHIWA is supported by the FP7 Capacities Programme under contract No. RI-261585

2. 22 Computer and Automation Research Institute, Hungarian Academy of Sciences MTA SZTAKIHungary University of InnsbruckUIBKAustria Charité - Universitätsmedizin BerlinC-UBGermany French National Centre for Scientific ResearchCNRSFrance University of WestminsterUOWUnited Kingdom Cardiff UniversityCUUnited Kingdom Academic Medical Centre of the University of AmsterdamAMCNetherlands University of Southern CaliforniaUSCUSA Sharing Interoperable Workflows through Large-Scale Scientific Simulation on Available DCIs (SHIWA) Partners: Duration: July 2010 – June 2012 (extended to September 2012)

3. 3 Workflow Interoperability Challenge

4. SHIWA Objectives and Services Objectives: To create an environment which enables seamless execution of workflows of different workflow systems through workflow interoperability Services: To support the whole workflow lifecycle: editing, uploading, browsing downloading and executing workflows To provide coarse- and fine-grained workflow interoperability solution To offer Distributed Computing Infrastructure interoperability solution To provide desktop computer and portal interfaces to manage workflows Key actors: researchersworkflow engine developers workflow developers 4

5. 555 SHIWA WF Repository SHIWA WF Repository SHIWA WE Repository E-Scientist WF Developer WE Developer fMRI MRI simulator Sequencing MOTEUR …. Triana Pegasus ASKALON P-GRADE … … Upload End User Portal Developer Portal Admin Portal Browse Run IWIR front,back end WF engine XML description WF engine binary Impor t IWIR Impor t IWIR Target WF System Target WF System Master WF System Master WF System Run DCI a Run DCI a SHIWA Science Gateway WF description WF executables Embedded WF System 2 Embedded WF System 2 Run DCI c Run DCI c Source WF System Source WF System Export IWIR Export IWIR Embedded WF System 1 Embedded WF System 1 Run DCI b Run DCI b Source WF System Source WF System Export IWIR Export IWIR Embedded WF System 1 Embedded WF System 1 Run DCI b Run DCI b CGI FGI NonLinSyst simulator NonLinSyst simulator SHIWA WF enactor SHIWA WF enactor IWIR SHIWA Solution

6. 6 Workflow description abstract workflow plain text which describes the inputs and outputs and explains what the workflow does Workflow implementations concrete workflow It contains all implementation files or references to these files (via eg. URLs) and also holds other data/metadata necessary to run the workflow on its associated workflow engine. Workflow configurations It contains pre-defined input files and parameters or references to these files and parameters and other data/metadata of a workflow. Workflow engines It contains files and other data/metadata necessary to execute a workflow engine on a grid site or references to them SHIWA Data Structure

7. 7 WF Submission client Submission Service Workflow Engine B Workflow of Workflow Engine B DCI Workflow Engine A Coarse-Grained Interoperability: Single Non-native Workflow

8. 8 J2 WF4 J1 J3 GEMLCA Client GEMLCA Submission Service Workflow Engine B Workflow of Workflow Engine B Workflow of Workflow Engine A Coarse-Grained Interoperability: Meta-Workflow DCI Resources native workflows:J1, J2, J3 non-native workflows:WF4 - black boxes which are managed as legacy code applications Workflow Engine A

9. 9 gLite DCI Globus DCI DCIs ARC DCI Unicore DCI Kepler WE GWES WE Taverna WE Triana WE MOTEUR WE ASKALON WE PGRADE WE SHIWA Portal GEMLCA admin SHIWA Science Gateway WE1WEp GEMLCA Repository WF1WFm SHIWA Repository WF1WFn WS-PGRADE Workflow engine WS-PGRADE Workflow editor GEMLCA Service GEMLCA with GIB SHIWA Proxy Server Proxy Server Galaxy WE Pegasus WE ProActive WE Workflow Engines SSP: CGI Infrastructure SHIWA Science GatewayResources native WEWS-PGRADElocal resources: invocation of locally deployed WEs portalWS-PGRADE v3.4.1WE submission to local cluster repositoryGEMLCA + SHIWA repo remote resources: through remotely pre-deployed submitterGEMLCA with GIBWEs to ARC, gLite, Globus Unicore DCIs proxy managementSHIWA Proxy Server

10. 10 SHIWA Portal: Abstract Workflow Jobs Ports

11. 11 SHIWA Portal: Configuring Workflow

12. 12 SHIWA Portal: Executing Workflow

13. 13 SHIWA Repository: Browse View domain search

14. 14 inputs outputs data sets implementation SHIWA Repository: Browse View

15. 15 Browse Workflows Browse Implementations Search Advanced Search Sort SHIWA Repository: Table View

16. 16 SHIWA Repository: Execution Support Execution data

17. 17 Taverna workflow SHIWA Portal: Legacy Code Explorer

18. 18 SHIWA Portal: Repository Portlet Workflow Repository

19. 19 CGI Developer Scenario: Specifying WE

20. 20 CGI Developer Scenario: Specifying WF through portal

21. 21 CGI Developer Scenario: Specifying WF through desktop Specifying WF through desktop

22. 22 CGI User Scenario: Native WE SHIWA Repository SHIWA Portal WF1 SHIWA Science Gateway GEMLCA Service WFn WE1WEp GEMLCA Repository WE + WF WF1WFm GEMLCA with GIB WF list WS-PGRADE Workflow engine WS-PGRADE Workflow editor user SHIWA Proxy Server Proxy Server step 1 search WF step 2 edit WF Kepler WE GWES WE Taverna WE Triana WE MOTEUR WE ASKALON WE PGRADE WE Galaxy WE Pegasus WE ProActive WE Workflow Engines gLite DCI Globus DCI DCIs ARC DCI Unicore DCI step 7 run WF step 6 retrieve proxy step 5 retrieve WF step 4 submit WF step 3 retrieve WF data

23. SHIWA Repository SHIWA Portal WF1 SHIWA Science Gateway GEMLCA Service WFn WE1WEp GEMLCA Repository WF1WFm GEMLCA with GIB WS-PGRADE Workflow Engine WS-PGRADE Workflow Editor WE + WF user GEMLCA Client MOTEUR Workflow Engine MOTEUR Workflow editor GEMLCA UI SHIWA Proxy Server Proxy Server CGI User Scenario: Non-native WE step 1 search WF step 2 edit WF step 3 submit WF step 6 retrieve proxy step 5 retrieve WF step 7 run WF Kepler WE GWES WE Taverna WE Triana WE MOTEUR WE ASKALON WE PGRADE WE Galaxy WE Pegasus WE ProActive WE Workflow Engines gLite DCI Globus DCI DCIs ARC DCI Unicore DCI step 4 forward WF

24. 24 SHIWA Repository SHIWA Portal WF1 SHIWA Science Gateway GEMLCA Service WFn WE1WEp GEMLCA Repository WF1WFm GEMLCA with GIB WS-PGRADE Workflow Engine WS-PGRADE Workflow Editor SHIWA Proxy Server Proxy Server gLite DCI Globus DCI DCIs ARC DCI Unicore DCI SHUWA Desktop WE user step 1 search WF step 2 retrieve WF step 3 forward WF bundle step 4 edit WF step 6 submit WF step 5 ask run WF CGI User Scenario: WF Execution via Desktop

25. 25 SHIWA Simulation Platform: Access SHIWA Science Gateway (SHIWA Portal + SHIWA Repository) joint portal and repository account given by the science gateway administrator DCI resources certificate given by the relevant DCI administrator, for example the VO sysadmin Access to the SHIWA Portal ssp.shiwa-workflow.eu Access to the SHIWA Repository repo.shiwa-workflow.eu

26. 26 Conclusions the simulation platform supports the whole lifecycle of workflows, i.e. creating, testing, uploading, browsing, downloading and running workflows users can use workflows of their own and of other workflow systems through the same user interface, i.e. they can re-cycle or share workflows created by other research teams or even other research communities workflow developers (or e-scientists) can create, run, upload workflows through the simulation platform researchers (astro physicists, earth scientists, bio scientists, etc.) can browse the workflow repository, select and run workflows through the simulation platform the simulation platform provides access to major European e- infrastructure, the European Grid Initiative (EGI) infrastructure which offers large pool of compute and storage resources