Building an European Research Community through Interoperable Workflows and Data ER-flow project Gabor Terstyanszky, University of Westminster, UK EGI Technical Forum September 2012 ER-flow is supported by the FP7 Capacities Programme under contract No. RI

2 University of WestminsterUoWUnited Kingdom Magyar Tudomanyos Akademia Szamitastechnikai es Automatizalasi Kutato Intezete MTA-SZTAKIHungary Centre National de la Recherche ScientifiqueCNRSFrance Stichting European Grid InitiativeEGI.euThe Netherlands Academic Medical Center of the University of Amsterdam AMCThe Netherlands Technische Universität DresdenTUDGermany Ludwig-Maximilians-Universität MünchenLMUGermany University College LondonUCLUnited KIngdom Trinity College DaudublinTCDIreland Istituto Nazionale di AstrofisicaINAFItaly Partners: Technology providers: CNRS, EGI.eu, MTA-SZTAKI, UoW Research Communities: Astro-PhysicsINAF Computational ChemistryLMU + TUD Helio-PhysicsTCD + UCL Life ScienceAMC Duration: September 2012 – August 2014 ER-flow Project

Project Aim and Services Aim: To provide a simulation platform for research communities to enable seamless execution of workflows of different workflow systems through workflow interoperability To investigate data interoperability issues in the workflow domain and propose solutions Services: To support the whole workflow lifecycle: editing, uploading, browsing downloading and executing workflows To provide coarse-grained workflow interoperability solution To provide GUIs to manage workflows Key actors: researchersworkflow engine developers workflow developers 3

44 Objective No. 1: To further build a European community of workflow developers and users involving a wide range of research communities which already use workflow systems and which are new to this technology. Objective No. 2: To migrate workflow based scientific applications of the supported research communities to the European Grid Infrastructure through the SHIWA Simulation Platform and to use these applications both for production runs and to promote e-Science workflow solutions for research communities. Objective No. 3: To disseminate the workflow interoperability solution of the SHIWA project among the selected research communities and identify further research communities that need the simulation platform to run their experiments. Objective No. 4: To define requirements of the supported research communities on interoperability of the scientific data in the workflow domain and to identify existing and missing protocols and standards needed to support this interoperability. Objective No. 5: To write a study on the interoperability of the scientific data in the workflow domain, make recommendations on how to achieve data and workflow interoperability with existing protocols and standard, and identify research, development and standardisation issues required to be solved in order to achieve workflow interoperability in data-intensive research. Project Objectives

55 WF submission client Submission Service Workflow Engine B Workflow of Workflow Engine B Workflow Engine A Coarse-Grained Interoperability: submitting non-native workflow DCI non-native workflow:WF - non-native workflows are black boxes which are managed as legacy code applications Workflow Engine A

Coarse-Grained Interoperability: meta-workflow submission WF2 WF4 J1 J3 submission client Submission Service Workflow Engine B Workflow of Workflow Engine B Workflows of Workflow Engine A DCI native workflows:J1, J3 and WF2 non-native workflows:WF4 Workflow Engine A

7 Data Interoperability in Workflow Domain WF2 J1 J3 WF2 ???? Challenges : How to manage different data formats of different workflow systems? Solution: Virtual Data Object concept – abstract data presentation How to transfer data among jobs and workflows? Solution: supporting standard data protocols to transfer data Meta-workflow

8 gLite DCI Globus DCI DCIs ARC DCI Unicore DCI Kepler WE GWES WE Taverna WE Triana WE MOTEUR WE ASKALON WE PGRADE WE SHIWA Portal GEMLCA admin SHIWA Science Gateway WE1WEp GEMLCA Repository WF1WFm SHIWA Repository WF1WFn WS-PGRADE Workflow engine WS-PGRADE Workflow editor GEMLCA Service GEMLCA with GIB SHIWA Proxy Server Proxy Server Galaxy WE Pegasus WE ProActive WE Workflow Engines SHIWA Science GatewayResources native WEWS-PGRADElocal resources: invocation of locally deployed WEs portalWS-PGRADE v3.5WE submission to local cluster repositoryGEMLCA + SHIWA repo remote resources: through remotely pre-deployed submitterGEMLCA with GIBWEs to ARC, gLite, Globus Unicore DCIs proxy managementSHIWA Proxy Server CGI Infrastructure

9 CGI Developer Scenario: Specifying Workflow Engine SHIWA Portal GEMLCA admin SHIWA Science Gateway WE1WEp GEMLCA Repository WF1WFm SHIWA Repository WF1WFn WS-PGRADE Workflow engine WS-PGRADE Workflow editor GEMLCA Service GEMLCA with GIB SHIWA Proxy Server Proxy Server step 1 specify WE data step 2 upload WE binary, dependencies step 3 deploy WE workflow engine developer

10 CGI Developer Scenario: Specifying Workflows SHIWA Portal SHIWA Science Gateway WE1WEp GEMLCA Repository WF1WFm SHIWA Repository WF1WFn WS-PGRADE Workflow engine WS-PGRADE Workflow editor GEMLCA Service GEMLCA with GIB SHIWA Proxy Server Proxy Server step 1 specify WF data step 2 upload WF step 3 deploy WF workflow developer

11 CGI User Scenario: Native WE - PGRADE SHIWA Repository SHIWA Portal WF1 SHIWA Science Gateway GEMLCA Service WFn GEMLCA Repository GEMLCA with GIB WE1WEp WF1WFm WS-PGRADE Workflow engine WS-PGRADE Workflow editor WF list user SHIWA Proxy Server Proxy Server step 1 search WF step 2 edit WF Kepler WE GWES WE Taverna WE Triana WE MOTEUR WE ASKALON WE PGRADE WE Galaxy WE Pegasus WE ProActive WE Workflow Engines gLite DCI Globus DCI DCIs ARC DCI Unicore DCI step 7 run WF step 6 retrieve proxy WE + WF step 5 retrieve WF step 4 submit WF step 3 retrieve WF data

12 CGI User Scenario: Native WE - MOTEUR SHIWA Repository WF1 SHIWA Science Gateway GEMLCA Service WFn WE1WEp GEMLCA Repository WF1WFm GEMLCA with GIB user SHIWA Proxy Server Proxy Server step 1 search WF Kepler WE GWES WE Taverna WE Triana WE MOTEUR WE ASKALON WE PGRADE WE Galaxy WE Pegasus WE ProActive WE Workflow Engines gLite DCI Globus DCI DCIs ARC DCI Unicore DCI step 5 retrieve proxy WE + WF step 4 retrieve WF GEMLCA Client MOTEUR Workflow Engine MOTEUR Workflow editor GEMLCA UI step 2 edit WF step 3 submit WF step 6 run WF

13 ER-flow Services Technical support: SHIWA Simulation Platform  workflow repository - uploading new workflows - browsing, searching, selecting and downloading workflows  portal - creating, editing, submitting and monitoring workflows Application support training e-scientists to use the simulation platform porting workflows to the simulation platform helping e-scientists to run workflows on the simulation platform

14 ER-flow & Research Communities Supported research communities: Astro-Physics14 workflows Computational Chemistry20 workflows Helio-Physics14 workflows Life Science20 workflows Supported research communities: number of users: minimum: 250 Number of executed workflow: 3000 Further research communities: at least four more research communities  Hydrometeorology  ???