ER-flow C. Vuerli Contributions by G. Terstyanszky, K. Varga

University of WestminsterUoWUnited Kingdom Magyar Tudomanyos Akademia Szamitastechnikai es Automatizalasi Kutato Intezete MTA-SZTAKIHungary Centre National de la Recherche ScientifiqueCNRSFrance Stichting European Grid InitiativeEGI.euThe Netherlands Academic Medical Center of the University of Amsterdam AMCThe Netherlands Technische Universität DresdenTUDGermany Ludwig-Maximilians-Universität MünchenLMUGermany University College LondonUCLUnited KIngdom Trinity College DaudublinTCDIreland Istituto Nazionale di AstrofisicaINAFItaly Partners: Technology providers: CNRS, EGI.eu, MTA-SZTAKI, UoW Research Communities: Astro-PhysicsINAF Computational ChemistryLMU + TUD Helio-PhysicsTCD + UCL Life ScienceAMC Duration: September 2012 – August 2014 ER-flow Project

SHIWA vs. ER-flow What are we going to use? –SHIWA  technology –ER-flow  dissemination, user support How and where should we use the SHIWA and the ER-flow name? –On leaflets, posters, presentations?

Project Aim and Services Aim: To provide a simulation platform for research communities to enable seamless execution of workflows of different workflow systems through workflow interoperability To investigate data interoperability issues in the workflow domain and propose solutions Services: To support the whole workflow lifecycle: editing, uploading, browsing downloading and executing workflows To provide coarse-grained workflow interoperability solution To provide GUIs to manage workflows Key actors: Researchers workflow engine developers workflow developers

Objective No. 1: To further build a European community of workflow developers and users involving a wide range of research communities which already use workflow systems and which are new to this technology. Objective No. 2: To migrate workflow based scientific applications of the supported research communities to the European Grid Infrastructure through the SHIWA Simulation Platform and to use these applications both for production runs and to promote e-Science workflow solutions for research communities. Objective No. 3: To disseminate the workflow interoperability solution of the SHIWA project among the selected research communities and identify further research communities that need the simulation platform to run their experiments. Objective No. 4: To define requirements of the supported research communities on interoperability of the scientific data in the workflow domain and to identify existing and missing protocols and standards needed to support this interoperability. Objective No. 5: To write a study on the interoperability of the scientific data in the workflow domain, make recommendations on how to achieve data and workflow interoperability with existing protocols and standard, and identify research, development and standardisation issues required to be solved in order to achieve workflow interoperability in data-intensive research. Project Objectives

No.descriptionM12M24milestonesWP 01new research communities of SSP 24M2.2/M2.3WP2 02number of researchers using SSP M5.2/M5.3WP5 Objective No. 1: To further build a European community of workflow developers and users involving a wide range of research communities which already use workflow systems and which are new to this technology. Objectives and Indicators Questions: -Which communities are we addressing? -How will we reach 250 users? Which communities will they represent? -Will they workflow developers or researchers with basic IT knowledge?

Objective No. 2: To migrate workflow based scientific applications of the supported research communities to the European Grid Infrastructure through the SHIWA Simulation Platform and to use these applications both for production runs and to promote e-Science workflow solutions for research communities. No.descriptionM12M24milestonesWP 03Astro-Physics workflows614M5.2/M5.3WP5 04Computational Chemistry workflows 1020M5.2/M5.3WP5 05Helio-Physics workflows614M5.2/M5.3WP5 06Life Science workflows1020M5.2/M5.3WP5 07number of executed workflows M5.2/M5.3WP5 Objectives and Indicators Questions: -Do we have these workflows? If we don’t have them how will we identify and find them? Will they non-native or meta-workflows? -Do we have “real” users who will run 3000 times workflows available in the repository?

Objective No. 3: To disseminate the workflow interoperability solution of the SHIWA project among the selected research communities and identify further research communities that need the simulation platform to run their experiments. No.descriptionM12M24milestonesWP 08dissemination & training events for supported communities 48M2.2/M2.3WP2 09dissemination & training events for supported communities 24M2.2/M2.3WP2 10European dissemination & training events 24M2.2/M2.3WP2 11users of the User Forum200400M2.2/M2.3WP2 12publications on research achievements 48M5.2/M5.3WP5 Objectives and Indicators Questions: -What kind of dissemination and training events will we run? How will identify the dissemination and training events? Who will run these events? -How will we run the User Forum? How will we attract users to the forum?

Objective No. 4: To define requirements of the supported research communities on interoperability of the scientific data in the workflow domain and to identify existing and missing protocols and standards needed to support this interoperability. No.descriptionM12M24milestonesWP 13requirements specification for data interoperability 22M3.1/M5.1WP2 WP5 Objectives and Indicators Questions: -What kind of data interoperability issues will we target? -How will we identify these requirements? Who identify these requirements?

Objective No. 5: To write a study on the interoperability of the scientific data in the workflow domain, make recommendations on how to achieve data and workflow interoperability with existing protocols and standard, and identify research, development and standardisation issues required to be solved in order to achieve workflow interoperability in data-intensive research. No.descriptionM12M24milestonesWP 14recommendations on data interoperability 01M4.2WP4 15publications on data interoperability 04M4.2WP4 Objectives and Indicators

Work Packages and Efforts No.titlem/mWP leader startingending WP1Project Management6UoW0124 WP2Knowledge Transfer29SZTAKI0124 WP3Infrastructure & Technical Support 20UoW0124 WP4Data Interoperability in Workflow Domain 30CNRS0124 WP5Application Support76AMC0124

partnerswp1wp2wp3wp4wp5 UoW MTA-SZTAKI8424 CNRS15151 EGI.eu624 AMC3319 partnerswp1wp2wp3wp4wp5 TUD29 LMU48 UCL215 TCD18 INAF3215 Partners and Efforts

NA2 Tasks Task 2.1: Maintaining the SHIWA website and producing dissemination material (M01-M24) Task 2.2: Running the SHIWA User Forum (M01-M24) Task 2.3: Organising dissemination events for European research communities (M01-M24) –One dissemination event for each supported research community in each project year –At least two events for new research groups in the second project year Task 2.4: Running training courses (M06-M24) –Organising one training course each project year

NA2 Strategy Organize user-community-targeted events/workshops Organize user-community-targeted training events Attend and organize scientific / non-scientific events Provide Web visibility (User Forum, ER-flow website, SHIWA website, SHIWA Facebook) –Improve the website to be more informative  feedback required Initiate collaborations with other projects (MoU) –SCI-BUS and EGI InSPIRE first

abstract workflow concrete workflow workflow engine workflow configuration concrete workflow engine abstract workflow engine workflow engine configuration workflow formal description workflow engine formal description CGI Concept

WF Submission Service Workflow Engine B Workflow of Workflow Engine B Workflow Engine A CGI: submitting non-native workflow DCI non-native workflow: WF non-native workflows are black boxes which are managed as legacy code applications Workflow Engine A

CGI: meta-workflow submission native workflows: J1, J3 and WF2 non-native workflows: WF4 black boxes which are managed as legacy code applications WF2 WF3 J1 WF4 Submission Service Workflow Engine B Workflow of Workflow Engine B Workflows of Workflow Engine A DCI Workflow Engine A Workflow Repository

gLite DCI Globus DCI DCIs ARC DCI Unicore DCI Kepler WE GWES WE Taverna WE Triana WE MOTEUR WE ASKALON WE PGRADE WE SHIWA Portal GEMLCA admin SHIWA Science Gateway WE1WEp GEMLCA Repository WF1WFm SHIWA Repository WF1WFn WS-PGRADE Workflow engine WS-PGRADE Workflow editor GEMLCA Service GEMLCA with GIB SHIWA Proxy Server Proxy Server Galaxy WE Pegasus WE ProActive WE Workflow Engines SHIWA Science GatewayResources native WEWS-PGRADElocal resources: invocation of locally deployed WEs portalWS-PGRADE v3.5WE submission to local cluster repositoryGEMLCA + SHIWA repo remote resources: through remotely pre-deployed submitterGEMLCA with GIBWEs to ARC, gLite, Globus Unicore DCIs proxy managementSHIWA Proxy Server SSP: CGI Infrastructure

CGI Developer Scenario: Specifying Workflow Engine SHIWA Portal GEMLCA admin SHIWA Science Gateway WE1WEp GEMLCA Repository WF1WFm SHIWA Repository WF1WFn WS-PGRADE Workflow engine WS-PGRADE Workflow editor GEMLCA Service GEMLCA with GIB SHIWA Proxy Server Proxy Server step 1 specify WE data step 2 upload WE binary, dependencies step 3 deploy WE workflow engine developer

CGI Developer Scenario: Specifying Non-Native Workflows SHIWA Portal SHIWA Science Gateway WE1WEp GEMLCA Repository WF1WFm SHIWA Repository WF1WFn WS-PGRADE Workflow engine WS-PGRADE Workflow editor GEMLCA Service GEMLCA with GIB SHIWA Proxy Server Proxy Server step 1 specify WF data step 2 upload WF step 3 deploy WF workflow developer

CGI User Scenario: PGRADE as Native WE SHIWA Repository SHIWA Portal WF1 SHIWA Science Gateway GEMLCA Service WFn WE1WEp GEMLCA Repository WF1WFm GEMLCA with GIB WS-PGRADE Workflow engine WS-PGRADE Workflow editor WF list e-scientists SHIWA Proxy Server Proxy Server step 1 search WF step 2 edit WF Kepler WE GWES WE Taverna WE Triana WE MOTEUR WE ASKALON WE PGRADE WE Galaxy WE Pegasus WE ProActive WE Workflow Engines gLite DCI Globus DCI DCIs ARC DCI Unicore DCI step 7 run WF step 6 retrieve proxy WE + WF step 5 retrieve WF step 4 submit WF step 3 retrieve WF data

SHIWA Science Gateway GEMLCA Service SHIWA Repository Submission service WE1WEn WF1WFm user SHIWA Proxy Server Proxy Server step 1 search WF step 2 edit WF Kepler WE GWES WE Taverna WE Triana WE MOTEUR WE ASKALON WE PGRADE WE Galaxy WE Pegasus WE ProActive WE Workflow Engines gLite DCI Globus DCI DCIs ARC DCI Unicore DCI step 6 run WF step 5 retrieve proxy WE + WF step 4 retrieve WF step 3 submit WF GEMLCA Client MOTEUR Workflow Engine MOTEUR Workflow editor GEMLCA UI CGI User Scenario: MOTEUR as Native WE

SSP: Execution Infrastructure

Export to IWIR Import from IWIR WF B WF A Interoperable Workflow Intermediate Representation IWIR Fine-grained interoperability (FGI)

Robot Certificate Management

Workflow Execution Statistics

User Communities and Simulation Platform Research communities supported by the project Astro-Physics Computational Chemistry HelioPhysics Life Science Further research communities at least four further research communities will be supported candidate communities Hydrometeorology Seizmology Further communities considered