Building an European Research Community through Interoperable Workflows and Data ER-flow project Gabor Terstyanszky, University of Westminster, UK EGI.

Slides:



Advertisements
Similar presentations
Enabling the execution of various workflows (Kepler, Taverna, Triana, P-GRADE) on EGEE Tamas Kukla, Tamas Kiss, Gabor Terstyanszky.
Advertisements

Running Workflows on Clouds and Grids Gabor Terstyanszky, University of Westminster T. Fahringer, P. Kacsuk, J. Montagnat, I. Taylor e-Science Workshop,
P. Kacsuk (MTA SZTAKI) – project co-ordinator on behalf of the SCI-BUS consortium SCIence gateway Based User Support (SCI-BUS)
ER-flow Purpose of the meeting C. Vuerli Contributions by G. Terstyanszky.
A Definition and Analysis of the Role of Meta-workflows in Workflow Interoperability Junaid Arshad, Gabor Terstyanszky, Tamas Kiss, Noam Weingarten Center.
EXTENDING SCIENTIFIC WORKFLOW SYSTEMS TO SUPPORT MAPREDUCE BASED APPLICATIONS IN THE CLOUD Shashank Gugnani Tamas Kiss.
Workflows Information Flows Prof. Silvia Olabarriaga Dr. Gabriele Pierantoni.
SCI-BUS is supported by the FP7 Capacities Programme under contract nr RI WS-PGRADE/gUSE Supporting e-Science communities in Europe Zoltan Farkas.
DCIs and Workflows C. Vuerli Contributions by G. Sipos, K. Varga, P. Kacsuk.
AHM /09/05 AHM 2005 Automatic Deployment and Interoperability of Grid Services G.Kecskemeti, Yonatan Zetuny, G.Terstyanszky,
ER-flow C. Vuerli Contributions by G. Terstyanszky, K. Varga.
Workflow sharing and integration services by the ER-flow project on behalf of the ER-flow consortium EGI Community Forum, Manchester,
New Communities: The Virtual Physiological Human Use Case Stefan Zasada University College London
A General and Scalable Solution of Heterogeneous Workflow Invocation and Nesting Tamas Kukla, Tamas Kiss, Gabor Terstyanszky.
1 Developing domain specific gateways based on the WS- PGRADE/gUSE framework Peter Kacsuk MTA SZTAKI Start date: Duration:
Sharing, integrating and executing different workflows in heterogeneous multi-cloud systems Peter Kacsuk MTA SZTAKI SCI-BUS is supported.
EGEE-III INFSO-RI Enabling Grids for E-sciencE Lessons learnt from the EGEE Application Porting Support activity Gergely Sipos Coordinator.
From P-GRADE to SCI-BUS Peter Kacsuk, Zoltan Farkas and Miklos Kozlovszky MTA SZTAKI - Computer and Automation Research Institute of the Hungarian Academy.
Sharing Workflows through Coarse-Grained Workflow Interoperability : Sharing Workflows through Coarse-Grained Workflow Interoperability G. Terstyanszky,
Introduction to SHIWA Technology Peter Kacsuk MTA SZTAKI and Univ.of Westminster
1 Introduction to SCI-BUS and WS-PGRADE/gUSE Peter Kacsuk MTA SZTAKI Start date: Duration: 36 months SCI-BUS is supported.
Introduction to WS-PGRADE and gUSE Tutorial Akos Balasko 04/17/
Grid Execution Management for Legacy Code Applications Grid Enabling Legacy Code Applications Tamas Kiss Centre for Parallel.
Workflow Level Grid Interoperability By GEMLCA and the P-GRADE Portal.
P-GRADE and GEMLCA.
1 P-GRADE Portal: a workflow-oriented generic application development portal Peter Kacsuk MTA SZTAKI, Hungary Univ. of Westminster, UK.
Grid Execution Management for Legacy Code Applications Grid Enabling Legacy Applications.
EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Services for advanced workflow programming.
1 RI WP4 version: 3.0 Introduction Author: Robert Lovas, Ad Emmen DEGISCO Desktop Grids for International Scientific Collaboration
Convert generic gUSE Portal into a science gateway Akos Balasko.
SHIWA and Coarse-grained Workflow Interoperability Gabor Terstyanszky, University of Westminster Summer School Budapest July 2012 SHIWA is supported.
11 Introduction to EDGI Peter Kacsuk, MTA SZTAKI Start date: Duration: 27 months EDGI.
SHIWA: Is the Workflow Interoperability a Myth or Reality PUCOWO, June 2011, London Gabor Terstyanszky, Tamas Kiss, Tamas Kukla University of Westminster.
EGI Technical Forum Amsterdam, 16 September 2010 Sylvain Reynaud.
European Middleware Initiative (EMI) Alberto Di Meglio (CERN) Project Director.
Application Specific Module Tutorial Zoltán Farkas, Ákos Balaskó 03/27/
1 SCI-BUS: building e-Science gateways in Europe: building e-Science gateways in Europe Peter Kacsuk and Zoltan Farkas MTA SZTAKI.
1 Porting applications to the NGS, using the P-GRADE portal and GEMLCA Peter Kacsuk MTA SZTAKI Hungarian Academy of Sciences Centre for.
1 SCI-BUS use case Peter Kacsuk, Sandor Acs, Mark Gergely MTA SZTAKI Start date: Duration: 36 months SCI-BUS is supported.
1 Further information and next steps Further information and next steps Gergely Sipos MTA SZTAKI
1 WS-PGRADE/gUSE generic DCI gateway framework for EGI user communities Zoltan Farkas and Peter Kacsuk MTA SZTAKI SCI-BUS is supported.
Introduction to the program of the summer school Peter Kacsuk MTA SZTAKI SCI-BUS is supported by the FP7 Capacities Programme under contract.
SCI-BUS is supported by the FP7 Capacities Programme under contract nr RI MTA SZTAKI background for the DARIAH CC Zoltan Farkas MTA SZTAKI LPDS,
Grid Execution Management for Legacy Code Architecture Exposing legacy applications as Grid services: the GEMLCA approach Centre.
1 Web: SGI - Science Gateways Initiative Delivering Science Gateway infrastructure components and solutions Stephen.
Introduction to SHIWA project EGI User Forum, Vilnius Peter Kacsuk MTA SZTAKI
Supporting Big Data Processing via Science Gateways EGI CF 2015, November, Bari, Italy Dr Tamas Kiss, CloudSME Project Director University of Westminster,
Porting workflows for the Heliophysics Community Dr. Gabriele Pierantoni Trinity College Dublin.
SHIWA Simulation Platform (SSP) Gabor Terstyanszky, University of Westminster EGI Community Forum Munnich March 2012 SHIWA is supported by the FP7.
Usage of WS-PGRADE and gUSE in European and national projects Peter Kacsuk 03/27/
1 Globe adapted from wikipedia/commons/f/fa/ Globe.svg IDGF-SP International Desktop Grid Federation - Support Project SZTAKI.
New developments of gUSE & WS-PGRADE to support e-science gateways Peter Kacsuk and Miklos Kozlovszky MTA SZTAKI - Computer and Automation Research Institute.
SHIWA project presentation Project 1 st Review Meeting, Brussels 09/11/2011 Peter Kacsuk MTA SZTAKI
SCI-BUS is supported by the FP7 Capacities Programme under contract RI ER-FLOW is supported by the FP7 Infrastructures under contract RI
Using SHIWA Workflow Interoperability Tools for Neuroimaging Data Analysis Applications Vladimir Korkhov 1, Dagmar Krefting 2, Tamas Kukla 3, Gabor Terstyanszky.
SCI-BUS project Pre-kick-off meeting University of Westminster Centre for Parallel Computing Tamas Kiss, Stephen Winter, Gabor.
1 SCI-BUS: building e-Science gateways in Europe: building e-Science gateways in Europe Peter Kacsuk MTA SZTAKI Start date:
Exposing WS-PGRADE/gUSE for large user communities Peter Kacsuk, Zoltan Farkas, Krisztian Karoczkai, Istvan Marton, Akos Hajnal,
Building an European Research Community through Interoperable Workflows and Data ER-flow Prof. Gabor Terstyanszky, University of Westminster, UK Heildelberg,
HELIOGATE – A USER EXPERIENCE PERSPECTIVE G.Pierantoni, E. Carley, B. Bentley S. Olabarriaga E. Sciacca, G. Taffoni, G. Castelli, U. Becciani.
Building an European Research Community through Interoperable Workflow and Data Gabor Terstyanszky University of Westminster.
1 SCI-BUS gateway technology for a large variety of user communities Peter Kacsuk MTA SZTAKI SCI-BUS is supported by the FP7 Capacities.
SHIWA SIMULATION PLATFORM = SSP Gabor Terstyanszky, University of Westminster e-Science Workflows Workshop Budapest 09 nd February 2012 SHIWA is supported.
EDGI European Desktop Grid Initiative
Tamas Kiss University Of Westminster
Workflows in Computational Chemistry Prof
Peter Kacsuk MTA SZTAKI
P-GRADE and GEMLCA.
Introduction to the SHIWA Simulation Platform EGI User Forum,
Presentation transcript:

Building an European Research Community through Interoperable Workflows and Data ER-flow project Gabor Terstyanszky, University of Westminster, UK EGI Technical Forum September 2012 ER-flow is supported by the FP7 Capacities Programme under contract No. RI

2 University of WestminsterUoWUnited Kingdom Magyar Tudomanyos Akademia Szamitastechnikai es Automatizalasi Kutato Intezete MTA-SZTAKIHungary Centre National de la Recherche ScientifiqueCNRSFrance Stichting European Grid InitiativeEGI.euThe Netherlands Academic Medical Center of the University of Amsterdam AMCThe Netherlands Technische Universität DresdenTUDGermany Ludwig-Maximilians-Universität MünchenLMUGermany University College LondonUCLUnited KIngdom Trinity College DaudublinTCDIreland Istituto Nazionale di AstrofisicaINAFItaly Partners: Technology providers: CNRS, EGI.eu, MTA-SZTAKI, UoW Research Communities: Astro-PhysicsINAF Computational ChemistryLMU + TUD Helio-PhysicsTCD + UCL Life ScienceAMC Duration: September 2012 – August 2014 ER-flow Project

Project Aim and Services Aim: To provide a simulation platform for research communities to enable seamless execution of workflows of different workflow systems through workflow interoperability To investigate data interoperability issues in the workflow domain and propose solutions Services: To support the whole workflow lifecycle: editing, uploading, browsing downloading and executing workflows To provide coarse-grained workflow interoperability solution To provide GUIs to manage workflows Key actors: researchersworkflow engine developers workflow developers 3

44 Objective No. 1: To further build a European community of workflow developers and users involving a wide range of research communities which already use workflow systems and which are new to this technology. Objective No. 2: To migrate workflow based scientific applications of the supported research communities to the European Grid Infrastructure through the SHIWA Simulation Platform and to use these applications both for production runs and to promote e-Science workflow solutions for research communities. Objective No. 3: To disseminate the workflow interoperability solution of the SHIWA project among the selected research communities and identify further research communities that need the simulation platform to run their experiments. Objective No. 4: To define requirements of the supported research communities on interoperability of the scientific data in the workflow domain and to identify existing and missing protocols and standards needed to support this interoperability. Objective No. 5: To write a study on the interoperability of the scientific data in the workflow domain, make recommendations on how to achieve data and workflow interoperability with existing protocols and standard, and identify research, development and standardisation issues required to be solved in order to achieve workflow interoperability in data-intensive research. Project Objectives

55 WF submission client Submission Service Workflow Engine B Workflow of Workflow Engine B Workflow Engine A Coarse-Grained Interoperability: submitting non-native workflow DCI non-native workflow:WF - non-native workflows are black boxes which are managed as legacy code applications Workflow Engine A

Coarse-Grained Interoperability: meta-workflow submission WF2 WF4 J1 J3 submission client Submission Service Workflow Engine B Workflow of Workflow Engine B Workflows of Workflow Engine A DCI native workflows:J1, J3 and WF2 non-native workflows:WF4 Workflow Engine A

7 Data Interoperability in Workflow Domain WF2 J1 J3 WF2 ???? Challenges : How to manage different data formats of different workflow systems? Solution: Virtual Data Object concept – abstract data presentation How to transfer data among jobs and workflows? Solution: supporting standard data protocols to transfer data Meta-workflow

8 gLite DCI Globus DCI DCIs ARC DCI Unicore DCI Kepler WE GWES WE Taverna WE Triana WE MOTEUR WE ASKALON WE PGRADE WE SHIWA Portal GEMLCA admin SHIWA Science Gateway WE1WEp GEMLCA Repository WF1WFm SHIWA Repository WF1WFn WS-PGRADE Workflow engine WS-PGRADE Workflow editor GEMLCA Service GEMLCA with GIB SHIWA Proxy Server Proxy Server Galaxy WE Pegasus WE ProActive WE Workflow Engines SHIWA Science GatewayResources native WEWS-PGRADElocal resources: invocation of locally deployed WEs portalWS-PGRADE v3.5WE submission to local cluster repositoryGEMLCA + SHIWA repo remote resources: through remotely pre-deployed submitterGEMLCA with GIBWEs to ARC, gLite, Globus Unicore DCIs proxy managementSHIWA Proxy Server CGI Infrastructure

9 CGI Developer Scenario: Specifying Workflow Engine SHIWA Portal GEMLCA admin SHIWA Science Gateway WE1WEp GEMLCA Repository WF1WFm SHIWA Repository WF1WFn WS-PGRADE Workflow engine WS-PGRADE Workflow editor GEMLCA Service GEMLCA with GIB SHIWA Proxy Server Proxy Server step 1 specify WE data step 2 upload WE binary, dependencies step 3 deploy WE workflow engine developer

10 CGI Developer Scenario: Specifying Workflows SHIWA Portal SHIWA Science Gateway WE1WEp GEMLCA Repository WF1WFm SHIWA Repository WF1WFn WS-PGRADE Workflow engine WS-PGRADE Workflow editor GEMLCA Service GEMLCA with GIB SHIWA Proxy Server Proxy Server step 1 specify WF data step 2 upload WF step 3 deploy WF workflow developer

11 CGI User Scenario: Native WE - PGRADE SHIWA Repository SHIWA Portal WF1 SHIWA Science Gateway GEMLCA Service WFn GEMLCA Repository GEMLCA with GIB WE1WEp WF1WFm WS-PGRADE Workflow engine WS-PGRADE Workflow editor WF list user SHIWA Proxy Server Proxy Server step 1 search WF step 2 edit WF Kepler WE GWES WE Taverna WE Triana WE MOTEUR WE ASKALON WE PGRADE WE Galaxy WE Pegasus WE ProActive WE Workflow Engines gLite DCI Globus DCI DCIs ARC DCI Unicore DCI step 7 run WF step 6 retrieve proxy WE + WF step 5 retrieve WF step 4 submit WF step 3 retrieve WF data

12 CGI User Scenario: Native WE - MOTEUR SHIWA Repository WF1 SHIWA Science Gateway GEMLCA Service WFn WE1WEp GEMLCA Repository WF1WFm GEMLCA with GIB user SHIWA Proxy Server Proxy Server step 1 search WF Kepler WE GWES WE Taverna WE Triana WE MOTEUR WE ASKALON WE PGRADE WE Galaxy WE Pegasus WE ProActive WE Workflow Engines gLite DCI Globus DCI DCIs ARC DCI Unicore DCI step 5 retrieve proxy WE + WF step 4 retrieve WF GEMLCA Client MOTEUR Workflow Engine MOTEUR Workflow editor GEMLCA UI step 2 edit WF step 3 submit WF step 6 run WF

13 ER-flow Services Technical support: SHIWA Simulation Platform  workflow repository - uploading new workflows - browsing, searching, selecting and downloading workflows  portal - creating, editing, submitting and monitoring workflows Application support training e-scientists to use the simulation platform porting workflows to the simulation platform helping e-scientists to run workflows on the simulation platform

14 ER-flow & Research Communities Supported research communities: Astro-Physics14 workflows Computational Chemistry20 workflows Helio-Physics14 workflows Life Science20 workflows Supported research communities: number of users: minimum: 250 Number of executed workflow: 3000 Further research communities: at least four more research communities  Hydrometeorology  ???