ER-flow C. Vuerli Contributions by G. Terstyanszky, K. Varga.

Slides:



Advertisements
Similar presentations
Enabling the execution of various workflows (Kepler, Taverna, Triana, P-GRADE) on EGEE Tamas Kukla, Tamas Kiss, Gabor Terstyanszky.
Advertisements

Running Workflows on Clouds and Grids Gabor Terstyanszky, University of Westminster T. Fahringer, P. Kacsuk, J. Montagnat, I. Taylor e-Science Workshop,
P. Kacsuk (MTA SZTAKI) – project co-ordinator on behalf of the SCI-BUS consortium SCIence gateway Based User Support (SCI-BUS)
© 2006 Open Grid Forum The Astro Community and DCIs in Europe and the role of Astro-CG C. Vuerli - INAF.
ER-flow Purpose of the meeting C. Vuerli Contributions by G. Terstyanszky.
A Definition and Analysis of the Role of Meta-workflows in Workflow Interoperability Junaid Arshad, Gabor Terstyanszky, Tamas Kiss, Noam Weingarten Center.
EXTENDING SCIENTIFIC WORKFLOW SYSTEMS TO SUPPORT MAPREDUCE BASED APPLICATIONS IN THE CLOUD Shashank Gugnani Tamas Kiss.
1 Application Specific Module for P-GRADE Portal 2.7 Application Specific Module overview Akos Balasko MTA-SZTAKI LPDS
Workflows Information Flows Prof. Silvia Olabarriaga Dr. Gabriele Pierantoni.
EUROPEAN UNION Polish Infrastructure for Supporting Computational Science in the European Research Space Cracow Grid Workshop’10 Kraków, October 11-13,
SCI-BUS is supported by the FP7 Capacities Programme under contract nr RI WS-PGRADE/gUSE Supporting e-Science communities in Europe Zoltan Farkas.
DCIs and Workflows C. Vuerli Contributions by G. Sipos, K. Varga, P. Kacsuk.
EGI: A European Distributed Computing Infrastructure Steven Newhouse Interim EGI.eu Director.
Workflow sharing and integration services by the ER-flow project on behalf of the ER-flow consortium EGI Community Forum, Manchester,
Privacy issues in integrating R environment in scientific workflows Dr. Zhiming Zhao University of Amsterdam Virtual Laboratory for e-Science Privacy issues.
A General and Scalable Solution of Heterogeneous Workflow Invocation and Nesting Tamas Kukla, Tamas Kiss, Gabor Terstyanszky.
1 Developing domain specific gateways based on the WS- PGRADE/gUSE framework Peter Kacsuk MTA SZTAKI Start date: Duration:
Sharing, integrating and executing different workflows in heterogeneous multi-cloud systems Peter Kacsuk MTA SZTAKI SCI-BUS is supported.
EGEE-III INFSO-RI Enabling Grids for E-sciencE Lessons learnt from the EGEE Application Porting Support activity Gergely Sipos Coordinator.
From P-GRADE to SCI-BUS Peter Kacsuk, Zoltan Farkas and Miklos Kozlovszky MTA SZTAKI - Computer and Automation Research Institute of the Hungarian Academy.
Sharing Workflows through Coarse-Grained Workflow Interoperability : Sharing Workflows through Coarse-Grained Workflow Interoperability G. Terstyanszky,
Introduction to SHIWA Technology Peter Kacsuk MTA SZTAKI and Univ.of Westminster
1 Introduction to SCI-BUS and WS-PGRADE/gUSE Peter Kacsuk MTA SZTAKI Start date: Duration: 36 months SCI-BUS is supported.
STAR net, Resources and VOs C. Vuerli, A. Costa, U. Becciani, P. Massimino, G. Castelli.
Introduction to WS-PGRADE and gUSE Tutorial Akos Balasko 04/17/
EMI INFSO-RI NA2 – Outreach and collaborations Status Report Emidio Giorgio (INFN Catania) Work Package Leader EMI First EC Review 22 June 2011,
Grid Execution Management for Legacy Code Applications Grid Enabling Legacy Code Applications Tamas Kiss Centre for Parallel.
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks EGEE – paving the way for a sustainable infrastructure.
Convert generic gUSE Portal into a science gateway Akos Balasko 02/07/
EGEE-III INFSO-RI Enabling Grids for E-sciencE Training services offered by SZTAKI for EGEE and EGI Gergely Sipos MTA SZTAKI (Hungarian.
Grid Execution Management for Legacy Code Applications Grid Enabling Legacy Applications.
EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Services for advanced workflow programming.
1 RI WP4 version: 3.0 Introduction Author: Robert Lovas, Ad Emmen DEGISCO Desktop Grids for International Scientific Collaboration
Convert generic gUSE Portal into a science gateway Akos Balasko.
SHIWA and Coarse-grained Workflow Interoperability Gabor Terstyanszky, University of Westminster Summer School Budapest July 2012 SHIWA is supported.
Building an European Research Community through Interoperable Workflows and Data ER-flow project Gabor Terstyanszky, University of Westminster, UK EGI.
SHIWA: Is the Workflow Interoperability a Myth or Reality PUCOWO, June 2011, London Gabor Terstyanszky, Tamas Kiss, Tamas Kukla University of Westminster.
Application Specific Module Tutorial Zoltán Farkas, Ákos Balaskó 03/27/
1 SCI-BUS: building e-Science gateways in Europe: building e-Science gateways in Europe Peter Kacsuk and Zoltan Farkas MTA SZTAKI.
1 SCI-BUS use case Peter Kacsuk, Sandor Acs, Mark Gergely MTA SZTAKI Start date: Duration: 36 months SCI-BUS is supported.
1 Further information and next steps Further information and next steps Gergely Sipos MTA SZTAKI
1 WS-PGRADE/gUSE generic DCI gateway framework for EGI user communities Zoltan Farkas and Peter Kacsuk MTA SZTAKI SCI-BUS is supported.
Introduction to the program of the summer school Peter Kacsuk MTA SZTAKI SCI-BUS is supported by the FP7 Capacities Programme under contract.
SCI-BUS is supported by the FP7 Capacities Programme under contract nr RI MTA SZTAKI background for the DARIAH CC Zoltan Farkas MTA SZTAKI LPDS,
Grid Execution Management for Legacy Code Architecture Exposing legacy applications as Grid services: the GEMLCA approach Centre.
Introduction to SHIWA project EGI User Forum, Vilnius Peter Kacsuk MTA SZTAKI
Supporting Big Data Processing via Science Gateways EGI CF 2015, November, Bari, Italy Dr Tamas Kiss, CloudSME Project Director University of Westminster,
Porting workflows for the Heliophysics Community Dr. Gabriele Pierantoni Trinity College Dublin.
SHIWA Simulation Platform (SSP) Gabor Terstyanszky, University of Westminster EGI Community Forum Munnich March 2012 SHIWA is supported by the FP7.
Usage of WS-PGRADE and gUSE in European and national projects Peter Kacsuk 03/27/
SAFE SSCs for A&A, Fusion and ES Coordinator: Claudio Vuerli, INAF, Italy.
1 Globe adapted from wikipedia/commons/f/fa/ Globe.svg IDGF-SP International Desktop Grid Federation - Support Project SZTAKI.
New developments of gUSE & WS-PGRADE to support e-science gateways Peter Kacsuk and Miklos Kozlovszky MTA SZTAKI - Computer and Automation Research Institute.
SHIWA project presentation Project 1 st Review Meeting, Brussels 09/11/2011 Peter Kacsuk MTA SZTAKI
SCI-BUS is supported by the FP7 Capacities Programme under contract RI ER-FLOW is supported by the FP7 Infrastructures under contract RI
Using SHIWA Workflow Interoperability Tools for Neuroimaging Data Analysis Applications Vladimir Korkhov 1, Dagmar Krefting 2, Tamas Kukla 3, Gabor Terstyanszky.
SCI-BUS project Pre-kick-off meeting University of Westminster Centre for Parallel Computing Tamas Kiss, Stephen Winter, Gabor.
1 SCI-BUS: building e-Science gateways in Europe: building e-Science gateways in Europe Peter Kacsuk MTA SZTAKI Start date:
Exposing WS-PGRADE/gUSE for large user communities Peter Kacsuk, Zoltan Farkas, Krisztian Karoczkai, Istvan Marton, Akos Hajnal,
Building an European Research Community through Interoperable Workflows and Data ER-flow Prof. Gabor Terstyanszky, University of Westminster, UK Heildelberg,
HELIOGATE – A USER EXPERIENCE PERSPECTIVE G.Pierantoni, E. Carley, B. Bentley S. Olabarriaga E. Sciacca, G. Taffoni, G. Castelli, U. Becciani.
Building an European Research Community through Interoperable Workflow and Data Gabor Terstyanszky University of Westminster.
EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI EGI Overview for ENVRI Gergely Sipos, Malgorzata Krakowian EGI.eu
1 SCI-BUS gateway technology for a large variety of user communities Peter Kacsuk MTA SZTAKI SCI-BUS is supported by the FP7 Capacities.
SHIWA SIMULATION PLATFORM = SSP Gabor Terstyanszky, University of Westminster e-Science Workflows Workshop Budapest 09 nd February 2012 SHIWA is supported.
EDGI European Desktop Grid Initiative
Workflows in Computational Chemistry Prof
Peter Kacsuk MTA SZTAKI
Introduction to the SHIWA Simulation Platform EGI User Forum,
Presentation transcript:

ER-flow C. Vuerli Contributions by G. Terstyanszky, K. Varga

University of WestminsterUoWUnited Kingdom Magyar Tudomanyos Akademia Szamitastechnikai es Automatizalasi Kutato Intezete MTA-SZTAKIHungary Centre National de la Recherche ScientifiqueCNRSFrance Stichting European Grid InitiativeEGI.euThe Netherlands Academic Medical Center of the University of Amsterdam AMCThe Netherlands Technische Universität DresdenTUDGermany Ludwig-Maximilians-Universität MünchenLMUGermany University College LondonUCLUnited KIngdom Trinity College DaudublinTCDIreland Istituto Nazionale di AstrofisicaINAFItaly Partners: Technology providers: CNRS, EGI.eu, MTA-SZTAKI, UoW Research Communities: Astro-PhysicsINAF Computational ChemistryLMU + TUD Helio-PhysicsTCD + UCL Life ScienceAMC Duration: September 2012 – August 2014 ER-flow Project

SHIWA vs. ER-flow What are we going to use? –SHIWA  technology –ER-flow  dissemination, user support How and where should we use the SHIWA and the ER-flow name? –On leaflets, posters, presentations?

Project Aim and Services Aim: To provide a simulation platform for research communities to enable seamless execution of workflows of different workflow systems through workflow interoperability To investigate data interoperability issues in the workflow domain and propose solutions Services: To support the whole workflow lifecycle: editing, uploading, browsing downloading and executing workflows To provide coarse-grained workflow interoperability solution To provide GUIs to manage workflows Key actors: Researchers workflow engine developers workflow developers

Objective No. 1: To further build a European community of workflow developers and users involving a wide range of research communities which already use workflow systems and which are new to this technology. Objective No. 2: To migrate workflow based scientific applications of the supported research communities to the European Grid Infrastructure through the SHIWA Simulation Platform and to use these applications both for production runs and to promote e-Science workflow solutions for research communities. Objective No. 3: To disseminate the workflow interoperability solution of the SHIWA project among the selected research communities and identify further research communities that need the simulation platform to run their experiments. Objective No. 4: To define requirements of the supported research communities on interoperability of the scientific data in the workflow domain and to identify existing and missing protocols and standards needed to support this interoperability. Objective No. 5: To write a study on the interoperability of the scientific data in the workflow domain, make recommendations on how to achieve data and workflow interoperability with existing protocols and standard, and identify research, development and standardisation issues required to be solved in order to achieve workflow interoperability in data-intensive research. Project Objectives

No.descriptionM12M24milestonesWP 01new research communities of SSP 24M2.2/M2.3WP2 02number of researchers using SSP M5.2/M5.3WP5 Objective No. 1: To further build a European community of workflow developers and users involving a wide range of research communities which already use workflow systems and which are new to this technology. Objectives and Indicators Questions: -Which communities are we addressing? -How will we reach 250 users? Which communities will they represent? -Will they workflow developers or researchers with basic IT knowledge?

Objective No. 2: To migrate workflow based scientific applications of the supported research communities to the European Grid Infrastructure through the SHIWA Simulation Platform and to use these applications both for production runs and to promote e-Science workflow solutions for research communities. No.descriptionM12M24milestonesWP 03Astro-Physics workflows614M5.2/M5.3WP5 04Computational Chemistry workflows 1020M5.2/M5.3WP5 05Helio-Physics workflows614M5.2/M5.3WP5 06Life Science workflows1020M5.2/M5.3WP5 07number of executed workflows M5.2/M5.3WP5 Objectives and Indicators Questions: -Do we have these workflows? If we don’t have them how will we identify and find them? Will they non-native or meta-workflows? -Do we have “real” users who will run 3000 times workflows available in the repository?

Objective No. 3: To disseminate the workflow interoperability solution of the SHIWA project among the selected research communities and identify further research communities that need the simulation platform to run their experiments. No.descriptionM12M24milestonesWP 08dissemination & training events for supported communities 48M2.2/M2.3WP2 09dissemination & training events for supported communities 24M2.2/M2.3WP2 10European dissemination & training events 24M2.2/M2.3WP2 11users of the User Forum200400M2.2/M2.3WP2 12publications on research achievements 48M5.2/M5.3WP5 Objectives and Indicators Questions: -What kind of dissemination and training events will we run? How will identify the dissemination and training events? Who will run these events? -How will we run the User Forum? How will we attract users to the forum?

Objective No. 4: To define requirements of the supported research communities on interoperability of the scientific data in the workflow domain and to identify existing and missing protocols and standards needed to support this interoperability. No.descriptionM12M24milestonesWP 13requirements specification for data interoperability 22M3.1/M5.1WP2 WP5 Objectives and Indicators Questions: -What kind of data interoperability issues will we target? -How will we identify these requirements? Who identify these requirements?

Objective No. 5: To write a study on the interoperability of the scientific data in the workflow domain, make recommendations on how to achieve data and workflow interoperability with existing protocols and standard, and identify research, development and standardisation issues required to be solved in order to achieve workflow interoperability in data-intensive research. No.descriptionM12M24milestonesWP 14recommendations on data interoperability 01M4.2WP4 15publications on data interoperability 04M4.2WP4 Objectives and Indicators

Work Packages and Efforts No.titlem/mWP leader startingending WP1Project Management6UoW0124 WP2Knowledge Transfer29SZTAKI0124 WP3Infrastructure & Technical Support 20UoW0124 WP4Data Interoperability in Workflow Domain 30CNRS0124 WP5Application Support76AMC0124

partnerswp1wp2wp3wp4wp5 UoW MTA-SZTAKI8424 CNRS15151 EGI.eu624 AMC3319 partnerswp1wp2wp3wp4wp5 TUD29 LMU48 UCL215 TCD18 INAF3215 Partners and Efforts

NA2 Tasks Task 2.1: Maintaining the SHIWA website and producing dissemination material (M01-M24) Task 2.2: Running the SHIWA User Forum (M01-M24) Task 2.3: Organising dissemination events for European research communities (M01-M24) –One dissemination event for each supported research community in each project year –At least two events for new research groups in the second project year Task 2.4: Running training courses (M06-M24) –Organising one training course each project year

NA2 Strategy Organize user-community-targeted events/workshops Organize user-community-targeted training events Attend and organize scientific / non-scientific events Provide Web visibility (User Forum, ER-flow website, SHIWA website, SHIWA Facebook) –Improve the website to be more informative  feedback required Initiate collaborations with other projects (MoU) –SCI-BUS and EGI InSPIRE first

abstract workflow concrete workflow workflow engine workflow configuration concrete workflow engine abstract workflow engine workflow engine configuration workflow formal description workflow engine formal description CGI Concept

WF Submission Service Workflow Engine B Workflow of Workflow Engine B Workflow Engine A CGI: submitting non-native workflow DCI non-native workflow: WF non-native workflows are black boxes which are managed as legacy code applications Workflow Engine A

CGI: meta-workflow submission native workflows: J1, J3 and WF2 non-native workflows: WF4 black boxes which are managed as legacy code applications WF2 WF3 J1 WF4 Submission Service Workflow Engine B Workflow of Workflow Engine B Workflows of Workflow Engine A DCI Workflow Engine A Workflow Repository

gLite DCI Globus DCI DCIs ARC DCI Unicore DCI Kepler WE GWES WE Taverna WE Triana WE MOTEUR WE ASKALON WE PGRADE WE SHIWA Portal GEMLCA admin SHIWA Science Gateway WE1WEp GEMLCA Repository WF1WFm SHIWA Repository WF1WFn WS-PGRADE Workflow engine WS-PGRADE Workflow editor GEMLCA Service GEMLCA with GIB SHIWA Proxy Server Proxy Server Galaxy WE Pegasus WE ProActive WE Workflow Engines SHIWA Science GatewayResources native WEWS-PGRADElocal resources: invocation of locally deployed WEs portalWS-PGRADE v3.5WE submission to local cluster repositoryGEMLCA + SHIWA repo remote resources: through remotely pre-deployed submitterGEMLCA with GIBWEs to ARC, gLite, Globus Unicore DCIs proxy managementSHIWA Proxy Server SSP: CGI Infrastructure

CGI Developer Scenario: Specifying Workflow Engine SHIWA Portal GEMLCA admin SHIWA Science Gateway WE1WEp GEMLCA Repository WF1WFm SHIWA Repository WF1WFn WS-PGRADE Workflow engine WS-PGRADE Workflow editor GEMLCA Service GEMLCA with GIB SHIWA Proxy Server Proxy Server step 1 specify WE data step 2 upload WE binary, dependencies step 3 deploy WE workflow engine developer

CGI Developer Scenario: Specifying Non-Native Workflows SHIWA Portal SHIWA Science Gateway WE1WEp GEMLCA Repository WF1WFm SHIWA Repository WF1WFn WS-PGRADE Workflow engine WS-PGRADE Workflow editor GEMLCA Service GEMLCA with GIB SHIWA Proxy Server Proxy Server step 1 specify WF data step 2 upload WF step 3 deploy WF workflow developer

CGI User Scenario: PGRADE as Native WE SHIWA Repository SHIWA Portal WF1 SHIWA Science Gateway GEMLCA Service WFn WE1WEp GEMLCA Repository WF1WFm GEMLCA with GIB WS-PGRADE Workflow engine WS-PGRADE Workflow editor WF list e-scientists SHIWA Proxy Server Proxy Server step 1 search WF step 2 edit WF Kepler WE GWES WE Taverna WE Triana WE MOTEUR WE ASKALON WE PGRADE WE Galaxy WE Pegasus WE ProActive WE Workflow Engines gLite DCI Globus DCI DCIs ARC DCI Unicore DCI step 7 run WF step 6 retrieve proxy WE + WF step 5 retrieve WF step 4 submit WF step 3 retrieve WF data

SHIWA Science Gateway GEMLCA Service SHIWA Repository Submission service WE1WEn WF1WFm user SHIWA Proxy Server Proxy Server step 1 search WF step 2 edit WF Kepler WE GWES WE Taverna WE Triana WE MOTEUR WE ASKALON WE PGRADE WE Galaxy WE Pegasus WE ProActive WE Workflow Engines gLite DCI Globus DCI DCIs ARC DCI Unicore DCI step 6 run WF step 5 retrieve proxy WE + WF step 4 retrieve WF step 3 submit WF GEMLCA Client MOTEUR Workflow Engine MOTEUR Workflow editor GEMLCA UI CGI User Scenario: MOTEUR as Native WE

SSP: Execution Infrastructure

Export to IWIR Import from IWIR WF B WF A Interoperable Workflow Intermediate Representation IWIR Fine-grained interoperability (FGI)

Robot Certificate Management

Workflow Execution Statistics

User Communities and Simulation Platform Research communities supported by the project Astro-Physics Computational Chemistry HelioPhysics Life Science Further research communities at least four further research communities will be supported candidate communities Hydrometeorology Seizmology Further communities considered