1. Workflow sharing and integration services by the ER-flow project kitti.varga@sztaki.mta.hu on behalf of the ER-flow consortium EGI Community Forum, Manchester, 8-12 April, 2013

2. Outline Motivations SHIWA  ER-flow Technology services SHIWA Simulation Platform Integrated workflow engines Example Integrating Moteur & WS-PGRADE workflows User support services 2

3. Motivation Isolation by technology –Many different workflow systems exists  We expect no change –Technological choice isolates users and user communities –Technological choice selects DCI/middleware (WS, gLite, Globus, …) Benefits of the SHIWA Simulation Platform –Share your own workflow, re-use workflows of others –Create and execute ‘meta-workflows’: built from smaller workflows that use different workflow languages/technologies –Combine workflows and DCIs SHIWA  ER-flow –SHIWA: FP7 R&D project. Created the SHIWA Simulation Platform (2010-2012) –ER-flow: FP7 support action project. Disseminates the SHIWA technology (2012- 2014). 3

4. Generic use-cases DCI #1 DCI #2 CE SE CE SE DCI – distributed computing infrastructure; CE/SE – computing/storage element

5. Generic use-cases DCI #1 DCI #2 CE SE CE SE combine WFs find and re-use with own data on own CE? DCI – distributed computing infrastructure; CE/SE – computing/storage element

6. ER-flow project 6 University of WestminsterUoWUnited Kingdom Magyar Tudomanyos Akademia Szamitastechnikai es Automatizalasi Kutato Intezete MTA-SZTAKIHungary Centre National de la Recherche ScientifiqueCNRSFrance Stichting European Grid InitiativeEGI.euThe Netherlands Academic Medical Center of the University of AmsterdamAMCThe Netherlands Technische Universität DresdenTUDGermany Ludwig-Maximilians-Universität MünchenLMUGermany University College LondonUCLUnited KIngdom Trinity College DaudublinTCDIreland Istituto Nazionale di AstrofisicaINAFItaly Technology providers + User support: MTA-SZTAKI, UoW, CNRS, EGI.eu (through the NGIs) Research Communities: Astro-physicsINAF Computational ChemistryLMU + TUD HeliophysicsTCD + UCL Life ScienceAMC + WE ARE LOOKING FOR NEW COMMUNITIES Duration: September 2012 – August 2014

7. Aims, services, actors 7 Aims: Provide a platform for research communities that enables seamless execution of workflows of different systems Investigate data interoperability issues in the workflow domain and propose solutions Services: SHIWA Simulation Platform Dissemination and outreach Training for workflow developers, workflow users Key actors: Workflow engine developers Workflow developers Researchers

8. ER-flow technology: SHIWA Simulation Platform 8

9. SHIWA Repository: –A database where workflows and meta-data are stored –Central repository to discover and share workflows within and among communities SHIWA Portal: –A web portal that is integrated with the SHIWA Repository and includes a workflow executor engine –To integrate and/or to execute workflows from the Repository SHIWA Desktop: –A desktop environment that provides similar access capabilities than the SHIWA Portal –Runs on the users’ desktops/laptops instead of a portal server Workflow systems already integrated: –ASKALON, Galaxy, GWES, Kepler, LONI Pipeline, MOTEUR, P-GRADE, PEGASUS, ProActive, Taverna, Triana, WS-PGRADE 9

10. Access to various workflow engines and DCIs where these WF applications can be run Clouds Local clusters Supercomputers Desktop grids (BOINC, Condor, etc.) Cluster grids (EGI, OSG, …) Supercomputer grids (PRACE, XSEDE) Grid systems Distributed Computing Infrastructures (DCIs) What does the developer of a (meta-)workflow need? SHIWA Repository Access to a large set of ready-to-run scientific workflow applications SHIWA Portal A portal/desktop to integrate, parameterize and run these applications Web services + Workflow engines

11. SHIWA Repository Facilitates publishing and sharing workflows Supports: Abstract workflows with multiple implementations of over 10 workflow systems Concrete workflows with execution specific data Available: From the SHIWA Portal http://ssp.shiwa-workflow.eu Standalone interface: http://repo.shiwa-workflow.eu

12. Example: Create meta-workflow from Moteur and WS-PGRADE workflows SHIWA Repository SHIWA portal DCI 1 gLite DCI 2 Cloud DCI n ARC 12 Access to Moteur WF Connected workflow engines Moteur WF engine

13. An example: SHIWA solution for LINGA Sub-Workflows Management 13 Multi- Workflow

14. Computational Chemistry Applications in ER-flow 14 workflow systemmiddlewareVO GROMACS Molecular Dynamic application Energy MinimisationWS-PGRADEgLiTE/UniCoreMoSGrid VO EquilibrationWS-PGRADEgLiTE/UniCoreMoSGrid VO Single TPRWS-PGRADEgLiTE/UniCoreMoSGrid VO CADDSuite docking application Docking with ligand generationWS-PGRADEgLiTE/UniCoreMoSGrid VO Docking without ligand generationWS-PGRADEgLiTE/UniCoreMoSGrid VO NWChem Quantum Chemistry/Molecular Dynamics application Geometry optimisationWS-PGRADEgLiTE/UniCoreMoSGrid VO Spectroscopic analysisWS-PGRADEgLiTE/UniCoreMoSGrid VO Parameter sweepWS-PGRADEgLiTE/UniCoreMoSGrid VO Transition state analysisWS-PGRADEgLiTE/UniCoreMoSGrid VO High throughputWS-PGRADEgLiTE/UniCoreMoSGrid VO Separate talks later today about Astro & Heliophysics workflows!

15. ER-flow user support SHIWA User Forum –www.erflow.eu/shiwa-user-forumwww.erflow.eu/shiwa-user-forum Workshops: –Workflow Application Porting Workshop, London 19-22 March 2013 http://www.erflow.eu/19-22-march-2013-application-porting-workshop http://www.erflow.eu/19-22-march-2013-application-porting-workshop 15

16. Conclusions Scientific workflow landscape is fragmented –Many workflow system with specialised & small user base –Many workflows with specialised & small user base SHIWA Simulation Platform enables integration –Within disciplines –Across disciplines ER-flow project provides user support –Consultancy & Workshops –User forum –Open for new communities 16

17. Questions? 17 www.erflow.eu/shiwa-user-forum www.erflow.eu