Building an European Research Community through Interoperable Workflows and Data ER-flow Prof. Gabor Terstyanszky, University of Westminster, UK Heildelberg, 16 May 2012 SHIWA is supported by the FP7 Capacities Programme under contract No. RI

2 Key Players and Challenges E-scientists - researchers of one particular research field, i.e. Astrophysicists, Computational Chemists, Heliophysicists etc. with basic computing knowledge - 10 or 100 thousands or millions Challenges: they are not familiar with the technology to run experiments on computing infrastructures and probably they will never learn it Workflow developers - they are familiar with both Computer Science and a particular research field - up to a few thousands Workflow engine developers - Computer Scientists with knowledge about data and compute technologies - up to a few hundreds

3 Technology Challenges

44 Clouds Local clusters Supercomputers Desktop grids (DGs) (BOINC, Condor, etc.) Cluster based service grids (SGs) (EGEE, OSG, etc.) Supercomputer based SGs (DEISA, TeraGrid) Grid systems E-science infrastructure What WE developers want and need Workflow Engine Repository They need - to publish WEs somewhere - to test access to WEs somewhere They want - to provide access to WEs

5 Clouds Local clusters Supercomputers Desktop grids (DGs) (BOINC, Condor, etc.) Cluster based service grids (SGs) (EGEE, OSG, etc.) Supercomputer based SGs (DEISA, TeraGrid) Grid systems E-science infrastructure What WF developers want and need Workflow Repository They need - to develop WFs somewhere - to publish WFs somewhere - to manage WFs somewhere - to execute WFs somewhere Science Gateway They want - to develop & publish WFs

66 Clouds Local clusters Supercomputers Desktop grids (DGs) (BOINC, Condor, etc.) Cluster based service grids (SGs) (EGEE, OSG, etc.) Supercomputer based SGs (DEISA, TeraGrid) Grid systems E-science infrastructure What e-scientists want and need Workflow Repository They need - to run experiments seamlessly i.e. executing workflows which access to data and compute resources hiding all technical details Science Gateway They want - to run experiments

gLite DCI Globus DCI DCIs ARC DCI Unicore DCI Kepler WE GWES WE Taverna WE Triana WE MOTEUR WE ASKALON WE PGRADE WE SHIWA Portal GEMLCA admin SHIWA Science Gateway WE1WEp GEMLCA Repository WF1WFm SHIWA Repository WF1WFn WS-PGRADE Workflow engine WS-PGRADE Workflow editor GEMLCA Service GEMLCA with GIB SHIWA Proxy Server Proxy Server Galaxy WE Pegasus WE ProActive WE Workflow Engines SHIWA Simulation Platform: Development Platform

88 SHIWA Portal: Editing Workflow

9 SHIWA Portal: Configuring Workflow

10 SHIWA Portal: Executing Workflow

11 SHIWA Workflow Repository

12 SHIWA Workflow Repository

13 WE Developer Scenario: SHIWA Portal GEMLCA admin SHIWA Science Gateway WE1WEp GEMLCA Repository WF1WFm SHIWA Repository WF1WFn WS-PGRADE Workflow engine WS-PGRADE Workflow editor GEMLCA Service GEMLCA with GIB SHIWA Proxy Server Proxy Server step 1 specify WE data step 2 upload WE binary, dependencies step 3 deploy WE workflow engine developer

14 WF Developer Scenario: SHIWA Portal SHIWA Science Gateway WE1WEp GEMLCA Repository WF1WFm SHIWA Repository WF1WFn WS-PGRADE Workflow engine WS-PGRADE Workflow editor GEMLCA Service GEMLCA with GIB SHIWA Proxy Server Proxy Server step 1 specify WF data step 2 upload WF step 3 deploy WF workflow developer

15 WF Developer Scenario: SHIWA Repository SHIWA Portal WF1 SHIWA Science Gateway GEMLCA Service WFn WE1WEp GEMLCA Repository WF1WFm GEMLCA with GIB WS-PGRADE Workflow engine WS-PGRADE Workflow editor WF list WF developer SHIWA Proxy Server Proxy Server step 1 search WF step 2 edit WF Kepler WE GWES WE Taverna WE Triana WE MOTEUR WE ASKALON WE PGRADE WE Galaxy WE Pegasus WE ProActive WE Workflow Engines gLite DCI Globus DCI DCIs ARC DCI Unicore DCI step 7 run WF step 6 retrieve proxy WE + WF step 5 retrieve WF step 4 submit WF step 3 retrieve WF data

16 gLite DCI Globus DCI DCIs ARC DCI Unicore DCI Kepler WE GWES WE Taverna WE Triana WE MOTEUR WE ASKALON WE PGRADE WE Galaxy WE Pegasus WE ProActive WE Workflow Engines Execution Infrastructure Astrophysics Science Gateway Computational Chemistry Science Gateway Heliophysics Science Gateway Life sciences Science Gateway SHIWA Repository WF1WFn

17 End User View

18 End User View

19 End User View

20 End User View

21 User Tailored View

22 User Tailored View

23 E-scientist Scenario: SHIWA Repository SHIWA Portal WF1 SHIWA Science Gateway GEMLCA Service WFn WE1WEp GEMLCA Repository WF1WFm GEMLCA with GIB WS-PGRADE Workflow engine WS-PGRADE End User View WF list WF developer SHIWA Proxy Server Proxy Server Select and run WF Kepler WE GWES WE Taverna WE Triana WE MOTEUR WE ASKALON WE PGRADE WE Galaxy WE Pegasus WE ProActive WE Workflow Engines gLite DCI Globus DCI DCIs ARC DCI Unicore DCI run WF retrieve proxy WE + WF retrieve WF submit WF

24 University of WestminsterUoWUnited Kingdom Magyar Tudomanyos Akademia Szamitastechnikai es Automatizalasi Kutato Intezete MTA-SZTAKIHungary Centre National de la Recherche ScientifiqueCNRSFrance Stichting European Grid InitiativeEGI.euThe Netherlands Academic Medical Center of the University of Amsterdam AMCThe Netherlands Technische Universität DresdenTUDGermany Ludwig-Maximilians-Universität MünchenLMUGermany University College LondonUCLUnited KIngdom Trinity College DaudublinTCDIreland Istituto Nazionale di AstrofisicaINAFItaly Partners: Technology providers: CNRS, EGI.eu, MTA-SZTAKI, UoW Research Communities: Astro-PhysicsINAF Computational ChemistryLMU + TUD Helio-PhysicsTCD + UCL Life ScienceAMC Duration: September 2012 – August 2014 ER-flow Project

25 nameshort descriptionWEmiddlewareVO ASTRO PHYSICS COMPCATcapture comets from the interstellar spaceWS-PGRADEgLITEAstro + VOCE VO FRANEC/BASTIevolutionary stellar modelsWS-PGRADEgLITEA&A VO LasMoGmodified gravity modelsWS-PGRADEgLITEAstro VO MESSTREAMmodelling the dynamic evolution of meteoroid streamWS-PGRADEgLITEAstro + VOCE VO PLANCKsimulation of the ESA Planck missionWS-PGRADEgLITEAstro + Planck VO VisIVOComputational CosmologyWS-PGRADEgLITEA&A VO COMPUTATIONAL CHEMISTRY GROMACS Molecular Dynamic application Energy MinimisationWS-PGRADEgLiTE/UniCoreMoSGrid VO EquilibrationWS-PGRADEgLiTE/UniCoreMoSGrid VO Single TPRWS-PGRADEgLiTE/UniCoreMoSGrid VO CADDSuite docking application Docking with ligand generationWS-PGRADEgLiTE/UniCoreMoSGrid VO Docking without ligand generationWS-PGRADEgLiTE/UniCoreMoSGrid VO NWChem Quantum Chemistry/Molecular Dynamics application Geometry optimisation = basic WF WS-PGRADEgLiTE/UniCoreMoSGrid VO Opt+freq WS-PGRADEgLiTE/UniCoreMoSGrid VO Freq WFWS-PGRADEgLiTE/UniCoreMoSGrid VO TD-DFT WFWS-PGRADEgLiTE/UniCoreMoSGrid VO Mulliken WFWS-PGRADEgLiTE/UniCoreMoSGrid VO Solvation WFWS-PGRADEgLiTE/UniCoreMoSGrid VO Spectroscopic analysis = MetaworkflowWS-PGRADEgLiTE/UniCoreMoSGrid VO Parameter sweep = Parameter WorkflowWS-PGRADEgLiTE/UniCoreMoSGrid VO Transition state seachWS-PGRADEgLiTE/UniCoreMoSGrid VO Transition state analysis = MetaWF of basic, transition state search and freq WFWS-PGRADEgLiTE/UniCoreMoSGrid VO High throughputWS-PGRADEgLiTE/UniCoreMoSGrid VO HELIO PHYSICS Monthly Event countsCounts the number of events for a period by querying the HEC catalogueTaverna 2web servicenone Detection of fastest type II CMEidentifying the fastest type II CME in timeTaverna 2web servicenone finding events in Xray and radioTaverna 2web servicenone Retrieve images for an eventretrieving solar monitor urls for votable eventsTaverna 2web servicenone Retrieve data for HEC event data Retrieves all instrument data for an event defined in the HEC catalogue and instrument defined in ICS ()Taverna 2web servicenone Associate flares with active regionsassociating flares with active regionsTaverna 2web servicenone Origin of solar wind events on EarthFinds the origins on the Sun of the Solar Wind detected on the EarthTaverna 2web servicenone LIFE SICENCE free surfermedical imagingWS-PGRADEgLITEvlemed VO double cross validationpattern classificationWS-PGRADE/gLITEvlemed VO /MOTEURgLITEvlemed VO protein dockingautodock vina with filteringWS-PGRADEgLITEvlemed VO SNP annotationDNA sequencing with AnnovarWS-PGRADE/gLITE, PBSvlemed VO /MOTEURgLITEvlemed VO SNP callinggenome data re-sequencingWS-PGRADE/gLITE, PBSvlemed VO /MOTEURgLITEvlemed VO sequence alignmentaligning sequence fragments to a reference databaseWS-PGRADE/gLITE, PBSvlemed VO /MOTEURgLITEvlemed VO indel callingcall insertions and deletions from sequence alignmentsWS-PGRADEgLITE, PBSvlemed VO sequence assembly of small genomesreconstruction of genomes by building contig sequencesWS-PGRADEgLITE, PBSvlemed VO simulation of low coverage sequence experimentsrandom samples alignments from a sequence alignment fileWS-PGRADEgLITE, PBSvlemed VO exome sequencingWS-PGRADEgLITE, PBSvlemed VO dti processingWS-PGRADEgLITEvlemed VO /MOTEURgLITE, PBSvlemed VO ER-flow Research Communities

26 SHIWA Simulation Platform: SHIWA Simulation Platform:Access SHIWA Science Gateway (SHIWA Portal + SHIWA Repository) joint portal and repository account given by the science gateway administrator DCI resources certificate given by the relevant DCI administrator, for example the VO sysadmin Access to the SHIWA Portal shiwa-portal.cpc.wmin.ac.uk/liferay-portal-6.05 Access to the SHIWA Repository shiwa-repo.cpc.wmin.ac.uk