New developments of gUSE & WS-PGRADE to support e-science gateways Peter Kacsuk and Miklos Kozlovszky MTA SZTAKI - Computer and Automation Research Institute of the Hungarian Academy of Sciences EGI User Forum 2011 Vilnius, Lithuania

2 Content 1.WS-PGRADE & gUSE in a nutshell 2.Basic and new services 3.Case studies 4.How to access

3 P-GRADE Portals around the globe National Grid portals (UK, Ireland, Croatia, Turkey, Spain, Belgium, Malaysia, Kazakhstan, Switzerland, Australia, ClGrid, etc.) HP-SEE/SEE-GRID-SCI, BalticGrid US Open Science Grid, TeraGrid IDGF (International Desktop Grid Federation) SHIWA Simulation Platform Economy-Grid, Swiss BioGrid, Bio és Biomed Virtual Organisations, MathGrid, etc. HP-SEE

4 P-GRADE Portal software family P-GRADE Portal 2.4 NGS P-GRADE portal P-GRADE Portal 2.5 P-GRADE Portal 2.8 P-GRADE Portal WS-PGRADE Portal 3.1 WS-PGRADE Portal 3.2 GEMLCA Grid Legacy Code Arch. GEMLCA, storage concepts Basic concepts WS-PGRADE Portal 3.3 P-GRADE Portal 2.10 Open source from 2008 stats: 2011

5 Motivations of creating gUSE To overcome (most of) the limitations of P-GRADE portal: To provide better modularity  to replace any service To improve scalability  to millions of jobs To enable advanced dataflow patterns To interface with wider range of resources To separate Application Developer view from Application User view WS-PGRADE (Web Services Parallel Grid Runtime and Developer Environment) and gUSE (Grid User Support Environment) architecture

6 gUSE and WS-PGRADE architecture Graphical User Interface: WS-PGRADE Workflow Engine Workflow storage File storage Application repository Logging gUSE information system Submitters Gridsphere  LIFERAY Large set of portlets Autonomous Services: high level middleware service layer Resources: middleware service layer Local resources, service grid VOs, Desktop Grid resources, Web services, Databases gUSE Meta-broker Submitters File storage Submitters

7 WS-PGRADE/gUSE Creating complex workflow and parameter sweeps Seamless access to various types of resources clusters, service grids, desktop grids, databases. Scalable architecture Advanced data-flows and workflow concept Creating complex applications using embedded workflows, legacy codes Comfort features Separated views for application developers and end-users Community components from workflow repository

8 WS-PGRADE in a nutshell General purpose, workflow-oriented portal. Supports the development and execution of workflow-based applications Based on Liferay Services supported by the portal: New functionalities Web services DB connectors Embedded workflows Job level PS Conditional jobs Recursive graph Multi-generator Multi-collector CROSS product PS DOT product PS Simultaneous use of various clusters, service and desktop Grids at workflow level Basic ServicesEGEE grids (LCG2,GLite) Globus grids (GT2,GT4) Desktop grids clusters Job execution File storage Certificate management Information system Brokering Job monitoring Workflow & job visualization

9 Latest developments of gUSE gUSE service orchestration direct remote access ASM (Application Specific Module) JAVA API New submitters ARC UNICORE + SAML (Security Assertion Markup Language) support LFS support

10 gUSE service orchestration gUSE WS-PGRADE ASM MOSGRID/x Remote client (CLI) Remote access API Prog. Env. (JAVA API) Remote access module (JAVA API) Chemaxon Ltd. SHIWA Accessing gUSE services remotely direct remote access ASM (Application Specific Module) JAVA API

11 WS-PGRADE in European and national projects CancerGrid (EU FP6) ProSim (UK) EDGeS and EDGI (EU FP7) MosGrid (D-Grid) Swiss Proteomics Portal HP-SEE (EU FP7) SHIWA (EU FP7) SCI-BUS (EU FP7)

12 The CancerGrid portal for anti-cancer drug research Workflow development & configuration Algorithms configuration Workflow execution Molecule database browser Integrated components of CancerGrid portal Structure viewer

13 ProSim workflow in g-USE UK project for Protein Molecule Simulation on the Grid a combination of GEMLCA and standard g-USE jobs Executed on 5 different sites of the UK NGS Parameter sweeps in phases 3 and 4 Status: actively used in production Phase 1 Phase 2

14 MosGrid Science Gateway for Molecular Simulations Requirement: Workflow enabled grid portal for UNICORE based on Liferay Solution: Adaptation of WS-PGRADE from GridSphere to Liferay Development of UNICORE Submitter Support for SAML Development of ASM support Status: It’s prototyped and ~15 community members tested Planned production provision from May 2011 See details: Previous presentation by Sandra Gesing: A Science Gateway for Molecular Simulations

15 Swiss Proteomics Portal 19/05/10 Step 1 Step 2 Result PS1&mzXM L_1 Result PS3&mzXM L_1 Result PS2&mzXM L_1 Result PS4&mzXM L_1 Result PS6&mzXM L_1 Result PS5&mzXM L_1 PS1 ParamA = A2 ParamB = B3 ParamC = C1 TPP Workflow Parameter Sets Data Set parameters.proper ties ParamA = A1; A2 ParamB = B1; B2; B3 ParamC = C1 combine INPUT file_list.txt mzXML_1 mzXML_2 mzXML_3 mzXML_2 mzXML_1 PS2 ParamA = A2 ParamB = B2 ParamC = C1 PS3 ParamA = A2 ParamB = B1 ParamC = C1 PS4 ParamA = A1 ParamB = B3 ParamC = C1 PS5 ParamA = A1 ParamB = B2 ParamC = C1 PS6 ParamA = A1 ParamB = B1 ParamC = C1 OUTPUT Result PS3&mzXM L_2 Result PS2&mzXM L_2 Result PS4&mzXM L_2 Result PS6&mzXM L_2 Result PS5&mzXM L_2 Result PS1&mzXM L_2 Result PS1 Result PS3 Result PS2 Result PS4 Result PS6 Result PS5 Result PS1&mzXM L_1 Result PS3&mzXM L_1 Result PS2&mzXM L_1 Result PS4&mzXM L_1 Result PS6&mzXM L_1 Result PS5&mzXM L_1 Status: Production since Dec 2010 in ETH Zürich See details: Previous presentation: Proteomics portal: an e- Science Gateway for Swiss Bioinformatics

16 Creating multi-workflow applications and executing them in the multi- middleware SHIWA VO SHIWA Simulation Platform v1 (P-GRADE) and v2 (WS-PGRADE) SHIWA RepositorySHIWA Portal WF1 e-scientist SHIWA Science Gateway invoke WE GEMLCA Service WFn WE1WEp search WF GEMLCA Repository edit WF WE + WF WF1WFm GEMLCA with GIB s1 s2 s3 s5 s2 s4 WF list PGRADE Workflow engine PGRADE Workflow editor gLite DCI MOTEUR WE GWES WE Globus DCI pre-deployed- WEs MOTEUR WE Kepler WE Taverna WE Triana WE local cluster ASKALON WE SHIWA VO ASKALON WE GEMLCA admin

18 Portal Access Portal related information Access to multi-grid environment WS-PGRADE/gUSE portal For application porting GASUC (Grid Appl. Support Centre) success stories Interoperable workflow and DCI support SHIWA Simulation platform GASUC

19 Invitation to related events PUCOWO (P-GRADE User COmmunity WOrkshop), London, 7-8 June IWSG-Life (International Workshop on Science Gateways for Life Sciences), London, 8-18 June uk/iwsg-life2011/ DCI Summer School: Grid and Cloud for e-science Budapest, July

20 Thank you for your attention! Questions?