EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI EGI for ELI-HU Gergely Sipos EGI.eu Technical Outreach Manager User Community Support Team European Grid Infrastructure

EGI-InSPIRE RI Outline EGI overview –Infrastructure, communities, projects,... EGI services for RIs –Infrastructure, collaboration, user tools, capacity Possible next steps for EGI – ELI-HU collaboration 2

EGI-InSPIRE RI European Grid Infrastructure EGI European –Over 35 countries Grid –Secure sharing of IT resources Infrastructure –Computers –Data –Applications –…. and beyond!! 3

EGI-InSPIRE RI EGI Resource Infrastructure Providers: National Grid Infrastructures (Snapshot: April 2012) 4

EGI-InSPIRE RI A multi-disciplinary e-infrastructure User VO Virtual Research Community Members Virtual Organisations Research communities Largest VO communities: High Energy Physics Astronomy & Astrophysics Life Sciences Earth Sciences Computational Chemistry Fusion … ~ users in ~200 Registered VOs Sites Current VRCs: wLCG - High Energy Physics WeNMR – Structural biology LSGC – Life sciences HMRC – Hydro-Meteorology CLARIN - Humanities

EGI-InSPIRE RI EGI’s Strategic Focus Operational Infrastructure –Operate a European wide grid infrastructure –Build a federated cloud infrastructure Virtual Research Environments –Integration & operation of scientific environments Community & Coordination –Events: Community Forum, Manchester, April –Marketing, technical outreach and support Focus on Reseach Infrastructures, such as ELI 6

EGI-InSPIRE RI EGI, EGI.eu, EGI-InSPIRE EGI – an open collaboration –To support the digital European Research Area through a pan- European research infrastructure based on services federed from the NGIs EGI.eu – a Dutch fundation owned by the NGIs –To coordinate the work of EGI (operations, technology, user support, policy, community & communications, administration) –Sustainable, small organisation EGI-InSPIRE – an FP7 project –Supports EGI.eu and EGI to transition to sustainable operational model –May 2010 – April 2014 –24 mEuro EC contribution, 72 mEuro in total 7

EGI-InSPIRE RI What can EGI offer to RIs? 1. Operation tools 2. Collaboration tools 3. Application developer & user tools 4. Computing, storage & hosting capacity (existing grid and cloud sites) NGIs and EGI.eu Middleware projects: EMI, IGE, EDGI,... VRC and VO projects: wLCG, WeNMR, ScalaLife HMRC,... EGI-InSPIRE project Technology projects: SCI-BUS, SHIWA, MAPPER,... Training Marketplace, Software registry & repository, VM image marketplace,... Monitoring, accounting, service registry,.... for a distributed infrastructure Registered in EGI Applications Database (mostly) From resource centres (Centrally managed resource pool is under setup) 8

EGI-InSPIRE RI Operation Tools for RIs 1.Service registry (GOCDB) –Repository to maintain topology and other type of basic information about distributed services 2.Accounting infrastructure –To gather, process and visualise usage of distributed services 3.Service Availability Monitor (SAM) –To gather, process and visualise availability and reliability information about distributed services 4.Service discovery capability (GLUE and LDAP) –To store detailed and dynamic information about services. To discover services by features. 5.Helpdesk (GGUS, xGUS) –To troubleshoot problems or provide guidance about products and services. 9 To operate a multi-site, multi-national IT infrastructure...

EGI-InSPIRE RI Collaboration Tools for RIs 1.Applications Database –Registry of high level software environments, software components, developers, publications 2.Training Marketplace –Registry of training events, training materials, training resources, university courses 3.Client Relationship Manager –To capture leads and needs of scientific communities 10 To exhange & integrate knowledge in a multi-national community...

EGI-InSPIRE RI Applications Database Benefits: Gives recognition to reusable scientific applications, application developer tools, portals, workflow systems, etc. Gives recognition to application developers (people profiles) Access through web page AND web gadget Community features such as commenting, rating, tag-based groupings 11

EGI-InSPIRE RI Benefits: Register & share training events, expertise, services, materials, resources, online courses, university courses Browse and search items Community features such as commenting, rating Access through web page and web gadget Training Marketplace 12

EGI-InSPIRE RI Client Relationship Manager To capture leads and needs of scientific communities EGI CRM – main capabilities (based on vTiger CRM): –Register projects/communities –Register personal leads –Register details about ‘Potential for EGI use’ and ‘Interest in e-infrastructures’ EGI instance is available to NGI International Liaisons and their collaborators at 13

EGI-InSPIRE RI Web Gadgets: Embed EGI collaboration services into Websites! The simplest way to reuse EGI services: embed EGI gadgets into your website Benefits –Customisability –Reusability –Compatibility Existing gadgets: –Application Database –Training Marketplace –Requirement Tracker –Grid Relational DBs Catalogue Use and develop gadgets & share them through EGI: 14

EGI-InSPIRE RI Application Developer and user tools for RIs EGI Applications Database stores –Ready-to-use solutions on EGI –Reusable frameworks –Check them at EGI provides these in partnership with projects. Some examples: –SCI-BUS: Grid/cloud portal framework –ER-flow: Workflow platform and repository –DIRAC: Application developer tool Numbers (February 2013) : –460 software in total –387 scientific application –70 developer tool/framework/toolkit –46 science gateway & framework –26 workflow system & workflow 15 Community specific Virtual Research Environment Domain services Service access Off-the- shelf and custom VM images Data bases and repositories Scientific applications and algorithms Service integration Science gateways, Portals Workflow systems Domain specific applications Domain specific workflows Domain specific databases To develop custom environments for scientists...

EGI-InSPIRE RI Community specific Virtual Research Environment 4. Grid and Cloud capacity for RIs Grid sites at ~350 sites OS Grid middleware: gLite, ARC, Unicore, Desktop Grid Local hypervisors OS Cloud sites at ~10 sites OS Domain specific services OS Community specific Virtual Research Environment Science gateways, Portals Workflow systems Science gateways, Portals Workflow systems EGI Cloud federation (IaaS) 16 Domain specific applications Domain specific workflows Domain specific databases

EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI Possible next steps for EGI – ELI-HU collaboration 17

Primary Storage & Archive Secondary Storage File & Metadata Catalog User (x10K) Instrument (x10s) Writes Data toReplicates Data to Downloads Data From Registers Data in Downloads Data From Searches for Data in Runs Analysis on Data Analysis Facility (100ks Cores) Controls GBs -1 PB GBs -1 CRISP reference architecture Real-time Queue Data Stager Collaboration Policies Data Management Data Policy Accounting and Billing DAQ Buffer Recording Buffer 2-3 Days PBs -1 Trigger Farm T0 Processing QA Calibration This is a slide from a recent CRISP presentation. Could we refine this architecture to specfy the ELI-HU IT infrastructure?

EGI-InSPIRE RI Follow up at national level Hungarian NGI: Magyar Grid Kompetencia Központ BME ELTE KFKI-RMKI MTA SZTAKI NIIFI 3D SOFT KFT. EGI projects with Hungarian partners EDGI - desktop grid SCI-BUS – portal framework for developers and users ER-flow – workflow platform & repository CRISP – common needs of physics ESFRIs

EGI-InSPIRE RI Follow-up at European level: EGI Virtual Team projects Virtual Team projects: –Short (up to 6 month) project with multiple NGIs involved –Setup through NGI International Liaisons and EGI.eu – Example VT projects that support RIs: –NGI - ELIXIR ESFRI collaboration (Started in October) –Technology study for CTA ESFRI (Started in January) –Towards a Chemistry, Molecular & Materials Science and Technology Virtual Research Community (Started in February) –Science Gateway Primer (Amost over) Setup a Virtual Team project for ELI? –With partners common to ELI – CRISP - EGI-InSPIRE? 20

EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI Extra slides 21

EGI-InSPIRE RI EGI Infrastructure: The platform model Community Platform EGI resources (cores & storage) VM EGI Core Infrastructure Platform EGI Cloud Infrastructure Platform Virtual Research Communities EGI Collaboration platform Community Platform (gLite Grid) Grid services 22

EGI-InSPIRE RI The current community platform: Unified Middleware Distribution (UMD) External technology providers: EMI (ARC, gLite, UNICORE, dCACHE), IGE (Globus), EDGI (desktop grids), SAGA, … UMD capabilities captured in UMD roadmap (last version 2012 May: Operations Provisioning Infrastructure Staged Rollout Staged Rollout Criteria Verification Criteria Verification Production Criteria Definition Criteria Definition External Technology Providers Deployed Software Helpdesk unit Deployed Software Helpdesk unit RequirementsSoftware Core infrastructure platform By EGI user communities and other stakeholders 23

EGI-InSPIRE RI EGI Grid use case 1: workload management with UMD Computing service Storage Service Site X of YOUR VO Information System Submit job query Retrieve output Create job definition Submit job (batch executable + small inputs) Broker service User environment publish state VO Management Service (VOMS) create Proxy (~login) process Retrieve status & (small) output files Logging and bookkeeping service Job status Logging Read/write files 24 Core infrastructure platform

EGI-InSPIRE RI Core infrastructure platform EGI grid use case 2: data & meta-data management with UMD Computing service Storage service Site X of YOUR VO Information System Query Command line or Portal publish state VO Management Service (VOMS) create proxy Upload file Download file File Catalog and/or Metadata catalog Register file Lookup file File content File references 25

EGI-InSPIRE RI Sept 2011 – March 2013: Federated Cloud Task Force –Identify open source technologies (OpenStack, OpenNebula,...) and open standard (OCCI, CDMI, GLUE2,...) –Deploy a testbed –Write blueprint for resource centres –Identify issues from non-technical areas (policy, dissemination) August 2012 – March 2013: Pilot use cases –Support early adopters –Collect and feedback requirements –Establish user-facing services April 2013 (EGI Community Forum): : Open the platform –Routine usage by users and providers EGI cloud roadmap 26

EGI-InSPIRE RI Structural biology – We-NMR project: Gromacs training environments Ecology – BioVel project: Remote hosting of OpenModeller service Linguistics – CLARIN project: Scalable ‘British National Corpus’ service (BNCWeb) Software development – SCI-BUS project: Simulated environments for portal testing Space science – ASTRA-GAIA project: Data integration with scalable workflows Some of the cloud infrastructure pilot use cases Join as pilot use case, or as provider of high level service on top of the EGI Cloud!\ 27

EGI-InSPIRE RI Services of the Collaborative platform 4: Requirements tracker Technology Coordination Board EGI Requirements Tracker NGIs VRCs VOs User Community Support Team of EGI.eu Technology providers projects events Input channels for community requirements EGI Helpdesk User Community Board Operations Management Board Resource centres Structured scientific communities NGIs & projects 28

EGI-InSPIRE RI EGI guiding Principles Be an open provider –Resource providers can federate their sites into the Core Platform –Software providers can join into the Core, Cloud and Collaboration Platforms Be a neutral provider –Collaboration Platform for the fair representation of various stakeholders and their achivements Be a flexible provider –Cloud and Core platform allows communities to deploy their own services  Community Platforms –Give communities the power to meet their own needs 29