1. www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE www.egi.eu EGI-InSPIRE RI-261323 The European Grid Infrastructure collaboration Gergely Sipos EGI.eu Technical Outreach Manager User Community Support Team gergely.sipos@egi.eu European Grid Infrastructure EGI Champions briefing webinar 20/Mar/2013

2. www.egi.eu EGI-InSPIRE RI-261323 Outline EGI overview –Infrastructure, communities, projects,... EGI services for research communities –Individual researchers  Research infrastructures Break EGI services for research communities –Details Homework –Next steps for Our Champions 2

3. www.egi.eu EGI-InSPIRE RI-261323 European Grid Infrastructure European –Over 35 countries Grid –Secure sharing of IT resources Infrastructure –Computers (clusters) –Data –Applications –…. and beyond!! 3

4. www.egi.eu EGI-InSPIRE RI-261323 Key resource providers in EGI: National Grid Infrastructures 4 Metric Value (March 2013) Sites315 Nb. of CPU cores372,612 Disk (PB)180 PB Tape (PB)167 PB

5. www.egi.eu EGI-InSPIRE RI-261323 5 A multi-disciplinary e-infrastructure http://www.egi.eu/community/ User VO Virtual Research Community Members Virtual Organisations Research communities Most active VO groups: High Energy Physics Astronomy & Astrophysics Life Sciences Earth Sciences Computational Chemistry Fusion … ~22.000 users in ~230 Registered VOs http://operations-portal.egi.eu/vo Sites Current VRCs: wLCG - High Energy Physics WeNMR – Structural biology LSGC – Life sciences (HMRC – Hydro-Meteorology) (CLARIN – Humanities) Sites

6. www.egi.eu EGI-InSPIRE RI-261323 EGI, EGI.eu, EGI-InSPIRE EGI – A collaboration. An ecosystem. –Operates the infrastructure – and does other things too! (see later) –Aim: Support the digital European Research Area through a pan-European research infrastructure based on services federed from the NGIs EGI.eu – A Dutch fundation owned by the NGIs –To coordinate the work of EGI (operations, technology, user support, policy, community & communications, administration) –Sustainable, small organisation EGI-InSPIRE – An FP7 project –Supports EGI.eu and EGI to transition to sustainable operational model –May 2010 – April 2014 6

7. www.egi.eu EGI-InSPIRE RI-261323 What is an NGI? http://www.egi.eu/about/ngis/ The country's single point of contact for government, research communities and resource centres as regards ICT services for e- science. Manage the computing resources and support services they provide to the European Grid Infrastructure. Has named contact for: –International liaison (NILs) –User support –Infrastructure Operation 7

8. www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE Project Integrated Sustainable Pan-European Infrastructure for Researchers in Europe A 4 year project with €25M EC contribution –Project cost €72M –Total Effort ~€330M –Effort: 9261PMs (192 FTE) Project Partners (50) EGI.eu (coordinator) 38 NGIs, 2 EIROs Asia Pacific (9 partners) 8

9. www.egi.eu EGI-InSPIRE RI-261323 EGI contributors National Grid Infrastructures (NGIs) Technologies: EMI, IGE, EDGI,... VRCs and VOs: wLCG, WeNMR, ScalaLife HMRC,... EGI-InSPIRE Specialised services: SCI-BUS, SHIWA, MAPPER,... EGI.eu Legal entities Projects Software + services MoUs, SLAs European Intergovernmental Research Organisations (EIROs) Infrastructures: CHAIN EUDAT... 9

10. www.egi.eu EGI-InSPIRE RI-261323 EGI’s strategic focus areas Operational Infrastructure Virtual Research Environments Community & Coordination Operate a European wide grid infrastructure Build a federated cloud infrastructure Integration & operation of scientific environments Governance Communication (inreach, outreach) 10

11. www.egi.eu EGI-InSPIRE RI-261323 EGI’s strategic focus areas Operational Infrastructure Virtual Research Environments Community & Coordination Operate a European wide grid infrastructure Build a federated cloud infrastructure Integration & operation of scientific environments Governance Communication (inreach, outreach) UCB TCB Press Policy Outreach AppDB CRM Training NILs VTs Design Gadgets Webinars OMB Consultance Newsletter Monitoring Accounting Registry Capacity UMD Case studies Requirements Core Cloud Platforms MoU Pilots Task forces middleware Marketplace 11

12. www.egi.eu EGI-InSPIRE RI-261323 EGI service offerings 5. Operation tools 4. Collab. & communic. services 1. Ready-to-use scientific applications 12 Individual researchers Research infrastructures 3. Application porting / develeloper tools 2. Grid & cloud sites Research groups.......

13. www.egi.eu EGI-InSPIRE RI-261323 1. Ready-to-use scientific applications Registered in EGI AppDB: http://appdb.egi.eu http://appdb.egi.eu –Browse by scientific discipline –Browse by Virtual Organisation Application access varies: –Public access –Via portal (username/pwd) –Register in VO (X.509 cert) Interact! –Follow –Rate –Comment –Contact Applications 13

14. www.egi.eu EGI-InSPIRE RI-261323 2/1. Grid sites Obtain an X.509 certificate: –http://www.igtf.net/http://www.igtf.net/ Join one of the existing VOs: –http://operations-portal.egi.eu/vohttp://operations-portal.egi.eu/vo –200-220 Virtual Organisations 8 science discipline specific VO groups 1 VO group for other disciplines 1 group for Multi-disciplinary VOs OR Setup a new VO: –http://operations- portal.egi.eu/vo/registrationWelcomehttp://operations- portal.egi.eu/vo/registrationWelcome –Request to support the VO is sent to site managers Get help from NGI User Support Teams 14 Grid sites at ~350 sites Linux OS Grid middleware: UMD Unified Middleware Distribution (gLite, ARC, Unicore, dCache, SAGA, Desktop Grid) Linux OS Domain specific applications Domain specific workflows Domain specific databases You may not need site-level access to use scientific applications on EGI. But if you do need then...

15. www.egi.eu EGI-InSPIRE RI-261323 EGI Grid use case example 1: workload management with UMD Computing service Storage Service Site X of YOUR VO Information System Submit job query Retrieve output Create job definition Submit job (batch executable + <20 MB inputs) Broker service User environment publish state VO Management Service (VOMS) Login to your VO (With X509 cert) job Retrieve status & (small) output files Logging and bookkeeping service Job status Logging Read/write files 15 EGI Virtual Organisation job Your files

16. www.egi.eu EGI-InSPIRE RI-261323 Implementation: Auger VO Serving the Pierre Auger Collaboration Computing service Storage Service Site of AugerVO Information System Submit job query Retrieve output Create job definition Submit job (batch executable + <20 MB inputs) Broker service User environment publish state VO Management Service (VOMS) Login to your VO (With X509 cert) job Retrieve status & (small) output files Logging and bookkeeping service Job status Logging Read/write files 16 Auger Virtual Organisation on EGI job Cosmic ray files Copied from Colorado (USA), Malargüe (Arg) to grid storage Lyon (Fr) (CCIN2P3) Processing software is distributed to consortium members CPU intensive simulations done at various sites and results copied back to Lyon Central services for the VO are in Czech Republic (by CESNET)

17. www.egi.eu EGI-InSPIRE RI-261323 EGI Virtual Organisation EGI grid use case example 2: data & meta-data management with UMD Computing service Storage service Site X of YOUR VO Information System Query User environment publish state VO Management Service (VOMS) Upload file Download file File Catalog and/or Metadata catalog Register file Lookup file File content Metadata 17 Login to your VO (With X509 cert) Your files

18. www.egi.eu EGI-InSPIRE RI-261323 Implementation: gLibrary http://glibrary.ct.infn.it 3TByte in total TIFF (100MB/A3 page) PDF (40-400MB/work) (1) Digitalise (2) Store (3) Catalogue + Meta-data (4) Browse De Roberto, an Italian writer of the XIX/XX century, born in Naples, but spending his life in Catania 18

19. www.egi.eu EGI-InSPIRE RI-261323 2/2 Cloud sites: EGI Federated Cloud Local hypervisors Cloud resources at ~10 EGI sites OS Domain specific services hosted on EGI as VM Images OS EGI FedCloud interfaces 19 OCCI: VM Image management BDII: Information system X509: Authentication VOMS: Authorization APEL: Accounting OpenStack OpenNebula EmotiveCloud (Spain) Okeanos (OpenStack impl. in GR) WNoDeS (Italy) Any service, not only “batch processing” and “file based data management”

20. www.egi.eu EGI-InSPIRE RI-261323 Sept 2011 – March 2013: Federated Cloud Task Force –Identify open source technologies (OpenStack, OpenNebula,...) and open standard (OCCI, CDMI, GLUE2,...) –Deploy a testbed –Write blueprint for resource centres –Identify issues from non-technical areas (policy, dissemination) August 2012 – March 2013: Pilot use cases –Support early adopters –Collect and feedback requirements –Establish user-facing services April 2013 (EGI Community Forum): : Open the platform –Routine usage by users and providers –Demo and discussions at EGI Community Forum 2013 Including early adopter use cases EGI Federated Cloud roadmap http://go.egi.eu/cloud 20

21. www.egi.eu EGI-InSPIRE RI-261323 3. Application porting / developer tools EGI Applications Database stores –Ready-to-use solutions on EGI –Reusable frameworks –Grouped in various categories (see below) Categories (numbers in February 2013) : –70 developer tool/framework/toolkit –46 science gateway & gateway framework –26 workflow system & workflow http://appdb.egi.eu 21 Domain specific services hosted on EGI Science domain specific services Service access Off-the- shelf and custom VM images Data bases and repositories Scientific applications and algorithms Service integration Science gateways (Portals) Workflow systems Scientific algorithms, applications Scientific workflows Scientific databases To create custom environments for scientists... EGI GridEGI FedCloud Very often only this part is unique Domain specific services hosted on EGI

22. www.egi.eu EGI-InSPIRE RI-261323 4/1. Communication services What we provide: Creation of content/materials –Case studies, news stories, newsletters, articles, prints, designs, blog posts, videos, leaflets, social network content, etc. Events –EGI annual meetings –F2f trainings and Webinars –Training Marketplace: http://training.egi.euhttp://training.egi.eu Outreach –International Science Grid This Week, HPCwire, HPCinthecloud and Datanami, etc. –Coordinated outreach (to ESFRIs) via the network of ‘NGI International Liaisons’ 22 To facilitate information exchange & knowledge integration in the community... In the pipeline: EGI Community Forum –Machester, 8-12 April: http://cf2013.egi.eu http://cf2013.egi.eu –Including several training sessions EGI Webinar Programme –DIRAC services, 22 March –https://wiki.egi.eu/wiki/EGI_Webinar_Programmehttps://wiki.egi.eu/wiki/EGI_Webinar_Programme European Geosciences Union –Vienna, 8-12 April: http://www.egu2013.euhttp://www.egu2013.eu –Exhibition booth –Presentation and PICO in ESSI2.8 session about EGI Federated Cloud Leaflets –EGI Service offerings

23. www.egi.eu EGI-InSPIRE RI-261323 4/2. Collaboration services Virtual Team projects –Short (up to 6 month) project with multiple NGIs involved –Setup through NGI International Liaisons and EGI.eu –https://wiki.egi.eu/wiki/Virtual_Team_Projectshttps://wiki.egi.eu/wiki/Virtual_Team_Projects Example VT projects that support Research Communities: –NGI - ELIXIR ESFRI collaboration (Started in October) –Technology study for CTA ESFRI (Started in January) –Towards a Chemistry, Molecular & Materials Science and Technology Virtual Research Community (Started in February) –Science Gateway Primer (Primer document in final editing) Setup a Virtual Team project for your community? –What would be the scope? Which countries should be involved? 23

24. www.egi.eu EGI-InSPIRE RI-261323 5. Operation Tools EGI’s services: 1.Service registry (GOCDB) –Repository to maintain topology and other type of basic information about distributed services 2.Accounting infrastructure –To gather, process and visualise usage of distributed services 3.Service Availability Monitor (SAM) –To gather, process and visualise availability and reliability information about distributed services 4.Service discovery capability (GLUE and LDAP) –To store detailed and dynamic information about services. To discover services by features. 5.Helpdesk (GGUS, xGUS) –To troubleshoot problems or provide guidance about products and services. 24 Services to operate a multi-site, multi-national Research Infrastructure... Local NGIs Service Registry Accounting Monitoring Heldesk Research infrastructure Scientific services Infrastructure services Community portals

25. www.egi.eu EGI-InSPIRE RI-261323 Take home messages EGI is based on a federated model –NGIs - national resource and service providers Established grid, emerging cloud infrastructure EGI is not just computing and storage –Scientific applications –Application developer tools & frameworks –Infrastructure operation tools –Collaboration and communication services EGI enables customised Virtual Research Environments 25

26. www.egi.eu EGI-InSPIRE RI-261323 Homework: What can our Champions do? 1.Ready-to-use applications: –Check the applications of your domain. Is there any key application missing? 2.Virtual Organisations: –Check the VOs that exist in EGI in your scientific domain. What services and processes do they provide to new users? Are these sufficient for a newbie to start? 3.Web gadgets: –EGI AppDB, Training Marketplace, etc. can be easily embedded into any website as ‘web gadgets’. Customise EGI gadgets for your community at http://go.egi.eu/gadgetshttp://go.egi.eu/gadgets 4.Events: –Is the local NGI aware of your community’s upcoming event? Will they send a representative? Get in touch with the NGI International Liaison: http://www.egi.eu/about/ngis/NILs.html http://www.egi.eu/about/ngis/NILs.html 5.Research Infrastructures: –Which research infrastructures (e.g. ESFRI) exist in your scientific discipline? What is their status? Which EGI services would these benefit from at this stage? Promote EGI within your scientific domain 26

27. www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE www.egi.eu EGI-InSPIRE RI-261323 Questions 27

28. www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE www.egi.eu EGI-InSPIRE RI-261323 Backup slides 28

29. www.egi.eu EGI-InSPIRE RI-261323 VO/community, user statistics (March 2012 – March 2013) 29 We need you to increase this!

30. www.egi.eu EGI-InSPIRE RI-261323 Web Gadgets: Embed EGI collaboration services into Websites! The simplest way to reuse EGI services: embed EGI gadgets into your website Benefits –Customisability –Reusability –Compatibility Existing gadgets: –Application Database –Training Marketplace –Requirement Tracker –Grid Relational DBs Catalogue Use and develop gadgets & share them through EGI: www.egi.eu/user-support/gadgets 30

31. www.egi.eu EGI-InSPIRE RI-261323 EGI Infrastructure: The platform model Community Platform EGI resources (cores & storage) VM EGI Core Infrastructure Platform (Operation tools) EGI Cloud Infrastructure Platform (FedCloud) Virtual Research Communities EGI Collaboration platform wLCG Community Platform 31 Community Platform (UMD) Collaboration Platform (AppDB, Training Marketplace, DocDB,...)

32. www.egi.eu EGI-InSPIRE RI-261323 The current community platform: Unified Middleware Distribution (UMD) External technology providers: EMI (ARC, gLite, UNICORE, dCACHE), IGE (Globus), EDGI (desktop grids), SAGA, … UMD capabilities captured in UMD roadmap (last version 2012 May: https://documents.egi.eu/document/612)https://documents.egi.eu/document/612 Operations Provisioning Infrastructure Staged Rollout Staged Rollout Criteria Verification Criteria Verification Production Criteria Definition Criteria Definition External Technology Providers Deployed Software Helpdesk unit Deployed Software Helpdesk unit RequirementsSoftware Core infrastructure platform By EGI user communities and other stakeholders 32

33. www.egi.eu EGI-InSPIRE RI-261323 Structural biology – We-NMR project: Gromacs training environments Ecology – BioVel project: Remote hosting of OpenModeller service Linguistics – CLARIN project: Scalable ‘British National Corpus’ service (BNCWeb) Software development – SCI-BUS project: Simulated environments for portal testing Space science – ASTRA-GAIA project: Data integration with scalable workflows Some of the EGI FedCloud pilot use cases Join as pilot use case, or as provider of high level service on top of the EGI Cloud! http://go.egi.eu/cloud 33