1. EMI INFSO-RI-261611 European Middleware Initiative (EMI) HSIN-YEN CHEN ASGC gWRF Workshop @ Thailand

2. EMI INFSO-RI-261611 Outline Grid Infrastructure and EMI Middleware EMI Vision and Objectives How does it work? Conclusions 06/10/2010 EMI Overview v. 3.0 2

3. EMI INFSO-RI-261611 History European DataGrid (EDG): April 2001 to March 2004 LHC Computing Grid (LCG): January 2002 onwards – First Grid deployment in March 2003, based on the EDG testbed – First LCG workshop in November 2004 Enabling Grids in E-sciencE (EGEE): April 2004 to April 2010 – Three separate two-year projects – Common infrastructure with LCG European Grid Infrastructure (EGI): May 2010 onwards – European Middleware Initiative (EMI) provides the middleware

4. EMI INFSO-RI-261611 EGEE→EGI transition planning 06/10/2010 EMI Overview v. 3.0 4

5. 5 www.egi.eu EGI-InSPIRE RI-261323 EGI Update - Prague 2012 Resource Infrastructure Providers 5 Resource Centres EGI-InSPIRE & EGI Council members319 Including integrated RPs351 Countries EGI-InSPIRE & EGI Council members42 Including integrated RPs54 Integrated EGI-InSPIRE Partners and EGI Council Members Internal/External Resource Providers (being integrated) External Resource Providers (integrated) Peer Resource Providers Registered Users: 21714 VOs: 233 LCPUs: 470,000 Disk: 143PB Tape: 138PB Jobs: 1.62 million/day

6. EMI INFSO-RI-261611 EMI Mission Statement 06/10/2010 EMI Overview v. 3.0 6 The European Middleware Initiative (EMI) project represents a close collaboration of the major European middleware providers - ARC, gLite, UNICORE and dCache - to establish a sustainable model to support, harmonise and evolve the grid middleware for deployment in EGI, PRACE and other distributed e-Infrastructures

7. EMI INFSO-RI-261611 A European Vision 06/10/2010 EMI Overview v. 3.0 7 Today Tomorrow Sustainability Persistence Interoperability Easier Access Sustainability Persistence Interoperability Easier Access

8. EMI INFSO-RI-261611 Partners (26) 06/10/2010 EMI Overview v. 3.0 8

9. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 9 Grid Concepts User Interface Grid services

10. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 Introductive concepts Computational Resources: machines where an user can execute a program (“job”), store or retrieve files Job: is a computational task (an executable, a script) that the user wants to be executed in the Grid, having finally the results back on it's machine VO: people owning at different institutions but working at similar problems sharing resources, applications, data, … 10

11. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 11 Gate Keeping: Identify users and give permissions Grid Security Users in many locations and organisations Data Storage Grid Services Access services (User Interface) : logon, upload credentials, run m/w commands Built on Grid Security Infrastructure (GSI) Encryption and Data Integrity, Authentication and Authorization Computation High Level Service Software Resources in many locations and organisations

12. 12 www.egi.eu EGI-InSPIRE RI-261323 12 Workload Management Data Management Security Information & Monitoring Access gLite Grid Middleware Services Metadata Catalog Storage Element Data Movement File & Replica Catalog Information & Monitoring Application Monitoring Authorization Authentication Auditing Computing Element Workload Management Job Provenance Package Manager APICLI Accounting Site Proxy Overview paper http://doc.cern.ch//archive/electronic/egee/tr/egee-tr-2006-001.pdfhttp://doc.cern.ch//archive/electronic/egee/tr/egee-tr-2006-001.pdf

13. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 13 Basic gLite use case: Job submission Computing Element Storage Element Site X Information System Submit job (executable + small inputs) Submit job query Retrieve output Resource Broker User Interface publish state File and Replica Catalog VO Management Service (DB of VO users) query create proxy process Retrieve status & (small) output files Logging and bookkeeping Job status Logging Input file(s) Output file(s) Register file

14. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 14 User Interface (UI) User Interface (UI): The place where users logon to the Grid Computing Element (CE) Computing Element (CE): A batch queue on a site’s computers where the user’s job is executed Storage Element (SE) Storage Element (SE): provides (large-scale) storage for files Resource Broker (RB) (Workload Management System (WMS) Resource Broker (RB) (Workload Management System (WMS): Matches the user requirements with the available resources on the Grid Main components Information System Information System: Characteristics and status of CE and SE File and replica catalog File and replica catalog: Location of grid files and grid file replicas Logging and Bookkeeping (LB) Logging and Bookkeeping (LB): Log information of jobs

15. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 15 User Interface (UI) User Interface (UI): The place where users logon to the Grid Computing Element (CE) Computing Element (CE): A batch queue on a site’s computers where the user’s job is executed Storage Element (SE) Storage Element (SE): provides (large-scale) storage for files Resource Broker (RB) (Workload Management System (WMS) Resource Broker (RB) (Workload Management System (WMS): Matches the user requirements with the available resources on the Grid Main components Information System Information System: Characteristics and status of CE and SE File and replica catalog File and replica catalog: Location of grid files and grid file replicas Logging and Bookkeeping (LB) Logging and Bookkeeping (LB): Log information of jobs All built upon authorisation, authentication, security

16. EMI INFSO-RI-261611 Outline Grid Infrastructure and EMI Middleware EMI Vision and Objectives How does it work? Conclusions 06/10/2010 EMI Overview v. 3.0 16

17. EMI INFSO-RI-261611 Primary Objectives Consolidate the existing middleware distribution simplifying services and components to make them more sustainable (including use of off-the-shelf and commercial components whenever possible) Evolve the middleware services/functionality following the requirement of infrastructure and communities, mainly focusing on operational, standardization and interoperability aspects Reactively and proactively maintain the middleware distribution to keep it in line with the growing infrastructure usage 06/10/2010 EMI Overview v. 3.0 17 Consolidate Evolve Support

18. EMI INFSO-RI-261611 Improved Usability One of the major complaints of research users about the middleware is about its limited “userfriendliness” – Deployment, configuration, service management, interoperability, security mechanisms, flexibility, etc. Unnecessary duplication of services and libraries must be avoided User requirements (ESFRI, VRCs) – EMI is requirement driven and will actively participate to the definition of user requirements with the major user communities and infrastructures Support the development of portals and domain- specific applications via clear APIs 06/10/2010 EMI Overview v. 3.0 18

19. EMI INFSO-RI-261611 Improved Security One of the most important and most difficult aspects of the middleware – Usability: existing certificate-based technologies are needed, but too complex to manage or use for the typical user or not easy to integrate in existing security contexts – Reliability: the increasing use of distributed computing and the handling of sensitive data require reliable and auditable security methods – Interoperability: the chosen methods must be common across all services and implementations 06/10/2010 EMI Overview v. 3.0 19

20. EMI INFSO-RI-261611 Standardization Very important to address a number of existing limitations – Interoperability, integration, extensibility and evolution, commercial usage All EMI services must: – Implement the relevant useful standards – Implement them in the same way – But, if no usable standard exists, EMI can propose solutions based on actual usage (de facto standards) EMI intends to be an active player in the standardization roadmap in collaboration with SIENA, the other DCI projects and the SDOs 06/10/2010 EMI Overview v. 3.0 20

21. EMI INFSO-RI-261611 Interoperability One of the major requirements of most user communities – Interoperability between different implementations of the same services or functionality (CE, Data, Info Systems) – Interoperability among HTC and HPC (MPI, security) – Interoperability between different infrastructures (EGI, WLCG, OSG, others) Also in this case, the widespread and formally correct adoption of standards is of primary importance 06/10/2010 EMI Overview v. 3.0 21

22. EMI INFSO-RI-261611 Integration with New Technologies Technology evolves continually Distributed computing middleware must capitalize on past achievements and learn from past lessons Using labels like Grids or Clouds can be misleading How can existing stable, reliable and secure services be made more dynamic and efficient? And again, standards can enable a smooth evolution to new technology 06/10/2010 EMI Overview v. 3.0 22

23. EMI INFSO-RI-261611 Integration with New Technologies Two of the main topics of interests are: – Messaging: there are a number of very practical use cases where Messaging can bring clear advantages (monitoring, accounting, distributed logging, service management, information systems, etc) – Virtualization and clouds: use of virtual machines can greatly simplify the issues of resource management, opening to more types of resource providers (including commercial), provide better independence of MW from OS management, make the whole distributed infrastructure more dynamic. But security, accounting, workflows are an issue. How to get the best of both worlds? 06/10/2010 EMI Overview v. 3.0 23

24. EMI INFSO-RI-261611 Outline Grid Infrastructure and EMI Middleware EMI Vision and Objectives How does EMI work? Conclusions 06/10/2010 EMI Overview v. 3.0 24

25. EMI INFSO-RI-261611 Applications Integrators, System Administrators EMI Middleware Evolution 05/10/2010 EMI Status Report for ECB 25 Standards, New technologies (clouds) Users and Infrastructure Requirements EMI Reference Services 3 yearsBefore EMIAfter EMI Specialized services, professional support and customization Standard interfaces

26. EMI INFSO-RI-261611 DCI collaborations Collaborations 05/10/2010 EMI Status Report for ECB 26 EMI EGI, PRACE, WLCG,OSG, etc ESFRI, VRCs ESFRI, VRCs StratusLab VENUS-C EDGI Requirements Releases Collaborations IGE SLAs & Support SIENA Standards Industry Standards Industry Collaborations

27. EMI INFSO-RI-261611 Release Plan 05/10/2010 EMI Status Report for ECB 27 01/05/2010 Start EMI 0 EMI 1 EMI 2 EMI 3 Support & Maintenance Supp. & Maint. 31/10/201030/04/201130/04/201228/02/2013 Major releases EMI Reference Services

28. EMI INFSO-RI-261611 Release Cycle 05/10/2010 EMI Status Report for ECB 28 Requirements Technical Plans Development and Test Plans Development Testing Certification Release Maintenance Support PTB JRA1 SA1 SA2 30/09/2010 31/10/2010 28/02/2011 30/04/2011 As needed EMI 1 Roll-out/DMSU TMB

29. EMI INFSO-RI-261611 Technical Areas 23/09/2010 EMI Overview - DECIDE Launch Event, Rome 29 Compute Services Data Services Security Services Infrastructure Services A-REX, UAS-Compute, WMS, CREAM, MPI, etc dCache, StoRM, UAS-Data, DPM, LFC, FTS, Hydra, AMGA, etc UNICORE Gateway, UVOS/VOMS/VOMS- Admin, ARGUS, SLCS, glExec, Gridsite, Proxyrenewal, etc Logging and Bookkeeping, Messaging, accounting, monitoring, virtualization/clouds support, information systems and providers Product Teams Dedicate teams of experts Fully responsible for development, maintenance and unit/system testing Product Teams Dedicate teams of experts Fully responsible for development, maintenance and unit/system testing 3 rd -level Support via the GGUS application

30. EMI INFSO-RI-261611 10/05/2010 EMI Hot Topic, JSC, FZJ 30 Auth X.509 Certificates Issued by Trusted Certification Authority (CA)‏ Proxy certificates for jobs Virtual Organisation Management Service (VOMS)‏ User permissions managed by the community to which they belong MyProxy Server Longer term certs and cert storage

31. EMI INFSO-RI-261611 10/05/2010 EMI Hot Topic, JSC, FZJ 31 User Interface (UI)‏ gLite command line tools glite-wms-job-* APIs For application development GUI Interfaces Sites typically provide a UI machine May also be installed locally

32. EMI INFSO-RI-261611 10/05/2010 EMI Hot Topic, JSC, FZJ 32 Compute Element (CE)‏ GridGate (headnode)‏ CREAM or LCG-CE Batch Local Ascii Helper (BLAH)‏ Common interface to multiple LMRS Local Resource Management System (LMRS)‏ PBS, LSF, Maui/Torque, Condor... Worker Nodes Cluster

33. EMI INFSO-RI-261611 10/05/2010 EMI Hot Topic, JSC, FZJ 33 Storage Element (SE)‏ Storage Resource Managers (SRM)‏ DPM (disks)‏ Castor (tape)‏ dCache etc.

34. EMI INFSO-RI-261611 10/05/2010 EMI Hot Topic, JSC, FZJ 34 File & Data management LHC File Catalog (LFC) stores information on Grid data files Each file can have many names... Grid Unique Identifier (GUID)‏ Logical File Name (LFN)‏ Storage URL (SURL)‏ Transport URL (TURL)‏

35. EMI INFSO-RI-261611 10/05/2010 EMI Hot Topic, JSC, FZJ 35 File & Data management 2 Unix-style lfc-* commands query the catalog lfc-ls, lfc-mkdir, lfc-rm, etc. lcg_util commands allow file and replica management lcg-cr, lcg-cp, lcg-rep, etc. File Transfer Service allows batch file transfers

36. EMI INFSO-RI-261611 10/05/2010 EMI Hot Topic, JSC, FZJ 36 Information Service (IS)‏ Berkeley Database Information Index (BDII)‏ LDAP-based information service GLUE Schema Resource discovery, Monitoring & Accounting Query via lcg-info lcg-infosites ldapsearch

37. EMI INFSO-RI-261611 10/05/2010 EMI Hot Topic, JSC, FZJ 37 Information Service (IS) 2 Resource level Grid Resource Information Server (GRIS)‏ One GRIS on top of each CE, SE, RB, MyProxy (no WNs). Site Level site BDII Top level Top BDII

38. EMI INFSO-RI-261611 Security Services Common Authentication Libraries for all EMI components, uniform authentication response from all components and removal of redundant authentication code/components Removal of old GSI from EMI components and installation to other O/S easier through use of standards-based code Common SAML profile and SAML assertions exchange throughout MW stack Adoption of SAML-enabled VOMS throughout Adoption of the Argus authorization system throughout Transparent AAI for users, usage of locally-based AAI systems, users can use more familiar credential interface, reduce/eliminate need for users to manage credentials. 06/10/2010 EMI Overview v. 3.0 38

39. EMI INFSO-RI-261611 Compute Services Define a unified specification for a generic Execution Service (EMI-ES) (existing standards are not usable and there is little possibility of extending and implementing them within the EMI lifetime) Provide a common framework for supporting MPI jobs – The many-core revolution (multi-core, many-core, GPUs etc.) deeply affects both HPC and HTC, effort activity to bridge HPC and HTC applications, which requires definition of fine-grained parameters Improve usability, maintainability and portability through a common set of APIs and user interfaces Comply to the unified EMI security model and use the EMI authorization service (Argus) throughout – This will allow central blacklisting, avoid inconsistent authorization decisions, ease deployment and maintenance Work on the integration of CEs with Virtualization Manager (OpenNebula, WNOD) using as much as possible some technology- independent abstraction layer Strategy for pilot jobs not clear, needs more discussion 06/10/2010 EMI Overview v. 3.0 39

40. EMI INFSO-RI-261611 Data Services Catalogue synchronization with direct interaction between FC and SE (possibly based on messaging) Extend FC to UNICORE Consolidation of SRM and move towards simplified SRM 2.2 specifications Replace Globus httpg with standard https (allows data access using https and WebDav) Standard data access through POSIX-compliant mountable file systems (NFS4.1, FUSE) Adoption of GLUE 2.0 Integration with ARGUS Consolidation of clients and APIs Improved monitoring and accounting support (Nagios, messaging-based solutions) 06/10/2010 EMI Overview v. 3.0 40

41. EMI INFSO-RI-261611 Infrastructure Services Common GLUE 2.0-based Information System for all MW Common Registry Service for end-point location Generalization of messaging framework to cover more use cases (monitoring, accounting, service management, info systems, File Catalogue synchronization) Investigation of virtualization and cloud technology and integration in the standard grid stack – Worker nodes (OpenNebula, WNOD, others) – Dynamic service provision (ad-hoc instanciation) (StratusLab/OpenNebula, VENUS-C, Google, others) 06/10/2010 EMI Overview v. 3.0 41

42. EMI INFSO-RI-261611 Outline Grid Infrastructure and EMI Middleware EMI Vision and Objectives How does it work? Conclusions 06/10/2010 EMI Overview v. 3.0 42

43. EMI INFSO-RI-261611 Conclusions EMI is a new reference in distributed computing middleware It brings together for the first time the expertise of the major European middleware providers EMI is very committed to provide a streamlined, standard set of services well integrated with the major OS and exploiting emerging technology as necessary 06/10/2010 EMI Overview v. 3.0 43

44. EMI INFSO-RI-261611 Thank you 06/10/201044 EMI Overview v. 3.0 EMI is partially funded by the European Commission under Grant Agreement INFSO-RI-261611