1. EMI INFSO-RI-261611 ARC on the European Distributed Computing Infrastructure (DCI) landscape Balázs Kónya (Lund University) NorduGrid Technical Workshop Kiev, 5-8 October 2011

2. EMI INFSO-RI-261611 outline Projects come and go, the Advanced Resource Connector (ARC)* stays * brought to you by the NorduGrid Collaboration NorduGrid and ARC in a nutshell A snapshot of the DCI landscape EGI EMI ARC development directions as part of broader EMI plans 5/10/2011 NorduGrid Workshop, Kiev 2

3. EMI INFSO-RI-261611 A slide from Sundvolden 5/10/2011 NorduGrid Workshop, Kiev 3 Since then: The NorduGrid Collaboration has been extended

4. EMI INFSO-RI-261611 ARC middleware A flexible, light-weight, reliable solution for distributed computing infrastructures 9/5/2011 www.nordugrid.org 4

5. EMI INFSO-RI-261611 The Distributed Computing Infrastructure Projects ProjectEDGIEGI-InSPIREEMIIGEStratusLabVENUS-C Websiteedgi- project.eu www.egi.euwww.eu- emi.eu www.ige- project.eu www.stratusla b.eu www.venus- c.eu Start Date01/06/201001/05/2010 01/10/201001/06/2010 Duration (months) 2448363024 Total Budget (€) 2,436,00072,000,00024,000,0003,693,0003,137,2218,803,046 Funding from the EC (€) 2,150,00025,000,00012,000,0002,350,0002,300,0004,500,000 Total effort (person months) 28192412319277340639 5/10/2011 NorduGrid Workshop, Kiev 5

6. EMI INFSO-RI-261611 The DCI landscape 5/10/2011 NorduGrid Workshop, Kiev 6 ARC DNA1.4 - EMI Roadmap and DCI Collaborations

7. EMI INFSO-RI-261611 EGI-InSPIRE Overview A 4-year €72M project ~190FTE with €25M EC contribution with ~€330M additional activities May 2010  April 2014 Project Partners (50) EGI.eu, 38 NGIs, 2 EIROs Asia Pacific (9 unfunded) EGI-InSPIRE Update, Lyon 2011 7

8. www.egi.eu EGI-InSPIRE RI-261323 European Grid Infrastructure (July 2011 and increase from Apr 2010) EGI-InSPIRE Update, Lyon 20118 Logical CPUs (cores ) 248,424 EGI (+29.3%) 337,608 All 106.7 PB disk and 112.8 PB tape Resource Centres 329 EGI 346 All (+6.8%) 93 supporting MPI (+6.8%) Countries (+11.5%) 50 EGI 57 All 38 NGIs providing resources 26 National Operations Centres 12 NGIs in 4 Federated Operations Centres 1 EIRO providing resources (CERN) 19 countries in 4 non-European Operations Centres Many of the sites are running ARC

9. www.egi.eu EGI-InSPIRE RI-261323 Project Objectives A sustainable federated production infrastructure –Resource providers in Europe and worldwide –With new technologies as they mature Support structured international research –Attract new international user communities –Technical services to ease wider adoption EGI-InSPIRE Update, Lyon 20119

10. EMI INFSO-RI-261611 EGI-InSPIRE EGI Ecosystem EGI-InSPIRE Update, Lyon 2011 10 Public Funding Bodies European Commission National Research Councils Service & Resource Providers EGI.eu Resource Infrastructure Providers Technology Providers Open Source Providers Commercial Providers User Community

11. EMI INFSO-RI-261611 EMI overview 11 What is EMI? Collaboration among the four major European middleware providers Three-year project, 24 M EUR, 50% from the European Commission, 50% from the partner Institutes 26 partners Work in three major areas: Consolidation of common libraries, clients, interfaces across the four MW stacks, decommissioning of unused services Development of new functionality based on user requirements User support - Reactively and proactively maintain the middleware distribution to keep it in line with the growing infrastructure usage ARC Consortia formed of 6 NorduGrid partners

12. EMI INFSO-RI-261611 The European Middleware Initiative (EMI) project represents a close collaboration of the major European middleware providers - ARC, gLite, UNICORE and dCache - to establish a sustainable model to support, harmonise and evolve distributed computing middleware for deployment in EGI, PRACE and other distributed e-Infrastructures EMI Mission Statement

13. EMI INFSO-RI-261611

14. EMI Workplan (3 years) Phase 1 (Kebnekaise): “planning” Software integration: created a distribution Agreements: EMI-ES, STAR, GSI-replacement, messaging use cases Design: EMI Authentication Library, EMI Registry Completed tasks: server-side GLUE2, consistent SRM,... Phase 2 (Matterhorn): “the YEAR of the Development” Functionality integration Agreements: compute accounting, delegation,... Implementation of all the agreements and first year designs Delivery of ”EMI products” Delivery of the Consolidation plans Design: AAI strategy, Cloud strategy Phase 3 (Monte Bianco) Full realization of the consolidation plans Implementation of AAI and Cloud strategy Finalization of new developments by bringing code to production level 5/10/2011 NorduGrid Workshop, Kiev 14

15. EMI INFSO-RI-261611 While climbing to the peaks.... In addition to development, EMI has a large ”base camp” to take care of Maintenance – bug fixing Support – issue resolution The ”base camp” has the highest priority 5/10/2011 NorduGrid Workshop, Kiev 15

16. EMI INFSO-RI-261611 EMI Baseline Services Applications Integrators, System Administrators ARC before, during and after EMI Users and infrastructure requirements, Standards, New technologies 3 yearsBefore EMIAfter EMI Specialized services, professional support and customization Standard interfaces

17. EMI INFSO-RI-261611 Dev highlightsfor Y2: compute Implementation of the agreed common job management methods (EMI-ES interface) – CREAM, UNICORE, A-REX Computing Elements – All EMI compute clients (arc* as well) Complete GLUE2 support – GLUE2 used in resource discovery, brokering, matchmking (arc*) Agreement and support for an EMI compute accounting record (JURA) ARGUS integration of computing elements (A-REX) Lots of product-level improvements 5/10/2011 NorduGrid Workshop, Kiev 17

18. EMI INFSO-RI-261611 Dev highlights Y2: data (1/2) GLUE2 support in data clients (arc*) EMI Data Access Library design and implementation (arc datalib) Storage Element and catalogue synchronization Better UNICORE data integration – SRM, LFC, AMGA NFS 4.1 / pNFS for EMI storage elements focus on standard protocols and procedures – POSIX file access and integration into SRM (File://) – http/WebDAV/Cloud storage note: ARC is note really involved in EMI data 5/10/2011 NorduGrid Workshop, Kiev 18

19. EMI INFSO-RI-261611 5/10/2011 NorduGrid Workshop, Kiev 19 Dev highlights: data (2/2) What we actually want (by the end of project) Google Storage Amazon S3 CloudFront EMI Storage FTS 3 Gs,S3 REST, WebDAV HTTP Gs,S3 REST, WebDAV HTTP EMI Storage

20. EMI INFSO-RI-261611 Dev highlights Y2: security Finalization of ”EMI profiles” – Common SAML profile (A-REX ?) – Common XACML profile (A-REX?) Security Token Service (STS) service – Simplified management of security credentials (AAI) – Authenticates and authorizes users based on security tokens – Transforms the security token into another security token – Implementation based on Shibboleth IDP & OpenWS/SAML v3 (Shib3) EMI Authentication Library implementation – JAVA, C, C++ APIs EMI delegation agreement – A delegation solution to replace ”GSI” in compute and data areas (A- REX) 5/10/2011 NorduGrid Workshop, Kiev 20

21. EMI INFSO-RI-261611 Dev highlight Y2: infra Full utilization of GLUE2 including validation EMI Service Registry – Full integration with ARC Delivery of service monitoring via NAGIOS – Each EMI service should come with probes Modified accounting publishers – DGAS, APEL – Accomodate new records – Utilize messaging 5/10/2011 NorduGrid Workshop, Kiev 21 EMI Service Registry

22. EMI INFSO-RI-261611 Dev highlight Y2: common/cross area Cloud strategy Consolidation plans for each area – What can be phased out? Additional platforms – Debian 6 (with an eye on latest Ubuntu) – SL6 Usability improvements – Command line parameters – Error messages 5/10/2011 NorduGrid Workshop, Kiev 22

23. EMI INFSO-RI-261611 Matterhorn in a nutshell The next Major release from the EMI project – Contains appx. 50-60 integrated products for compute, data, security and infra capabilities – Backward incompatibility allowed! The most development-dense EMI release Will be available on multiple platforms Planned release date: 26 April 2012 Support will run well after the end of EMI project Will contain substantial ARC 5/10/2011 NorduGrid Workshop, Kiev 23

24. EMI INFSO-RI-261611 NorduGrid package update 1 of 11.05 ARC Release (22.07.2011) nordugrid-arc-1.0.1, nordugrid-doc-1.0.2, nordugrid-compat-1.0.0, arcjobtool-0-3-0 EMI Kebnekaise Update 5 (4.8.2011) ships: ARC CE v.1.0.1 – “The Computing Element” ARC Clients v. 1.0.1 – Old and new CLIs (ng*, arc*), all client libs – Compute, data, infosys, security ARC Core v. 1.0.1 – ”common” stuff, the HED ARC gridftp server v. 1.0.1 – The “Classic SE” ARC Infosys v. 1.0.1 – ARIS, EGIIS, Grid Monitor Warning: Kebnekaise does not contain everything from ARC 11.05 – And also as UMD 1.2.0 (13.09.2011) Distribution of the “ARC products” in EMI 1 “Kebnekaise” and UMD 1.2 A new update is on the way: ARC 1.1.0

25. EMI INFSO-RI-261611 Welcome to the Technical meeting 5/10/2011 NorduGrid Workshop, Kiev 25

26. EMI INFSO-RI-261611 About the NG technical meetings 5/10/2011 NorduGrid Workshop, Kiev 26 Informal open discussions Sometimes going deep into the details Time to collect ideas Opportunity to ask the questions Give feedback, come with problems