EMI INFSO-RI EMI 2 Matterhorn roadmap Balázs Kónya (Lund University) EMI Technical Director EGI Technical Forum, Lyon, 20 September 2011

EMI INFSO-RI outline EMI is about software, the mission of the project is to deliver software for DCIs EMI development cycle Development highlights EMI 2 ”Matterhorn” – Content – Timeline Without dwelling into the details 20/09/2011 EGI TF, Lyon 2

EMI INFSO-RI EMI on the DCI landscape 20/09/2011 EGI TF, Lyon 3 EMI DNA1.4 - EMI Roadmap and DCI Collaborations

EMI INFSO-RI First peak has been conquered: EMI 1 20/09/2011 EGI TF, Lyon Kebnekaise (2104 m), 12 May Support to be provided until

EMI INFSO-RI Next peak to climb: EMI 2 20/09/2011 EGI TF, Lyon Matterhorn (4448 m), April

EMI INFSO-RI We have started the journey 20/09/2011 EGI TF, Lyon We defined the Plan (what to climb, how to go, what to carry) DNA1.3.2 ”Technical development plan” We have the necessary tools build & integration, task tracking We are on the way 6

EMI INFSO-RI From requirements to released products SA1 (PTs) Released products JRA1 (PTs) Development tasks Area Leaders Area work plans for the PTs PTB Tech Objectives PTB Prioritized, assessed requirements Customers Requirements 20/09/2011 EGI TF, Lyon 7

EMI INFSO-RI While climbing to the peak.... In addition to EMI 2 development, EMI has a large ”base camp” to take care of Maintenance – bug fixing Support – issue resolution The ”base camp” has the highest priority 20/09/2011 EGI TF, Lyon 8

EMI INFSO-RI Matterhorn: requirements Sources of requirements – EGI-TCB: formal requests communicated via EGI Tracker – WLCG: informal requests communicated verbally or via – Users: direct communication with PTs Handling of requirements – Requirement collection and processing for EMI 2: CLOSED – No major changes in EMI 2 directions Last minute case-by-case adjustments are possible – Requirement collection for EMI 3: OPEN Requirement assessment is a continously running process Everything is recorded and tracked: – – note: requirements from the project proposal phase are not there 20/09/2011 EGI TF, Lyon 9

EMI INFSO-RI Matterhorn: requirements Endorsed top priority requirements: WMS should provide better feedback about jobs, automated resubmission of stuck jobs Release EMI clients on SL5/32 Standard file locations for logs and temporary files Publish mw service version Provide Nagios probes for EMI services Better error messages (more verbose, informative) Coherency of command line commands parameters Coherency of APIs Support for Debian 6 Support for SL6 Ubuntu support at least for EMI clients CEs should support a set of LRMS Reduce memory consumption of CREAM 20/09/2011 EGI TF, Lyon 10

EMI INFSO-RI Matterhorn: requirements and some non-accepted ones: Fixing known bugs before adding new features Uniform logging Improving (automatization) yaim configuration Unicore GUI improvements 20/09/2011 EGI TF, Lyon 11

EMI INFSO-RI EMI Workplan (3 years) Phase 1 (Kebnekaise): “planning” Software integration: created a distribution Agreements: EMI-ES, STAR, GSI-replacement, messaging use cases Design: EMI Authentication Library, EMI Registry Completed tasks: server-side GLUE2, consistent SRM,... Phase 2 (Matterhorn): “the YEAR of the Development” Functionality integration Agreements: compute accounting, delegation,... Implementation of all the agreements and first year designs Delivery of ”EMI products” Delivery of the Consolidation plans Design: AAI strategy, Cloud strategy Phase 3 (Monte Bianco) Full realization of the consolidation plans Implementation of AAI and Cloud strategy Finalization of new developments by bringing code to production level 20/09/2011 EGI TF, Lyon 12

EMI INFSO-RI Dev highlights: compute (1/2) Implementation of the agreed common job management methods (EMI-ES interface) – CREAM, UNICORE, A-REX Computing Elements – All EMI compute clients Complete GLUE2 support – GLUE2 used in resource discovery, brokering, matchmking Agreement and support for an EMI compute accounting record (UR) ARGUS integration of computing elements Lots of product-level improvements – see next slide as an example 20/09/2011 EGI TF, Lyon 13

EMI INFSO-RI Dev highlights: compute (2/2) gLite compute PT plans: CREAM High Availability – Pool of CREAM CEs seen as a single logical CE (S)GE support for CREAM WMS Support for DAG jobs without using Condor DAGMan – This will enable the submission to CREAM CEs also for DAG jobs 20/09/2011 EGI TF, Lyon 14

EMI INFSO-RI Dev highlights: data (1/2) GLUE2 support in data clients EMI Data Access Library design and implementation Storage Element and catalogue synchronization – Will not be released with EMI 2 Better UNICORE data integration – SRM, LFC, AMGA NFS 4.1 / pNFS for EMI storage elements focus on standard protocols and procedures – POSIX file access and integration into SRM (File://) – http/WebDAV/Cloud storage details: Monday Data Workshop presentations 20/09/2011 EGI TF, Lyon 15

EMI INFSO-RI /09/2011 EGI TF, Lyon 16 Dev highlights: data (2/2) What we actually want Google Storage Amazon S3 CloudFront EMI Storage FTS 3 Gs,S3 REST, WebDAV HTTP Gs,S3 REST, WebDAV HTTP EMI Storage

EMI INFSO-RI Dev highlights: security Finalization of ”EMI profiles” – Common SAML profile – Common XACML profile Security Token Service (STS) service – Simplified management of security credentials (AAI) – Authenticates and authorizes users based on security tokens – Transforms the security token into another security token – Implementation based on Shibboleth IDP & OpenWS/SAML v3 (Shib3) EMI Authentication Library implementation – JAVA, C, C++ APIs EMI delegation agreement – A delegation solution to replace ”GSI” in compute and data areas Details: Wednesday ”EMI Security” session 20/09/2011 EGI TF, Lyon 17

EMI INFSO-RI Dev highlight: infra Full utilization of GLUE2 including validation EMI Service Registry Delivery of service monitoring via NAGIOS – Each EMI service should come with probes Modified accounting publishers – DGAS, APEL – Accomodate new records – Utilize messaging 20/09/2011 EGI TF, Lyon 18 EMI Service Registry

EMI INFSO-RI Dev highlight: common/cross area Cloud strategy (see next talk) Consolidation plans for each area – What can be phased out? Additional platforms – Debian 6 (with an eye on latest Ubuntu) – SL6 Usability improvements – Command line parameters – Error messages 20/09/2011 EGI TF, Lyon 19

EMI INFSO-RI Matterhorn: what will be inside the box? EMI 2 will be largely based on products taken from EMI 1 – EMI 1 contains 54 products from ”AMGA to WMS” Product updates with possible backward incompatible changes New products: – STS, Pseudonimity – EMI Service Registry, EMI_datalib, EMI_authlib Components to be phased out – Not yet decided, consolidation plans are due in October – There’ll be products tagged as ”discontinued” – EMI project is contractually obliged to cut its portfolio The content will be finalized as part of the release preparation – We are strict on deadlines, delayed products can and will drop out 20/09/2011 EGI TF, Lyon 20

EMI INFSO-RI Matterhorn: release timeline (1/2) Development tasks defined and being tracked - July, 2011 Some task are not yet recorded BUILD environment ready for all platforms: – August, 2011 Not fully completed QA policies and procedures ready – September, 2011 RC1 (1 st code freeze) – 14 th Dec, 2011 – Successful build and packages available – All foreseen functionalities should be implemented RC2 (2 nd code freeze) – 1 st Feb, 2012 – Bug fixes – Available test reports for in-house deployment of RC1 20/09/2011 EGI TF, Lyon 21

EMI INFSO-RI Matterhorn: release timeline (2/2) RC3 ( 3 rd code freeze) - 29 th Feb, 2012 – RC2 functionality and config testing, deployment on EMI testbed RC4 (final release) - 28 th Mar, 2012 – Full test reports for RC3 – Fully deployed RC3 on the EMI testbed – Only authorized bug fixes (tracked) Final Release available in testing repository - 30 th Mar, 2012 – Complete testing and certification reports available – All tasks “Certified” Announce EMI 2 (Matterhron) - 26 th Apr, 2012 Preparation, planning for EMI 3 starts – January /09/2011 EGI TF, Lyon 22

EMI INFSO-RI Matterhorn in a nutshell The next Major release from the EMI project – Contains appx integrated products for compute, data, security and infra capabilities – Backward incompatibility allowed! The most development-dense EMI release Will be available on multiple platforms Planned release date: 26 April 2012 Support will run well after the end of EMI project 20/09/2011 EGI TF, Lyon 23

EMI INFSO-RI Thank you 20/09/2011 EGI TF, Lyon EMI is partially funded by the European Commission under Grant Agreement INFSO-RI