1. Enabling Grids for E-sciencE www.eu-egee.org EGEE-III INFSO-RI-222667 EGEE and gLite are registered trademarks Francesco Giacomini JRA1 Activity Leader INFN EGEE-III First Review, 24-25 June, 2009 JRA1 Middleware Re-engineering Status Report

2. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 2 Activity Overview *includes 18 PMs of voluntary effort

3. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 3 Effort spent at M12 Some overspending by all partners – All understood – No impact is expected for their contribution during the rest of the project Voluntary effort Anticipation of work

4. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 4 Tasks TJRA1.1 Middleware support – “maintain and gradually improve the reliability, performance, usability, and manageability of the existing services”  Subset of gLite – Support via product-specific mailing lists, the gLite bug tracker (Savannah) and the general EGEE ticketing system (GGUS, on a best-effort basis) TJRA1.2 Research & development and standardisation – “development of components needed for an effective usage of the production resources and [...] adoption of consolidated international standards” – Main development is a coherent Authorization Framework TJRA1.3 Activity Management – Includes Security Architect

5. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 5 gLite Middleware General Services LHC File Catalogue Hydra Workload Management Service File Transfer Service Logging & Book keeping Service AMGA Storage Element Disk Pool Manager dCache Information Services BDII MON User Access User Interface Security Services Virtual Organisation Membership Service Authz. Service SCAS Proxy Server LCAS & LCMAPS Compute Element CREAMLCG-CE gLExec BLAH Worker Node

6. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 6 Products Security – VOMS and VOMSAdmin – Proxy and attribute certificate renewal – Shibboleth interoperability – LCAS, LCMAPS – gLExec – Delegation framework – CGSI_gSOAP – gsoap-plugin – Trustmanager, Util-Java – GridSite – Authorization Service Information, Monitoring – GLUE Schema, BDII – R-GMA Job Management – BLAH – CREAM, CEMon – WMS – Logging and Bookkeping Data Management – DPM – LFC – FTS – EDS, Hydra – GFAL – lcg_utils

7. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 7 Major achievements Middleware Support – Goal: to keep the infrastructure running  Infrastructure has doubled every year – Releases for all the services/components have been delivered to certification and production  Bug fixes  Improvements in terms of reliability, performance, usability, and manageability  Minor functional enhancements New services in production – CREAM  gLite interface to a Computing Element – HYDRA  A keystore for Medical Data Management

8. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 8 Major achievements /2 Development – The new Authorization service (Argus) is ready for certification Standardization – The GLUE Schema v. 2 has become an OGF standard – SRM v. 2.2 adopted by the gLite Storage Element (DPM) – JSDL support by the gLite meta-scheduler (WMS) – BES/JSDL adoption under discussion for the gLite CE (CREAM)  Co-chairing the OGF PGI WG – SAML support by VOMS

9. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 9 Organization The organization is based on largely autonomous development teams – Usually co-located with SA3 people for testing, forming clusters of competence In general a service (node) integrates components provided by multiple teams

10. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 10 Workplan Integrated in Savannah – Links to other tasks, patches, bugs

11. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 11 SW Engineering with ETICS Managed with ETICS – ~300 components, 75% gLite, 25% externals – Platforms, dependencies, node types, etc.

12. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 12 ETICS: Benefits Single place to manage many aspects of software engineering Remote builds, done on a trusted third-party infrastructure Comprehensive reports Availability of an artifact repository after a build – Readily installable packages Integrated testing facility – From unit to system testing Support for multi-platform portability Support for QA plugins – Checks on code – Metrics on code

13. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 13 Automatic testing: VOMS System testing, library testing and regression testing – Requires deployment of a functional VOMS server instance, including a MySQL database Run remotely on ETICS after a build – Results integrated in the build report

14. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 14 Portability to multiple platforms Joint SA3/JRA1 effort: – Coordination by SA3, changes in the code by JRA1 Slow progress – Lower priority than other needs – Lack of effort dedicated to integration activity Getting better – gLite 3.2 integrated builds on many platforms – Nightly builds available through ETICS – Build is not release! Multi-platform strategy – Few certified platforms – Uncertified packages available for more platforms  Help from the community

15. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 15 IPv6 Collaboration with SA2 – They ran the IPv6 static checker as an ETICS plugin and submitted 110 bugs  75 for JRA1 components Bugs are being fixed – Important but not urgent task Final testing to be done by SA2 on their IPv6 testbed Some services are IPv6 ready – Production LFC, DPM – Prototype BDII

16. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 16 Activity Management All-Hands meetings – Every about six months – 5-7 November 2008, Prague – 6-8 May 2009, Nicosia – Together with SA3  One day in common, one day Activity-specific EMT – Technical management, short-term view Security coordination – For operations and middleware TMB – Technical management, mid-term view

17. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 17 Goals for Y2 Organization – In preparation for EGI, from Clusters of Competence to Product teams  A team is responsible for all aspects of a software product (development, testing, certification, release, support,...) – Need to adapt the existing processes and procedures – Establishment of a gLite Consortium to provide a long-term sustainable roadmap for the gLite software Consolidation of the software – Multi-platform support – Review of error messages and logging – Documentation, especially of interfaces – Appropriate copyright and license attributions – Minimization of dependencies – Provision of a gLite SDK

18. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 18 Summary All Activity tasks proceed according to the plan – Deviations in effort consumption are not worrying Collaboration with SA3 and the ETICS team allows us to overcome difficulties The software process and management procedures are increasingly based on the available tools – Mainly ETICS and Savannah Transition to EGI requires changes both in organization and focus of technical activity