ETICS Services Management SA1 Status Report ETICS Services Management ETICS2 First Review Alberto Aimar CERN Bruxelles, 3 April 2009

Contents Goals Major Achievements Challenges Metrics and Statistics Conclusions WPX WP name Bruxelles, 3 April

Goals WPX WP name Bruxelles, 3 April 3 3

Goals SA1 – Service Management will ensure that the core ETICS services and infrastructure are reliable, stable and well documented, providing industry-grade software lifecycle and release management, service deployment and monitoring tools and second-level user support (first-level support is provided by SA2). responsibility of improving the services especially in the security and reliability areas, assessing the extensions to the ETICS services developed internally by SA2, JRA1 and JRA2 or by the ETICS user community at large and adding them to the stable core of services after submitting them to rigorous quality assurance procedures. WPX WP name Bruxelles, 3 April

Deliverables and Activities DSA1.1 Execution plan for first 12 months of infrastructure operation M03 DSA1.2 ETICS Core Services Design Specification M06 DSA1.3 ETICS Site Service Level Agreement M09 DSA1.4 Execution plan for second 12 months of infrastructure operation M12 SA1.1 Work Package coordination SA1.2 Core service maintenance and extensions SA1.3 Core service documentation SA1.4 Infrastructure deployment, maintenance and upgrades SA1.5 Core service certification WPX WP name Bruxelles, 3 April

Major Achievements WPX WP name Bruxelles, 3 April 6 6

ETICS SA1 Services SA1 Status Report ETICS2 All Hands Meeting

1 – Performance and Speed ETICS Client Improvements The system was designed for integrators and managers and the speed of execution of individual commands was not a priority Over time it has been used more and more by individual developers, whose primary concern is performance of single builds or tests Improves performance from 200% to 900% depending on the task to be executed and the available hardware Old client New client Speed-up Modules gLite ~35h ~4h 875% 384 WMS 1h 43m 41s 14m 16s 735% 110 Data Management 1h 12m 18s 10m 34s 720% 104 Security 29m 38s 5m 45s 483% 65 LB 14m 32s 2m 51s 460% 42

2 – Scalability and Extensibility Worker Nodes Virtualization Phase 1 (now): Moved to Virtual Machines CERN has moved to double 4-core nodes (8 cores/each machine) ETICS and other projects had to move to virtual images Now the nodes in the WN pool are virtual machines Static creation of virtual machines Prepared page describing the Virtual Images management (updates, modification) Phase 2: Dynamic Deployment of Virtual Images ETICS Bootstrapper A common repository of official images will be set up and maintained The ETICS bootstrapper will download and start a virtual machine directly on the WN Custom-configured virtual machines and is also needed to be able to use on other grid infrastructure (Submissions eng.) Xen is the hypervisor that is more likely to be found (if any) in other infrastructures not at CERN

3 – Data Custody ETICS Repository The ETICS Repository has been reorganized Major important improvement Scalable and faster statistics New version of the tools used (Java, etc) A new browser interface and addressing based on REST Presented to the user with a more intuitive tree of directories and files with icons. The reports and the packages are now stored on a HA file system (AFS). Before were on disk maintained and backed up by us SA1 Status Report ETICS2 All Hands Meeting

3b – Data Custody Repository of Build Results/Reports Generation of RPM and Tar packages was already available but the Debian users and gLite needed other distribution formats Dynamic APT/YUM Repositories were requested and are now available Permanent APT/YUM repository for the registered repository Repository is well implemented and has no known issues of scalability, etc Reports are very clear and useful No need of major development in next months SA1 Status Report ETICS2 All Hands Meeting

3 – Userfriendly Web Client Not many resources for this fundamental component until October 2008 Web App (restarted Oct 2008) Porting to Firefox 3 was the major Achievement Fixing bugs in the Web Apps Changes required by others (multi-packaging, etc) Portal (restarted Sept 2008) Improved the External Requests and Submission web interface Streamline interface for repetitive non-expert tasks (re-run build, test, etc) vs. more expert tasks (new package, configuration, etc) Disseminator (restarted Oct 2008) Deployed on an internal INFN machine to be tried and tested, Need to be completed as the metrics are a cornerstone of many ETICS activities (Plug-ins, QA, A-QCM, gLite) SA1 Status Report ETICS2 All Hands Meeting

WPX WP name Bruxelles, 3 April

WPX WP name Bruxelles, 3 April

4 – Organization and Monitoring Release Procedures and Deployment Production Installation: “The” officially released supported ETICS Release Candidate (RC) Installation: “Next” production installation available for final certification and test by selected users Integration Testing Installation: All the release candidates of the packages are tagged at project level and installed for integration Development Installation: A shared installation where developers can test their packages with the release candidates of other packages. Individual Development: Installations: developers or teams can instantiate they infrastructure, often in reduced scale for individual development and testing. Automatic Deployment Automatic deployment allows the instantiation the ETICS Server and the ETICS Repository on given hosts SA1 Status Report ETICS2 All Hands Meeting

4b – Organization and Monitoring Monitoring Dashboard (CERN SLS) SA1 Status Report ETICS2 All Hands Meeting

4c – Organization and Monitoring ETICS Service Level Agreement Yearly Availability Yearly Reliability Access to Project Binary packages 98% 99% Access to Build Reports and Metrics Repository 97% Build and Configuration Portal 95% Support requests (creation of projects, new users, etc). 100% Note: Availability and reliability values are determined by taking into account issues due to the ETICS Services functions; but not those caused by the services used by ETICS. E.g. if there is no network connectivity at CERN for 24h, those 24h will not be considered an ETICS downtime. SA1 Status Report ETICS2 All Hands Meeting

5 – Integration of the ETICS development SA1 in ETICS 2 Collaboration with other activities New Submission Engines  SA2 New plug-ins + Integration  JRA2 Multi-node Distr. Testing  JRA2 A-QCM + Metrics  NA2 Cross Submission  JRA1 Dissemination Material  NA2 Documentation and Support  SA2 SA1 Status Report ETICS2 All Hands Meeting

Metrics and Statistics WPX WP name Bruxelles, 3 April 19 19

SA1 Deliverables DSA1.1 – Execution plan for 12 months of infrastructure operation M03 This deliverable describes the execution plan for the first half of the ETICS 2 project, including the core service roadmap and the infrastructure deployment plan. DSA1.2 – ETICS Core Services Design Specification M06 This deliverable describes the overall ETICS 2 core services. DSA1.3 – ETICS Site Service Level Agreement M09 This deliverable describes the Service Level Agreements upon which the ETICS service will be provided. The SLAs will define the service level the users can expect from the service in terms of availability, accessibility and support. DSA1.4 – Execution plan for second 12 months of infrastructure operation M12 This deliverable describes the execution plan for the second half of the ETICS 2 project, including the core service roadmap and the infrastructure deployment plan. DSA1.5 – Infrastructure and core services certification and usage report M21 This deliverable reports on the release management cycles and certification of the ETICS 2 infrastructure and core services, including lessons learned and corrective action to apply. OK TBD SA1 Status Report ETICS2 All Hands Meeting

Usage of the ETICS Services Metrics Build/test type Q2 Q3 production 13703 builds 17121 builds Test ~600 builds ~3000 builds other ~300 builds ~650 builds Project Q2 Q3 org.glite 7464 3423 org.etics 3672 org.glite.testsuites 2154 21 org.gcube 135 521 torquemaui 35 132 externals 34 68 unicore 33 131 root-project 26 - castor 19 quattor-compiler 7 vdt WPX WP name Bruxelles, 3 April

Challenges WPX WP name Bruxelles, 3 April 22 22

SA1 Challenges Resources (late hiring) What did we do Difficult to hire people joined at M7/10 or later (after some milestones were due) What did we do Hire more people with money unspent Be 100% over short periods (100% for 3 months vs. 25% for one year) Working in the same location for some periods CERN late of 6 months  Hired a Technical student Service to run, upgrades to perform, gLite requests Major problems with AFS and with new hardware Receive show-stoppers requests from EGEE Sometimes conflicting, sometimes not using the solution Some re-planning needed and is being done  Are all milestones still valid in the year?  Are the ETICS Services sustainable outside research? SA1 Status Report ETICS2 All Hands Meeting

Conclusions WPX WP name Bruxelles, 3 April 24 24

Conclusions Goals Deliverables Key Achievements Speed Scalability and Extensibility Data Custody Organization and Monitoring Integration of other Development Recruiting takes time and delay ALL next activities Planning focusing on the needs of future sustainability WPX WP name Bruxelles, 3 April

SA1 Execution Plan Summary SA1 Status Report ETICS2 All Hands Meeting

Add support for SLC5 gcc 4.1.3 and gcc 4.3.3 (gLite) In addition or instead we did: Add support for SLC5 gcc 4.1.3 and gcc 4.3.3 (gLite) YUM Repository (gLite) Virtualization of WN and servers (CERN) ETICS Bootstrapper (to use other infrastructures) Regression tests for the Client New Submission Tab (needed by users and better management of the Services) New people in the projects (late recruiting (M7-M8) WPX WP name Bruxelles, 3 April