1. ETICS Services Management
SA1 Status Report
ETICS Services Management
ETICS2 First Review
Alberto Aimar
CERN
Bruxelles, 3 April 2009

2. Contents Goals Major Achievements Challenges Metrics and Statistics
Conclusions
WPX WP name
Bruxelles, 3 April

3. Goals
WPX WP name
Bruxelles, 3 April 3 3

4. Goals SA1 – Service Management
will ensure that the core ETICS services and infrastructure are reliable, stable and well documented,
providing industry-grade software lifecycle and release management, service deployment and monitoring tools and second-level user support (first-level support is provided by SA2).
responsibility of improving the services especially in the security and reliability areas,
assessing the extensions to the ETICS services developed internally by SA2, JRA1 and JRA2 or by the ETICS user community at large and adding them to the stable core of services after submitting them to rigorous quality assurance procedures.
WPX WP name
Bruxelles, 3 April

5. Deliverables and Activities
DSA1.1
Execution plan for first 12 months of infrastructure operation
M03
DSA1.2
ETICS Core Services Design Specification
M06
DSA1.3
ETICS Site Service Level Agreement
M09
DSA1.4
Execution plan for second 12 months of infrastructure operation
M12
SA1.1
Work Package coordination
SA1.2
Core service maintenance and extensions
SA1.3
Core service documentation
SA1.4
Infrastructure deployment, maintenance and upgrades
SA1.5
Core service certification
WPX WP name
Bruxelles, 3 April

6. Major Achievements
WPX WP name
Bruxelles, 3 April 6 6

7. ETICS SA1 Services
SA1 Status Report
ETICS2 All Hands Meeting

8. 1 – Performance and Speed ETICS Client Improvements
The system was designed for integrators and managers and the speed of execution of individual commands was not a priority Over time it has been used more and more by individual developers, whose primary concern is performance of single builds or tests
Improves performance from 200% to 900% depending on the task to be executed and the available hardware
Old client
New client
Speed-up
Modules
gLite
~35h
~4h
875%
384
WMS
1h 43m 41s
14m 16s
735%
110
Data Management
1h 12m 18s
10m 34s
720%
104
Security
29m 38s
5m 45s
483% 65 LB
14m 32s
2m 51s
460% 42

9. 2 – Scalability and Extensibility Worker Nodes Virtualization
Phase 1 (now): Moved to Virtual Machines
CERN has moved to double 4-core nodes (8 cores/each machine)
ETICS and other projects had to move to virtual images
Now the nodes in the WN pool are virtual machines
Static creation of virtual machines
Prepared page describing the Virtual Images management (updates, modification)
Phase 2: Dynamic Deployment of Virtual Images
ETICS Bootstrapper
A common repository of official images will be set up and maintained
The ETICS bootstrapper will download and start a virtual machine directly on the WN
Custom-configured virtual machines and is also needed to be able to use on other grid infrastructure (Submissions eng.)
Xen is the hypervisor that is more likely to be found (if any) in other infrastructures not at CERN

10. 3 – Data Custody ETICS Repository
The ETICS Repository has been reorganized
Major important improvement
Scalable and faster statistics
New version of the tools used (Java, etc)
A new browser interface and addressing based on REST
Presented to the user with a more intuitive tree of directories and files with icons.
The reports and the packages are now stored on a HA file system (AFS). Before were on disk maintained and backed up by us
SA1 Status Report
ETICS2 All Hands Meeting

11. 3b – Data Custody Repository of Build Results/Reports
Generation of RPM and Tar packages was already available but the Debian users and gLite needed other distribution formats
Dynamic APT/YUM Repositories were requested and are now available
Permanent APT/YUM repository for the registered repository
Repository is well implemented and has no known issues of scalability, etc
Reports are very clear and useful
No need of major development in next months
SA1 Status Report
ETICS2 All Hands Meeting

12. 3 – Userfriendly Web Client
Not many resources for this fundamental component until October 2008
Web App (restarted Oct 2008)
Porting to Firefox 3 was the major Achievement
Fixing bugs in the Web Apps
Changes required by others (multi-packaging, etc)
Portal (restarted Sept 2008)
Improved the External Requests and Submission web interface
Streamline interface for repetitive non-expert tasks (re-run build, test, etc) vs. more expert tasks (new package, configuration, etc)
Disseminator (restarted Oct 2008)
Deployed on an internal INFN machine to be tried and tested,
Need to be completed as the metrics are a cornerstone of many ETICS activities (Plug-ins, QA, A-QCM, gLite)
SA1 Status Report
ETICS2 All Hands Meeting

13. WPX WP name
Bruxelles, 3 April

14. WPX WP name
Bruxelles, 3 April

15. 4 – Organization and Monitoring Release Procedures and Deployment
Production Installation: “The” officially released supported ETICS
Release Candidate (RC) Installation: “Next” production installation available for final certification and test by selected users
Integration Testing Installation: All the release candidates of the packages are tagged at project level and installed for integration
Development Installation: A shared installation where developers can test their packages with the release candidates of other packages.
Individual Development: Installations: developers or teams can instantiate they infrastructure, often in reduced scale for individual development and testing.
Automatic Deployment
Automatic deployment allows the instantiation the ETICS Server and the ETICS Repository on given hosts
SA1 Status Report
ETICS2 All Hands Meeting

16. 4b – Organization and Monitoring Monitoring Dashboard (CERN SLS)
SA1 Status Report
ETICS2 All Hands Meeting

17. 4c – Organization and Monitoring ETICS Service Level Agreement
Yearly Availability
Yearly Reliability
Access to Project Binary packages
98%
99%
Access to Build Reports and Metrics Repository
97%
Build and Configuration Portal
95%
Support requests (creation of projects, new users, etc).
100%
Note: Availability and reliability values are determined by taking into account issues due to the ETICS Services functions; but not those caused by the services used by ETICS. E.g. if there is no network connectivity at CERN for 24h, those 24h will not be considered an ETICS downtime.
SA1 Status Report
ETICS2 All Hands Meeting

18. 5 – Integration of the ETICS development SA1 in ETICS 2
Collaboration with other activities
New Submission Engines  SA2
New plug-ins + Integration  JRA2
Multi-node Distr. Testing  JRA2
A-QCM + Metrics  NA2
Cross Submission  JRA1
Dissemination Material  NA2
Documentation and Support  SA2
SA1 Status Report
ETICS2 All Hands Meeting

19. Metrics and Statistics
WPX WP name
Bruxelles, 3 April 19 19

20. SA1 Deliverables
DSA1.1 – Execution plan for 12 months of infrastructure operation M03
This deliverable describes the execution plan for the first half of the ETICS 2 project, including the core service roadmap and the infrastructure deployment plan.
DSA1.2 – ETICS Core Services Design Specification M06
This deliverable describes the overall ETICS 2 core services.
DSA1.3 – ETICS Site Service Level Agreement M09
This deliverable describes the Service Level Agreements upon which the ETICS service will be provided. The SLAs will define the service level the users can expect from the service in terms of availability, accessibility and support.
DSA1.4 – Execution plan for second 12 months of infrastructure operation M12
This deliverable describes the execution plan for the second half of the ETICS 2 project, including the core service roadmap and the infrastructure deployment plan.
DSA1.5 – Infrastructure and core services certification and usage report M21
This deliverable reports on the release management cycles and certification of the ETICS 2 infrastructure and core services, including lessons learned and corrective action to apply. OK
TBD
SA1 Status Report
ETICS2 All Hands Meeting

21. Usage of the ETICS Services Metrics
Build/test type Q2 Q3
production
13703 builds
17121 builds
Test
~600 builds
~3000 builds
other
~300 builds
~650 builds
Project Q2 Q3
org.glite
7464
3423
org.etics
3672
org.glite.testsuites
2154 21
org.gcube
135
521
torquemaui 35
132
externals 34 68
unicore 33
131
root-project 26 -
castor 19
quattor-compiler 7
vdt
WPX WP name
Bruxelles, 3 April

22. Challenges
WPX WP name
Bruxelles, 3 April 22 22

23. SA1 Challenges Resources (late hiring) What did we do
Difficult to hire people joined at M7/10 or later (after some milestones were due)
What did we do
Hire more people with money unspent
Be 100% over short periods (100% for 3 months vs. 25% for one year)
Working in the same location for some periods
CERN late of 6 months  Hired a Technical student
Service to run, upgrades to perform, gLite requests
Major problems with AFS and with new hardware
Receive show-stoppers requests from EGEE
Sometimes conflicting, sometimes not using the solution
Some re-planning needed and is being done
 Are all milestones still valid in the year?
 Are the ETICS Services sustainable outside research?
SA1 Status Report
ETICS2 All Hands Meeting

24. Conclusions
WPX WP name
Bruxelles, 3 April 24 24

25. Conclusions Goals Deliverables Key Achievements
Speed
Scalability and Extensibility
Data Custody
Organization and Monitoring
Integration of other Development
Recruiting takes time and delay ALL next activities
Planning focusing on the needs of future sustainability
WPX WP name
Bruxelles, 3 April

26. SA1 Execution Plan Summary
SA1 Status Report
ETICS2 All Hands Meeting

27. Add support for SLC5 gcc 4.1.3 and gcc 4.3.3 (gLite)
In addition or instead we did:
Add support for SLC5 gcc and gcc (gLite)
YUM Repository (gLite)
Virtualization of WN and servers (CERN)
ETICS Bootstrapper (to use other infrastructures)
Regression tests for the Client
New Submission Tab (needed by users and better management of the Services)
New people in the projects (late recruiting (M7-M8)
WPX WP name
Bruxelles, 3 April