JRA2: Quality Assurance Gabriel Zaquine - BT EGEE-II Final EU Review (CERN) 8-9 July 2008

Contents Goals and organisation Processes and procedures Metrics programme Industry QA standards follow-up Summary JRA2- G. Zaquine - EGEE-II Final EU Review - 8-9 July 2008

Goals Coordinate the overall Quality Assurance (QA) aspects of the project to ensure its processes, services and deliverables are of high-quality Establish quality procedures, methods and associated tools for all project activities Establish a metrics programme for key targets of the project and each activity including measures of the performance of each partner through the project activities and their contribution to the programme of work Foster industry QA standards best practices JRA2- G. Zaquine - EGEE-II Final EU Review - 8-9 July 2008

JRA2 in numbers Manpower: 3 partners, 2 countries, 3 FTE EGEE-II Budget JRA2- G. Zaquine - EGEE-II Final EU Review - 8-9 July 2008

QA organisation The EGEE-II Quality Assurance activity is organised through the Quality Assurance Group (QAG) composed of QA representatives from each activity coordinated by the JRA2 Quality coordinator The main role of the Activity QA representatives is to: define the Activity Quality & Measurement Plans ensure that quality measures as agreed are applied inside their activity The execution of the verification and testing activities is the responsibility of the individual activities JRA2- G. Zaquine - EGEE-II Final EU Review - 8-9 July 2008

Processes and Procedures Capitalize on best practices JRA2 Quality Assurance Processes and Procedures Capitalize on best practices JRA2- G. Zaquine - EGEE-II Final EU Review - 8-9 July 2008

Processes / Procedures Used by several other projects Fully documented JRA2- G. Zaquine - EGEE-II Final EU Review - 8-9 July 2008

Project Management The Project management procedures and related material have proven successful in managing a large project since EGEE phase I In particular, many mechanisms have been established by the project to assess project progress: PEB weekly meetings, all-activity meetings (AAM) on a quarterly basis Quarterly reports, periodical reports, cost claims Activities’ execution plans and associated project tracking tool (PPT) Deliverables and milestones reviews Users’ surveys Metrics program: Overall metrics, Activity Metrics, Partner activity review External Advisory Committee reports EU reviews Many of this material has been used by related projects For EGEE-III, in order to monitor the progress of the programme of work in each country and the involvement of individual partners, a country-based review process will be used systematically across the whole project JRA2- G. Zaquine - EGEE-II Final EU Review - 8-9 July 2008

Middleware As a means of reinforcing the long-term stability of gLite, several actions have been performed by the project, such as: gLite restructuring including dependency challenge and dependency review (cross activity code review) More robust software and better maintainability & portability Review of gLite authorization (MJRA1.7) Framework re-design Adoption of the ETICS build system More control of the software structure Moving to a continuous release process Allowing independent component releases JRA1 and SA3 will continue Working closely with the ETICS-2 project particularly with the automated build systems Automating the testing and certification procedures will continue The acceptance criteria will be enforced with a check list that defines the criteria at each stage in the release process JRA2- G. Zaquine - EGEE-II Final EU Review - 8-9 July 2008

Operations (1/2) Quality Assurance is an inherent part of the everyday work of the Operations Operations have shown a continual improvement in the quality of the service with clarified work procedures of deployment and day-to-day operations of both production and pre-production services as well as the improvement of various automated tools A wide range of metrics are available in a metrics portal, and used in reporting and as part of assessments of both sites and services Several mechanisms have been put in place to gather feedback weekly operations meeting operations workshops feedback from GGUS for user support survey of the user community on all aspects of the operations of the infrastructure JRA2- G. Zaquine - EGEE-II Final EU Review - 8-9 July 2008

Operation (2/2) The SA1 activity is introducing SLAs and the means of collecting the metrics that will be used to monitor them A general strategy of the Service Activities in EGEE-III will be to optimise the activities in order to reduce the overall level of effort required in the future to manage a sustainable infrastructure improving tools needed to support operations monitoring automation of alarms develop a full set of Service Level Agreements (SLA) at several levels, associated with mechanisms to monitor and verify these SLAs JRA2- G. Zaquine - EGEE-II Final EU Review - 8-9 July 2008

Security Management Best practices have been considered and described in MJRA2.2.1 “Security audit, strategy and plan”, and progressively implemented Security audits have been performed on aspects of Grid operations and middleware by various groups within the project as well as external third parties (BARC/EUIndiaGrid and CNES) Security Service Challenges (SSC) is an ongoing process on many grid sites, running in its 3rd phase http://cern.ch/osct/ssc.html The goal of theses challenges is to test the incident of our security tools, policies and procedures and identify where improvements can be made Procedures generally understood Proposed enhancements for Access control (ex: blocking a user at the authZ level) and tractability - Logging and audit trails (logs) Review of gLite authorization (MJRA1.7) Framework re-design In addition, the OSG is part of the overall work as partner in the security groups, with exchange of experiences from similar efforts within the US This will be expended to other projects for the future JRA2- G. Zaquine - EGEE-II Final EU Review - 8-9 July 2008

JRA2 Quality Assurance Metrics programme Provide a powerful tool for assessment at each level : - Project overall metrics - Activity metrics - Partner metrics JRA2- G. Zaquine - EGEE-II Final EU Review - 8-9 July 2008

Overall main metrics The project identified overall metrics, with “target” values defined in the Technical Annex http://egee-jra2.web.cern.ch/EGEE-JRA2/EGEE-II/QoS/Follow-up/Overall.htm The project has exceeded all targets set for the project JRA2- G. Zaquine - EGEE-II Final EU Review - 8-9 July 2008

Activity metrics A full set of activity metrics have been provided by each activity. These are grouped into themes, namely: project management, training, dissemination, usage, operation, middleware JRA2- G. Zaquine - EGEE-II Final EU Review - 8-9 July 2008

SA1 metrics … as part of everyday monitoring tool The project has implemented the metrics defined in MSA1.1 More than 40 metrics including timeseries data as well as snapshot data providing more finely-grained data http://egee-docs.web.cern.ch/egee-docs/list.php?dir=./mig/production/& Size metrics (CPU, Storage, Users) Operation metrics (Site availability, Site reliability) Usage metrics (Jobs, Data transfer, CPU, Storage) User support tickets (Numbers, Types, Response times) Services metrics (CE, SE, SRM, BDII, RB, LFC, FTS, VOMS, etc) The site availability and reliability metrics calculated by SAM & GridView are used for sites’ SLA conformance JRA2- G. Zaquine - EGEE-II Final EU Review - 8-9 July 2008

Availability of a Site / Services Site availability is calculated as: logical AND of all service types If a site has several instances of a service type (e.g. Computing Elements), the service is deemed to be available if any of the instances are available (logical OR) https://edms.cern.ch/document/860386 ..... █ D  = Down (unscheduled) █ NA = No test result Available █ M = Maintenance (scheduled) █ U = Up JRA2- G. Zaquine - EGEE-II Final EU Review - 8-9 July 2008

From Partner Activity Review to Federation & Country review Given the size of the EGEE-II (more than 90 partners), the partners’ metrics programme was devised to ensure that all partners are performing as expected The Partner Activity Reviews have been organized with a format depending on the nature of each activity http://egee-jra2.web.cern.ch/EGEE-JRA2/EGEE-II/QoS/Follow-up/PartnerMetrics.htm The Activity reviews went well. The Activity Managers found these reviews useful. It was a forum to discuss issues, examine progress & contributions and establish better communication between partners This is being expanded to “federation reviews” in EGEE-III Monitor the progress of the programme of work in each country and the involvement of individual partners Examine progress and contribution and identify and promote regional successes and initiatives for re-use in other countries Verify the state of readiness for the transition to EGI JRA2- G. Zaquine - EGEE-II Final EU Review - 8-9 July 2008

JRA2 Quality Assurance Industry QA standards follow-up “Evaluate best practices that could be progressively applied to EGEE” - ITIL group - Relationships with ETICS JRA2- G. Zaquine - EGEE-II Final EU Review - 8-9 July 2008

Following up industry standards and best practices (1/2) The EGEE ITIL group has been set up with the goal to assess ITIL best practices Trainings “Introduction to ITIL” and dedicated presentations at EGEE conferences ITIL Services Level Management (SLA modeled on ITIL recommendations) ROC / Site HARDWARE AND CONNECTIVITY CRITERIA DESCRIPTION OF SERVICES COVERED SERVICE HOURS AVAILABILITY SUPPORT SERVICE REPORTING AND REVIEWING https://edms.cern.ch/document/860386 GGUS/ROCs for users support service https://edms.cern.ch/document/888089 ITIL Service Desk and Incident Management Evaluation for EGEE Networks Operations Centre (ENOC) Importance of a common terminology e.g. Incident, Problem, Known error Evaluation vs GGUS Out of 103 ITIL items analysed from the ITIL “Incident Management”, most are fulfilled by the present arrangements of GGUS Improvement of Interface between GGUS central part and regional parts in order to improve Problem Management Enforcement of Incident Manager ITIL will be useful at EGI stage as a set of best practice and common terminology The QAG will pursue the work on ITIL based on ITILV 3 within EGEE-III ITIL 3 JRA2- G. Zaquine - EGEE-II Final EU Review - 8-9 July 2008

Following up industry standards and best practices (2/2) Relationships with ETICS have continued JRA2 has followed the work of the ETICS QA software certification model based on ISO and CMMi standards, including: automated evaluation modules for Static analysis Coding style Structural testing Functional testing Standards compliance For EGEE-III, the QAG will encourage ISO and CMMi best practices toward Software Engineering part JRA2- G. Zaquine - EGEE-II Final EU Review - 8-9 July 2008

Summary The QA objectives planned for the project have been achieved, and QA is in active use across all activities The quality of the service of the EGEE infrastructure has been continuously increased, with clarified work procedures, as well as the improvement of various automated tools The project has exceeded all targets set for the second year of the project In the list of measures for continuous quality improvement foreseen within EGEE-III Quality Assurance will be pervasive throughout all EGEE-III activities and each activity will plan for their QA measures as part of the activity tasks and milestones Further implement the Services Level Management process with a full set of Service Level Agreements (SLA) at several levels, associated with mechanisms to monitor and verify these SLAs The acceptance software component criteria will be enforced at each stage Security auditing to cooperate more with other projects To monitor the progress of the programme of work in each country and the involvement of individual partners, a federation reviews process will be used systematically across the whole project The QAG group will also pursue the work on ITIL based on ITIL 3, and the work with ETICS concerning Software Engineering, encouraging ISO and CMMi best practices JRA2- G. Zaquine - EGEE-II Final EU Review - 8-9 July 2008