1. JRA2: Quality Assurance
Gabriel Zaquine - BT
EGEE-II Final EU Review (CERN)
8-9 July 2008

2. Contents Goals and organisation Processes and procedures
Metrics programme
Industry QA standards follow-up
Summary
JRA2- G. Zaquine - EGEE-II Final EU Review July 2008

3. Goals
Coordinate the overall Quality Assurance (QA) aspects of the project to ensure its processes, services and deliverables are of high-quality
Establish quality procedures, methods and associated tools for all project activities
Establish a metrics programme for key targets of the project and each activity
including measures of the performance of each partner through the project activities and their contribution to the programme of work
Foster industry QA standards best practices
JRA2- G. Zaquine - EGEE-II Final EU Review July 2008

4. JRA2 in numbers Manpower: 3 partners, 2 countries, 3 FTE
EGEE-II Budget
JRA2- G. Zaquine - EGEE-II Final EU Review July 2008

5. QA organisation
The EGEE-II Quality Assurance activity is organised through the Quality Assurance Group (QAG)
composed of QA representatives from each activity
coordinated by the JRA2 Quality coordinator
The main role of the Activity QA representatives is to:
define the Activity Quality & Measurement Plans
ensure that quality measures as agreed are applied inside their activity
The execution of the verification and testing activities is the responsibility of the individual activities
JRA2- G. Zaquine - EGEE-II Final EU Review July 2008

6. Processes and Procedures Capitalize on best practices
JRA2 Quality Assurance
Processes and Procedures Capitalize on best practices
JRA2- G. Zaquine - EGEE-II Final EU Review July 2008

7. Processes / Procedures
Used by several other projects
Fully documented
JRA2- G. Zaquine - EGEE-II Final EU Review July 2008

8. Project Management
The Project management procedures and related material have proven successful in managing a large project since EGEE phase I
In particular, many mechanisms have been established by the project to assess project progress:
PEB weekly meetings, all-activity meetings (AAM) on a quarterly basis
Quarterly reports, periodical reports, cost claims
Activities’ execution plans and associated project tracking tool (PPT)
Deliverables and milestones reviews
Users’ surveys
Metrics program: Overall metrics, Activity Metrics, Partner activity review
External Advisory Committee reports
EU reviews
Many of this material has been used by related projects
For EGEE-III, in order to monitor the progress of the programme of work in each country and the involvement of individual partners, a country-based review process will be used systematically across the whole project
JRA2- G. Zaquine - EGEE-II Final EU Review July 2008

9. Middleware
As a means of reinforcing the long-term stability of gLite, several actions have been performed by the project, such as:
gLite restructuring including dependency challenge and dependency review (cross activity code review)
More robust software and better maintainability & portability
Review of gLite authorization (MJRA1.7)
Framework re-design
Adoption of the ETICS build system
More control of the software structure
Moving to a continuous release process
Allowing independent component releases
JRA1 and SA3 will continue
Working closely with the ETICS-2 project particularly with the automated build systems
Automating the testing and certification procedures will continue
The acceptance criteria will be enforced with a check list that defines the criteria at each stage in the release process
JRA2- G. Zaquine - EGEE-II Final EU Review July 2008

10. Operations (1/2)
Quality Assurance is an inherent part of the everyday work of the Operations
Operations have shown a continual improvement in the quality of the service
with clarified work procedures of deployment and day-to-day operations of both production and pre-production services
as well as the improvement of various automated tools
A wide range of metrics are available in a metrics portal, and used in reporting and as part of assessments of both sites and services
Several mechanisms have been put in place to gather feedback
weekly operations meeting
operations workshops
feedback from GGUS for user support
survey of the user community on all aspects of the operations of the infrastructure
JRA2- G. Zaquine - EGEE-II Final EU Review July 2008

11. Operation (2/2)
The SA1 activity is introducing SLAs and the means of collecting the metrics that will be used to monitor them
A general strategy of the Service Activities in EGEE-III will be to optimise the activities in order to reduce the overall level of effort required in the future to manage a sustainable infrastructure
improving tools needed to support operations
monitoring automation of alarms
develop a full set of Service Level Agreements (SLA) at several levels, associated with mechanisms to monitor and verify these SLAs
JRA2- G. Zaquine - EGEE-II Final EU Review July 2008

12. Security Management
Best practices have been considered and described in MJRA “Security audit, strategy and plan”, and progressively implemented
Security audits have been performed on aspects of Grid operations and middleware by various groups within the project as well as external third parties (BARC/EUIndiaGrid and CNES)
Security Service Challenges (SSC) is an ongoing process on many grid sites, running in its 3rd phase
The goal of theses challenges is to test the incident of our security tools, policies and procedures and identify where improvements can be made
Procedures generally understood
Proposed enhancements for Access control (ex: blocking a user at the authZ level) and tractability - Logging and audit trails (logs)
Review of gLite authorization (MJRA1.7)
Framework re-design
In addition, the OSG is part of the overall work as partner in the security groups, with exchange of experiences from similar efforts within the US
This will be expended to other projects for the future
JRA2- G. Zaquine - EGEE-II Final EU Review July 2008

13. JRA2 Quality Assurance
Metrics programme Provide a powerful tool for assessment at each level : Project overall metrics Activity metrics Partner metrics
JRA2- G. Zaquine - EGEE-II Final EU Review July 2008

14. Overall main metrics
The project identified overall metrics, with “target” values defined in the Technical Annex
The project has exceeded all targets set for the project
JRA2- G. Zaquine - EGEE-II Final EU Review July 2008

15. Activity metrics
A full set of activity metrics have been provided by each activity. These are grouped into themes, namely: project management, training, dissemination, usage, operation, middleware
JRA2- G. Zaquine - EGEE-II Final EU Review July 2008

16. SA1 metrics … as part of everyday monitoring tool
The project has implemented the metrics defined in MSA1.1
More than 40 metrics including timeseries data as well as snapshot data providing more finely-grained data
Size metrics (CPU, Storage, Users)
Operation metrics (Site availability, Site reliability)
Usage metrics (Jobs, Data transfer, CPU, Storage)
User support tickets (Numbers, Types, Response times)
Services metrics (CE, SE, SRM, BDII, RB, LFC, FTS, VOMS, etc)
The site availability and reliability metrics calculated by SAM & GridView are used for sites’ SLA conformance
JRA2- G. Zaquine - EGEE-II Final EU Review July 2008

17. Availability of a Site / Services
Site availability is calculated as:
logical AND of all service types
If a site has several instances of a service type (e.g. Computing Elements), the service is deemed to be available if any of the instances are available (logical OR)
.....
█ D  = Down (unscheduled) █ NA = No test result Available █ M = Maintenance (scheduled) █ U = Up
JRA2- G. Zaquine - EGEE-II Final EU Review July 2008

18. From Partner Activity Review to Federation & Country review
Given the size of the EGEE-II (more than 90 partners), the partners’ metrics programme was devised to ensure that all partners are performing as expected
The Partner Activity Reviews have been organized with a format depending on the nature of each activity
The Activity reviews went well. The Activity Managers found these reviews useful. It was a forum to discuss issues, examine progress & contributions and establish better communication between partners
This is being expanded to “federation reviews” in EGEE-III
Monitor the progress of the programme of work in each country and the involvement of individual partners
Examine progress and contribution and identify and promote regional successes and initiatives for re-use in other countries
Verify the state of readiness for the transition to EGI
JRA2- G. Zaquine - EGEE-II Final EU Review July 2008

19. JRA2 Quality Assurance
Industry QA standards follow-up “Evaluate best practices that could be progressively applied to EGEE” - ITIL group - Relationships with ETICS
JRA2- G. Zaquine - EGEE-II Final EU Review July 2008

20. Following up industry standards and best practices (1/2)
The EGEE ITIL group has been set up with the goal to assess ITIL best practices
Trainings “Introduction to ITIL” and dedicated presentations at EGEE conferences
ITIL Services Level Management (SLA modeled on ITIL recommendations)
ROC / Site
HARDWARE AND CONNECTIVITY CRITERIA
DESCRIPTION OF SERVICES COVERED
SERVICE HOURS
AVAILABILITY
SUPPORT
SERVICE REPORTING AND REVIEWING
GGUS/ROCs for users support service
ITIL Service Desk and Incident Management
Evaluation for EGEE Networks Operations Centre (ENOC)
Importance of a common terminology e.g. Incident, Problem, Known error
Evaluation vs GGUS
Out of 103 ITIL items analysed from the ITIL “Incident Management”, most are fulfilled by the present arrangements of GGUS
Improvement of Interface between GGUS central part and regional parts in order to improve Problem Management
Enforcement of Incident Manager
ITIL will be useful at EGI stage as a set of best practice and common terminology
The QAG will pursue the work on ITIL based on ITILV 3 within EGEE-III
ITIL 3
JRA2- G. Zaquine - EGEE-II Final EU Review July 2008

21. Following up industry standards and best practices (2/2)
Relationships with ETICS have continued
JRA2 has followed the work of the ETICS QA software certification model based on ISO and CMMi standards, including:
automated evaluation modules for Static analysis
Coding style
Structural testing
Functional testing
Standards compliance
For EGEE-III, the QAG will encourage ISO and CMMi best practices toward Software Engineering part
JRA2- G. Zaquine - EGEE-II Final EU Review July 2008

22. Summary
The QA objectives planned for the project have been achieved, and QA is in active use across all activities
The quality of the service of the EGEE infrastructure has been continuously increased, with clarified work procedures, as well as the improvement of various automated tools
The project has exceeded all targets set for the second year of the project
In the list of measures for continuous quality improvement foreseen within EGEE-III
Quality Assurance will be pervasive throughout all EGEE-III activities and each activity will plan for their QA measures as part of the activity tasks and milestones
Further implement the Services Level Management process with a full set of Service Level Agreements (SLA) at several levels, associated with mechanisms to monitor and verify these SLAs
The acceptance software component criteria will be enforced at each stage
Security auditing to cooperate more with other projects
To monitor the progress of the programme of work in each country and the involvement of individual partners, a federation reviews process will be used systematically across the whole project
The QAG group will also pursue the work on ITIL based on ITIL 3, and the work with ETICS concerning Software Engineering, encouraging ISO and CMMi best practices
JRA2- G. Zaquine - EGEE-II Final EU Review July 2008