1. EMI INFSO-RI-261611 EMI Quality Assurance Processes (PS06-4-499) Alberto Aimar (CERN) CERN IT-GT-SL Section Leader EMI SA2 QA Activity Leader

2. EMI INFSO-RI-261611 EMI context and QA objectives Guidelines and Metrics Tools and Testbeds Reports and Reviews Conclusions CHEP 2010, Taipei EMI QA Activities - A.Aimar (CERN) 2 Outline

3. EMI INFSO-RI-261611 EMI Mission Statement CHEP 2010, Taipei EMI QA Activities - A.Aimar (CERN) 3 The European Middleware Initiative (EMI) project represents a close collaboration of the major European middleware providers - ARC, gLite, UNICORE and dCache - to establish a sustainable model to support, harmonise and evolve the grid middleware for deployment in EGI, PRACE and other distributed e-Infrastructures

4. EMI INFSO-RI-261611 EMI Project Structure NA1 - Administrative and Technical Management NA2 – Outreach and Collaborations SA1 - Maintenance and Support JRA1 - Development, Integration and Evolution SA2 - Quality Assurance 4 EMI QA Activities - A.Aimar (CERN) CHEP 2010, Taipei

5. EMI INFSO-RI-261611 CHEP 2010, Taipei DRAFT - EMI QA Activities - A.Aimar (CERN) - DRAFT 5 The Product Teams

6. EMI INFSO-RI-261611 Four large MW projects developing several products – Many development teams > 25 Product Teams Request from the grid infrastructures and the EU funding – Have more uniform releases and interoperable middleware distributions – Provide a common degree of verification & validation – Have metrics and objective criteria to compare product quality and evolution over time No resources to have independent QA efforts – Dev teams focus on development (JRA1) and maintenance (SA1) – No time to set up report on metrics, install tools, maintain testbeds Provide homogeneous packaging, reporting and reviewing of the products – Have limited impact on the way development teams work – Let them still use their build systems, dev tools, bug trackers, etc CHEP 2010, Taipei EMI QA Activities - A.Aimar (CERN) 6 The Challenge

7. EMI INFSO-RI-261611 Quality Assurance Process Definition and Monitoring – Standards-compliant software engineering process. – Continual activity of monitoring its application Metrics Definition and Reporting – Definition, collection and reporting of software quality metrics. – Reports information on the status of the software to take corrective actions QA Implementation Review and Support – Review activities of the QA, test and certification implementations. – Sample review of test plans, compliance, porting guidelines, documentation, etc – Supporting the Product Teams in implementation of tests and use of testing tools Tools and Repositories, Maintenance and Integration – Definition and maintenance of tools required to support QA process – Supporting activity to software providers to integrate external tools – Repositories for the EMI software packages, tests, build and reports Testbeds Setup, Maintenance and Coordination – Setup and maintenance of distributed testbeds for continuous integration testing – Coordination and provision of larger-scale testbeds from collaborating providers CHEP 2010, Taipei EMI QA Activities - A.Aimar (CERN) 7 EMI QA (SA2) Tasks and Objectives

8. EMI INFSO-RI-261611 CHEP 2010, Taipei DRAFT - EMI QA Activities - A.Aimar (CERN) - DRAFT 8 Software Quality Plan

9. EMI INFSO-RI-261611 Software Quality Assurance Plan (SQAP) – Document that specifies the tasks needed to define and measure quality, responsibilities for quality monitoring, documentation required and procedures – Plan that will be followed to manage the QA process SQAP specifies the manner in which the EMI project is going to achieve its software quality goals. – Metrics and measurements are associated in order to evaluate the quality of the EMI software and lifecycle SQA tasks, roles and responsibilities – EMI technical activities (SA1, SA2 and JRA1) – EMI technical bodies (PTB and EMT) – Even of specific individuals/roles in EMI The SQAP covers documentation, reporting and reviewing tasks – Describes the metrics that will be used for the QA reporting and reviews Will be updated regularly, based on experience and needs It is complemented by a set of guidelines (on dev and doc) CHEP 2010, Taipei EMI QA Activities - A.Aimar (CERN) 9 Software Quality Plan

10. EMI INFSO-RI-261611 A set of very practical Guidelines for QA and Software Development is available: – Configuration and Integration – Packaging and Releasing – Change Management (bugs, patches, etc) – Certification and Testing – Metrics Generation Move towards a uniform setup and common definitions and conventions in the project – Releases, patches, versions – Packaging, repositories, distributions – User support, documentation Advantages for e-infrastructures are obvious but it requires some work and little changes by the dev teams in EMI – The guidelines were defined taking the best practices from the 4 middlewares and are mostly agreed by all dev teams CHEP 2010, Taipei EMI QA Activities - A.Aimar (CERN) 10 Development Guidelines

11. EMI INFSO-RI-261611 CHEP 2010, Taipei DRAFT - EMI QA Activities - A.Aimar (CERN) - DRAFT 11 Development Guidelines

12. EMI INFSO-RI-261611 Definition of the Minimum Required Documentation for each software component. – Installation Guide and Troubleshooting guide – User Documentation – Software Requirements Specifications – Software Design Description – Software Verification and Validation Plan and Report General Project-wide Required Documentation – Technical Development Plan – Software Release Plan and Schedule – Software Maintenance and Support Plan – QA Tools Documentation – Continuous Integration and Certification Test beds Documentation – Security Assessments CHEP 2010, Taipei EMI QA Activities - A.Aimar (CERN) 12 Documentation Guidelines

13. EMI INFSO-RI-261611 Metrics are needed to quantify and qualify the status of the software components Use as much as possible numerical metrics – All automated and extracted in the same exact way, by the same tool(s) Starting with a selection of useful and simple metrics – Tools available give a common environment to extract metrics and test – Same metrics for all components, in order to have fair reports Types of SQA metrics – Metrics on code, process, support, documentation – Internal and external metrics (code and process) – Language dependent (Java, C++, Python) Examples of metrics – Reaction time to critical bugs, delays in releases – Complexity, bug density, test coverage We refer to QA standards, but use what is realistically applicable – ISO /IEC 9126-1,-2,-3,-4 and ISO/IEC 25000 CHEP 2010, Taipei EMI QA Activities - A.Aimar (CERN) 13 Quality Metrics

14. EMI INFSO-RI-261611 CHEP 2010, Taipei DRAFT - EMI QA Activities - A.Aimar (CERN) - DRAFT 14 Metrics

15. EMI INFSO-RI-261611 Metric, Testing and Packaging Tools – Compliance and interoperability of the software products – Integration builds of all middleware components – Same identical platforms for all builds, use standard packages on platforms – Automatic deployment and distributed testing of software products Integration of data coming from different sources and tools – Common report of metrics from different static and dynamic software QA tools – Collection of data from several req and bug trackers used by dev teams – Using same tool for packaging and testing tools – Use of an exchange format for different sources Support for repositories and distribution – Common software repository for all EMI middleware – Use the standard RHEL/SL and Debian repositories and formats Generation of QA reports – Metrics extraction, storage and archival – Graphs and reports at all levels of detail – Comparison tables and plot trends CHEP 2010, Taipei EMI QA Activities - A.Aimar (CERN) 15 Tools

16. EMI INFSO-RI-261611 CHEP 2010, Taipei DRAFT - EMI QA Activities - A.Aimar (CERN) - DRAFT 16 Tools

17. EMI INFSO-RI-261611 EMI SA2 provides a distributed testbed – Real hw resources (+ obviously using virtualization) – For integration testing in EMI project – For the product teams testing – Distributed over the sites of the middleware partners Testbed available to Dev teams for testing their software – The teams can easily test their components with what is in production or will soon be (RC) – Production and RC available for all components, servers, etc – Product Team can configure the services needed for its specific tests Provide support for the typical scenarios – Integration testing within a minor release – Integration testing for a major release – Cross middlewares tests across the network – Large-scale tests, real usage scenarios CHEP 2010, Taipei EMI QA Activities - A.Aimar (CERN) 17 Testbeds

18. EMI INFSO-RI-261611 CHEP 2010, Taipei DRAFT - EMI QA Activities - A.Aimar (CERN) - DRAFT 18 Testbeds

19. EMI INFSO-RI-261611 QA reports objectively describe the quality of the product – Only clear metrics are included, e.g. number of bugs/SLOC, reaction time, trends over time, successful/failed releases, etc – Reports both on the product but also on how the team works – Not the goal of the QA activity, a mean to see strengths and weaknesses The same type report for all components – Allows comparisons among components – Trends over time of the same components Fully automated. Dev teams can have their report weekly – See the results and execute corrective actions SA2 reviews of the QA reports and supports the teams – Provides the general reviewing every Month and formally every Quarter – Reports to the EMI mgmt in case of issues – Supports the dev team that need help with metrics, tools, testbeds CHEP 2010, Taipei EMI QA Activities - A.Aimar (CERN) 19 QA Reports and Reviews

20. EMI INFSO-RI-261611 CHEP 2010, Taipei DRAFT - EMI QA Activities - A.Aimar (CERN) - DRAFT 20 QA Reports

21. EMI INFSO-RI-261611 Dedicated QA is a useful activity in large projects, but guidelines, procedures should not overload the developers Challenge of implementing QA in a distributed and heterogeneous environment – Different kind of sw products, different tools, distributed teams, etc Collected experience from the developers in order to define realistic and shared solutions – Never seen a top down or theoretical software approach working...... so we try a (slightly) different one. Define metrics and automate report generation Provide also practical services, not just procedures – to developers: supported and automated tools, testbeds, packaging – to e-infrastructures: verified and homogeneous sw, doc, repositories https://twiki.cern.ch/twiki/bin/view/EMI/SA2 CHEP 2010, Taipei DRAFT - EMI QA Activities - A.Aimar (CERN) - DRAFT 21 Conclusions

22. EMI INFSO-RI-261611 Thank you CHEP 2010, Taipei22 EMI QA Activities - A.Aimar (CERN) EMI is partially funded by the European Commission under Grant Agreement INFSO-RI-261611