1. EMI INFSO-RI-261611 EMI Quality Assurance Tools Lorenzo Dini (CERN) SA2.4 Task Leader

2. EMI INFSO-RI-261611 Task Goals Initial situation Build, Integration, Packaging Build and Test Infrastructure Software Repositories Continuous Testing Quality Assurance – Metrics Workflow – QA Reports 12/04/2011 EGI User Forum - EMI Technical Forum - April 11-14 Vilnius 2 Outline

3. EMI INFSO-RI-261611 EMI QA tools not only for Quality Assurance Single tool-chain and workflow for – Build, integration, Packaging, Testing, QA Unified project builds in a single box Single release tool Single dependency management Uniform and compliant packaging Tests stored together with configuration QA metrics created in builds 12/04/2011 EGI User Forum - EMI Technical Forum - April 11-14 Vilnius 3 SA2.4 - EMI QA Tools

4. EMI INFSO-RI-261611 LanguagesRelease Integration Dependency Management PackagingTestingBug Tracker 12/04/2011 EGI User Forum - EMI Technical Forum - April 11-14 Vilnius 4 Initial Situation

5. EMI INFSO-RI-261611 12/04/2011 EGI User Forum - EMI Technical Forum - April 11-14 Vilnius 5 Survey Results 1

6. EMI INFSO-RI-261611 12/04/2011 EGI User Forum - EMI Technical Forum - April 11-14 Vilnius 6 Survey Results 2

7. EMI INFSO-RI-261611 12/04/2011 EGI User Forum - EMI Technical Forum - April 11-14 Vilnius 7 Survey Results 3

8. EMI INFSO-RI-261611 Different cultures, environments, requirements and priorities led to different tools difficult to merge Uniform the build environment to provide a common software engineering workflow without affect developer efficiency Provide a comprehensive infrastructure able to satisfy the totality of needs Introduce testing, QA metrics generation and reporting in the same tool-chain Support everybody on a new system 12/04/2011 EGI User Forum - EMI Technical Forum - April 11-14 Vilnius 8 Challenges

9. EMI INFSO-RI-261611 Taken the versatility of the gLite ETICS system as a general framework Adopted the ARC dependency management with EPEL and packaging with Mock (Debian with PBuilder) Configured minimal worker nodes as dCache with packages installed as privileged user at build start and build in user space Kept Maven to uniformly and effectively build Java components as UNICORE 12/04/2011 EGI User Forum - EMI Technical Forum - April 11-14 Vilnius 9 Build, Integration and Packaging

10. EMI INFSO-RI-261611 Every EMI software component is configured in a single ETICS project and organized in releases The whole release is built from source in a single minimal clean node 4 times a day ETICS checks-out the source code, installs the dependencies using YUM/APT and triggers the build systems (Autotools, Ant, Maven) in user space ETICS creates missing SPEC files and produces RPMs and DEBs ETICS triggers Mock/PBuilder in a pristine environment for the packaging 12/04/2011 EGI User Forum - EMI Technical Forum - April 11-14 Vilnius 10 Build, Integration and Packaging

11. EMI INFSO-RI-261611 12/04/2011 EGI User Forum - EMI Technical Forum - April 11-14 Vilnius 11 Build, Integration and Packaging

12. EMI INFSO-RI-261611 12/04/2011 EGI User Forum - EMI Technical Forum - April 11-14 Vilnius 12 Build, Integration and Packaging

13. EMI INFSO-RI-261611 Virtual infrastructure based on CERN Microsoft HyperV and Condor >50 VMs in >10HVs available at any time for builds and tests VMs are automatically scratched after jobs to ensure clean environments Some high performance nodes have been dedicated to project builds to minimize the build cycle 12/04/2011 EGI User Forum - EMI Technical Forum - April 11-14 Vilnius 13 Build and Test Infrastructure

14. EMI INFSO-RI-261611 12/04/2011 EGI User Forum - EMI Technical Forum - April 11-14 Vilnius 14 Build and Test Infrastructure

15. EMI INFSO-RI-261611 EMI software repositories are provided for each release and release candidate These repositories are automatically populated after builds An EMI external repository is also provided for packages not available in the OS (SL, EPEL, Debian) Automatic ETICS repositories are also generated with the contents of the builds 12/04/2011 EGI User Forum - EMI Technical Forum - April 11-14 Vilnius 15 Software Repositories

16. EMI INFSO-RI-261611 12/04/2011 EGI User Forum - EMI Technical Forum - April 11-14 Vilnius 16 Continuous Testing “The gLite Data Management Continuous Integration and Testing Process” – 14:30

17. EMI INFSO-RI-261611 12/04/2011 EGI User Forum - EMI Technical Forum - April 11-14 Vilnius 17 Quality Assurance The goal of Process Metrics is to identify improvement opportunities in the way the software is developed and tested, from a process point of view. In our context the main sources of process metrics are the Defect Tracking tools used in EMI. Product (or Internal) Metrics measure certain characteristics of the product like complexity, changeability and testability. At the moment we rely on static code analyzers to measure these characteristics. ”Software Metrics Defines, Reports and Analysis in EMI” – 12:00

18. EMI INFSO-RI-261611 QA Metrics Workflow http://gridka- school.scc.kit.edu/img/dCache_logo.gif XSLT XSD

19. EMI INFSO-RI-261611 12/04/2011 EGI User Forum - EMI Technical Forum - April 11-14 Vilnius 19 Bug Tracker exporters “Software Quality Assurance in EMI” – 11:00 XSLT XSD The middleware distributions decided not to merge their tracking systems Agreements on fields, states, workflow and interfaces to track the process Exporters query each system and provide a standard XML document with the data This documents are stored and backed up on servers provided by distributions

20. EMI INFSO-RI-261611 12/04/2011 EGI User Forum - EMI Technical Forum - April 11-14 Vilnius 20 QA Plug-ins QA Plug-ins are ETICS modules that run during the build and trigger common static analysis tools for QA SLOCCount, FindBugs, PMD, Checkstyle, PyLint, RPMLint, CPPCheck, etc... Product QA Metrics are produced and stored permanently in the ETICS repository

21. EMI INFSO-RI-261611 12/04/2011 EGI User Forum - EMI Technical Forum - April 11-14 Vilnius 21 QA Report Generator Queries the XML for the process metrics and the ETICS repository for the product metrics creating aggregated metrics. Generates charts, plots and tables with QA information and fills in QA templates with the dynamic information. Reports can be produced daily for Product Teams and Release Manager Produces drafts for EU QA Deliverables

22. EMI INFSO-RI-261611 12/04/2011 EGI User Forum - EMI Technical Forum - April 11-14 Vilnius 22 QA Reports: Process Metrics

23. EMI INFSO-RI-261611 12/04/2011 EGI User Forum - EMI Technical Forum - April 11-14 Vilnius 23 QA Reports: Product Metrics

24. EMI INFSO-RI-261611 A very heterogeneous starting point Uniform software engineering tool-chain Successfully integrated EMI 0 and EMI 1 New integrated testing and QA process All performed improvements benefit all teams at once. Specific solution can make a single team happier but are globally more expensive and not maintainable. 12/04/2011 EGI User Forum - EMI Technical Forum - April 11-14 Vilnius 24 Conclusions

25. EMI INFSO-RI-261611 Thank you 12/04/201125 EGI User Forum - EMI Technical Forum - April 11-14 Vilnius EMI is partially funded by the European Commission under Grant Agreement INFSO-RI-261611