1. EMI INFSO-RI-261611 SA2 - Quality Assurance Alberto Aimar (CERN) SA2 Leader EMI First EC Review 22 June 2011, Brussels

2. EMI INFSO-RI-261611 Outline Objectives Achievements KPIs Lessons Learned Outlook 1st EMI Periodic Review 22/06/20112

3. EMI INFSO-RI-261611 SA2 in EMI 3 Quality Assurance Process Policies Metrics Reviews Tools Infrastructure Support SA2 1st EMI Periodic Review Requirements Technical Plans Development and Test Plans Development Testing Certification Validation Release Maintenance Support NA1, JRA1 JRA1 SA1 SA2 Software & Services Requirements Defines ImplementsCertifies Release Candidate Process definition Process monitoring NA1, NA2 22/06/2011

4. EMI INFSO-RI-261611 SA2 QA Objectives Processes and Metrics Establish common software quality assurance process and metrics Tools and Testbeds Provide tools and resources for building and testing Support a continuous integration and testing infrastructure Monitoring and Review Monitor metrics trends, reviewing and improving software quality Customers QA Criteria Allow passing customer acceptance criteria User Support on QA Support and consultancy within the project 4 Define and establish a common software quality assurance process and metrics for all engineering activities. Enable a continuous integration and testing process by selecting and maintaining tools and resources for building and testing software either within the project of in collaboration with external resource providers Allow the EMI middleware to consistently pass the customer acceptance criteria and continually improve the software quality and the process itself by monitoring the metrics value trends, reviewing quality control activities and related tests, providing support and consultancy in QA matters. [DoW] Define and establish a common software quality assurance process and metrics for all engineering activities. Enable a continuous integration and testing process by selecting and maintaining tools and resources for building and testing software either within the project of in collaboration with external resource providers Allow the EMI middleware to consistently pass the customer acceptance criteria and continually improve the software quality and the process itself by monitoring the metrics value trends, reviewing quality control activities and related tests, providing support and consultancy in QA matters. [DoW]

5. EMI INFSO-RI-261611 SA2 QA Strategy EMI is merging 4 established Middleware projects, each with its own QA practices Work with a 3-years vision Y1: explain, define and implement Y2: review and automate Y3: consolidate and optimize Use established QA standards ISO /IEC 9126-1-4 and 25000 Leverage on existing QA practices Use existing QA tools, resources and expertise 1st EMI Periodic Review 522/06/2011

6. EMI INFSO-RI-261611 SA2 Management Strategy Strategy applied for all SA2 activities and tasks Involve PTs, SA1, JRA1 - Merging 4 products, tools, platforms, etc is a challenge, we involved all teams. Provide early solutions - We wanted to provide policies, tools, repositories, etc because teams were waiting for them Explain QA constraints - Make aware all developers and teams that QA will requires changes for converging on common solutions Be transparent – Work closely with other activities (SA1, JRA1, EMI EMT Product teams, EGI QA, PO), all SA2 material public Aim at automated solutions – They provide exponential benefits compared to any specific or manual solutions Support tools and testbed – Not just policies and rules to follow, but also give useful infrastructure and services to the developers 1st EMI Periodic Review 622/06/2011

7. EMI INFSO-RI-261611 SA2 Information 1st EMI Periodic Review 7 https://cern.ch/twiki/bin/view/EMI/SA2 (bit.ly/emisa2) 22/06/2011

8. EMI INFSO-RI-261611 Objective: Common QA Process Software Quality Assurance Plan defined early SQA factors, SQA management Roles and procedures Standard Practices and conventions Detailed QA Policies Release and Change Management Configuration and Integration Packaging and Distribution Testing and Certification Documentation 8 Requirements Technical Plans Development and Test Plans Development Testing Certification Validation Release Maintenance Support SA2 Define and establish a common software quality assurance process and metrics for all engineering activities [DoW] SO 1.4 22/06/2011 1st EMI Periodic Review EMI QA policies are used by all product teams to achieve a uniform process and presentation of EMI software, documentation, etc

9. EMI INFSO-RI-261611 Some QA Policies 22/06/2011 1st EMI Periodic Review 9

10. EMI INFSO-RI-261611 Objective: Common QA Metrics Selected Metrics to objectively qualify status of software Automated numerical metrics Cover different requirements of EMT, SA1, PT, SA2 Provide reports, trends and dashboards Generate a wide range of metrics to cover several QA aspects Metrics on code, process, documentation, support Internal and external metrics (code and process) Language dependent (Java, C++, Python), object-oriented Examples of metrics Reaction time to critical bugs, delays in releases SLOC, code complexity, bug density, test coverage 22/06/2011 1st EMI Periodic Review 10 Continually improve the software quality and the process itself by monitoring the metrics value trends [DoW] SO 1.4 Automated tools to collect the same QA metrics across all EMI products. QA data generated to be analysed & key metrics selected

11. EMI INFSO-RI-261611 11 Objective: Provide EMI QA Tools Initial situation: All different tools, dev environments, trackers, etc Final Goals: Move towards a single tool set, workflow and data Approach Followed Define common software engineering workflow - Uniform build, testing, packaging trying not to affect the developers’ efficiency Provide build and test infrastructure - Provide a comprehensive infrastructure able to support the policies (build, packaging, releasing, etc). This infrastructure is ETICS. Introduce progressively testing, QA metrics – provide automated test execution, metrics generation and reporting Focus on Support - Support integrated in GGUS, 50% of users were new to the tools 22/06/2011 Enable a continuous integration and testing process selecting and maintaining tools and resources for building and testing software [DoW] SO 1.5 1st EMI Periodic Review All EMI software is built, packaged and distributed with the same tools and is all following same policies All EMI software is built, packaged and distributed with the same tools and is all following same policies

12. EMI INFSO-RI-261611 MiddlewareLanguagesRelease Integration Dependency Management PackagingTestingTrackers Tools - Initial Situation 22/06/2011 1st EMI Periodic Review 12

13. EMI INFSO-RI-261611 MiddlewareLanguagesRelease Integration Dependency Management PackagingTestingTrackers Tools - Current Status 22/06/2011 1st EMI Periodic Review 13 XML Exchange Format

14. EMI INFSO-RI-261611 Objective: Common QA Testbed Created the EMI Testbed for integration and testing process Single Testing Infrastructure within Quality Assurance Product Team – centric model for their individual certification Support Internal Inter-Component Testing Focus on interaction among different clients and components provided by other Product Teams and for release candidates verifications Defined Large Scale Testing infrastructure Scalability and interoperability testing on real environment Involved external customers, resources and their verifications (EGI QA) Installed all EMI Services both production and release candidates Agreed participation of EGI sites to large scale testing 14 Enable a continuous integration and testing process either within the project or in collaboration with external resource providers [DoW] SO 1.4 22/06/2011 1st EMI Periodic Review All EMI Software (~90 installations) production and release candidates on EMI testbed (7 sites) for testing & certification

15. EMI INFSO-RI-261611 SA2 – Year 1 22/06/2011 1st EMI Periodic Review 15 Quality Assurance Process Policies Metrics Reviews Tools Infrastructure Support SA2 Requirements Technical Plans Development and Test Plans Development Testing Certification Validation Release Maintenance Support NA1, JRA1 JRA1 SA1 SA2 Software & Services Requirements Defines ImplementsCertifies Release Candidate Process definition Process monitoring In Y1 SA2 established the foundations of a measurable QA process based on common policies, tools and infrastructure

16. EMI INFSO-RI-261611 Outline Objectives Achievements KPIs Lessons Learned Outlook 1st EMI Periodic Review 22/06/201116

17. EMI INFSO-RI-261611 SA2 KPIs 1st EMI Periodic Review 17 KPIDescriptionTargetQ1Q2Q3Q4P1 KSA2.2 - Services Availability ETICS 97% -99.7 %95.6 %98.6 %98.0 % Testbed 97% -97.5 %98.7 %98.8 %98.3 % KSA2.3 - Testbed Size 50 CPUs-70609073 KSA2.6 - No of Requests ETICS --39277547 Testbed ---82215 KSA2.7 - Average Response Time ETICS --8.2 h5.2 h5.9 h6.4 h Testbed ---6.0 h4.3 h5.2 h KSA2.8 - Average Solution Time ETICS --17.4 h10.8 h37.6 h21.9 h Testbed ---11.2 h29.5 h20.3 h 22/06/2011

18. EMI INFSO-RI-261611 Lessons Learned Merging QA for four products all with different processes, tools is really complicated, technically and humanly Strong support of all partners was crucial for adoption Combine top-down QA constraints & bottom-up needs Involve teams experts and whoever want to contribute. Constantly explain the goods of shared common solutions Start with simple metrics, same for all, explain and discuss details. Select metrics suitable for different audiences Provide early tools and testbed infrastructure with useful services in order to support the policies and the process Periodically review and adapt keeping in mind the overall QA goals of the project and developers needs 1st EMI Periodic Review 1822/06/2011

19. EMI INFSO-RI-261611 SA2 in EMI Year 2 Main QA Focus for Year 2 (in addition to Y1 activities) Review and adapt QA policies, tools, testbed after Year 1 Select Key Metrics and work with PTs to improve on them Distribute standard QA reports automatically generated Provide SL6 and Debian 6 new platforms on ETICS, Testbed Improve on Unit Testing PTs should add tests in ETICS Add Tests on testbed PTs provide automated tests Progress with Large Scale Testbed with volunteer EGI Sites Management Approach for Year 2 Follow the QA strategy with same approach used in Y1 Worked very well. Y2: review and automate. Involve PTs in QA matters in order to have their support 1st EMI Periodic Review 1922/06/2011

20. EMI INFSO-RI-261611 1st EMI Periodic Review 20 Thank you EMI is partially funded by the European Commission under Grant Agreement INFSO-RI-261611 22/06/2011