1. Quality Assurance and Testing in LCG CHEP 2004 Interlaken, Switzerland 30 September 2004 Manuel Gallas, Jakub MOSCICKI CERN http://spi.cern.ch mgallas@mail.cern.ch jakub.moscicki@cern.ch

2. M.Gallas, J.Moscicki CERN PH-SFT SPI - Software Process & Infrastructure 2 Outline: Quality Assurance and Testing in LCG LCG SPI Context SPI Services Overview Sw-Testing: Test Frameworks Test Documents User support Quality Assurance: QA activities Bug tracker statistics Test coverage LCG App.Area policies Summary

3. M.Gallas, J.Moscicki CERN PH-SFT SPI - Software Process & Infrastructure 3 LCG SPI context LCG Infrastructure Common services Similar ways of working (process) Tools, templates, training General QA, tests, integration, release LCG Application Area software projects POOL: Persistency SEAL: Core common software PI: Physics Interfaces SIMU: Simulation …etc… LHC grid software applications (LHC experiments, projects, etc) LCG SPI project LCG Application Area SIMULATIO N

4. M.Gallas, J.Moscicki CERN PH-SFT SPI - Software Process & Infrastructure 4 SPI Services Overview Provide General Services needed by each project CVS repository, Web Site, Software Library Mailing Lists, Bug Reports, Task Management, Collaborative Facilities Provide solutions specific to the Software Development phases Tools, Templates, Policies, Support, Documentation, Examples Software Testing and QA CodingAnalysis and Design Development Release Specifications Testing Build systems Deployment and Installation Documentation Quality Assurance Software Development General Services CVS service Collaborative FacilitiesTask ManagementMailing Lists Web Portal External Software SPI project  The rest of the talk describes QA and Sw-Testing services SPI Web Site - http://spi.cern.ch

5. M.Gallas, J.Moscicki CERN PH-SFT SPI - Software Process & Infrastructure 5 Software testing should be an integral part of the software development in the LCG App Area. All level of software testing should be run as part of an automatic process. SW-Testing Overview - http://spi.cern.ch/testing Software testing Integration test System test Acceptance test Unit test Sw-testing team Integration Tests System Tests Examples LHC experiments Use in Exp. Test FrameWorks SW-testing doc SW- testing support SPI SW-Testing SW-testing polices Code Documentation Software developer CODE Work Package Test Automated testing Nightly Building System

6. M.Gallas, J.Moscicki CERN PH-SFT SPI - Software Process & Infrastructure 6 LCG-SPI SW-Testing Aim: to help developers: to help projects: to produce test code suitable to be run in automatic way to document their tests to plan their test phase to run all their test in automatic way to browse and scan easily the test results and failures Our inputs: Contacts within HEP-community. What is available as free open source code. Our constraints: Avoid commercial software and licensing problems. Avoid “do it yourself solutions” Try to adopt commonly used open-source software. Requirements from the users : Work with different languages (C++, Python, …) and in different environments (platforms & compilers) Allow two ways of testing: (1) “check the output of the test-code” (2) “check the test results inside the test-code”. Provide a easy way to integrate existing tests. Allow the creation of dependencies among tests. Easily establish a criteria if a test should pass or fail in a given release Organize the tests by components or packages / integration / system /acceptance. Regression testing and check respect to the previous sw-release. Easy integration with the Nightly Building System. Provide also a graphical interface to run the tests and examine the test results.

7. M.Gallas, J.Moscicki CERN PH-SFT SPI - Software Process & Infrastructure 7 Test Frameworks: Global picture X-Unit family JUnitQtUnit CppUnitPyUnit Oval Test Scripts Old tests SW Product Examples Unit testing Acceptance testing QA activity End User Installation checks Nightly Building Project Release Sw-Testing QMTest Uses a GUI for creating and running tests (also in batch). Can run tests in parallel, supports execution of a single test or many at once (test-cases & test-suites) Organizes tests hierarchically Records dependencies among tests Top layer Bottom layer - Adaptable to the programming language and developer - Prepared to be run in automatic way - Integrates different ways to test. - Common environment to run the tests and to access the test results.

8. M.Gallas, J.Moscicki CERN PH-SFT SPI - Software Process & Infrastructure 8 Main Scope: Unit–testing The same “assertion style” in different languages. Provides:- The name of the test case that failed. -The name of the source file that contains the test. -The line number where the failure occurred Different platforms/compilers: (Linux/Solaris/Windows) Test Frameworks: X-Unit family X-Unit family JUnit QtUnit CppUnit PyUnit A simple test: 1.Subclass the TestCase CppUnit or PyUnit class 2.Override the method runTest(). 3.When you want to check a value, call the (CppUnit or PyUnit) ASSERT(bool) and pass in an expresion that is true if the test succeeds

9. M.Gallas, J.Moscicki CERN PH-SFT SPI - Software Process & Infrastructure 9 Test Frameworks: Oval Oval: validation and regression developed & used within CMS experiment Can be use for Unit-test to Validation tests. Compare the output log file with a given reference file (Smart comparison of those lines which start with “[Oval]”) It is possible to set different run environments. Can run external scripts and external binaries. Ovalfile (configuration) Executable Test Source Code Reference file Log file Oval run Oval diff Oval build

10. M.Gallas, J.Moscicki CERN PH-SFT SPI - Software Process & Infrastructure 10 Test Frameworks: QMTest QMTest Uses a graphical interface for creating and running tests The configuration files are in XML and can be created from the GUI. We provide also script to do it Runs tests in parallel Organizes tests hierarchically Supports execution of a single test or many at once Records dependencies among tests Can be run in batch mode -> easy integration with the Nightly-Building systems

11. M.Gallas, J.Moscicki CERN PH-SFT SPI - Software Process & Infrastructure 11 Test Documents For all project is needed a Test Plan in order to have a clear idea of the type of tests, schedule, environment and responsibilities. We use a templatetemplate Test Case template collects information about the existing tests and how they cover the expected functionality of the software.

12. M.Gallas, J.Moscicki CERN PH-SFT SPI - Software Process & Infrastructure 12 User support: Web information httphttp://spi.cern.ch/ Test Frameworks HowTo Test doc Policies httphttp://spi.cern.ch/testing

13. M.Gallas, J.Moscicki CERN PH-SFT SPI - Software Process & Infrastructure 13 User support: How-To * * * * * 3 Follow sw-testing policies Automatic testing requires some standardization 1 2

14. M.Gallas, J.Moscicki CERN PH-SFT SPI - Software Process & Infrastructure 14 Quality Assurance Service - http://spi.cern.ch/qa The main goal of QA activity is help LCG projects assess and improve the quality of the software provide tools to collect useful metrics/statistics which help to asses quality; generate reports; verify if project setup is correct with LCG policies. QA Tools and Focus Automatic reports Development/integration of automatic tools LCG Policies agreed and defined by AF SPI supports them in the tools and procedures and only helps to work them out

15. M.Gallas, J.Moscicki CERN PH-SFT SPI - Software Process & Infrastructure 15 Quality Assurance Activities QA Checklist on each Release Build the release Run automatic tests Metrics Test Inventory & Metrics Documentation/Examples Inventory Bug tracker Statistics Code Inventory Rule Checker, Logiscope LCG Policies Configuration of a build system CVS directory structure Software testing polices Automatic QA Reports Generated at every release Published on the SPI web site External tools Test coverage SLOC, Valgrind, ignominy QA procedure: well-defined, transparent, open -clear rules and checklist of assessed items -anybody at anytime may see statistics -anybody may create reports themselves -anybody may contribute

16. M.Gallas, J.Moscicki CERN PH-SFT SPI - Software Process & Infrastructure 16 QA Reports Bug tracker statistics Testing inventory by package Results of the automatic testing Test coverage Source code statistic (n files, n lines, sloc,.. ) CVS structure

17. M.Gallas, J.Moscicki CERN PH-SFT SPI - Software Process & Infrastructure 17 QA reports: Bugtracker Statistics Defect Analysis in Time  Defect Reparation Trends  Activity of the project / users  Flexible category selection subsystems, severity,...  Caveat: -not all bug reports are “real” bugs Correlation of defects and tests  Traceability oEvery bug fix should be accompanied with a test case oTest case should contain a bug reference #  Avoiding reappearing bugs

18. M.Gallas, J.Moscicki CERN PH-SFT SPI - Software Process & Infrastructure 18 QA reports: Test Coverage Coverage of Unit tests Integration tests Validation tests At the level of Project Package File Source Lines Tools gcc3.2 –gcov LTP scripts

19. M.Gallas, J.Moscicki CERN PH-SFT SPI - Software Process & Infrastructure 19 LCG Policies CVS and Build Directory Policy Software Testing Policies Version Numbers, Tagging and Release Procedure Installation Directory Structure Platform string, binary names, debug flags and more They are a needed by the LCG They are defined by the LCG projects (AF), collected by SPI SPI supports them in the tools and procedures and only helps to work them out If everything is different is too difficult to use and to automate compromising on our habits, for project needs tell when they are not followed First time that this exists at this extent, and that is checked QA tools verify the compliance of software to the Policies

20. M.Gallas, J.Moscicki CERN PH-SFT SPI - Software Process & Infrastructure 20 QA activities designed to match the HEP environment -focused more on testing and bug tracking -less on sophisticated source code metrics integrated and automated into the LCG development environment QA infrastructure will be reused in EGEE Summary Quality Assurance and Testing in LCG LCG SPI Context SPI Services Overview Sw-Testing: Test Frameworks Test Documents User support Quality Assurance: QA activities Bug tracker statistics Test coverage LCG App.Area policies Summary Software testing activities a complete and coherent automatic testing infrastructure was deployed based on existing tools in the open source community. It covers covers from unit to validation tests. a unique entry point (QMTest) can be use to run all the tests at the release time, in the nightly building system, for the QA reports or by the developer.