LCG-SPI: SW-Testing LCG AppArea internal review (20/10/03) Manuel.Gallas@cern.ch LCG/SPI LCG Software Process & Infrastructure M Gallas CERN EP-SFT LCG-SPI: SW-Testing

Outline SPI SW-Testing SW-Testing Overview LCG-SPI SW-Testing Test Frameworks Test Documents User support M Gallas CERN EP-SFT LCG-SPI: SW-Testing

SW-Testing Overview GOAL (from RTAG2) SPI SW-Testing Software testing Software testing should be an integral part of the software development in the LCG App Area. All level of software testing should be run as part of an automatic process. Code Documentation Software developer CODE Work Package Test Automated testing Nightly Building System Test FrameWorks SW-testing doc SW- testing support SPI SW-Testing SW-testing polices Sw-testing team Integration Tests System Examples LHC experiments Use in Exp. Software testing Integration test System test Acceptance test Unit test M Gallas CERN EP-SFT LCG-SPI: SW-Testing

LCG-SPI SW-Testing Aim: to help developers: to help projects: to produce test code suitable to be run in automatic way to document their tests to plan their test phase to run all their test in automatic way to browse and scan easily the test results and failures Requirements from the users : Work with different languages (C++, Python, …) and in different environments (platforms & compilers) Allow two ways of testing: (1) “check the output of the test-code” (2) “check the test results inside the test-code”. Provide a easy way to integrate existing tests. Allow the creation of dependencies among tests. Easily establish a criteria if a test should pass or fail in a given release Organize the tests by components or packages / integration / system /acceptance . Regression testing and check respect to the previous sw-release. Easy integration with the Nightly Building System. Provide also a graphical interface to run the tests and examine the test results. Our inputs: Contacts within HEP-community. What is available as free open source code. Our constraints: Avoid commercial software and licensing problems. Avoid “do it yourself solutions” Try to adopt commonly used open-source software. M Gallas CERN EP-SFT LCG-SPI: SW-Testing

Test Frameworks: Global picture QA activity End User Installation checks Nightly Building Project Release Sw-Testing Top layer Bottom layer - Adaptable to the programming language and developer - Prepared to be run in automatic way - Integrates different ways to test. - Common environment to run the tests and to access the test results. QMTest Uses a GUI for creating and running tests (also in batch). Can run tests in parallel, supports execution of a single test or many at once (test-cases & test-suites) Organizes tests hierarchically Records dependencies among tests X-Unit family JUnit QtUnit CppUnit PyUnit Oval Test Scripts Old tests SW Product Examples Unit testing Acceptance testing M Gallas CERN EP-SFT LCG-SPI: SW-Testing

Test Frameworks: X-Unit family JUnit QtUnit CppUnit PyUnit A simple test: Subclass the TestCase CppUnit or PyUnit class Override the method runTest(). When you want to check a value, call the (CppUnit or PyUnit) ASSERT(bool) and pass in an expresion that is true if the test succeeds Main Scope: Unit–testing The same “assertion style” in different languages. Provides:- The name of the test case that failed. -The name of the source file that contains the test. -The line number where the failure occurred Different platforms/compilers: (Linux/Solaris/Windows) M Gallas CERN EP-SFT LCG-SPI: SW-Testing

Test Frameworks: Oval Test Source Executable Code Ovalfile Oval diff validation and regression developed & used within CMS experiment Ovalfile (configuration) Executable Test Source Code Reference file Log Oval run Oval diff Oval build Can be use for Unit-test to Validation tests. Compare the output log file with a given reference file (Smart comparison of those lines which start with “[Oval]”) It is possible to set different run environments. Can run external scripts and external binaries. M Gallas CERN EP-SFT LCG-SPI: SW-Testing

Test Frameworks: QMTest Uses a graphical interface for creating and running tests The configuration files are in XML and can be created from the GUI. We provide also script to do it Runs tests in parallel Organizes tests hierarchically Supports execution of a single test or many at once Records dependencies among tests Can be run in batch mode -> easy integration with the Nightly-Building systems M Gallas CERN EP-SFT LCG-SPI: SW-Testing

Test Documents For all project is needed a Test Plan in order to have a clear idea of the type of tests, schedule, environment and responsibilities. We use a template Test Case template collects information about the existing tests and how they cover the expected functionality of the software. M Gallas CERN EP-SFT LCG-SPI: SW-Testing

User support: Web information http://spi.cern.ch/ http://spi.cern.ch/testing Test Frameworks HowTo Test doc Policies M Gallas CERN EP-SFT LCG-SPI: SW-Testing

Automatic testing requires User support: How-To * * * * 1 2 3 Follow sw-testing policies Automatic testing requires some standardization * M Gallas CERN EP-SFT LCG-SPI: SW-Testing

Final Remarks SPI SW-Testing Software testing is a needed step in the software development process. SPI provides test frameworks, templates and help to have a coherent automatic testing procedure with a common interface to run and browse the test results. The proposed testing tools were chosen following the criteria of a simple approach, tools in use in the experiments and freely available. SW-Testing Overview LCG-SPI SW-Testing Test Frameworks Test Documents User support SPI SW-Testing Following standards makes life easier to SPI and the users. Automatic testing is always needed and is a way to consolidate the software. Bug reports and typical use-cases from the users should be converted into tests SPI SW-testing has to evolve and consolidate following the needs of the LCG projects. All SPI sw-testing tools, templates, documentation can be used by any project. Thanks to: LCG-PI team LCG-POOL team LCG-SEAL team LCH-experiments M Gallas CERN EP-SFT LCG-SPI: SW-Testing