1. LCG-SPI: SW-Testing LCG AppArea internal review (20/10/03)
LCG/SPI
LCG Software Process & Infrastructure
M Gallas CERN EP-SFT
LCG-SPI: SW-Testing

2. Outline SPI SW-Testing SW-Testing Overview LCG-SPI SW-Testing
Test Frameworks
Test Documents
User support
M Gallas CERN EP-SFT
LCG-SPI: SW-Testing

3. SW-Testing Overview GOAL (from RTAG2) SPI SW-Testing Software testing
Software testing should be an integral part of the software development in the LCG App Area.
All level of software testing should be run as part of an automatic process.
Code Documentation
Software
developer
CODE
Work Package
Test
Automated testing
Nightly Building
System
Test FrameWorks
SW-testing doc
SW- testing support
SPI SW-Testing
SW-testing polices
Sw-testing
team
Integration
Tests
System
Examples
LHC
experiments
Use in Exp.
Software testing
Integration test
System test
Acceptance test
Unit test
M Gallas CERN EP-SFT
LCG-SPI: SW-Testing

4. LCG-SPI SW-Testing Aim: to help developers:
to help projects:
to produce test code suitable to be run in automatic way
to document their tests
to plan their test phase
to run all their test in automatic way
to browse and scan easily the test results and failures
Requirements from the users :
Work with different languages (C++, Python, …) and in different environments (platforms & compilers)
Allow two ways of testing: (1) “check the output of the test-code”
(2) “check the test results inside the test-code”.
Provide a easy way to integrate existing tests.
Allow the creation of dependencies among tests.
Easily establish a criteria if a test should pass or fail in a given release
Organize the tests by components or packages / integration / system /acceptance .
Regression testing and check respect to the previous sw-release.
Easy integration with the Nightly Building System.
Provide also a graphical interface to run the tests and examine the test results.
Our inputs:
Contacts within HEP-community.
What is available as free open source code.
Our constraints:
Avoid commercial software and
licensing problems.
Avoid “do it yourself solutions”
Try to adopt commonly used
open-source software.
M Gallas CERN EP-SFT
LCG-SPI: SW-Testing

5. Test Frameworks: Global picture
QA activity
End User
Installation
checks
Nightly
Building
Project Release
Sw-Testing
Top layer
Bottom layer
- Adaptable to the
programming language
and developer
- Prepared to be run in automatic way
- Integrates different ways to test.
- Common environment to run the tests and to access the test results.
QMTest
Uses a GUI for creating and running tests (also in batch).
Can run tests in parallel, supports execution of a single
test or many at once (test-cases & test-suites)
Organizes tests hierarchically
Records dependencies among tests
X-Unit family
JUnit
QtUnit
CppUnit
PyUnit
Oval
Test
Scripts
Old
tests
SW Product Examples
Unit testing
Acceptance
testing
M Gallas CERN EP-SFT
LCG-SPI: SW-Testing

6. Test Frameworks: X-Unit family
JUnit
QtUnit
CppUnit
PyUnit
A simple test:
Subclass the TestCase CppUnit or PyUnit class
Override the method runTest().
When you want to check a value, call the (CppUnit or PyUnit) ASSERT(bool) and pass in an expresion that is true if the test succeeds
Main Scope: Unit–testing
The same “assertion style” in different languages.
Provides:- The name of the test case that failed.
-The name of the source file that contains the test.
-The line number where the failure occurred
Different platforms/compilers: (Linux/Solaris/Windows)
M Gallas CERN EP-SFT
LCG-SPI: SW-Testing

7. Test Frameworks: Oval Test Source Executable Code Ovalfile Oval diff
validation and regression
developed & used within
CMS experiment
Ovalfile
(configuration)
Executable
Test
Source
Code
Reference
file
Log
Oval run
Oval
diff
Oval build
Can be use for Unit-test to Validation tests.
Compare the output log file with a given reference file (Smart comparison of those lines which start with “[Oval]”)
It is possible to set different run environments.
Can run external scripts and external binaries.
M Gallas CERN EP-SFT
LCG-SPI: SW-Testing

8. Test Frameworks: QMTest
Uses a graphical interface for creating and running tests
The configuration files are in XML and can be created from the GUI. We provide also script to do it
Runs tests in parallel
Organizes tests hierarchically
Supports execution of a single test or many at once
Records dependencies among tests
Can be run in batch mode -> easy integration with the Nightly-Building systems
M Gallas CERN EP-SFT
LCG-SPI: SW-Testing

9. Test Documents
For all project is needed a Test Plan in order to have a clear idea of the type of tests, schedule, environment and responsibilities.
We use a template
Test Case template collects information about the existing tests and how they cover the expected functionality of the software.
M Gallas CERN EP-SFT
LCG-SPI: SW-Testing

10. User support: Web information
Test Frameworks
HowTo
Test doc
Policies
M Gallas CERN EP-SFT
LCG-SPI: SW-Testing

11. Automatic testing requires
User support: How-To * * * * 1 2 3
Follow sw-testing
policies
Automatic testing requires
some standardization *
M Gallas CERN EP-SFT
LCG-SPI: SW-Testing

12. Final Remarks SPI SW-Testing
Software testing is a needed step in the software
development process.
SPI provides test frameworks, templates and help to
have a coherent automatic testing procedure with a
common interface to run and browse the test results.
The proposed testing tools were chosen following the
criteria of a simple approach, tools in use in the
experiments and freely available.
SW-Testing Overview
LCG-SPI SW-Testing
Test Frameworks
Test Documents
User support
SPI SW-Testing
Following standards makes life easier to SPI and the users.
Automatic testing is always needed and is a way to consolidate
the software.
Bug reports and typical use-cases from the users
should be converted into tests
SPI SW-testing has to evolve and consolidate following
the needs of the LCG projects.
All SPI sw-testing tools, templates, documentation can be
used by any project.
Thanks to:
LCG-PI team
LCG-POOL team
LCG-SEAL team
LCH-experiments
M Gallas CERN EP-SFT
LCG-SPI: SW-Testing