How to Contribute to System Testing and Extract Results Gunter Folger CERN/PH/SFT Geant4 workshop 2010/ ESA

Content STT ⇔ System Testing Team Mandate, or Tasks How does it work? How to contribute

Mandate To improve the quality and robustness of GEANT4 code at the "System Level", To ensure it compiles and runs without crashing on all supported platforms in tests which cover a wide range of physics, geometries, particles, processes over many events.  Although physics validation is not our aim, we will obviously be aware of the physics quality and seek cooperation with the category teams.

Tasks Integration testing of proposed tags Q/A for release Regular - ~ daily - testing of new code Currently no backup person Help is needed in this routine job Q/A for release Using tools checking memory use, Checking code for std::, ... Physics validation test suite HEP oriented

Regular integration testing Propose Tags Tags Select Testing Analyse results Accept / Reject

Steps – prepare testing Tag(s) are proposed by developer Tags are selected by STT Taking into account dependencies Avoiding sets of tags risking confusing results Bonsai interface makes it easy to do mistake in this step Two/three step process per tag Some pitfalls or bugs Some mistakes are be hard to correct, only in db itsself. Replacement for web interface in development

Testing Testing automatic – unless there is a problem Configure environment using cmt Checkout – querying bonsai db for tags Compile/Build using Geant4 make system Testing, QMtest runs all tested defined in test suite Failure defined as sizeof(stderr) != 0, or crash. In case of crash, core file is created Results of build & test copied to web Crash require manual intervention Core file can only be inspected on testing machine No trace back generated

Steps – analyze results Reject failing tag(s) Compilation error/warning Runtime error messages, crash Accept tags if no (new) problem

Current difficulties to share work Bonsai web interface Cumbersome to select tags correctly Easy to make mistakes Debugging of crashes requires access to testing machines Still fairly large number of failures on Windows Replacement under development Alpha version available at https://sftweb.cern.ch/gea nt4/geant4tags Access possible Improve handling of crash? Already greatly reduced Difficulty Remedy

How can you contribute? Assuming Bonsai web interface is replaced Running Shifts for testing? Starting with few volunteers

How can you contribute (2) Short term help Several tests fail on windows 6 months ago, the number was much higher So far, all were due to bugs in code Help in finding problems will be welcome.

How to access information From bonsai, there is link to results page, or Link to testing results

Not covered Extending test suite, include new tests or examples cmt configuration for Geant4

Summary Improvements in testing procedures and tools will allow to share testing work A few volunteers initially needed to test Help with debugging on windows is welcome

Backup Slides shown in Catania/2009

Overview Move of Geant4 integration testing from TINDERBOX to SPI nightly build system Pop versus push Bonsai untouched Recent changes in nightly build system Server approach with goal to run several “jobs” in parallel Geant4 contributed new display of results Future planned changes More involvement of developers More frequent tests with direct feedback Shifts of running testing?

What has changed for Geant4 developer? Did you notice a change Compilation warnings no longer tolerated More examples included in testing MAC fully integrated Windows still requires work

Move to SPI nightlies (1) Supported platform include Linux, Mac, and Windows Previous setup worked for Linux, but Mac and Windows not integrated Regular build on set of machines once per night Machine pulls information on what to build Build is completely on local disk Source is checked out by each machine, possibly more than once Performance, and less affected by afs problems Network timeout, quota,... Support for testing different configurations on one machine Currently two: Current selected tags Reference tag, patch, release, as soon as preliminay tag exists Started to test multiple compilers on SLC5

Move to SPI nightlies (2) Build configured via CMT was setup scripts Single configuration file for all systems, yet allows for specialisation. Standard Geant4 make based build system used to build libraries

Move to SPI nightlies (3) Test tool is QMTest Tinderbox no longer in use ( ~March 09) Setup in cvs under tests/tools/qmtest Based on python, easily extended Allows to easily run tests, examples, but also benchmarks, without using cryptic symlinks from test to examples. A test consists of 2+ tests Build the executable compilation warnings flagged as FAILURE Run the executable with possible different input file(s)  multiple tests for same executable

Move to SPI nightlies (4) Log files copied to web server parsed and available from http://lcgapp.cern.ch/spi/cgi-bin/nightlies Or http://lcgapp.cern.ch/spi/cgi-bin/nightly.py

Inventory of examples tested See also http://geant4.cern.ch/collaboration/working_groups/testing/tests.shtml

Improvements of SPI nightlies(1) Original SPI display of results difficult to analyse Failing tests difficult to correlate across platforms Addressed by storing results in database Offering lots of possibilities to view results Web page allows to analyse failures across platforms, days, ... Work of technical student Victor Diez Gonzales

Improvement of SPI nightlies(2) Distributions of builds to machines completely revised A server distributes build to Client machines Capabilities of client in central configuration file Server may run several build in parallel, if resources allow Number of CPUs available and allowed to use

Summary Migration to SPI nightlies successful Effort to run testing significantly reduced Select tag in bonsai Check results Accept/reject tags, eventually debug on build machine What is next? Ideas: Add yet more tests, e.g. benchmark directory Automatic generation of stack trace for crash Create tests database