1. Geant4: System Testing Team
Steve O’Neale
CERN IT/ASD and University of Birmingham
LAL-Orsay, France Oct. 2000
1/18/2019

2. STT Activities Shrunk from 14 names to 6 yielding 1 fte
Bonsai recently introduced (Serguei Sadilov)
gui to track state of testing from developers cvs tag
internal
proposed for testing
selected for testing
accepted
rejected
global
Testing [the subject of this presentation]
Tinderbox, to view results, next (Fred Jones)
LXR to browse source code in “internal” use
1/18/2019

3. Current Platforms At CERN from stesting account
can login from fukui, babar-online, G4Orsay etc.
DXplus(5) afs iso/non-iso opt/deb 4
HPplus (5) afs opt/deb 2
SUNgeant nfs opt/deb 2
PCgeant nfs Linux opt/deb 2
NT (LHCb on reftags)
1/18/2019

4. Platforms: January 2001 CERN RISC Reduction Planning
DX plus Frozen, only for LEP use
HP plus Frozen, only for LEP use
NT (PCSF) convert to Linux
(after Feb LHCb production, if linux drivers available)
LXBatch many machines
SUN adequate for development
Atlas, Alice and CMS to prepare requirements
GEANT4 CB/TSB Change Supported Platforms ?
Drop some risc architectures or ask principal users to host tests
Add (maybe) newer SUN system, newer Linux, IA-64.
1/18/2019

5. STT Procedures (before bonsai)
Documented with few changes
Worked at the HEAD (with bonsai we work from a list of tags)
prod and two dev areas, per platform, in afs at cern
Some links to local disks (need more )
cvs update for tags prepared by hand
make and scripts compile, make libraries, executables, run tests and diff outputs
data file prepared by hand to drive scripts for each/all platforms
may submit all libraries and/or tests to build and run
or select just one test at high statistics (Large_N)
inspect results (1000 file in three different file systems)
report by mail and editing web page (updated on cvs commit)
1/18/2019

6. Automation and Monitoring(1)
Prototypes in perl (in cvs, undocumented), no major changes to previous scripts, now need to change procedures, review platforms, sites and stt activities, specify reporting to Tinderbox.
In prod/dev1/dev2 dir, source setup.©sh to establish environment.
Hasami.plx
queries bonsai database to extract last reference tag, accepted tags and tags for testing
generates serial number (123), driver files for testing (reftag+tags123)
prunes bonsai123.sdb for current e&o in bonsai,
hosts123.data according to available test platforms
run existing update.sh script yielding one line per file update123.log
ReviewUpdate.plx
check update log (for clashes, modified code, errors)
produce short summary, could launch tests if no problems found
we sometimes debug or work interactively in the source tree
ExtractLastLog.plx
FilterMake.plx
ShowFiles.plx
1/18/2019

7. Automation and Monitoring (2)
Launch tests g4allsys.sh < hosts123.data
ExtractLastLog.plx (not a great name)
inspects gmake.log for compilation and build (size and age)
g4allsys.sh log for tests (Started, Finished, Disabled etc)
generates web page
could be run (every n minutes) via acrontab at cern.
FilterMake.plx
more sophisticated than grep (E)rror
attempts to classify each line in gmake.log
might be developed to tabulate known and new warning
does Tinderbox already have such analysis tools ?
1/18/2019

8. Automation and Monitoring (3)
ShowFiles.plx
examines the expected test log (and diff) files
reports missing files (i.e. test not run)
reports G4 error messages per test
excludes differences wrt reference output due to cvs version tags, directories used, date, time, iso/non-iso formatting, some cases where many tracking steps are printed and so on
shows number of differences (if any) per test per platform
1/18/2019

9. STT results for the developer
Tag is accepted or rejected
may reverts to proposed if co-working tags need work.
Compilation or load failure
on developer’s least favourite platform
small error can generate 20Mb of logs for hp “future” compiler
Tests results
run to completion - no change wrt reference output
run to completion - developer validates change to reference output
(new) warnings and/or fatal errors found
test loops or exhausts resources in some way
1/18/2019

10. STT Operational Problems
Few tests of a tagset have finished cleanly in 2000
Modifying the scripts to make use of local (nfs mounted) disk space on PUBLIC machines will help a lot.
Independent monitoring is needed as there are many ways in which NO output is obtained
machine re-booted to fix (say) an afs hang in atlas/delphi build
machine down, has too many processes, disks full, token expires
test program loops, or waits, or fills disks (selective switched off)
afs can hang waiting for a server response at any time
from after 10 seconds when clearing old test files (rm bin/*)
to writing (ld) the last test executable after hours of building
1/18/2019

11. Next Steps
Bonsai, tinderbox, lxr, changing the testtools, monitoring and automating the procedures, as suggested, can reduce the stt manpower needs and improve the presentation to the developers.
Before proceeding, we have some questions for G4
Will we soon drop platforms (Dec, Hp, NT) ?
Will we have test sites other than CERN ?
Will we have more than 1 person to run the tests ?
We could then take on more interesting activities
timing, profiling, leak detection, code metrics, regression analysis, extend test cases to more physics and detectors …..
1/18/2019