SYSTEM INTEGRATION TESTING Getting ready for testing shifts Gunter Folger CERN PH/SFT Geant4 Collaboration Workshop 2011 SLAC

Tasks of STT working group  Integration testing of proposed tags  Regular - ~ daily - testing of new code  Shifts planned for this work  Q/A for release  Using tools checking memory use,  Checking code for std::,...  Physics validation test suite  HEP oriented 19-Sep Gunter Folger - CERN PH/SFT

Contents  “shift duties”  Testing cycle  Tags management tool  Selecting, rejecting, accepting tags  Checking results  Logging results or decisions  Documentation 19-Sep Gunter Folger - CERN PH/SFT

Shift Duties  Routine work within collaboration should be shared across collaboration  Daily running of system integration testing is one of these routine work items  like reviews for papers  Not included  Integrating more tests  Managing configuration 19-Sep Gunter Folger - CERN PH/SFT

Who does what? What is integration testing 19-Sep Gunter Folger - CERN PH/SFT

Regular integration testing Developer propose Tags Tester selects Tags Testing in “nightlies” Tester analyse results Tester accepts / rejects tags 19-Sep Gunter Folger - CERN PH/SFT

Steps – prepare testing  Tag(s) are proposed by developer  Tags are selected by STT tester  Use “full interface” of geant4tags  Taking into account dependencies on co-working tags  Avoiding sets of tags risking confusing results Hints: NO: development in different (sub)categories Yes: bug fixes in a category, or tags in tests and examples 19-Sep Gunter Folger - CERN PH/SFT

Geant4tags – full interface 19-Sep Gunter Folger - CERN PH/SFT

Selecting tags  Mark tags and click “Change Status”  Check and adapt settings in pop-up, and click “Change”  Taglist is a grouping of tags  Tags selected in one day  Optional name 19-Sep Gunter Folger - CERN PH/SFT

Starts short past midnight at CERN Lasts few hours Where? How? What ? Wait for testing 19-Sep Gunter Folger - CERN PH/SFT

19-Sep Gunter Folger - CERN PH/SFT

Testing - How 19-Sep-2011 Gunter Folger - CERN PH/SFT 12  Testing using SPI nightlies is automatic – unless there is a rare problem  Configure environment using CMT environment variables, compiler Geant4 specific setup in Geant4 CMT requirements file  Checkout automatic – querying db for tags  Compile/Build using Geant4 build system  Testing uses QMtest tool run tests as defined in test suite, setup in geant4/tests/tools/qmtest Failure defined as sizeof(stderr) != 0, or crash. In case of crash, core file is created, but is local to testing machine  Results of build & test copied to web  Crash require manual intervention  Core file can only be inspected on testing machine  No trace back generated

Testing - What 19-Sep-2011 Gunter Folger - CERN PH/SFT 13  We test different configurations in ‘slots’  Testing selected tags using Normal GNU-make build & test Cmake build & test using GNUmake and QMtest Internal CLHEP & Energy conservation checks in G4Hadronic Process enabled  Testing reference/patch tag using GNU-make build  And using differing OS/compiler combinations  Linux, Mac, Windows Sometimes several compilers  Sunday is Large_N test suite is run

Nightlies result page 19-Sep Gunter Folger - CERN PH/SFT

Analyse Results  Reject tag causing compilation error or warning  Any (new) tests failing in any configuration  If clear which tag causes this, reject tag  If unclear, one can postpone a tag  Run debugger on testing machine  Record result in online document 19-Sep Gunter Folger - CERN PH/SFT

Accept/reject tags  Important Do not let increase the number tests failing  Accept tags only if no  Compilation error or warning Do reject any tag with compilation warning or error  No test newly failing Unless a change in one category uncovers an old problem somewhere else. 19-Sep Gunter Folger - CERN PH/SFT

How to Reject Tags  Geant4tags web page, as for selection  Rejecting:  Select tag to reject, one by one  Change status  in popup, select reject and in new field fill in reason and click “Change” 19-Sep Gunter Folger - CERN PH/SFT

How to Accept tags  From full interface in Geant4tags  Mark all tags you want to accept, and click change Can do this on the tags tab Or on taglist tab, select taglist, and mark tags Click in header row will mark all  In popup, check that ‘Accepted’ is selected and click change 19-Sep Gunter Folger - CERN PH/SFT

Documentation  Working group web page has links to  Documentation on test procedures Needs improvements, please give feedback  List of tests and examples being used in testing Actual list is in log file of test Web page listing exists Sep Gunter Folger - CERN PH/SFT

Shift duties - Summary  For about one week, once per day: 1. Checks results and document results 2. Reject or accept tags 3. Select proposed tags 19-Sep Gunter Folger - CERN PH/SFT

Shift duties - Summary WhatTime CERNSLAC ( -9 )Japan ( +8 ) Checks results and document results Morning >~ 8:00 MorningLate afternoon >~ 16:00 Reject or accept tagsAs above Select proposed tagsBefore midnight Before 3:00 pmFrom 9:00 until 8:00 next day 19-Sep Gunter Folger - CERN PH/SFT For about one week of shift, once per day

Proposal  Start with testing shifts during “normal” weeks  Release phase depending on experience  Shifts of two testers, one novice, one expert  During release period, local expert will join  Need close collaboration with release manager 19-Sep Gunter Folger - CERN PH/SFT

Summary  Shift system for regular integration testing possible  Tools have been refined  No failing tests on supported platforms  Propose to start shifts now, and gather experience  Steering board first?  Improve documentation with experience and feedback  Andrea will continue to show  tools to document results  Subscribe for shifts 19-Sep Gunter Folger - CERN PH/SFT