1. SYSTEM INTEGRATION TESTING Getting ready for testing shifts Gunter Folger CERN PH/SFT Geant4 Collaboration Workshop 2011 SLAC

2. Tasks of STT working group  Integration testing of proposed tags  Regular - ~ daily - testing of new code  Shifts planned for this work  Q/A for release  Using tools checking memory use,  Checking code for std::,...  Physics validation test suite  HEP oriented 19-Sep-2011 2 Gunter Folger - CERN PH/SFT

3. Contents  “shift duties”  Testing cycle  Tags management tool  Selecting, rejecting, accepting tags  Checking results  Logging results or decisions  Documentation 19-Sep-2011 3 Gunter Folger - CERN PH/SFT

4. Shift Duties  Routine work within collaboration should be shared across collaboration  Daily running of system integration testing is one of these routine work items  like reviews for papers  Not included  Integrating more tests  Managing configuration 19-Sep-2011 4 Gunter Folger - CERN PH/SFT

5. Who does what? What is integration testing 19-Sep-2011 5 Gunter Folger - CERN PH/SFT

6. Regular integration testing Developer propose Tags Tester selects Tags Testing in “nightlies” Tester analyse results Tester accepts / rejects tags 19-Sep-2011 6 Gunter Folger - CERN PH/SFT

7. Steps – prepare testing  Tag(s) are proposed by developer  Tags are selected by STT tester  Use “full interface” of geant4tags https://sftweb.cern.ch/geant4/geant4tagsadmin  Taking into account dependencies on co-working tags  Avoiding sets of tags risking confusing results Hints: NO: development in different (sub)categories Yes: bug fixes in a category, or tags in tests and examples 19-Sep-2011 7 Gunter Folger - CERN PH/SFT

8. Geant4tags – full interface 19-Sep-2011 8 Gunter Folger - CERN PH/SFT

9. Selecting tags  Mark tags and click “Change Status”  Check and adapt settings in pop-up, and click “Change”  Taglist is a grouping of tags  Tags selected in one day  Optional name 19-Sep-2011 9 Gunter Folger - CERN PH/SFT

10. Starts short past midnight at CERN Lasts few hours Where? How? What ? Wait for testing 19-Sep-2011 10 Gunter Folger - CERN PH/SFT

11. 19-Sep-2011 11 Gunter Folger - CERN PH/SFT

12. Testing - How 19-Sep-2011 Gunter Folger - CERN PH/SFT 12  Testing using SPI nightlies is automatic – unless there is a rare problem  Configure environment using CMT environment variables, compiler Geant4 specific setup in Geant4 CMT requirements file  Checkout automatic – querying db for tags  Compile/Build using Geant4 build system  Testing uses QMtest tool run tests as defined in test suite, setup in geant4/tests/tools/qmtest Failure defined as sizeof(stderr) != 0, or crash. In case of crash, core file is created, but is local to testing machine  Results of build & test copied to web  Crash require manual intervention  Core file can only be inspected on testing machine  No trace back generated

13. Testing - What 19-Sep-2011 Gunter Folger - CERN PH/SFT 13  We test different configurations in ‘slots’  Testing selected tags using Normal GNU-make build & test Cmake build & test using GNUmake and QMtest Internal CLHEP & Energy conservation checks in G4Hadronic Process enabled  Testing reference/patch tag using GNU-make build  And using differing OS/compiler combinations  Linux, Mac, Windows Sometimes several compilers  Sunday is Large_N test suite is run

14. Nightlies result page 19-Sep-2011 14 Gunter Folger - CERN PH/SFT

15. Analyse Results  Reject tag causing compilation error or warning  Any (new) tests failing in any configuration  If clear which tag causes this, reject tag  If unclear, one can postpone a tag  Run debugger on testing machine  Record result in online document 19-Sep-2011 15 Gunter Folger - CERN PH/SFT

16. Accept/reject tags  Important Do not let increase the number tests failing  Accept tags only if no  Compilation error or warning Do reject any tag with compilation warning or error  No test newly failing Unless a change in one category uncovers an old problem somewhere else. 19-Sep-2011 16 Gunter Folger - CERN PH/SFT

17. How to Reject Tags  Geant4tags web page, as for selection  Rejecting:  Select tag to reject, one by one  Change status  in popup, select reject and in new field fill in reason and click “Change” 19-Sep-2011 17 Gunter Folger - CERN PH/SFT

18. How to Accept tags  From full interface in Geant4tags  Mark all tags you want to accept, and click change Can do this on the tags tab Or on taglist tab, select taglist, and mark tags Click in header row will mark all  In popup, check that ‘Accepted’ is selected and click change 19-Sep-2011 18 Gunter Folger - CERN PH/SFT

19. Documentation  Working group web page has links to  Documentation on test procedures http://geant4.cern.ch/collaboration/working_groups/testing/restricted/geant4/tests/results/procedures.html http://geant4.cern.ch/collaboration/working_groups/testing/restricted/geant4/tests/results/procedures.html Needs improvements, please give feedback  List of tests and examples being used in testing Actual list is in log file of test Web page listing exists http://geant4/collaboration/working_groups/testing/tests.shtml http://geant4/collaboration/working_groups/testing/tests.shtml 19-Sep-2011 19 Gunter Folger - CERN PH/SFT

20. Shift duties - Summary  For about one week, once per day: 1. Checks results and document results 2. Reject or accept tags 3. Select proposed tags 19-Sep-2011 20 Gunter Folger - CERN PH/SFT

21. Shift duties - Summary WhatTime CERNSLAC ( -9 )Japan ( +8 ) Checks results and document results Morning >~ 8:00 MorningLate afternoon >~ 16:00 Reject or accept tagsAs above Select proposed tagsBefore midnight Before 3:00 pmFrom 9:00 until 8:00 next day 19-Sep-2011 21 Gunter Folger - CERN PH/SFT For about one week of shift, once per day

22. Proposal  Start with testing shifts during “normal” weeks  Release phase depending on experience  Shifts of two testers, one novice, one expert  During release period, local expert will join  Need close collaboration with release manager 19-Sep-2011 22 Gunter Folger - CERN PH/SFT

23. Summary  Shift system for regular integration testing possible  Tools have been refined  No failing tests on supported platforms  Propose to start shifts now, and gather experience  Steering board first?  Improve documentation with experience and feedback  Andrea will continue to show  tools to document results  Subscribe for shifts 19-Sep-2011 23 Gunter Folger - CERN PH/SFT