SDD/DFS A. Modigliani VLT 2 nd Generation Instrumentation Pipelines, 19 Apr ACCEPTANCE TESTS Andrea Modigliani

SDD/DFS A. Modigliani VLT 2 nd Generation Instrumentation Pipelines, 19 Apr OUTLINE Why/When Priorities Acceptance tests Experience & feedback

SDD/DFS A. Modigliani VLT 2 nd Generation Instrumentation Pipelines, 19 Apr Why/When Why:  Need to define a clear and compact list of requirements and tests to verify the compliance of DRL deliveries to PAE/COM1.  Standardise and make transparent the review process.  Possibly distribute the review across a long period.  Ensure that the DRL not only meets scientific specs but also is easy to maintain, portable, well documented, efficient, robust. When:  Every three months to have at least 4-6 iterations in the period FDR-PAE.  We encourage the consortium to apply the tests during development.  Tests are performed by ESO after a given DRL version has been certified/provided on an appropriate data set.

SDD/DFS A. Modigliani VLT 2 nd Generation Instrumentation Pipelines, 19 Apr Priorities The next slides list several requirements organized by priority: Priority 1 tests will be checked on every delivery, Priority 2 tests are performed from time to time, Priority 3 tests are verified at the end (PAE & end of commissioning)…together with priority 1 and priority 2 Each test is listed and a possible verification criteria is indicated in parenthesis, references to other presentations are indicated as [Author]

SDD/DFS A. Modigliani VLT 2 nd Generation Instrumentation Pipelines, 19 Apr Compliance and initial verifications The acceptance test should be part of the PAE process (executed on DRL 0.5). Any action should to be completed before the last commissioning run (in DRL 1.0). See also [R. Palsa, Y. Jung]. Does the pipeline follow the recipe template [Y. Jung]? (static checks). Is the documentation in English? (manual inspection) Does the software conform to ESO coding standards [R. Palsa]? (manual inspection, compilation with no warning with strict compiler options). Does the pipeline use only CPL and agreed upon libraries [C. Izzo, S. Castro] ? (static checks). Does the code contain only CPL functions/objects? If something is missing, ask (compute ratio CPL/SLOC). Are exported function namespace protected? (static checks).

SDD/DFS A. Modigliani VLT 2 nd Generation Instrumentation Pipelines, 19 Apr Execution tests: to verify the delivery is complete and installable/executable (1) Do the recipes exist in the expected version? (esorex –recipes, esorex –man-page) Is there a test package covering each combination of recipes and major instrument setting? (manual inspection, compare with DRL doc) Are test data appropriate to verify each recipe and DRL relevant function? (manual inspection, compare with DRL doc) Are recipes executable? (verify on provided test package) Do the recipes create the expected products [S. Castro]? (compare output versus DRL doc) Do the recipes create the QC they should [P. Ballester, S. Castro]? (compare output versus DRL doc)

SDD/DFS A. Modigliani VLT 2 nd Generation Instrumentation Pipelines, 19 Apr Execution tests (2) Do the expected DRL functions exist? (nm libiiinstrument.so) Is there a unit test for each DRL function covering each instrument setting and parameter space [L. Lundin]? (manual inspection, compare with DRL doc) Are there memory leaks [J. M. Larsen]? (esorex –mem-check recipe_name recipe_name.sof) Are there memory errors [J.M. Larsen]? (valgrind –tool=memcheck) Is the recipe interface implemented in a standard way [Y. Jung]? (runtest.pl, test with Gasgano) Is the doxygen doc complete? (make html, check results) Do all source files contain addtogroup/defgroup? (static checks, make html) Are all recipes documented, including In/Out tags? (esorex –man-page)

SDD/DFS A. Modigliani VLT 2 nd Generation Instrumentation Pipelines, 19 Apr Detailed validation: are results correct? Do recipes give correct results (FITS products + QC parameters)? (run recipe on test data and check In/Out, compare with DRL doc) Do the interfaces fit together (is it possible to run a recipe cascade?) Are the final results (after running the cascade) correct? (manual inspection, compare with DRL doc) Does each recipe terminate in a reasonable time ? (exposure time/execution time >1). [To profile see J.M. Larsen] Does the pipeline work and give results as defined in the DRL design doc? (manual inspection, compare results with DRL doc)

SDD/DFS A. Modigliani VLT 2 nd Generation Instrumentation Pipelines, 19 Apr Detailed validation (2) Does the unit test implement the quality assessment defined in the DRL doc? (manual inspection, compare with DRL doc) [L. Lundin] Does the unit test check the basic validity of the function results (including QC parameters) and that the accuracy is as required? (manual inspection, compare with DRL doc) Does the unit test cover relevant parts of input parameter space, such as true/false for Boolean parameters? (manual inspection) Do the unit tests pass? (manual inspection, make check) Does the recipe verify it receives proper input (check for input TAGs)? Is the FITS format as described in the DRL doc (extensions/keywords)? Are the product valid FITS files? (fitsverify) Is the documentation and code understandable? (manual inspection)

SDD/DFS A. Modigliani VLT 2 nd Generation Instrumentation Pipelines, 19 Apr Experience and feedback Communication is a crucial factor Acceptance tests should be part of the implementation schedule presented at FDR. Acceptance tests may involve additional work in the initial implementation phase but will save a lot of time for PAE and commissioning and help to provide a better product to the users. Acceptance tests are in addition to other tests which may be applied by the instrument SV team.