Standard Scripts Project 2 2014 - Proposal for Qualification of Standard Scripts

Main Sections Summary of prior proposal, 2013 hidden slides, to preserve history Updated proposal, July 2014

Main Sections Summary of prior proposal Updated proposal Objectives Definitions: Qualification, States, Roles Metadata and Test data State Transitions

Proposal 2014 - Objectives For End-users Clear overview of resources available, incl. purpose & state of each Inspire confidence from first user experience Ease-of-use: clear messaging from scripts Reproducible results in users’ own environments Consistency of scripts, learning first one makes remaining familiar Ease of converting users to contributors For Contributors & Standard Scripts Team Ease of contribution: Modularize & standardize workflows & checklists Modularize core components (e.g., FUTS for dependency checking?) Automate testing, issue identification (e.g., concept similar to Spotfire/R compatibility) Centralize & consolidate information & results DDT comment on "Ease of converting users ..." Contribution should be easy! And should accommodate the willing contributor, however much or little time they have available.

Qualification Proposal meaningful terms in blue http://www. phusewiki Qualification Instructions See template on PhUSE Wiki, link to file "Qualification of CSS scripts.xlsx" Certification applies to new scripts and functionality Confirmation applies to updates of existing script States: Contributed, Develop, Review, Qualified Roles Contributor: Anyone with appropriate skills & interests Developer: A volunteer in CSS Working Group 5, familiar with our objectives** Tester: A volunteer in CSS WG 05 who is familiar with our objectives** Environment Tester: Anyone in community who is able to set up automatic test replication in their work environment Reviewer: Author of white papers, designers of script targets** ** suggests a quick-start onboarding page in CSS Phusewiki Qualification phase vs. State: First time a script is developed/tested, we perform Certification phase Upon later updates, developer/tester perform Confirmation phase

Qualification Proposal Metadata for scripts should indicate: YML proposal Whitepaper & output ID uniquely identify the target <Script: Source, Title, Target> Programming language & version <Language: name, version> Type of output: <Script: Type> Study design: <Script: StudyDesign> State of qualification <Stage>: <Qualification: Stage> OS is not included, since should be indicated in OS compatibility report Test Data requirements available as a program or script (text, not binary) based on expected standards (SDTM, ADaM) data requirements clearly & accurately specified for each script include expected results from these data for defined tests/checks YML Metadata template in Phuse-scripts google code repository: https://code.google.com/p/phuse-scripts/source/browse/trunk/ nb: <Script: StudyDesign> tag does not yet exist in the YML Metadata template YML specifications: http://www.yaml.org/spec/1.2/spec.html "OS is not included ..." See related concepts of automated testing and Environment Testers, above

Qualification Proposal Transitions "Contributed" is the original State of all scripts to Develop, checklist includes by Developer & Reviewer R & D confer on suitability of contribution. Suitable starting point? [ may require analysis details, specs, from contributor ] D reviews components (checklist to be completed) D works with Contributor to complete minimum components [ including Test Data and Coverage of defined tests ] D adds standard parameter, dependency checking D writes Qualification of CSS scripts.xlsx, see PhUSE Wiki Qual Workbook to Review, checklist includes by Tester Review Qualification instructions, consider coverage of tests Execute Qualification instructions Work with Developer to complete execution successfully R & D confer: This is essentially an investment decision. Does the contribution warrant development by the Standard Scripts volunteers? Is it a sufficient starting point? D reviews components ... partial list: Clear scope & requirements for target output (from White Paper?) Good Programming Practices Program header Documentation (just in header?) Test Data D adds standard parameter, ... See for e.g., http://thotwave.com/resources/futs-framework-unit-testing-sas/ ThotWave may be interested to contribute with FUTS, their Framework for Unit Testing SAS. We could probably use much of this framework & components D writes Qualification instructions ... See "Qualification of CSS scripts.xlsx" on PhUSE Wiki http://www.phusewiki.org/wiki/index.php?title=File:Qualification_of_CSS_scripts.xlsx

Qualification Proposal Transitions continued to Qualified by Tester & Environment Tester & Reviewer T updates reference test outputs from certification/confirmation E updates & executes local tests (posting PASS/FAIL results) R confirms script output matches intention R confirms qualification process covers important elements and considerations. R also provides user (rather than technical) feedback? Achieve "Qualified" state when all tests in all test environments PASS (i.e., match outputs that T has certified and/or confirmed) and that R agrees that target is achieved

Qualification Proposal Efforts Required Top priority Finalize Qualification states, roles, workflow, checklists, and templates Code criteria for "acceptable" (link to GPP, PhUSE) Test definition, Test confirmation (also from White Paper team) Test data Next priorities Design test structure in google code Develop scripts that will allow Environment Testing Develop general components (e.g. parameter, dependency checking) Identify Environment Testers based on Host environment SAS or R version Identify opportunities to automate qualification. E.g., Environment Testers to post results back as machine readable Script green-light/red-light qualification matrix on Phusewiki