#PhUSE Standard Scripts Project Proposal for Qualification of Standard Scripts

#PhUSE Main Sections  Summary of prior proposal, 2013  Updated proposal, July 2014

#PhUSE Main Sections  Summary of prior proposal – Concepts, definitions & metadata – Test data considerations – Heavy vs. Light qualification  Updated proposal

#PhUSE Proposal from 2013 Anyone can submit a script, according to a check list Categorize scripts by complexity – Complexity: low, medium, high, software – Output: tabulated data, analysis data, table, figure, listing Metadata for script should indicate – Type of output: tabulated data, analysis data, table, figure, listing – Study design: parallel, crossover, etc – State of qualification:?

#PhUSE Proposal from 2013 Test data – Overall project should have minimum test data (SDTM & ADaM) – Scripts can propose new test data, must pass (Data fit? Open CDISC?)Open CDISC – Share program to produce test data, never binary test data 2 levels of qualification – Light vs. Heavy: according to complexity of script & output – Common elements include header adhere to good programming practicesgood programming practices clear scope of script (e.g., study design(s)) test data matche scope & pass "FDA Data Fit" assessment (Open CDISC?) documentation available for: usage, inputs, outputs, dependencies

#PhUSE Proposal from 2013 Heavy qualification – Beta package includes:minimal elements for contribution Specification & Documentation (could be in pgm header) Test data (Data Fit? or Open CDISC?) Tests & Expected results defined Peer Review: GPP, Specs & Docn reviewed, Tests reproducedGPP – Draft Write qualification plan, Review tests for completeness/suitability (e.g., Branch testing – are all conditional blocks/combos tested?) – Test Peer Review: Write qualification report, incl. log/output from tests – Final

#PhUSE Proposal from 2013 Light qualification – Beta package includesskip if >1 yr production use without ERROR – Draftminimal elements for contribution Specification & Documentation (could be in pgm header) Test data (Data Fit? or Open CDISC or other, as appropriate) Tests & Expected results defined Peer Review: GPP, Specs & Docn reviewed, Tests reproducedGPP Write qualification plan, Review tests for completeness/suitability (e.g., Branch testing – are all conditional blocks/combos tested?) – Test Peer Review: Write qualification report, incl. log/output from tests – Final

#PhUSE Proposal through CSS 2104 Peer Review ChecklistHeavyLight Requirement specificationX? Documented or perhaps only documented in headerX User GuideXX SDTM/ADaM used in input/outputXX Open CDISC validator or Data Fit used to check input/outputXX GPPGPP in sourceXX Run according to Requirement specificationX? Tested by qualification plan, tests & results all Peer reviewedX? Tested by End usersX? Robust without red errors in contributor's production environmentXX Robust and used in FDA (other) scripts repository, ranked ******X

#PhUSE Main Sections  Summary of prior proposal  Updated proposal – Objectives – Definitions: Qualification, States, Roles – Metadata and Test data – State Transitions

#PhUSE Proposal Objectives For End-users – Clear overview of resources available, incl. purpose & state of each – Inspire confidence from first user experience – Ease-of-use: clear messaging from scripts Reproducible results in users’ own environments Consistency of scripts, learning first one makes remaining familiar – Ease of converting users to contributors For Contributors & Standard Scripts Team – Ease of contribution: Modularize & standardize workflows & checklists – Modularize core components (e.g., FUTS for dependency checking?)FUTS – Automate testing, issue identification (e.g., concept similar to Spotfire/R compatibility ) concept similar to Spotfire/R compatibility – Centralize & consolidate information & results

#PhUSE Qualification Proposal meaningful terms in blue Qualification Instructions (see embedded template  ) – Certification applies to new scripts and functionality – Confirmation applies to updates of existing script States:Contributed, Develop, Review, Released Roles – Contributor: Anyone with appropriate skills & interests – Developer: A volunteer in CSS Working Group 5, familiar with our objectives** – Tester: A volunteer in CSS WG 05 who is familiar with our objectives** – Environment Tester: Anyone in community who is able to set up automatic test replication in their work environment – Reviewer: Author of white papers, designers of script targets** ** suggests a quick-start onboarding page in CSS Phusewiki

#PhUSE Qualification Proposal Metadata for scripts should indicate:YML proposal – Whitepaper & output ID uniquely identify the target – Programming language & version – Type of output: – Study design: – State of qualification : – OS is not included, since should be indicated in OS compatibility report Test Data requirements – available as a program or script (text, not binary) – based on expected standards (SDTM, ADaM) – data requirements clearly & accurately specified for each script – include expected results from these data for defined tests/checks

#PhUSE Qualification Proposal Transitions"Contributed" is the original State of all scripts – to Develop, checklist includesby Developer & Reviewer R & D confer on suitability of contribution. Suitable starting point? [ may require analysis details, specs, from contributor ] D reviews components (checklist to be completed) D works with Contributor to complete minimum components [ including Test Data and Coverage of defined tests ] D adds standard parameter, dependency checking D writes Qualification of CSS scripts.xlsx (see template embedded above)Qualification of CSS scripts.xlsx – to Review, checklist includesby Tester Review Qualification instructions, consider coverage of tests Execute Qualification instructions Work with Developer to complete execution successfully

#PhUSE Qualification Proposal Transitionscontinued – to Releasedby Tester & Environment Tester & Reviewer T updates reference test outputs from certification/confirmation E updates & executes local tests (posting PASS/FAIL results) R confirms script output matches intention R confirms qualification process covers important elements and considerations. R also provides user (rather than technical) feedback? Achieve “Released" state when all tests in all test environments PASS (i.e., match outputs that T has certified and/or confirmed) and that R agrees that target is achieved

#PhUSE Qualification Proposal Efforts Required – Top priority Finalize Qualification states, roles, workflow, checklists, and templates Code criteria for "acceptable" (link to GPP, PhUSE)link to GPP, PhUSE Test definition, Test confirmation (also from White Paper team) Test data – Next priorities Design test structure in google code Develop scripts that will allow Environment Testing Develop general components (e.g. parameter, dependency checking) Develop general components Identify Environment Testers based on – Host environment – SAS or R version Identify opportunities to automate qualification. E.g., – Environment Testers to post results back as machine readable – Script green-light/red-light qualification matrix on Phusewiki

#PhUSE