WG5 P02 Proposal2014 Qualification of Standard ScriptsStandard Scripts

Proposal through CSS Anyone should be able to submit a script, according to a check list Categorize scripts according to complexity –Complexity:low, medium, high, software –Output:tabulated data, analysis data, table, figure, listing Metadata for script should indicate –Type of output:tabulated data, analysis data, table, figure, listing –Study design:parallel, crossover, etc –State of qualification:?

Proposal through CSS 2104 Test data –Overall project should have minimum test data (SDTM & ADaM) –Scripts can propose new test data, must pass (Data fit? Open CDISC?)Open CDISC –Share program to produce test data, never binary test data 2 levels of qualification to match script complexity/output –Light vs. Heavy qualification –Common elements include header good programming practices clearly declared scope of script (e.g., study design(s)) test data matches scope & passes "FDA Data Fit" assessment (?) documentation inputs/outputs/dependencies/usage

Proposal through CSS 2104 Heavy qualification –Beta package includesminimal elements for contribution Specification & Documentation (could be in pgm header) Test data (Data Fit? or Open CDISC or other, as appropriate) Tests & Expected results defined Peer Review: GPP, Specs & Docn reviewed, Tests reproducedGPP –Draft Write qualification plan, Review tests for completeness/suitability (e.g., Branch testing – are all conditional blocks/combos tested?) –Test Peer Review: Write qualification report, incl. log/output from tests –Final

Proposal through CSS 2104 Light qualification –Beta package includesskip if >1 yr production use without ERROR –Draftminimal elements for contribution Specification & Documentation (could be in pgm header) Test data (Data Fit? or Open CDISC or other, as appropriate) Tests & Expected results defined Peer Review: GPP, Specs & Docn reviewed, Tests reproducedGPP Write qualification plan, Review tests for completeness/suitability (e.g., Branch testing – are all conditional blocks/combos tested?) –Test Peer Review: Write qualification report, incl. log/output from tests –Final

Proposal through CSS 2104 Peer Review ChecklistHeavyLight Requirement specificationX? Documented or perhaps only documented in headerX User GuideXX SDTM/ADaM used in input/outputXX Open CDISC validator or Data Fit used to check input/outputXX GPPGPP in sourceXX Run according to Requirement specificationX? Tested by qualification plan, tests & results all Peer reviewedX? Tested by End usersX? Robust without red errors in contributor's production environment XX Robust and used in FDA (other) scripts repository, ranked ******X

Proposalmeaningful terms in blue Qualification Certification phase of Qualification applies to new scripts and tests Confirmation phase applies to updates of existing scripts States:Contributed, Development, Testing, Qualified Roles –Contributor: Anyone with appropriate skills & interests –Developer: CSS Working Group 5 volunteer** familiar with objectives –Tester: CSS WG 05 volunteer** –Environment Tester: Anyone in industry community able to set up automatic test replication in their work environment –Reviewer: Author of white papers, designers of script targets ** suggests an quick-start onboarding page in CSS Phusewiki

Proposal Qualification Metadata for script should indicate –Whitepaper & output ID –Programming language & version (e.g., R 3.1.1) –Type of output:tabulated data, analysis data, table, figure, listing –Study design:parallel, crossover, etc –State of qualification:Contributed, Development, Testing, Qualified

Proposal Qualification End-user Objectives –Clear overview of purpose and resources –Inspire confidence from first sight –Ease of use, clear messaging from first run –Reproducible results –Consistency of scripts, learning first one makes remaining familiar –Ease of converting users to contributors Contributor Objectives –Standardize routine steps –Modularize routine components –Automate testing, issue identification –Centralize & consolidate information & results

Proposal Qualification TransitionsContributed is the original State of all scripts –to Development, checklist includesby Developer & Reviewer R confirms contributed output matches/approximates target [ may require analysis details, specs, from contributor ] D reviews components D works with Contributor to complete minimum components [ including Test Data and Coverage of defined tests ] D adds standard parameter, dependency checking D writes Qualification instructions.docx (see template, above)Qualification instructions.docx –to Testing, checklist includesby Tester Review Qualification instructions, consider coverage of tests Execute Qualification instructions Work with Developer to complete execution successfully

Proposal Qualification Transitionscontinued –to Qualifiedby Tester & Environment Tester & Reviewer T updates posted test outputs from certification/confirmation E updates local tests and executes (posting PASS/FAIL results) R confirms script output matches intention & qualification process covers important elements and considerations. Also provides user (rather than technical) feedback? Achieve "Qualified" state when all tests in all test environments PASS (i.e., match outputs that T has certified and/or confirmed) and that R agrees that target is achieved

Proposal Qualification Efforts Required –Finalize Qualification states, roles, workflow, checklists, and templates –Design test structure in google code –Develop scripts that will allow Environment Testing –Develop general components (e.g. parameter, dependency checking)Develop general components –Identify Environment Testers based on Host environment SAS or R version –Identify opportunities to automate qualification. E.g., Docx format for Qualification instructions is not easily machine readable Environment Testers to post results back as machine readable Script green-light/red-light qualification matrix on Phusewiki

Doing now what patients need next