1. Standard Scripts Project 2
Proposal for Qualification of Standard Scripts

2. Main Sections Summary of prior proposal, 2013
Updated proposal, July 2014

3. Main Sections Summary of prior proposal Updated proposal
Concepts, definitions & metadata
Test data considerations
Heavy vs. Light qualification
Updated proposal

4. Proposal from 2013
Anyone can submit a script, according to a check list
Categorize scripts by complexity
Complexity: low, medium, high, software
Output: tabulated data, analysis data, table, figure, listing
Metadata for script should indicate
Type of output: tabulated data, analysis data, table, figure, listing
Study design: parallel, crossover, etc
State of qualification: ?

5. Proposal from 2013 Test data 2 levels of qualification
Overall project should have minimum test data (SDTM & ADaM)
Scripts can propose new test data, must pass (Data fit? Open CDISC?)
Share program to produce test data, never binary test data
2 levels of qualification
Light vs. Heavy: according to complexity of script & output
Common elements include
header
adhere to good programming practices
clear scope of script (e.g., study design(s))
test data matche scope & pass "FDA Data Fit" assessment (Open CDISC?)
documentation available for: usage, inputs, outputs, dependencies

6. Proposal from 2013 Heavy qualification
Beta package includes: minimal elements for contribution
Specification & Documentation (could be in pgm header)
Test data (Data Fit? or Open CDISC?)
Tests & Expected results defined
Peer Review: GPP, Specs & Docn reviewed, Tests reproduced
Draft
Write qualification plan, Review tests for completeness/suitability (e.g., Branch testing – are all conditional blocks/combos tested?)
Test
Peer Review: Write qualification report, incl. log/output from tests
Final

7. Proposal from 2013 Light qualification
Beta package includes skip if >1 yr production use without ERROR
Draft minimal elements for contribution
Specification & Documentation (could be in pgm header)
Test data (Data Fit? or Open CDISC or other, as appropriate)
Tests & Expected results defined
Peer Review: GPP, Specs & Docn reviewed, Tests reproduced
Write qualification plan, Review tests for completeness/suitability (e.g., Branch testing – are all conditional blocks/combos tested?)
Test
Peer Review: Write qualification report, incl. log/output from tests
Final
DDT comment on "light" qualification plan:
I think the suggestion was to replace this with a statement of ERROR-free production use >1 yr prior to contribution to PhUSE Standard Scripts

8. Proposal through CSS 2104 Peer Review Checklist Heavy Light
Requirement specification X ?
Documented or perhaps only documented in header
User Guide
SDTM/ADaM used in input/output
Open CDISC validator or Data Fit used to check input/output
GPP in source
Run according to Requirement specification
Tested by qualification plan, tests & results all Peer reviewed
Tested by End users
Robust without red errors in contributor's production environment
Robust and used in FDA (other) scripts repository, ranked ******
DDT comment on "light" qualification plan:
I think the suggestion was to replace this with a statement of ERROR-free production use >1 yr prior to contribution to PhUSE Standard Scripts

9. Main Sections Summary of prior proposal Updated proposal Objectives
Definitions: Qualification, States, Roles
Metadata and Test data
State Transitions

10. Proposal 2014 - Objectives For End-users
Clear overview of resources available, incl. purpose & state of each
Inspire confidence from first user experience
Ease-of-use:
clear messaging from scripts
Reproducible results in user's own environment
Consistency of scripts, learning first one makes remaining familiar
Ease of converting users to contributors
For Contributors & Standard Scripts Team
Standardize workflows and checklists
Modularize routine components (e.g., FUTS for dependency checking?)
Automate testing, issue identification (e.g., concept similar to Spotfire/R compatibility)
Centralize & consolidate information & results
DDT comment on "Ease of converting users ..."
Contribution should be easy!
And should accommodate the willing contributor, however much or little time they have available.

11. Qualification Proposal meaningful terms in blue http://www. phusewiki
Qualification Instructions (see embedded template ð)
Certification phase of Qualification applies to new scripts and tests
Confirmation phase applies to updates of existing scripts
States: Contributed, Development, Testing, Qualified
Roles
Contributor: Anyone with appropriate skills & interests
Developer: CSS Working Group 5 volunteer familiar with objectives**
Tester: CSS WG 05 volunteer familiar with objectives**
Environment Tester: Anyone in industry community able to set up automatic test replication in their work environment
Reviewer: Author of white papers, designers of script targets**
** suggests a quick-start onboarding page in CSS Phusewiki
Qualification phase vs. State:
First time a script is developed/tested, we perform Certification phase
Upon later updates, developer/tester perform Confirmation phase

12. Qualification Proposal
Metadata for script should indicate
Whitepaper ID & output ID
Programming language & version (e.g., R v3.1.1, SAS v9.4)
Type of output: tabulated data, analysis data, table, figure, listing
Study design: parallel, crossover, etc
State of qualification: Contributed, Development, Testing, Qualified
OS is not included, since should be indicated in OS compatibility report
Test Data requirements
available as a program or script (text, not binary)
based on expected standards (SDTM, ADaM)
data requirements clearly & accurately specified for each script
include expected results from these data for defined tests/checks
"OS is not included ..."
See related concepts of automated testing and Environment Testers, above

13. Qualification Proposal
Transitions "Contributed" is the original State of all scripts
to Development, checklist includes by Developer & Reviewer
R & D confer on suitability of contribution. Suitable starting point? [ may require analysis details, specs, from contributor ]
D reviews components (checklist to be completed)
D works with Contributor to complete minimum components [ including Test Data and Coverage of defined tests ]
D adds standard parameter, dependency checking
D writes Qualification instructions .docx (see template, above)
to Testing, checklist includes by Tester
Review Qualification instructions, consider coverage of tests
Execute Qualification instructions
Work with Developer to complete execution successfully
R & D confer:
This is essentially an investment decision.
Does the contribution warrant development by the Standard Scripts volunteers?
Is it a sufficient starting point?
D reviews components ... partial list:
Clear scope & requirements for target output
(from White Paper?)
Good Programming Practices
Program header
Documentation (just in header?)
Test Data
D adds standard parameter, ...
See for e.g.,
ThotWave may be interested to contribute with FUTS, their Framework for Unit Testing SAS.
We could probably use much of this framework & components
D writes Qualification instructions ...
See Word docx template, embedded above

14. Qualification Proposal
Transitions continued
to Qualified by Tester & Environment Tester & Reviewer
T updates reference test outputs from certification/confirmation
E updates & executes local tests (posting PASS/FAIL results)
R confirms script output matches intention
R confirms qualification process covers important elements and considerations.
R also provides user (rather than technical) feedback?
Achieve "Qualified" state when all tests in all test environments PASS (i.e., match outputs that T has certified and/or confirmed) and that R agrees that target is achieved

15. Qualification Proposal
Efforts Required
Top priority
Finalize Qualification states, roles, workflow, checklists, and templates
Next priorities
Design test structure in google code
Develop scripts that will allow Environment Testing
Develop general components (e.g. parameter, dependency checking)
Identify Environment Testers based on
Host environment
SAS or R version
Identify opportunities to automate qualification. E.g.,
Environment Testers to post results back as machine readable
Script green-light/red-light qualification matrix on Phusewiki