Standard Script All-Hands meeting September 29, 2014 1.

Slides:



Advertisements
Similar presentations
GEOSS Data Sharing Principles. GEOSS 10-Year Implementation Plan 5.4 Data Sharing The societal benefits of Earth observations cannot be achieved without.
Advertisements

Test Automation Success: Choosing the Right People & Process
TITLE OF PROJECT PROPOSAL NUMBER Principal Investigator PI’s Organization ESTCP Selection Meeting DATE.
Fit to Learn Using the Employability Skills Framework to improve your performance at College The Employability Skills Framework has been developed by business.
Certified Business Process Professional (CBPP®)
TS16949 requirements Subjects –Audit planning –Recertification audit requirements –Auditing Remote supporting functions.
FDA scripts. Validation of script/programs. Heavy and light validation Check list FDA scripts Basis for discussion FDA: WG5 Project 02 18/8/2015 WG5 Project.
#PhUSE Standard Scripts Project Proposal for Qualification of Standard Scripts.
Bay Area CDISC Implmentation Network – July 13, 2009 How a New CDISC Domain is Made Carey Smoak Team Leader CDISC SDTM Device Team.
JumpStart the Regulatory Review: Applying the Right Tools at the Right Time to the Right Audience Lilliam Rosario, Ph.D. Director Office of Computational.
WG5 P02 Proposal2014 Qualification of Standard ScriptsStandard Scripts.
WG5 P02 Proposal2014 Qualification of Standard ScriptsStandard Scripts.
Standard Script All-Hands meeting September 29,
Standard Script All-Hands meeting September 29,
Development of Standard Scripts for Analysis and Programming Working Group – White Papers Project PhUSE Webinar January 2015 Mary Nilsson 1.
Qualification Process for Standard Scripts Hosted in the Open Source Repository ABSTRACT Dante Di Tommaso 1 and Hanming Tu 2 Tehran 1 F. Hoffmann-La Roche.
MODULE B: Case Report Forms Jane Fendl & Denise Thwing April 7, Version: Final 07-Apr-2010.
Encounter Data Validation: Review and Project Update August 25, 2015 Presenters: Amy Kearney, BA Director, Research and Analysis Team Thomas Miller, MA.
Second Annual Japan CDISC Group (JCG) Meeting 28 January 2004 Julie Evans Director, Technical Services.
PhUSE Computational Science Working Groups Solutions Through Collaboration.
InWEnt | Qualified to shape the future1 Internet based Human Resource Development Management Platform Human Resource Development Programme in Natural Disaster.
1 What’s Next for Financial Management Line of Business (FMLoB)? AGA/GWSCPA 6 th Annual Conference Dianne Copeland, Director, FSIO May 8, 2007.
Management & Development of Complex Projects Course Code MS Project Management Perform Qualitative Risk Analysis Lecture # 25.
Current and Future Applications of the Generic Statistical Business Process Model at Statistics Canada Laurie Reedman and Claude Julien May 5, 2010.
WG4: Standards Implementation Issues with CDISC Data Models Data Guide Subteam Summary of Review of Proposed Templates and Next Steps July 23, 2012.
Optimizing Data Standards Working Group Meeting Summary
Draft Transition Plan for the Transfer of the Drug Medi-Cal Treatment Program Fourth Series: Stakeholder Meetings Department of Health Care Services Department.
Standard Scripts - Project 2 Proposal for Qualification July 2014 Project 2 Update.
Fundamentals of Information Systems, Second Edition 1 Systems Development.
Health eDecisions Use Case 2: CDS Guidance Service Strawman of Core Concepts Use Case 2 1.
Development of Standard Scripts for Analysis and Programming PhUSE Annual Conference October 2014 Mike Carniello 1.
FDA / PhUSE CSS Initiative. Purpose The FDA wishes to establish collaborative working groups to address current challenges related to the access and review.
Slide: 1 CEOS SIT Technical Workshop |Caltech, Pasadena, California, USA| September 2013 CEOS Work Plan Section 6.1 G Dyke CEOS ad hoc Working Group.
PhUSE Computational Science Working Group Update 24 November 2015 Development of Standard Scripts for Analysis and Programming.
White Papers to Fill the Gaps of Standardization of Tables, Figures, and Listings (Creating Standard Targets) Analyses and Displays for Vital Signs, ECG,
Emerging Technologies Semantic Web and Data Integration This meeting will start at 5 min past the hour As a reminder, please place your phone on mute unless.
Oman College of Management and Technology Course – MM Topic 7 Production and Distribution of Multimedia Titles CS/MIS Department.
Development of Standard Scripts for Analysis and Programming 17 March 2014 PhUSE Computational Science Symposium Conference Summary.
PhUSE Computational Science Working Groups Solutions Through Collaboration.
WG5 P02 Proposal2014 Qualification of Standard ScriptsStandard Scripts.
Lead from the front Texas Nodal 1 Early Delivery System Testing Environments & Planning – Involving Market Participants TPTF May.
How Good is Your SDTM Data? Perspectives from JumpStart Mary Doi, M.D., M.S. Office of Computational Science Office of Translational Sciences Center for.
Standard Design Process Overview
The PhUSE Therapeutic Area Wiki Page Angelo Tinazzi 1, Oliver Wirtz 2, Christian Mueller 3, Sascha Ahrweiler 2 1 Cytel Inc, Geneva (Switzerland), 2 UCB.
Advanced Higher Computing Science
Principal Investigator ESTCP Selection Meeting
Principal Investigator ESTCP Selection Meeting
GEOSS Data Sharing Principles
Standard Analyses and Code Sharing Summary for the Best Practices for Data Collection Instructions Project Team Mary Nilsson 20 March 2017.
Traceability between SDTM and ADaM converted analysis datasets
Standard Analyses and Code Sharing Working Group
In-Depth Report from Optimizing Data Standards Working Group
PhUSE Computational Science
Standard Scripts Project 2
PhUSE Computational Science Working Groups
Standard Analyses and Code Sharing
Overview of the Standard Analyses & Code Sharing Working Group
SDTM and ADaM Implementation FAQ
An Update on the CS Standard Analyses and Code Sharing Working Group Nancy Brucken, Principal Statistical Programmer, Syneos Health; Jared Slain, Associate.
Standard Analyses and Code Sharing
Nonclinical Working Group Update CSS 2014
Development of Standard Scripts for Analysis and Programming
Standard Scripts Project 2
WG5 P02 Proposal 2014 Qualification of Standard Scripts
Standard Scripts Project 2
Safety Analytics Workshop – Computational Science Symposium 2019
PhUSE Standard Analyses and Code Sharing Working Group
Standard Scripts Project 2
WG5 P02 Proposal 2014 Qualification of Standard Scripts
Optimzing the Use of Data Standards Calling for Volunteers
Presentation transcript:

Standard Script All-Hands meeting September 29,

Computational Science Symposium (CSS) Working Groups – 5th year in a Series of Annual Computational Science Meetings established to support the work of FDA’s Center for Drug Evaluation and Research (CDER), Office of Computational Science (OSC) – 3rd Year working with PhUSE (first two were with the DIA) – FDA’s motivation: To work collaboratively with industry, vendors and regulators in a “non-competitive space” to find solutions and advance the computational sciences/technology associated with the development and regulation of new medical products (drugs, biologics, etc.) – Perceived benefits of collaboration and crowd sourcing PhUSE CSS Background 2

PhUSE CSS Working Groups Optimizing Use of Data Standards Standard Scripts for Analysis and Reporting White Papers on Analysis and Displays Script Repository (4 projects) Communications Emerging Technologies Semantic Technologies Non-Clinical 3

Summary of Standardization Data Collection Systems Observed Datasets Analysis Datasets Tables, Figures and Listings Clinical Data Flow Trial Design PRM SDTM ADaM No TFL Stds Exist Industry Standards Alignment CDASH ? TransCelerate (TAs)

Script Repository

P01: Look for existing scripts and store them in the repository - Adrienne Bonwick and Aiyu Li P02: Define qualification steps for scripts in the repository - Dante Di Tommaso P03: Maintain and enhance platform (repository) for sharing scripts - Mike Carniello/Hanming Tu P04: Legal ownership and issues in open source repository - Sally Cassells P05: Create templates and metadata for documenting scripts and coding practices - Jean-Marc Ferran/Eric Sun P06: Refine process for creating and editing scripts in Google Code P07: Implement and further develop communication plan for standard scripts - Dirk Spruck P08: Create white papers providing recommended display and analysis including Table, List and Figure shells - Mary Nilsson Projects:

Remit: Look for existing scripts and store them in the repository Progress Received the “Demographics” package from the FDA Reviewed and updated to work on any PC SAS package using FDA Pilot Data Provided working instructions Added to Code Repository Starting to investigate MAED P01: Adrienne Bonwick and Aiyu Li

Standard Scripts Project Proposal for Qualification of Standard Scripts

Main Sections  Summary of prior proposal, 2013  Updated proposal, July 2014

Main Sections  Summary of prior proposal – Concepts, definitions & metadata – Test data considerations – Heavy vs. Light qualification  Updated proposal

Proposal from 2013 Anyone can submit a script, according to a check list Categorize scripts by complexity – Complexity: low, medium, high, software – Output: tabulated data, analysis data, table, figure, listing Metadata for script should indicate – Type of output: tabulated data, analysis data, table, figure, listing – Study design: parallel, crossover, etc – State of qualification:?

Proposal from 2013 Test data – Overall project should have minimum test data (SDTM & ADaM) – Scripts can propose new test data, must pass (Data fit? Open CDISC?)Open CDISC – Share program to produce test data, never binary test data 2 levels of qualification – Light vs. Heavy: according to complexity of script & output – Common elements include header adhere to good programming practicesgood programming practices clear scope of script (e.g., study design(s)) test data matche scope & pass "FDA Data Fit" assessment (Open CDISC?) documentation available for: usage, inputs, outputs, dependencies

Proposal from 2013 Heavy qualification – Beta package includes:minimal elements for contribution Specification & Documentation (could be in pgm header) Test data (Data Fit? or Open CDISC?) Tests & Expected results defined Peer Review: GPP, Specs & Docn reviewed, Tests reproducedGPP – Draft Write qualification plan, Review tests for completeness/suitability (e.g., Branch testing – are all conditional blocks/combos tested?) – Test Peer Review: Write qualification report, incl. log/output from tests – Final

Proposal from 2013 Light qualification – Beta package includesskip if >1 yr production use without ERROR – Draftminimal elements for contribution Specification & Documentation (could be in pgm header) Test data (Data Fit? or Open CDISC or other, as appropriate) Tests & Expected results defined Peer Review: GPP, Specs & Docn reviewed, Tests reproducedGPP Write qualification plan, Review tests for completeness/suitability (e.g., Branch testing – are all conditional blocks/combos tested?) – Test Peer Review: Write qualification report, incl. log/output from tests – Final

Proposal through CSS 2104 Peer Review ChecklistHeavyLight Requirement specificationX? Documented or perhaps only documented in headerX User GuideXX SDTM/ADaM used in input/outputXX Open CDISC validator or Data Fit used to check input/outputXX GPPGPP in sourceXX Run according to Requirement specificationX? Tested by qualification plan, tests & results all Peer reviewedX? Tested by End usersX? Robust without red errors in contributor's production environmentXX Robust and used in FDA (other) scripts repository, ranked ******X

Main Sections  Summary of prior proposal  Updated proposal – Objectives – Definitions: Qualification, States, Roles – Metadata and Test data – State Transitions

Proposal Objectives For End-users – Clear overview of resources available, incl. purpose & state of each – Inspire confidence from first user experience – Ease-of-use: clear messaging from scripts Reproducible results in user's own environment Consistency of scripts, learning first one makes remaining familiar – Ease of converting users to contributors For Contributors & Standard Scripts Team – Standardize workflows and checklists – Modularize routine components (e.g., FUTS for dependency checking?)FUTS – Automate testing, issue identification (e.g., concept similar to Spotfire/R compatibility ) concept similar to Spotfire/R compatibility – Centralize & consolidate information & results

Qualification Proposal meaningful terms in blue Qualification Instructions (see embedded template  ) – Certification phase of Qualification applies to new scripts and tests – Confirmation phase applies to updates of existing scripts States:Contributed, Development, Testing, Qualified Roles – Contributor: Anyone with appropriate skills & interests – Developer: CSS Working Group 5 volunteer familiar with objectives** – Tester: CSS WG 05 volunteer familiar with objectives** – Environment Tester: Anyone in industry community able to set up automatic test replication in their work environment – Reviewer: Author of white papers, designers of script targets** ** suggests a quick-start onboarding page in CSS Phusewiki

Qualification Proposal Metadata for scripts should indicate:YML proposal – Whitepaper & output ID uniquely identify the target – Programming language & version – Type of output: – Study design: – State of qualification : – OS is not included, since should be indicated in OS compatibility report Test Data requirements – available as a program or script (text, not binary) – based on expected standards (SDTM, ADaM) – data requirements clearly & accurately specified for each script – include expected results from these data for defined tests/checks

Qualification Proposal Transitions"Contributed" is the original State of all scripts – to Development, checklist includesby Developer & Reviewer R & D confer on suitability of contribution. Suitable starting point? [ may require analysis details, specs, from contributor ] D reviews components (checklist to be completed) D works with Contributor to complete minimum components [ including Test Data and Coverage of defined tests ] D adds standard parameter, dependency checking D writes Qualification instructions.docx (see template, above)Qualification instructions.docx – to Testing, checklist includesby Tester Review Qualification instructions, consider coverage of tests Execute Qualification instructions Work with Developer to complete execution successfully

Qualification Proposal Transitionscontinued – to Qualifiedby Tester & Environment Tester & Reviewer T updates reference test outputs from certification/confirmation E updates & executes local tests (posting PASS/FAIL results) R confirms script output matches intention R confirms qualification process covers important elements and considerations. R also provides user (rather than technical) feedback? Achieve "Qualified" state when all tests in all test environments PASS (i.e., match outputs that T has certified and/or confirmed) and that R agrees that target is achieved

Qualification Proposal Efforts Required – Top priority Finalize Qualification states, roles, workflow, checklists, and templates – Next priorities Design test structure in google code Develop scripts that will allow Environment Testing Develop general components (e.g. parameter, dependency checking) Develop general components Identify Environment Testers based on – Host environment – SAS or R version Identify opportunities to automate qualification. E.g., – Environment Testers to post results back as machine readable – Script green-light/red-light qualification matrix on Phusewiki

Remit: Maintain and enhance script repository Progress CSS 2014 Scriptathon work catalogued and displayed (see next slide) PharmaSUG Scriptathon work to be added to repository PhUSE 2014 Scriptathon planned What’s next? Do we use Scriptathons as vehicle for adding code? Should we spend effort in updating/perfecting what we have? Scriptathon targets are White Paper displays … should we add SDTM and ADaM scripts as well? P03: Hanming Tu and Mike Carniello

P04: Legal ownership and issues in the Open Source Repository Summary 16 Sep 2014

Background Members of WG5, the PhUSE Standard Script Development group, wanted to ensure that users of posted scripts are protected from any potential claims of ownership from organizations employing individuals who contribute scripts. – The group recommended that a ‘click through’ agreement be obtained from a lawyer and posted on the GoogleCode site. This became P04 for our group.

Assignment March 2013 Get more scripts in the Script Repository – Networking – Broad Communications Address any remaining validation, legal, process issues Finalize first white paper – Goal to finalize within a couple of months Begin work on the remaining white papers – Looking for more active participants – Goal to be final or near-final by next year

Summary of Next Steps (2013) The work has now been divided into 8 projects – P01: Look for existing scripts and store them in the repository (Kevin Kane) – P02: Define validation steps for scripts in the repository (Lina Jørgensen) – P03: Maintain and enhance platform (repository) for sharing scripts (Hanming Tu) – P04: Legal ownership and issues in open source repository (Kevin Kane/Sally Cassells)

Summary of Next Steps (2013) 8 projects (cont’d) – P05: Create templates for documenting scripts and coding practices (Jean-Marc Ferran) – P06: Refine process for creating and editing scripts in Google Code (Kevin Kane) – P07: Implement and further develop communication plan for standard scripts (Dirk Spruck) – P08: Create white papers providing recommended display and analysis including Table, List and Figure shells (Mary Nilsson)

Actions for P04 Sally contacted Chris Decker to request that PhUSE provide a legal contact who could develop the agreement. Chris provided draft agreement in May 2013 for review. The agreement is currently posted on the GoogleCode site and in the PhUSE Wiki. The document name is SS UKGROUPS Harmony Individual Contributor License AgreementDOC.

Issues, Discussions, Decisions UK vs. US – The Draft Agreement cites UK law rather than US (Section 8). Adding similar clauses for the US would require the services of a US lawyer. Individually owned tools – Contributor grants full ’worldwide, non exclusive, royalty free, irrevocable license to the IP’. – If tool was developed collaboratively, contributor should get agreement from collaborators before posting. Click through – Not yet technically click through but contributors can read and agree to the terms.

Next Steps, Recommendations Investigate possibility of modifying the language in Section 8 to indicate that the agreement applies fully within the US. Consider a leadership team vote to finalize the draft agreement. Link to agreement PhUSE Standard Scripts Contributor AgreementPhUSE Standard Scripts Contributor Agreement

Project Description Create white papers providing recommended displays and analyses including Table, Figure, and Listing shells for clinical trial study reports and summary documents. The intent is to begin the process of developing industry standards with respect to analysis and reporting for measurements that are common across clinical trials and across therapeutic areas. Script developers could then create scripts consistent with the recommendations for all to use, improving efficiency and safety signal detection. P08: Mary Nilsson

Progress - The following white papers have been finalized: Analyses and Displays Associated with Measures of Central Tendency – With a Focus on Vitals, ECGs, and Labs in Phase 2-4 Clinical Trials and Integrated Summary Documents (finalized in October 2013) Analyses and Displays Associated with Non-Compartmental Pharmacokinetics – With a Focus on Clinical Trials (finalized in March 2014) Note: Final white papers can be found on the PhUSE website, Publications section. ( P08: Mary Nilsson

Progress - The following white papers are in progress: Analyses and Displays Associated with Outliers or Shifts from Normal to Abnormal – With a Focus on Vitals, ECGs, and Labs in Phase 2-4 Clinical Trials and Integrated Summary Documents (third draft targeted for public comment in September 2014) Analyses and Displays Associated with Adverse Events – With a Focus on Phase 2-4 Clinical Trials and Integrated Summary Documents (first draft targeted for public comment in September 2014) Analyses and Displays Associated with Demographics, Disposition, and Medications – With a Focus on Phase 2-4 Clinical Trials and Integrated Summary Documents (final white paper targeted for October 2014) Analyses and Displays Associated with Hepatotoxicity – With a Focus on Phase 2-4 Clinical Trials and Integrated Summary Documents (first draft targeted for public comment by the end of 2014) Analyses and Displays Associated with QT Studies (first draft targeted for public comment by the end of 2014) Note: Draft white papers (where applicable) can be found on the PhUSE Wiki, Development of Standard Scripts Project 08 wiki page ( P08: Mary Nilsson

How you can help – When a white paper has been finalized, or when a white paper is ready for broad review, please help spread the word within your organization and across your contacts. Please include your medical colleagues. Thanks! P08: Mary Nilsson