PhUSE Standard Analyses and Code Sharing Working Group All Hands Working Group Meeting 29 November 2018

PhUSE Working Groups Standard Analyses and Code Sharing Working Group Leads: Hanming Tu, Mary Nilsson From the PhUSE Working Groups home page – The Standard Analyses and Code Sharing WG is one of 6 WGs as part of the FDA/PhUSE collaboration.

SACS Working Group Vision Vision: Leverage crowd-sourcing to improve the content and implementation of analyses for medical research, leading to better data interpretations and increased efficiency in the clinical development and review processes SACS = Standard Analyses and Code Sharing

SACS Working Group Goals Establish and maintain a publicly available repository for storing program code to be used as analytical tools for medical research Where gaps exist, develop recommendations for analyses and displays in areas that could benefit from crowd-sourcing Where gaps exist, develop code for recommended analyses and displays that could benefit from crowd-sourcing (to reside in the repository) Note: In early 2019, we should decide whether to add goals to cover our additional project teams

Current Projects Analysis and Display White Papers (ADW) Lead: Mary Nilsson Best Practices for Quality Control and Validation Leads: Maria Dalton and Jane Marrer New Code Sharing (Repository) Lead: Hanming Tu Good Programming Practices Lead: Ninan Luke (Macro Development) New Communication, Promotion, Education (CPE) Leads: Wendy Dobson, Jared Slain Test Data Factory Lead: Peter Schaefer Note: We’re still working out the details around the Good Programming Practices project team. We will likely have more sub-projects. Note: Most meetings occur at the project team and sub-project team level. We rarely meet as a full Working Group. We do have

ADW Project Team Lead: Mary Nilsson FDA Liaison: Nhi Beasley Description: This project includes the development of white papers that provide recommended Tables, Figures, and Listings for clinical trial study reports and submission documents. The intent is to begin the process of developing industry standards with respect to analysis and reporting for measurements that are common across clinical trials and across therapeutic areas. ADW = Analysis and Display White Papers

ADW Project Team Goals Define a set of recommended tables, figures, listings and associated analyses Ensure reviewers receive clinically relevant and meaningful analyses for benefit-risk assessment Harmonization leads to efficiencies in both creation and review of analyses Improve expertise in safety analytics across multiple disciplines involved with planning, interpreting, and report safety analyses Promote good analytical practices, avoid poor practices Facilitate cross-functional engagement in analytical planning ADW = Analysis and Display White Papers Excerpt from the Adverse Events White Paper: The development of standard TFLs and associated analyses will lead to improved standardization of AE data from collection through data display and analysis. The development of standard TFLs will also lead to improved and harmonized product lifecycle management across therapeutic areas by ensuring that reviewers receive clinically relevant and meaningful analyses of patient safety for benefit-risk assessment. This white paper reflects recommendations that would lead to more consistent TFLs, but the recommendations should not be interpreted as “required” by any regulatory agency. Another purpose of this white paper (along with the other white papers from the project team) is to improve expertise in safety analytics across the multiple disciplines involved with planning, interpreting, and reporting safety analyses. Statisticians can and should assist cross-disciplinary teams to understand and reduce bias in analysis planning and reporting. This assistance is important even when inferential statistics are not used. The potential for biased comparisons is especially a concern when multiple studies are combined (eg, via poor pooling practices for integrated summaries). Safety physicians have relied on qualitative analyses of case reports, looking at individual or small clusters of events. Recently, there has been an increased emphasis on aggregate reviews of safety data. As noted in Section VI of the CIOMS Working Group VI report [1], while medical judgment remains critical in the interpretation of safety data, descriptive and inferential statistical methods can help medical personnel decide whether chance variation is a possible explanation for what is observed or whether it is more likely that some genuine drug effect has occurred. This requires statisticians to increase their engagement and help cross-disciplinary safety management teams to think more quantitatively, This also requires nonstatistical disciplines to obtain a higher level of analytical expertise. For example, all members of a safety management team should understand that improper pooling can lead to biased and even paradoxical results. See Table 10.1 for an illustrative example of paradoxical results and see Section 10.1 for other areas where increased expertise is needed.

Vision: Fill the Gap on Analysis and Display Standards Data Collection Systems Observed Datasets Analysis Datasets Tables, Figures and Listings Clinical Data Flow Trial Design PRM SDTM ADaM No Standards Exist Industry Standards Alignment CDASH A lot of progress has been made with respect to standardization – mostly in the collection and data space. There’s a gap with respect to analyses and displays. CFAST = Coalition for Accelerating Standards and Therapies (Therapeutic Areas)

ADW – Final Deliverables All final white papers from the project team are in the Working Group Deliverables Catalog White Papers section ADW = Analysis and Display White Papers

ADW - Timeline for Future Deliverables ADW Project Team – Timeline for Future Deliverables

ADW Project Team – Get Involved Co-author a white paper Medical and statistical participants Approximately 8 hours a month for 6 months Participate in reviews Medical and statistical reviewers across business units Approximately 2 hours Help recruit reviewers Help with action items (follow-up tasks) where relevant Communicate the availability of the white papers Promote following the recommendations in the white papers If you receive push-back from internal colleagues or from a regulator, let us know Reference white papers in Statistical Analysis Plans ADW = Analysis and Display White Papers

Code Sharing (Repository) Project Team Lead: Hanming Tu FDA Liaison: Mat Soukup Description: Establish and maintain a collaboration platform for leveraging crowd-sourcing to improve the content and implementation of analyses for medical research and leading to better data interpretations and increased efficiency in the clinical drug development and review processes.

Code Sharing Project Goals Establish and maintain a publicly available repository for storing program codes to be used as analytical tools for medical research Develop guidance on managing repositories and script metadata for sharing scripts Improve frontend for accessing and searching the repository Review contributed scripts such as FDA scripts Develop scripts based on white papers

Vision: Open Tool Box Shared tools Text Shared Value Code Sharing (Repository) Project Team

Vision: Script Repository FDA Industry Academia Script Repository in Github Shared tools Script Metadata Test Data SAS, R, Spotfire, Etc. White Paper Scripts Contributed Scripts Code Test Environment Code Sharing (Repository) Project Team

Code Sharing – Final Deliverables Script Repository in Github created Qualification guidelines created Scripts from other groups have been contributed FDA, Non-clinical, Data handling, Spotfire safety templates Scripts from within the project have been created Tables and figures from the 2013 labs/vitals/ECGs white paper White paper showing the displays created from the FDA-contributed scripts FDA: https://github.com/phuse-org/phuse-scripts/wiki/JumpStart-Scripts Non-clinical: https://github.com/phuse-org/phuse-scripts/tree/master/contributed/Nonclinical Data Handle: https://github.com/phuse-org/phuse-scripts/tree/master/lang/SAS/datahandle Spotfire Templates: https://github.com/phuse-org/phuse-scripts/tree/master/contributed/Spotfire

Code Sharing – Timeline for Future Deliverables Script Metadata white paper – Currently under public review The review period closes 13th December More code creation planned for 2019 Will likely focus on tables and figures that are recommended in a white paper but not part of predominant practice today

Code Sharing – Get Involved Participate in the public review of the Script Metadata white paper Go to phuse.eu, click on the Resources tab, click on Working Group Deliverables, click on Deliverables Under Review Share existing code Create code Use code from the repository Help identify areas where crowd-sourcing the creation of a shared tool makes sense Interactive safety review? Reduce some common errors in e-submission?

Communication, Promotion, Education (CPE) Project Team Leads: Wendy Dobson, Jared Slain FDA Liaison: Mat Soukup, Susan Duke Description: The success of working group “Standard Analyses and Code Sharing” relies on the acceptance, input, feedback and further development from/by the user community. A main priority will be to develop a Communication Plan that conceptualizes efficient ways to communicate working group progress and results, e.g. white papers, and the call for scripts. It will define target groups, timing, communication channels, and the presentation.

CPE Project Goals Facilitate communication of deliverables Improve navigation to information Facilitate presentations/posters at multiple conferences Keep wiki up-to-date Facilitate creation of educational video clips Study-size adjusted percentages completed Help define places to mention deliverables Write journal article(s) Create a Safety Analytics site Host a Safety Analytics workshop Crowd-source educational library

CPE Project – Final Deliverables Multiple presentations at various conferences Educational video on study-size adjusted percentages

CPE Project – Timeline for Future Deliverables Safety Analytics Educational Workshop – CSS 2019 DIA presentation – June 2019 Journal article – targeting end of 2019 Additional educational video clips – throughout 2019 and beyond Safety Analytics education library - Create front-end in early 2019

CPE Project – Get Involved Help with communication Help with improving navigation Volunteer to be a liaison with other groups Present at professional meetings where PhUSE work can fit Even a couple of slides would help! Help with crowd-sourcing educational information Primarily interested in safety analysis educational information Share existing educational material Help create missing material

Best Practices for Quality Control and Validation Project Team New Leads: Maria Dalton and Jane Marrer FDA Liaison: TBD Description: Quality and accuracy is essential in the health and life sciences industries. Patients and regulators must be able to trust analyses of clinical data. A white paper will be created to provide an in-depth review of best practices for robust quality control. The scope of the paper will be quality control of analysis programming of clinical data in health and life sciences organisations. It is applicable to the organisations who produce analyses of clinical data, including Contract Research Organisations. The paper will not discuss oversight of outsourced programming.

Quality Control and Validation – Get Involved Join the project team Co-author sections of the white paper Review the white paper Recruit project team members/reviewers

Good Programming Practices (GPP) Project Team New Lead: Ninan Luke (Macro Development) FDA Liaison: TBD Description: Macros provide an effective way to automate and reuse code in a standard and consistent manner across SAS programs. This ability to reuse code means that the use of GPP is particularly important in macro code and we think that there is a need to develop a consensus and document good programming prairies specifically for macro programming. Note: We will likely have more GPP-related sub-projects. Macro development will likely become a sub-project within a broader project team.

Good Programming Practices – Get Involved Join the project team Co-author deliverable Review deliverable Recruit project team members/reviewers Propose a sub-project related to GPP

Test Data Factory (TDF) Project Team Lead: Peter Schaefer CDISC Liaison: TBD Description: Several CS Projects develop and specify medical research methods, features, or processes, and some even create software components or subsystems for common tasks in drug development. As part of these efforts, a variety of SDTM or ADaM test datasets are required. The typical fallback position of project teams is to use data from the CDISC pilot project and/or anonymized study data that are provided by project team members. The Test Data Factory project aims at providing test data formatted in SDTM and ADaM that support a more systematic and comprehensive testing of these concepts and scripts. Note: We will likely have more GPP-related sub-projects. Macro development will likely become a sub-project within a broader project team.

Test Data Factory – Deliverables Initial Approach: Start with existing CDISC pilot datasets, and update them to comply with more up-to-date SDTM standards Reasoning: Gain experience, provide some results sooner rather than later Completed update of SDTM and ADaM CDISC Pilot datasets The datasets and related documentation are in final review. Should be released before the end of the year Next step: Implement scripts/applications that generate Test Data based on user input for specific testing tasks.

Test Data Factory – Get Involved Join the project team Review SDTM and ADaM test data packages when out for public review Recruit project team members/reviewers Volunteer to help with the next phase of the project – simulated test datasets

How to Participate Sign up for the PhUSE working group mailings From phusewiki.org, click “Join a Working Group Now” Standard Scripts Groups CSS-WG-Standard-Scripts (Entire Working Group) CSS-WG-Standard-Scripts-WhitePapers (White Paper Project Team) CSS-WG-Standard-Scripts-Platform (Script Repository) CSS-WG-SS-WhitePaperReviewers (Notified when a white paper is ready for review) See wiki pages for all projects (www.phusewiki.org) Contact any of the WG or project leads Note: We’ll likely need additional group mailing lists for the new projects