Improving PCOR Methods: Causal Inference Jason Gerson, PhD Program for Clinical Effectiveness and Decision Science CIMPOD 2017 February 27, 2017

Improving PCOR Methods: Why Methods Matter PCORI was created to support research that provides high-integrity, evidence-based information to patients, clinicians, and the broader healthcare community. PCORI’s Mission Rigorous research methods are needed to produce relevant, trustworthy findings that can improve patients’ healthcare outcomes. Why Methods Matter Methods include systematic processes, designs, tools, and techniques used to generate the evidence needed to answer questions about which healthcare options work best for particular patients. Methods for PCOR

Improving PCOR Methods: Program Goals Identify methodological gaps relevant to the conduct of PCOR Identify Methods Gaps Fund high impact studies which address gaps in methodological research Fund Research Disseminate and facilitate the adoption of new methods to improve the conduct of PCOR Disseminate Promising/Best Practices

Methods Program Overview 92 funded projects, $93 million awarded, $750,000/3 years 29 states, DC, and Quebec Methods for Patient and Stakeholder Engagement Methods for Patient-Centered Outcomes and Patient-Reported Outcomes Research Related to the Ethical Conduct of PCOR/CER Methods to Improve Study Design Methods to Improve Validity and Efficiency of Analyses Methods to Support Data Research Networks

Importance of CI Methods for PCOR/CER Many important clinical CER questions cannot be answered with RCTs (impossible/undesirable/unethical) – need for observational studies and to support improvements in analytic methods. Comparison of complex treatment strategies sustained over long periods of time using data collected from routine clinical practice (or EHRs). Need to “transport” findings from our funded methodological projects to other parts of the PCORI portfolio, esp. large pragmatic studies and PCORnet. Intent of our investments in this area is to provide guidance/ recommendations to research community about what CI approaches are more or less optimal for a particular problem– toolkits, decision trees, standards, best practices.

Methods for Comparative Effectiveness and Safety Analyses in a High-Dimensional Covariate Space with Few Events Potential Impact Could change PCOR by improving the ability to evaluate treatments soon after they are made available, evaluate treatment effects in patient subgroups, and evaluate treatments for rare diseases Engagement Patients participated in preliminary interviews, and an advisory panel of 10 patients and other stakeholder representatives will meet five times per year to provide feedback Methods Evaluate the analytic strategies through simulation studies and apply analytic methods to three real-world patient-centered outcomes research sample studies Evaluates and improves analytic strategies for nonrandomized studies with few outcome events and many potential confounders, such as studies of treatment effects in patient subgroups and studies of treatments for rare diseases. Jessica M. Franklin, PhD Brigham and Women’s Hospital Boston, MA CER Methods and Infrastructure Cycle: Inaugural Methods awarded September 2013

Causal analyses of electronic health record data for assessing the comparative effectiveness of treatment regimens Potential Impact Could change PCOR by improving existing causal inference methods for comparing complex treatment strategies and informing decision making by clinicians and patients for the management of chronic conditions Engagement A multi-stakeholder committee will inform and prioritize the development and dissemination of the proposed statistical methods Methods Simulation studies to evaluate the statistical methods; application of the methods to analyze EHR data from a recent observational study of type II diabetes Advances and adapts existing causal inference methods to improve evaluation of realistic adaptive treatment strategies that reflect real-world adherence to treatment decisions and recommended monitoring schedules. Romain Neugebauer, PhD Kaiser Foundation Research Institute Oakland, CA CER Methods and Infrastructure, awarded September 2014

Learning within Health Care Delivery Systems: Design, Analysis, and Interpretation of Longitudinal Cluster Randomized Trials Potential Impact Improves methods for the design and analysis of stepped wedge designs for the longitudinal study of clusters (e.g., providers or clinics), including analysis of heterogeneity of treatment effects (HTE) Engagement Advisory committee and engagement with clinical researchers and statisticians (including the NIH Collaboratory) Methods Theoretical development, simulation studies, and empirical analyses to evaluate the performance of methods Development and validation of open- source software to assist in the design of stepped wedge trials Develops a causal inference framework, operational knowledge, and study-planning tools necessary to design and conduct longitudinal studies of novel heath care delivery methods within a collection of delivery systems. Patrick Heagerty, PhD, MS, University of Washington Seattle, WA CER Methods and Infrastructure Cycle 2 2015 (Fall 2015) awarded April 2016

Patient Centered Adaptive Treatment Strategies (PCATS) using Bayesian Causal Inference Potential Impact Could change PCOR by improving methods for analyzing data in cases where the treatment/exposures varies over time and improve clinical practice by developing evidence-based shared decision making tools for identifying optimal PCATS at the point-of- care. Engagement A stakeholder advisory panel including two parent representatives, and a partnership with PR-COIN (part of PCORnet) will ensure patient and stakeholder engagement throughout the research process, including in the design, conduct, analysis, and dissemination. Methods Bayesian modeling strategies, simulation studies, and empirical analyses of PCATS for polyarticular Juvenile Idiopathic Arthritis (pJIA). Develops Bayesian double robust causal inference methods for use in analyzing large registry and electronic health records to evaluate the clinical effectiveness of treatments and inform patient centered adaptive treatment strategies (PCATS). Bin Huang, PhD Cincinnati Children’s Hospital Medical Center Cincinnati, OH CER Methods and Infrastructure, awarded May 2015

Modeling Strategies for Observational CER: What Works Best When? Potential Impact Could change PCOR by helping researchers select the best method(s) for a particular observational dataset and CER question Methods Systematic review of existing literature on relevant analytical methods; simulations to assess performance of methods in cases where the literature is insufficient Develops a decision tool for use in selecting the most appropriate analytical methods to control bias from treatment self-selection (“confounding by indication”) in analyses of observational data for CER. Douglas Landsittel, PhD University of Pittsburgh Pittsburgh, PA CER Methods and Infrastructure, awarded December 2013

Causal Inference Guidelines for Pragmatic Clinical Trials Potential Impact Provides investigators with better tools to estimate patient-centered effects in pragmatic clinical trials. Engagement Patients and stakeholders will be engaged in identifying the effects of interest in pragmatic trials and developing standards for estimating these effects. Methods Systematic review, focus groups, and case studies with supporting software will be used to create and implement a revised framework for the design and conduct of randomized pragmatic trials. Develops a set of causal inference standards specifically designed for the analysis of pragmatic clinical trials. Miguel Hernan, MPH, MD, DPH Harvard University School of Public Health Boston, MA CER Methods and Infrastructure, awarded September 2015

PCORI Dissemination & Implementation (D&I) Awards Purpose: Designed to give PCORI awardee teams an opportunity to propose investigator-initiated strategies for disseminating and implementing findings from their PCORI funded studies in the context of existing evidence undertake the next step(s) for making their research results more useful, actionable, accessible and available to targeted end users

Supported D&I Activities Develop, demonstrate, and evaluate approaches for incorporating PCORI research results in specific decision-making settings Adapt the content, format, or vehicle for delivering CER research evidence, to improve its use for different populations and across settings. Take results and products found effective “to scale” in diverse settings and populations. De-implement or reduce the use of interventions that are not evidence-based, have been prematurely widely adopted, or are harmful or wasteful. Note: All D&I projects must actively disseminate/implement findings to targeted end users and evaluate the success of the dissemination and implementation strategy.

D&I Award: "Comparison of Dynamic Treatment Strategies in Patient-Centered Outcomes” (PI: Zhang) Award is intended to build on findings from a PCORI Pilot Project in order to increase access, adoption, uptake, and sustainability of CI methods, specifically IPW and the g-formula among knowledge-users Researchers of 11 selected CER observational studies from 4 healthcare organizations Provide in-depth, ongoing consultation with clinical researchers Study design - guidance on how to define and emulate a target trial, how to specify the research question identify all required data elements, and examine sources of biases and confounds to select one or both g-methods Training and TA - hands-on training in the understanding and implementation of appropriate applications of the g-methods End-of-Project Knowledge Translation – range of collaborative activities that ensures sustainability of approach at the 4 sites and dissemination to wider audience

Questions? jgerson@pcori.org