Improving PCOR Methods: Causal Inference Jason Gerson, PhD Associate Director CIMPOD 2016 February 25, 2016

Improving PCOR Methods: Why Methods Matter PCORI’s Mission PCORI was created to support research that provides high-integrity, evidence-based information to patients, clinicians, and the broader healthcare community. Why Methods Matter Rigorous research methods are needed to produce relevant, trustworthy findings that can improve patients’ healthcare outcomes. Methods for PCOR Methods include systematic processes, designs, tools, and techniques used to generate the evidence needed to answer questions about which healthcare options work best for particular patients.

Improving PCOR Methods: Program Goals Identify Methods Gaps Identify methodological gaps relevant to the conduct of PCOR Fund Research Fund high impact studies which address gaps in methodological research Disseminate Promising/Best Practices Disseminate and facilitate the adoption of new methods to improve the conduct of PCOR

Methods Program Overview 73 funded projects, $76 million awarded 25 states, DC, and Quebec Methods for Patient and Stakeholder Engagement Methods for Patient-Centered Outcomes and Patient-Reported Outcomes Research Related to the Ethical Conduct of PCOR/CER Methods to Improve Study Design Methods to Improve Validity and Efficiency of Analyses Methods to Support Data Research Networks

CER Methods Program Portfolio (73 projects as of 2/25/2016) Causal inference Heterogeneity of treatment effect Missing data Treatment adherence Some projects are classified in ≥1 category

Importance of CI Methods for PCOR/CER Many important clinical CER questions cannot be answered with RCTs (impossible/undesirable/unethical) – need for observational studies and to support improvements in their analytic methods. Comparison of complex treatment strategies sustained over long periods of time using data collected from routine clinical practice (or EHRs). Need to “transport” findings from our funded methodological projects to other parts of the PCORI portfolio, esp. large pragmatic studies and PCORnet. Intent of our investments in this area is to provide guidance/ recommendations to research community about what CI approaches are more or less optimal for a particular problem– toolkits, decision trees, standards, best practices.

Causal Inference for Effectiveness Research in Using Secondary Data Potential Impact Could change PCOR by improving methods for adjusting for confounding in observational studies, thereby improving the ability of those studies to generate robust findings Engagement A stakeholder advisory panel of 10 patients and will meet 5 times per year to provide feedback Methods Expansive simulation studies and clinically relevant sample studies to test and improve approaches for adjusting for confounding in database analyses Provides guidance on the optimal use of two innovative and highly promising approaches (high-dimensional propensity scoring and targeted maximum likelihood estimation) for robust confounding adjustment in database analyses. Sebastian Schneeweiss, MH, ScD MD, Brigham and Women’s Hospital Boston, MA CER Methods and Infrastructure, awarded September 2013

Causal analyses of electronic health record data for assessing the comparative effectiveness of treatment regimens Potential Impact Could change PCOR by improving existing causal inference methods for comparing complex treatment strategies and informing decision making by clinicians and patients for the management of chronic conditions Engagement A multi-stakeholder committee will inform and prioritize the development and dissemination of the proposed statistical methods Methods Simulation studies to evaluate the statistical methods; application of the methods to analyze EHR data from a recent observational study of type II diabetes Advances and adapts existing causal inference methods to improve evaluation of realistic adaptive treatment strategies that reflect real-world adherence to treatment decisions and recommended monitoring schedules. Romain Neugebauer, PhD Kaiser Foundation Research Institute Oakland, CA CER Methods and Infrastructure, awarded September 2014

Patient Centered Adaptive Treatment Strategies (PCATS) using Bayesian Causal Inference Potential Impact Could change PCOR by improving methods for analyzing data in cases where the treatment/exposures varies over time and improve clinical practice by developing evidence-based shared decision making tools for identifying optimal PCATS at the point-of- care. Engagement A stakeholder advisory panel including two parent representatives, and a partnership with PR-COIN (part of PCORnet) will ensure patient and stakeholder engagement throughout the research process, including in the design, conduct, analysis, and dissemination. Methods Bayesian modeling strategies, simulation studies, and empirical analyses of PCATS for polyarticular Juvenile Idiopathic Arthritis (pJIA). Develops Bayesian double robust causal inference methods for use in analyzing large registry and electronic health records to evaluate the clinical effectiveness of treatments and inform patient centered adaptive treatment strategies (PCATS). Bin Huang, PhD Cincinnati Children’s Hospital Medical Center Cincinnati, OH CER Methods and Infrastructure, awarded May 2015

Modeling Strategies for Observational CER: What Works Best When? Potential Impact Could change PCOR by helping researchers select the best method(s) for a particular observational dataset and CER question Methods Systematic review of existing literature on relevant analytical methods; simulations to assess performance of methods in cases where the literature is insufficient Develops a decision tool for use in selecting the most appropriate analytical methods to control bias from treatment self-selection (“confounding by indication”) in analyses of observational data for CER. Douglas Landsittel, PhD University of Pittsburgh Pittsburgh, PA CER Methods and Infrastructure, awarded December 2013

Development of a Causal Inference Toolkit for Patient- Centered Outcomes Research Potential Impact Could change PCOR by: 1) improving the validity of analytic methods for common applications; 2) enhancing the reproducibility, transparency, and replication of PCOR; and 3) improving the utility of “real-world” data in conducting PCOR studies Engagement A technical advisory panel will participate in approximately three conference calls per year to provide advice on this project Methods Develop guidelines for the use of inverse probability weighting and g-formula analytical techniques and software to facilitate the use of these techniques Develops a causal inference toolkit to provide a comprehensive, practical, and accessible guide to implementing inverse-probability weighting and the g-formula analytical techniques for PCOR studies using large observational data. Yi Zhang, MS, PhD Medical Technology & Practice Patterns Institute Bethesda, MD CER Methods and Infrastructure, awarded September 2013

Causal Inference Guidelines for Pragmatic Clinical Trials Potential Impact Provides investigators with better tools to estimate patient-centered effects in pragmatic clinical trials. Engagement Patients and stakeholders will be engaged in identifying the effects of interest in pragmatic trials and developing standards for estimating these effects. Methods Systematic review, focus groups, and case studies with supporting software will be used to create and implement a revised framework for the design and conduct of randomized pragmatic trials. Develops a set of causal inference standards specifically designed for the analysis of pragmatic clinical trials. Miguel Hernan, MPH, MD, DPH Harvard University School of Public Health Boston, MA CER Methods and Infrastructure, awarded September 2015

Thank you