Use of PRECIS-2 Ratings to Advance Understanding of Pragmatic Trial Design Domains: NIH Funded Trials (USA) Paula Darby Lipman, Ph.D. Westat Rockville, Maryland, USA Workshop: PRECIS-2 - Precisely how can this tool help investigators design trials to achieve practical answers to “real-world” questions? Society for Clinical Trials Annual Meeting Liverpool, United Kingdom May 8, 2017

Presentation Objectives Briefly introduce the National Institutes of Health (NIH) Pragmatic Trials Collaborative Project Describe how we incorporated PRECIS-2 Present results of a study using PRECIS-2 ratings to advance understanding of pragmatic trial design domains

Acknowledgments Pragmatic Trials Collaborative Project Investigators and research staff National Heart, Lung, and Blood Institute (NHLBI) and National Institute on Aging (NIA) program officers Co-investigator: Sean Tunis, MD (Center for Medical Technology Policy) NHLBI Program Officer: Kate Stoney, PhD Supported by NHLBI Grant 1R01HL125114-01 to Westat – Rockville, Maryland Health research organization

RFA-HL-14-019 Low-Cost, Pragmatic, Patient-Centered Randomized Controlled Intervention Trials Purpose: To encourage investigators to use existing resources to: Conduct low-cost, pragmatic, patient-centered randomized controlled trials of interventions Targeted at patients, families, health care providers, communities or health care systems through the integration of RCTs into existing clinical practice settings  Award Budget: $2.3M Total Direct Cost (5 years) Project Kick Off: November 14, 2014

2013: Statement of the Problem Only a small part of routine medical therapy provided to individual patients is based on the highest level of evidence. WHY? Studies with highly restrictive inclusion/exclusion criteria Limited testing of health care delivery strategies High cost of RCTs as they are currently conducted Conclusion: To meet the increasing demand for high-quality evidence in the health care arena, efficient, low-cost strategies for the conduct of large-scale, high-impact RCTs are needed.

Awardees of RFA HL-14-019 Principal Investigator(s) Study Title Agency Michael Simon Avidan, MD, Washington University Electroencephalograph Guidance of Anesthesia to Alleviate Geriatric Syndromes NIA Daniel Buysse, MD, University of Pittsburgh Pragmatic Trials of Behavioral Interventions for Insomnia in Hypertensive Patients NHLBI Michelle Gong, MD, Einstein College of Medicine Ognjen Gajic, MD, Mayo Clinic Prevention of Severe Acute Respiratory Failure in Patients with PROOFCheck- an Electronic Checklist to Prevent Organ Failure Scott David Halpern, MD, PhD, University of Pennsylvania Default Palliative Care Consultation for Seriously Ill Hospitalized Patients Henry Wang, MD, University of Alabama at Birmingham Pragmatic Trial of Airway Management in Out-of-Hospital Cardiac Arrest Studies are very different. Early adopters in this arena.

Westat’s Role Monitored achievement of trial planning milestones Quarterly calls with awardees to review progress Plan agenda for annual in-person meetings Bring together PIs and members of their research teams Includes NIH Program Officers Support collaborative activities Innovative use of the Pragmatic-Explanatory Continuum Indicator Summary (PRECIS-2) Instrument

Pragmatic Trials Project: PRECIS-2 Study Objectives Assess the degree to which the Pragmatic Trials were pragmatic at the planning phase Study whether and how trial design changed from the planning phase to the implementation phase Identify domains that may be more challenging and why (similar to Johnson, et al., 2016 design) – NIH collaboratory

Methods Design Quantitative data collection Mixed-methods [quan → QUAL] with sequential collection and analysis of quantitative data (PRECIS-2 ratings) and qualitative data (interviews with trial PIs) Quantitative data collection Time 1 (planning) – following a brief training (Feb ‘15) Time 2 (implementation) – after a refresher session (Apr ‘16) Qualitative data collection Interviews with each PI (Summer ‘16) Primary goals: Discuss experiences and impressions of the tool Discuss domains with a change in rating

Findings Experience with the tool Two of the PIs had used the original PRECIS tool to assess the design of their protocols. The other three PIs were not familiar with the tool (or its predecessor) prior to the project. All PIs agreed that the tool would have been helpful at the design phase of their trials. How pragmatic were the trials at the planning phase?

PRECIS-2 Results: Planning Phase

Findings (continued) As reflected in rating changes, how did trial design change over time? Total of 45 paired ratings (5 trials x 9 domains) Of these, 24 had a rating change Focus of the qualitative data collection and analysis One or more rating shifts occurred for each trial and across all nine domains Flexibility of Adherence and Follow-up = most shifts

Findings (continued) What did the shift in rating reflect? Only 3 represented a true change in trial design 21 shifts reflected understanding of the PRECIS-2 tool Three domains requiring clarification: Eligibility Flexibility of Adherence Follow-up

Eligibility #1 PI: “It’s a little more work to figure out patients who are…excluded…I just thought it was going to be very, very easy and you don’t have to think about it. But it turns out actually I have to have my staff validate it.” Less pragmatic due to additional effort needed to identify appropriate patients and validate correct identification. Eligibility refers to the extent to which the trial population matches the population intended for the intervention. The issue of effort to engage participants is more relevant to Recruitment Path, which addresses whether effort to recruit participants is greater than for patient engagement in usual care.

Eligibility #2 PI: “Once we started applying the criteria, we recognized… there are some people who we’ve excluded and I think they’re for good reasons. We haven’t changed the criteria, it’s just that as we’ve been applying them, we realized that it excludes a larger percentage of people perhaps than we thought.” Less pragmatic due to a higher proportion of patients excluded than originally anticipated. Volume of patient exclusion is not a consideration for pragmatism though it reflects the extent to which the trial participants are representative of most patients. For the Eligibility domain, one should consider the extent to which trial participants are similar to those who would receive the intervention if it were part of usual care.

Flexibility - Adherence #1 PI: “When we were in the planning phase, [it wasn’t] clear to us exactly how we were going to notify the clinicians…” Resource requirements are addressed under Organization. This domain should not have been rated, as there is no monitoring of patient adherence. More pragmatic because clinician notification [part of the intervention] was more automated than anticipated (requiring less effort).

Flexibility - Adherence #2 PI: “We’re not excluding anybody based on adherence, but we are encouraging adherence and are providing feedback on adherence.” This domain addresses how flexibly participants in the trial are monitored and encouraged compared to usual care. More pragmatic because no participants are excluded due to adherence or adherence is minimal.

Flexibility - Adherence #3 PI: “Our intervention really is executed and then it’s done, so the adherence of it is actually very minimal and the remainder of care given thereafter is just standard of care.” For surgical trials or trauma care, where there is no compliance issue after consent has been given, the domain is not applicable and should be left blank (unrated). More pragmatic because no participants are excluded due to adherence or adherence is minimal.

Follow-up #1 PI: “In clinical care, one would not necessarily seek out follow-up on patients, such as their primary outcome or survival… what made us think that it was less pragmatic was the manner by which you seek out that information.” Does not apply to this domain, which is concerned only with burden of follow-up on the participants, not whether the follow-up data are routinely collected. Less pragmatic as data needed from medical records are not routinely collected.

Follow-up #2 PI: “I have to apply in a separate IRB to a statistics department to get that long term follow-up. And that requires linking of the patient’s data. So that’s just a little less automatic…more work for me. For patients it’s the same.” Less pragmatic as collection of follow-up data is less automatic than anticipated. This domain is concerned only with burden of follow-up on the participants, not burden on research team or effort needed to collect the follow-up data.

Conclusions Not surprising to find that PRECIS-2 tool definitions and instructions were misunderstood or misapplied Positive response to using PRECIS-2 to guide conversations around trial design Provided valuable feedback to inform future trainings on the tool Conclusion that trial design decisions may be relatively stable is in need of further examination PRECIS-2 provides a valuable framework for discussion of critical elements underlying design decisions Includes communicating more effectively with each other and with their trial stakeholders

Poster Session 3 - #298 Framing the Conversation: Use of PRECIS-2 ratings to advance understanding of pragmatic trial design domains 10:00 a.m. Tuesday, May 9

Thank you!