1. Use of PRECIS-2 Ratings to Advance Understanding of Pragmatic Trial Design Domains: NIH Funded Trials (USA)
Paula Darby Lipman, Ph.D.
Westat
Rockville, Maryland, USA
Workshop: PRECIS-2 - Precisely how can this tool help investigators design trials to achieve practical answers to “real-world” questions?
Society for Clinical Trials Annual Meeting
Liverpool, United Kingdom
May 8, 2017

2. Presentation Objectives
Briefly introduce the National Institutes of Health (NIH) Pragmatic Trials Collaborative Project
Describe how we incorporated PRECIS-2
Present results of a study using PRECIS-2 ratings to advance understanding of pragmatic trial design domains

3. Acknowledgments Pragmatic Trials Collaborative Project
Investigators and research staff
National Heart, Lung, and Blood Institute (NHLBI) and National Institute on Aging (NIA) program officers
Co-investigator: Sean Tunis, MD (Center for Medical Technology Policy)
NHLBI Program Officer: Kate Stoney, PhD
Supported by NHLBI Grant 1R01HL to Westat – Rockville, Maryland
Health research organization

4. RFA-HL-14-019 Low-Cost, Pragmatic, Patient-Centered Randomized Controlled Intervention Trials
Purpose: To encourage investigators to use existing resources to:
Conduct low-cost, pragmatic, patient-centered randomized controlled trials of interventions
Targeted at patients, families, health care providers, communities or health care systems through the integration of RCTs into existing clinical practice settings 
Award Budget: $2.3M Total Direct Cost (5 years)
Project Kick Off: November 14, 2014

5. 2013: Statement of the Problem
Only a small part of routine medical therapy provided to individual patients is based on the highest level of evidence. WHY?
Studies with highly restrictive inclusion/exclusion criteria
Limited testing of health care delivery strategies
High cost of RCTs as they are currently conducted
Conclusion: To meet the increasing demand for high-quality evidence in the health care arena, efficient, low-cost strategies for the conduct of large-scale, high-impact RCTs are needed.

6. Awardees of RFA HL-14-019 Principal Investigator(s) Study Title Agency
Michael Simon Avidan, MD,
Washington University
Electroencephalograph Guidance of Anesthesia to Alleviate Geriatric Syndromes
NIA
Daniel Buysse, MD,
University of Pittsburgh
Pragmatic Trials of Behavioral Interventions for Insomnia in Hypertensive Patients
NHLBI
Michelle Gong, MD,
Einstein College of Medicine
Ognjen Gajic, MD,
Mayo Clinic
Prevention of Severe Acute Respiratory Failure in Patients with PROOFCheck- an Electronic Checklist to Prevent Organ Failure
Scott David Halpern, MD, PhD,
University of Pennsylvania
Default Palliative Care Consultation for Seriously Ill Hospitalized Patients
Henry Wang, MD,
University of Alabama at Birmingham
Pragmatic Trial of Airway Management in Out-of-Hospital Cardiac Arrest
Studies are very different.
Early adopters in this arena.

7. Westat’s Role Monitored achievement of trial planning milestones
Quarterly calls with awardees to review progress
Plan agenda for annual in-person meetings
Bring together PIs and members of their research teams
Includes NIH Program Officers
Support collaborative activities
Innovative use of the Pragmatic-Explanatory Continuum Indicator Summary (PRECIS-2) Instrument

8. Pragmatic Trials Project: PRECIS-2 Study Objectives
Assess the degree to which the Pragmatic Trials were pragmatic at the planning phase
Study whether and how trial design changed from the planning phase to the implementation phase
Identify domains that may be more challenging and why
(similar to Johnson, et al., 2016 design) – NIH collaboratory

9. Methods Design Quantitative data collection
Mixed-methods [quan → QUAL] with sequential collection and analysis of quantitative data (PRECIS-2 ratings) and qualitative data (interviews with trial PIs)
Quantitative data collection
Time 1 (planning) – following a brief training (Feb ‘15)
Time 2 (implementation) – after a refresher session (Apr ‘16)
Qualitative data collection
Interviews with each PI (Summer ‘16)
Primary goals:
Discuss experiences and impressions of the tool
Discuss domains with a change in rating

10. Findings Experience with the tool
Two of the PIs had used the original PRECIS tool to assess the design of their protocols.
The other three PIs were not familiar with the tool (or its predecessor) prior to the project.
All PIs agreed that the tool would have been helpful at the design phase of their trials.
How pragmatic were the trials at the planning phase?

11. PRECIS-2 Results: Planning Phase

12. Findings (continued)
As reflected in rating changes, how did trial design change over time?
Total of 45 paired ratings (5 trials x 9 domains)
Of these, 24 had a rating change
Focus of the qualitative data collection and analysis
One or more rating shifts occurred for each trial and across all nine domains
Flexibility of Adherence and Follow-up = most shifts

13. Findings (continued) What did the shift in rating reflect?
Only 3 represented a true change in trial design
21 shifts reflected understanding of the PRECIS-2 tool
Three domains requiring clarification:
Eligibility
Flexibility of Adherence
Follow-up

14. Eligibility #1
PI: “It’s a little more work to figure out patients who are…excluded…I just thought it was going to be very, very easy and you don’t have to think about it. But it turns out actually I have to have my staff validate it.”
Less pragmatic due to additional effort needed to identify appropriate patients and validate correct identification.
Eligibility refers to the extent to which the trial population matches the population intended for the intervention.
The issue of effort to engage participants is more relevant to Recruitment Path, which addresses whether effort to recruit participants is greater than for patient engagement in usual care.

15. Eligibility #2
PI: “Once we started applying the criteria, we recognized… there are some people who we’ve excluded and I think they’re for good reasons. We haven’t changed the criteria, it’s just that as we’ve been applying them, we realized that it excludes a larger percentage of people perhaps than we thought.”
Less pragmatic due to a higher proportion of patients excluded than originally anticipated.
Volume of patient exclusion is not a consideration for pragmatism though it reflects the extent to which the trial participants are representative of most patients.
For the Eligibility domain, one should consider the extent to which trial participants are similar to those who would receive the intervention if it were part of usual care.

16. Flexibility - Adherence #1
PI: “When we were in the planning phase, [it wasn’t] clear to us exactly how we were going to notify the clinicians…”
Resource requirements are addressed under Organization.
This domain should not have been rated, as there is no monitoring of patient adherence.
More pragmatic because clinician notification [part of the intervention] was more automated than anticipated (requiring less effort).

17. Flexibility - Adherence #2
PI: “We’re not excluding anybody based on adherence, but we are encouraging adherence and are providing feedback on adherence.”
This domain addresses how flexibly participants in the trial are monitored and encouraged compared to usual care.
More pragmatic because no participants are excluded due to adherence or adherence is minimal.

18. Flexibility - Adherence #3
PI: “Our intervention really is executed and then it’s done, so the adherence of it is actually very minimal and the remainder of care given thereafter is just standard of care.”
For surgical trials or trauma care, where there is no compliance issue after consent has been given, the domain is not applicable and should be left blank (unrated).
More pragmatic because no participants are excluded due to adherence or adherence is minimal.

19. Follow-up #1
PI: “In clinical care, one would not necessarily seek out follow-up on patients, such as their primary outcome or survival… what made us think that it was less pragmatic was the manner by which you seek out that information.”
Does not apply to this domain, which is concerned only with burden of follow-up on the participants, not whether the follow-up data are routinely collected.
Less pragmatic as data needed from medical records are not routinely collected.

20. Follow-up #2
PI: “I have to apply in a separate IRB to a statistics department to get that long term follow-up. And that requires linking of the patient’s data. So that’s just a little less automatic…more work for me. For patients it’s the same.”
Less pragmatic as collection of follow-up data is less automatic than anticipated.
This domain is concerned only with burden of follow-up on the participants, not burden on research team or effort needed to collect the follow-up data.

21. Conclusions
Not surprising to find that PRECIS-2 tool definitions and instructions were misunderstood or misapplied
Positive response to using PRECIS-2 to guide conversations around trial design
Provided valuable feedback to inform future trainings on the tool
Conclusion that trial design decisions may be relatively stable is in need of further examination
PRECIS-2 provides a valuable framework for discussion of critical elements underlying design decisions
Includes communicating more effectively with each other and with their trial stakeholders

22. Poster Session 3 - #298
Framing the Conversation: Use of PRECIS-2 ratings to advance understanding of pragmatic trial design domains
10:00 a.m. Tuesday, May 9

23. Thank you!