1. MAKING PRAGMATIC TRIALS WORK operational challenges & decision support tool ‘PragMagic’
So John Norrie discussed why there is a need for pragmatic trials, what pragmatic trials are and, at least as important, what they are not. He also already mentioned some of the operational challenges of more pragmatic design choices,
As part of the IMI GetReal project,
which focussed how new methods of RWE collection and synthesis could be adopted earlier in the drug development and healthcare decision making process
we looked at the operational side of pragmatic trials in which at least one of the arms involve a drug component.
I just want to make that focus clear because I realized that most pragmatic trials discussed at this conference are not drug trials.
So, in the next 15 minutes I'll discuss in more detail why the translation of the pragmatic trial principles into a methodologically sound and feasible trial protocol is not always straightforward,
what the forces at work are in this field
and I’ll show some of the issues in more detail using the decision support tool for pragmatic trial design that we have developed as part of the getreal project, which is currently being piloted.
So why did we, within wp3 of getreal, focus on pragmatic trials? Not because…
Mira Zuidgeest, Assistant Professor & co-lead WP3 GetReal
University Medical Center Utrecht, the Netherlands

2. Why focus on pragmatic trials?
Research question
Decision making routine clinical practice
PragMagic tool
Study design
Pragmatic trial
Continuum
Explanatory trial
NOT because we think they are better than explanatory trials or observational studies, because they are not.
They simply answer a different research questions
so it is the research question that determines which study design is most suitable.
If you aim to guide decision making in routine clinical practice then a more pragmatic trial may be most suitable and the pragmagic tool may be of help.
It is also good to remember that explanatory and pragmatic trials should not be seen as separate entities but as the extremes of a continuum and that most trials are somewhere in between these extremes.
It is also not because PT are so much harder to perform than more explanatory trials because again this depends on the situation. A post launch point of care pragmatic trial with follow up through the EHR may be much simpler than a more traditinal explanatory trial whereas a prelaunch pragmatic trial me be a lot more challenging.
So the reason we focus on pragmatic trials is…

3. Why focus on pragmatic trials?
opportunity to generate RWE, also in early stage of drug development
limited experience & unanticipated operational challenges
need for
guidance and tools in designing pragmatic trials with high generalisability to setting of interest while ensuring operational feasibility and validity
We focus on pragmatic trials because they provide a great opportunity to generate RWE, also in the early stage of drug development.
But limited experience with these trials make people hestitant to embark on this road, also because the OC are different than the ones in more traditional explanatory trials and therefore often unanticipated.
Thats why, at this stage, there is more need for guidance and tools in designing pragmatic trials with high generalisability to setting of interest while ensuring operational feasibility and validity
So let's look at an example of a pragmatic trial to help us identify some of the issues...it is a pre launch pragmatic trial for which the results have recently been published...

4. Pragmatic trial example
The aim was to studie the effectiveness of a once daily combinationinhaler versus usual care for copd
The trial design was pragmatic on many aspects....

5. Pragmatic trial example
Design choices Operational challenges
On the left hand side you see some of the key deisn choices.
As already mentioned pre launch trial
Comparing not the effect of a drug but of a Treatment strategy
Broad inclusion of patients in setting of interest
Usual care comparator
Clinically relevant outcome
So the design sounds great.
However, they had to overcome a lot of operational challenges to be able to succesfully run the study, such as...
How to ensure timely safety reporting of an experimental drug without interfering too much with routine clinical practice?
How to supply the drugs as would be done in routine clinical practice (which included the training of many many pharmacy teams)
How to ask for informed consent in line with ethical and legal requrements without overburdening already busy gp practices
Gcp training of gp sites, which were often research naive sites without changing routine clinical practice too much
Technical and privacy challenges regarding linkage of ehr
Many of these challenges were different from the ones that one would encounter in a more taditional expl trial
So this example brings us to the reason we started to develop a decision support tool for pragmatic trial design...
Nawar 2013; New 2014

6. Translation of pragmatic trial concept to trial protocol
Concept of pragmatic trials
methodology,
feasibility,
stakeholder/ethical acceptability
Pragmatic trial protocol
Generalisability of study results to population and setting of interest
Schwartz & Lellouch 1967
PRECIS Thorpe 2009/Loudon 2015
Recruit real world population
Setting of care reflecting routine clinical practice
Operationalise treatment strategies (e.g. blinding, co-medication)
Choose appropriate usual care comparator & outcome
Trial conduct should not influence routine clinical practice
Comparison of treatment strategies (drug effect + extraneous factors)
Because the translation of the concept of pragmatic trials to a pragmatic trial protocol which is methodologically sound AND feasible AND acceptable by stakeholders and ethical principles is challenging and involves a new way of thinking we are not yet familiar with.
Schwartz and Lellouch have set the scene for pragmatic trials
the PRECIS authors developed a tool to show how pragmatic your trial design is.
We want to take this one step further and move into the practice of trial conduct. So we put more emphasis on how to balance between the aim to be pragmatic and the feasibility but also for example validity of the trial results.
I'll quickly summarize the main pragmatic trial principles with their main challenges, some of which have been mentioned during the previous talk already...
Main aim of a more pragmatic trial is to maximise generalisability of study results to the population of interest
But it is not enough to say that you use wide eligibility criteria. You need to make sure you actually recruit a trial population representative of the real world pop in sites that reflect routine clinical practice?
Second is that in a pragmatic trial you want to compare treatment strategies, which includes placebo effects and other extraneous factors... But how to operationalise this with repect to eg blinding and comedication
To be able to guide prescribing the comparator and outcome in pragmatic trials should be clinically relevant, but how to choose the appropriate comparator if there are many usual care options available?
And on top of this, a very specific point for pragmatic trials. Since they aim to measure effectiveness in routine clinical practice... All aspects of trial conduct should influence routine clinical practice as little as possible, which has implications for data collection and safety monitoring as well.
So what are the forces at work when designing a more pragmatic trial?....
Clinically relevant comparators and outcome(s)

7. Forces at work TRIAL DESIGN ELEMENTS OPERATIONAL CHALLENGES
Retropro & eLung
Van Staa 2014*
Possible solution?
Appoint dedicated study staff
TRIAL DESIGN ELEMENTS
OPERATIONAL CHALLENGES
Choice of sites:
GENERAL PRACTICES
training requirements & data collection  change in usual practice & low/selective participation
59%
of sites expressed interest in trial participation 4%
of sites actually recruited patients
generalisability ↓
& feasibility ↓
participating GPs differed from those not participating
eLung trial did not recruit sufficient N of patients
As mentioned before
Starting point are the trial design elements you can choose towards a more pragmatic trial.
The aim is to increase generalisability to the setting of interest.
But these design elements also have other implications for example with respect to precision or costs.
Also a design element can have specific operational challenges
Which are, al already mentioned, less often anticipated than in tradition more explanatory trials.
These operational challenges can have their own impact on generalisabaility and the other implications.
Let’s walk through an example.
If these operational challenges lead to a change in routine clinical pratice or low or selective participation of practices this influences the gen & feas of your trial
This example has been described by van staa and others in 2014….
So these are the forces we try to capture in a systematic way in the decision support tool for pragmatic trial design pragmagic
GENERALISABILITY, VALIDITY, PRECISION
STAKEHOLDER & ETHICAL ACCEPTABILITY
RESOURCES
generalisability ↑
generalisability ↓
feasibility ↓
* Van Staa et al. The opportunities and challenges of pragmatic point-of-care randomised trials using routinely collected electronic records: evaluations of two exemplar trials. Health Technol Assess. 2014;18(43);1-146

8. PRAGMAGIC a decision support tool for pragmatic trial design
We try to raise awareness of the the possible consequences of more pragmatic trial design choices so they can be better anticipated. I’ll show you what this looks like in the tool
The tool has been developed in colaboration with a serious gaming company called Ijsfontein, which explains the nice visuals in the tool and the built in gaming elements..
So let me show you around quickly
Intro page
Generalisability has been put central in tool
Main screen shows city --? Light up first row
Navigation wheel – 7 domains with all main ques you have to think about when designing a more pragmatic trial
Progress
In the end you can export your choices and notes
I’ll show you three examples of issues we previously touched upon.
1. Recruit real world population in real world setting - D participant Q1
2. Choosing the appropriate usual care comparator – D comparator – Q7 - One arm with mix of all usual care options may be the most generalisable to the setting of interest on a population level. But may be less acceptable by stakeholders if risk-benefit ratio differs between usual care options in mix because difficult to interpret results for specific patients. May also be less acceptable from ethical standpoint if mix includes suboptimal care
3. How to operationalise the treatment strategy – various things to consider including start of treatment, randomization level, blinding, switching, dosing of drug and concomitent interventions, supply and reimbursement of the drug and treatment adherence. If we look at Q9 – dosing in trial – A at discretion of treating physician is most generalisable but may decrease precision & ethical acceptability and is hard to implement for experimental arm in prelaunch trial.
4. Which clinically relevant outcome to meause? Q4 PROM– clinically relevant outcome which may be prefered (acceptability) but may not be part of routine clinical practice and thus lower generalisability & increase costs of trial.
5. All elements of trial conduct should interfere with or change routine clinical practice & usual care as little as possible -- other considerations need to be made with regard to data collection, monitoring & safety than for traditional more explanatory trials so that’s why all these trial design elements need to be thought through as well.
So I hope this presentation gave you a good impression of some of the issues involved in designing and performing more pragmatic trials and how the pragmagic tool can help you get insights in these issues.
Some closing points…

9. Closing points
Research question drives study design & specific design elements (e.g. switching between treatment arms)
Minimal interference with routine clinical practice requires thought shift during design phase of pragmatic trial
Balance methodology, feasibility & stakeholder and ethical acceptability
Is it possible to design a 100% pragmatic AND feasible clinical trial?

10. Launch of PragMagic tool June 8th (webinar), see… Contact us at… WWW
Launch of PragMagic tool June 8th (webinar), see… Contact us at…
PragMagic tool: Decision support tool for pragmatic trial design.
Focus: Randomised pragmatic trials with
drug component.
NOT: decision-making tool; checklist to assure regulatory/ethical compliance; quality verdict on study design.

11. Backup slides

18. Recruit real world population in real world setting
Example 1
Recruit real world population in real world setting
Eligibility criteria need to be as in routine clinical practice
BUT this does not automatically lead to representative study population.
who do you reach with which recruitment strategies?
who gives consent?
Are participating health care providers representative?
Who do they ask to participate in the trial?

19. Choosing the appropriate usual care comparator
Example 2
Choosing the appropriate usual care comparator
What if…
Several usual care options exist?
Guidelines differ from usual care?
Variations exist in usual care across centres/counties?
Usual care changes during trial conduct?
Example: Comparator Q7 –One arm with mix of all usual care options may be the most generalisable to the setting of interest on a population level.
But
may be less acceptable by stakeholders if risk-benefit ratio differs between usual care options in mix because difficult to interpret results for specific patients
May also be less acceptable from ethical standpoint if mix includes suboptimal care

20. How to operationalise the treatment strategies?
Example 3
How to operationalise the treatment strategies?
Blinding
Dose and co-medication
Random allocation & level of randomization
Dealing with treatment preferences
Supply & reimbursement of medication/treatments
Dealing with switches
Example: Comparator Q9 – dosing in trial – A at discretion of treating physician is most generalisable but may decrease precision & ethical acceptability and is hard to implement for experimental arm in prelaunch trial.

21. Which clinically relevant outcome to measure?
Example 4
Which clinically relevant outcome to measure?
Patient Reported Outcomes are clinically relevant
May be preferred  acceptability
Not part of routine clinical practice  generalisability & costs
Example: Outcome Q4
Which clinically relevant outcome to meause?
PROM– clinically relevant outcome which may be
prefered (acceptability) but
may not be part of routine clinical practice and thus lower generalisability & increase costs of trial

22. Example 5
All elements of trial conduct should interfere with or change routine clinical practice & usual care as little as possible.
Other considerations need to be made with regard to data collection, monitoring & safety than for traditional more explanatory trials
Data Q1, Safety Q3, Quality Q2 – other considerations need to be made with regard to data collection, monitoring & safety than for traditional more explanatory trials