MAKING PRAGMATIC TRIALS WORK operational challenges & decision support tool ‘PragMagic’ So John Norrie discussed why there is a need for pragmatic trials, what pragmatic trials are and, at least as important, what they are not. He also already mentioned some of the operational challenges of more pragmatic design choices, As part of the IMI GetReal project, which focussed how new methods of RWE collection and synthesis could be adopted earlier in the drug development and healthcare decision making process we looked at the operational side of pragmatic trials in which at least one of the arms involve a drug component. I just want to make that focus clear because I realized that most pragmatic trials discussed at this conference are not drug trials. So, in the next 15 minutes I'll discuss in more detail why the translation of the pragmatic trial principles into a methodologically sound and feasible trial protocol is not always straightforward, what the forces at work are in this field and I’ll show some of the issues in more detail using the decision support tool for pragmatic trial design that we have developed as part of the getreal project, which is currently being piloted. So why did we, within wp3 of getreal, focus on pragmatic trials? Not because… Mira Zuidgeest, Assistant Professor & co-lead WP3 GetReal University Medical Center Utrecht, the Netherlands

Why focus on pragmatic trials? Research question Decision making routine clinical practice PragMagic tool Study design Pragmatic trial Continuum Explanatory trial NOT because we think they are better than explanatory trials or observational studies, because they are not. They simply answer a different research questions so it is the research question that determines which study design is most suitable. If you aim to guide decision making in routine clinical practice then a more pragmatic trial may be most suitable and the pragmagic tool may be of help. It is also good to remember that explanatory and pragmatic trials should not be seen as separate entities but as the extremes of a continuum and that most trials are somewhere in between these extremes. It is also not because PT are so much harder to perform than more explanatory trials because again this depends on the situation. A post launch point of care pragmatic trial with follow up through the EHR may be much simpler than a more traditinal explanatory trial whereas a prelaunch pragmatic trial me be a lot more challenging. So the reason we focus on pragmatic trials is…

Why focus on pragmatic trials? opportunity to generate RWE, also in early stage of drug development limited experience & unanticipated operational challenges need for guidance and tools in designing pragmatic trials with high generalisability to setting of interest while ensuring operational feasibility and validity We focus on pragmatic trials because they provide a great opportunity to generate RWE, also in the early stage of drug development. But limited experience with these trials make people hestitant to embark on this road, also because the OC are different than the ones in more traditional explanatory trials and therefore often unanticipated. Thats why, at this stage, there is more need for guidance and tools in designing pragmatic trials with high generalisability to setting of interest while ensuring operational feasibility and validity So let's look at an example of a pragmatic trial to help us identify some of the issues...it is a pre launch pragmatic trial for which the results have recently been published...

Pragmatic trial example The aim was to studie the effectiveness of a once daily combinationinhaler versus usual care for copd The trial design was pragmatic on many aspects....

Pragmatic trial example Design choices Operational challenges On the left hand side you see some of the key deisn choices. As already mentioned pre launch trial Comparing not the effect of a drug but of a Treatment strategy Broad inclusion of patients in setting of interest Usual care comparator Clinically relevant outcome So the design sounds great. However, they had to overcome a lot of operational challenges to be able to succesfully run the study, such as... How to ensure timely safety reporting of an experimental drug without interfering too much with routine clinical practice? How to supply the drugs as would be done in routine clinical practice (which included the training of many many pharmacy teams) How to ask for informed consent in line with ethical and legal requrements without overburdening already busy gp practices Gcp training of gp sites, which were often research naive sites without changing routine clinical practice too much Technical and privacy challenges regarding linkage of ehr Many of these challenges were different from the ones that one would encounter in a more taditional expl trial So this example brings us to the reason we started to develop a decision support tool for pragmatic trial design... Nawar 2013; New 2014

Translation of pragmatic trial concept to trial protocol Concept of pragmatic trials methodology, feasibility, stakeholder/ethical acceptability Pragmatic trial protocol Generalisability of study results to population and setting of interest Schwartz & Lellouch 1967 PRECIS Thorpe 2009/Loudon 2015 Recruit real world population Setting of care reflecting routine clinical practice Operationalise treatment strategies (e.g. blinding, co-medication) Choose appropriate usual care comparator & outcome Trial conduct should not influence routine clinical practice Comparison of treatment strategies (drug effect + extraneous factors) Because the translation of the concept of pragmatic trials to a pragmatic trial protocol which is methodologically sound AND feasible AND acceptable by stakeholders and ethical principles is challenging and involves a new way of thinking we are not yet familiar with. Schwartz and Lellouch have set the scene for pragmatic trials the PRECIS authors developed a tool to show how pragmatic your trial design is. We want to take this one step further and move into the practice of trial conduct. So we put more emphasis on how to balance between the aim to be pragmatic and the feasibility but also for example validity of the trial results. I'll quickly summarize the main pragmatic trial principles with their main challenges, some of which have been mentioned during the previous talk already... Main aim of a more pragmatic trial is to maximise generalisability of study results to the population of interest But it is not enough to say that you use wide eligibility criteria. You need to make sure you actually recruit a trial population representative of the real world pop in sites that reflect routine clinical practice? Second is that in a pragmatic trial you want to compare treatment strategies, which includes placebo effects and other extraneous factors... But how to operationalise this with repect to eg blinding and comedication To be able to guide prescribing the comparator and outcome in pragmatic trials should be clinically relevant, but how to choose the appropriate comparator if there are many usual care options available? And on top of this, a very specific point for pragmatic trials. Since they aim to measure effectiveness in routine clinical practice... All aspects of trial conduct should influence routine clinical practice as little as possible, which has implications for data collection and safety monitoring as well. So what are the forces at work when designing a more pragmatic trial?.... Clinically relevant comparators and outcome(s)

Forces at work TRIAL DESIGN ELEMENTS OPERATIONAL CHALLENGES Retropro & eLung Van Staa 2014* Possible solution? Appoint dedicated study staff TRIAL DESIGN ELEMENTS OPERATIONAL CHALLENGES Choice of sites: GENERAL PRACTICES training requirements & data collection  change in usual practice & low/selective participation 59% of sites expressed interest in trial participation 4% of sites actually recruited patients generalisability ↓ & feasibility ↓ participating GPs differed from those not participating eLung trial did not recruit sufficient N of patients As mentioned before Starting point are the trial design elements you can choose towards a more pragmatic trial. The aim is to increase generalisability to the setting of interest. But these design elements also have other implications for example with respect to precision or costs. Also a design element can have specific operational challenges Which are, al already mentioned, less often anticipated than in tradition more explanatory trials. These operational challenges can have their own impact on generalisabaility and the other implications. Let’s walk through an example. If these operational challenges lead to a change in routine clinical pratice or low or selective participation of practices this influences the gen & feas of your trial This example has been described by van staa and others in 2014…. So these are the forces we try to capture in a systematic way in the decision support tool for pragmatic trial design pragmagic GENERALISABILITY, VALIDITY, PRECISION STAKEHOLDER & ETHICAL ACCEPTABILITY RESOURCES generalisability ↑ generalisability ↓ feasibility ↓ * Van Staa et al. The opportunities and challenges of pragmatic point-of-care randomised trials using routinely collected electronic records: evaluations of two exemplar trials. Health Technol Assess. 2014;18(43);1-146

PRAGMAGIC a decision support tool for pragmatic trial design We try to raise awareness of the the possible consequences of more pragmatic trial design choices so they can be better anticipated. I’ll show you what this looks like in the tool The tool has been developed in colaboration with a serious gaming company called Ijsfontein, which explains the nice visuals in the tool and the built in gaming elements.. So let me show you around quickly Intro page Generalisability has been put central in tool Main screen shows city --? Light up first row Navigation wheel – 7 domains with all main ques you have to think about when designing a more pragmatic trial Progress In the end you can export your choices and notes I’ll show you three examples of issues we previously touched upon. 1. Recruit real world population in real world setting - D participant Q1 2. Choosing the appropriate usual care comparator – D comparator – Q7 - One arm with mix of all usual care options may be the most generalisable to the setting of interest on a population level. But may be less acceptable by stakeholders if risk-benefit ratio differs between usual care options in mix because difficult to interpret results for specific patients. May also be less acceptable from ethical standpoint if mix includes suboptimal care 3. How to operationalise the treatment strategy – various things to consider including start of treatment, randomization level, blinding, switching, dosing of drug and concomitent interventions, supply and reimbursement of the drug and treatment adherence. If we look at Q9 – dosing in trial – A at discretion of treating physician is most generalisable but may decrease precision & ethical acceptability and is hard to implement for experimental arm in prelaunch trial. 4. Which clinically relevant outcome to meause? Q4 PROM– clinically relevant outcome which may be prefered (acceptability) but may not be part of routine clinical practice and thus lower generalisability & increase costs of trial. 5. All elements of trial conduct should interfere with or change routine clinical practice & usual care as little as possible -- other considerations need to be made with regard to data collection, monitoring & safety than for traditional more explanatory trials so that’s why all these trial design elements need to be thought through as well. So I hope this presentation gave you a good impression of some of the issues involved in designing and performing more pragmatic trials and how the pragmagic tool can help you get insights in these issues. Some closing points…

Closing points Research question drives study design & specific design elements (e.g. switching between treatment arms) Minimal interference with routine clinical practice requires thought shift during design phase of pragmatic trial Balance methodology, feasibility & stakeholder and ethical acceptability Is it possible to design a 100% pragmatic AND feasible clinical trial?

Launch of PragMagic tool June 8th (webinar), see… Contact us at… WWW Launch of PragMagic tool June 8th (webinar), see… Contact us at… WWW.PRAGMAGIC.EU info@pragmagic.eu PragMagic tool: Decision support tool for pragmatic trial design. Focus: Randomised pragmatic trials with drug component. NOT: decision-making tool; checklist to assure regulatory/ethical compliance; quality verdict on study design.

Backup slides

Recruit real world population in real world setting Example 1 Recruit real world population in real world setting Eligibility criteria need to be as in routine clinical practice BUT this does not automatically lead to representative study population. who do you reach with which recruitment strategies? who gives consent? Are participating health care providers representative? Who do they ask to participate in the trial?

Choosing the appropriate usual care comparator Example 2 Choosing the appropriate usual care comparator What if… Several usual care options exist? Guidelines differ from usual care? Variations exist in usual care across centres/counties? Usual care changes during trial conduct? Example: Comparator Q7 –One arm with mix of all usual care options may be the most generalisable to the setting of interest on a population level. But may be less acceptable by stakeholders if risk-benefit ratio differs between usual care options in mix because difficult to interpret results for specific patients May also be less acceptable from ethical standpoint if mix includes suboptimal care

How to operationalise the treatment strategies? Example 3 How to operationalise the treatment strategies? Blinding Dose and co-medication Random allocation & level of randomization Dealing with treatment preferences Supply & reimbursement of medication/treatments Dealing with switches Example: Comparator Q9 – dosing in trial – A at discretion of treating physician is most generalisable but may decrease precision & ethical acceptability and is hard to implement for experimental arm in prelaunch trial.

Which clinically relevant outcome to measure? Example 4 Which clinically relevant outcome to measure? Patient Reported Outcomes are clinically relevant May be preferred  acceptability Not part of routine clinical practice  generalisability & costs Example: Outcome Q4 Which clinically relevant outcome to meause? PROM– clinically relevant outcome which may be prefered (acceptability) but may not be part of routine clinical practice and thus lower generalisability & increase costs of trial

Example 5 All elements of trial conduct should interfere with or change routine clinical practice & usual care as little as possible. Other considerations need to be made with regard to data collection, monitoring & safety than for traditional more explanatory trials Data Q1, Safety Q3, Quality Q2 – other considerations need to be made with regard to data collection, monitoring & safety than for traditional more explanatory trials