Stavros Petrou 26 th November 2010 Adaptive designs for NIHR funded trials: A health economics perspective

Outline Brief introduction to trial-based economic evaluation Economic perspectives on adaptive designs driven primarily by clinical, statistical and regulatory considerations: - Eligibility criteria- Statistical analysis plan - Maintain study power- Others - Outcomes unrelated to efficacy - Group sequential methods Adaptive designs driven primarily by economic considerations Implications for research funders

Trial-based economic evaluation Aim: to maximise health gain given scare resources Method: compare cost and consequences of interventions Economic evaluation: explicit criteria for making choices Incremental cost-effectiveness ratio (ICER): ( C T - C C ) / (E T - E C ) Mean net benefit = Rc.  E -  C

Selecting treatment options Source: Gray A, Clarke P, Wolstenholme J, Wordsworth S. Applied methods of cost-effectiveness analysis in health care. Oxford University Press, 2010.

Cost-effectiveness frontier Source: Gray A, Clarke P, Wolstenholme J, Wordsworth S. Applied methods of cost-effectiveness analysis in health care. Oxford University Press, 2010.

Adaptation of study eligibility criteria Examination of baseline information blinded to treatment assignment Modification of eligibility criteria to increase numbers of patients with desired characteristics May impair relationship between treatment effect and changed patient characteristic(s) →Estimate net benefits from observed costs and effects, construct regression model with a treatment variable and covariates for patient characteristic(s) →Magnitude and significance of coefficients on interaction terms provide estimates of cost- effectiveness by patient characteristic

Adaptations to maintain study power (I) Blinded interim analyses of aggregate data Event rate may be below initial assumption Increase sample size/period of follow-up → Trialists interested in events/time to event → Unlikely to collect comprehensive data on final outcomes, e.g. mortality →Economists interested in all events →Need to link intermediate endpoints to the long- term outcomes of interest →Extrapolation into life expectancy and quality adjusted life expectancy using survival models

Adaptations to maintain study power (II) Stratification of patients at baseline Following interim analysis focus remainder of study on subgroups with greatest event rates, lowest variance Subgroups no longer the focus may contain important economic information: → most trials underpowered on economic endpoints → collect follow-up event, resource use and HRQL data in refined sub-groups Analyse cost-effectiveness data by sub-group avoiding the danger of spurious differences in treatment effects due to chance

Adaptations to maintain study power (III)

Adaptations based on outcomes unrelated to efficacy Dropping dose group(s) with unacceptable rates of adverse effects Requirement that adverse effect is not composite of or directly related to efficacy outcome: → patients in dropped dose group(s) may contain important economic information → follow-up →clinical assessment of risk-benefit comparison may be opaque →utility measures could synthesise risks and benefits →need for sensitive and validated utility measures

Adaptations using group sequential methods Alpha spending approaches to control for Type 1 errors Compelling ethical and statistical evidence needed for terminating early: (i) Futility (ii) Demonstrated efficacy Is there a case for applying group sequential design and analysis methods to economic endpoints? → Frequentist and Bayesian approaches available for estimating sample size requirements for economic endpoints → Very complex: Economists tend to focus on estimation and not hypothesis testing → Typically large variability in economic measures might dictate very large sample sizes, which may be neither financially nor ethically acceptable

Adaptations to statistical analysis plan Prospective SAP generally remains unaltered Limited changes may be acceptable Analytical plan for economic evaluation usually separate Is there a case for introducing a decision criterion for extrapolation modelling for cost-effectiveness? → TOBY trial: Total body hypothermia for neonatal encephalopathy: pCE = 0.69 at 18 months, 0.99 at 18 years → REFLUX Trial: Minimal access surgery amongst people with gastro-oesophageal reflux disease: pCE = 0.46 at 12 months, 0.74 over lifetime → When does it become ‘futile’ to extrapolate cost-effectiveness?

Other unblinded adaptive designs Adaptive designEconomic considerations Dose selection studiesDropped dose group(s) may contain important economic information. Need to take account of adherence and completion. Randomisation based upon relative treatment group responses Regression model required for net benefits with a treatment variable and covariates for patient characteristic(s). Sample size based on interim effect size estimates Statistical adjustments required to protect against bias. May require additional funding/time. Patient population based on treatment-effect estimates Statistical adjustments required to protect against bias. Analyse cost-effectiveness data by subgroup. Endpoint selection based on interim estimate of treatment effect Relationship of new endpoint to economic outcomes may need exploration if subsequent cost-effectiveness modelling is planned

Adaptive designs driven primarily by economic considerations

Value of Information Expected value of perfect information (EVPI) –Equals net benefit from the best decision we could make, minus the net benefit of the decision made based on current information –Perfect information is unattainable; EVPI is therefore the absolute maximum we should be willing to spend on research Expected value of sample information (EVSI) –Represents expected value of conducting a study of a specific size –Can take account of trial costs, opportunity cost of delaying adoption, cost of reversing initial decision, etc –Can be calculated analytically or using simulation Relies on many assumptions

Application of VOI to adaptive trial design Developed by Andrew Willan Sample sizes not dependent on type I and II errors Aims to maximise ENG, i.e. difference between cost of trial and EVSI Single stage EVSI design: sample size driven by potential no of patients and expected values Incorporates pilot data on INBs (means, variances, between-patient variances). Also requires no of beneficiaries; fixed, variable & analytical costs for trial. Two stage EVSI design: requires further complex assumption on optimal proportion of patients in first stage.

Adaptive design of early ECV trial Pilot study: 323 pregnant women presenting in breech position randomised to early v late ECV CIHR funded trial required 730 women per arm to have 80% probability to reject Ho if treatments differed by 8% using two-sided type 1 error of 0.05 Single stage EVSI: assumes b 0 =69, v 0 =3725, σ 2 =217,227, N=1,000,000, C f =$498K,000, C v =$1,600 and C a =$2,000; n=345 Two stage EVSI: assumes α 0 =0.45; n=290

Financial Cost ($) Opportunity Cost ($) EVSI ($)ENG ($)Yield* (%) N=750,000 CIHR (n=730)2,836,00050,3452,298, , Single-stage (n=279) 1,392,80019,2411,585,729173, Two-stage (E(n)=239) 1,265,33516,4831,982,742700,92455 N=1,000,000 CIHR (n=730)2,836,00050,3453,066,003179, Single-stage (n=345) 1,604,00025,7932,364,176736,38345 Two-stage (E(n)=290) 1,533,20020,0012,942,8631,386,66289 N=2,000,000 CIHR (n=730)2,836,00050,3456,136,4883,250, Single-stage (n=547) 2,250,40037,7245,665,4873,377, Two-stage (E(n)=448) 1,934,13330,8976,399,9144,434, Source: Willan A, Kowgier M, Determining optimal sample sizes for multi-stage randomized clinical trials using value of information methods. Clinical Trials 2008;5:

Implications for research funders Few economic perspectives on adaptive trial designs Adaptive designs driven primarily by clinical, statistical and regulatory considerations will have economic implications Only one study has generated an adaptive design driven primarily by economic considerations Need for prospective experiments and validation Adaptive designs for complex interventions, public health interventions