Economic evaluations alongside clinical trials: what they contribute, how they are performed and their limitations Dr. Stavros Petrou Presentation to Nottingham CTU 19 th June 2008

What is economic evaluation? Premise: scarce (health care) resources Aim: to maximise health gain with the available resources Method: compare cost and consequences of interventions Balance: about costs and consequences, inputs and outputs Economic evaluation: explicit criteria for making choices.

Definition of economic evaluation Definition of economic evaluation: The comparative analysis of alternative courses of action in terms of both their costs and their consequences (Drummond et al, 1997) Requires: –a comparison of two or more alternatives –examination of both costs and consequences

Types of economic evaluation NO Costing study YES Is effectiveness of interventions equal? YESNO Cost minimization study Can all outcomes be valued in monetary terms ( e.g. willingness to pay)? YES Cost benefit analysis NO Can outcomes be measured as quality adjusted life years? YES NO Cost-effectiveness analysis Cost-utility analysis Is there good evidence on effectiveness of interventions being compared?

Economic evaluation alongside trials Two independent groups Control groupTreatment group Patient(Cost, Effect) 1( C C 1, E C 1 ) 2( C C 2, E C 2 ) 3( C C 3, E C 3 ). n C ( C C n, E C n ) Mean:( C C, E C ) Patient(Cost, Effect) 1( C T 1, E T 1 ) 2( C T 2, E T 2 ) 3( C T 3, E T 3 ). n T ( C T n, E T n ) Mean:( C T, E T )

C T - C C E T - E C Economic evaluation alongside trials Two independent groups Control groupTreatment group Mean:( C C, E C ) Mean:( C T, E T ) Incremental cost-effectiveness ratio

The cost-effectiveness plane C New treatment more costly New treatment more effective New treatment less effective New treatment less costly NE NW SWSE Existing treatment dominates New treatment dominates New treatment less costly but less effective New treatment more effective but more costly

New treatment cost-effective New treatment cost-ineffective The cost-effectiveness plane C New treatment more costly New treatment more effective New treatment less effective New treatment less costly NE NW SWSE Maximum acceptable ICER

NICE: method of operation Preferred measure of cost-effectiveness: –Quality-adjusted life year (QALY) –Alternatives - e.g. cost per life year gained - acceptable No absolute threshold for level of acceptability: –no empirical basis for setting a value –may in some circumstances want to ignore threshold –A set threshold implies efficiency has absolute priority over other objectives (e.g. fairness) –many technology suppliers are monopolies; a threshold would discourage price competition

Cost per QALY results from NICE Those in bold: rejected Source: N Devlin, D Parkin. Does NICE have a cost-effectiveness threshold and what other factors influence its decisions? A binary choice analysis. Health Economics 2004: 13(5):

Why QALYs as a measure of outcome? To use cost-effectiveness as a guide to decision- making, we need to compare the c-e of different uses of resources Therefore we need an effectiveness measure that can be used in a wide range of settings: Life-years gained –but only where survival is main outcome Quality adjusted life years (QALYs) –Composite of survival and quality of life

Time in years t Health profile with intervention Health profile without intervention QALYs gained Quality of life valuation u

The EuroQol EQ-5D The following questions are designed to tell us about your state of health today. Please look at each group of questions, and then tick the statement which best describes you own health state today. You should make five ticks in all, one for each group. Mobility I have no problems walking about____ I have some problems walking about____ I am confined to bed____ Self-care I have no problems with self-care____ I have some problems washing or dressing myself____ I am unable to wash or dress myself____ Usual activities I have no problems performing my usual activities____ I have some problems with performing my usual activities____ I am unable to perform my usual activities____ Pain/discomfort I have no pain or discomfort____ I have moderate pain or discomfort____ I have extreme pain or discomfort____ Anxiety/depression I am not anxious or depressed____ I am moderately anxious or depressed____ I am extremely anxious or depressed____

Tariffs for the EuroQol EQ-5D Coefficient Constant0.081 Mobility - Some problems Confined to bed0.314 Self care - Some problems Unable to wash/dress0.214 Usual activities - Some problems Unable to perform0.094 Pain/discomfort - Moderate Extreme0.386 Anxiety/depression - Moderate Extreme0.236 N3 (Level 3 at least once)0.269

Why randomised trials? Most treatments do not have large effects; reliably detecting moderate effects requires studies that simultaneously avoid: –moderate bias proper randomisation intention-to-treat analysis avoidance of inappropriate sub-group analysis –moderate random error adequate size

Why economic assessment in clinical trials? Many health economists advocate models using lots of data sources: trial, non-trial, summary data etc But...issues of bias and random error also affect incremental resource use and health outcomes And, trials provide patient-level data, useful for: –Dealing with patient heterogeneity –Examining covariance, e.g. between costs and outcomes –Building and validating models, eg to extrapolate Trials allow prospective measurement of resources & outcomes of interest Incremental cost of economic evaluation alongside trials is low

UK Collaborative ECMO Trial Pragmatic RCT 185 mature ( 35 weeks, 2kgs) infants with severe respiratory failure (ox. index 40) Infants recruited from 55 centres in Randomisation to ECMO: 1 of 5 specialist centres, cannulated, ECMO support Randomisation to CM: conventional care Outcomes: survival without severe disability up to 7 years of age

Design and analytical issues Costs: measurement and valuation Consequences: measurement and valuation Analytical issues: within and beyond RCTs

Perspectives and types of costs Direct costs –Health care system –Other care inputs, e.g. social services –Patient, family, carer expenses Informal care costs –Opportunity cost of unpaid informal care Indirect costs –Time off work, reduced productivity –Early retirement –Premature mortality Transfer payments –Payments such as social security benefits that redistribute output with no exchange of goods or services

Three elements of cost Resource use (cost generating event) –a day in hospital –a GP consultation Unit cost –cost per in-patient day / per hospitalisation –cost per GP consultation / per GP minute Cost –the product of resource use and unit costs

Measurement of resource use All significant resource inputs during first 7 years of life Trial data collection forms: –transport (mode, distances) –duration and intensity of neonatal care Observational research for infant death Postal questionnaires, validated by information from hospital records: – post-discharge hospital and social service utilisation GP records: –community service utilisation

Valuation of resource use Unit costs employed to value resource use Neonatal care – top down methodology Readmissions - Reference cost schedules Community service utilisation - Published cost data (previous studies, PSSRU, etc.) Drugs – BNF £, prices

Time horizon and discounting Should extend far enough into the future to capture all costs and consequences of interventions being evaluated ECMO Study – time horizon initially reflected that of RCT Costs (and consequences) occurring beyond the first year of life must be reduced to present values Rationale for discounting - Time preference - Opportunity cost – market basis

Mean costs and mean cost differences Cost categoryECMO (n=93)CM (n=92) Mean(SD)Mean(SD)Mean difference P value Death434(665)846(714)-412< Transport2147(3080)296(777)1851< Initial hospitalisation22996(21618)6143(13144)16853< Hospital readmissions1518(3868)1127(3063) Outpatient hospital care 970(1592)496(1008) Community care1976(2882)1247(2647) Other costs229(1255)74(182) Total costs30270(24380)10229(18356)20041< Source: Petrou S, Bischof M, Bennett C, Elbourne D, Field D, McNally H. Cost-effectiveness of neonatal ECMO based on seven year results from the UK Collaborative ECMO Trial. Pediatrics 2006; 117(5):

Bootstrap mean cost differences Cost category Mean cost difference (ECMO – CM) Bootstrap mean cost difference (95% CI) Death (-614, -209) Transport (1251, 2497) Initial hospitalisation ,826 (11,775, 22,044) Hospital readmissions (-603, 1334) Outpatient hospital care (97, 875) Community care (-62, 1549) Other costs (-13, 443) Total costs (13690, 26318) Source: Petrou S, Bischof M, Bennett C, Elbourne D, Field D, McNally H. Cost-effectiveness of neonatal ECMO based on seven year results from the UK Collaborative ECMO Trial. Pediatrics 2006; 117(5):

Measurement of outcomes at 7 years Survival period life years gained Neurodevelopmental assessments performed by developmental psychologist across 6 domains: –cognitive ability- behaviour –neuromotor skills- hearing –general health- vision Each domain defined as normal, impaired or mild, moderate or severely disabled Overall status defined by highest degree of impairment or disability in any domain disability-free life years gained Limitations of QALYs in childhood context

ECMO (n=93) CM (n=92) Deaths Before discharge28 (30.1%)54 (58.7%) Between discharge and one year2 (2.2%)0 (0.0%) Between one years and four years1 (1.1%)0 (0.0%) Between four years and seven years0 (0.0%) Total31 (33.3%)54 (58.7%) Loss to follow-up Between discharge and one year1 (1.1%) Between one year and four years2 (2.2%) Between four years and seven years3 (3.2%)1 (1.1%) Total6 (6.5%)4 (4.3%) Final assessment at seven years of age Severe disability3 (3.2%)0 (0.0%) Moderate disability9 (9.7%)6 (6.5%) Mild disability13 (14.0%)11 (12.0%) Impairment only21 (22.6%)15 (16.3%) No abnormal signs or disability10 (10.8%)2 (2.2%) Known survivors with no disability31/56 (55.4%)17/34 (50.0%)

Cost-effectiveness of neonatal ECMO Incremental cost-effectiveness ratio = C / E = £13,385 per life year gained or = £23,566 per disability-free life year gained NB: Natural or physical unit of outcome, which ignores full range of consequences.

Mean net benefits of neonatal ECMO Mean net benefits = Rc. E - C RcLife year gainedDisability-free life year gained Mean net benefit 95% CIMean net benefit 95% CI 0-20,130(-26,027, -13,522)-19,949(-25,997, -13,375) 10,000-5,299(-14,475, 3,750)-11,387(-20,394, -244) 20,0009,532(-6,691, 25,351)-2,825(-18,377, 13,804) 30,00024,362(1,474, 47,314)5,737(-16,245, 29,393) 40,00039,193(8,926, 69,513)14,299(-14,863, 44,781) 50,00054,024(15,496, 91,567)22,861(-13,246, 60,319)

Are trial-based economic evaluations sufficient? Under the right circumstances, Yes Like clinical evidence, economic evidence can stand alone or be synthesised Requirements: reasonable comparators, adopts final outcomes, collects data on a sufficiently broad set of services, adequately powered, sufficiently long follow-up, representative patient population Problems: Use of inappropriate statistical tests, lack of power, failure to handle missing data, lack of intention- to-treat analysis, follow-up too short, lack of transferability

Was the ECMO trial-based economic evaluation sufficient? Reasonable comparator Representative patient population Intention to treat analysis Adequately powered Appropriate perspective Sufficiently long follow-upx Adopts final outcomes? Appropriate statistical tests Handled missing data

Longer-term cost-effectiveness Simple decision-analytic model Assumptions: –restricted to first 18 years of life –survivors to 7 years survive to 18 years –disability status at age 7 does not vary –excess annual costs during years 4-7 continue during years 8-18 ICER: £7344 per life year gained £11802 per disability-free life year gained

Vehicles for economic evaluation Prospective collection of data alongside randomised controlled trials –Least subject to bias, control over instruments, low incremental cost. May need to supplement. Prospective collection of data alongside non- randomised studies –More subject to possible bias, control over instruments, low cost. Retrospective analysis of available data –Low control over design and data, subject to bias. Modelling study –Data from different sources, combined in decision- analytic models. Hard to validate. Useful and often unavoidable adjunct to trials.

Limitations of trials as a vehicle for economic evaluation Trial limitations Inappropriate or partial comparisons More than one trial Partial measurement Unrepresentative practice Intermediate outcomes Limited follow-up No trials NICE Examples Temozolomide (recurrent malignant glioma) Drugs for Alzheimers Riluzole (resource use) Glycoproteins Beta interferon (MS) Implantable cardioverter defibrillators Liquid-based cytology

Modelling Hence modelling aims to address all these questions and can be used instead of or as a complement to trial evidence –Structure the economic question –Extrapolate beyond observed data –Links intermediate and final endpoints –Generalizes results to other settings/patient groups –Synthesises evidence and can create head-to-head comparisons where RCTs dont exist –Can indicate the need for further research

Models should: Represent a simplification of the real world Encourage decision-makers to be explicit Reflect current clinical practice and use appropriate comparators Be based on the best quality data available Cover the appropriate time period Include sensitivity analysis to explore uncertainty of data inputs and model structure Be transparent and reproducible Have internal and external validity

The realities of research funding A large proportion of funding for economic evaluation is attached to trials Three options available to analysts: –Satisfy expectation of standard trial-based economic evaluation and risk misleading results –Refuse to collaborate –Pragmatic collaboration : Seek opportunities to use modelling to inform trial design Ensure sufficient budget for analysis which includes synthesis and modelling

A pragmatic way forward? Synthesis and modelling with updated evidence Identify decision problems Synthesis and modelling given available evidence Setting of research priorities Primary research (e.g. RCTs)

Suggested checklist for assessing economic evaluations 1)Was question well-defined? 2)Were alternatives clearly described? 3)What evidence on effectiveness was used? 4)Were resources measured and valued fully & appropriately? 5)Was discounting necessary and was it performed? 6)Were incremental costs and outcomes analysed? 7)Was an adequate sensitivity analysis performed? 8)Are the results adequate to inform purchasing? 9)Are the conclusions justified? 10)Are the results applicable to the local population? Source: Drummond MF, OBrien, B, Stoddart GL, Torrance GW. Methods for the economic evaluation of health care programmes. 2nd edition. Oxford: Oxford University Press, Drummond MF, Jefferson T, et al. Guidelines for authors and peer reviewers of economic submissions to the BMJ. BMJ 1996; 313: