Paul Tappenden Jim Chilcott Health Economics and Decision Science (HEDS) School of Health and Related Research (ScHARR) 25 th July 2005 Consensus working group on the use of evidence in economic decision models “Ideal” versus practical - issues of using all available evidence to inform decision model parameters

Presentation focus Key questions 1.“How is evidence currently identified and used to inform model parameters within NICE assessments?” 2.“What would be the implications of implementing NICE’s recommendation regarding the use of all relevant evidence?” 3.“How should we search for, select and use evidence to inform model parameters?” Illustrative case study – irinotecan, oxaliplatin and raltitrexed for advanced colorectal cancer –Update of 2000 assessment

Key stages in assessment process 1.Systematic review of clinical effectiveness 2.Identification, selection and critical appraisal of existing economic analyses 3.Model structuring 4.Identification, selection and ‘appraisal’ of evidence to inform model parameters Use of evidence extends beyond model parameterisation alone

Systematic review of clinical effectiveness Process –Identification, selection and critical appraisal of evidence relating to clinical effectiveness of novel therapies vs. standard treatment –Systematic searches to identify available evidence relating to intervention, disease domain, (comparator), RCTs. Routine searches of grey literature –Broader inclusion of non-RCTs in the absence of strong RCTs –Methodological quality appraised using accepted checklists ACRC case study –Updated systematic review of clinical effectiveness of irinotecan, oxaliplatin and raltitrexed compared to 5-FU/FA in patients with advanced colorectal cancer –Phase 2 trials excluded –17 RCTs of varying methodological quality –Randomisation to first-line / second-line / sequence

Issues for systematic reviews Issues –Definitions of “strong” RCTs? –Potentially relevant but weaker studies RCTs may be excluded from the decision model due to internal and external biases –Should we really use these to inform parameters in the model? If so how should we account for bias and confounding? –Broader inclusion criteria may lead to substantial increases in time required for systematic reviewers and modellers

Identification and use of existing economic studies Process –Existing economic evidence is typically used by TAGs for 3 purposes 1. Evaluating the need for independent economic assessment 2. Informing structural assumptions within the model 3. Informing model parameters –Identified by replacing RCT filter with economic filter ACRC case study –11 cost-effectiveness studies of irinotecan, oxaliplatin and raltitrexed in ACRC –Cost-effectiveness analyses all based on trials in which crossovers following progression were unplanned and unrecorded. –Effectiveness and cost-effectiveness estimates based on overall survival were confounded –Problematic outcomes e.g. cost per progression-free life year gained –Existing economic studies flawed but used to inform types of parameter to be included parameter values

Issues –Systematic searches, and hence, cost-effectiveness reviews may be restricted to intervention under appraisal –Broader economic (or clinical) studies may inform structural assumptions and suggest data to inform model parameters –Additional searches → additional time and resource requirements Issues for economic reviews

Model structuring Process –Parameters used depend on model structure –No guidance on the use of relevant evidence to inform model structure or types of parameters to be included –In practice, model structure guided by What others have done before What you’ve done before What clinical experts say ACRC case study –Economic model framework based upon criticisms of previous analyses –Led by economic outcomes to be evaluated –Suggested methodology formed from review of earlier studies –Survival modelling using indirect comparison to estimate cost per life year gained & cost per QALY gained

Issues for model structuring Issues –May be guided by access to data –“Relevant evidence” for this stage is typically clinical opinion –Whose opinion(s)? –Dealing with lack of clinical consensus between stakeholders –Absence of formal methods for problem-structuring in health economic modelling

Model parameterisation Process –Evidence used to inform model parameters is typically obtained from diverse range of sources –Parameters for treatment effect obtained directly from systematic review of clinical effectiveness –Other parameters may identified by systematic/topic searches –Typical sources include literature clinical opinion cross sectional studies & surveys registry data audit studies etc. –Sources usually reported –Search methods not usually reported –The use of this evidence is usually informally appraised –Selection criteria for such evidence is not usually fully reported

ACRC model parameters Chemotherapies for ACRC Effectiveness parameters –Overall survival Kaplan Meier curves for baseline treatment (weibull parameters) –Hazard ratios for other treatments –Utility scores Resource use & cost parameters –Mean dosage per cycle / RDI –Number of cycles of each chemotherapy regimen actually received –Acquisition costs –Administration resource use & costs

Effectiveness parameters 17 Phase 3 RCTs included in review “Best available evidence” identified from systematic searches and review → 15 RCTs not used in model due to confounding of overall survival benefits and absence of resource use data Overall survival and progression-free survival data made available to ScHARR-TAG by MRC CTU, and from trial publication (Tournigand et al)

Resource use and cost parameters Chemotherapy resource use (no. cycles, RDI) obtained directly from MRC presentations, Tournigand paper, personal communication Evidence relating to costs of line insertion, AEs, consultation, diagnostic tests, pharmacy preparation and dispensing etc. identified from systematic search of cost and cost-effectiveness studies Parameter estimates were usually hidden away in tables or text – not the subject of the study High cost assumption for multiple estimates Evidence on inpatient/outpatient administration obtained from sponsor submission. Checked with clinical advisors Pharmacy cost revised following peer review comments Drug costs from BNF Administration resource use assumptions from advisors, costed using PSSRU Results and review of systematic searches not formally reported

Issues for model parameterisation Issues –Limited of methods for selection of non-clinical evidence –Difficult to gauge reliability –Subjective judgement How well written the paper is How recent the evidence is Internal/external validity (?) Consistency between multiple sources –Identifying all available evidence doesn’t necessarily solve the problem –Systematic searches and review of all model parameters could have considerable impact on time and resource requirements for model population process

Summary issues Decreasing marginal return from additional review effort –At what point do we stop looking for evidence… When we have found multiple sources? When we have found nothing? –Parsimony Definition of ‘relevant’ evidence is highly subjective Data access Identifying, selecting and appraising all relevant evidence is not a panacea. There is always a gap between –the parameters that the data tell us about; and –the parameters that we need to populate the model –need to allow for additional uncertainty