Identifying evidence for decision-analytic models Suzy Paisley DoH Research Scientist in Evidence Synthesis Consensus Working Group on the Use of Evidence in Economic Decision Models : Establishing the Current Situation MRC HSRC Workshop 25 July 2005

Consensus Working Group on the Use of Evidence in Economic Decision Models DoH Award To develop a systematic approach to searching for evidence to inform decision-analytic models  Characteristics and sources of evidence to inform parameter values  Use of evidence in informing other parts of the modelling process

Consensus Working Group on the Use of Evidence in Economic Decision Models Today’s presentation  Why an issue  Current practice –Identification of evidence –Use of evidence  Implications of current practice

Consensus Working Group on the Use of Evidence in Economic Decision Models Why an issue?  Environmental issues  Own experience –Undertaking HTA searches –Peer review of HTA reports

Consensus Working Group on the Use of Evidence in Economic Decision Models ‘Typical’ HTA  Decision problem / research question  Syntheses of existing evidence  Systematic review of clinical- effectiveness  Decision-analytic model of cost- effectiveness

Consensus Working Group on the Use of Evidence in Economic Decision Models Decision problem  search question P =colorectal cancer I =irinotecan C=any relevant O=survival #1colorectal #2irinotecan #3rct #4#1 and #2 and #3

Consensus Working Group on the Use of Evidence in Economic Decision Models Modelling decision problems  Effectiveness  Costs  Resource use / activity  Health states  Utility values  Indirect comparators  Longer term outcomes  ‘Other’ interventions  Natural history  Epidemiology

Consensus Working Group on the Use of Evidence in Economic Decision Models Current practice  Analysis of 27 technology assessment reports (TARs)  Review of HTA guidelines  Content analysis of modelling guidelines

Consensus Working Group on the Use of Evidence in Economic Decision Models Current practice: Reports of models  Use evidence which cannot be identified systematically by PICO focused searches (21/27 (78%) assessments for NICE Paisley, 2001)  Report PICO focussed searches (4/21 (19%) reported additional ‘ad hoc’ searches Paisley, 2001)  Issues of transparency

Consensus Working Group on the Use of Evidence in Economic Decision Models Current practice: Reports of models Types Effectiveness Costs Resource use / activity Health states Utility values Indirect comparators Longer term outcomes ‘Other’ interventions Natural history Epidemiology Sources ‘Published papers’ Routine data Reference sources Local / clinical / expert opinion Sponsor submissions Uses Parameter values Model structure Sensitivity analysis Validation / consistency / calibration

Consensus Working Group on the Use of Evidence in Economic Decision Models Current practice: Methodological guidance  HTA Guidelines  Modelling guidelines

Consensus Working Group on the Use of Evidence in Economic Decision Models Current practice: HTA guidelines (n=32) Identification of evidence: No mention13(41%) Systematic review approach13(41%) Broad approach2(6%) Approach unclear2(6%) Not available2(6%) Name data sources21(66%)

Consensus Working Group on the Use of Evidence in Economic Decision Models Current practice: Modelling guidelines ISPOR Principles of Good Practice (Weinstein et al, 2003)  1 explicit reference to systematic reviews of the literature  41 implicit references to a process of seeking evidence  30/41 (73%) references form statements within recommendations

Consensus Working Group on the Use of Evidence in Economic Decision Models Current practice: Modelling guidelines Model structure10(33%) Data identification7(23%) Data modelling7(23%) Data incorporation0(0%) Model validation6(20%)

Consensus Working Group on the Use of Evidence in Economic Decision Models Current practice: Conclusions  Reports of models –Report PICO focussed searches –Use a wide range of evidence from a wide range of sources throughout the modelling process  Modelling and HTA guidelines –Give limited explicit guidance –Give an implicit indication that range of sources should be used throughout the modelling process

Consensus Working Group on the Use of Evidence in Economic Decision Models Implications for searching: Types of evidence  Effectiveness  Costs  Resource use / activity  Health states  Utility values  Indirect comparators  Longer term outcomes  ‘Other’ interventions  Natural history  Epidemiology  Defining search questions  Defining the evidence base  Transferable / ‘proxy’ data  Grouping questions

Consensus Working Group on the Use of Evidence in Economic Decision Models Implications for searching: Sources of evidence  ‘Published papers’  Routine data  Reference sources  Local / clinical / expert opinion  Sponsor submissions  Prioritising appropriate sources

Consensus Working Group on the Use of Evidence in Economic Decision Models Implications for searching: Uses of evidence  Parameter values  Model structure  Sensitivity analysis  Validation / consistency / calibration  Iterative nature of modelling process  Matching retrieval of evidence to use of evidence

Consensus Working Group on the Use of Evidence in Economic Decision Models Other issues  Aim of the search –What evidence? –What for? –How much?  Methods of selecting evidence  Alternative search methods  Reporting standards

Consensus Working Group on the Use of Evidence in Economic Decision Models Conclusions The process of identifying evidence for models needs to take account of the following issues:  The range of types, sources and uses of evidence  The process of identifying relevant evidence bases and defining search questions  The iterative nature of model development  How much evidence is required  Reporting standards

Consensus Working Group on the Use of Evidence in Economic Decision Models Conclusions Identifying evidence for models is different to identifying evidence for systematic reviews.