Guidelines for the reporting of evidence identification in decision models: observations and suggested way forward Louise Longworth National Institute for Health and Clinical Excellence

Outline of presentation Background NICE Methods Guide to Technology Appraisal Suggested framework for reporting Discussion

Background Previous meeting –Difficulty of judging methods of evidence identification/selection as these are not currently always fully reported Why do we need guidelines on reporting? –Transparency –Consistency –Reproducibility Content of presentation: –The reporting of evidence identification and selection for economic models. –NOT addressing wider issues of appropriate methods of identifying or selecting evidence. –NOT addressing reporting of aspects of economic modelling other than evidence identification and selection.

NICE Guide to Methods of TA - Health effects Section “All health effects should be identified and quantified, with all data sources clearly described. As a reference case, all evidence on outcomes should be obtained in a systematic review, which can be defined as the systematic location, appraisal and synthesis of evidence in order to obtain a reliable overview.”

NICE Guide to Methods of TA - Valuation of health effects Section “For the reference case, a standardised and validated generic (non-disease-specific) instrument is required to quantify the effects of technologies in terms of HRQL for patients. The value of changes in patients’ HRQL (that is, utilities) should be based on public preferences elicited using a choice-based method. Evidence should be presented to indicate that any data taken from the literature have been identified systematically.”

NICE Guide to Methods of TA – Resource use and costs Section “The methods of identification of resource use and unit cost data are not as well defined as for evidence for the identification of clinical effectiveness. Where cost data are taken from the literature, the methods used to identify the sources should be defined. Where several sources are available, a justification for the costs chosen should be provided…”

NICE Guide to Methods of TA – Valuation of resource use Section “…As far as possible, estimates of unit costs and prices for particular resources should be used consistently across appraisals. A first point of reference in identifying such costs and prices should be any current official listing published by the Department of Health and/or the Welsh Assembly Government.”

Suggested framework for reporting of evidence identification and selection 1.Method/s of identification of evidence 2.Details of identification/selection of evidence & rationale 3.Quality and relevance of selected evidence

Syst. review Non-syst. review Specific to other parameter Standard/ recommend Identified by expert/ consultee Expert opinion Primary research Natural history RR AEs Resource Use Unit costs H-R utilities Chosen methods of identification

Identification of evidence Syst. review Non-syst. review Specific to other parameter Standard/ recommend Identified by expert/ consultee Expert opinion Primary research Natural history X √ XXXXX RR √ XXXXXX AEs XX √ XXXX Resource Use X √ XX √ XX Unit costs XXX √ XXX H-R utilities √ XXXX √ X

Identification of evidence Syst. review Non-syst. review Specific to other parameter Standard/ recommend Identified by expert/ consultee Expert opinion Primary research Natural history X 1 XXXXX RR 1 XXXXXX AEs XX 1 XXXX Resource Use X 1 XX 2 XX Unit costs XXX 1 XXX H-R utilities 1 XXXX 2 X

Syst. review Non-syst. review Specific to other parameter Standard/ recommend Identified by expert/ consultee Expert opinion Primary research RR 1 XXXXXX Natural history ? ? ??? 2 X Resource use - events X 1 XXXXX Resource use – monitoring XXX 1 X 2 X Unit costs XX 11 XXX H-R utilities 1 XXXX2X Example – Statins TAR

Syst. review Non-syst. review Specific to other parameter Standard/ recommend Identified by expert/ consultee Expert opinion Primary research Treatment effect 1 XXXXXX Natural history X X 1 XX X X Adverse events XX 1 XXXX Resource use – treatment XX 1 X X XX Resource use – adverse events XXXX 1 XX Resource use – monitoring XXX 1 X 2 X Cost data XXX 11 XX H-R utilities 1 XXXXXX Example – Advanced ovarian cancer TAR

Details of identification/selection Rationale for choice of method –E.g. If specific to alternative parameter state why Process of identification/selection of evidence –Details of method employed (e.g. methods of systematic review) –Details of how final source was selected from all the identified evidence (e.g. inclusion/exclusion criteria, most relevant population) –Details of selected evidence –Details of evidence synthesis (if appropriate)

Example 1: Systematic review (for health effects?) Search strategy Data sources searched Inclusion criteria Exclusion criteria Process of review (e.g. no of reviewers, solution in case of disagreement between reviewers) Details of excluded studies Details of included studies Methods of data synthesis if appropriate

Example 2: source specific to other parameter For example, the source/s of evidence for adverse event data may be limited to link to the source used for the effectiveness data. Rationale for choice of method Rationale for choice of data source Description of data source

Example 3: Expert opinion Rationale for using expert opinion as the source of evidence Rationale for choice of expert/s Methods of elicitation –What was asked? –How was it asked? Framework for elicitation (e.g. Delphi panel) Description of data

Quality and relevance of selected evidence –Comment on the quality of the evidence using validated checklists as appropriate –Comment on the generalisability of the evidence to the specific decision problem

Issues for consideration This information is presented in TARs for most (although not consistently all) parameters This presentation focuses on the methods of reporting the selection of evidence to inform model parameters –What about the selection of evidence to inform model structures? Are we able to produce guidelines for reporting data selection, without agreeing appropriate methods of data identification and selection? Suggested framework –Methods of identification –Details of identification and selection –Statement of quality and relevance