Systematising the process : the role of PSMs in informing model structure Jim Chilcott Technical Director, ScHARR-TAG Suzy Paisley DoH Research Scientist in Evidence Synthesis Paul Tappenden Senior OR Analyst Consensus Working Group on the Use of Evidence in Economic Decision Models MRC HSRC Workshop 26 September 2005

Before we start…

Consensus Working Group on the Use of Evidence in Economic Decision Models Today’s presentation  Define ‘structuring’ within the modelling process  Describe current practice –Evidence used –Tasks undertaken  Discuss ‘What role problem structuring methods (PSMs)?’

Consensus Working Group on the Use of Evidence in Economic Decision Models Defining model structuring  Disease and treatment pathway  Model scope  Modelling method

Consensus Working Group on the Use of Evidence in Economic Decision Models Current practice: HTA Guidelines HTA guidelines(n=32) None17(53%) Description2(6%) Justification4(13%) Sensitivity analysis5(16%) Not available4(13%)

Consensus Working Group on the Use of Evidence in Economic Decision Models Current practice: Modelling guidelines (Philips et al.,2004)  Statement of problem  Justification of approach  Scope / perspective  Structural assumptions  Consistency  Strategies / comparators  Model type  Time horizon  Health states  Cycle length  Parsimony

Consensus Working Group on the Use of Evidence in Economic Decision Models Current practice: Reported (Cooper et al., 2005) HTA monographs (n=42) Diagram of model26(62%) Model structure and assumptions discussed5(12%)

Consensus Working Group on the Use of Evidence in Economic Decision Models Current practice: Observed  8 case study projects  Description model structuring process  Short, semi-structured interviews  Examination of corresponding study reports

Consensus Working Group on the Use of Evidence in Economic Decision Models Evidence consulted SubjectClinical literature Clinical expert (one-off) Clinical expert (ongoing) Existing models 1  2  3  4  5  6  7  8  TOTAL8 (100%)1 (13%)5 (63%)8 (100%)

Consensus Working Group on the Use of Evidence in Economic Decision Models Reporting SubjectMethodology justifiedHealth states /pathways justified? Source of structural assumptions reported? 1Report not yet available 2   3  4    5  6   7  8  TOTAL 5 (71%)2 (29%)3 (43%)

Consensus Working Group on the Use of Evidence in Economic Decision Models Summary of themes  Model structure amended according to data availability  Clinician input (1 st order validation)  Existing models consulted as 1 st step  Iterative structuring process  Complexity of structure shifted both ways  Very limited reporting process of development  MSc dissertation provided most transparent reporting

Consensus Working Group on the Use of Evidence in Economic Decision Models Current practice: Evidence used  Clinical expertise  Published evidence  Existing models  Natural history  Treatment pathway  Modelling method  Effectiveness evidence

Consensus Working Group on the Use of Evidence in Economic Decision Models Current practice: Tasks undertaken  Iteration  Consultation  Evaluation and comparison  Feedback and clarification  Decisions regarding simplicity, complexity, feasibility and inconsistency

Consensus Working Group on the Use of Evidence in Economic Decision Models Why PSMs?  Lack of transparency  Data modelling vs problem modelling  Validity of model (unquestioned question)  Issues that ‘can’t be modelled’  ‘Not qualified to comment’

Consensus Working Group on the Use of Evidence in Economic Decision Models Why PSMs?  Data led models vs evidence informed models  Use of expert opinion  Implicit use of PSM methods  Observed usefulness of action research process

Consensus Working Group on the Use of Evidence in Economic Decision Models PSMs / MSMs MessesProblems SODA SSM SCA Robustness Analysis Drama Theory Decision Conferencing ? Decision analysis Markov Queuing theory Simulation Game Theory

Consensus Working Group on the Use of Evidence in Economic Decision Models Characteristics of PSMs  Techniques to facilitate exploration / definition of a problem situation  Group based  Uncovering options  Enabling group action  Accepting uncertainty  Managing conflict  Facilitator - analyst

Consensus Working Group on the Use of Evidence in Economic Decision Models SODA Individual interviews => Merge maps => Group workshops => Commitment to action Cognitive maps Group map Expand/refine map Group ownership

Consensus Working Group on the Use of Evidence in Economic Decision Models SSM

Consensus Working Group on the Use of Evidence in Economic Decision Models Pros  Problem structuring, not problem solving  Gathering information, generating ideas  Retain flexibility in early stages  Demonstrate understanding of complexity, diversity and inconsistency  Prioritising and simplifying

Consensus Working Group on the Use of Evidence in Economic Decision Models Cons  Methods development required  Adaptation to HTA context  New skills  Ownership  Facilitation of group decision making  Reproducibility

Consensus Working Group on the Use of Evidence in Economic Decision Models Developing a mixed method ‘Soft’ qualitative structuring process ‘Hard’ analytic methods Qualitative model structuring process to inform quantitative decision-making process.