Prior Elicitation: Teaching Old Dogs New Tricks PSI Annual Conference 15th – 17th May 2017 Kimberley Hacquoil, Statistics, Programming and Data Strategy GSK Maria Costa, Statistics, Programming and Data Strategy GSK

Outline Introduction Background Prior Elicitation eLearning Overview eLearning Development Process eLearning Demonstration Challenges & Learnings Conclusion Prior Elicitation: Kimberley Hacquoil & Maria Costa PSI 2017

Introduction Prior elicitation is the process through which expert knowledge about a quantity of interest (e.g., treatment effect) is elicited from a subject matter expert and represented through a probability distribution In order to participate, experts need to understand basic probability concepts What is a subjective probability? How can uncertainty be represented through a probability distribution? Which subjective judgements should be made and how to assess these? For a prior elicitation to be effective it is important to train experts (typically non-statisticians) in these concepts This training component is a challenging and time-consuming task In this talk we will share GSK’s experience in developing and implementing an eLearning to train experts in the fundamental statistical concepts underlying a prior elicitation Objective: “To create a fit-for-purpose eLearning which would train experts for prior elicitation using Roulette and Quartile methods” In collaboration with Tony O’Hagan and Grifo Multimedia Prior Elicitation: Kimberley Hacquoil & Maria Costa PSI 2017

Decision to conduct elicitation Background Post-elicitation phase (facilitator) Elicitation phase (experts + facilitator) Pre-elicitation phase (project statistician & physician + facilitator) Select experts Documentation Decision problem or statistical model Limited /conflicting evidence; high uncertainty Problem definition (project team) Select method Frame problem Decision to conduct elicitation Prepare evidence dossier Training Carry out elicitation Since 2014 GSK have conducted ~ 35 Prior Elicitation sessions Currently, training happens in a separate ~45 min meeting that occurs prior to the actual elicitation session This can be challenging as experts are unlikely to be available at the same time (particularly if in multiple time zones) leading to multiple training sessions being required Development of eLearning streamlines training process and allows “just in time” approach to training Prior Elicitation: Kimberley Hacquoil & Maria Costa PSI 2017

Prior Elicitation eLearning Overview The eLearning curriculum is composed of 4 modules... Module 0: Introduction Introduction to eLearning and the concept of expert judgement elicitation Overview of eLearning modules Module 1: Probabilities How subjective probabilities can be used to represent knowledge and uncertainty about events of interest How uncertainty can be described by a probability distribution and how to interpret it Module 2: Elicitation Methods for expert elicitation and overview of structure of elicitation session How the Quartile and Roulette methods work – which subjective judgements are required? Group elicitation: the consensus prior and the role of the rational impartial observer Module 3: Assessment Assessment: is the expert able to make the necessary judgements using either the Quartile or Roulette methods and does he/she understand the implications of those judgements? Prior Elicitation: Kimberley Hacquoil & Maria Costa PSI 2017

eLearning Development Process Roll out and Embedding Integration Development Maintenance Scope of work Storyboards Planning and Design Create Modules Test Modules GSK Systems UAT Implement Prior Elicitation: Kimberley Hacquoil & Maria Costa PSI 2017

eLearning Development Process Roll out and Embedding Integration Development Maintenance Scope of work Storyboards Planning and Design Create Modules Test Modules GSK Systems UAT Implement Prior Elicitation: Kimberley Hacquoil & Maria Costa PSI 2017

eLearning Development Process Roll out and Embedding Integration Development Maintenance Scope of work Storyboards Planning and Design Create Modules Test Modules GSK Systems UAT Implement Prior Elicitation: Kimberley Hacquoil & Maria Costa PSI 2017

eLearning Development Process Roll out and Embedding Integration Development Maintenance Scope of work Storyboards Planning and Design Create Modules Test Modules GSK Systems UAT Implement Prior Elicitation: Kimberley Hacquoil & Maria Costa PSI 2017

eLearning Development Process Roll out and Embedding Integration Development Maintenance Scope of work Storyboards Planning and Design Create Modules Test Modules GSK Systems UAT Implement Prior Elicitation: Kimberley Hacquoil & Maria Costa PSI 2017

eLearning Development Process Roll out and Embedding Integration Development Maintenance Scope of work Storyboards Planning and Design Create Modules Test Modules GSK Systems UAT Implement Communication of new eLearning to target audience Documentation of eLearning completion to ensure compliance Gather feedback on content and usability Prior Elicitation: Kimberley Hacquoil & Maria Costa PSI 2017

eLearning Development Process Roll out and Embedding Integration Development Maintenance Scope of work Storyboards Planning and Design Create Modules Test Modules GSK Systems UAT Implement Prior Elicitation: Kimberley Hacquoil & Maria Costa PSI 2017

eLearning Demonstration Prior Elicitation: Kimberley Hacquoil & Maria Costa PSI 2017

Challenges & Learnings Teaching in eLearning environment Balancing teaching, examples, assessment Balancing length of eLearning and information overload Ideal vs too hard/long to implement/program Coordination issues Integrating different systems Understanding what people need and when Learnings Project management Regular meetings/interactions Talking to the right people Capture comments in a more structured way Prior Elicitation: Kimberley Hacquoil & Maria Costa PSI 2017

Conclusion Where we are... Currently going through User Acceptance Testing stages of implementation Selected 2 clinical experts: one with and one without experience in prior elicitation Also included experienced statisticians (experienced facilitator) as part of UAT Plan is to roll out eLearning by end of May/ early June (depending on UAT outcome) Prior Elicitation: Kimberley Hacquoil & Maria Costa PSI 2017

Acknowledgements Tony O’Hagan Bill O’Shea, Denise Bird, Kate Foster Nigel Dallow, Nicky Best, Tim Montague Duncan Richards, Rajnish Saini Tony O’Hagan Antonio De Girolamo Livio Melfi Roberta Memeo Ed Morris Jeremy Oakley Prior Elicitation: Kimberley Hacquoil & Maria Costa PSI 2017

Thank You

Back Up

Screen Shots Module 0 - Introduction Prior Elicitation: Kimberley Hacquoil & Maria Costa PSI 2017

Screen Shots Module 1 – Subjective Probabilities Prior Elicitation: Kimberley Hacquoil & Maria Costa PSI 2017

Screen Shots Module 1 – Subjective Probabilities Prior Elicitation: Kimberley Hacquoil & Maria Costa PSI 2017

Screen Shots Module 1 – Representing Knowledge through Probability Distributions Prior Elicitation: Kimberley Hacquoil & Maria Costa PSI 2017

Screen Shots Module 1 – Interpreting Probability Distributions Prior Elicitation: Kimberley Hacquoil & Maria Costa PSI 2017

Screen Shots Module 1 – Assessing Understanding of Probability Distributions Prior Elicitation: Kimberley Hacquoil & Maria Costa PSI 2017

Screen Shots Module 2 – Elicitation Methods Prior Elicitation: Kimberley Hacquoil & Maria Costa PSI 2017

Screen Shots Module 2 – Example to Illustrate Elicitation Methods Prior Elicitation: Kimberley Hacquoil & Maria Costa PSI 2017

Screen Shots Module 2 – Quartile Method Prior Elicitation: Kimberley Hacquoil & Maria Costa PSI 2017

Screen Shots Module 2 – Quartile Method Prior Elicitation: Kimberley Hacquoil & Maria Costa PSI 2017

Screen Shots Module 2 – Roulette Method Prior Elicitation: Kimberley Hacquoil & Maria Costa PSI 2017

Screen Shots Module 2 – Roulette Method Prior Elicitation: Kimberley Hacquoil & Maria Costa PSI 2017

Screen Shots Module 2 – Structure of an Elicitation Session Session starts with individual elicitation: Each expert is asked to represent their prior knowledge through a distribution Facilitator then leads group discussion: Objective is to reach a consensus prior, representing the views of a Rational Impartial Observer Prior Elicitation: Kimberley Hacquoil & Maria Costa PSI 2017

Screen Shots Module 3 – The Quantity of Interest Prior Elicitation: Kimberley Hacquoil & Maria Costa PSI 2017

Screen Shots Module 3 – The Evidence Dossier Prior Elicitation: Kimberley Hacquoil & Maria Costa PSI 2017

Screen Shots Module 3 – Assessing the Quartile Method using MATCH Prior Elicitation: Kimberley Hacquoil & Maria Costa PSI 2017

Screen Shots Module 3 – Assessing the Quartile Method using MATCH Prior Elicitation: Kimberley Hacquoil & Maria Costa PSI 2017

Screen Shots Module 3 – Assessing the Roulette Method Prior Elicitation: Kimberley Hacquoil & Maria Costa PSI 2017

Screen Shots Module 3 – Feedback Prior Elicitation: Kimberley Hacquoil & Maria Costa PSI 2017