1. Prior Elicitation: Teaching Old Dogs New Tricks PSI Annual Conference 15th – 17th May 2017
Kimberley Hacquoil, Statistics, Programming and Data Strategy GSK
Maria Costa, Statistics, Programming and Data Strategy GSK

2. Outline Introduction Background Prior Elicitation eLearning Overview
eLearning Development Process
eLearning Demonstration
Challenges & Learnings
Conclusion
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017

3. Introduction
Prior elicitation is the process through which expert knowledge about a quantity of interest (e.g., treatment effect) is elicited from a subject matter expert and represented through a probability distribution
In order to participate, experts need to understand basic probability concepts
What is a subjective probability?
How can uncertainty be represented through a probability distribution?
Which subjective judgements should be made and how to assess these?
For a prior elicitation to be effective it is important to train experts (typically non-statisticians) in these concepts
This training component is a challenging and time-consuming task
In this talk we will share GSK’s experience in developing and implementing an eLearning to train experts in the fundamental statistical concepts underlying a prior elicitation
Objective: “To create a fit-for-purpose eLearning which would train experts for prior elicitation using Roulette and Quartile methods”
In collaboration with Tony O’Hagan and Grifo Multimedia
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017

4. Decision to conduct elicitation
Background
Post-elicitation phase (facilitator)
Elicitation phase (experts + facilitator)
Pre-elicitation phase (project statistician & physician + facilitator)
Select experts
Documentation
Decision problem or statistical model
Limited /conflicting evidence;
high uncertainty
Problem definition (project team)
Select method
Frame problem
Decision to conduct elicitation
Prepare evidence dossier
Training
Carry out elicitation
Since 2014 GSK have conducted ~ 35 Prior Elicitation sessions
Currently, training happens in a separate ~45 min meeting that occurs prior to the actual elicitation session
This can be challenging as experts are unlikely to be available at the same time (particularly if in multiple time zones) leading to multiple training sessions being required
Development of eLearning streamlines training process and allows “just in time” approach to training
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017

5. Prior Elicitation eLearning Overview
The eLearning curriculum is composed of 4 modules...
Module 0: Introduction
Introduction to eLearning and the concept of expert judgement elicitation
Overview of eLearning modules
Module 1: Probabilities
How subjective probabilities can be used to represent knowledge and uncertainty about events of interest
How uncertainty can be described by a probability distribution and how to interpret it
Module 2: Elicitation
Methods for expert elicitation and overview of structure of elicitation session
How the Quartile and Roulette methods work – which subjective judgements are required?
Group elicitation: the consensus prior and the role of the rational impartial observer
Module 3: Assessment
Assessment: is the expert able to make the necessary judgements using either the Quartile or Roulette methods and does he/she understand the implications of those judgements?
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017

6. eLearning Development Process
Roll out and Embedding
Integration
Development
Maintenance
Scope of work
Storyboards
Planning and Design
Create Modules
Test Modules
GSK Systems
UAT
Implement
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017

7. eLearning Development Process
Roll out and Embedding
Integration
Development
Maintenance
Scope of work
Storyboards
Planning and Design
Create Modules
Test Modules
GSK Systems
UAT
Implement
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017

8. eLearning Development Process
Roll out and Embedding
Integration
Development
Maintenance
Scope of work
Storyboards
Planning and Design
Create Modules
Test Modules
GSK Systems
UAT
Implement
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017

9. eLearning Development Process
Roll out and Embedding
Integration
Development
Maintenance
Scope of work
Storyboards
Planning and Design
Create Modules
Test Modules
GSK Systems
UAT
Implement
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017

10. eLearning Development Process
Roll out and Embedding
Integration
Development
Maintenance
Scope of work
Storyboards
Planning and Design
Create Modules
Test Modules
GSK Systems
UAT
Implement
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017

11. eLearning Development Process
Roll out and Embedding
Integration
Development
Maintenance
Scope of work
Storyboards
Planning and Design
Create Modules
Test Modules
GSK Systems
UAT
Implement
Communication of new eLearning to target audience
Documentation of eLearning completion to ensure compliance
Gather feedback on content and usability
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017

12. eLearning Development Process
Roll out and Embedding
Integration
Development
Maintenance
Scope of work
Storyboards
Planning and Design
Create Modules
Test Modules
GSK Systems
UAT
Implement
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017

13. eLearning Demonstration
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017

14. Challenges & Learnings
Teaching in eLearning environment
Balancing teaching, examples, assessment
Balancing length of eLearning and information overload
Ideal vs too hard/long to implement/program
Coordination issues
Integrating different systems
Understanding what people need and when
Learnings
Project management
Regular meetings/interactions
Talking to the right people
Capture comments in a more structured way
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017

15. Conclusion Where we are...
Currently going through User Acceptance Testing stages of implementation
Selected 2 clinical experts: one with and one without experience in prior elicitation
Also included experienced statisticians (experienced facilitator) as part of UAT
Plan is to roll out eLearning by end of May/ early June (depending on UAT outcome)
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017

16. Acknowledgements Tony O’Hagan Bill O’Shea, Denise Bird, Kate Foster
Nigel Dallow, Nicky Best, Tim Montague
Duncan Richards, Rajnish Saini
Tony O’Hagan
Antonio De Girolamo
Livio Melfi
Roberta Memeo
Ed Morris
Jeremy Oakley
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017

17. Thank You

18. Back Up

19. Screen Shots Module 0 - Introduction
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017

20. Screen Shots Module 1 – Subjective Probabilities
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017

21. Screen Shots Module 1 – Subjective Probabilities
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017

22. Screen Shots
Module 1 – Representing Knowledge through Probability Distributions
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017

23. Screen Shots Module 1 – Interpreting Probability Distributions
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017

24. Screen Shots
Module 1 – Assessing Understanding of Probability Distributions
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017

25. Screen Shots Module 2 – Elicitation Methods
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017

26. Screen Shots Module 2 – Example to Illustrate Elicitation Methods
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017

27. Screen Shots Module 2 – Quartile Method
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017

28. Screen Shots Module 2 – Quartile Method
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017

29. Screen Shots Module 2 – Roulette Method
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017

30. Screen Shots Module 2 – Roulette Method
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017

31. Screen Shots Module 2 – Structure of an Elicitation Session
Session starts with individual elicitation: Each expert is asked to represent their prior knowledge through a distribution
Facilitator then leads group discussion: Objective is to reach a consensus prior, representing the views of a Rational Impartial Observer
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017

32. Screen Shots Module 3 – The Quantity of Interest
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017

33. Screen Shots Module 3 – The Evidence Dossier
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017

34. Screen Shots Module 3 – Assessing the Quartile Method using MATCH
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017

35. Screen Shots Module 3 – Assessing the Quartile Method using MATCH
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017

36. Screen Shots Module 3 – Assessing the Roulette Method
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017

37. Screen Shots Module 3 – Feedback
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017