1. Multiple Mini Interviews in the UK
Jon Dowell - University of Dundee
Aileen O’Brien - St George's University of London
Multiple Mini Interviews in the UK
Welcome, thanks for coming, thanks to Martin
Hope you enjoy hearing more about changes
Presented in the spirit of ‘work in progress’. Every school struggling with same issues and we do not claim to have ‘the answer’ 1

2. Rationale
Personal qualities considered ‘important’ but not very reliably assessed by interview and without much predictive validity. (Goho + Blackman cognitive r 0.06, clinical r 0.17)
Introduced for assessing medical school applicants at McMaster University, Canada. (Reiter, Eva et al)
Piloted 2002 – 12 x 8 min. Generalisability since +/- 0.7
Predicted OSCE performance ß 0.44 (interview nil)
Clerkship ratings ß 0.57 (interview nil)
MCC Part 1 (selected components) ß (interview nil or negative)
Supporting evidence from
Canberra - Harris and Owen 2007
Calgary - Brownell et al 2007
Dundee - Ponnamperuma 2008 / 4 station trial 2

3. Domains assessed Dundee Georges Interpersonal and communication skills
Critical thinking, problem solving
Team work
Robustness against stress
Ethical/moral reasoning
Integrity/honesty
Preparation and motivation for medical school
Georges
Empathy
Communication
Organisation and problem solving
Team work
Initiative and resilience
Insight and integrity
Effective learning style
Academic ability & intellect
ability & intellect
(already tested)
Suggest Aileen presents 1-1/2 way through this one. Jon then describes Dundee content and process in slides 5-7. 3

4. Dundee Stations 2010 Role play OCD discussion 1:1
Professional Integrity
Task
Sort it out - cards
Issues in Medicine
Puzzle
Fitness to Practice
Medicine as a career
About you - UCAS
Medications Muddle
Moral Dilemmas

5. How it runs 5

6. Marking Training 30 min general 20 min station specific
Optimal criteria
for each domain
defined
Name Label 6

7. Stations St George’s Standard interview questions Prioritising
Scenario/ breaking bad news
Presentation
Observation
Communication/ giving instructions

8. George’s Marking Scheme
Candidates name………………………………………………
Motivation to do medicine
What are you looking forward to about being a doctor?
Points for guidance:
Personal satisfaction from helping people
Satisfaction from achieving something worthwhile
Working with a diverse range of people
Combining an interest in science with caring for people
Variety of career opportunities once qualified
Job status and security
Good income
The opportunity to work abroad
Intellectual rigour
Lifelong learning and clear professional career structure
Camaraderie from clinical teams
Research collaboration
Please mark out of 5
Poor motivation good motivation
Please also assign an overall impression of the candidate’s suitability to be offered a place at medical school based on their performance at this station (please circle and comment as appropriate).
EXCELLENT ACCEPTABLE CAUSE FOR CONCERN
Red card Comments:

9. George’s results Graduates and non graduates did equally well
No correlation with
Gender
Age
4 year and 5 year course
UKCAT and GAMSAT scores
Reliable (kappa 0.68)

10. George’s Conclusions Successful pilot 2009
Ran this year with 1000 candidates
Smooth (two breakdowns!)
Graduates and school leavers same cut off score for admission

11. Dundee results 2009 n = 452 Cronbach’s Alpha = .70
No correlation in 2009 with
Gender
Age
Overseas candidates
School type
Small age gender interaction

12. 2010 MMIs: Gender Females performed significantly better than males.
t(475) = 3.19, p = < .01
Effect size: 0.14

13. 2010 MMIs: Applicant Group Significant differences between groups.
F(4) = 2.50, p < .05
Graduates/Mature perform significantly better than all other groups

14. Domain Reliability Number of Stations Number of Scores
Cronbach’s alpha
Communication 9 12
.767
Critical Thinking 6
.453
Integrity 1 -
Moral Reasoning 4
.170
Motivation 3
.515
Teamwork 2
.395

15. Correlations between scales
No correlation between UKCAT and MMI scores
No correlation between our UCAS form and MMI scores

16. Dundee MMI - Conclusions
Good reliability in last two years.
All stations were reliable enough. No negative Item- total correlations; no reason to reject any station.
Successfully ran for over 900 candidates with no significant problems (no breakdowns, tears etc!)
Putting more weight on MMI may increase female and graduate/mature candidates compared to the Pre-Interview Score

17. MMI/Pre-Interview 40/60 60/40 80/20 88 Selection Difference Male 129
40/60
60/40
80/20
Male
129
122
121
Female
201
208
209
A-Level 81 79
Scottish
189
182
179
G/M 27 35 38 EU 16 17 OS 18 48 40
Selection
Difference 88

18. So what do the punters think?

19. Assessors’ Survey Dundee
116 /157 assessors (88% of students and 50% of staff) completed online survey to find out how they viewed MMI.
91% thought the process fair.
88% thought stress on applicants ‘moderate’ or less. 19

20. Candidates’ Survey Dundee
324 /433 (75%) of Candidates completed online survey.
94% felt MMI was ‘fair’.
90% felt it is a ‘valid way to assess candidates’.
71% preferred MMI to traditional interview.
‘Student Counselor’ (role play) cited as the most enjoyable (153/172). 20

21. Assessors’ Survey George’s
Equally fair
Equally accurate
Equally able to pick best candidate
Interviews “coached”
MMIs can be boring
MMIs good distinguishing tool

22. Candidates’ survey George’s
5 years- MMI more fair
-MMI more accurate
-interview more difficult
4 years- MMI more difficult
-equally fair and accurate

23. Conclusions and next steps
MMIs becoming an interview tool at many schools. Is it justifiable not to use them?
How to develop new stations? Collectively?
Reliability potentially increased (above 0.8) using Rasch analysis – alters decision for 10%
Predictive validity in UK? – could UKCAT help?
Questions...

24. ? Rasch Modeling
Options to improve?

25. Examiner Track A B C D E Station
Variation in outcomes can be enhanced by using multiple tracks
Examiner 1 2 3 4 5 6 7 8 9 10 A B C D E 4
Track 5 7 4 25

26. Yardstick

27. Effect of using Rasch Reliability based on ‘fair scores’ 0.84
10% different candidates would be given offers. (+ 10% not)
Only the top 169 ‘stable’.
Could not be 100% confident in process this year, so not applied. (complex black box stuff)