1. First use of the multiple mini- interview (MMI) system in a UK veterinary school Carol Gray MRCVS Programme Director BVSc University of Liverpool

2. The “Old” System Stage 1Academic achievements Stage 2Personal statement (x 2 scorers) Stage 3Panel interview (2 interviewers)

3. How important is academic ability? Very important Important Not very important

4. The answer? Previous academic performance → –23% variance in medical school performance –<3% variance in postgraduate performance (Ferguson, James & Madeley, BMJ 2002)

5. HEFCE study (all courses) 79,005 18 yr olds who went to University in 1997-8, followed through to 2000-1

6. Copyright ©2005 BMJ Publishing Group Ltd. McManus, I C et al. BMJ 2005;331:555-559 Fig 1 Outcome of students in relation to A level grades in all subjects. Grades are based on best three. A=10, B=8, C=6, D=4, and E=2 points

7. So why not just rely on academic achievement? Widening participation issues Poor correlation with postgraduate performance Are we just selecting for success on the course?

8. The personal statement “ Virtually no research has examined the predictive power of personal statements” (Ferguson, James, Madeley, BMJ 2002) “We are now admitting cohorts of medical students who………. have been constructing their CVs since the age of 9” (Greenhalgh, BMJ 2004)

9. Pros and cons Think about the advantages and disadvantages of personal statements

10. Why use personal statements? Evidence of work experience, motivation BUT –Plagiarism –Coaching

11. What else could we use? Work experience reports?

12. Problems with interviews Poor inter-rater reliability on PS scores Subjective interview scores Lack of consistency regarding questions asked at interview Disagreement between panel members

13. Possible Solutions? Structure the interview Train the interviewers Increase number of people on panel OR change the system

14. How about a lottery for places? It DOES happen! (some EU veterinary schools) May be used as sole method or combined with interview

15. OR….. Psychometric testing Problem – need expert input

16. A different system First reported by Eva et al, McMaster University (Medical Education 2004) Multiple mini-interview, based on OSCE format

17. The Set-up Station 1 Station 2 Station 3 - rest Station 4 Station 5 Station 6 Station 7 - rest Station 8 Station 9

18. But what are we looking for? What makes a good vet?

19. How did we decide? We copied medical schools!

20. The stations 1Presentation (motivation) 2Ethics 4Case study (critical thinking) 5Welfare issues (ethics) 6Knowledge of the profession 8Scientific paper (critical thinking) 9Stress in the profession

21. Example – station 2 Both you and a friend have been picked to play hockey for your country. Your friend has made the first team, and you are first reserve. However, a few weeks before the game, your friend confides in you that he/she is struggling with a few minor injuries, so has started taking anabolic steroids to help with recovery from these injuries. Use of anabolic steroids is prohibited in all sports in the UK and Europe, but hockey players are rarely tested.

22. Scoring system COMMUNICATION SKILLS 1 2 3 4 5 6 Provided very Argued well from poor argumentsa number of viewpoints AWARENESS OF RELEVANT ETHICAL ISSUES 1 2 3 4 5 6 Completely unaware of Aware of a complete range ofethical issues SUITABILITY FOR PROFESSION (“GLOBAL SCORE”) 1 2 3 4 5 6 Completely unsuitableExtremely suitable

23. Statistical interpretation

24. Anonymous post-interview questionnaire – responses on 5-point Likert scale 1.The interview allowed me to demonstrate my knowledge of animals and the veterinary profession 2.The interview stations were fair to everyone 3.The interviewers encouraged me to put forward my own opinions 4.The interview set-up was better than a “traditional” interview panel 5.I enjoyed the interview today

25. Feedback results

26. Positive comments I feel it was a fair judgment on the character of the interviewees. V. progressive – reinforces my opinion of Liverpool keeping up with the times….. ….good that every student was asked same questions, giving a fair test A refreshing change. The fact that many people interviewed you was fair.

27. Negative comments I didn’t get to talk about my personal statement or work experience very much I felt 5 min wasn’t quite long enough to settle into each station It might have been nice to know beforehand that the interview was not going to be traditional – I might have changed the way I prepared

28. The future of the admissions process Personal statement to provide evidence of circumstances? More emphasis on work experience (but see above) via questionnaire More interviews?

29. The future of this study Comparison between 2005-06 (traditionally selected) and 2006-07 (MMI selected) cohorts Success on course Success in career