1. The Effect of Interviewer on Rank List: An Imperfect Science Becomes More Imperfect Daniel Vargo, MD Program Director, General Surgery Associate Professor, Dept. of Surgery University of Utah School of Medicine

2. Disclosures None

3. Background Applicant Selection: Surgery –Job Description Website “Red Book” –Applications 526 last year –Nebulous scoring system –Interviews

4. Background Interviews –Interviewers not HR trained –Go on “gut feeling” “I wanted to like/not like this candidate” –Spend interview trying to validate feelings or impressions Trip up questions –Base opinion on “unusual” things…..

5. His socks didn’t match his pants Who wears a pants suit to an interview He had this weird look in his eyes

6. PGY-1 Summative Meeting Interview Comments vs. Performance –No correlation –Lowest scored intern last two years highest performance

7. “Six Sigma” Evaluation Took process apart Biggest perceived variable –Interviews

8. Question How variable are the interviews? What effect does this variability have on process?

9. Methods 5 years data Interviewers and scores –“Easy Scorers” –“Hard Scorers” Applicants –Strong –Average –Weak

10. Methods Applicant group ranking –Compared with interview panel composition

11. Results 30 Interviewers 303 applicants –909 interviews

12. Applicant Distribution Top 10 Appl. >40 Or NR

13.  “ES” “HS”

14. Interview Scores: All Candidates 303 Appl Strong (70) Average (200) Weak (33) ES (20) 959182 HS (10) 918171 ES= Easy Scorer HS= Hard Scorer

15. Scoring: Strong Applicants HS=Hard Scorer P= NS 70 Appl 0-1 HS (44) 2-3 HS (26) Ave. Int. Score 9592 Ave. Rank Position 67

16. Scoring: Average Applicants 200 Appl. 0 HS (34) 1 HS (85) 2 HS (60) 3 HS (21) Ave Int Score 91898276 Ave Composite Score 830817789740 Rank List Position 22242631

17. Scoring: Average Applicants 200 Appl. 0 HS (34) 3 HS (21) Ave Int Score 9176* Ave Composite Score 830740* Rank List Position 2231 * p<0.05

18. 200 Appl 134 Discussed 66 Not Discussed

19. 200 Appl 134 Discussed 24 No HS106 ≥ 1 HS 66 Not Discussed

20. 200 Appl 134 Discussed 24 No HS106 ≥ 1 HS 66 Not Discussed 6 No HS60 ≥ 1 HS

21. 200 Appl 134 Discussed 24 No HS106 ≥ 1 HS 66 Not Discussed 6 No HS60 ≥ 1 HS

22. Results ≥ 1 HS –Lower interview scores –Lower composite scores –Lower position on rank list –Less likely to be discussed at rank meeting

23. Assumptions Candidate pools are equally distributed Interviewer “toughness” did not vary Other variables in score calculation consistent

24. Conclusions Interviewers do vary in type Scores effect applicants Another area of variability to be addressed in the interview process