The Effect of Interviewer on Rank List: An Imperfect Science Becomes More Imperfect Daniel Vargo, MD Program Director, General Surgery Associate Professor, Dept. of Surgery University of Utah School of Medicine

Disclosures None

Background Applicant Selection: Surgery –Job Description Website “Red Book” –Applications 526 last year –Nebulous scoring system –Interviews

Background Interviews –Interviewers not HR trained –Go on “gut feeling” “I wanted to like/not like this candidate” –Spend interview trying to validate feelings or impressions Trip up questions –Base opinion on “unusual” things…..

His socks didn’t match his pants Who wears a pants suit to an interview He had this weird look in his eyes

PGY-1 Summative Meeting Interview Comments vs. Performance –No correlation –Lowest scored intern last two years highest performance

“Six Sigma” Evaluation Took process apart Biggest perceived variable –Interviews

Question How variable are the interviews? What effect does this variability have on process?

Methods 5 years data Interviewers and scores –“Easy Scorers” –“Hard Scorers” Applicants –Strong –Average –Weak

Methods Applicant group ranking –Compared with interview panel composition

Results 30 Interviewers 303 applicants –909 interviews

Applicant Distribution Top 10 Appl. >40 Or NR

 “ES” “HS”

Interview Scores: All Candidates 303 Appl Strong (70) Average (200) Weak (33) ES (20) HS (10) ES= Easy Scorer HS= Hard Scorer

Scoring: Strong Applicants HS=Hard Scorer P= NS 70 Appl 0-1 HS (44) 2-3 HS (26) Ave. Int. Score 9592 Ave. Rank Position 67

Scoring: Average Applicants 200 Appl. 0 HS (34) 1 HS (85) 2 HS (60) 3 HS (21) Ave Int Score Ave Composite Score Rank List Position

Scoring: Average Applicants 200 Appl. 0 HS (34) 3 HS (21) Ave Int Score 9176* Ave Composite Score * Rank List Position 2231 * p<0.05

200 Appl 134 Discussed 66 Not Discussed

200 Appl 134 Discussed 24 No HS106 ≥ 1 HS 66 Not Discussed

200 Appl 134 Discussed 24 No HS106 ≥ 1 HS 66 Not Discussed 6 No HS60 ≥ 1 HS

200 Appl 134 Discussed 24 No HS106 ≥ 1 HS 66 Not Discussed 6 No HS60 ≥ 1 HS

Results ≥ 1 HS –Lower interview scores –Lower composite scores –Lower position on rank list –Less likely to be discussed at rank meeting

Assumptions Candidate pools are equally distributed Interviewer “toughness” did not vary Other variables in score calculation consistent

Conclusions Interviewers do vary in type Scores effect applicants Another area of variability to be addressed in the interview process