1. Choosing our Residents Michael J. Ruckenstein M.D. Otorhinolaryngology: Head and Neck Surgery at PENN Excellence in Patient Care, Education and Research since 1870

3. Residency Selection 2 Anecdotes The ‘Superstar’ The ‘Sub’ David Kennedy quoting ?Mike Johns “If you are picking 4 residents, 1 will be a superstar, 2 will be average, and 1 will be an underperformer”

4. Shouldn’t we be doing better? Entering medical school is a highly selective process Otolaryngology – HNS is a highly competitive match Penn is highly coveted, ‘top 10’ residency Shouldn’t we be able to select 4 ‘superstars’?

5. How we are choosing - all

6. How we are choosing - OTO

7. Cognitive Abilities USMLEs Clerkship grades AOA status Correlate with performance on inservice scores Do not correlate with resident performance during residency

8. Letters of Rec Not standardized Written in ‘code’ that isn’t always shared Recapitulate ‘cognitive’ skills May or may not correlate on all relevant factors Vary according to who is writing them Vary on who is reading them!

9. Dean’s Letter (MSPE) Include the positive, not the negative Fantasyland – ‘Jewish Day School phenomenon’ 90% of the applicants are above average!

10. The Interview Unstructured Individual No set criteria to evaluate Designed to ‘get the candidate talking’ The unstructured, casual conversation interview is the MOST invalid method of accessing candidates!

11. Most valid interview Individual or group Should assess a specific area Structured Situational > Job-Related> Psychological

12. How can we improve? Define characteristics (Dimensions) of an ideal resident Evaluate for factors that may predict for these qualities Dimensions Cognitive Abilities Personality Technical Abilities Research Abilities

13. Cognitive Abilities These can be readily assessed using existing data USMLEs Clerkship grades (highly variable between institutions) AOA What do they mean in terms of performance? We may wish to look for a ‘minimum’ as opposed to selecting for the highest scores

14. Personality Conscientiousness Dependability Attention to detail Follow through Collegiality/agreeableness Ability to work well with others Ability to take direction Being nice Empathy Ability to read in emotions and take their perspectives

15. Personality - 2 Communication Skills Ability to communicate with colleagues Ability to communicate with patients GRIT Perseverance and passion for long term goals Clutch Ability to function and act under pressure Moral and Ethical Standards

16. Personality - 3 Some data to indicate this Dimension may be the most predictive of success How do you assess for all these Facets Self reporting – prone to confabulation Scales Can be included in standardized LOR Interview – structured to assess one or more of these facets

17. Research Research productivity prior to entering residency not correlated with performance during residency May be desirable for a program to include research skills in their residency ‘mix’ Some evidence that research may be misrepresented in application Likely good topic for structured interview Assess performed research Presentation of a ‘research problem’

18. Technical Skills Innate ability to perform surgical tasks Age, sex, gender, grades, manual dexterity do not correlate with operative competence Visual spatial abilities – the skill of mentally manipulating an object in 3-D correlates with initial acquisition of a surgical skill Knot tying Z-plasty

19. Mental Rotation Test

20. Surgical Abilities Technical ability not the sole factor in making a ‘good surgeon’ Surgical decision making Preoperative determination of indications for surgery Intraoperative decisions Post-operative management

21. Multicenter Studies Retrospective Graduates Prospective Incoming residents/applicants

22. Protocol Cognitive Assessment USMLE Part 1 Clerkship marks AOA Research Numeric Scale Graduate degree Publications OTO-HNS related Independent research (creativity)

23. Protocol 2 Personality NEO PI-R Best test of 5 Factor Model of Personality MSCEIT Emotional Intelligence Technical Mental Rotation Test SIM (e.g. M&T)

24. Protocol 3 For prospective study, a standardized letter of recommendation would be very advantageous

25. Conclusions Most of the criteria we currently rely on to select resident candidates are not valid We are proposing retrospective and prospective studies to quantify factors that may predict an applicant’s future success as a resident