Measuring Non-technical Aspects of Surgical Clinician Development in an Otolaryngology Residency Training Program Jennifer J. Shin, M.D. S.M. Michael J. Cunningham, M.D. Kevin G. Emerick, M.D. Stacey T. Gray, M.D. Society for University Otolaryngologists November 13, 2015

The Constitution of a Surgeon It takes five years to learn how to operate and twenty years to learn when and when not to. -Anonymous

The Constitution of a Surgeon It takes five years to learn how to operate and twenty years to learn when and when not to. -Anonymous I would like to see the day when somebody would be appointed surgeon somewhere who had no hands, for the operative part is the least part of the work. -Harvey Williams Cushing, Letter to Dr Henry Christian, Quoted in 'The Best Hope of All', Time (3 May 1963).

Validated Instruments …Clinical Outcomes …Evidence-based Practice

Validated Instruments …Educational Outcomes …Evidence-based Residency NoviceIntermediateCompetent Instrument to measure the development of clinical practice ability?

Instrument Design  Rapid/real time evaluation  Insight into each trainee’s thought process  Concrete feedback and data compilation  Benefits beyond the learning/measurement itself

Instrument Administration Mock oral board exam  Case criteria:  1) Single patient with a single main diagnostic issue.  2) Case culminates in an intervention, with one resulting complication  3) Enough detail available so that a full history, physical examination, and complement of diagnostic testing may be obtained -- actual radiologic images, lab values, audiometric studies, or other diagnostic test results are available

Instrument Development  Iterative process:  22 successive drafts  311 candidate questions  Face validity evaluation:  Residency program director  Division chairman  Professionalism director  Resident education curriculum supervisor  Epidemiologist/Instrument validation scientist  Validation phase:  Instrument assessment

Inter-rater Reliability Cohen Kappa 0.66 (SE 0.03) 72.5% agreement Internal Consistency Cronbach alpha >0.87 Responsive to Change

OBJECTIVES To utilize the clinical practice instrument (CPI) to measure non-technical diagnostic and management skills during otolaryngology residency training To determine whether there is demonstrable change in these skills between PGY-2, 4, and 5 residents To evaluate whether results vary according to subspecialty topic or method of administration.

METHODS Prospective study of an otolaryngology residency training program Institutional review board approved n=248 evaluations of 45 otolaryngology resident trainees at regular intervals Analysis of variance with nesting and post-estimation pairwise comparisons to evaluate total and domain scores according to training level, subspecialty topic, and method of administration Examination preparation and security

PGY-4 & 5

RESULTS Total scores were significantly different among PGY-levels of training, with lower scores seen in the PGY-2 level compared to the PGY-4 or PGY-5 level (p<0.0001). PGY Level nMean Standard Deviation MinimumMaximum 25 th Percentile 75 th Percentile All

RESULTS

Residents scored higher in general otolaryngology than in the subspecialties (ANOVA, p<0.003). There was a significant difference in between general otolaryngology and pediatric otolaryngology (p<0.0001), and between general otolaryngology and head and neck surgery (post- ANOVA pairwise comparison, p<0.0033) RESULTS

Administering the examination with an electronic scoring system, rather than a paper-based scoring system, did not affect these results. The calendar year of administration did not affect these results. RESULTS

CONCLUSIONS Standardized interval evaluation with the CPI demonstrates improvement in qualitative diagnostic and management capabilities as PGY-levels advance. Administration of the CPI has been formally incorporated into the Harvard otolaryngology residency curriculum. The CPI can potentially be adapted for use in any otolaryngology training program, and is potentially applicable to any surgical specialty.

FUTURE DIRECTIONS Short-term educational missions… Milestones… Electronic case library… Program update follow up? Benchmarks? Other sites?

Thank you