1. Jonathan Fryer, Noreen Corcoran, Brian George, Ed Wang, Deb DaRosa Northwestern University Feinberg School of Medicine, Chicago Illinois

2. Screening of Applications Assessment of Candidates Ranking of Candidates

3.  1) How effective is the ranking process in selecting residents who will perform well?  2) How effective is the ranking process in predicting subsequent resident performances?  3) How effective are “adjustments” made to the preliminary rank list in improving the selection of residents who will perform well?

4.  General surgery categorical resident recruitment between 2002-2011 inclusive (n=46). ◦ 4 categorical residents 2002-2004. ◦ 5 categorical residents 2005-2011.  Residents who dropped out (n=1) or who were recruited after the PGY1 year (n=2) were excluded from the analyses.

5.  We compared how successful candidates were ranked during recruitment with their subsequent performance in our program.

6.  1) USMLE˄ Scores alone  2) Unadjusted Ranking Score (URS˄): based on sum of 3 assessment scores ◦ Academic Profile (Coordinators)  Medical school rank, USMLE Step 1, Class Rank, Honors in Surgery ◦ Program Director Review (PDs)  Research experience, extracurricular/community involvement, LORs, Personal Statement, Dean’s Letter ◦ Faculty Interview score (Faculty)  Averaged for 2 independent faculty interview scores. ˄ Higher score is better

7.  3) Final Adjusted Ranking (FAR˅) ◦ Modification of the preliminary rank list generated by the URS ◦ Based on additional insights about specific candidates provided by the resident selection committee and/or leadership  Endorsements from trusted colleagues  Negative interactions with staff  Concerns raised by residents, coordinators  Other? ˅ Lower is better

8. Screening Academic Profile * Assessment URS * Ranking FAR *USMLE included

9.  1) ABSITE˄ percentile alone  2) Resident Evaluation Grade (REG˄) ◦ Semiannual evaluation scores (Letter grade: A-F) ◦ Group discussion and grade assignment based on:  Clinical Evaluations (360°): faculty, peers, med students, nurses, patients, etc.  Compliance: evaluations, case log, duty hour log, conference attendance, etc.  ABSITE, Mock Oral, PAME scores. ˄ higher is better

10.  3) Independent Faculty Rating/Ranking (IFRR˅) ◦ Confidential survey with faculty independently rating all residents using a 7-point Likert scale and ranking resident within their PGY1 recruitment cohorts. ◦ * not part of standard resident evaluation at NU ˅ lower is better

11.  Full IRB approval was obtained.  All resident ranking and performance data was de-identified after collection and aggregated to protect resident confidentiality.

12.  Semiannual Resident Evaluation Grades (A-F) were converted to numerical values (5-0, respectively) and averaged for analyses.  Data from Individual Faculty Rating/Ranking surveys were averaged for individual residents.

13.  Statistical Analyses performed using SAS 9.2 software (Cary, NC). ◦ Associations between ranking and performance parameters were analyzed using Spearman’s correlation coefficient. ◦ Comparison of ranking parameters between poor and satisfactory performance used student t-test. ◦ Differences in performance based on ranking range were compared using F-test.

14. Poor Resident Performance Parameter # (%) comments Drop outs1 of 46 (2.2%)excellent evaluations ABS QE/CE fail0 of 13 (0%) REG < C *1 of 46 (2.2%)Probation IFFR > 4.06 of 46 (13.0%)>4.0 = below average ABSITE < 35% *12 of 46 (26.1%) * Occurring at any time during residency training

15. Recruitment Ranking USMLE˄URS˄FAR˅ Resident Performance R*p p p ABSITE (%tile)˄ 0.61<0.0001 0.060.69520.090.5891 PGY1 REG˄0.12<0.4087 0.400.0058 0.170.2597 Overall Grade˄0.16<0.2783 0.340.0219 0.160.2783 Independent Faculty Rating˅ 0.22<0.14090.020.9020-0.120.4245 * Spearman correlation coefficient ˄ higher is better ˅ lower is better

16. USMLE˄URS˄FAR˅ Poor Resident Performance Criteria MeanSDMeanSDMeanSD *REG ˄< C (n=1)239.00-111.00-27.00- ≥ C (n=46)238.4215.27110.446.0018.4811.83 p value0.80080.6041<0.0001 IFRR ˅ ≥ 4.0 (n=6)232.5017.44111.335.2013.149.12 < 4.0 (n=41)239.3314.76110.336.0819.5212.00 p value0.30730.70230.2199 *ABSITE ˄ % < 35 (n=12)228.9217.70110.255.9319.2513.59 ≥ 35 (n=35)243.4813.05110.326.3717.9511.95 p value0.00570.97520.7625 * Occurring at any time during the residency

17. USMLE˄URS˄FAR˅ Poor Resident Performance Criteria MeanSDMeanSDMeanSD *REG ˄< C (n=1)239.00-111.00-27.00- ≥ C (n=46)238.4215.27110.446.0018.4811.83 p value0.80080.6041<0.0001 IFRR ˅ ≥ 4.0 (n=6)232.5017.44111.335.2013.149.12 < 4.0 (n=41)239.3314.76110.336.0819.5212.00 p value0.30730.70230.2199 *ABSITE ˄ % < 35 (n=12)228.9217.70110.255.9319.2513.59 ≥ 35 (n=35)243.4813.05110.326.3717.9511.95 p value0.00570.97520.7625 * Occurring at any time during the residency

18. Performance criteria ABSITE (% tile) PGY1 REGOverall REGIFRR Rank range ( % tile)˅ MeanSDMeanSDMeanSDMeanSD <15 % (n=14)61.6928.824.570.554.500.443.100.78 15 -30% (n=17)65.5721.774.290.794.260.643.160.80 >30% (n=22)66.4323.734.000.764.070.563.220.68 P value (F-test)0.88310.11060.13380.9158

19. Performance criteria ABSITE˄PGY1 REG˄Overall REG˄ IFRR˅ Rank Range (% tile)˅ MeanSDMeanSDMeanSDMeanSD <15% (n=19)60.8426.404.630.624.420.503.130.83 15-30% (n=9)66.1722.263.810.663.920.613.520.65 >30% (n=22)70.0028.814.360.714.540.382.700.43 P value (F-test)0.6475 0.00250.00510.0156

20. Performance criteria ABSITE˄PGY1 REG˄Overall REG˄ IFRR˅ Rank Range (% tile)˅ MeanSDMeanSDMeanSDMeanSD <15% (n=19)60.8426.404.630.62 4.42 0.503.130.83 15-30% (n=9)66.1722.263.810.66 3.92 0.613.520.65 >30% (n=22)70.0028.814.360.71 4.54 0.382.700.43 P value (F-test)0.6475 0.00250.00510.0156

21.  Single center study  URS confounded by USMLE score  REG confounded by ABSITE  No formal faculty orientation for IFRR survey

22.  USMLE scores were predictive of subsequent ABSITE performance only.  Unadjusted Ranking Scores (URS) were predictive of subsequent performance based on resident evaluation grades (REGs), while Final Adjusted Rankings (FAR) were not.

23.  Our resident selection process has generally been successful in providing us with residents who perform well.  Our unadjusted ranking score appears to be a better predictor of subsequent resident performance than our final adjusted ranking… ….therefore caution should be exercised when considering adjustments to the preliminary rank list, as they may not engender selection of better performing residents.  Effectively defining a reliable rank list “cutoff”, beyond which performance will predictably decrease, may not be possible in our system.

27. ◦ Drop outs: 1 (after PGY1 despite excellent performance) ◦ ABS exams first try pass rate (n=13):  100% ◦ Ever with REG <C 100% (i.e. probation): 1 ◦ IFR > 4.0 (i.e. below average): 6 ◦ Ever with ABSITE scores < 35%tile ever: 12

29. USMLEURSFAR Poor Resident Performance Criteria MeanSDMeanSDMeanSD REG < C (n=1)239.00-111.00-27.96- ≥ C (n=46) 238.4215.27110.446.0018.4811.83 p value0.80080.6041 <0.000 1 IFR ≥ 4.0232.5017.44111.335.2013.149.12 < 4.0239.3314.76110.336.0819.5212.00 p value0.30730.70230.2199 ABSITE % < 35 228.9217.70110.255.9319.2513.59 ≥ 35 243.4813.05110.326.3717.9511.95 p value 0.0057 0.97520.7625

30.  Absolute ranking correlation with resident performance: ◦ Absite ◦ Semiannual evaluation grade ◦ Faculty survey rating

31.  Absolute ranking (AR): Ranking among entire candidate group (n= 60-80).  Relative ranking (RR): Ranking among cohort of successful PGY1 applicants (n=4 or 5).

32.  Within resident cohorts FAR did not correlate significantly with subsequent: ◦ ABSITE scores (r=0.22; p=0.1760) ◦ Semi-annual evaluation scores (r=0.20; p=0.1987) ◦ Faculty survey cohort rankings (r=0.23;0.1175)  Conversely, USMLE scores exhibited a significantly positive correlation with subsequent: ◦ ABSITE scores (r=0.46; p=0.0022), ◦ Semi-annual evaluation scores (r=0.41; p=0.0163) ◦ Faculty cohort rankings (r=0.35; p=0.163)

33.  Resident recruitment involves a formal evaluation of candidates where a variety of objective and subjective criteria are used to rank candidates from best to worst.  Preliminary rank lists are often subsequently “adjusted” based on additional insights about the candidates.

34. Screening of Applications Assessment and Preliminary Ranking of Candidates URS Final Ranking of Candidates FAR