1. Curriculum Change & Student Learning Outcomes: Internship Prof Tony Chung, Associate Dean (Clinical) Faculty of Medicine, Chinese University of Hong Kong Prof Tony Chung, Associate Dean (Clinical) Faculty of Medicine, Chinese University of Hong Kong

2. Research Objective Impact of curriculum changes in terms of student learning outcomes

3. Measured Outcomes Curriculum changes assessed by: 1.Summative assessments during clinical years 2.Surveys of final year students re: instructional quality, curriculum coverage & clinical skill development 3.Interns’ self-assessment of their generic skills 4.Internship performances as rated by clinical supervisors

4. Historical Controls Compared Interns from new curriculum (2006) with Interns from 1995 – 2005 (all interns from old curriculum) Interns from 2002 – 2005 (senior classmates trained under old curriculum)

5. Up to 11 Cohorts Trained Under Old Curriculum 1 Cohort Trained Under New Curriculum Clinical Supervisors Ratings of Interns Clinical Supervisors Ratings of Interns

6. Measured Outcomes Preceptor Ratings of 11 Intern Performance Criteria –Professional Knowledge –Clinical Skill –Clinical Judgment –Use of Medical Language –Attitude to Patients –Attitude to Medical Staff –Willingness to Learn –Organizational Ability –Attendance at Educational Activities –Communication Skill –Sense of Responsibility attributes given increased emphasis in new curriculum

7. Additional Measures Additional assessments from Internship - Content analyses of clinical supervisors’ comments compared 2003/04 - 2005/06 (old) & 2005/06 (new) - Frequency of medical incident reports related to interns compared 2005/06 (old) & 2006/07 (new)

8. Statistical Adjustments Covariates Type of hospital (Teaching, Teaching affiliated) Unit (Med, Surg, Paed, Psy, OBG, Ortho) Gender (Female / Male) Rotation (1 st to 4 th ) Initial graduating status (Pass/ Fail) Inflation in supervisor ratings over 11 yrs (all significant polynomial terms for trends)

9. Results

10. In terms of room for performance improvement in reference to all interns educated & trained in the old curriculum (1995 to 2001) improvements by new interns ranged from 2.3% to 39.5% across the 11 performance attributes If controls restricted to last 4 cohorts trained in the old curriculum improvements by new interns ranged from 2.7% to 51.8% across the 11 performance attributes Three attributes statistically significant after covariate adjustment (function of statistical power/sample size)

11. Curriculum Changes & Performances % Gained within Region of Performance Attribute Effect p that Could be Improved Given Previous Interns (1995 - 2005) (2001 - 2004) Attitude to medical staff+ 0.316 0.00739.5%51.8% Organization ability+ 0.259 0.01820.2%23.3% Communication skill+ 0.230 0.03720.1%23.7% Use of medical language+ 0.157 0.10313.1%15.1% Willingness to learn+ 0.112 0.358 12.5%16.2% Clinical judgment+ 0.151 0.13412.1%14.0% Attendance at educ. act.+ 0.132 0.21811.1%13.4% Attitude to patients+ 0.091 0.42110.4%13.2% Sense of responsibility+ 0.076 0.532 9.1%11.2% Professional knowledge+ 0.030 0.751 2.3% 2.7% Clinical skill+ 0.029 0.772 2.3% 2.7% Covariates: gender, hospital, unit, P/F status, rotation & polynomial terms for trends

12. Content Analysis of Supervisors’ Comments ~ 2/3’s of the supervisors include written comments compared cohorts from old curriculum, comments regarding the 2005/06 interns (new curriculum) were –more detailed (+10.5%) –more complimentary & more interns described as good/better

13. Rotation Curriculum None Poor Average Good 1 st New 26.8% 1.4% 4.2% 67.6% * Old 33.3% 1.4% 16.0% 49.3% 2 nd New 33.1% 0.7% 9.2% 57.0% * Old 35.4% 2.1% 27.8% 34.7% 3 rd New 37.3% 1.4% 7.0% 54.2% * Old 35.4% 3.5% 18.1% 43.1% 4 th New 42.3% 2.8% 10.6% 44.4% * Old 45.1% 6.9% 23.6% 24.3% N New = 142 * c 2 1st rotation = 14.86 (p=0.002); c 2 2nd rotation = 22.23 (p<0.001) N Old = 144 c 2 3rd rotation = 10.04 (p=0.018); c 2 4th rotation = 18.12 (p<0.001) Nature of Clinical Supervisors’ Comments Concerning CUHK Interns 2006/07 (New) : 2005/06 (Old)

14. Proportion of Interns Whose Performance was described as Good 2003/04 – 2005/06 (Old) vs 2006/07 (New) % No trend of increasingly positive comments across rotation & years. There is a statistically significant trend during 4 rotations of more new interns being described as good compared to any of the last three cohorts from the old curriculum.

15. Among Clinical Supervisors’ Comments: Proportion Indicative of good Intern performance 2005/06 vs 2006/07

16. Two Negative Comments “…potential to further improve his clinical skill and judgment; appears to be too passive; may need longer time to be accustomed to a new system/ environment; needs reinforcement in social interaction among colleague” “…complaint from patient, incident happened... in PWH. Tried to use the same syringe with normal saline to flush the heparin blocks of two separate patients. This was noted by the 2 nd patient and when patient refused to be treated with the same syringe, Dr. xxx refused to flush the patient's heparin block. Dr. xxx was counseled by Risk Management Team of the Prince of Wales Hospital”

17. Example Comments for ‘Average’ “conscientious young doctor, willing to learn” “quiet, pleasant and organised” “satisfactory performance” “satisfactory performance, MRCOG part I examination” “Quiet, and gentle. Willing to learn”

18. Example Comments for ‘Good’ “Strong sense of responsibility; professional knowledge, skills and clinical judgment are excellent for her seniority; humble, quick and eager to learn. Overall excellent performance” “Excellent attitude to patients; appreciation letters received from patients she cared for” “Demonstrated impressive clinical acuity in psychiatry, astute and logical clinician, sensitive intuition, appropriate empathy and scientific objectivity; identified relevant research questions in his daily clinical encounters”

19. “Did a fantastic job at her first rotation, keen to learn and willing to take responsibility, no reservation in recommending her to extend her future training in oncology and in medicine” “Particularly appreciated the clinical records and documentation written by Dr. xxx, which were very neat, well organized and comprehensive” “One of the very best interns we ever had. He has sound general medical knowledge and can practically apply this to patients; hard working, reliable & totally committed to his work; well liked by his colleagues. He is someone with great potential!!” Example Comments for Good (cont’d)

20. 3 rd Outcome from Supervisors: Medical Incident Reports 11 hospitals, 58 specialty units & 4 rotations in 2003-6 (old curriculum) –Average 20 interns had at least one medical incident report –Average 26 medical incident reports were filed In 2006/07 (new curriculum) –10 interns had at least one medical incident report (50% reduction) –14 medical incidents were filed (46 % reduction) Frequency of medical incident reports significantly less

21. Discussion

22. Interns’ performance as a measure Interns assessed since 1995 –many attributes matched additional learning goals emphasized in new curriculum –comparisons during medical school limited to assessments used in old & new curriculum –usually did not include the generic & attitudinal measures Third party assessment (purchaser) –less potential for bias Assessments related to professional work as a front-line doctor –not restricted to within-school clinical tasks

23. Limitations Historical controls Blinding not uniform across clinical supervisors Sample size of intervention group small (i.e., one year of interns) 1 st cohort of a new curriculum often does better than subsequent cohorts

24. Conclusion Little evidence to support knowledge of the new cohort has significantly changed as seen in exam results comparing cohorts in years 4 & 5 Preliminary evidence supports new graduates have some enhanced generic skills and professionalism (a goal of the new curriculum)

25. Conclusion Importance of Establishing Evidence Ongoing monitoring and evaluation Draw firmer conclusions when more new graduates assessed